Do certain spices really go bad? or will people without refined palates usually not tell the difference, specifically chili powder?

tl;dr : it's complicated.

If you're relying on cooking by blindly following recipes, and hoping that they come out the same every time, then yes, you probably want to replace old spices. So professionals are going to give that advice, as it's easier for every chef in the restaurant to be able to re-create a given dish, or for home chefs to have a chance for the recipe to come out like the recipe writer intended.

But the thing is -- even different brands have different potency.

You may know how long it's sat on the shelf since you bought it -- but how long was it at the store? How long was the jar sitting in a warehouse before it was sent to the store? How long were the whole spices sitting around before they were ground and put into the jar?

And what type of jar, even? A glass jar is going to hold in essential oils better than thin plastic bag or a plastic jar. And an almost empty jar is going to have more air for the essential oils to mix with and disperse each time you open it vs. a mostly full jar.

Really, you need to smell and/or taste your spices. Don't just throw them away because you've had them for a year -- they might be fresher than the ones that you replace them with if you buy it from a store that doesn't have good turn over.

If your spices seem to have no scent, then you can try using more to try to get the flavors back in balance. Although beware, because sometimes an herb or spice might lose one chemical (that's more volatile), but still have others -- so if you get to the point where you're more than doubling it, you might start to notice off flavors (because of other less volatile chemicals that are now in a higher concentration)

If it has an off scent (I had it happen with za'atar blend, as it has oil and sesame seeds in it) or seems weird for some other reason (like absorbing too much moisture from the air and clumping), then you should consider replacing it.

I'm not sure where I'd put color changes -- I'd probably smell and/or taste it and see if that qualifies as 'losing potency' or 'has gone off'.

And sometimes, as was your case with the paprika that you thought was too strong, sometimes people prefer the older stuff. I remember someone telling me that he had a mustard that he really liked, but he got more, and it was insanely hot. (because he had bought the first one on sale, as they were trying to clear out old stock).

So what matters is -- did you like it? If so, it's fine.

In general you can expect better results from fresher ingredients and spices. In my favorite Pakistani restaurant they grind their spices on a daily basis and it really makes a difference. Even without a refined palate you will be easily able to taste a considerable difference between freshly ground pepper and pre-ground pepper that has been sitting on the shelf for more than a year.

So if you are aiming for good results and your current stock is several years old my recommendation would be to get fresh supply. For me this would not necessarily mean to dump the old stock as there are some kinds of recipes (e.g. spreads) where I consider this subpar quality acceptable.

I think that definition of 'good' is 'it won't kill you'.
I wouldn't even keep supermarket spices 2 to 4 years, let alone the good stuff.

By extrapolation [though I haven't used a pre-mixed 'chilli con carne' mix in decades] anything you get from the supermarket is going to be half-dead before you open it.

Simple cayenne powder doesn't have much aroma anyway, so you can keep that long after the initial bloom of it has gone [though you want some better grade New Mexico in a decent chilli], but such as cumin [vital in "chilli con carne"] I find if I buy it in a supermarket all the high aromatics are already gone, compared to my usual online supplier.
If I put that in a similar jar to the supermarket stuff*, I reckon I've 3 to 4 months before it gets as lacklustre.

After that 3 months or so it takes for the higher aromatic oils to evaporate off the good stuff, they then tend to last about the same length of time. I wouldn't keep a high aromatic such as cumin any longer than 6 months. It's pretty much dead by then [though it still won't kill you, it's just lost the ability to make great chilli.]

Don't get me wrong - I do have things in the cupboard over a year or so old & do use them… until I get to the point I eventually realise I'm just not enjoying what I make with it as much as I did. Then I have a clear out & replace all the things I should have replaced last year.
I also used to think supermarket herbs & spices were exactly what those herbs & spices should taste like. They're not. I swapped to an online 'fresh' supplier & my eyes [& nostrils & tastebuds] were finally opened.

*I do sometimes have both 'the good stuff' & supermarket in quick succession. My online guys charge quite a bit for postage so if I only need one new spice I get it from the supermarket. I get the 'good stuff' every 6 months or so when I do a clearout & restock. Some I just let run out & don't replace until I've got a large enough order to be worth the postage. [I'm a bit of an 'ingredients freak' so I have two entire cupboards full of individual herbs & spices rather than ever buy any mix.] I'm the kind of guy who has both Mexican and European Oregano in the cupboard & knows why ;))

Oh… btw, a 'chilli con carne trick' I learned a while back is that a lot of the supermarket blends use masa harina to add a bit of 'extra Mexico'… whether or not you'd have put it in that particular dish if making it from a trad recipe. It took me years to figure that out, because masa harina is just not something you ever find in a UK supermarket to be able to recognise the smell. My online guys sorted me out on that one too. Bonus… home made corn tortillas are absolutely nothing like any you can buy in a supermarket. They have flavour & aroma.

This answer suppliments Joe's, specifically for chili powder.

A good chili powder* has three groups of flavors: heat from the capsaicin, fruity flavors from the pepper, and smoky flavors from toasting, drying, and/or smoking.

Over time and air exposure, both the fruity and smoky flavors fade, leaving only the capsaicin. So the old chili powder will still be "hot" but will lack other flavors, and thus deprive chili made from it of depth and richness. While less experienced chili-eaters won't know what's missing, they will know something is. The statement from Kate's Conscious Kitchen is correct.

How long that takes depends multiple factors, including storage conditions (particularly exposure to air and sun), type of chili, how it was originally dried/smoked and ground, humidity and temperature. In a worst case -- openly exposed to air in a hot humid place with some direct sun -- chili powder could go "stale" in a week. In a more normal case, a closed spice bottle out of the sun stored at around 21C and infrequently opened, I've found it's more like a year to eighteen months. Vacuum-packed, and kept in the dark at 15C, chili powder could be good for 2-5 years in my personal experience.

But like Joe said in his answer: there's absolutely no substitute for tasting it to see if it's stale.

(* by "chili powder" here I am referring to spices that are just ground, dried chili peppers. There's also spice mixes sold as "chili powder" that contain additional spices, such as cumin and oregano. Those would also suffer from the fading of those additional spices, and as a result could lose flavor even faster)

I won't say that the spices spoil, but I do think spices can go stale. It is like bread there is nothing wrong with yesterday's bread, but there is magic to a bun fresh out of the oven. Some spices give oil and fragrance of when crushed freshly, there is a remarkable difference in powder peppercorns and freshly grinded pepper. That is why tempering of spices is such an integral part of Indian cuisine, it gets the best out of them.

As for powdered chilli that is the dried fruit of a chilli plant that is crushed into a powder. It can go stale but that is not anything 30 seconds in a warm and dry frying pan cannot cure.