When is a cooking oil not appropriate to substitute for another?

Actually, there are really only a few oils you can substitute for each other, at least without any significant side effects.

The oils which generally are used interchangeably are peanut oil, canola/rapeseed oil, and sunflower oil. These oils have similar smoke points, don't impart any really noticeable flavour, and tend to be used primarily for high-heat cooking (pan-frying, deep-frying), so if you're paranoid about saturated fat for instance, you can substitute sunflower oil for peanut oil. Corn oil is in the same group, but I rarely see that used anymore. You can also use the "light" olive oil, but that will change the flavour of the dish. I believe walnut oil has similar properties, but it's considerably harder to find.

But keep in mind that oils are used for far more than frying. Many have highly-specialized uses:

• Extra virgin olive oil is most commonly used in sauces and salad dressings ("oil and vinegar" almost always means olive oil, there really is no substitute);

• Chili oil is really more of a condiment than a cooking oil. Even if you could cook with it, the result would be inedible due to the heat.

• Toasted sesame oil is used as a flavourings in Asian dishes. It's useless as a cooking oil (and cooking with it would be a terrible waste). Regular sesame oil, on the other hand, is often bought in a refined form and is generally used as a cooking oil.

• There are a lot of other more esoteric types of oil such as palm oil and coconut oil, which you really don't want to use unless you know what you're doing (you can ruin the flavour).

I could go on, but for now I'll refer you to the Types of oils and their characteristics as a starting point. Cooking oils really aren't freely interchangeable in all situations; even if you've accounted for smoke point and flavour, sometimes a significantly different fat content (i.e. grapeseed oil which is mostly polyunsaturated vs. canola oil which is mostly monounsaturated) can seriously mess up a delicate recipe.

It's better to be asking which oils you can substitute in a specific situation than to assume everything goes and list the "exceptional" circumstances.

It's very important to consider smoke point. If you're using it in a high heat application, make sure that the oil you choose won't burn. Besides that, make sure the people eating your food won't be allergic to the oil you choose, for example peanut.

I'd also like to point out that you can often but not always substitute clarified butter or rendered animal fats (bacon! duck!) as an alternative to boring neutral oils! They can't take quite as high a temperature as canola oil or peanut oil, they tend to go rancid more quickly, and they don't work in some applications, like mayonnaise. But damn they taste good...

There are definitely cases where substituting oils is NOT appropriate. For example, I wouldn't use olive oil for making pancakes, as olive oil impart a specific flavor. There are many other examples of flavorful oils, such as peanut.