Wikipedia lists a few substitutes, some of which are fairly readily available. These may be based on roasted grains (like barleycup), chicory (though this is often combined with coffee), or both.

There are several processes for decaffeinating coffee. Not all rely on the organic solvents you're trying to avoid. The Swiss water process avoids then completely and is used commercially so you should be able to order some. CO₂ decaffeinated coffee is rarer.

Roasted barley tea

Mugi-cha, or roasted barley tea, is my favorite substitute because it also has a roasted, slightly bitter taste. It is commonly consumed in China, Korea, and Japan and is widely available elsewhere as well. There are also roasted barley and chicory blends marketed specifically as "coffee substitute."

Strictly speaking, caffeine is a poison that certain plants produce to ward off insects and other animals. Other popular examples of chemical defences that humans like to consume are nicotine and capsaicin.

I've noticed that there are certain teas that are naturally caffeine free

Tea - in contrast to cofee - can be made of any number of different plants like various fruits, herbs, tree bark (cinnamon), roots and leaves of different species. The traditional black, green, yellow and white teas all contain leaves of Camellia sinensis, the tea plant. Since this plant uses caffeine as a chemical defence like the coffee plant, there are no naturally caffeine free teas of it. The caffeine free teas you mean are all made of different plants like peppermint, rosehip or hibiscus, to name just a few.

Removing caffeine from plant material is just as much a chemical process as the (natural) synthesis that creates caffeine in a plant. Since millions of people around the world drink decaf each day, you can be reasonably sure not to be poisoned. The taste, however, will differ slightly from unprocessed coffee.

Decaffeinated coffee is generally produced using organic solvents. However, it isn't produced using harmful (carcinogenic) benzene anymore, so stories you might have read about how poisonous decaf coffee is are probably wrong. While not something you would want to consume in a high concentration, the solvents used today are harmless at the tiny concentrations that remain in the coffee after decaffeination.

Not all decaffeinated coffee is produced with organic solvents though, so you can avoid them entirely if you want to. Look for decaffeinated coffee which has been produced using the "Swiss water process" or "mountain water process", which only use water to draw out the caffeine. There is also decaf coffee which is produced using a carbon dioxide process, but that appears to be quite rare as it is a more expensive method.