Why is lye used in some recipes for cookies?

First of all, lye is not "water boiled with ash". You might be thinking of potash, which used to be used as lye, but virtually all food-grade lye today is sodium hydroxide.

In terms of its function, it largely raises the alkalinity (pH). Baking soda does too, but sodium hydroxide is far more potent - let's just say you don't want to get any on your hands by accident.

The reason some recipes use lye is that alkalinity is a major promoter of the Maillard reaction, which is responsible for the crisping and browning you see in many baked goods - most notably pretzels. I haven't seen cookie recipes using it but I imagine that those recipes would produce very crisp and hard cookies - not my cup of tea but maybe that's what you want.

The only ingredient I know of that you can use instead is baking soda, but I wouldn't call that a replacement, more like a substitution. It will work the same way, but in the end you'll have something much softer and less browned.

There is actually another substance that is in between the pH of sodium bicarbonate (baking soda) and sodium hydroxide (lye), and that is sodium carbonate AKA "washing soda". It's listed as a food additive (E500) but it is rather difficult to find in food-grade forms. As its name implies, it's more commonly used as a detergent. However, it would be a better substitute than baking soda due to its higher pH.

One can make food grade sodium carbonate (Na₂CO₃) from food grade baking soda, sodium bicarbonate (NaHCO₃), by baking it an oven spread out on a cookie sheet at 350° F for one hour; baking longer won't hurt. This drives off water and carbon dioxide to form food grade washing soda (sodium carbonate) which increases the pH more than baking soda.

Here's the reaction for the above process:

2NaHCO₃ ---> Na₂CO₃ + CO₂ + H₂O

Alkaline solutions are used in different qualities in doughs. I am afraid that you mix something up here, so I don't know which you mean.

One use is to enhance the Maillard reaction, which Aaronut already described. This is indeed done with lye. But nobody is incorporating lye into the dough (this would be quite dangerous). Instead, the formed pieces are immersed in boiling lye before being baked. This gives them a very intensive brown crust, but it isn't crispy, it is soft. The texture is like glossy magazine paper. It also gives a subtle taste of its own. I have only encountered this used with yeast dough, like pretzels or breadrolls. I haven't seen it used with cookies. I think that this is what Aaronut's answer refers to. As you don't have lye at home, it is acceptable to immerse your pretzels into a very concentrated solution of baking soda. The pictures look similar, I've never done it myself.

The second thing you could mean: as Aaronut pointed out, ash boiled with water and filtered isn't lye, it is potash. It isn't used for the Maillard reaction, or not primarily for that. A very refined version of it is incorporated into the dough itself, including cookie dough, as a primitive leavening method. Because of its alkalinity, it reacts with sour elements of the dough. But this is quite a crude method (you never know the concentration, plus, you risk off-tastes and maybe coloring from whatever parts of the ash didn't get filtered well enough). So it fell in disuse since pure alkaline salts (like baking soda and amonium bicarbonate) became available. If you really have a recipe which requires it, it is probably from before the 19. century. And given the progress of cookie recipes since then, I'd just use a modern recipe and forget the potash. It is easier than figuring out the correct substitution ratio. Unless you want to recreate the taste as it used to be, but then no substitution would to, you'll have to secure the real thing from somewhere (probably untreated wood in your own fireplace + ancient filtering methods).

Another reason for strong alkalis in cooking is to quickly breaks down the flour gluten, instead of having the dough sit around for a long time to 'soften' it

This is used in hand pulled noodles and in cookie dough that is extruded or piped

Common bottled cooking Lye water in supermarkets is mostly potassium carbonate and some sodium biphosphate. In powder form it is referred to as ???? (Kansui powder). For noodles they typically use a 1:200 to 1:100 ratio

When using Lye make sure it doesn't get in direct contact with fats or oils or you will have a soapy taste. Mix it with water and other ingredients first, then add fats or oils

As an alternative you can buy the "Natron" mineral. It's mined in Egypt and many other parts of the world, including the UK. It has been used as a food additive for centuries. Sold like packs of salt, contains around 80% sodium carbonate

To further answer the part of your question about replacing lye, you might also be able to use culinary lime from Anson Mills (and apparently some Walmarts according to the article). I'm pretty sure it is made of calcium hydroxide, also known as "cal" to Mexicans.

Concerning the purpose of lye in your recipe, in addition to the affect it would have on the dough (as already explained by others), it may also have a nutritional purpose.

According to an article on wikipedia about Hominy, Cherokees made hominy grits by soaking corn in a weak lye solution obtained by leaching hardwood ash with water

When combined with unmilled corn, it makes niacin (a B vitamin) and possibly other vitamins more available to the body. Although wheat flour, especially whole wheat, contains some niacin, I don't know if the nutritional benefits would translate, especially if it has already been milled into flour.

Could you please let us know what the instructions were? Curious to know if it goes into the dough (as perhaps a softener) or on the outside (to make it more crisp) and if it is a traditional recipe referring to wood ash lye or something more recent.

For moon cakes, lye water helps keep the baked dough soft and make it brown nicely when baked.