When is a food considered a delicacy?

who determines that an item should be classified as this

You do.

There is no official list of delicacies. No authority on deciding what food is tasty. If, to you, lavender chocolate is a delicacy, then you can say so. Others will say that it isn't, that it tastes like soap and, and they can't bring themselves to eat it. Saying "I like bacon" is just a personal statement about your preferences. Saying "Bacon is a delicacy" is a personal statement which carries a bit more information, namely that you find bacon tasty, but also consider it a luxury food item, as opposed to an everyday food (although this second semantic distinction is somewhat lost nowadays).

As tastes tend to be shared within a culture, you will probably find that the people around you tend to agree somewhat on what is a delicacy, although it's not a complete overlap. This can appear as if there a universal agreement on what is a delicacy. And I don't doubt that somebody, somewhere has compiled something they call a list of delicacies, the same way that a movie fan will compile a list of "really funny comedies". Both have equally little validity - what is funny to that person will not always be funny to you, even though there will be a correlation if you grew up in the same culture.

I found your question very interesting as, even though we know there are delicacies, I never thought about how something came to be considered a delicacy.

The following excerpt from an article in psmag.com sheds a little light on the subject. The article in its entirety is very interesting and demonstrates how the same thing may or may not be a delicacy today and the opposite tomorrow.

Cultural influences lead to the difference in the habitual consumption of certain foods and in traditions of preparation, and in certain cases can lead to restrictions such as exclusion of meat and milk from the diet. Cultural influences are however amenable to change: when moving to a new country individuals often adopt particular food habits of the local culture.

Social influences on food intake refer to the impact that one or more persons have on the eating behavior of others, either direct (buying food) or indirect (learn from peer’s behavior), either conscious (transfer of beliefs) or subconscious. Even when eating alone, food choice is influenced by social factors because attitudes and habits develop through the interaction with others. However, quantifying the social influences on food intake is difficult because the influences that people have on the eating behavior of others are not limited to one type and people are not necessarily aware of the social influences that are exerted on their eating behavior.

Delicacy's are traditionally based on food availability and cultural and personal taste which is also related to geography and culture. However modern refrigeration and transportation has made food from all over the world much more available then it once was. So delicacies have become much more personal then cultural as one persons stable diet of chocolate will be another persons delicacy.

Cultural delicacies are often in abundance around important cultural events like Christmas. I remember the first time I had turkey and cranberry sauce when it was not Christmas. It was incredible and to me because for me that was a delicacy only available at Christmas. And then of course one cultures delicacies like triple smoked crispy fried bacon is another cultures garbage.

So delicacies are influenced by personal taste and experience to cultural ritual.