What makes cake a Sponge Cake? And what doesn't?

You are asking for a firm definition where none exists.

Baking is not mathematics, and it does not have a heap of literature where everybody has formally agreed to use the same terminology. Thus classification of baking goods is done into intuitive categories. This means it doesn't work by strictly defining where the border of a category lies (everything that has feature X is a member of the category "sponge cake", and everything that doesn't have feature X is not a member). Rather, it works by defining a center of the category, or a prototype for what a sponge cake should be like. The more a cake reminds one of the prototypical sponge cake, the more likely it is that this person will consider it to belong into the category of sponge cakes. And through social interaction, people will partly synchronize their ideas of what is considered "prototypical", and how close is "close enough" to still be considered part of the category.

WIth that in mind, the best answer that can be given is to describe what a prototypical sponge cake is like. It has a specific texture, not that much different from other things called a "sponge", such as contemporary plastic sponges for dishwashing. It is basically a firmed-up foam, with many small, regular holes inside. It has a characteristic elasticity - not as brittle as a meringue, but not as soft as a marshmallow, it is rather in the middle of these two cooking extremes. It also has a characteristic type of being "moist" - it is not as dry as a baguette, but also not super moist like a brownie. It is chemically leavened, somewhat sweet, and vanilla flavored. (While there are variants like "chocolate sponge", they are not the category prototype). It is not noticeably oily in the mouth the way a chiffon cake can be.

There is a chef named Ruhlmann who has tried to standardize some recipe types by giving a basic formula for creating a prototypical dish of each type. The book is called "Ratio", because the formulas are represented as the ratio of the ingredients used. For sponge cake, he gives a ratio of 1:1:1:1 flour:eggs:butter:sugar, and he requires that the cake uses the creaming method. A cake created with the same ratio, but using the muffin method, is given in the book as "pound cake".

Following this ratio indeed produces a cake whose qualities are very close to what most people would call a prototypical sponge cake (with some cultural differences - I think that traditionally German and Austrian "bisquitboeden" are ligher on fat but else they are used interchangeably with Anglosaxon sponges). Making his pound cake (by mixing the same ingredients without creaming) results in a cake with a different, heavier texture with less regular holes. However, this does not preclude other recipes from creating a cake with a typical "sponge" texture without making use of a creaming method, or even from people finding the difference so insignificant that they would consider Ruhlman's pound cake to be a sponge cake too.

So, it is not about any specific method, or ingredient, or ratio. It is about whether people who eat it will agree that they associate it with the term, based on whatever triggers their feeling of similarity.

As an English person who moved to the USA as an adult. I thought I might give my side of what I have seen here to explain the difference in wording. Firstly I would highly recommend reading the Wikipedia article on this as it includes a discussion of why "pound cakes" and other similar cakes are considered sponge cake in the UK but "foam cakes" are in fact not, while the situation is largely reversed in the USA.

British Cakes

In my experience in the UK, cakes encompass a similar but significantly different set of foods then they do in the USA. These include far denser baked goods than would normally be considered cakey in the USA. This includes rich fruit cake, eccles cake, welsh cake and various other heavy, rich or chewy baked goods. In general it feels to me that cake is more about the role the cake fills in a meal than it's specific texture. Sponge cakes are light and spongey compared to most of the other things called cakes so the term is still appropriate, and helps distinguish these lighter cakes from other cakes.

American Cakes

Since coming to the US I have noticed that the range of goods people assume to be cake tends to be much more on the lighter fluffy side, including angel food cake, chiffon cake, but also includes things like funnel cake, which does not seem intuitively cakey to my British palette. It feels like cake in the USA depends much more on the texture, with most cakes being much lighter, fluffier affairs, with denser or more chewy cakes being the exception that requires specific calling out.

Sponge Cake

So, down to the actual question. The British definition of sponge cake, I think comes down to the typical style of the lighter end of cakes in the UK, which are mostly pound cake like, cakes which one would often eat with tea or coffee, including Victoria sponge, Batternberg cake, etc. The style of very light fluffy cakes common in the USA seems to be a much more modern trend in the UK so are not what one would jump to when thinking of sponge.

Conversely sponge cake in the USA occupies a different space, though shares the fact it describes cakes on the lighter, fluffier end of the spectrum of typical cakes in the USA. Comparatively pound cake and other batter cakes are actually fairly heavy on the spectrum of common cakes in the USA (excluding celebration, wedding and birthday cakes). I also noted that batter style cakes seem somewhat less common in general in the US, and are often seen as antiquated or old fashioned.

TLDR

In summary, I believe the term "Sponge Cake" has entirely different origins in the UK and USA, with the definition being just similar enough to be confusing. The spectrum of cakes available is generally very different between the two countries as well as what is a cake at all.

Basically, to make things simple, consider "American Sponge Cake" and "British Sponge Cake" to have different meanings, and avoid conflating the two, as trying to combine the two definitions results in a helplessly vague term with little agreement on what it means.

General Note

Very many terms in British and American cooking vary, dumplings, biscuits and many other things with the same name are totally different foods, roasting, baking, grilling and broiling seem to have significantly different meanings. In essence try not to assume a food, cooking method or other cooking related term means the same thing in a UK or USA context. This is very often mistaken and in fact the similarity of language and words is often a false friend, misleading one to assume the same words likely have the same meaning where they do not.

@rumtscho has a great answer for what a sponge cake actually, technically is, but I don't think they've answered your underlying question.

As I understand it, "sponge" is British-English (slang?) for any kind of cake, whether the cake in question is technically a sponge cake or not. I've also watched a lot of Bake Off, and you rarely even hear them use the word "cake"-- everything is "sponge."

So in the context of watching the show, when you ask

what's the difference between cake and sponge, and how can I tell?

the answer is

they're the same thing.

Sponge cake is the "raw" cake without any other flavors or icing or any decorations. Baking cake is always made of vanilla and chocolate flavor. Apart from this 'Cake' is everything you can think of.