Is there a rule of thumb when replacing flour with whipped egg whites in a cake/muffin recipe?

While I suppose this is technically possible, I would never consider trying to make this substitution.

Muffins are are form of quick bread. Their structure is based on gelatinized starch from the flour, leavened by chemical leavers such as baking powder or baking soda.

In chiffon cakes, sponge cakes, and angel food cakes, the whipped egg whites or meringue form the primary basis of the structure as a foam with mechanically beaten in air. These batters are much much thinner than quick bread batters, generally, so that the other ingredients can be folded with the egg whites while maintaining the foam.

Instead, I suggest you take a different approach:

• Find a better quick bread recipe--there are many, many good banana muffin recipes easily searchable. If you don't find a muffin recipe you like, you can always use any banana bread recipe for muffins as well, since quick breads and muffins are the same thing, just baked in different shapes.
• Find a recipe for a banana sponge cake or banana chiffon cake. You can always bake these recipes in cupcake form if you like.

Finding a good coherent recipe will almost certainly by easier than trying to adapt a muffin recipe you are not happy with into something else.

If you are really wanting to transform recipes more generally, you need to learn the basis on which they work. You might want to read Michael Ruhlman's Ratio, as a starting point--however, the ratios of ingredients are only part of the story. The other part of the story is the method or technique by which they are prepared.

In the world of cakes and muffins and quick breads, these include the muffin method, the creaming method, the reverse creaming method from Rose Levy Berenbaum, and--I don't know a general name for this method--but the whipped egg whites folded with other ingredients technique.

Each of these methods works optimally with different ratios of ingredients, but also achieves different outcomes with the same ingredients.

So you would have to analyze the individual recipe, adjust the ratio of ingredients, and possibly modify the method by which the ingredients are prepared or combined.

Frankly, to lighten a muffin recipe, I would first ask: are you measuring correctly? Too much flour is going to lead to a dense outcome, and flour is very easy to mismeasure. I would suggest always baking by weight, not volume when you can. Reputable recipe sites for baked goods will tell you the weight of flour in addition to the volume measurement, or at least tell you what the standard weight of a cup of flour is. If no such guide is provided, I find that using a 4.5 oz (US) standard cup for flour is pretty generally successful.

My second approach would be to very, very slightly increase the amount of baking powder, maybe by 1/4 tsp increments in two different trials.

If that fails, you might try reducing the flour by 25% or 50%, but not increasing the number of eggs; instead, change your method from the muffin method to the whipped egg white method, separating the eggs, whipping the egg whites, and then folding them into the other wet ingredients, then gently folding in the dry ingredients.

Still, when you do these kinds of changes, you need to test your recipes, keep careful notes, and so on. I still would start by researching already tested recipes that achieve the outcome you desire.