What strength of brine will completely inhibit growth of lactobacillus (and friends)?

In brief, your question has no possible general answer for the kind of scenario you posit (where you add a certain amount of salt to a certain volume of food) or even a scenario where you add a brine of concentration X to a certain amount of food.

Most vegetable (and animal) sources for food contain significant amounts of water, and some of that water will be released when placed into a brine solution. The amount of water that is released will alter the final brine concentration, often significantly. Reputable recipes for food preservation and brining will take into account the average amount of water released to arrive at a final brine concentration that will ensure food safety (while not making the food unpalatable and perhaps allowing bacteria to grow if fermentation is desired). The safety of such recipes is generally determined empirically through extensive testing to ensure the final products are stable. There's no general formula you can use.

Now, if you actually have a way of determining the final brine concentration after the food has released all of its water, you can answer your question for various bacteria strains. This paper has a summary of minimum levels of water activity (in Table 5) that will allow bacteria to grow. For Lactobacillus, using Table 4 in the paper, you can see that the maximum salt concentration which will allow growth could be anywhere from 7% to 16.5%, depending on the particular strain of Lactobacillus.

However, note that these are absolute maximum salt concentrations that can allow any growth. At high concentrations, the activity of Lactobacillus and other fermentation bacteria will slow significantly. By the time you get above a 10% solution or so, you're going to see very slow fermentation, and it is possible that other bacteria, yeast, and mold that can survive such high salt concentrations might grow first. (Lactic acid fermentation works by using both salt and resulting acidity as preservatives; if the lactic acid bacteria cannot grow fast enough, the lower acidity may allow other bad things to grow.)

Regarding your specific situation:

There are a number of things which are unclear here. First, cabbage generally is not fermented with any significant amount of brine. It is generally dry salted, and the water which comes out of the cabbage serves as the brine. Brine is generally only added if needed to keep the cabbage submerged. (In my experience, this is almost never necessary assuming firm packing of the cabbage.)

So, I'm a bit confused by the idea of a recipe which said to ferment cabbage with 3.5% brine by weight. Most recipes will tell you an amount of salt to add per weight of cabbage. As mentioned above, reputable recipes (tested for food safety in preservation) make assumptions about the amount of water that comes out of different fruits and vegetables when put in a salt solution. I'm also not even sure how you calculated "the correct amount of salt for a 3.5% brine that completely filled both jars" -- do you mean that you calculated the amount of salt to add to a weight of water that filled both jars?

In any case, this is NOT the right way to determine salt concentration for fermentation. Brine percentages should only be used when they are actually referring to brine. You generally cannot take a brine percentage and apply it to a volume that will primarily be filled up with something else. Yes, most fruits and vegetables are mostly water, but the amount of water that actually comes out of the food can significantly vary and will alter brine strength, which can change fermentation and safety.

Here's an example of a tested recipe put out by a reputable food preservation source. It recommends 3 tablespoons of pickling salt per 5 pounds of cabbage. That comes out to roughly 2.5% salt by weight of cabbage. That's a pretty typical proportion for sauerkraut recipes. As the link states near the top, recipes can vary from 2 to 3%, but 2.25-2.5% is most typical. Anyhow, the recipe also states that if extra brine is needed to cover the cabbage, it should be produced by adding 1.5 tablespoons to a quart of water, which comes out to roughly 3% brine.

Bottom line: yes, assuming I understand what you did, lactic acid bacteria will probably still be able to grow in your mixture. Whether they grow fast enough to ferment well and produce acid before other things grow -- that's harder to predict. In the future, I'd suggest using a specific recipe from a food safety and preservation organization. Unless you REALLY know what you're doing, you shouldn't be modifying recipes or performing your own calculations to do uncontrolled fermentations.