Can I set a creme brulee with agar agar?

Removing a custard (which is what creme brulee) is from its form or mold does not definitely require agar agar. Flan, which is famous from a number of cuisines, is an unmolded custard.

While experimentation would be required, it is highly likely that if you use a silicone based flexible form, and make a fairly stiff custard, you will be able to gently and carefully unmold it. You may wish to lightly grease the form before baking the custard to help facilitate release. This Chest of Books article recommends two eggs per cup if milk for a custard that requires unmolding.

Many molded custard recipes exist, although a good number of them have gelatin as an additional stabilizing ingredient. You may wish to google "molded custard recipe" or "Charlotte recipe" for inspiration (Charlottes tend to have a lady finger garnish around the edges).

If you are interested in using agar agar or other modernist hydro-colloids in your recipe development, you may wish to read the excellent primer at Cooking Issues.

Note that they characterize the texture of agar agar gels as somewhat brittle, which may not be ideal for your application.

See also Can I substitute agar agar for gelatin in pudding which is a very similar application. Michael at Herbivoracious indicates that agar agar may not be suitable due to the texture. There are some alternate ideas there that may help you get started.

SAJ14SAJ has a great answer. Creme Brulee is very similar to Creme Caramel which is a molded custard.

You could try Brulee-ing the unmolded creme caramel, either the top or making a complete shell. Agar shouldn't be necessary. Sprinkle the outside with super fine sugar (not icing sugar) and use a plumber's torch. It could turn out to be an awesome creation!