This is a very thoughtful take on a difficult situation. I'm especially impressed that you were able to think through what it might look like from another perspective.

But I wonder--if the problem is the fears, maybe that can be addressed directly.

For example, if the standard for places that have gender-specific spaces is to also have gender neutral spaces, then there is no reason the fear ever needs to be provoked, at least not in any obvious way.

If you pass as one gender or the other, use their space. Nobody feels threatened. (Unless statistics were to show that despite passing, trans women perpetrate considerably higher than cis-woman levels of violence against women...if that was true, you'd need to pass really well in order for it to work, since any hint would be scary.) If you don't easily pass (or can pass as the one you don't wish to go to), use the gender-neutral space.

Furthermore, if it is provably true that with gender-affirming therapy, trans women have rates of aggression etc. against women similar to the rates displayed by cis women (rather than by cis men), then documenting this really well may in time allow even victimized women to relax as long as you pass not as a woman but as someone deep enough into gender-affirming therapy to fall into the low-danger category. There may also be other ways to give a trustworthy signal that indicates that danger is unlikely.

(If the statistics don't work out the way one wants, it's worth subdividing--for instance, rather than just considering trans people as one block, one could consider those with strong gender dysphoria vs those without.)

So I think in the case of a fear-based reaction, there's quite a lot that can be done that as far as I can tell isn't being pursued. (Indeed, I think some of the vehemence of trans activism is actively stoking fear-based reactions through its stridency and insensitivity.)

Then the question is what the relative rates are of fear to other bases for opposition. If fear is a big one, then the overall condition for everyone might be improved considerably. If it exists but is comparatively rare, well, maybe still good to address, but most people's lives would not be materially affected.