This is badly wrong. I guess since Lise Eliot, who really ought to know what she's talking about, recently wrote a paper claiming exactly this, I can't be too hard on you for accepting it.
But it's wrong. It's just plain wrong, because people have looked really carefully at the structure and it's not the same: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6418873/.
The differences aren't huge, and the distributions overlap substantially, so it's not "Men are from Mars, Women are from Venus". But the structures are absolutely there, and they vary in both directions which makes Eliot's charge of not correcting for volume (which is true) less compelling. "Hey, you think you can tell male and female brains apart, but you didn't check for size" doesn't show that you can't tell them apart! It only tells you you can't be sure you can.
Anyway, it turns out that Eliot was probably overzealous in her criticisms. Since the 2021 paper came out, people corrected for volume and from volume-corrected brains had an 86% success rate of predicting who was male and who was female: https://www.medrxiv.org/content/10.1101/2022.04.27.22274355v1.full.
But wait, there's more! If you analyze brain structure of trans people as well, you can tell all four groups apart! Trans men, trans women, cis men, and cis women all can be identified at way above chance, and though brain size may play some role it can't be all of it because you couldn't tell trans from cis on the basis of that: https://pubmed.ncbi.nlm.nih.gov/31999324/
Not reproduced yet, so there's always a chance of systematic error, but there aren't any obvious flaws.
And, of course, if you drink alcohol or smoke pot or whatever, your behavior changes a lot, so it's not all just a matter of structure, either. Even if the morphology is identical, brains could be functionally gendered.
If you find a structural difference, however, note that it also doesn't prove that there's a functionally gendered brain. Only a morphologically structured gendered brain. But it does suggest that a difference is likely.
Nonetheless, your characterization of transmeds being wrong that there are structural differences is itself wrong. Eliot correctly noted that the studies on differences weren't very good and some weren't consistent, but that didn't prove that there was no difference, only that we shouldn't think we know...and in the meantime, we've come up with better evidence and a difference is apparent.