No, people who are strongly gender dysphoric--those are more common than those with intersex morphology.

If it is true that there is a brain sex that goes beyond a generic brain responding to hormones, and that brain has a strong affinity for a body of the same type, then if there is a mismatch, it's not clear that "sex" unqualified has a clear meaning.

The reason is that we're used to everything aligning reasonably well. We expect that. If there are rare cases where it doesn't, it's at least worth re-examining the terminology to decide if changes are needed.

We might decide that, no, the best way to resolve the lack of alignment is to say body morphology wins: it is reproductive organs, or what the reproductive organs would have been if not for a developmental anomaly, that we use to categorize. But I don't think it's fair to just pre-judge and say that it has to be this way. You don't have to accept silliness like "sex is a spectrum" (it really isn't--there's a great deal of machinery devoted to ensuring, as best as a biological system can, sexual dimorphism) in order to at least have the conversation.

For instance, one resolution is to say that sex refers to body and gender refers to mental states, including those with a biological basis. So gender isn't just culture--it's culture plus any biologically-driven mental factors.

Another resolution is that body sex and mental sex each refer to purely biology, absent any cultural influences, but then in the rare cases where the two don't align well, "sex" becomes an ambiguous term.

The "assigned male at birth" stuff seems like a rather poor resolution to me. But I don't think that being skeptical of the resolution means that there was no issue to begin with (even if the issue was rare).