The problem with your "rebunking" of the concept--which you treat very nicely qualitatively--is that you don't actually get into numbers. Given that biology is messy, either you throw away the ability to make distinctions between hands and feet and say "everything is a spectrum", or you accept that any dichotomy will have some number of exceptions and special cases that might not fit any typical pattern. It's hard enough to understand things without making easy things hard, so let's do the latter, hm?

When you add numbers back in, the number of cases of intersex conditions that are spectrum-like, it's really very small--a small fraction of 1%. (Most intersex conditions are clearly "more male-like" or "more female-like". Only a very few leave one scratching one's head even after careful consideration--like the two examples you gave.)

If you look at individual traits--height, for instance--you find that the very strongly dichotomous regulation nonetheless produces overlapping distributions where "spectrum" is a fine way to describe things even if you mix the two differentially-regulated distributions from males and females (and the very small number of "um, this is different enough from the norm that there isn't even a clear choice of which word to use" cases).

But overall, if "sex" is supposed to mean the totality of differentially expressed phenotypes, it's not meaningfully a spectrum. It's like saying that the side of the road you drive on (in one country) is a spectrum. Yes, occasionally you turn, go around a double-parked car, or if traffic is low drift a little over the center line to get farther from parked cars. But you drive on the right. Or left. "Spectrum" is a spectacularly bad way to understand the kind of variation from the norm that you get regarding where on the road you drive. Likewise with sex.

This says nothing about whether or not we should treat people well, of course. This is just about being honest with ourselves. If we have to delude ourselves to be nice, I would argue that we're not actually nice at all.