That you can force something into a binary if you try really hard does not make the natural division "binary", nor does it demonstrate that the division you are using matches in detail the divisions that other people use. When you have a large number of covarying traits with two main modes, once you find something where the covariance is broken, you really need to be more specific about what you're talking about.

Although this article has several problems, its description of intersex conditions is pretty good, and illustrates why the forced dichotomizing is just not a very astute thing to do: https://medium.com/starts-with-a-bang/ask-ethan-what-is-a-woman-1f94e7abcfbe.

Note that people do also tend to fail to dichotomize where they could; "intersex" includes a large majority of people who are by any reasonable definition biologically female or biologically male. But there are also some (even more rare) cases where saying that the biological sex is "oh, wow, that didn't come out the normal way at all" is drastically more accurate than "well, if you look really really hard and decide that this is more important than that, and imagine that these regulatory processes went differently than they actually did, this one is 'male'".