I tend not to agree with Sam Harris on a lot of things because I think his viewpoints, while perceptive enough as far as they go, often aren't sufficiently perceptive to handle the actual complexity of a lot of situations.
But here, I at least see his point pretty clearly, and it's a good one.
Scientific rationality: because it deliberately obscures a huge and important correlation. Obscuring relationships and complicating language in order to meet ideological ends (even meritorious ones) cripples our ability to think. Scientific thinking already has a perfectly effective way to handle exceptions: you talk about them as exceptions, with their own names. Then you can think clearly and reason about the ordinary cases, and you can also think clearly and reason about the unusual cases.
The more demands you have to cater to to obscure common relationships and highlight unusual ones, the more awkward everything becomes. The harder it becomes to think clearly. The more prone you are to making cognitive errors. And, if externally imposed through social pressure, the more likely it is that a mis-step triggers castigation, reducing your ability to actually explore reality as it is. Thought policing is incredibly bad for rationality.
And let's be very very clear about this: it is not scientifically or medically necessary or useful to have as a category "bodies with vaginas" which is used outside of extremely rarified contexts. The pressure for this is entirely imposed for social ends.
You might argue that it's worth it; that the social ends are so important it's worth the downside. But you can't argue that this is the natural way to organize things for rational manipulation, because it isn't.
As far as why it's bad for the English language, in addition to it being an almost direct parallel to 1984 (Orwell), you say why yourself: it is clunky and impersonal. Language thrives when it is elegant, and it is one of our most powerful ways to personalize and make human connections, which is extremely important for us to have compassion for each other.
So on this one point alone, I think Harris' position has validity. The counter, if one is to be successful, would need to be of the form "yes, but". (E.g. "but the impact is modest and the benefit is large", plus details justifying that view.)