The problem with this is that people will start to look at external phenotypes to make assumptions about hormones and genotype. Someone is going to 'look' male or female, and then we are going to make assertions because 'biology says'

Well, yes, people always oversimplify things. But if you instead make the assumption that there are no differences, and that expectations cannot vary at all, is that actually any better?

It seems to me that there are two separate problems here.

(1) What can you know about someone based on superficial characteristics--is it enough to be informative?

(2) Whether or not you think you can know anything, are you sensitive to individual differences?

I don't deny that people have historically decided the answer to (1) is "you know almost everything" and to (2) is "no, don't bother with being sensitive". But it's not clear to me that we can't get both right.