The problem with saying that "sex is a spectrum" is that it's not really any more accurate than saying "sex is binary" because the variation does not occur along a single axis. Take chromosomes, for instance. Men are normally XY, and women XX. But you can find a whole diversity of people who survived having all kinds of different numbers, not just XXY (which in some sense is "in between" and therefore spectrum like). What about X (Turner syndrome), or XXYY? And then there are other genetic disorders that result in absence of some sexually dimorphic features without the appearance of any sexually dimorphic features of the opposite sex, and yet others that express sexually dimorphic features of one sex without diminishing those of the opposite sex. So, really, "sex is a spectrum" is an drastic oversimplification if you are specifically talking about the whole diversity of possibilities that are observed in the human population. Would you say that "literature is a spectrum" or "there are a spectrum of sports"? This seems a poor way to use the word "spectrum". On the other hand, all of these things are very rare (well under 1%), and there is a huge amount of genetic regulation devoted to maintaining clear sexual dimorphism (both that necessary for reproduction and that for signaling), so in that sense saying "sex is a spectrum" is very unlike a real spectrum, where the different colors are all reasonably frequently found. So I just don't see any condition under which "sex is a spectrum" is a useful thing to say. You woefully misrepresent the big picture if you're not careful, while still woefully underexpressing the diversity of actual phenotypes and conditions.

Regarding performance--of course there's a wide range of levels of capability among trans people. The point is not to let outliers determine things, but rather the effects of gender-affirming care. If we were to find a sport--heighting, let's say, where you win based solely on how tall you are--where trans women were closer to cis men than to cis women on average (or in distribution), then it would not be fair to maintain a difference between men and women while placing trans women with cis women. In order to know to what extent this is the case, at present anyway, there seems no option but to go check sport by sport, because the physiological changes induced by HRT are complex, and how biomechanics impacts sport performance are not sufficiently well understood to predict in advance.