Maybe the reason you find that you don't have to walk on eggshells is because of the inversion of privilege in these spaces: you have traditionally-oppressed-group privilege?

You say that not seeing color is a problem because then people won't notice if people of color are absent. Well...yes...but isn't that worth it if they also won't notice when people of color are present? (Of course, the worst is: don't notice when they're absent, but do notice when present and be jerks in the latter case--obviously this can't be the ideal, and it is, sadly, all too common.)

It's a false dichotomy to think that we can't address racial disparities unless individually we train ourselves to be hyper-aware of race. You can't see disparities as an individual, most of the time--your perspective is too limited. But you can express prejudice and bias as an individual every single time. So this suggests that the ideal would be to de-emphasize racial differences on a personal level, while still investing heavily in analyzing trends to look for racial disparities.

There are a few exceptions (e.g. if group X has a cultural habit of doing something that group Y finds offensive, group X needs to pay attention to who group Y is so they can cut it out). But in terms of what people actually complain about, I see lots of "they were mean because of my race" and a fair bit of "it is exhausting being always viewed as a representative of my race" and not very much "they treated me like everyone else despite my race and it was horrible".