Um, in reality, when you have differential treatment on the basis of appearance (e.g. taking a preferential attitude towards the majority, or those who look superficially most like you), and you consider race as part of appearance, yes, in reality it is because of the "color of your skin". That's it. That's all there is to it.

Some of your other points are good, but this is absolutely not. You could argue about which biases are triggered by skin color alone and how bad the impact is, but not that people don't notice or aren't biased: the evidence is just too overwhelming that they do and they are (often enough for it to matter).