Yes, exactly, because some resources are limited and those best able to take advantage of them should do so.

The gatekeeping can be (and often is) done improperly, but it is a very important function.

"It's a lottery for who gets to go to med school"--yes, that's going to work out well, isn't it?

"Navy SEALS are chosen at random from the U.S. population." Maybe not?

It's also very odd to in one breath say you've been excluded by claims of "merit" (presumably because people are not colorblind) and in the next say that colorblindness is the problem.

Fake colorblindness is a problem. "We don't see color--we just act to exclude it" is terrible. But actual colorblindness solves most of the problems you complain about. (It's mostly been about bias, differential treatment, and the like.)

Now, actual colorblindness might not be achievable. But you shouldn't argue against the concept but rather about the implementability, unless you're claiming that the color-evoked bias and differential treatment is no big deal.