Just as structural racism provides differential harm to one race based on ostensibly race-blind structures, so too can structural affirmative action provide differential aid to one race based on ostensibly race-blind structures.
If it's bad enough to do something about when it's racism, maybe it's good enough to serve the need when it's affirmative action?