Since you mentioned California's ban on the use of race in admissions for its public university system, it's worth noting that nobody has to go with their hat in hand in the California system. There are numerous community colleges in the state, their mandate to take anyone who is prepared to learn (and charge only a modest fee), and the top students are guaranteed a spot in the prestigious University of California system to finish a bachelor's degree--and it's been this way for decades!
California's community college system has long been a path for students who have potential but haven't had the opportunity to learn and prove themselves in their secondary schooling. Did you know that Jackie Robinson, Steve Jobs, and Beverly Cleary all went through the California Community College system?
It's great to have the HBCUs to provide additional wonderful educational opportunities in a setting that may help students succeed who otherwise might have struggled or been unmotivated to try. But it's worth noting that students of all races in some places have high-quality opportunities to catch up on the education that they were denied. And there's no reason why this shouldn't be the norm everywhere.
We do need to fix the educational disparities. As Harlem Children's Zone shows, the biggest problem we face is not investing belief and money into teaching everyone. But we have good ways to provide an alternative route to education, as California demonstrates, which do not require either affirmative action or de-facto racially-segregated colleges and universities.