Depending on the details, objecting to a certain kind of casting can be story-based, racist, or anti-racist.
I'm all for shutting down the people who object on racist grounds, even if an anti-racist argument can be made, because bigotry deserves to be opposed and called out for the small-minded hostility that it is.
However, let's think a little bit more about all three cases, so we don't mix up legitimate concerns with hatefulness.
First, the anti-racist objection, when appropriate (generally present-day fairly realistic settings): here you have a white character doing white things in a white setting and all you did is change skin color and pretend that you're representing the black experience. No, you're not! It's the most egregious and patronizing type of tokenism. Although it's possible for black people to do exactly the same things as white people, in the real world there are real differences and if you hide those differences you're hiding the real experience. It's just one step from blackface--yes, a black actor played the role, but everything else about the black experience was elided and that is wrong.
Then, the story-based objection: the imaginary (or historical) world presented has its own setting and rules within that setting, and everything that seems out of place is jarring and makes it hard to suspend disbelief, which is key for enjoyment of many forms of entertainment. For instance, if you see Native Americans in the 1400s using chopsticks, if Brutus uses a flintlock pistol to kill Caesar, if you see palm trees in the antarctic, or an ethnically Zulu king of the Saxons, it simply doesn't fit the picture. We don't object to the Zulu king because we object to black people; we object to things that don't fit. If you want to do the work of establishing a believable (not painfully strained) alternate history of the Saxons where a Zulu king could reasonably be there, great! But don't just stick them there and expect us not to complain, any more than you should expect us not to complain that because your set was in Hollywood and there were palm trees in the background that we can't believe it's Antarctica.
Finally, the racist objection: I wanted this character to be white, and they're black. No way!
So, back to the book. It's not clear to me whether the story-based objection works or not, but it seems like it might. Annabeth Chase the a daughter of Athena and Frederick Chase. Athena is Greek. So if Annabeth Chase is played by Leah Jeffries, our choices are (1) Athena is black (Zeus is black? Zeus's headache was black? Hephastus' axe blackened her when she was ejected from Zeus's head? This isn't the Athena of mythology?); (2) Frederick Chase is black; or (3) since Annabeth was formed from Athena's thoughts, not sex, she's whoever Athena was thinking of...but why was Athena thinking of a black girl--needs explanation?
If it makes sense in the context of the TV series, great! But it seems like it's going to take some work to be convincing. Also, fans of the series are used to Annabeth as who she was, and you don't automatically get to carry over any fondness for Annabeth to someone who you have labeled Annabeth and stuck in something labeled Percy Jackson and the Olympians, even if Rick Riorden says he wants it that way. For instance, I love the immersiveness of the Lord of the Rings. I was outraged by the hasty ents in the movies and put off by the exessive fighting and minimal fear, planning, uncertainty, and tension.
So, maybe racists are livid because the character is black.
But maybe people who fell in love with one character aren't too fond of what looks like it's going to be palm trees in Antarctica, or simply not the character who they loved.
However, given that many fans were nonplussed by the movies, which didn't really involve Riorden much, I think he has plenty of opportunity to win over the fans quickly with the kind of storytelling they love and characters whose souls they recognize even if they might look a little different.
At that point, we'll be able to who the real racists are.