Black people throughout the world have an incredibly diverse history that is absolutely not the same across different parts of itself. It's as different from itself as anyone is from anyone else. Not to mention that "People of Color" have their own widely divergent histories also (throughout the world and within the United States). I certainly agree that CHOSSA people have a different history within the U.S. than does any other group, but unless you call a recent Nigerian immigrant to the U.S. a "Person of Color" and not "Black" (?!), I really don't see how this makes enough sense as the justification for a linguistic distinction.
I get your hesitation to identify with a group from which you sometimes receive racist attitudes--but really, don't you think you've got to cover at least everyone in the United States (including people who recently came from sub-Saharan Africa), and preferably the whole world?