Antifa is a fringe group. "Let's be ready to start fights in the street a la Weimar Germany, because obviously that worked out so well," is a pretty absurd starting point.
BLM isn't and shouldn't be a fringe idea, though, because it is a response to the reality that all lives are supposed to matter, yet if you look at how things play out it sure looks like people are putting very little value on black lives specifically.
So "Black Lives Matter" is a way to help actually make all lives matter. If you have a goal, and you're partway there, focusing on the parts that remain is the way to accomplish the goal.
Now, as with anything, you'll find excesses that cause problems, bad ideas about how to accomplish goals, self-serving individuals exploiting things, and so on. All that is apparent within BLM as a whole, but the thing to do is to push back on each one of the problems, not the idea.
The idea is fine.