"Belief in human rights equals leftism" sounds more like classical liberalism to me, which is also strongly endorsed (to an extent) by the right in many ways. For instance, a vigorous defense of merit is far more likely to be found on the right than on the left, and yet the underpinnings of the argument is (when well-constructed) usually rights-based.

Also, where you prioritize social justice vs human rights will have a profound impact on your ideals. (Details of how the impact plays out will depend on your conception of social justice, but note that "human rights" are usually envisioned as individual rights not group or society rights.)

If you put human rights first then you find ideological opposition both on the left and on the right; they just have different reasons for curtailing (or failing to recognize) human rights.

Full disclosure: I don't think human rights are always the best way to think about moral obligations towards each other.