Yes, that makes a lot of sense, and I agree (and had thought about it before)--human desires and needs (the actual needs, not the "I need a new iPhone" needs) come before both.
So I think it's fair to say that they're both aspects of how we can organize our thoughts around developing and maintaining a social contract.
However, I also don't think it's too misleading--and is better at reshaping one's outlook given the normal way we think about rights--to say that responsibilities come before rights because without first considering the associated responsibilities, thinking about rights is vacuous: the only path to get rights is through responsibility. However, we can perfectly well take on responsibilities without thinking about rights at all. So rights appear to me to be the more derived concept: you only acquire a right in practice after the necessary responsibilities have been accepted.
I wouldn't advise going overboard there, though. Of course, once we have both ideas in mind, we can work with either and sculpt the other to match. For instance, we can think about rights and decide "we gave these rights to some people but actually we should give them to everyone". Then we change responsibilities accordingly.
Thank you for the interesting discussion!