It's about as clearly defined as anything is in biological science. Just because there are cases where you can't say much beyond, "Well, this is weird!" doesn't mean that the vast majority of cases fall into the clearly-defined pattern.

In cases where there is no clear differentiation into egg-producing and sperm-producing phenotypes, we don't assign sex. For instance, budding yeast has a and alpha mating types, but as they are otherwise identical (save that a and alpha haploids combine to make a new diploid yeast, not a and a or alpha and alpha), we don't assign male and female.

Otherwise, the producer of the egg-like-thing is the female, however it happens to be regulated, and the one that produces the sperm-like-thing is male; and if there is a strong correlation between the non-gamete phenotypes and the gametes--and there almost always is--if something goes wrong with the production of gametes but the other phenotypes are the same (especially if the regulatory pathways are the same but broken/altered at some point), then we call the thing that would-have-had-eggs female and the thing that would-have-had-sperm male.

It's a very robust pattern, very clear in the a huge number of cases, and wherever it's not clear, you just don't assign a sex.

I really don't see, given how messy biology is, how anything can be terribly much better defined than that.