Think about it, they'd have the ability to mate with anyone else regardless of the partners gender where-as single genders can only reproduce with members of the opposite sex, their population would be double that of the single gender in no time, after a while single genders would be bred out of existence with maybe a few here and there due to recessive genes and all that jazz. Unless of course hermaphroditism itself would be a recessive gene, in which case it would be more of a rarity, however since this is completely hypothetical since herms are incapable of actual reproduction there's no way to determine if herms would be a recessive or dominant trait.