Sex is indeed a spectrum. Intersex presentation makes up a meaningful though small percentage, somewhere around the 0.05% range. If it were a binary there would be two options, mutually exclusive. This is a bimodal distribution, with two very strong peaks for XX or XY karyotype and a bunch of variation around either different karyotypes, XXY etc, or differing activation or expression of those karyotypes, eg androgen insensitivity etc.

On top of that, what would you say sex is exactly? Which gamete is larger? In seahorses the males have the smaller gametes but the females use something very similar to a penis to deposit the egg into the male who then raises it and performs all the roles we associate with females in humans.

Is it based on which chroonosomes? In some animals it is a WZ or W karyotypes, so that can't be it. In others it is just a presence or absence of a sex chromosome. In some plants they have more than two sets of everything, like strawberries with 7 copies of each chromosome. In others they have one, two, or four in some parts of the life cycle, but sometimes the thing we see is the higher number, sometimes it is the lower number. Some have a mix of male and female parts, having sperm and egg producers on the same plant but separated, some have both right next to each other in groupings. Some animals can undergo sex changing due to environmental factors.

Nothing in biology is as simple as the models we use to represent them. Sex is complex and while it sometimes seems simple that is the less common state. Genes are not often all the way on or all the way off, they are usually moderated and running at different levels across the organism cell by cell, and changing with time. The same goes for traits.

I would recommend learning more about ut biology if you really do believe sex is a simple binary. The world of biology is far more complex and varied than that idea can capture and honestly it is fascinating, I find it extremely exciting to find the examples of my own ignorance, they are usually super cool. Good luck!