I've had some similar experiences. My partner and I are straight presenting though my partner is trans, so I guess that is temporary. I've been told that I am straight because I have sex with women, but also told I am gay because I have sex with men. Honestly it is just bigotry pure and simple, but it can get inside your head and mess you up.
At this point in time trans people are the target of the day, but once they are dealt with the bigots will come for the bisexuals, then for the gays and lesbians. The only way to protect all of us is to band together and protect our most vulnerable members.