People are strange. NT people are stranger. I too fail to move from positive first meeting to friend (whatever that is). - to be honest it's something I'm not uncomfortable with. I don't know what is said behind my back at work, but as I can be rather ferocious at times people tend to be careful. Being good at my job probably helps too.
One thing that did occur to me, and this is not meant to be offensive is that the uncanny valley may play a part. If you don't know what this is, it started with humanoid robots that looked too human and they freaked people out, while non human robots were fine. If the masking isn't perfect NTs can have a hard time with accepting, however people on the other side of the valley can be 'endearingly eccentric' and get along fine. I don't mask anymore, I go for eccentric, it's easier. It is however a fine line and even I have to pretend to be normal sometimes.