The problem with non-visible disabilities like Asperger's Syndrome is that people seem disposed to forgetting about them, and dare I say even deny them. If I had no legs and I got into a relationship with someone they wouldn't think to themselves "Sure he has no legs now, and needs a wheelchair, but after going out with me for a few months he'll grow those legs back." "Sure he's blind now, but once he meets my friends he'll be seeing in no time."
Because no-one can see your disability they soon forget all about it, maybe even think it's an act, but the overall and lasting impression seems to be that you're just shy and just need to come out of your shell. That being dragged here and there and thrown into deep-ends will have you waving and not drowning. As the months drag on they begin to realise that it's not an act, that you'll never change, that this really is just how you are.