It's really weird that these things go around. I think at the end it just a fallacy of association. Unfortunatly not only anti-vegans use that crap, in russia it is quite common for vegan groups to claim that queer folk (whether queer through sexuality, gender, or both) are like that due to the meat consumption. They make arguments like "In USSR most people were too poor to eat lots of meat, and there were less homosexuals (transsexuals, etc), therefore it's eating meat that makes people sick".
This stuff really reminds me of "Did you know that Hitler was a vegetarian" type of crap (which is not only false, but also completely irrelevant). And i try to stay away from the arguments like that all together, or if i join in them i try to argue against this even being a topic of discussion.
P.S. I hope that there'd be more pansexual vegans.