Personally, I'm kind of used to it. I wore the U.S. Army uniform voluntarily for 21 years through the Viet Nam years when they used to spit on it, for those of you who don't remember those years.
I take it in stride now as I did then. There are those types that just don't get it. There aren't very many these days but they're there and as you said, they dress very sloppy and yet feel they have some right to critique someone elses style.
By the way, those that used to attempt to belittle me for wearing a uniform and being a soldier? I see them every once in a while on the streets, drunk or burned out from drugs. They have nothing going for them and never will so I guess that fate has a way of dealing with them, it doesn't make me happy or sad one way or the other but I still take it in stride.
Like my mother used to say, "Consider the source".

Chris.