Its more than likely what town or area of town you are in. In my area people area very accepting of the kilt, but the closer you get to hillbilly central the less accepting people are. Its like a threat to their norm.
I wouldnt worry about it. Youre obviously the much bigger man.

Craig