I think if your going to step outside the sphere of what (in most other peoples perception) is "normal" then you have to expect some kind of reaction- its only natural for people to wonder why your wearing the kilt, and whilst you answer to no one as to what you can and cant wear, most people outside of Scotland dont know anything about the history or facts about the kilt.

people may think your a cross dresser or whatever, but why does that matter?, I dont mind what people do; cross dress/gay/bi/whatever, I know im none of these things, and if I was then I would be proud to be, you have to accept what you are, recently a Scots guest house owner was criticied for banning two gay man from sharing a bed in his hotel, he was stuck off by the Scottish tourist board but received much support from "the man in the street" his point was he doesnt mind what goes on but not under his roof, some people would use the word "homophobe" I cant stand this word its bandied about to much and doesnt mean anything, other than someone with an irrational fear of the word "homo"

You have to be comfortable with what you are, if people think your a cross dresser -then the world doesnt stop spinning! let them swirl around in a pool of ignorance, Sadly two men were beaten up by neds last year in Scotland their crime- wearing the kilt!!
be happy with what you are and let people think what they want.