They probably are simply not used to the idea of a kilt that is not tartan. Arm yourself with a little education about solid color kilts, so you can let people know when they ask that yes, it is indeed a legitimate and traditional option.
http://albanach.org/solid_kilt.html