Why not just say loud enough for everyone near him to hear "Intelligent people call it a kilt."
Sorta makes him look the fool and in most cases people around will end up defending your right to wear one.