It might help if you dress yourself up as nice as possible as you wear it casually. It's hard to be too self-conscious when you know you look great! It also makes it more easily identifiable as a kilt so if you're worried about people taking it as a skirt that'll help you too.