I'm hardly a fashion maven, but I have a bowler with a silver concho hat band I've been known to wear at the, ahem, drop of a hat. I think, so long as you're comfortable with it, what other people think is of secondary importance.