Wear the Fez by all means. It is the symbol that you are using to show others the reason you are making the visit.