I say go for it! Most folks don't observe the formal rules of general dress in society any way (i.e. no white after Labor Day, etc.).