Definetly, take off the best. Sleeve shirt would look fine without.

Does one have to be a gentleman to look nice? Or to be traditional?