Yes Bill... I often do that to informal events where I'm sure a few other people will be wearing shirts without ties. When I get to these events I normally take my jacket off right away anyway.
Just wear whatever feels right!