Pretty much what Dave stated, I would have replied, that one does not have to wear underwear anytime as that is a presonal preference. With your wife saying what she did probably threw him for a loop.

Sometimes keeping them guessing, by not saying anything, is one great way to respond.