You could have just told him that it is personal choice, just like with any other article of clothing a person might wear. After all, there is no requirement to wear underwear with jeans, or any other pants. Then, if he did indeed ask the question, you could fall back on any of the many answers from here on the board.