Depends on the cut of the vest for me. I have two that sort of sweep away at the very bottom so I don't button those. If it is cut straight then I will button all of them.

For me comfort would certainly be the deciding factor.