First of all it is clothing not a uniform, belt / not belt or vest / no vest is more your choice for casual wear. A belt is normally not worn with a vest because the belt pushes the bottom of the vest out and it looks funny. There are vests that are cut short and end just above the belts and then a vest belt combo may look OK. If the kilt fits correctly you don't need to wear either a vest or a belt.

The major question, especially for casual wear is what makes you feel comfortable.