I do understand exactly your problem, there is nothing between ultra-casual and ultra-formal. Hot climates like Georgia or SoCal make jackets beyond uncomfortable, life-threatening from heat injuries is more like it.

First be under no false idea that I know anything about fashon but I will take a shot here:

First, below the waist, good stick with it.

Second, no waist coat or vest as they look a little odd alone.

Third, a nice shirt with a standing collar designed to be worn without a tie OR a fold down collar with a tie. DO NOT wear a fold down collar shirt without a tie. This would be way too informal. I often see colored shirts sold with a matching tie as a set, look around.

Fourth, the time of day is problematic, for day wear you should wear brown belt, sporran and shoes after 5PM black belt, sporran and shoes should be worn. I would wear black.