For casual kilting, wear what you'd normally wear with a pair of blue jeans or shorts. T-shirt, sweater, polo, no shirt...sandals, tennis shoes, boots... socks, hose, bare feet...
Here are a few of my opinions, which is what most of the discussion here is about, anyway. T-shirts are perfectly acceptable for casual kilting. They look better tucked in, so as to not hide the pleats, but are more comfortable untucked. To my eye, sandals and socks don't mix, just as boots and kilt hose don't mix. If you're wearing tennis shoes or boots, wear the socks you would wear with them if you were wearing shorts. I think a kilt needs a sporran, so I'd encourage you to get a nice casual leather sporran. There are several sporran makers around here, and I can personally endorse the work of "A_Hay!"
I'm not touching the hat issue. Too many strong feelings here.

Please notice that this is just my opinion. You asked about wearing the kilt casually, and there are not as many "rules" about casual kilting as there are about formal or traditional kilting. The only rule is, you have to post pictures so we can steal your style ideas.