It's worth a try though it might be kind of bulky in the pleats, depending on how thick the fleece is...

Sounds like you're interested in making your own version of the "Comfy Kilt" by Sportkilt, which is sewn up in cotton flannel. Check it out:

http://www.sportkilt.com/category/57/Comfy-Kilt.html