What you want is fabric suitable for making trousers. I've made non traditional kilts of cotton duck, cotton twill, cotton-poly blends, wool, wool blends, and the ones I like best are, in order, the all wool, the cotton poly, and the wool blend.

Cotton poly has a bunch of advantages, it's hard wearing, it will take a crease, but it doesn't wrinkle as much as pure cotton does, it's available in a range of weights, colors, and can be had reasonably cheaply. Some of it's better than others, of course.

The all cotton twill I made were always wrinkled, got shiny spots at wear marks quickly, and didn't hang as well. (The same fabric made up into a nice pair of trousers, which weren't terribly prone to wrinkling.)

You don't say where you are, so I can't make suggestions about sources.