If WORSTED is not important, I would suggest 96 District Storehouse. Their selection isn't big (usually 5 or less at any one time), but rather cheap: usually $10-15/yard. All they sell is reenactment fabrics, so NO polyblends.
They usually have Black Watch (or a close semblance) in stock in a decent weight wool for a kilt. Normally, it is intended for guys making Highlander regiment plaids for Amer. War (of Independ.) reenacting. So, the owners make a great effort to have some stocked. That was what I used for my first kilt.
(Those are ALL -usually- SAXONY wool, but good for the price-> GREAT for cadadh.)
They also have a LARGE amount of solid wools (and a few that look like tweeds), usually in the $10-15 range (some more). They also stock various linens, cottons, and the like in period appropriate prints and hues (generally, Amer. Rev. to Amer. Civil War). I get A LOT of stuff from them. I would also recommend their Woolsey (Wool and linen blend) hose, which are relatively durable, cheap, and a nice "oatmeal-esque" tone. (I where mine COMMONLY with my kilt, and have several pair.)

96 District Storehouse
http://www.96districtstorehouse.com/
111 Trinity St.
Abbeville, SC 29620
864-366-9600
sales@96districtstorehouse.com

(NO association besides customer)