I often wear my kilt when going "out-on-the-town". I live by the St. Lawrence Market area of Toronto and regularly shop, walk and "tipple" while kilted. I can tell you, however, that the pubs are many, the food plentiful and varied and you may require an extra hole or two in your kilt strap should you have a prolonged stay. Also, check out the Distillery District.
I hope your visit will not conflict with the G20 Conference here in the city. There is a ton of extra security, traffic control and general inconvenience in the downtown area that could limit your ease of getting about and you enjoyment of our city.