If the jacket were tweed, or just grey or navy I think it would be fine. The black seems to my eyes a tad too formal for the polo.

Cheers