I'm having a tough time with this.

First off I would not go for it personally. It somehow does not seem right to have jackets and the like in leather.

Now I will go along with Matt in saying that a more casual denim or leather jacket may be a good idea.