Actually depending on how it fits you (and how it was fitted to you), it can be tricky.
It would have to be something that needs to be evaluated. You can't just tack or pin it because it would be affected by movement.
A stiffener, like an interfacing could work, or could make it worse!

It's a tough call