You know, I think you could make it work even very casually. Spats are the kind of thing that work best with a confident attitude and a well matched outfit.
I say go for it.