I've had similar encounters myself. I think (IMHO) that it's the similarity to a cassock that draws notice.