I must admit that I haven't had that happen in a mighty long time. However I have thought of saying something like "you look lost, do you need help with directions?"

Yes it is frustrating.