Welcome to the wonderful world of merchandising.
I usually wear jeans, a black t-shirt, and look nothing like store staff, but it gets so bad some days I want to hang a sign around my neck that says, "I don't work here, don't ask me for help."
For the record (and giggles) I've been talking with store managers, refused to help customers, the customer then demands I am fired, and the manager then tells the customer s/he can't fire me because I don't work there, and the customer still doesn't believe them. I swear how these people have money to shop is beyond me.