We do not always wear our histories or identities or residency status as signs.
If "we know" that "white guys" are privileged and approach people on that basis--on the basis of appearance or demographic characteristics--that's . . . problematic.
You can't always know when you are "punching up" vs. "punching down."
And "racial shorthand" is dangerous.