You said the two guys—to use your language—were “fooling around.” That’s a form of sex.
You’re really hung up on race aren’t you. What do you think my race is?
Maybe it’s that he doesn’t like your body. Or your nose. Or your potbelly. Or your skinny legs. Or your age. Or whatever. He made it clear he didn’t want you around while he was “fooling around” with another guy. Rather than closing your eyes you might have been polite and backed away rather than in effect calling others racists.