Next time just walk away or change the subject if they start talking.
This ruin the sexual fantasy? Have you heard about anonymous sex? Just wait for them naked in your room.
You're the client, express yourself, specially if this problem has happened already 4 times out of 6 and bothered you so much you're posting on here seeking advice about how to stop them from being so attached to you.