A disgruntled escort could publish your phone number online somewhere where you have no power to get it deleted (yes, it has happened).
Then when a current or future employer does a google search on the same phone number that you gave them on your CV, one of the things they will find is an account of you meeting an escort, or attempting to meet an escort.
Same with friends and family, they know your number. Even if you change number, they might be curious as to why you are changing and could google the old phone number, and they could find that same account of you meeting an escort.