You're not firing him, you're certainly now willing to hire him again because you didn't like him.
tell him, you moved out of town
block him
tell him, you're no longer hiring
tell him you have a regular in town and you're not interested in seeing other guys (including him).
No big deal!
Many escorts/masseurs if they don't hear from a client after a couple of trips simply remove it from their contact list. It happens in all professions!