What's allowable is what two people agree on. Translation: Communicate
As to my own experiences, when I travel with a client, I am exclusively with him. I'm on his dime, his clock and don't look for play elsewhere. My only requirements for 'me' time are daily gym visits, but he's always more than welcome to join me for a workout.