I can understand how you feel. I usually start off by talking a bit, getting them a drink, normal getting to know someone a little, etc.. However, I consider us to be "on the clock" once I let him in. Also, after a few minutes, I usually say something like "shall we go get more comfortable?". I don't live in the city center and I couldn't imagine having them drive this way and then not counting the whole time they are with me.
You may need to guide the direction of your time together. They aren't mind readers.