If I have only booked an hour, five minutes is more than enough chit chat. If they haven't initiated I do. I have friends I can talk to.
If they want to keep talking, it should be off the clock. Let them know you are ready to go.
I hate to say it but it's partly your fault. They might be thinking you are waiting to be comfortable.
Especially a first meeting I like to book more than an hour. Gives a bit more time.