Most of the time, you get what you pay for. Even if he did agree to a reduced rate, you were relying on him having the same enthusiasm for the encounter and you were short timed, timed and ultimately not happy with the encounter. That said, you are partly to blame as it saved you (x-y) dollars but you wound up with a lesser product.
As far as returning his belongings, I am not leaving the cozy confines of my home to return them. I might consider it if I were happy with the service, but since you were not, let him walk the walk. In fact, to be especially uncooperative, I might watch the clock and tell him he had a specific amount of time to get them because you had other plans and would be leaving on time.