I think, depending on your relative ages, sizes, muscle, etc. and whether this is an in-call (your place) outcall (his place) or a hotel, there is the issue of your personal safety if you were to refuse to pay. The OP said the RM was high. Do you really want to confront him? I think the better path is to pay and be done with him.