In most cases, it is the price that will cause me to say "no thanks!" I generally do a bit of homework about the escort, and have a pretty good idea about the anticipated experience. However, from time to time, they come back with a very high rate, or higher than others have indicated it might be.
While there could be several reasons, price is frequently the #1 rejection factor, IMHO.