When i was younger and not fully comfortable in my skin an older coworker would greet me; hey handsome. I had no idea how to take it. Spent many hours thinking of what that meant. Many options went through my mind. I ended up processing it as a kind greeting. That kind greeting actually made me see myself differently. It gave me confidence and taught me how my words can impact people. I use that greeting as well. Its amazing on the smile i receive in return. Most people appreciate the kind words.
With that, you live once. Be brave, if you want something more offer to grab coffee or a smoothie after working in the yard. You'll be able to gage wjere he's coming from. Next time he says a flirty comment bring it back to him. Such as.... not as sexy as your body but thanks....