On the other side of the pejorative side, one unexpected benefit of the apps in recent years is that older men are no longer "invisible" to younger men, as was the case for decades. Quite out of the blue, as I have gotten older, I have noticed the large percentage of men in their 30s and 40s (even their 20s) who profess an attraction to older men. At first being called "daddy" seemed offensive and insulting to me, but the first time I realized it was not being said to put me down but because the younger man was attracted, I was shocked.
So it makes sense that it means something specific to you, given your actual parent status, but take a moment and consider what it might mean to the younger man who is calling you that. You might find a very positive mutual connection there.