When you have to ask others if what you are wearing (fill in the blanks) "makes you look fat", "looks ridiculous", does nothing for you" etc., etc. then you should forget the clothing, be it Speedos, skinny jeans, tight polos, baseballcaps, and so on.
Confidence may make you FEEL great, it doesn't make you LOOK great. After a certain age, there are parts which will betray you---no matter how much you exercise, diet, go under the knife. Accept it and most of all enjoy the luxury of being comfortable in your skin and not making others' crawl.