I think this statement really highlights your naiveness. While it's cool that you like to play dress up, you have to understand that some people might be speaking from experience.
You don't want to be that dude who is always trying to look better than everyone else and then fail to get a promotion/advancement as a result.
You say you don't worry about other people, which is great, but you don't work for yourself. You work for other people, so you have to always be taking their opinion of you into consideration, especially if they have control over your livelihood.