If you believe that you're a person who can talk to any person of any age, skin color, physical quality (hot woman, old man, ugly child, etc.) without the fear that you're not worthy enough, I think height will not matter.
Confidence comes from being comfortable with the body you have and...