It doesnt sound like youre missing anything, and you should just move along and ignore this unfortunate social phenomenon which has been documented to exist not only across age ranges, but in a wide variety of different cultures around the world.
There is no medical reason for penis size to matter. And the fact is that men and women seem to differ on the question does penis size matter.
One of my big problems with the whole question is that it tries to homogenize us into one or two categories, when sexual tastes are far more complicated than that. The importance of penis size also seems tied to the idea that bigger must be better. One of the problems with this line of thinking is that we forget that bigger is comparative, and men are very bad at making accurate comparisons about penis size, in fact few men have an accurate idea of what is average penis size. And for that matter, medical researchers arent all that sure about average penis size either.

