As you point out obsession with penis size seems to be mostly a guy thing. You aren't missing anything at all here, and I'd suggest treating it like a car accident, if there isn't anyone screaming for help all you can do is look straight ahead and keep going.
There is absolutely no medical reason for penis size to be any more or less important to someone than other physical attributes. Our bodies are capable of feeling pleasure in a million ways, and while a penis can be used to produce pleasure, frankly, it's not necessary and there's no size limit. There are more specific arguments you could make with people if you wanted to. For example, if you are using a penis for vaginal or anal intercourse most of the nerve endings and most sensitive spots are on the outside and just inside the vagina and rectum. So you don't need a big penis to produce pleasure there. And for the penis owner, there isn't any connection between size of penis and sensitivity.
It's true that some people like being penetrated by big things, but I've never heard of someone who was happy in their sex life in all ways other than penis size and decided to end it. Usually if penis size is an important issue, there are other sex issues in the mix.
One of my big problems with the whole question is that it tries to homogenize us into one or two categories, when sexual tastes are far more complicated than that. The importance of penis size also seems tied to the idea that bigger must be better. One of the problems with this line of thinking is that we forget that bigger is comparative, and men are very bad at making accurate comparisons about penis size, in fact few men have an accurate idea of what is average penis size. And for that matter, medical researchers aren't all that sure about average penis size either.