Yeah it's sometimes hard to understand that evolution is not a perfection machine, it's not survival of the fittest, and it does not select the "best" outcome and discard the second best. It's survival of the least unfit, and it selects any option that doesn't cause the individual to die before reproducing. Just because our ancestors did something and nature didn't punish them with death because of it does not mean it's the optimal way to go about life. It merely means that doing that thing didn't cause our ancestor to die before their kids were born. Evolution doesn't select for perfection, just reproduction.
Same argument when someone says "when I was a kid we did X and we lived". Doesn't mean sleeping on the back dashboard of the car on the freeway is a good idea, just that you in particular did not die from doing so.
The point of that argument is never "This is perfect." It's exactly as you say - "This is not bad enough to cause serious harm and therefore" (they're saying) "not bad enough to get worked up over it."
"It survived evolution" = "It won't kill you" = "It's good enough"
Same argument when someone says "when I was a kid we did X and we lived". Doesn't mean sleeping on the back dashboard of the car on the freeway is a good idea, just that you in particular did not die from doing so.