>people downloading public keys over insecure networks and not verifying them at all

And note that SSL download public keys over insecure networks. Verifying them is why certificates were created