I'm really glad people don't leave comments for downvoting. Then this forum would be littered (heh) with completely useless 3-word comments "downvoted for x".
It's not that I'm so conserned with my points really. Just sometimes it sucks if people disapprove of what you said and you're not sure why. It's usually enough if just one person leaves a comment. Sometimes these things are just hard for me to interpret and it frustrates me a bit.
Be less concerned with your internet points.