If empathy were a limited resource that demanded to be spent only in descending order of the seriousness of the potential injury, then you might have a point.
Time is a limited resource. But fair point, as one could spend less time being unsympathetic towards others and instead spend it more valuably being empathetic towards legitimate causes.