KE isn't really relevant except in repeated braking.
It's air resistance scaling as F=kv^2 or P=kv^3 that does it. Doubling the effort drops time by just over 20% in the aero dominated regime. Perceived doubling of effort is more likely 30-50% more power which is 8-13% less time.
You're often faster saving those muscles for taking off from the lights or climbing the hill in a commute type situation than adding 2-7kph to the straight.
To go a little bit faster you need way more energy or vice-versa: going a bit slower saves you a lot of energy.