More troubling to me than the closed source firmware is that the bug in TFA seems like something that the most basic of a test suite should be catching.
It’s reminiscent of Apple’s “goto fail” lack of certificate checking - another easily testable case that simply wasn’t.
The test authors don’t even need to be on the same team/manager. They can just write black box tests to the spec, like the author of this post did.
I’m not even some big TDD guy. It just seems to me that in these core security-critical libraries/functions that should be pretty side-effect-free that you should have some basic “receive x, produce y” functional tests to make sure the API is doing what it claims to do on the tin.
the most basic of a test suite should be catching.
A most basic test suite is not likely to wait some arbitrary amount of time (2 seconds, as the author found by trial and error) between calls to the HSM.
The images in the 'Digging deeper' section suggest otherwise. They appear to show a successful run followed by one that fails because the 'encrypted' value is garbage. Where am I missing the instantly reproducible failure?
In what way? 'Temporal' fuzzing to an eon-like range of two seconds seems, naively at least, entirely impractical.
Edit: a somewhat different way of putting this concretely - what is a practical stochastic testing regime that can reasonably be expected to find this bug?
Well designed systems have a mockable clock and timer subsystem.
They could easily test "Do X, wait 100 years, do Y".
You find all kinds of wired bugs when you do that - things that poll, for example Cron daily, will have to be run 36500 times. Certificates expire. Counters overflow. Date systems can't convert the date to and from strings. Logfiles get too big. Etc.
The premise was that basic automated testing by some other team at Google (to whom the HSM is a black box) would catch this. I don't see how that's obvious. Then you're all 'but fuzzing!' and I'm like 'wat' and now you're asking me if I know you can set the clock on the HSM? I don't think I know that. It's a black box.
It’s reminiscent of Apple’s “goto fail” lack of certificate checking - another easily testable case that simply wasn’t.
The test authors don’t even need to be on the same team/manager. They can just write black box tests to the spec, like the author of this post did.
I’m not even some big TDD guy. It just seems to me that in these core security-critical libraries/functions that should be pretty side-effect-free that you should have some basic “receive x, produce y” functional tests to make sure the API is doing what it claims to do on the tin.