This is a race triggered by a particular reording of memory accesses as seen by different cores. It's the kind of thing that doesn't necessarily show up in a unit test anyway.
Exactly. This kind of bugs require stress testing to reveal. You might need to run it for minutes, hours or days to get it reproduced. And this bug might not come up with the hardware you are running.
Code dealing with memory barriers in SMP systems is non trivial to write, review and test. Everything is hardware specific, timing dependent and non-deterministic. Simple unit tests are useless for this kind of tasks, it needs stress testing on different hardware and a variety of workloads.
Reminds me of a passage from Tracy Kidders "Soul of a new Machine" [1]. The guys at Data General were implement one of the first pipelined 32bit processors for a mini computer in the late 70's. In the book Kidder talks about how they had a gate level simulator implemented in software that allowed them trouble shoot timing issues. Makes me wonder if a similar simulator could be useful to test and/or debug these types of issues.
Great book, highly recommend it. It won the Pulitzer.
In case you're not trolling: not really, not officially.
Unpopular features on less common architectures are frequently broken for large stretches of time, and go unnoticed until someone complains. Open source really exemplifies the squeaky wheel getting the grease, which is kind of sad.
Places where Linux is popular undoubtedly have their own internal private test suites, especially for features less popular on bleeding edge kernels (eg S390 arch support or Infiniband)
It would be hard to get any sort of good coverage with unit tests, too, but that shouldn't be a reason to avoid trying.
> It would be hard to get any sort of good coverage with unit tests, too, but that shouldn't be a reason to avoid trying.
Could a large but spotty unit test suite inspire false confidence that led to be being less careful about signing off on changes and thus decrease overall quality?
It is difficult to design meaningful unit tests for preemptively multitasking, protected memory operating system kernels. I don't know that it's actually impossible, but difficult.
While there are many advantages to unit testing, kernels are typically tested from userspace.
Some tests are laughably simple. I panicked the OS X kernel a while back with a shell script that repeatedly loaded and unloaded my kernel extension. Only a minute or two was required for the panic.
Apple fixed the panic but never told me how they screwed up.
EDIT: Of significant concern is how the kernel deals with the electrical circuitry. While the kernel is implemented in software, the reason we even have kernels, is so that end-user code doesn't have to understand much about physics.
AMCC - since acquired by LSI - sold some high-end RAID Host Bus Adapters. We had quite a significant problem with motherboard support. We had to test our cards on a whole bunch of different motherboards as well as PCI expansion chassis.
One might protest that "PCI is a standard!" but what we have is what we can buy at Microcenter. :-/
While not all of the kernel is concerned with physical hardware, much of it is. It's not really possible to write unit tests for the parts that have to deal with edge cases in electrical circuitry.
