This isn't a software bug, but since a lot of these aren't, I thought I'd share because it was a fun one with an unexpected cause.
I worked on a floor with about 10 people that was entirely occupied by phone switch equipment (raised floor, wires/racks, Halon fire suppression and big enough to seat several hundred people were it not for the equipment). For two weeks, about every 3 days or so in the middle of the night, the power would cut off. This was particularly surprising since the entire floor had dedicated battery and diesel backup (regularly checked/tested) and they never kicked on. Our facilities guy was going bald troubleshooting it -- brought in electricians and had the techs checking everything. There was just no explanation.
In a last ditch effort to try to get some information, he setup a laptop with a built in webcam and placed it high enough in the air so as to get most of the site[0].
A little history is necessary for the facility's design to make sense. At one point this room was used for our mainframe -- we were a local phone company and had a ton of data. This necessitated having a very elaborate near-line storage device custom built for the company. It consisted of a multi-million dollar robot (the exact kind you see on commercials building CARS, an arm about the size of an adult man coming out of the floor which ran on a track from one wall to about the middle of the space). It was enclosed in glass and would move tapes from a large shelving unit into drives and back but it was an open loop system: it never truly knew if it got a tape or if the tape made it to the drive and back and being an imperfect mechanical device, every once in a while it dropped a tape and someone would have to disable it, go in and pick the tape up off the floor (or, more often, the pieces of what was once a tape in some cases). This robot moved very fast and was very powerful so in a scenario where it's a person vs. "big moving robot"... well, there'd be pieces of person on the floor instead of tape. Since we liked our employees (and OSHA probably mandated it), the interior of the robot housing was filled with exposed "big red buttons" that would cut the power in an emergency. The exterior walls of the switch room had the same switches, though these buttons had a large acrylic cover with a hole in it so that you couldn't accidentally power down anything. A choice few of these killed power to the entire site and had a sign indicating that with something along the lines of "OH PLEASE GOD DON'T TOUCH THIS BUTTON"
Janitorial staff had been used to turning the lights out on their way out if they were left on and a new member of janitorial staff discovered, at some point, that hitting that big red button took care of all of the lights at once (along with all of the normally blinking LEDs on the thousands of switch cards, but hey -- it got dark at least!). So on his way out the door, he'd walk over to it, look at it for a second, then push it ... powering down ... everything.
The workaround was easy. We were now responsible for taking care of our garbage, dusting and cleaning from that point forward (which I think during my 7 or so years on that floor happened once) and a permanent camera was installed in the ceiling which was powered on a circuit not affected by the buttons. The buttons remained, though.
[0] I think after ruling out everything else he had suspected sabotage of some kind was responsible. Our doors used RFID badges and visitor logs were accessible, but at that time the doors that were interior to our office space didn't require badge access and there were no entries for the doors that one would have checked.
First thing that came to my mind - check the logs. What, no access logs for critical infrastructure, no physical access control, "anybody could use the door, no biggie"? I had a hunch about your issue from sentence 3 onwards - I thought the story "janitor unplugs server, plugs in vacuum, replugs server when done" was universally known. Apparently, "those who don't know history are doomed to repeat it." ;)
Yeah, that was the painful part. Almost nobody had access to that entire suite and those that did underwent stringent background checks and were very technical, so physical security once you were in the suite was limited.
IIRC, I believe it was discovered that the janitorial staff used building keys rather than the RFID locks so they weren't even logged when they arrived in suite.
I was a little surprised that hitting the emergency button didn't trigger an alarm of some kind, but that's how it was installed in the 80s and I'm fairly certain it's still that way, today (though I don't work there any longer).
Outside of those omissions, things really were kept in order: monthly battery tests, quarterly diesel/full system and disaster recovery tests. It's right when you think you have a solid process that someone comes along and pushes the wrong button, or burns some toast and triggers a floor evacuation/unexpected Halon test (that happened, too -- at some point they took away all of our nice things).
I worked on a floor with about 10 people that was entirely occupied by phone switch equipment (raised floor, wires/racks, Halon fire suppression and big enough to seat several hundred people were it not for the equipment). For two weeks, about every 3 days or so in the middle of the night, the power would cut off. This was particularly surprising since the entire floor had dedicated battery and diesel backup (regularly checked/tested) and they never kicked on. Our facilities guy was going bald troubleshooting it -- brought in electricians and had the techs checking everything. There was just no explanation.
In a last ditch effort to try to get some information, he setup a laptop with a built in webcam and placed it high enough in the air so as to get most of the site[0].
A little history is necessary for the facility's design to make sense. At one point this room was used for our mainframe -- we were a local phone company and had a ton of data. This necessitated having a very elaborate near-line storage device custom built for the company. It consisted of a multi-million dollar robot (the exact kind you see on commercials building CARS, an arm about the size of an adult man coming out of the floor which ran on a track from one wall to about the middle of the space). It was enclosed in glass and would move tapes from a large shelving unit into drives and back but it was an open loop system: it never truly knew if it got a tape or if the tape made it to the drive and back and being an imperfect mechanical device, every once in a while it dropped a tape and someone would have to disable it, go in and pick the tape up off the floor (or, more often, the pieces of what was once a tape in some cases). This robot moved very fast and was very powerful so in a scenario where it's a person vs. "big moving robot"... well, there'd be pieces of person on the floor instead of tape. Since we liked our employees (and OSHA probably mandated it), the interior of the robot housing was filled with exposed "big red buttons" that would cut the power in an emergency. The exterior walls of the switch room had the same switches, though these buttons had a large acrylic cover with a hole in it so that you couldn't accidentally power down anything. A choice few of these killed power to the entire site and had a sign indicating that with something along the lines of "OH PLEASE GOD DON'T TOUCH THIS BUTTON"
Janitorial staff had been used to turning the lights out on their way out if they were left on and a new member of janitorial staff discovered, at some point, that hitting that big red button took care of all of the lights at once (along with all of the normally blinking LEDs on the thousands of switch cards, but hey -- it got dark at least!). So on his way out the door, he'd walk over to it, look at it for a second, then push it ... powering down ... everything.
The workaround was easy. We were now responsible for taking care of our garbage, dusting and cleaning from that point forward (which I think during my 7 or so years on that floor happened once) and a permanent camera was installed in the ceiling which was powered on a circuit not affected by the buttons. The buttons remained, though.
[0] I think after ruling out everything else he had suspected sabotage of some kind was responsible. Our doors used RFID badges and visitor logs were accessible, but at that time the doors that were interior to our office space didn't require badge access and there were no entries for the doors that one would have checked.