The server got wedged: it couldn't write to the log file because there were too many files open. Still not sure why there were so many files open. I've never seen that error before. But the program that notices when the server is wedged and kills it didn't catch it in this case, because instead of not responding it would respond but with an empty page.

What was the exact error? What kind of files were open? If you want to share, I'm sure lots of us would be interested to hear the details so we know what to look out for.

Sounds like the limit for the maximum number of open File Descriptors (FDs) was reached. On Linux, you can check the limit by doing:

  cat /proc/sys/fs/file-max

You can change this limit:

  echo 100000 > /proc/sys/fs/file-max

This limit is also available through the "fs.file-max" sysctl.

The "lsof" tool is great for seeing what FDs a process has open.

Note that FDs are used for more than just regular files: directories, sockets, pipes, and other file-like objects too.

Also ulimit and /etc/security/limits.conf

The default for max open files (1024) is ridiculously low. First thing to do when installing linux, up the max open file limits.

netstat is also handy here (e.g. sometimes you find a single remote IP is holding open way more sockets than it should be).

It looks as if most of the file descriptors in use at any given time refer to closed tcp connections. We're still trying to figure out what this means.

Last time I saw too many open files errors was because of too many open sockets (some piece of code was leaking socket handles).

This was the bane of my existence at one point.

"I've never seen that error before."

I feel like nothing better describes keeping a web app up and running than Murphy's Law. I amazed at how many little things have developed over time and temporarily brought down my sites over the years.

In other news, productivity across the globe spiked today...

i have to admit, i only found myself slightly more productive because i was distracted wondering what was wrong with HN and periodically checking to see if it was back

You should ask for you money back.

One good reason to ask about downtime is to learn from other people's troubles and about their tools.

Welcome back. I missed you.

us hackers wouldn't get paid much if computers always behaved themselves.

Is fixing someone's mess really "hacking"?

People will need new stuff regardless of whether or not the old stuff works -- writing the new stuff is "hacking".

Hacking's meaning naturally stems from manipulating previously existing stuff.

Had the same exact problem about 10 days ago, as well.