Thought Torrent - HDD and mod_fcgid Frustration

I talked about having trouble with the new hardware for Dent. Since that post, I've:

• checked the partition tables for overlap, twice
• run badblocks -w over every single partition
• made a RAID 5 at least three times and a RAID 0 once
• put ext3 on the RAIDs at least twice per construction
• tried both the Gentoo 2005.1 and Knoppix 4.0.2 LiveCDs

I finally got some I/O errors from the kernel (dmesg) while using Knoppix. I am absolutely certain that reiserfsck did not stand a chance of recovering the filesystem. The data was corrupted, usually silently, and e2fsck failed every time. One time ext3 freaked out and remounted the filesystem as read-only after printing some errors about an aborted journal (yay).

The only remaining step is to put two of the drives in my desktop and see if I get corruption there. My motherboard has built-in support for SATA, cutting out the possibly bad controller, and the power supply (and so on) are obviously different.

Yesterday, I started setting up mod_fcgid on the atrus.org server. Using it should improve performance—dropping Apache's footprint from 13MB to 8MB by ridding it of mod_php baggage—and allow me to run rails apps.

It works beautifully on my Gentoo desktop; I haven't seen a zombie process from it yet. Granted, my desktop isn't a high-traffic server :p However, mod_fcgid fails to do anything useful in Debian Sarge. PHP scripts just die. Ruby on Rails applications, uh, don't run at all. Instead the source is served up as text/plain. I've fiddled with the configuration for all day and not even LogLevel debug is telling me anything useful.

Pertinent version info:
apache2 2.0.54-5
libapache2-mod-fcgid 1.05-1 (also tried 1.08 from source)
php4 4.3.10 (unpatched, built from source to enable fastcgi, also tried 4.4.1)
ruby 1.8.2-1

Partial /etc directory. No, you may not have my SSL/TLS cert ;)

Example of PHP death:
[Thu Feb 16 04:32:05 2006] [notice] Apache/2.0.54 (Debian GNU/Linux) mod_ssl/2.0.54 OpenSSL/0.9.7e configured -- resuming normal operations
[Thu Feb 16 04:32:05 2006] [info] Server built: Sep 5 2005 11:15:09
[Thu Feb 16 04:32:05 2006] [debug] /home/adconrad/build/apache2/security/sarge/apache2-2.0.54/build-tree/apache2/server/mpm/prefork/prefork.c(956): AcceptMutex: sysv
(default: sysvsem)
[Thu Feb 16 04:32:09 2006] [notice] mod_fcgid: server /home/atrus/websites/atrus/www/hosted/aan/index.php(21813) started
[Thu Feb 16 04:32:09 2006] [error] [client 129.2.229.48] Premature end of script headers: index.php
[Thu Feb 16 04:32:12 2006] [notice] mod_fcgid: process /home/atrus/websites/atrus/www/hosted/aan/index.php(21813) exit(communication error), terminated by calling ex
), return code: 255

My php.ini has a file for error_log, but it's empty.

I assume I'm doing something stupid ... If you spot what's wrong, that's great. I'll also happily take debugging tips. My current idea is to make an alternate config that would run one process listening to an odd port and using gdb, strace, or whatever. It seems like a gigantic pain in the ass, but I really need more information.

[edit: Rails is working now. I just needed to comment out AddHandler fastcgi-script .fcgi in the generated public/.htaccess. It clobbers the correct handler (fcgid-script)]