CSS aggregation with Zen-based theme caused one particular server to stop serving Drupal pages

Weirdest. Server. Drupal. Zen. Bug. Ever. Our client's site suddenly stopped working. With a server error.

Safari can’t open the page. Safari can’t open the page “http://www.example.com/landing” because the server unexpectedly dropped the connection. This sometimes occurs when the server is busy. Wait for a few minutes, and then try again.

Opera was just a blank white screen. But other sites on the server were still working. Huh? Including a development version of the same exact site for which every single visitor-visable page somehow caused a server error. And admin pages on this site were still working. (We use an admin theme.) Exact same database, exact same code on our server worked fine and dandy. Rolling back the code and the database to 24 hours ago when things were definitely working did nothing. Changing the theme made the site work again (well, "serve up pages", not work in any acceptable business sense). Changing to non-zen based theme made things work. Same sitewide errors using Zen itself. Including a fresh-downloaded copy of Zen. (Everything made a bit more interesting by a bunch of table crash errors in the middle and not being sure what the client's managed hosting was or was not doing-- they said they wouldn't touch InnoDB, still wondering if they broke it before saying that. Fun stuff like this: warning: mysql_query() [function.mysql-query]: Unable to save result set in /usr/www/users/example/example-v3/drupal/includes/database.mysql.inc on line 108. user warning: Table 'system' is marked as crashed and should be repaired query: DELETE FROM system WHERE type = 'theme' in /usr/www/users/example/example-v3/drupal/modules/system/system.module on line 815. user warning: Duplicate entry 'themes/pushbutton/pushbutton.info' for key 1 query: INSERT INTO system (name, owner, info, type, filename, status, throttle, bootstrap) VALUES ('pushbutton', 'themes/engines/phptemplate/phptemplate.engine', ......) in /usr/www/users/example/example-v3/drupal/modules/system/system.module on line 822. ... Oh, and a red herring (maybe?) of watchdog errors indicating that Views could not include one of its files: include(./sites/default/themes/example/views/products/views-view-field--field-product-nid.tpl.php) [function.include]: failed to open stream: No such file or directory in /usr/www/users/genarts/example-v3/drupal/includes/theme.inc on line 1020. Well, that file really wasn't there (but, one cache clear -- and believe me, i did dozens, views cache, everything -- and the error should have been gone). Put the file there, Views couldn't find it's own stuff, which was there: require_once(./sites/all/modules/views/includes/view.inc) fatal error in /usr/www/users/genarts/example-v3/drupal/sites/all/modules/views/views.module Finally, after hours of this sort of randomly disabling features, modules like Views, disabling boost, messing with .htaccess, and hacking at our theme and then discovering any Zen involvement caused it, all without anywhere near adequate access to error logs or support from Pair (for their managed dedicated server, this just retroactively justified years of us managing our own servers), i stumbled onto the immediate cause. CSS aggregation. Disable CSS aggregation, and it worked again.

Drupal can automatically optimize external resources like CSS and JavaScript, which can reduce both the size and number of requests made to your website. CSS files can be aggregated and compressed into a single file, while JavaScript files are aggregated (but not compressed). These optional optimizations may reduce server load, bandwidth requirements, and page loading times. These options are disabled if you have not set up your files directory, or if your download method is set to private. Optimize CSS files: Disabled Enabled This option can interfere with theme development and should only be enabled in a production environment.

Gzipping not the cause. JS aggregation not a problem. Let's go over that again. Something that Pair changed on their server at about 11 a.m. Caused any use of the Zen theme (directly or as a base theme) to trigger some weird server error only when CSS aggregation was enabled. Random googling way before reaching this level of understanding of the cause of the problem suggested that a site somehow sending a character or something that is somehow suspect could make mod_security put the kibosh on the whole thing. That's still the only theory i have on this whole thing.

Resolution

The cause: Drupal core bug Optimize CSS option causes php cgi to segfault in pcre function "match" Hat tip to: CSS aggregation fails with Zen theme...