I have mod_fcgid for my PHP sites, and after my Lucid upgrade I noticed that PHP uploads such as images, photos and documents were being corrupted quite frequently, and it was happening on both WordPress and Simple Machines Forum (SMF). I spent a lot of time troubleshooting it to no avail. I was about to give up and either roll back to Hardy LTS or put PHP on Windows and run my sites there when I stumbled across this bug report: Uploads greater than 64 kilobytes corrupted when using Apache mod_fcgid

Lucid LTS’s mod_fcgid has this bug, and to automatically get the patch you have to have security.ubuntu.com in your /etc/apt/sources.list, for example:

deb http://security.ubuntu.com/ubuntu lucid-security universe

Then run updates and upgrades.

I suppose it’s partially my fault for not having security there in the first place, but I thought the security patches would eventually show up on the distribution mirrors. Apparently not yet. And also this seems more like a bug fix than a security issue.

The hosted VPS was an old stable Debian version running under Virtuozzo with a 2.4 kernel. For years it has been running flawlessly. During a recent upgrade it started installing a new glibc and other programs relying on a 2.6 kernel. I guess my old Debian stable went out of support, but I’m surprised it tried going 2.6. Of course I have no control over the kernel, so I was in a bad spot. The critical services were still running, so I kept them running. (I am pretty sure most services would fail upon restart.) I had some problems with mismatched libraries failing, but overall the web server, DNS servers and sites kept running. Since I was planning on migrating to my home server anyway, I decided to keep it limping along while I prepared my home server environment.

My home environment was a Hardy 8.04 LTS Ubuntu server running OpenVZ containers. This has been stable for at least a couple of years, but in anticipation of moving my web server into a container there I decided to upgrade to Lucid 10.04 LTS. My mistake was not previewing major changes that would affect me. Each one probably deserves its own blog, so I’ll summarize here for now.

OpenVZ deprecated, LXC is the new paravirtualization component. Good news: it’s in the main kernel tree. Bad news: it’s different enough from OpenVZ to introduce a new learning curve, and the documentation and support tools are immature as of mid-2011.

mod_fcgid behaves differently: On my small VPS I had apache and mod_fcgid configured for optimal efficiency in a small RAM footprint, but the same configuration options caused failures on the new mod_fcgid. The short story is that mod_fcgid became an Apache project in 2009, and they’ve made some changes. They’re actually decent changes, but they don’t work with my old config options when trying to minimize process count and RAM usage. mod_fcgid will now by default avoid closing the first three processes for each class, “class” being defined in my case as the php file called in the URL. Since RAM is not an issue on the home server I removed my process count constraints, but I intend to toy with the settings until I can get it like I used to have it: one to four php_cgi processes handling all php requests.

SysV init -> Upstart init: Lucid init is a hybrid between the new Upstart init and the old Sys V init. In many ways I like this, and when installing Ubuntu packages this wouldn’t really be an issue, but inside the LXC containers it has thrown me for a few loops because Upstart keys on events that may not happen inside a container the same way they do on the host. The good news is that in-container init scripts and service starts that key on filesystem mounting and network interface availability can be changed to “start on startup” since you can assume a container will have its filesystem and network set up before the init system is called. In particular you might want to be sure that /etc/init/rc-sysinit.conf is changed to “start on startup” inside LXC containers, and I had to do the same for /etc/init/mysql.conf .

I haven’t yet converted my real web server from Apache/mod_fcgid to Apache/mod_fastgi with an externally spawned FastCGI process. It is doable and works fine, but it is a pain to configure it how I want it to work.

I’m not sure I can blame Apache or FastCGI. It actually makes sense how they are doing it. However, I tend to install PHP software packages like Drupal, Simple Machines Forum, Mambo/Joomla, Gallery2, WordPress and so forth. Generally these applications are built with Apache/mod_php in mind and take some tweaking to make work on other setups. But I don’t want to tweak, especially if I have to do it periodically when upgrading.

I switched from lighttpd/ModFastCGI to Apache/mod_php/Squid because I was tired of fiddling with lighttpd rewrites to make pretty urls for Drupal and SMF which include ready-made Apache pretty rewrite rules.

Apache/mod_fcgid has worked pretty well so far, and with the AddHandler directive it behaves like mod_php: Any .php files under the document root will be parsed by the PHP interpreter. The downside is that any caching–like APC–is not shared across the PHP processes.

So I wanted to spawn my own PHP FastCGI process with several child processes sharing a cache like I did with lighttpd and then have Apache use it. That works, but the mod_fastcgi FastCgiExternalServer directive behaves much like a ScriptAlias directive: if you specify a folder, anything under that folder will be processed by the FastCGI server, so you can’t easily mix static files with PHP script files. I tried hacking my way around this with various combinations of symlinks, aliases, handlers and rewrites, and I even got a couple of the combinations to work, but I would have to make changes to each Apache virtual host (or enforce a strict naming scheme for my vhosts), and if I missed something it would be too easy to accidentally serve up the source of a .php file.

Now if I were making an application from scratch it would be fine to put the PHP files in a script folder and the static files elsewhere, and I could alter premade software like that with rather simple aliases, but I would have to do it for every installed instance and then re-tweak every time I updated the software. No thanks.

I was able to spawn a separate PHP FastCGI server with children and have Apache connect to it. It was trickier than I thought. The big benefit is that one shared APC cache can serve all the PHP child processes and being able to use a multithreaded Apache without worrying about whether my PHP dependencies are thread safe.

The trick isn’t getting it working, but getting it working the way I wanted. I want “.php” files to be processed by the FastCGI server and have the other files sent by Apache. Without some tricky configuration, Apache’s mod_fastcgi can only send specified file requests or specified directories–plus all their contents–to the external FastCGI server.

But I am getting ahead of myself. Let me back up to my old lighttpd setup: I had lighttpd installed, and a script that launced several php-cgi processes and listened on a network socket. Lighttpd would connect to the php-cgi processes and let them handle PHP processing. Apache can do this, too, but it was hard for me to easily find out how online.

As it turns out, the spawn-fcgi program from lighttpd that I used to start the FastCGI server is now a project on its own. Supposedly the mod_fastcgi developers have a launcher program, too, but I couldn’t easily find it, and I was already familiar with spawn-fcgi and was happy to see it’s being maintained. I downloaded the source package from the site, extracted it and then did the usual “./compile”, “make” and “sudo make install”. So now I have /usr/local/bin/spawn-fcgi installed.

There is some good info on lighttpd’s ModFastCGI documentation site on launching a PHP server with spawn-fcgi and various helper scripts. I modified one slightly to make it use a unix socket instead of a network tcp socket:

#!/bin/bash

## ABSOLUTE path to the spawn-fcgi binary
SPAWNFCGI="/usr/local/bin/spawn-fcgi"

## ABSOLUTE path to the PHP binary
FCGIPROGRAM="/usr/bin/php-cgi"

## TCP port to which to bind on localhost
FCGIPORT="1026"

## bind to unix domain socket
FCGISOCKET="/tmp/php.sock"

## number of PHP children to spawn
PHP_FCGI_CHILDREN=4

## maximum number of requests a single PHP process can serve before it is restarted
PHP_FCGI_MAX_REQUESTS=1000

## IP addresses from which PHP should access server connections
FCGI_WEB_SERVER_ADDRS="127.0.0.1"

# allowed environment variables, separated by spaces
ALLOWED_ENV="ORACLE_HOME PATH USER"

## if this script is run as root, switch to the following user
USERID=www-data
GROUPID=www-data

################## no config below this line

if test x$PHP_FCGI_CHILDREN = x; then
  PHP_FCGI_CHILDREN=5
fi

export PHP_FCGI_MAX_REQUESTS
export FCGI_WEB_SERVER_ADDRS

ALLOWED_ENV="$ALLOWED_ENV PHP_FCGI_MAX_REQUESTS FCGI_WEB_SERVER_ADDRS"

### This if-then-else is for opening a network TCP port
#if test x$UID = x0; then
#  EX="$SPAWNFCGI -n -p $FCGIPORT -f $FCGIPROGRAM -u $USERID -g $GROUPID -C $PHP_FCGI_CHILDREN"
#else
#  EX="$SPAWNFCGI -n -p $FCGIPORT -f $FCGIPROGRAM -C $PHP_FCGI_CHILDREN"
#fi

### This if-then-else is for opening a unix socket
if test x$UID = x0; then
  EX="$SPAWNFCGI -n -s $FCGISOCKET -f $FCGIPROGRAM -u $USERID -g $GROUPID -C $PHP_FCGI_CHILDREN"
else
  EX="$SPAWNFCGI -n -s $FCGISOCKET -f $FCGIPROGRAM -C $PHP_FCGI_CHILDREN"
fi

# copy the allowed environment variables
E=

for i in $ALLOWED_ENV; do
  E="$E $i=${!i}"
done

# clean the environment and set up a new one
exec env - $E $EX

In the above script I had to use /bin/bash instead of Ubuntu’s default /bin/sh as it uses some of bash’s features. Also note that with spawn-fcgi you can have a network tcp socket or a unix socket, but not both. On my test server I just simply ran the above script as root; it won’t restart itself if the VPS is restarted or if the script crashes. I have daemontools on my real server, and I’ll use that to start and monitor the launcher script. The link to lighttpd’s site has other startup scripts worht looking at.

You can’t use mod_fcgid to connect to the externally spawned FastCGI process. It can only launch and manage the processes itself. So I loaded mod_fastcgi and used the FastCgiExternalServer directive:

<IfModule mod_fastcgi.c>
  FastCgiExternalServer /srv/www/site/fcgi -socket /tmp/php.sock
</IfModule>

That tells Apache that any request under the /srv/www/site/fcgi directory gets passed to the FastCGI process with a unix socket at /tmp/php.sock. Unfortunately there is not an simple configuration to have it just run php files, and the FastCGI server may not know what to do with static files like pictures or .css files.

There is a good article explaining the FastCgiExternalServer directive. Its solution to having just the .php files be handled by the external server involve adding a handler, assigning an action to the handler pointing to a nonexistent script and then aliasing the nonexistent script back to a folder symlinked to the original directory. The only way I could find to simplify that was to use a ReWriteRule. In either case we need to unfortunately modify the configuration for each vhost to make it work.

I have several vhosts under /srv/www/. Following the articles example I created a symlink /srv/fcgi pointing to /srv/www . Then I modified my mod_fastcgi configuration as such:

<IfModule mod_fastcgi.c>
  FastCgiExternalServer /srv/fcgi -socket /tmp/php.sock
  ReWriteEngine On
  ReWriteCond %{DOCUMENT_ROOT} ^/srv/www/(.*)
  ReWriteRule ^/(.*\.php(3|4)?(\?.*)?)$ /srv/fcgi/%1/$1
</IfModule>

Now the external FastCGI server is invoked whenever a file under /srv/fcgi is accessed, but /srv/fcgi is just a symlink to /srv/www. Instead of the above article’s gyrations I figured out the above rewrite rules that will rewrite any request for a .php file to /srv/fcgi/(rest-of-document-root)/(request_URL) . So in effect the rewrite points back to the original file, but through a symlink that makes Apache use the FastCGI server to process it. The ReWriteCond shown doesn’t actually make a decision; it is giving me a reference to use when constructing my rewritten path name.

Now I have to modify my vhosts. Rewrite rules don’t carry over to vhosts by default. For each VirtualHost section I have to add the following which allows the server rewrite rules inherit to the vhost:

ReWriteEngine On
ReWriteOptions Inherit

Alternately I could just put the rewrite rules in each VirtualHost section. In fact I may need to if I have other rewrite rules for pretty URLs.

With FastCGI–whether externally spawned or managed by mod_fcgid or mod_fastcgi–you also need ExecCGI enabled in the Options directive.

I used Apache benchmark and verified that all the child proceses are being used concurrently. And now the APC cache is shared among all the child processes.

July 2011 Note: mod_fcgid has changed its behavior in opening and closing processes since I wrote this article. mod_fcgid is now an Apache project, and limiting it to 4 processes with the configuration I have below caused me problems when I migrated to a server with the newer mod_fcgid. mod_fcgid now opens a new process for each “class” which for our php purposes means each .php file location requested by a URL, and it tries to keep 3 processes open for each class by default. Since I limited the max processes to 4, it will not close enough idle processes to allow new classes to open processes, and the pages fail. I think this may be fixable with settings, but I had other upgrade frustrations during the migration and am not now constrained by a small VPS, so I haven’t yet been motivated to figure out how to get the newer mod_fcgid to operate well in a small VPS.

My best-performing small VPS setup was with lighttpd and FastCGI PHP, but I got tired of trying to make rewrites work in lighttpd and switched to a two-process prefork Apache with mod_php and Squid as a web accelerator. That worked pretty well, but not as fast as lighttpd and FastCGI. What I really want is a multithreaded Apache and a FastCGI PHP that will fit in my small, cheap VPS.

I had tried Apache’s worker MPM and FastCGI before, but at the time both Apache and the PHP FastCGI process bloated and took up all my RAM despite my settings. Recently I decided to try again and was able to find out how to make it work.

Under Linux, by default each thread is assigned 8MB of stack memory, so an Apache process with 25 threads would try to take up 25*8=200MB of RAM!!! Plus the size of the Apache parent process, plus anything else that runs on my VPS. Not going to work in my small VPS. However, each thread doesn’t really need that much RAM. In fact, 128k is working fine for me so far. Apache 2.2 has a new directive ThreadStackSize for the worker MPM, and I set mine to “ThreadStackSize 131072″, and now I can have two Apache processes with 25 threads each taking up about 25MB worth of privvmpages. Another way to accomplish this is add “ulimit -s 128″ to the Apache startup scripts. For Apache 2.0 you have to do it this way. Since I am using Apache 2.2 I didn’t have to use ulimit, but when I was testing the effects of changing the stack size I used this script which worked as a temporary measure:

#!/bin/sh

ulimit -s 128
/usr/sbin/invoke-rc.d apache2 restart

My 25 MB RAM usage above is without mod_php, though. Google searches lead to conflicting information about whether PHP is thread safe, so I want to use FastCGI. My problem with Apache FastCGI before was that it spawned several times as many PHP processes as I thought I had told it to. I was using mod_fcgid and pointing it to the same FastCGI PHP wrapper script that I had used for lighttpd. But that script set PHP to launch child processes, and I have since learned that mod_fcgid does not multiplex and therefore will not use the child processes. Instead it launches as many processes as it sees fit, and my configuration had each of those launching 4 children. No wonder my RAM got chewed up so quickly. So now I am letting mod_fcgid call /usr/bin/php-cgi directly:

<IfModule mod_fcgid.c>
        AddHandler fcgid-script .fcgi .php
        # Where to look for the php.ini file?
        DefaultInitEnv PHPRC        "/etc/php5/cgi"
        # Maximum requests a process should handle before it is terminated
        MaxRequestsPerProcess       1000
        # Maximum number of PHP processes
        MaxProcessCount             4
        # Number of seconds of idle time before a php-cgi process is terminated
        IPCCommTimeout              120
        IdleTimeout                 120
        #Or use this if you use the file above
        FCGIWrapper /usr/bin/php-cgi .php
</IfModule>

Unfortunately, since each PHP processes is launched separately, any caching such as eAccelerator or APC will not be shared across each process. And each process uses up another X MB of RAM for the cache. So if I’m using APC with the default 30 MB cache and have 4 FastCGI PHP processes going, my APC caches are taking up 120 MB all by themselves! At the moment this is exactly what I’m doing, because I’ve moved up from a 256 MB VPS to a 390 MB VPS, and my total memory usage seems to be hovering near 256 MB when all processes are running. However, when PHP processes aren’t needed, mod_fcgid will kill them off to save memory, so most of the time I’m using much less RAM. I will see if I can set up the FastCGI processes like I did with lighttpd and then connect to it from Apache. I think it’s doable, but I haven’t tried yet.

I like using the worker MPM and FastCGI better than using the prefork MPM, mod_php and Squid. First of all the log files are a lot easier to parse. Apache (as I have it configured) can handle 50 concurrent static requests including 4 concurrent PHP requests, and I was able to enable KeepAlives again so my sites feel more responsive. With prefork Apache and Squid I could see a row of GIF smileys load up left-to-right when posting a reply in one of my forums. Now it happens so fast I can’t see it anymore. If I can get the PHP cache–previously eAccelerator, but I just switched to APC–to share itself across all my PHP processes then the overal RAM usage will be much lower, and I’ll be able to have more PHP processes.

And I like the new setup better than lighttpd because I can use the Apache rewrite rules provided by software programs like Drupal and SMF rather than try to translate them into lighttpd rewrites.