I have mod_fcgid for my PHP sites, and after my Lucid upgrade I noticed that PHP uploads such as images, photos and documents were being corrupted quite frequently, and it was happening on both WordPress and Simple Machines Forum (SMF). I spent a lot of time troubleshooting it to no avail. I was about to give up and either roll back to Hardy LTS or put PHP on Windows and run my sites there when I stumbled across this bug report: Uploads greater than 64 kilobytes corrupted when using Apache mod_fcgid

Lucid LTS’s mod_fcgid has this bug, and to automatically get the patch you have to have security.ubuntu.com in your /etc/apt/sources.list, for example:

deb http://security.ubuntu.com/ubuntu lucid-security universe

Then run updates and upgrades.

I suppose it’s partially my fault for not having security there in the first place, but I thought the security patches would eventually show up on the distribution mirrors. Apparently not yet. And also this seems more like a bug fix than a security issue.

The hosted VPS was an old stable Debian version running under Virtuozzo with a 2.4 kernel. For years it has been running flawlessly. During a recent upgrade it started installing a new glibc and other programs relying on a 2.6 kernel. I guess my old Debian stable went out of support, but I’m surprised it tried going 2.6. Of course I have no control over the kernel, so I was in a bad spot. The critical services were still running, so I kept them running. (I am pretty sure most services would fail upon restart.) I had some problems with mismatched libraries failing, but overall the web server, DNS servers and sites kept running. Since I was planning on migrating to my home server anyway, I decided to keep it limping along while I prepared my home server environment.

My home environment was a Hardy 8.04 LTS Ubuntu server running OpenVZ containers. This has been stable for at least a couple of years, but in anticipation of moving my web server into a container there I decided to upgrade to Lucid 10.04 LTS. My mistake was not previewing major changes that would affect me. Each one probably deserves its own blog, so I’ll summarize here for now.

OpenVZ deprecated, LXC is the new paravirtualization component. Good news: it’s in the main kernel tree. Bad news: it’s different enough from OpenVZ to introduce a new learning curve, and the documentation and support tools are immature as of mid-2011.

mod_fcgid behaves differently: On my small VPS I had apache and mod_fcgid configured for optimal efficiency in a small RAM footprint, but the same configuration options caused failures on the new mod_fcgid. The short story is that mod_fcgid became an Apache project in 2009, and they’ve made some changes. They’re actually decent changes, but they don’t work with my old config options when trying to minimize process count and RAM usage. mod_fcgid will now by default avoid closing the first three processes for each class, “class” being defined in my case as the php file called in the URL. Since RAM is not an issue on the home server I removed my process count constraints, but I intend to toy with the settings until I can get it like I used to have it: one to four php_cgi processes handling all php requests.

SysV init -> Upstart init: Lucid init is a hybrid between the new Upstart init and the old Sys V init. In many ways I like this, and when installing Ubuntu packages this wouldn’t really be an issue, but inside the LXC containers it has thrown me for a few loops because Upstart keys on events that may not happen inside a container the same way they do on the host. The good news is that in-container init scripts and service starts that key on filesystem mounting and network interface availability can be changed to “start on startup” since you can assume a container will have its filesystem and network set up before the init system is called. In particular you might want to be sure that /etc/init/rc-sysinit.conf is changed to “start on startup” inside LXC containers, and I had to do the same for /etc/init/mysql.conf .

I run several Debian-based Linux machines and virtual machines at home and periodically install or reinstall one to test something. They all need updates—and mostly the same updates—so I wanted to cache the updates locally rather than download them several times when I upgrade.

There is an apt-proxy package, and although I can’t recall the problems with it I remember deciding it was not going to work well for me. I could rsync the entire package archive, but that’s just wasteful. I finally decided on setting up a Squid proxy dedicated—by intent, not controls—to caching deb packages from Debian and Ubuntu archives. And rpm’s and such if I should use other distro’s.

So I set up Squid and looked through the configuration options. Squid is by default set up to be most efficient at getting cache hits. I wanted to be sure it doesn’t expire the seldom-accessed large deb files to make room for tiny files, so I changed the cache replacement policy to LFUDA to optimize byte hit rate. I also increased the maximum object size to 100 megabytes from the default 4096 kilobytes. In a typical Squid cache the larger files aren’t cached because they often aren’t requested as often as smaller files by web surfers, however my cache’s purpose is to save these large files locally for updating several machines.

Now I needed to make my machines use the proxy for apt. For that I just added a code snippet to each /etc/apt/apt.conf, or in my cases I just slipped this file named jimproxy into /etc/apt/apt.conf.d/ :

Acquire {
        Retries "0";
        HTTP {
                Proxy "http://address-or-URL-of-squid-proxy.example.tld:3128/";
        };
};

Now when I run apt or aptitude or any manager that uses apt, they will use my Squid proxy to obtain the distribution packages.

This worked quite well, but I recently noticed some problems. The issue appeared to be that there were missing deb files from the archives, but what really was happening was that new Package.bz2 lists were on the archives, but my Squid cache was serving older lists it had cached. It listed some older packages which were no longer there. So my “apt-get update” would read an old package list and then “apt-get -u upgrade” wouldn’t find those older packages. So I need to tell Squid to be sure to check for new package lists. To do that I changed the refresh pattern option for “refresh-ims”. Voilà, it works properly now.

Squid.conf lines before:

# maximum_object_size 4096 KB
# cache_replacement_policy lru
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern .               0       20%     4320

Squid.conf lines after:

maximum_object_size 100 MB
cache_replacement_policy heap LFUDA
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern .               0       20%     4320 refresh-ims

I turned on refresh-ims for everything, but I probably would have been fine with turning it on for just the frequently-changing files as shown in the following code. But in my case I don’t think turning it on for all files will adversely affect things.

maximum_object_size 100 MB
cache_replacement_policy heap LFUDA
refresh_pattern ^ftp:          1440    20%     10080
refresh_pattern ^gopher:       1440    0%      1440
refresh_pattern Packages\.bz2$ 0       20%     4320 refresh-ims
refresh_pattern Sources\.bz2$  0       20%     4320 refresh-ims
refresh_pattern Release\.gpg$  0       20%     4320 refresh-ims
refresh_pattern Release$       0       20%     4320 refresh-ims
refresh_pattern .              0       20%     4320

You may also be interested in using Squid in web accelerator mode in a small VPS to boost performance.