I run several Debian-based Linux machines and virtual machines at home and periodically install or reinstall one to test something. They all need updates—and mostly the same updates—so I wanted to cache the updates locally rather than download them several times when I upgrade.
There is an apt-proxy package, and although I can’t recall the problems with it I remember deciding it was not going to work well for me. I could rsync the entire package archive, but that’s just wasteful. I finally decided on setting up a Squid proxy dedicated—by intent, not controls—to caching deb packages from Debian and Ubuntu archives. And rpm’s and such if I should use other distro’s.
So I set up Squid and looked through the configuration options. Squid is by default set up to be most efficient at getting cache hits. I wanted to be sure it doesn’t expire the seldom-accessed large deb files to make room for tiny files, so I changed the cache replacement policy to LFUDA to optimize byte hit rate. I also increased the maximum object size to 100 megabytes from the default 4096 kilobytes. In a typical Squid cache the larger files aren’t cached because they often aren’t requested as often as smaller files by web surfers, however my cache’s purpose is to save these large files locally for updating several machines.
Now I needed to make my machines use the proxy for apt. For that I just added a code snippet to each /etc/apt/apt.conf, or in my cases I just slipped this file named jimproxy into /etc/apt/apt.conf.d/ :
Acquire {
Retries "0";
HTTP {
Proxy "http://address-or-URL-of-squid-proxy.example.tld:3128/";
};
};
Now when I run apt or aptitude or any manager that uses apt, they will use my Squid proxy to obtain the distribution packages.
This worked quite well, but I recently noticed some problems. The issue appeared to be that there were missing deb files from the archives, but what really was happening was that new Package.bz2 lists were on the archives, but my Squid cache was serving older lists it had cached. It listed some older packages which were no longer there. So my “apt-get update” would read an old package list and then “apt-get -u upgrade” wouldn’t find those older packages. So I need to tell Squid to be sure to check for new package lists. To do that I changed the refresh pattern option for “refresh-ims”. Voilà, it works properly now.
Squid.conf lines before:
# maximum_object_size 4096 KB # cache_replacement_policy lru refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern . 0 20% 4320
Squid.conf lines after:
maximum_object_size 100 MB cache_replacement_policy heap LFUDA refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern . 0 20% 4320 refresh-ims
I turned on refresh-ims for everything, but I probably would have been fine with turning it on for just the frequently-changing files as shown in the following code. But in my case I don’t think turning it on for all files will adversely affect things.
maximum_object_size 100 MB cache_replacement_policy heap LFUDA refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern Packages\.bz2$ 0 20% 4320 refresh-ims refresh_pattern Sources\.bz2$ 0 20% 4320 refresh-ims refresh_pattern Release\.gpg$ 0 20% 4320 refresh-ims refresh_pattern Release$ 0 20% 4320 refresh-ims refresh_pattern . 0 20% 4320