When deciding to create the IT Know-It-All site I reviewed my options. My goal was to write about my IT experiences—discovered solutions, lab tests, and reports on my research—to share them and to create a navigable knowledgebase for myself. I had originally envisioned a site with a knowledgebase engine, but later realized  blog with search and tags would work just fine for my purposes and be easier to update and maintain. Wordpress does this “out of the box” with no additional modules, is quite popular and well-maintained, so I chose Wordpress for the new site.

I found no reliably simple way to import two Drupal sites’ articles into Wordpress. I didn’t have that many articles, and none with photos or other media, so I viewed them with Firefox with “no style” chosen and simply copied each article to the clipboard and then pasted it into a new Wordpress post on the new site, then adjusted the publish date. That worked surprisingly well. (I then set up redirects, but that is out-of-scope for this post.)

So far I really like using Wordpress. Drupal can do more and is more flexible, but without thinking about it I had reduced my Drupal sites to a level of simplicity that Wordpress does better. I have two more simple Drupal blog sites, and I think I will convert them to Wordpress, too. My Drupal sites are at least one major version behind because upgrading would cause module issues, so I might as well update (for my purposes) to Wordpress.

For a year or two I was successfully running Drupal with lighttpd and fastcgi. Lighttpd is very efficient, and having 4 fastcgi PHP processes let me limit the memory php used while keeping my small sites responsive enough. But redirects and URL rewriting are done differently, and it was tricky at times to get it to work the way I wanted with Drupal. I eventually got tired of wrestling with rewrites and redirects with every new non-Drupal php-based app I wanted to try, so I started thinking about how to make Apache work for a small site.

Now I am running Apache2/mod_php with a Squid front-end cache. If you’re not familiar with Squid, in brief it is a web proxy that caches web requests. It is often used to speed up clients but can be reversed to cache requests on the server end in web accelerator mode. Apache uses more RAM than lighttpd/fastcgi, and Squid uses RAM for itself and the cache, so I had to cut back two 2 Apache processes.

That may sound like too little, but here’s how it works well: the threads aren’t stuck delivering a request to a slow remote client because the local Squid cache accepts the request locally and then delivers it to the client freeing up the Apache process to handle the next request. 2 processes can handle my traffic because they can quickly deliver their payload to the cache and move on, and Squid can handle the delivery over the network to the client.

Of course sometimes the processes get hung up on slow MySQL queries or slow PHP queries, so occasionally I had delays. In particular I got rid of my RSS requests, because the Apache process had to wait while requesting RSS feeds from other sites, and sometimes that got slow or even timed out leaving just one Apache process handling requests, and if it stumbled on something then I’d have client requests waiting in line not getting served. So I got rid of my news feeds. Note that I am talking about my web site pulling feeds from other sites; of course I can offer RSS feeds for my Drupal blog with no such PHP delays. I also strive to keep my MySQL running smoothly, but everyone should do that, anyway.

The downside of using Squid is logging. Since Squid is my front-end web server I have Apache listen on 127.0.0.1:80, and Squid accesses it there. So my Apache log files show all requests coming from 127.0.0.1, and many static page or image requests don’t come through because Squid has them cached. However I configured Squid to log in Apache log format and just use those logs instead.

Of course dynamic content from Drupal has the nocache header, so Squid isn’t caching the dynamic content for future requests, but it still frees up Apache while delivering it to the client. It does cache the static files like images, style sheets and javascript files, so the Apache threads mostly focus on dynamic content only.

Another way I keep memory usage down is with eaccelerator. It caches PHP scripts so they don’t have to recompile every time they’re run. This can save memory in addition to processor time. After changing Drupal or any of my scripts I usually delete the cache and click around my sites to force all the php to run so eaccelerator will cache it. Then I restart my php processes (Apache2 in the case of Apache/mod-php or the fastcgi server if using fastcgi) to lower their memory usage. After that the cached scripts should run and the PHP processes shouldn’t bloat as much. Note that every time PHP is updated eaccelerator must be recompiled. In older PHP versions it would crash if you didn’t, but now it just silently (except for a log entry) fails to cache your scripts if you forget to recompile after a PHP update.

With lighttpd/fastcgi I was able to run 4 PHP processes (memory_limit from 8MB – 16MB), lighttpd, MySQL and Exim (my mail daemon) in a 256mb VPS with good speed. With Apache2/mod_php I am running 2 Apache2/mod_php processes, Squid (8 MB cache memory), MySQL and Exim in a 256mb VPS. Having only two processes forces me to watch for slow requests like a hawk, but Squid takes care of slow clients. I still ran into memory problems occasionally, but now I have a 384mb VPS and haven’t had a privvm failure yet.

I’m currently running Drupal 4.6.6, Drupal 4.7 RC3 and CivicSpace 0.8.3 . I preferred to have one set of source files, but I wanted each site (Apache vhost) to have its own root folder.

So, with some symlink magic, here’s what I do. Each version of Drupal has its own directory under my servers web root. I have the following directories and symlinks:

• drupal-4.6.6
• drupal-4.7.0-rc3
• civicspace-0.8.3

• drupal-4.6 -> drupal-4.6.6
• drupal-4.7 -> drupal-4.7.0-rc3
• civicspace -> civicspace-0.8.3

Each vhost has its own DocumentRoot with the following symlinks, substituting drupal-4.6 or civicspace for drupal-4.7 as appropriate:

• cron.php -> ../drupal-4.7/cron.php
• database -> ../drupal-4.7/database
• includes -> ../drupal-4.7/includes
• index.php -> ../drupal-4.7/index.php
• misc -> ../drupal-4.7/misc
• modules -> ../drupal-4.7/modules
• scripts -> ../drupal-4.7/scripts
• sites -> ../drupal-4.7/sites
• themes -> ../drupal-4.7/themes
• update.php -> ../drupal-4.7/update.php
• xmlrpc.php -> ../drupal-4.7/xmlrpc.php

Each vhost gets its own files/ directory, robots.txt, .htaccess and favicon.ico files.

Now, when the next version (e.g. 4.7.1) is released I can unpack the files, copy the sites folder from the previous version, and then update the version symlink (drupal-4.7 -> drupal-4.7.1) and run the update.php script.

If I want to change a particular site from Drupal 4.6 or CivicSpace 0.8.3 to Drupal 4.7 I update the symlinks in the DocumentRoot to point to the 4.7 series, copy the appropriate site config file over and run the update.php script.

This is working well so far, but in my recent “Memory Hogging” blog I mention that having too many modules installed (even if not activated) make the admin/modules screen take up tons of RAM. My sites have different modules needs, so I think I’m going to give each vhost its own modules directory and then symlink the modules under that. This way I’ll have only one module version per drupal version in my filesystem and be able to remove modules I know don’t need from individual sites.

BBCode: (Input format / filter.) I love this module. Most of my intended audience is familiar with BBSes that use BBCode. With auto-url recognition and line breaks this is almost perfect for my needs. Perfect would include formatting buttons. Note that it also allows image links. (Images are so complicated in Drupal, but that’s another topic…)

BBCode_wysiwyg: (Formatting buttons.) Sounded like a great addition to the BBCode module, but it’s just 4 pushbuttons that don’t work as the buttons on the forums. I quickly turned it off. phpBB is GPL; I wonder how hard it would be to use their formatting toolbuttons for Drupal? UPDATE: I looked at phpBB’s form, and it’s also just html pushbuttons. I had remembered it being fancier.

Textile: (Input format / filter, included with CivicSpace.) Cute idea. *This text bolded* _this text underlined_ ; simple markup to produce valid xhtml. I especially like the links: “link text”:http://URL . Two problems: 1: it’s different than what my audience is used to. 2: you have to use white space to separate the markup from other text, so a sentence ending in bold, for example, would have a space before the period. Or comma, etc. That bugs me. I converted all content to BBCode and disabled Textile.

Short note on altering input formats: While playing with input formats I wound up with existing content using several different types of input formats. I wanted to simplify choices for users to BBCode or Plain Text (yeah, it’s a tad redundant), but in doing so it turns out the users can’t edit content created in that input format! A node administrator has to edit each bit of content and change the input format (and formatting, if necessary) to allow the users to edit their content again. (This is only if the users can’t currently use the input format the content was originally created in.) Moral of the story: Decide early in your site history what input options you want to use. Alternately offer tons of input types, but my users aren’t generally tech savvy and confuse with too many choices.

tagadelic: Very nifty. It lists your categories sorted as you want, but categories with more nodes in them have a bigger font indicating more content. I’ve only peeked at this module a little bit. At first it seemed like it shows all terms for all vocabularies, but then I noticed it created a nested menu item with links for each vocabulary. I wasn’t ready for that to be there but couldn’t disable the stupid thing! I got clever and moved it to my Secondary Links menu (Drupal 4.7) which I don’t display on my site.

votingapi: I just got this. It doesn’t do anything on its own, but it is the power behind simple voting and latest and greatest. After upgrading to Drupal 4.7 I played with free tagging taxonomies and was disappointed to learn you can’t have users (or guests) apply tags to existing content (a la rating/moderating/evaluating). I think I can work up something with votingapi to do this, though.

TinyMCE: (Formatting toolbar.) This thing and I got off to a bad start and just didn’t hit it off. I turned it off pretty quickly and haven’t tried it since. It reformatted my text box and removed all line/paragraph breaks in the box and in the output. I tried this as one of the first things I did with CivicSpace, and now that I know more about input formatting I may have to give this a try again as I’d really like some basic formatting buttons for my posters. UPDATE: I’m highly suspicious of an HTML editor, though…hackers could bypass the editor in a hand-crafted POST request. I’m much more comfortable using BBCode and escaping all HTML. Besides, I don’t really care for WYSIWYG; I just want an easy-to-use toolbar.

Although a relative newbie, I have some observations on CivicSpace 0.8.3 as compared to a base Drupal install:

• The “plain text” input filter allows unfiltered html!
• The bug list isn’t closely monitored. I submitted the unfiltered html issue to CivicSpace as a bug, and 4 1/2 days later it is still unassigned and has no replies.

• The zadministration panel has a cute front page but mucks up the administration navigation menu in a way that’s confusing to me, different from Drupal (and therefore Drupal documentation and discussion) and when you turn off the zadministration module your administer settings are scattered amongst the top level of your navigation menu; in other words you don’t go back to Drupal normal admin when disabling zadministration. EDIT Dec 20, 2006: I just noticed how to avoid this problem when disabling zadministration: Before disabling the zadmin module, from the zadmin panel, click “Activate/Deactivate this page” and then push the “Deactivate” button. Then the admin menus return to their Drupal state and you can safely disable the zadmin module.
• CivicSpace seems to be a memory hog compared to Drupal. CivicSpace recommends setting php.ini to use 24M of memory, but on my 256M virtual server I keep running out of VM pages when I do that. I’ve tried different settings and had problems at 8M and 12M; I’m at 16M now but I haven’t had a problem with a Drupal site at 8M with the same modules enabled yet. I think the enabled statistics module may be to blame, but I’m not sure yet. (I have CivicCRM turned off. I could also probably tune apache and MySql to not use as much RAM, but I haven’t tried that yet.)
• After upgrading CivicSpace to Drupal 4.7 rc3 with a mix of Drupal 4.7 and cvs modules, the “my profile” page has some odd characters on it.

These past two or three weeks I’ve been working hard on figuring out CivicSpace / Drupal. I’m ditching CivicSpace 0.8.3–based on Drupal 4.6–and going with Drupal 4.7 rc3.

Drupal 4.7 is very new. In fact it hasn’t been officially released yet. But there are enough changes I think it’s important to go ahead and switch now while my websites are young and shy on content. The big feature for me is the free tagging additions to taxonomy. I don’t fully understand these yet but have some ideas on how I want to use them. As I understand it, the backported free tagging / folksonomy patches for 4.6 are sufficiently different from 4.7 as to cause upgraded headaches.

At first CivicSpace sounded like an easy-to-install Drupal distribution that included a robust default configuration with popular modules. I had used Drupal 4.6, but hadn’t quite figured it out, and CivicSpace seemed slicker. Perhaps it helped; I don’t know. But I’ve already outgrown the need for the preselected modules. As far as the easy install, it was easy, but I didn’t have a problem installing Drupal in the first place. As I learned to use CivicSpace I decided I didn’t like their “zadministration” modified administration menu, didn’t use the features (CivicCRM) that set CivicSpace apart from Drupal plus modules and started to suspect that CivicSpace is enough different from Drupal that “upgrading” to Drupal could be a problem, especially with established content.

So, on the the upgrade. I installed the drupal 4.7 files in a new directory, copied my files/ directory and configuration directory over and checked the .htaccess and robots.txt files. I downloaded and installed the modules that were enabled in CivicSpace but not included with Drupal. Some modules have a 4.7 version, some I had to get the cvs build. Actually I left out the zadministration panel module because I couldn’t find it on Drupal’s site, and I hate it, anyway.

EDIT Dec 20, 2006: I just noticed how to avoid the administration menu problem when disabling zadministration: Before disabling the zadmin module, from the zadmin panel, click “Activate/Deactivate this page” and then push the “Deactivate” button. Then the admin menus return to their Drupal state and you can safely disable the zadmin module. (/EDIT)

Drupal 4.6/CivicSpace don’t have a “site maintenance” mode; I wanted to prevent possible changes to the database during the upgrade. I found suggestions on using conditional apache redirects, but it occurred to me I could just remove authenticated users’ permission to access content. To make a friendly message, I created a new block for the top of the left column explaining the site was briefly down for maintenance. After enabling that and disabling “access content” under the node section of the access control page everybody except user #1 gets “Access Denied”, the login box and my site mx message block. I patted myself on the back for being so clever and knowing how to do this.

I backed up the databse. The files hadn’t changed since the daily backup, and I didn’t intend to delete or overwrite anything, so I didn’t do any specific file backup. I logged in as user #1 as per the upgrade instructions. I disabled the zadministration module because I wasn’t going to use it in 4.7. I used the unix “mv” command to move the existing install to an old/ directory and move the new directory in its place. Already logged in as user #1, I browsed to /update.php . Drupal 4.7 is pretty slick in that it has upgrade hooks for its modules, meaning you don’t need to update modules individually. This screen handles upgrading the datbase for the system and all hooked modules.

That was easy, and then the site worked fine. I poked around a bit and then reenabled user access to content and disabled the site mx message block. There were a couple of oddities at first, but they cleared and were apparently caching issues. Only two issues remained: the admin menu was messed up due to disabling zadministration; the primary links were missing (I don’t use secondary).

EDIT Dec 20, 2006: The zadministration menu issues in the previous and following paragraphs are avoidable by “deactivating” zadmin before disabling the module. See my earlier edit above for details. I haven’t upgraded a deactivated zadmin site yet, so I’m not sure yet if the Primary Links problem will still happen.(/EDIT)

These issues are actually the same. Drupal 4.7’s main menu is called “Primary Links” and is propogated to the Primary Links area on the theme as well. But due to the zadministration influence your “upgraded” menu doesn’t fit the scheme. I had done a Drupal 4.6->4.7 upgrade before the CivicSpace upgrade, so I knew what the menu should look like. Rather than rebuild it by hand I used mysqldump and mysql to dump the menu table from the “good” site’s menu and import it into my upgraded CS site, overwriting the existing structure. (Lots of backups and careful typing here, of course.) It worked great! I then had to delete an item I had added to the other site’s menu and manually add links to my aggregator sources. Interestingly it automatically deleted the invalid aggregator source links that were imported from the other board. After this my navigation menu, administration menu and Primary Links along the top of the page were working spiffy and the menu adapted to new modules I enabled. UPDATE: Drupal 4.7 doesn’t add menu items under aggregator/sources anymore; that’s probably part of why I had to manually re-add the menu items after doing my menu trick. Come to think of it, I think I’ll take the aggregator/sources subitems back off again as I have too many feeds to make sense of; I’ll set up the categories instead.

The only two remaining glitches I know of are that the profile page is slightly garbled and the “1,2,3 next page” links are squished together for some reason. The squished links are on my 4.6->4.7 upgrade site, too, so I don’t know if that’s an rc3 issue yet or an upgrade artifact. UPDATED: The “squished links” were apparently also a caching issue (cached CSS file?). It looks fine now for both the 4.6->4.7 site and the CivicSpace->Drupal 4.7 site, and I haven’t changed anything.