• Searches insist on changing my search query – I cannot search for e.g. “Home Depot” because search substitutes the nearest home depot location for my search query. I am often not looking for the closest location, and sometimes autocomplete misses what I mean entirely
• Gallery can’t find my photos I’ve taken before the last reboot, even though they are still in storage
• Netflix app does not work, it freezes on a black screen when I start it
• Calendar quit syncing with Exchange

I am very angry to say that the reset has not fixed the search problem. I turned off Google search suggestions in Settings -> Search -> Searchable Items, but freakin’ Maps insists on not letting me search for what I want to search for. Sooner or later I will break my phone in anger over this. Now I am live-blogging my search for a fix: Settings -> Language & Keyboard -> Touch Input -> Text input: uncheck Prediction and Word completion. I try “home depot” and “fedex” in maps and in both cases it searches for what it thinks I want instead of what I typed. >:( This phone may not survive the rest of this blog post. I go into settings and clear data from the app. No help. Time to go scream bloody murder on some Google/Android help forums.

Okay, next issue: I fire up Gallery. WTF?!?! There are images here. I suppose the fact that camera shots from before the factory reset indicate the index problem is solved, but I clearly told it to wipe all phone storage and SD card, and clearly it didn’t. It has my camera photos, my videos and some photos that were attached to SMS messages. It is not showing the icons and other non-gallery stuff it was showing before the reset. I wonder if it only wiped its own private storage areas and left the rest of the browseable storage alone? Well, I am confused, but I will call this issue fixed.

The Netflix app works after the reset. I was able to log in and start a movie.

My calendar wouldn’t sync with my last employer’s Exchange for the past couple of months even though it used to do so, but since I just left that job I can’t test to see if it works now.

So in summary the most infuriating issue is still with me, so it is apparently the way the search software is written, or perhaps there is a third place to look for how to change autocomplete/autocorrect. I find it very curious and unexpected that my files (including music, movies, documents and photos) are still on the phone and SD storage after all the warnings that all storage would be wiped out.

I now realize I did not even think to backup my call history. Probably not important, but with my last employer I used this phone for work a lot, and it was occasionally very useful to use call history to seen when I last spoke with someone. And now if I get a missed call that I don’t recognize, I can’t see if I have spoken with that number in the past. This might be a bigger deal if I weren’t changing employers right now.

I will not restore my old SMS messages by choice, but they are backed up in an XML file in Google docs if I need them.

Restoring bookmarks was pretty easy. I just searched for the “easy” sites (e.g. NOAA Weather, ESPN) and then read my email to click the links I emailed myself as backup, then bookmarked those (longer URL) links (e.g. Weather for my area, ESPN Dallas). Of course the “favorite” section is not populated yet, but that will restore itself over time. I didn’t care to back up or restore browser history or any passwords or cookies I had.

I took photos of my home screen layout, and I will manually set it back up as I use each app. I might look at the photos, or I might just icons and widgets where I feel like it at the time. Since my phone and app use is changing, my placement priority may change, too.

I backed up my files manually, and I don’t see a pressing need to copy any back to the phone just now. I had some music and movies on there before, but I usually use another device for playing media. I’ll probably load some entertainment on it for my next vacation, but for now I’ll only copy files when I need them. My documents were related to my old job, so no need to put those back on the phone.

Since I used the same Google account on the phone as I did before the reset, my Maps saved locations/My Places are back with no further action on my part.

I noticed some of my built-in apps are out-of-date after the reset, so before reinstalling my downloaded apps I am updating the built-in apps. Interesting: Amazon MP3 is installed; I didn’t think it was a built-in app. After updates I go to My Android Apps on Google Play (formerly Android Market) and start clicking the seemingly misinformed “installed” link and directing each app to install on my phone. I skipped a few that I no longer need or trust. They are installing to my phone with no further action on my part. Cool!

I will manually set up my Tweetdeck settings today and add my WiFi encryption keys and passphrases as I revist my WiFi spots.

Now for the real question: Did this fix my issues?

After a minute or so a “Sorry!” dialog came up about a service running and asked me if I wanted to wait or force close. While I was pondering the conundrum it started to reboot itself. For a while a rather interesting icon of some green arrows in a circle above an image of a droid with light coming out of it appears, but it went away and rebooted before I could get the webcam set up to take a photo.

The next reboot looks normal but is taking longer than usual.

After only 4-5 minutes it asks me which language and then tries to activate itself. It is unable to do so, then I recall I have a network extender, so I unplug it in case activation can’t be completed through the extender. Sure enough, now that the extender is off the activation call goes through, I complete activation and the phone reboots. (I have good signal, but I bought a network extender because my last residence had poor signal.)

It asks again for my language, offers a tutorial on use and starts asking questions about settings, eventually getting to Google account and all the features that require all the companies in the world to track me everywhere I go.

Now it’s time to restore data!

Even though I have autocomplete turned off, the phone will always substitute what it thinks I meant when searching. This really makes me angry when searching maps and the internet. If I search for “Home Depot” it for some reason substitutes the name and address of the nearest Home Depot when I hit enter, when those results are almost always different than what I wanted. It does the same thing for internet search; I cannot search for what I type in, I can only search for what it thinks I meant. The other big issue is that it does not index any photos I took since the phone last rebooted. They are all still in storage, but Gallery does not show them. I tried moving all the photos off, deleting the thumbnails and any index-looking files I found, but that has not helped. As far as Gallery knows the only camera photos that exist were taken since last reboot. It does go and find all the random images, icons and photos elsewhere in storage, though.

A couple of other issues are that the Netflix app quit working and the calendar stopped syncing with Exchange. But I didn’t use the phone for Netflix much, and changing jobs removes the need to sync the work calendar. I’m not sure if these issues coincided with the last Android update or if they happened at a different time.

I have done some internet searching and tried some things, but I have been unable to fix it. In the corporate PC world the fix for all desktop software issues is to reimage the PC, so I figured this would eventually require a factory reset of the phone which would make me lose all my apps, contacts, bookmarks, settings, SMS messages and home screen layout. I have been dreading the thought.

This weekend I am between jobs as I just finished my employment Friday and will start with my new employer Monday, so now my phone can forget all the data and sync settings for my old company’s email and saved documents. My new company will issue me a company phone, so my personal Android phone usage will now change greatly. Also, I traveled a lot in the old job but will travel very little in the new job, so the application use pattern for my Droid will change a lot.

Changing jobs and roles doesn’t require me to reset the phone; I could just delete the data and accounts associated with the old company. But give that I already have problems, why not do the factory reset now and start fresh?

In the Windows PC world, when you migrate or reimage a user’s PC you generally catch everything the user wants by backing up My Documents, Favorites and Desktop from their profile, and then ensuring their local email is backed up which for Outlook is usually under application data or local settings and for Notes is usually under C:\Notes. (Bonus points for hunting down and backing desktop wallpaper, Outlook signature/proof/stationery files, email autocomplete settings and quicklaunch bar shortcuts.) So, I organized my thoughts about what I want to back up from my Android and worked on doing so:

• SMS Messages – Not critical, but I have occasionally found myself referring to old texts
• Contacts – Gotta have these backed up or I wouldn’t know how to get hold of anyone!
• Bookmarks – Not critical, but annoying to not keep them
• Home screen layout – This sounds whiny, but you really do get used to a layout, and 20-40 icons and widgets are a pain to recreate
• Files – I have photos, music downloads, movie downloads, eBooks and a lot of document downloads.
• Maps saved locations – I would rather back up and keep my starred locations
• Downloaded apps – I would like to have a list of apps I downloaded so I know to reinstall them again
• Tweetdeck settings – I want my Tweetdeck to look the same
• WiFi encryption keys and passphrases – I thought about this right before resetting my phone. I can reenter the phrases manually, but a couple of them are a pain on the on-screen keyboard. I don’t see an immediate and easy way to view or copy them for backup, though.
• What I forgot – I’m adding this after restoring my phone: My phone call history would have been nice to keep if I thought about it beforehand. I don’t care about the following items, but others might: browser “favorites” sites, media playlists, browser history, downloaded-but-later-deleted apps (Google Play shows all previously downloaded apps whether they’ve been deleted or not).

So there is what I would like to back up and, in some cases, restore. Now, how am I going about doing this? There seems to be no built-in, catch-all backup for all these things. But since some are provided by the phone manufacturer, some by the service carrier, some by Google and some by third parties, I guess that makes sense. There are many backup apps, specialized and broad, free and for-pay. I’m not going to review all of them. Heck, I didn’t even look at more than two or three. I’m just going to write about what I did.

You would think SMS messages would be found somewhere on the phone storage or sandisk storage and be accessible when hooking the droid up to a PC and browsing. But I never found them. So I found and used a free app SMS Backup & Restore to back up my text messages to an XML file and then uploaded the XML file to Google Docs. I could have emailed the file, copied over USB or one of many other transfers. I don’t intend to restore my SMS messages after the reset, but the app will let you read a backup file just like it was your main SMS messages, so I will be able to read the messages without restoring. There is probably a way to view the XML messages on the PC, but I haven’t even bothered to try yet.

There are many options for contacts, so many that any given user may already have theirs unknowingly backed up. First, note that contacts are a combined list of contacts from multiple possible sources: phone contacts, Google account contacts and mail app contacts. Google account contacts sync automatically by default and therefore are already “backed up” to Google. (These contacts are viewable online in Google Mail / GMail.) Mail app contacts should be synced and therefore “backed up” with the mail server, in my case my old company’s Exchange server, but I deleted that account on my last day. The phone contacts need to be backed up, though. There is a “Backup Assistant” under settings -> Accounts and Sync that runs automatically by default, but as far as I can tell you have to pay the cell phone service provider a fee to restore that data using Backup Assistant. I used HTC Sync which is installable when I plug my phone in via USB, but that version didn’t work anymore so I downloaded the one from HTC’s site. It allowed me to sync my contacts with Outlook. The old version said it could sync to Windows 7 Contacts, but it wouldn’t sync and the new version didn’t offer to sync to Windows 7 Contacts. Oh well.

I don’t have many bookmarks in my Android browser, but there were a few I wanted to be sure to keep. I didn’t want to install an app to back them up, I couldn’t find the bookmarks while browsing through phone storage and I saw no option to export the list. So I just wrote down what I had bookmarked (ESPN Dallas, NOAA Weather, other easy-to-recreate bookmarks), but there were a couple of links to articles I wanted to refer back to. Instead of writing those down, I touched-and-held the bookmarked, chose “share” and emailed the link to myself so I can later read the email, click the link and bookmark it. It’s not as simple as export-import, but it’s simple enough for as few bookmarks as I have, and I don’t have to install an app to do it.

Home screen layout: I think what I am going to do is use a webcam and just take a photo of each screen and then recreate it manually. Since my app use will change with the job changes, it might make sense to rearrange things, anyway.

I back up all the files by hooking the phone to my PC via USB, enabling drive access and just copying everything over in Windows Explorer. All photos, music, documents and videos are locatable with browsing and searching, but phone settings, app settings and app data don’t seem to be represented in the copied files. But I have a lot of data files, so this is definitely worth doing. I won’t need all the files after the reset, so I’ll selectively restore individual files. It should be noted that I manage my music and video files myself. If you are using an application to organize them and make playlists you may want to be sure those things are backed up.

Perhaps the easiest backup for me was to realize that my Maps saved locations/starred places are already synced in my Google Maps with the Google account I use on the phone. (My Places -> Starred in Google Maps.) Assuming I use the same Google account after resetting the phone, these places will be “restored”.

Getting a list of downloaded apps was also easier than expected. Google Play (formerly Android Market) has My Android Apps which shows all the apps I’ve ever downloaded. But it currently shows them as “installed” even if I have uninstalled them on the phone. No biggie, because if I click on “installed” the next screen knows whether or not it’s actually installed and lets me install it there. So my plan is to reinstall apps using this web page.

For my Tweetdeck settings I think I’ll just list the columns I have and recreate them manually. I do not use a Tweetdeck account, just Twitter accounts, otherwise it might remember my accounts for me.

So there we go. I’ve backed up my data as best as I think I can (or am willing to do), I’ve blogged about it, so now I’ll perform the factory reset. And write another blog about that.

• PXE Boot: There is no Preboot eXecutable Environment boot standard for IPv6 yet, but there are some developments towards this goal. PXE v2.1 is a 1999 non-open standard, and it appears to me the momentum to move away from the PXE standard and add other abilities to network boot such as HTTP, iSCSI and other alternative file transport and the ability for one network boot standard to work with multiple architectures.
• Intel’s response to this subject indicates they expect network IPv6 booting to be done via UEFI. I am seeing UEFI BIOS in newer business x86-64 workstations, but I do not know if they are currently capable of IPv6 remote boot. UEFI options I’ve seen are disabled by default in BIOS.
• What used to be etherboot—an open-source network boot firmware / chain-loadable boot program—became gPXE, and in 2010 it forked into gPXE and iPXE. iPXE’s site says all new development is being done on iPXE, but there was a Google Summer of Code project to add IPv6 boot to gPXE, so I am currently confused as to which project to try first if I were prototyping IPv6 network booting.
• In my view, dual-stack IPv6-IPv4 networking will be a reality for at least several years, so I expect IPv4 PXE boot to be the norm until IPv6 network boot standards are mature.
• NAT: Network Address Translation was created to slow down IPv4 address exhaustion, so it is not needed for IPv6. However, many users seem to think that NAT enhances security (I largely disagree), and some have tried to develop a form of NAT during the IPv4-to-IPv6 transition phase so a group of IPv4-only hosts might communicate over a NAT device with IPv6-only hosts. But the transition NAT attempts have run into problems and aren’t considered general-purpose transition solutions. Ideally all hosts on the internet can directly address each other, so NAT should disappear when IPv4 does.
• WINS: Windows Internet Name Service maps NetBIOS names to IPv4 addresses, but Microsoft has moved to DNS for client-server name resolution and is developing PNRP for peer name resolution. Do not expect WINS to be implemented for IPv6 name resolution or to use IPv6 to transport queries.

Original article published February 1, 2010, 10:00 AM, later edited for updates.

My SixXS tunnel won’t accept packets from my HE subnet, and my HE tunnel won’t accept packets from my SixXS subnet, but I can only route by destination…with normal routing.

The answer—besides living happily with one subnet and tunnel—is policy routing. With policy routing I can set up more than one routing table and choose which routing table to use based on the source of the packet, so I should be able to send SixXS-subnet-sourced packets to the SixXS tunnel and HE-subnet-sourced packets to the HE tunnel.

Unfortunately my DD-WRT version doesn’t support policy routing for IPv6. So I designated my Linux server as the DMZ host under the DMZ tab under the NAT / QoS tab and set up my tunnels and routing on the Linux box.

Instead of setting up a piece at a time and testing it as I go, I brazenly typed up my whole configuration before trying any of it out on the Linux box. That cost me a few hours of troubleshooting, but the funny part is I mostly had it right the first time. The problem was that policy rule flushing works slightly differently with IPv4 and IPv6, at least on Ubuntu Lucid 10.04.2.

The default rules are as shown:

$ /sbin/ip -6 rule show
0: from all lookup local
32766: from all lookup main

The “gotcha” is that I decided to flush the rules before adding my own. I did this because I was duplicating rules when I brought the interface down and up again.

$ /sbin/ip -6 rule flush
$ /sbin/ip -6 rule show
0: from all lookup local

Flush works a little too well as it takes away the rule that uses the main routing table! Flushing the IPv4 rules puts the default rules back, but not so with IPv6. So when flushing IPv6 rules remember to add the main rule back:

$ /sbin/ip -6 rule flush
$ /sbin/ip -6 rule add priority 32766 from all table main

Here is my working policy routing setup that routes to the proper tunnel based on the source address. I have chosen to set up the HE tunnel and subnet as normal and make policy routing decisions for the SixXS tunnel. Other things I did to make this work was to enable IPv6 routing in sysctl and add “200 sixxs” to /etc/iproute2/rt_tables so I could have a routing table named “sixxs”, but I could have used a numbered table instead. Ubuntu/Debian /etc/network/interfaces file (partial):

auto he-ipv6
iface he-ipv6 inet6 v4tunnel
        endpoint 216.218.224.42
        address 2001:470:1f0e:b56::2
        netmask 64
        ttl 64
          # Null route HE /48 to prevent sending back to internet
        post-up /sbin/ip -6 route add unreachable 2001:470:b967::/48 || true
          # Null route HE /64 to prevent sending back to internet
        post-up /sbin/ip -6 route add unreachable 2001:470:1f0f:b56::/64 || true
          # Global unicast range route (effective default)
        post-up /sbin/ip route add 2000::/3 dev he-ipv6 src 2001:470:1f0e:b56::2 || true

auto sixxs
iface sixxs inet6 v4tunnel
        endpoint 216.14.98.22
        address 2001:4978:f:178::2
        netmask 64
        ttl 64
        # doesn't work: mtu 1280
        post-up /sbin/ip link set $IFACE mtu 1280 || true
          # Null route SixXS /48 to prevent sending back to internet
        post-up /sbin/ip -6 route add unreachable 2001:4978:192::/48 || true
          # Set up SixXS routing table
        post-up /sbin/ip -6 route flush table sixxs || true
        post-up /sbin/ip -6 route add 2000::/3 dev $IFACE src 2001:4978:f:178::2 table sixxs || true
          # Set up routing table rules, he-ipv6 is default, sixxs is sixxs-sourced
        post-up /sbin/ip -6 rule flush || true
          # To local prefixes, use main routing table
        post-up /sbin/ip -6 rule add priority 100 to 2001:4978:192::/48 table main || true
        post-up /sbin/ip -6 rule add priority 200 to 2001:470:b967::/48 table main || true
        post-up /sbin/ip -6 rule add priority 300 to 2001:470:1f0f:b56::/64 table main || true
          # To nonroutable "global" prefixes, use main routing table
            # 6to4. Uncomment this if I implement local 6to4 conversion
        #post-up /sbin/ip -6 rule add priority 400 to 2002::/16 table main || true
            # Teredo. Uncomment this if I implement local Teredo conversion
        #post-up /sbin/ip -6 rule add priority 500 to 2001::/32 table main || true
            # Documentation: range reserved for dummy addy's in documentation
        post-up /sbin/ip -6 rule add priority 600 to 2001:db8::/32 table main || true
          # From SixXS subnet addr to global unicast (last rule), use sixxs routing table
        post-up /sbin/ip -6 rule add priority 32000 from 2001:4978:192::/48 to 2000::/3 table sixxs || true
          # Need the default main; flushing seems to delete the ip6 main rule
        post-up /sbin/ip -6 rule add priority 32766 from all table main || true
          # Flush routing cache to enable new routing info
        post-up /sbin/ip -6 route flush cache || true
          # Flush rules and re-add main table rule
        post-down /sbin/ip -6 rule flush || true
        post-down /sbin/ip -6 rule add priority 32766 from all table main || true

I null-route my assigned prefixes/subnets to avoid sending inappropriate traffic back into the internet which will just get routed right back to me. When I add routes for my /64 prefixes they will override the null route because they more specifically match the destination addresses. I am also routing to 2000::/3 (all currently assigned global unicast addresses) instead of a default route so I don’t send out multicasts or other address ranges I don’t intend.

If you’re wondering about “|| true”, if I don’t have that and the command generates an error code, the interface set-up would stop. The “|| true” causes the command to return a “success” code to the ifup/ifdown scripts even if the command fails so that the ifup/ifdown script can continue with the rest of the setup.

My plan is that the main routing table will handle all cases except from the SixXS subnet to  the SixXS tunnel, in my case from the subnet to external global unicast addresses. I could just make sure each table has all the routes I need, but in my case I think it’s easier this way. Here are the resulting rules from the above configuration:

$ ip -6 rule
0:      from all lookup local
100:    from all to 2001:4978:192::/48 lookup main
200:    from all to 2001:470:b967::/48 lookup main
300:    from all to 2001:470:1f0f:b56::/64 lookup main
600:    from all to 2001:db8::/32 lookup main
32000:  from 2001:4978:192::/48 to 2000::/3 lookup sixxs
32766:  from all lookup main

Rules 100, 200 and 300 are my local prefixes. They don’t need to go out the SixXS tunnel, so I divert them to the main routing table. Rule 600 is the reserved documentation prefix. I think I may try to use that address range when making how-to videos, so I want to divert that to the main routing table and null route it there to prevent sending invalid traffic out to the Internet. Rule 32000 catches all remaining packets from my SixXS subnet that are to the global unicast address range and tells Linux to use the routing table I named “sixxs”. The last rule is what sends everything else to the main routing table. The last rule should exist by default, but “ip -6 rule flush” deletes it, and that’s the “gotcha” that cost me a few hours’ troubleshooting.

The sixxs routing table as configured above:

$ ip -6 route show table sixxs
2000::/3 dev sixxs  metric 1024  mtu 1280 advmss 1220 hoplimit 4294967295

Downloads:
/videos/view-http-headers-with-chrome.mp4
/videos/view-http-headers-with-chrome.webm
/videos/view-http-headers-with-chrome.ogv

Whoops! I messed up those headers! I fixed them after I made the video, and then I used Chrome’s F12 to check them, and this time I got them right.

Sean and Jen McCown, the MidnightDBAs started recording a weekly video in 2009 and later started live broadcasting the show and posting the recording on their website later.

When I joined the show they were using Ustream.tv for live broadcast and chat while using a second webcam and Debut video capture software to record the show for republication. I was able to change to widescreen video and update the quality and features within the Ustream controls, but Ustream had a habit of frequently changing the user interface and increasing ad intrusion until we finally got fed up and looked for other options.

I will write up articles on how we live stream self-hosted video as IT Know-It-All, but I decided I will also post weekly show production notes.

So please allow myself (IT Know-It-All) to introduce myself (midnightFreddie) as our new blogger!

I’ve had bad experiences with other hosting providers, both mine and for my clients. With other hosts I’ve often had host-related problems while their reporting services claim that their services are up and their support line doesn’t answer, later to find out they’ve had major problems. At least in the age of Twitter it’s easy to find other clients of the same host and verify the problem is with the hosting company and not with my or my client’s server.

But when I decided to leave my previous ill-behaved host who had my VPS down for 5 days and couldn’t tell me if or when it would be back up—thankfully I keep my own backups—I looked for a VPS hosting provider that had been around for many years, appeared to care about existing customers and offered a good value. I picked JaguarPC partially because of some recommendations and partially because they had a special running where I could get my umanaged VPS with 390 MB of RAM for the price of 256 MB.

Well, the special never ended like I thought it might. In fact, three years later they—on their own initiative—upgraded me to a new plan that had over three times more storage, three times the RAM (four times the RAM of the base plan I had) and ten times the monthly bandwidth at a lower plan cost. They explained in the email that they want to thank and retain their existing customers. The actual upgrade itself caused no issues, outages or reboots on my VPS. (This is possible due to the VPS solution they are using.)

I have not known of any outages or slow server issues since I’ve been with them. I haven’t had to contact them once since I opened the account. However, they proactively notified me of a DDoS issue that they were experiencing and thought might affect me, and they kept an updated status report available. I checked my server and didn’t seem to be affected, but it was very refreshing to hear from the hosting provider and having regular updates. That is vastly superior to any other host I’ve dealt with.

I semi-publicly shared the adventure with some online friends. I posted the following on May 10, 2008 about my leaving my previous host and starting with JaguarPC:

When I suggested he [the VPS reseller] set up a new VPS for me he liked the idea at first but then asked if I wanted to go direct with the company that hosted my old server. That’s the second time he’s tried to pawn me off onto another company, and I suddenly recalled I had been concerned about performance issues, anyway, and now he suggests I go with the company who apparently lost a host server and can’t seem to get it back up? Nope, time to go with someone totally new. JaguarPC apparently has been doing this 10 years, so hopefully after 3 years they’ll still want my business. Plus they were having a sale that’s supposed to give me more RAM at the same price which should help with the concurrent requests I’ve talked about before.

I haven’t closed the old account yet. I don’t really expect for them to bring my VPS back up as it was, but I’d like to see how my backups restore before I totally give up on it. So far I think I’m missing a few emails, a static website I forgot to back up (no biggie) and perhaps some config info that might be handy to look at but isn’t critical.

I posted the following on January 19, 2011:

Well, so far I’m pleased with JaguarPC. There have been no major issues for me overall, and they emailed me an outage notice yesterday and linked to a support thread that was keeping running updates on a DDoS attack that was affecting several of their servers and networks. And as far as I can tell my server didn’t even go down, but I got the updates anyway.

It’s so nice to find out about a problem from the hosting company and be offered a place to find updates before I even know there is a problem.

The last two major host company problems I’ve had (with other host companies) usually involved no public mention of the issue for at least an hour or two after a major outage, and support lines and support system jammed and giving no information or false positive information. Here they have proactively notified me, so if I start having problems I know what’s probably going on and know where to look for updates rather than waste hours trying to troubleshoot something I have no control over.

I posted the following on February 4, 2011:

Wow! JaguarPC just auto-upgraded me to a new plan that is cheaper and has 3-10 times the specs the old plan has! I’ll post a redacted form of the email later when I’m not on my Droid. The jist is they want to treat existing customers as well as new ones.

Hoo boy, now I can increase the concurrency here and give some more cache RAM to the database.

So yeah, apparently JaguarPC does still want my business after nearly 3 years.

This is the (redacted) email they sent me about the February 2011 upgrade:

We are very pleased to inform you of an immediate upgrade of your VPS package. This free automatic update is to express gratitude to our loyal customers like yourself. We are upgrading all our VPS plans in next several weeks and today your VPS was upgraded on the new plan.

No action is required on your end. There is no other change to your hosting, domains, or IP.

“It is common in the web hosting industry to promote plans to attract new customers, but we really wanted to do something that was a way of saying thank you to our existing customers,” says Greg Landis , CEO of JaguarPC. “We could not imagine a better way to do that to give more for less. Our philosophy has always been to treat our customers with utmost respect regardless of their size, and we hope to maintain our strong customer loyalty with this movement.”

The upgrade details are as follows:

[server account name]

Your old VPS plan name: VPS – Freedom Plan
Your old VPS plan cost: $[...] + addons
Your old VPS plan specs: 15GB Disk, 256MB RAM, 300GB Bandwidth + addons

Your new VPS plan name: Linux VPS Pro
Your new VPS plan cost: $[...] + addons
Your new VPS plan specs: 50GB Disk, 1024MB RAM, 3000GB Bandwidth + addons

[VPS management products info]

We just turned 12 years old in last October and getting stronger in 13th year of business! This is a tribute to the loyalty of our clients and reflects our business concepts of good value, integrity, and honesty. It wouldn’t be possible without the ongoing trust and business from clients such as yourself. Thank you all again for your business and referrals. We wish you all the very best.

First, it should be noted that when we talk about video we usually mean video and audio together. Video and audio are different forms of information, but they have to be delivered together for the expected experience. The overall conceptual breakdown:

• Video source: Most likely a USB webcam for a small web show broadcast
• Audio source: Often integrated with the webcam, but could be a discrete microphone
• Encoder: For our small live web show this is a laptop or PC that the webcam and microphone plug into. On professional setups this could be a piece of professional hardware as part of a camera system or a standalone device. The encoder will encode the audio and video streams and multiplex them into a container format. More on these terms later, but the data stream that leaves the encoder will be readable by viewer clients.
• Media server: The encoder sends the video/audio data stream to the media server to be delivered to all the viewer clients. It’s possible the encoder machine could also be the media server, but it’s more flexible and scalable if the media server is a separate server box.
• Client: The viewers are looking at their PCs, Macs, tablets or phones which are pulling the video (and audio) from the media server.

As the audio and video moves through the above progression, there are several properties that transform it, store it or transport it:

• Video codec: Codec is “coder/decoder”. It is the algorithm or language, if you will, that turns what the camera sees into computer information. At its very basic, video is a series of still photographs played after one another, but that would take up a prohibitive amount of disk space and network bandwidth to store and transmit, so there are various competing and evolving codecs to encode quality video in a small data space. The encoder you choose and configure will determine the video codec, and the media server you choose may only support certain codecs even though the media server itself doesn’t need to encode or decode the video. (Examples: h.264, VP8, VC-1, Theora)
• Audio codec: Same concept as video, but it applies to audio instead. Your choices in components and software may dictate which audio codec you can use. (Examples: MP3, AAC, WMA)
• MUX or multiplex: When playing video (with audio), you are playing two different types of information at the same time: audio and video. But you are playing only one data file or one live data stream. How do both audio and video get delivered in one data stream? Multiplexing. In brief, each elemental data stream (video, audio, etc.) is split into data chunks, and chunks of each individual stream take turns being delivered through the actual one data stream. There are different standards for multiplexing, and again your choice of components may dictate a particular MUX. Note also that other data may be multiplexed in your streams, like subtitles, alternate video or audio streams or metadata.
• Container format: At its most basic, this is the file storage type, and is usually indicated by the file extension like .AVI, .MOV, .MP4 and .M4V. Conceptually this would seem to blend with MUX, but somehow in ways not completely clear to me they are different from MUX, and some container types work with different MUX types.
• Streaming protocol: This is the method the data stream is transported or delivered over a network. (Examples: HTTP, RTP, RTSP, RTMP, MMS)

So, the encoder transforms the information from the video and audio sources using particular codecs and MUX’s them into a container format, then uses a streaming protocol to deliver the container stream to the media server, and clients use a streaming protocol—not necessarily the same one used between the encoder and media server—to receive the container stream from the media server.

But you can’t just pick whatever codecs, MUX, container and protocols you want. They don’t all work together. The real pain of the situation is that your audience’s clients may need particular codecs, MUX, container and protocols. The really real pain is that if your audiences’ clients are different, they may be incompatible with your choices and incompatible with each other.

I’ll remind the reader that the following are described from the point of view of a small, not-for-profit web show that wishes to grow from a viewership of 20-50 live viewers with no or minimal cost, assuming an existing 25mbps upstream connection and existing Windows 2008 Server and Linux server available. The organized portion of our show is recorded and made available for free download later, so content protection is not a concern for us. The community is rooted in Microsoft-based applications and therefore are generally assumed to have Windows, IE and Windows Media Player available, but in practice many of them own iPhones, iPads or iPods and desire to view the show on them. I also want to avoid clients having to install extra plugins or software to be able to view the live web show.

I’ll use a team metaphor for the types of clients the audience may have. There are three big teams, one fading team and one emerging team:

• Team Flash video: The ubiquitous Adobe Flash browser plugin supports video in certain formats across multiple platforms and browsers. Flash is almost assumed for general web surfing, and our existing broadcasts over Ustream require Flash, making Flash a likely choice for us since our audience already has it. There is a free version of the encoder, limited but useful, and a free Flash-based player whose only restriction is an unobtrusive logo, but the Flash Media Server software costs at least $1000. I have yet to find a free and simple substitute for Flash Media Server, but I have managed with much difficulty to use VLC as encoder and media server. I will blog about that separately, and I think there are other possibilities, but the free options require a lot of tinkering and testing and time. However, the popular iPads, iPhones and iPods cannot use flash and aren’t otherwise natively compatible
• Team Windows Media: If you’re sure all your viewers will be using Windows and Internet Explorer to view your show, this may be the option for you. Microsoft offers Windows 2008 Media Services (media server) as a free download, and there is a free version of Microsoft Expression Encoder. I found them easy to set up and use, but with the free tools most of the touted neat features like forward error correction, smooth streaming, adaptive streaming and multi-bitrate encoding aren’t available with the free tools. Still, with the free tools I was able to easily set up a live webcast viewable by Windows/IE/Windows Media Player users. However, those using other browsers or other platforms couldn’t view the stream. There may be a Java applet that will play the streams on other platforms, but I moved on to other teams before I really tried to make this work.
• Team Apple: Even though our web show is inseparably tied to Microsoft and Windows, a vocal number of the audience want to use their iPads and iPhones to view the show. At first glance team Apple and team Flash seem to overlap on compatible codecs, but they use different streaming protocols and possibly different muxes and containers. Since our primary target is Wintel platforms this does not seem to be the best server platform for us. However the Quicktime encoder pricing did seem to be quite reasonable, and Darwin Streaming Server is freely available and compatible with Windows, although I found the download page excruciatingly hard to find.
• Team RealMedia: I consider this a fading team. That may or may not be fair, but the RealPlayer plugin or player has to be installed for clients to use this, and I don’t believe most have this installed already, so for me it was a nonstarter. I didn’t look into pricing.
• Team HTML5: The great news: an open standard, and a choice of codecs including an open free video codec (VP8 / WebM). The bad news: this is an emerging standard, and I don’t think I can assume my viewers will be using an HTML5-compliant browser, although I could be wrong about that. And apparently the current VP8 encoder is not fast enough for live webcasts. HTML5 with its video tags brings hopes that the proprietary video format wars may be over in a few years and streaming video will be easier and cheaper, but it’s not here yet.