We've been working on a new web-based management console.  This will not replace, but will rather supplement and rely upon, the console-gui and API. This is an in-development preview of the new interface.  It is a good time to make your voice heard, if there is something you'd like to see, or changes made.  Clearly, we'll be making many changes yet, and some features may be renamed, moved, or may not make it into the immediate release.  Screenshots below.

Technical details:  It will use our new RESTful API and is expected to be entirely self-contained within HTML, Javascript, and CSS.  This will allow for a high-level of customizability and as a reference application for those looking to utilize our API.

As many of our customers are aware, we've had problems with stability on some of our VPS nodes.  Specifically, our newest and largest nodes.  Most of these issues have been related in one way or another to our iSCSI SAN.  We understand how significant these problems are for our customers, and we hope customers realize how significant these problems are for us -- clearly, problems for customers are our problems too. We don't like dealing with down machines any more than necessary.  In fact, as a systems administrator, I've always advocated laziness -- the good kind of laziness: do it once, do it right.  I feel the the best systems administrators keep things running smoothly by doing the least amount of work possible.  The point is that we want things to work smoothly even more than you do.  The reality is that we're simply hitting a number of edge cases of the relatively new technology, both hardware and software, which we deploy.

The problems we have had have been numerous and varied.  Some crashes have left little information behind, some with detailed information, others even happened before our own eyes upon running some relatively safe and mundane command.  Some have had interesting causes, some have had interesting solutions.  Some solutions have been deployed already, some will automatically deploy upon reboot (i.e. after the next crash), and others are planned for future maintenance windows.

We're working to resolve these issues as quickly as possible and regain a reputation for reliable service.

Below is a list of issues which we're recognizing as either currently, or recently, problematic:

IO errors & read-only filesystems:  Customers have at times noticed IO errors and read-only filesystems when the SAN has "disappeared".  Sometimes this happens after a simple hiccup, sometimes it is a serious outage, but either way, if the error hits a customer instance, they're going to need a reboot.  To resolve this, we're changing the iSCSI timeouts.  It has been recently discovered that the default timeout settings were not appropriate for our configuration.  This change should cause customer's machines to block and wait for the storage to return.  However, really, these timeout settings are only a concern when something else goes wrong, but it might be the difference between a hiccup and a disaster.

Swap performance: There are obvious concerns about the performance of the SAN itself, but recently, the performance of swap -- which is stored on local arrays, per-node, seems to be a great issue in relation to the IO wait problems.  In fact, some SAN issues have resulted from problems with local storage.  Poor performance of local disks used for swap have caused high-IO wait on the entire machine, causing time-outs to the SAN.  This is particularly troublesome when customers run out of RAM in their instance, triggering page thrashing.  In this case, one or two customers can ruin it for everyone.  To reduce customer page thrashing, we've been raising the resource ceiling, giving (all) customers more RAM.  One intended improvement will be improved swap devices.  The best long-term solution, however, would be a capability to constrain IO per instance, which is not currently possible in kernel space.  Piping swap access through userspace will be slower, but the increased reliability might be more than worth it and is under consideration.

Dropping disks: Our SAN filer had has been dropping disks left and right.  At one point, it lost an entire mirror set, although we were able to recover it.  Some of these problems have been due to the controller itself, and we've upgraded the firmware to resolve those issues.  However, some of the disks themselves seem to be suffering from "old age" after a single year.  SMART returns "good", badblock scans are clean, but it appears that "old age" may have affected latency as drives encounter more internal errors.  Disk latency is bad, it triggers drive removals by our "intelligent" disk controller.  We've replaced 1/3rd of the disks in our array and moved to three-way mirroring (adding disks) to protect against multiple disk failures.  Unfortunately, it seems that we still need to replace more disks.  I think in this regard, we're learning a lot of the same issues that Google has according to their study, "Failure Trends in a Large Disk Drive Population".  As a result of these replacements, we've had to run a lot of background reconstructions, affecting performance and IO-wait.

SAN OS crashes: A few times, the SAN itself has crashed. Initially, it happened due to controller & os crashes upon drive hotswap, even from automatic drive removal, an issue now resolved by a firmware update.  Then, we experienced, of all things, a CPU fan failure, also resolved.  Finally, this past week, we've experienced what appears to have been a hangup in the controller's kernel driver.  The only improvements which seem possible here are controller redundancy and OS upgrades.  Most immediately, we'll upgrade the OS.  We'll probably add another controller soon.

iSCSI+ZFS is slow: Our current version of the OpenSolaris OS has bugs and limitations which affect performance for our use case.  The actual explanation for this is beyond this article, but essentially, limitations in the iSCSI implementation are preventing the system from benefiting from OS, disk, or controller cache.  Worse, it results in additional, unnecessary writes to disk which would otherwise not occur.  This means that performance suffers significantly during normal operation, let alone during background reconstructions or other high-load periods.  A new release of OpenSolaris resolving these issues is scheduled to be released this month (March 2010), we are lab-testing and will schedule an upgrade when appropriate.

CLVM: Some of our nodes are using CLVM. We're not happy with LVM or the RedHat cluster suite. It hasn't scaled and has been a source of crashes and reboots.  In January, we were able to complete software modifications which allow us to use iSCSI directly and new accounts are no longer stored on LVM.  Luckily for those still on the nodes using CLVM, we've addressed most/all of the issues and have recently made a change that may drastically improve reliability and stability.  Still, we're actively seeking to migrate those customers still on CLVM away from it.

Node crashes:  Some of our nodes have simply proven unstable.  Some of the problems we've run into have been strictly software, such as when a node crashes upon performing a "shutdown force" of an instance.  Some however, have been clearly hardware, or kernel drivers.  For those issues which are strictly software, we've been upgrading those machines, especially those most frequently affected by crashes, with newer kernels.  As for hardware, we had made the decision to replace some machines.  At this time, we're putting a temporary hold on that idea for further evaluation before we mindlessly throw money at a problem, but may likely proceed.

Instances are reinstalled poorly: Due to legacy concerns, reimaging is performed in a sub-optimal way.  While this hasn't been explicitly linked to widespread stability problems and to no issues recently, it has been clearly linked to a few isolated cases from 2008.  It is clear that even if our current solution is not a cause of instabilities, that it carries a great potential for causing instabilities and must be improved.  In February, we rewrote our installation script in a more stable fashion.  This was not sufficient to solve the (potential) problem, but was a necessary step towards this goal.

We frequently get asked how to upgrade kernels, how to upgrade modules, etc.  This is actually in our knowledge-base, but I thought it would be useful to toss this onto the blog since we just pushed out a very important kernel upgrade, and want to make sure that this information is as accessible as possible!

1. Simply shutdown your VPS.  You must do a full shutdown and not a "reboot". 

2. Then, boot the machine from the management console.  IMPORTANT: If you have a 32-bit OS, you must use the command "boot32", this will be unnecessary in a future update.

3. Update your modules:

     wget -O - http://ftp.grokthis.net/pub/linux/modules/install_modules.sh | /bin/bash

4. You're done!  You might want to reboot if you depend on specific modules being available during boot.

We want to let everyone know of a free software cloud management framework we've been building. If you've read the CCIF mailing lists or our twitter, then you're probably already aware of it. This project is called Annelidous (www.annelido.us). It enables the building of public and private cloud infrastructure services (IaaS), API agnostic frontends, and API proxies. It is licensed under the AGPLv3, more information regarding the AGPLv3 license can be found on the annelido.us website and at www.fsf.org.

So far, I'm running Annelido.us to run the services of GrokThis.net and VPS Village, but it has potential beyond simply hosting. Runnable code is now available for managing Xen clusters over SSH, and an initial frontend based on xen-shell has been completed. Some work has already begun on backend connectors to EC2 and Vertebra-xen. All of this code is available in a public git repository.

The design goals include the potential to build proxies between various IaaS APIs. As an example, a proxy could be built that allows a frontend with support for the OCCI API to communicate to a cloud which offers the EC2 API. This might also then include the ability to automatically and transparently allow OVF files to be used on EC2.

Frontend applications will be able to use this framework to access a variety of 'Connectors' through a common Perl API. In this sense, I intend for it to provide something analogous to what DBI provides for databases.

It should be noted that support for billing/accounting modules is being built in as well. So far, it integrates with Ubersmith, a billing manager oriented for web hosting operations.

I have very noble goals with this project, but as of yet, it has only scratched the service of its potential. There is currently an IRC channel on irc.freenode.org, #annelidous, and a public git repository. You can accept this as a call for both awareness and for assistance, so that we can have a free, open, and interoperable solution for building, connecting to, and supporting future IaaS solutions.

Recently, some users of vps008 have noticed that they cannot complete a reinstall/reimage of Debian or Ubuntu.  We resolved the initial complaints by a reboot.   Unfortunately, on a VPS server with many customers on it, it can take over an hour to complete a reboot cycle.  We hoped that this would not be a recurring issue, but could not be certain of it.

Now, we are noticing that this behavior is, in fact, recurring.  Shortly after a reboot, for the following days or weeks, the server will operate normally, but will eventually begin to hang running the configuration for console-setup during a debootstrap installation.

At this point, most customers will eventually close their SSH sessions, and try again to reimage their VPS.  When reimaging again, they will get an error that their filesystem is already mounted.   This happens because the previous installation never completed.

When our support is contacted, we can stop the prior installation, but without a (time-consuming, service affecting) reboot we cannot provide much further immediate assistance.  We can then schedule to reboot the server at a more appropriate time when the service interruption would be less significant to the many other customers that are sharing that host.

We are currently investigating alternative solutions for running the reimages that would not trigger, or be affected by, this bug.  Additionally, we are attempting to obtain some assistance from the Xen and Linux developers.