Blog

New API & command interface documentation preview

Eric Windisch — Fri, 25 Jun 2010 00:42:36 -0400

Very, very soon now, we'll be releasing a new management console. Currently, customers may login to manage individual instances, with each instance having a unique login. Our new "Cloud Admin Shell" will offer the ability to manage multiple instances under a single account -- using the same credentials as for their billing account. It will also support creating and removing instances on-the-fly! This will be possible, as currently, via interactive shell, RESTful API, and via an SSH "API".

Below, we are supplying a preview of the new command/API documentation. We will also provide an end-user manual at the time of release. This is being provided for the sake of public comment -- so let us know what you think!

NAME

cloud-admin-shell - Provide a console interface to control instances.

Command Table

instance [id]

boot

Issues a boot command to an instance. If the string argument "console" is provided, it will also start a serial console. Returns 1 on success, or undef on failure.

status

Indicates if the instance is running or not. Returns 1 on success, or undef on failure.

shutdown [graceful | force]

Sends a shutdown command. By default, this is a graceful shutdown. If a non-graceful shutdown is not possible, the instance will remain running. To force a non-graceful shutdown, specify the string argument, force. This is non-blocking, use instance status to determine if the instance is still running. The instance console command may be used to interactively view the status of the shutdown.

reboot

Sends a reboot command, a graceful instance shutdown followed by an instance boot. Will fail if cannot be performed gracefully. If so, the command instance shutdown force must be utilized.

reimage

On an instance whose status is shutdown, install a new operating system. Implementation and availability depends on the cloud specified.

console

Launch an interactive serial console to the instance. Only available on interactive shells.

instance [id] config [noun]

password [pass] [pass-repeat]

Set a new instance-password. This may be used to login for management of this instance, separate from the account-wide management offered by cloud-admin-shell. Useful for when looking to authorize users to an instance under an account, while not giving access to the entire account.

cloud [id]

Change the cloud on which this instance operates. Different clouds may operate within separate geographic areas and may incur different usage charges.

add

- ipv4

Add a new IPv4 address

- ipv6

Add a new IPv6 address

script [noun]

Commands useful and specific to scripting.

fetchuri [uri]

Fetches a URI. HTTP and FTP have been tested. Servers must specify a Document Length. If the Document Length is not provided by the server, this will return -2 If the Document Length is larger than 2MB, this will return -1 Otherwise, this will download and return as a string the content of the provided, downloaded URI.

language [interpreter]

Changes the language of the current shell. This is only temporary. To set the shell permanently, use config interactive_shell.

This may be used from scripts if a script should contain multiple languages, but is intended primarily for interactive use.

exit

Exits the shell.

quit

Exits the shell.

help

Returns helpful information.

create [noun]

instance

Create/add an instance.

show [noun]

instance

Show information about an instance. If no arguments are provided, lists all instances.

- config

Shows the configuration & preferences for an instance.

author

Displays the authors of this application.

version

Displays the current version number.

whoami

Tells you who the current user is.

storage

Accesses the storage of an instance.

message [message-id]

Displays system messages. If no message is indicated, it will return a summary of the last 10 messages. If a message id is specified, it will return the full contents of the requested message.

cloud

List available clouds, given no arguments. Given the argument of a cloud identifier, will provide information regarding that cloud.

config [noun]

storage

Configure storage.

passwd [new-pass] [new-pass-repeat]

Set or change an account password.

sshkey [key]

Install an SSH key for the current user. Must be provided in OpenSSH format.

pref [variable] [value]

Set preferences for the current user.

Web Management Console PREVIEW

Eric Windisch — Wed, 12 May 2010 23:36:44 -0400

We've been working on a new web-based management console. This will not replace, but will rather supplement and rely upon, the console-gui and API. This is an in-development preview of the new interface. It is a good time to make your voice heard, if there is something you'd like to see, or changes made. Clearly, we'll be making many changes yet, and some features may be renamed, moved, or may not make it into the immediate release. Screenshots below.

Technical details: It will use our new RESTful API and is expected to be entirely self-contained within HTML, Javascript, and CSS. This will allow for a high-level of customizability and as a reference application for those looking to utilize our API.

The new RESTful API PREVIEW

Eric Windisch — Thu, 06 May 2010 00:39:53 -0400

For over a year, we've had an API available based on XML::RPC. It wasn't popular and it was difficult to support and keep up to date. However, today I present to you a *PREVIEW* of our RESTful API. Results are returned in JSON. You can now authenticate and push requests to the cloud-shell interface via HTTP.

The interface is in extreme-beta-state for the moment, as it only provides RAW access to the STDOUT of the methods as run through the interactive cloud-shell. The output must be cleaned up, the proper return-values and data-types must be returned. Interpreting HTTP-POST data as script should also be implemented. Once this is all complete, we'll mark it as stable. I'm estimating a week.

Example:

https://secure.grokthis.net/manage/vps/rest/status

Enter your username and password for the cloud-shell management interface (HTTP BASIC auth) and you should retrieve the status of your default instance.

Other commands of the cloud-shell will also work. Not yet supported via the REST interface are 'fetchuri', 'reimage', and 'console'.

Cloud-shell Scripting! [PART 2]

Eric Windisch — Fri, 23 Apr 2010 15:08:00 -0400

A short while ago, we added interpreters to the cloud-shell. However, there were still many things it could not do, limitations in loading externally generated scripts, etc. Well, since our initial announcement, we've added a number of really cool features which we hope you'll love.

Bug Fixes

First, we've fixed bugs. The setpref command should now be much more reliable and changes will take effect immediately, even changing the interactive_shell will have an immediate effect. You'll also know which interpreter you're using according to the prompt: "cloud-shell[tcl]>" or "cloud-shell[ruby]>"

Command Passing

You can now launch commands via SSH's command passing mechanism. To demonstrate:

$ ssh -l cdemo manage.vps.grokthis.net status

Guest : Running

All commands executed this way are done via the 'simple' interpreter, there is no smart interpreter when executing commands this way. Remember, we support SSH keys, so you can use this in an automated process.

Script Piping

Finally, if command passing isn't cool enough, you can now pipe scripts:

$ ssh -l cdemo manage.vps.grokthis.net ruby < do_this_stuff.rb

$ echo "setpref interactive_shell ruby" | ssh -l cdemo manage.vps.grokthis.net tcl

$ cat <<EOF | ssh -l cdemo manage.vps.grokthis.net javascript
whoami()
EOF

Note that to pipe a script, you must (currently) specify the interpreter on the SSH commandline. We intend to add support for a hash-bang (#!) to specify the script type in the script itself...

Chef / CFEngine / Puppet

We haven't done anything with these DevOps engines yet, but we see the potential to do some really great stuff... some sort of integration and/or cookbooks are planned. Keep tuned!

Storage problems finally resolved (we think!)

Eric Windisch — Fri, 23 Apr 2010 14:52:34 -0400

So, we've had a number of problems with the storage. A few months ago, to help minimize the effect, we had moved to three-way mirroring. It turns out that not only did it fail to improve the situation, it only made it worse. Quite simply, we had several drives which were experiencing repeat "failures" despite testing good and running the latest firmware. It turns out, however, that there was a "super secret" firmware which our drive manufacturer would not inform us of, this was only discovered by research and communication with other customers of this particular disk manufacturer. Having the three-way mirroring had made this worse for another reason altogether, ZFS will either refuse to write to, or will put a heavy bias against, any *mirror* sets which are marked degraded. Normally, I'd expect that read performance would be affected negatively, but write performance would continue as normal on a degraded set. In ZFS, this is quite the contrary, but write performance suffers quite horribly.

In the end, we've managed to get the drives operating stably and now know that a three-way mirror when degraded will not equal the performance of a non-degraded 2-way mirror as I might otherwise have expected with the Linux software RAID.

The remaining thing to do, yet, is to migrate Rorschach off the clustered LVM volume, moving it to the direct-iSCSI volumes. Rorschach is the last machine using clustered LVM. Then, we'll have to migrate a few hosts that never made it onto shared storage. All in due time...

Fixing storage hotspots...

Eric Windisch — Fri, 02 Apr 2010 13:24:22 -0400

We have become aware that one of the most significant performance issues with our system and the most immediate cause of IO-wait problems is related to storage hotspots. Currently, 90% of our disk IO is being handled by 25% of our disk array. The storage system we are using should normally avoid this, but we are an edge-case. Our immediate goal is to rectify this problem, spreading the load across the entire array. It has also come to our attention that additional performance-benefiting changes we had previously planned may not be beneficial without first addressing this serious issue.

To combat hot-spots in our SAN which are key to the performance problems noted recently, we must shutdown and (safely) re-provision a little more than 100 customers. This re-provisioning will require the recreation of customer filesystems, while retaining customer data. Customers with partitioned block storage will simply have their systems copied bit-by-bit for convenience. Customers looking to receive a bit-by-bit filesystem copy should notify us of this request. We also intend to modify customer's /etc/fstab file to indicate a default mount option of 'noatime'. We understand that these changes are somewhat invasive, so please do notify support if you have any concerns!

This change will simplify future maintenance, recovery, and backup operations. It will greatly reduce the complexity of operations performed by individual host nodes, leading to more stable systems. It will offer greater data security. Additionally, it will help us in implementing further performance-boosting strategies. Finally, it should provide a less invasive methodology for eliminating future hotspots when/if they develop. Approximately 10% of our customers are already configured in this manner with great success.

Cloud-shell Scripting!

Eric Windisch — Fri, 19 Mar 2010 17:14:42 -0400

We now have basic Tcl, Javascript, and Ruby functionality built into cloud-shell! This allows you to script basic functionality from the interactive terminal. User feedback is encouraged, let us know what you think!

What does this mean for you? Well, power users will be able to do a whole lot of really neat things as we expand the library of available methods. Currently, though, you may access any of the existing shell commands from code.

What *will* it mean? Well, we're going to expand the available methods so that you'll be able to write scripts for any sort of neat things. Solaris or Macintosh users might be familiar with OpenFirmware and EFI -- the goal is to make our cloud-shell similar to, but more powerful than, these environments.

UPDATE: Michael Szul (@szul) asked if we had any particular reason for choosing Tcl. Well, for the interactive shell, it simply makes a lot of sense because from now on, all commands you run are actually Tcl procedures. We can do this without a change in the typical user expectations or experience. Stored scripts will be able to use the same familiar syntax you've always used, making it a close analog to a unix shell or powershell script. Essentially, it is easy for the user, despite being a relatively dated and obscure language in 2010. That said, I'm already working on Javascript support, which I feel won't be very useful for interactive use, but would be of great use for non-interactive code, especially for those from a web background.

UPDATE 2: Javascript now supported! Do 'setpref interactive_shell javascript' from the Tcl shell. Note that from then on, everything will be in Javascript syntax, i.e. to get back to Tcl: "setpref('interactive_shell', 'tcl')". We've mapped write/writeln/document.write/document.writeln such that they print to your terminal, for convenience.

Update 3: You may now load script files from the network. Use the fetchuri method to download a script (remote server MUST set Content-Length). You can then pass that to an eval function, i.e. in javascript: eval(fetchuri("http://ftp.grokthis.net/pub/installers/karmic.js")); Allowing one to download & exec scripts in languages other than that of the interactive shell should be complete sometime tomorrow.

Update 4: Now supporting Ruby!!! Finally, customers that, for whatever reason, want to disable interactive features entirely can revert to the old non-interactive shell by choosing the 'simple' shell.

Example Script (tcl):

Currently, many of the features listed below are not yet incorporated into your shell. This is an example of how scripting will eventually empower users. This example pseudo-script reimages the OS while also setting the CPU architecture and boot methods, and finally, boots the OS. It is a simple example, but hopefully, a powerful one. Ultimately, we intend to base our reimaging software on this concept.

    # user settings
    set cpu_arch 64
    set boot_method pvgrub
    set url "ftp://ftp.grokthis.net/disk_images/fedora-12_64.tar.bz2"

    # set the guest
    set guest [get_guest]

    # Program:
    foreach disk [get_storage] {
     if { [string compare $disk(desc) 'root'] == 0 } {
      reimage $disk $url
     }
    }

    setpref cpu_arch $cpu_arch
    setpref boot_method $boot_method

    boot

    # Notify a remote machine of our IP address
    fetchuri "http://your-server/instance-manager/callback/create/$guest(ip)"

Bandwidth monitoring

Eric Windisch — Mon, 15 Mar 2010 23:59:14 -0400

We've added a new bandwidth command to the cloud-shell management console. It is not active for everyone yet, but for over 50% of the customer base. Enjoy!

cloud-shell> bandwidth

        cdemo

           month        rx      |       tx      |    total
        ------------------------+---------------+---------------
          Mar '10         0 MB |    158.76 MB |    158.76 MB
        ------------------------+---------------+---------------
        estimated         0 MB |       327 MB |       327 MB
Connection closed.
cloud-shell>

Karmic upgrade mini-howto

Eric Windisch — Sat, 13 Mar 2010 13:07:12 -0500

This is not yet user-friendly. Let us know if you need help! We'll soon develop a script to automate this upgrade process. Note that these instructions should be more-or-less suitable for migrating any Debian OS, such as Lenny, to running a distribution kernel (rather than using our managed kernel)

Upgrade from Jaunty with apt-get. Note that *currently* installing Jaunty and upgrading is the easiest way to run Karmic.
***DO NOT REBOOT / SHUTDOWN***
Copy /etc/event.d/tty1 to /etc/event.d/xvc0 and replace all 'tty1' with 'xvc0'. Remove /etc/event.d/tty*
Install the packages "linux-virtual" and "grub". Note that "grub-pc" (GRUBv2) will NOT work.
Modify /boot/grub/menu.lst so that you use 'root = (hd0)' (default is hd0,0) Append console=xvc0 to the kernel line.
Modify /etc/fstab, replace 'sda' with 'xvda'. (sda1 becomes xvda1, etc)
Shutdown.
Login to cloud-shell (management console)
Run following commands in the cloud-shell:

cloud-shell> setpref boot_method pvgrub
cloud-shell> setpref disk_namespace xen

Logout of the cloud-shell
Log back into the cloud-shell (this reloads your preferences)
Execute:

cloud-shell> boot console


Note: if you build a partitioned instance then your menu.lst may contain the default (hd0,0) entry and you should set boot_method to 'pvgrub_part'. Partitioned instance support has not yet been fully tested.
WARNING: Do NOT upgrade to ext4 if using grub, as per these instructions as it does not support ext4.  If you must use ext4, you may attempt a partitioned installation; only the first partition must be ext2/3.
WARNING #2: Archive / restore support is not tested with grub. We should test this.  If it works, let us know.



SAN status
Eric Windisch — Sat, 13 Mar 2010 11:18:23 -0500
As many of our customers are aware, we've had problems with stability on some of our VPS nodes.  Specifically, our newest and largest nodes.  Most of these issues have been related in one way or another to our iSCSI SAN.  We understand how significant these problems are for our customers, and we hope customers realize how significant these problems are for us -- clearly, problems for customers are our problems too. We don't like dealing with down machines any more than necessary.  In fact, as a systems administrator, I've always advocated laziness -- the good kind of laziness: do it once, do it right.  I feel the the best systems administrators keep things running smoothly by doing the least amount of work possible.  The point is that we want things to work smoothly even more than you do.  The reality is that we're simply hitting a number of edge cases of the relatively new technology, both hardware and software, which we deploy.

The problems we have had have been numerous and varied.  Some crashes have left little information behind, some with detailed information, others even happened before our own eyes upon running some relatively safe and mundane command.  Some have had interesting causes, some have had interesting solutions.  Some solutions have been deployed already, some will automatically deploy upon reboot (i.e. after the next crash), and others are planned for future maintenance windows.
We're working to resolve these issues as quickly as possible and regain a reputation for reliable service.


Below is a list of issues which we're recognizing as either currently, or recently, problematic:

IO errors & read-only filesystems:  Customers have at times noticed IO errors and read-only filesystems when the SAN has "disappeared".  Sometimes this happens after a simple hiccup, sometimes it is a serious outage, but either way, if the error hits a customer instance, they're going to need a reboot.  To resolve this, we're changing the iSCSI timeouts.  It has been recently discovered that the default timeout settings were not appropriate for our configuration.  This change should cause customer's machines to block and wait for the storage to return.  However, really, these timeout settings are only a concern when something else goes wrong, but it might be the difference between a hiccup and a disaster.

Swap performance: There are obvious concerns about the performance of the SAN itself, but recently, the performance of swap -- which is stored on local arrays, per-node, seems to be a great issue in relation to the IO wait problems.  In fact, some SAN issues have resulted from problems with local storage.  Poor performance of local disks used for swap have caused high-IO wait on the entire machine, causing time-outs to the SAN.  This is particularly troublesome when customers run out of RAM in their instance, triggering page thrashing.  In this case, one or two customers can ruin it for everyone.  To reduce customer page thrashing, we've been raising the resource ceiling, giving (all) customers more RAM.  One intended improvement will be improved swap devices.  The best long-term solution, however, would be a capability to constrain IO per instance, which is not currently possible in kernel space.  Piping swap access through userspace will be slower, but the increased reliability might be more than worth it and is under consideration.
Dropping disks: Our SAN filer had has been dropping disks left and right.  At one point, it lost an entire mirror set, although we were able to recover it.  Some of these problems have been due to the controller itself, and we've upgraded the firmware to resolve those issues.  However, some of the disks themselves seem to be suffering from "old age" after a single year.  SMART returns "good", badblock scans are clean, but it appears that "old age" may have affected latency as drives encounter more internal errors.  Disk latency is bad, it triggers drive removals by our "intelligent" disk controller.  We've replaced 1/3rd of the disks in our array and moved to three-way mirroring (adding disks) to protect against multiple disk failures.  Unfortunately, it seems that we still need to replace more disks.  I think in this regard, we're learning a lot of the same issues that Google has according to their study, "Failure Trends in a Large Disk Drive Population".  As a result of these replacements, we've had to run a lot of background reconstructions, affecting performance and IO-wait.
SAN OS crashes: A few times, the SAN itself has crashed. Initially, it happened due to controller & os crashes upon drive hotswap, even from automatic drive removal, an issue now resolved by a firmware update.  Then, we experienced, of all things, a CPU fan failure, also resolved.  Finally, this past week, we've experienced what appears to have been a hangup in the controller's kernel driver.  The only improvements which seem possible here are controller redundancy and OS upgrades.  Most immediately, we'll upgrade the OS.  We'll probably add another controller soon.

iSCSI+ZFS is slow: Our current version of the OpenSolaris OS has bugs and limitations which affect performance for our use case.  The actual explanation for this is beyond this article, but essentially, limitations in the iSCSI implementation are preventing the system from benefiting from OS, disk, or controller cache.  Worse, it results in additional, unnecessary writes to disk which would otherwise not occur.  This means that performance suffers significantly during normal operation, let alone during background reconstructions or other high-load periods.  A new release of OpenSolaris resolving these issues is scheduled to be released this month (March 2010), we are lab-testing and will schedule an upgrade when appropriate.
CLVM: Some of our nodes are using CLVM. We're not happy with LVM or the RedHat cluster suite. It hasn't scaled and has been a source of crashes and reboots.  In January, we were able to complete software modifications which allow us to use iSCSI directly and new accounts are no longer stored on LVM.  Luckily for those still on the nodes using CLVM, we've addressed most/all of the issues and have recently made a change that may drastically improve reliability and stability.  Still, we're actively seeking to migrate those customers still on CLVM away from it.

Node crashes:  Some of our nodes have simply proven unstable.  Some of the problems we've run into have been strictly software, such as when a node crashes upon performing a "shutdown force" of an instance.  Some however, have been clearly hardware, or kernel drivers.  For those issues which are strictly software, we've been upgrading those machines, especially those most frequently affected by crashes, with newer kernels.  As for hardware, we had made the decision to replace some machines.  At this time, we're putting a temporary hold on that idea for further evaluation before we mindlessly throw money at a problem, but may likely proceed.
Instances are reinstalled poorly: Due to legacy concerns, reimaging is performed in a sub-optimal way.  While this hasn't been explicitly linked to widespread stability problems and to no issues recently, it has been clearly linked to a few isolated cases from 2008.  It is clear that even if our current solution is not a cause of instabilities, that it carries a great potential for causing instabilities and must be improved.  In February, we rewrote our installation script in a more stable fashion.  This was not sufficient to solve the (potential) problem, but was a necessary step towards this goal.



User preferences in Cloud Shell
Eric Windisch — Wed, 03 Mar 2010 15:42:29 -0500
VPS/Cloud customers may now set user preferences via the cloud-shell application.  Currently, we're supporting two such preferences and will continue to expand this list to support a wide array of user-configurable options...
The current variables include:

 cpu_arch, which should be '32' or '64'. Default is 64.

 autoheal, which should be 1 or 0.  Default is 1.

The cpu_arch setting will be used to determine which architecture kernel should be loaded on boot.  Please note that the boot_method variable may override this setting.  WARNING: the legacy command 'boot32' has been removed in favor of this user-preference.

The autoheal setting will determine whether or not the configured instance pool will be automatically restored/rebooted upon failure.

To set or get these variables, use the commands 'setpref' and 'getpref' as such:

cloud-shell> setpref cpu_arch 32
cloud-shell> setpref autoheal 0
cloud-shell> getpref
$VAR1 = { 
          'client:NNN:pref:cpu_arch' => '32',
          'client:NNN:pref:autoheal' => '0'
        };



It should be noted that reimaging your OS currently cannot modify these variables.  Thus, if you switch from a 32-bit or 64-bit OS, you must modify the cpu_arch variable manually. This is a forthcoming improvement we must make to the system.

UPDATE:  New 'boot_method' preference available!
Currently accepted boot methods are:

pv - Paravirtualized (default)
pv_part - Paravirtualized, Partitioned disk
pvgrub - Paravirtualized, with GRUB
rescue - Boot into rescue image
Boot methods internally implement CPU architecture support and may optionally ignore or reject a configured cpu_arch setting.



Snapshots: Archiving and restore from Cloud-Shell
Eric Windisch — Tue, 10 Nov 2009 13:43:54 -0500
We are now allowing customer testing of Archive and Restore from our cloud-shell management console (https://secure.grokthis.net/manage/vps).  Currently, only our FURY and RORSCHACH nodes have this feature enabled, other nodes will be upgraded to support this during a future scheduled maintenance window.

The process is quite simple, once you login to the shell you should see the standard cloud-shell command prompt:

cloud-shell>
To archive and snapshot your instance, simply run, archive.  Note that this will briefly take your instance offline.
cloud-shell> archive
Archiving instance: 000
Instance running. Proceeding.
NOTE: Your instance will now be paused.
 This will briefly take your instance offline (~0-60 min)
4 GB 01:01:01 [17 MB/s] [======> ]  100%  ETA 0:00:00
192 MB 01:01:01 [17 MB/s] [=====>]  100%  ETA 0:00:00
0+N records in
0+N records out
Your instance has been resumed.
cloud-shell>

To restore from your archive, first shutdown the instance, then execute restore:
cloud-shell> shutdown
cloud-shell> status
Guest: shutdown
cloud-shell> restore
4 GB 01:01:01 [17 MB/s] [======> ]  100%  ETA 0:00:00
192 MB 01:01:01 [17 MB/s] [=====>]  100%  ETA 0:00:00
0+N records in
0+N records out
Your instance has been restored to its prior state.
cloud-shell>

The restore command will return your instance to the last stored archive.  This includes disk block storage, swap, and RAM.  There should be no need to manually boot the instance after a restore.



Booting with Grub (custom kernels!)
Eric Windisch — Thu, 17 Sep 2009 15:39:29 -0400
To run a custom kernel, you need only two things, a kernel (of course!) and a GRUB configuration.

Lets start with GRUB.  This assumes that you will not be using an initrd file, but you may use one, and that your kernel image will be installed as /boot/vmlinuz:
# mkdir -p /boot/grub
# cat <<EOF > /boot/grub/menu.lst
default		0
timeout		5

title		Linux Default
root		(hd0)
kernel		(hd0)/boot/vmlinuz root=/dev/sda1 ro console=xvc0 clocksource=jiffies
EOF

That is it for GRUB!  Now, as for your kernel, you just need to make sure that it is compiled to run as a Xen DomU with paravirtualization.  This requires that you use a recent Linux kernel with support for Xen enabled, or you use a patched kernel such as provided by XenSource (http://xenbits.xensource.com/linux-2.6.18-xen.hg) 

The compiled kernel image may be placed anywhere inside your filesystem as long as GRUB is configured to point to it.  The above example assumes this location is /boot/vmlinuz.
Finally, from the management console, execute the command, "boot-grub".  This command is API-accessible.  Unfortunately, for now, you will need to manually specify this whenever booting. Your VPS will boot with a system-default image by default, if for instance, the node crashes.  This caveat will disappear once we complete a migration of all accounts to a grub-based configuration.


Today's outage
Eric Windisch — Sat, 12 Sep 2009 03:22:21 -0400
Today, we experienced an outage.  Technically, we experienced a few outages.
We initially discovered that our Galadriel server was offline, and furthermore, that the VPS node that it ran upon was also down.  We began working on restoring this machine when we were distracted by an incoming DoS attack.
Working (remotely) to extinguish the attack, we found ourselves locked out of the network and triggered a reload of the router.  Upon reloading, the system brought up the services in an atypical order triggering a bug/caveat/quirk previously unknown to us.  While everything looked good on our side, discussions with upstream carriers indicated that all was not as it appeared.  With this input, we quickly located a solution which we began to apply, until...
A storm knocked out electricity in parts of our neighborhood.  While we retained power, internet connectivity was lost to our offices (note, we have no servers located in these offices).
Once our branch office internet connectivity was restored, were were able to complete the process of restoring internet connectivity to all servers, restoration of the crashed VPS node and its guest accounts, including the galadriel server.
We apologize for the inconvenience that this caused.  This explaination seeks only to provide transparency to our operations so that our customers may know the details of service affecting events.



Free incoming calls via Skype now available!
Eric Windisch — Fri, 28 Aug 2009 16:08:49 -0400
Please note that in addition to our international (+1) number and US toll-free (+1 800) numbers, we can now accept Skype-to-Skype calls to SkypeID "grokthisnet".   Skype-to-Skype calls forward directly into our standard telephone and answering systems, providing an inexpensive option for international customers to contact us by voice without expensive international calling rates.
Please note that due to forwarding delays, Skype calls may require up to 8-10 rings before receiving an answer.  If no one is available to take your call, you will (eventually) be forwarded to our answering system.

Blog

New API & command interface documentation preview

Web Management Console *PREVIEW*

The new RESTful API *PREVIEW*

Example:

Cloud-shell Scripting! [PART 2]

Bug Fixes

Command Passing

Script Piping

Chef / CFEngine / Puppet

Storage problems finally resolved (we think!)

Fixing storage hotspots...

Cloud-shell Scripting!

Bandwidth monitoring

Karmic upgrade mini-howto

SAN status

User preferences in Cloud Shell

Snapshots: Archiving and restore from Cloud-Shell

Booting with Grub (custom kernels!)

Today's outage

Free incoming calls via Skype now available!

Web Management Console PREVIEW

The new RESTful API PREVIEW