Very, very soon now, we'll be releasing a new management console.  Currently, customers may login to manage individual instances, with each instance having a unique login.  Our new "Cloud Admin Shell" will offer the ability to manage multiple instances under a single account -- using the same credentials as for their billing account.  It will also support creating and removing instances on-the-fly!  This will be possible, as currently, via interactive shell, RESTful API, and via an SSH "API".

Below, we are supplying a preview of the new command/API documentation.  We will also provide an end-user manual at the time of release.  This is being provided for the sake of public comment -- so let us know what you think!

NAME

cloud-admin-shell - Provide a console interface to control instances.

Command Table

instance [id]

boot

Issues a boot command to an instance. If the string argument "console" is provided, it will also start a serial console. Returns 1 on success, or undef on failure.

status

Indicates if the instance is running or not. Returns 1 on success, or undef on failure.

shutdown [graceful | force]

Sends a shutdown command. By default, this is a graceful shutdown. If a non-graceful shutdown is not possible, the instance will remain running. To force a non-graceful shutdown, specify the string argument, force. This is non-blocking, use instance status to determine if the instance is still running. The instance console command may be used to interactively view the status of the shutdown.

reboot

Sends a reboot command, a graceful instance shutdown followed by an instance boot. Will fail if cannot be performed gracefully. If so, the command instance shutdown force must be utilized.

reimage

On an instance whose status is shutdown, install a new operating system. Implementation and availability depends on the cloud specified.

console

Launch an interactive serial console to the instance. Only available on interactive shells.

instance [id] config [noun]

password [pass] [pass-repeat]

Set a new instance-password. This may be used to login for management of this instance, separate from the account-wide management offered by cloud-admin-shell. Useful for when looking to authorize users to an instance under an account, while not giving access to the entire account.

cloud [id]

Change the cloud on which this instance operates. Different clouds may operate within separate geographic areas and may incur different usage charges.

add

- ipv4

Add a new IPv4 address

- ipv6

Add a new IPv6 address

script [noun]

Commands useful and specific to scripting.

fetchuri [uri]

Fetches a URI. HTTP and FTP have been tested. Servers must specify a Document Length. If the Document Length is not provided by the server, this will return -2 If the Document Length is larger than 2MB, this will return -1 Otherwise, this will download and return as a string the content of the provided, downloaded URI.

language [interpreter]

Changes the language of the current shell. This is only temporary. To set the shell permanently, use config interactive_shell.

This may be used from scripts if a script should contain multiple languages, but is intended primarily for interactive use.

exit

Exits the shell.

quit

Exits the shell.

help

Returns helpful information.

create [noun]

instance

Create/add an instance.

show [noun]

instance

Show information about an instance. If no arguments are provided, lists all instances.

- config

Shows the configuration & preferences for an instance.

author

Displays the authors of this application.

version

Displays the current version number.

whoami

Tells you who the current user is.

storage

Accesses the storage of an instance.

message [message-id]

Displays system messages. If no message is indicated, it will return a summary of the last 10 messages. If a message id is specified, it will return the full contents of the requested message.

cloud

List available clouds, given no arguments. Given the argument of a cloud identifier, will provide information regarding that cloud.

config [noun]

storage

Configure storage.

passwd [new-pass] [new-pass-repeat]

Set or change an account password.

sshkey [key]

Install an SSH key for the current user. Must be provided in OpenSSH format.

pref [variable] [value]

Set preferences for the current user.

We've been working on a new web-based management console.  This will not replace, but will rather supplement and rely upon, the console-gui and API. This is an in-development preview of the new interface.  It is a good time to make your voice heard, if there is something you'd like to see, or changes made.  Clearly, we'll be making many changes yet, and some features may be renamed, moved, or may not make it into the immediate release.  Screenshots below.

Technical details:  It will use our new RESTful API and is expected to be entirely self-contained within HTML, Javascript, and CSS.  This will allow for a high-level of customizability and as a reference application for those looking to utilize our API.

For over a year, we've had an API available based on XML::RPC.  It wasn't popular and it was difficult to support and keep up to date.  However, today I present to you a *PREVIEW* of our RESTful API.  Results are returned in JSON.  You can now authenticate and push requests to the cloud-shell interface via HTTP. 

The interface is in extreme-beta-state for the moment, as it only provides RAW access to  the STDOUT of the methods as run through the interactive cloud-shell.   The output must be cleaned up, the proper return-values and data-types must be returned.  Interpreting HTTP-POST data as script should also be implemented.  Once this is all complete, we'll mark it as stable.  I'm estimating a week.

Example:

 https://secure.grokthis.net/manage/vps/rest/status

Enter your username and password for the cloud-shell management interface (HTTP BASIC auth) and you should retrieve the status of your default instance.

Other commands of the cloud-shell will also work. Not yet supported via the REST interface are 'fetchuri', 'reimage', and 'console'.

A short while ago, we added interpreters to the cloud-shell. However, there were still many things it could not do, limitations in loading externally generated scripts, etc.  Well, since our initial announcement, we've added a number of really cool features which we hope you'll love.

Bug Fixes

First, we've fixed bugs. The setpref command should now be much more reliable and changes will take effect immediately, even changing the interactive_shell will have an immediate effect. You'll also know which interpreter you're using according to the prompt: "cloud-shell[tcl]>" or "cloud-shell[ruby]>"

Command Passing

You can now launch commands via SSH's command passing mechanism.  To demonstrate:

$ ssh -l cdemo manage.vps.grokthis.net status

Guest : Running

All commands executed this way are done via the 'simple' interpreter, there is no smart interpreter when executing commands this way.  Remember, we support SSH keys, so you can use this in an automated process.

Script Piping

Finally, if command passing isn't cool enough, you can now pipe scripts:

$ ssh -l cdemo manage.vps.grokthis.net ruby < do_this_stuff.rb

$ echo "setpref interactive_shell ruby" | ssh -l cdemo manage.vps.grokthis.net tcl

$ cat <<EOF | ssh -l cdemo manage.vps.grokthis.net javascript
whoami()
EOF

Note that to pipe a script, you must (currently) specify the interpreter on the SSH commandline. We intend to add support for a hash-bang (#!) to specify the script type in the script itself...

Chef / CFEngine / Puppet

We haven't done anything with these DevOps engines yet, but we see the potential to do some really great stuff... some sort of integration and/or cookbooks are planned.  Keep tuned!

So, we've had a number of problems with the storage.  A few months ago, to help minimize the effect, we had moved to three-way mirroring.  It turns out that not only did it fail to improve the situation, it only made it worse.  Quite simply, we had several drives which were experiencing repeat "failures" despite testing good and running the latest firmware.  It turns out, however, that there was a "super secret" firmware which our drive manufacturer would not inform us of, this was only discovered by research and communication with other customers of this particular disk manufacturer.  Having the three-way mirroring had made this worse for another reason altogether, ZFS will either refuse to write to, or will put a heavy bias against, any *mirror* sets which are marked degraded.  Normally, I'd expect that read performance would be affected negatively, but write performance would continue as normal on a degraded set.  In ZFS, this is quite the contrary, but write performance suffers quite horribly.

In the end, we've managed to get the drives operating stably and now know that a three-way mirror when degraded will not equal the performance of a non-degraded 2-way mirror as I might otherwise have expected with the Linux software RAID.

The remaining thing to do, yet, is to migrate Rorschach off the clustered LVM volume, moving it to the direct-iSCSI volumes.  Rorschach is the last machine using clustered LVM.  Then, we'll have to migrate a few hosts that never made it onto shared storage.  All in due time...

We have become aware that one of the most significant performance issues with our system and the most immediate cause of IO-wait problems is related to storage hotspots.  Currently, 90% of our disk IO is being handled by 25% of our disk array.  The storage system we are using should normally avoid this, but we are an edge-case.  Our immediate goal is to rectify this problem, spreading the load across the entire array.  It has also come to our attention that additional performance-benefiting changes we had previously planned may not be beneficial without first addressing this serious issue.

To combat hot-spots in our SAN which are key to the performance problems noted recently, we must shutdown and (safely) re-provision a little more than 100 customers.  This re-provisioning will require the recreation of customer filesystems, while retaining customer data.  Customers with partitioned block storage will simply have their systems copied bit-by-bit for convenience.  Customers looking to receive a bit-by-bit filesystem copy should notify us of this request.  We also intend to modify customer's /etc/fstab file to indicate a default mount option of 'noatime'.  We understand that these changes are somewhat invasive, so please do notify support if you have any concerns!

This change will simplify future maintenance, recovery, and backup operations. It will greatly reduce the complexity of operations performed by individual host nodes, leading to more stable systems.  It will offer greater data security.  Additionally, it will help us in implementing further performance-boosting strategies.  Finally, it should provide a less invasive methodology for eliminating future hotspots when/if they develop.  Approximately 10% of our customers are already configured in this manner with great success.

We now have basic Tcl, Javascript, and Ruby functionality built into cloud-shell! This allows you to script basic functionality from the interactive terminal. User feedback is encouraged, let us know what you think!

What does this mean for you? Well, power users will be able to do a whole lot of really neat things as we expand the library of available methods. Currently, though, you may access any of the existing shell commands from code.

What *will* it mean? Well, we're going to expand the available methods so that you'll be able to write scripts for any sort of neat things. Solaris or Macintosh users might be familiar with OpenFirmware and EFI -- the goal is to make our cloud-shell similar to, but more powerful than, these environments.

UPDATE: Michael Szul (@szul) asked if we had any particular reason for choosing Tcl. Well, for the interactive shell, it simply makes a lot of sense because from now on, all commands you run are actually Tcl procedures.  We can do this without a change in the typical user expectations or experience.  Stored scripts will be able to use the same familiar syntax you've always used, making it a close analog to a unix shell or powershell script.  Essentially, it is easy for the user, despite being a relatively dated and obscure language in 2010.  That said, I'm already working on Javascript support, which I feel won't be very useful for interactive use, but would be of great use for non-interactive code, especially for those from a web background.

UPDATE 2: Javascript now supported! Do 'setpref interactive_shell javascript' from the Tcl shell.  Note that from then on, everything will be in Javascript syntax, i.e. to get back to Tcl: "setpref('interactive_shell', 'tcl')".  We've mapped write/writeln/document.write/document.writeln such that they print to your terminal, for convenience.

Update 3: You may now load script files from the network. Use the fetchuri method to download a script (remote server MUST set Content-Length). You can then pass that to an eval function, i.e. in javascript: eval(fetchuri("http://ftp.grokthis.net/pub/installers/karmic.js"));  Allowing one to download & exec scripts in languages other than that of the interactive shell should be complete sometime tomorrow.

Update 4: Now supporting Ruby!!!  Finally, customers that, for whatever reason, want to disable interactive features entirely can revert to the old non-interactive shell by choosing the 'simple' shell.

Example Script (tcl):

Currently, many of the features listed below are not yet incorporated into your shell.  This is an example of how scripting will eventually empower users. This example pseudo-script reimages the OS while also setting the CPU architecture and boot methods, and finally, boots the OS. It is a simple example, but hopefully, a powerful one.  Ultimately, we intend to base our reimaging software on this concept.

    # user settings
    set cpu_arch 64
    set boot_method pvgrub
    set url "ftp://ftp.grokthis.net/disk_images/fedora-12_64.tar.bz2"

    # set the guest
    set guest [get_guest]

    # Program:
    foreach disk [get_storage] {
     if { [string compare $disk(desc) 'root'] == 0 } {
      reimage $disk $url
     }
    }

    setpref cpu_arch $cpu_arch
    setpref boot_method $boot_method

    boot

    # Notify a remote machine of our IP address
    fetchuri "http://your-server/instance-manager/callback/create/$guest(ip)"

We've added a new bandwidth command to the cloud-shell management console. It is not active for everyone yet, but for over 50% of the customer base.  Enjoy!

cloud-shell> bandwidth

        cdemo

           month        rx      |       tx      |    total
        ------------------------+---------------+---------------
          Mar '10         0 MB  |    158.76 MB  |    158.76 MB
        ------------------------+---------------+---------------
        estimated         0 MB  |       327 MB  |       327 MB
Connection closed.
cloud-shell>

This is not yet user-friendly. Let us know if you need help!  We'll soon develop a script to automate this upgrade process.  Note that these instructions should be more-or-less suitable for migrating any Debian OS, such as Lenny, to running a distribution kernel (rather than using our managed kernel)

1. Upgrade from Jaunty with apt-get.  Note that *currently* installing Jaunty and upgrading is the easiest way to run Karmic.
2. ***DO NOT REBOOT / SHUTDOWN***
   
3. Copy /etc/event.d/tty1 to /etc/event.d/xvc0 and replace all 'tty1' with 'xvc0'.  Remove /etc/event.d/tty*
4. Install the packages "linux-virtual" and "grub". Note that "grub-pc" (GRUBv2) will NOT work.
5. Modify /boot/grub/menu.lst so that you use 'root = (hd0)'  (default is hd0,0)   Append console=xvc0 to the kernel line.
6. Modify /etc/fstab, replace 'sda' with 'xvda'. (sda1 becomes xvda1, etc)
   
7. Shutdown.
8. Login to cloud-shell (management console)
9. Run following commands in the cloud-shell:
cloud-shell> setpref boot_method pvgrub
cloud-shell> setpref disk_namespace xen

10. Logout of the cloud-shell
11. Log back into the cloud-shell (this reloads your preferences)
    
12. Execute:
cloud-shell> boot console


Note: if you build a partitioned instance then your menu.lst may contain the default (hd0,0) entry and you should set boot_method to 'pvgrub_part'. Partitioned instance support has not yet been fully tested.

WARNING: Do NOT upgrade to ext4 if using grub, as per these instructions as it does not support ext4.  If you must use ext4, you may attempt a partitioned installation; only the first partition must be ext2/3.

WARNING #2: Archive / restore support is not tested with grub. We should test this.  If it works, let us know.