Bad day. Fri, May 8, 2009

Bad day. Fri, May 8, 2009
7:27AM - our data center informed us we had a "new" user name and password.
8:05AM - we are told to check the ticket center where we found our server had been disconnected from the internet at 7:31AM
8:05AM - we are told to install a "recovery tool", backup our data and reinstall. Our new username and password will not permit installation of the recovery tool

We point DNS servers to the backup server in Galax. The new addresses start working their way through the Internet but it will take hours before most servers send viewers to the backup server.

11:55AM - we are told our account had been reenabled but we must not put the server back online.

We gain access through the recovery tool, compress the database and start a download. That process takes about 3 hours. We expand the database files, slip them under a running database server on a workstation, patch the passwords to match the original and the database is up to date. Now we slip the database under the backup server and we are over that hurdle.

We start the same process on the user file system. It is twelve times the size of the database and the download is a little over 50% complete on Saturday at 9:00AM. If all goes well the user files will be up-to-date sometime on Sunday.

In the interim we have been exploring Amazon Web Services - S3 the Simple Storage Solution and EC2 - the Elastic Compute Cloud. Early explorations are most encouraging. We will continue this week.

A painful experience for both Woody and I and our web users. It would have been much worse if Woody had not rebuilt the backup server in mid-April. Bad days do happen.