Last week I was attending a Virtual Technology Users Group (http://www.VTUG.com) meeting out of state and so far the week had resulted in my being called only a small handful of times and for nothing overly serious. That changed Friday evening after I had gotten home. I had just crawled into bed and was drifting into blissful rest in my own bed for the first time in nearly a week. Although I do not recall hearing it, the telephone rang and it was my job calling. Power had gone out for about an hour due to high winds and rain storms. Somehow my drifting into dreamland made me oblivious but it was enough to knock over a few trees onto some electric lines. There was nothing but wet roads as I drove in the dark to my office at about 9pm.
When I went to the server room everything appeared to be fine. Everything seemed to be powered so at first I was a little confused. When I had checked the machine states remotely from home prior to leaving, I saw a few of our desktop computers on the internet so I knew more or less that the infrastructure hardware was fine. Ultimately, the issue was that the VMs did not start after the power was restored .. including Domain Controllers and DNS servers, Virtual Center servers, nothing! I logged into each of my VMware hosts individually and manually powered on all of the machines. It seemed like the APC power agent had successfully powered everything off for me. That first impression was oh so incorrect though.
Somehow the company’s entire email database (both the public and private information store) was corrupted and unusable. I tried ISINTEG, the usual MS Exchange tool for such an issue, and it did not work. It was hopeless. The production folks were back working again at this time but I was getting nowhere with this Exchange issue. All I had left that I could do was make a call to Microsoft paid support. At this time it was after midnight and I was simply exhausted. I decided that tomorrow would be soon enough and that the email downtime was just going to have to be inevitable. I knew anything else that needed to be done could be done remotely so, satisfied that we had no further issues in production I went home to sleep.
As I was driving back home my thoughts drifted to Veeam. My internal dialog suggested that the previous even’s backup should have occurred a short while before the power outage had occurred. I resolved that in the morning I would verify and if I was correct and it was successful, an Instant VM Recovery and a VMotion would be the simplest way to save our email system and the cost of a paid call to Microsoft. At 8am the next morning I logged into my backup server remotely from home. I was right, the backup was successful and it had occurred about an hour-and-a-half prior to the power outage. It took about 30-minutes for me to successfully restore and migrate the virtual machine back into production. Email was again flowing perfectly and the actual downtime was minimal. We even saved a paid Microsoft support call.
Thanks Veeam!
-Paul Martin
and thriller it was, for you...