Recoving from a SAN disaster

[jgremillion]

Our Clariion CX4 experienced a hardware failure a few weeks ago that caused our array to reboot multiple times. Needless to say, our VM farm was in a mess. Now that I finally have our environment back together, I want to take a moment to share a few thoughts with everyone.

The Good:
Veeam pretty much works. We had problems restoring some our EMC Recoverpoint Images for some unknown reasons but never had a problem when I tried to restore from a Veeam backup image. In a few cases the VBK on disk seem to be corrupt. In that case I just imported a VBK from our tape archive and kept rolling along.

There are a whole lot of ways to us VIR. That technology is really helpful. How did I use it? Before I restored a Groupwise Post Office from backup, I used VIR to let our GW admin look at the PO and make sure all was right with it before I took the time to restore it. (Groupwise on Windows is temperamental sometimes.)

I wonder how many ways is there to restore a VM from a Veeam image? Replicate, Full VM, VM files, guest file level, VIR with storage vMotion, VIR and Replicate, VIR with VCenter Converter ( Client or Standalone), FLR appliance, FLR appliance and copy VMDKs around, VM copy. Obviously some are better then others, but when you are really trying to recover from a disaster with very little data loss it's nice to know you have options.

It's a good thing I archive VBKs images to tape once a week.

[jftuga]

As we don't use tape, your statement about the VBK files being corrupt in a few cases really concerns me. Can you please elaborate on this?

Is there anyway to test the integrity of the VBK & VIB files?

Thanks,
-John

[jgremillion]

A backupjob was running when my array crashed. That corrupted a VBK. I am using the reversed incremantal feature. If I was using the traditional incramental feature, I would hve not had a problem with the VBK. The problem with have just been in the VIB.

I am thinking about changing my backup jobs to use the traditional incremental feature.

[Gostev]

Generally speaking, even if VBK is corrupt, you still should be able to restore from previous restore point, for which backup completed successfully. Our reversed incremental engine is designed to handle job termination (including abnormal), because in reality this happens very often. As long as your target storage is not affected by disaster in any way and continues to run fine, source storage crash should not really matter. Even if in case of some weird type of failure when source storage starts to send random data, this only trashes the latest restore point - and not all previous ones. All VRBs are static, just like increments.

If you still have those backups, it would be interesting to investigate if they are recoverable (earlier restore points). If not, we would definitely be interested to take a look at what happened.