Help! Replica VMs Corrupted!

[CaptainFred]

Today after my replication job had a run I started up some VMs at DR to give them a test. I haven't setup SureReplica yet so I first started my replica DC (2008 R2). It started into Windows ok, did it DC recovery stuff in the background, rebooted and when I tried to login to it blue screened. It rebooted into Windows ok but as soon as I try and login it blue screens with the generic STOP 0x0000007B error. I tried rolling back to various snapshot points and same thing happens. So I tried a few other VMs, some 2008 R2, some 2003, various sizes, all seems to be corrupt in one way or another! Rolling back snapshots, even to the first doesn't help either. I'm using VBR 7.0.0.833. vSphere 5.5.0 Build 1474528 in both Production and DR. The ESXi servers at DR and Production are different but that shouldn't matter!

I deleted and re-replicated the DC again and that's fine for now, but what about everything else? I can't re-replicate everything again (20TB of VMs!!!)

What should I do? Please help!

Thanks

[Gostev]

You should open a support case for assistance with ANY technical issues, as explained when you click New Topic.
That said, if you replicas are corrupted, then I don't think there are other option but re-replicating.
However, it would still be nice to investigate the cause with support to prevent similar issues in future. Thanks.

[CaptainFred]

Ok I will do. But how could this have happened?!

This is supposed to be the disaster recovery solution for my company and be reliable!!!

Some start doing a chkdsk immediately after starting, so that's a bad sign.

[Gostev]

I cannot guess the reason. Each particular case must be investigated properly, but most common reason for such issues is storage-level corruption of replica VM datastore.

Our solution is as reliable as the underlying storage. There is nothing we can do if storage starts to misbehave and corrupts replica VM disks. We can only detect this and notify you, this is exactly the reason why we have introduced SureReplica.

[lp@albersdruck.de]

I cannot guess the reason. Each particular case must be investigated properly, but most common reason for such issues is storage-level corruption of replica VM datastore.

Another possible issue can be corrupted or wrong CBT information from vSphere. But as Gostev said, contact support and let them investigate.

[CaptainFred]

I've raised Case # 00538718.

As it looks as though I may have to re-replicate some or all our VMs (or take a copy of the backup file to the DR site), would it be quicker to setup WAN accelerators at both ends and run a backup copy job to get the data there and then use that to re-seed the new replica's? From what I've read, using a backup copy job with WAN accelerators may not be any faster (and quicker) overall compared to a multi threaded high compression replication job?

[foggy]

What kind of link do you have to the remote location?

[CaptainFred]

50mbit with a site to site VPN over it using Cisco ASA firewalls so loose some bandwidth with that overhead.

Also, regarding why the VMs are corrupted, is it not likely that it's anything to do the link or the firewalls because otherwise the receiving Veeam proxy would have identified corruption in it's error checking?

[Gostev]

Yes, the target proxy validates all traffic received, and will automatically address any in-flight data corruption by requesting the source proxy to re-send "bad" blocks again.

[CaptainFred]

Ok so would it be quicker to use a backup copy job with WAN accelerators or just re-replicate everything again?

[veremin]

Most likely, the direct replication job will be quicker in this case, because initial run of wan accelerated backup copy job might take some time, indeed. Just be sure to select proxy servers at both ends. Thanks.