Replicated VMs Start with bluescreen. v7 [Case # 00826751]

[ori]

Hi,

In our latest DR drill, we have found out that 7 of our replicated VMs (out of 162) got bluescreen at the very start of the boot process.
By using diskpart I saw that for all of these VMs the C partition was there but it had no NTFS structure - like it's not formatted. In addition, several more VMs were starting with the chkdsk process - a behavior we have never seen before.
Re-replicating incrementally did not help, only full replication fixed it.
It's interesting that all of these VMs are Server 2003, while we have a lot of other OSs too.

We are not using the replica failover process, but manually start the VMs at the DR site.
The support answer was that manually powering on replicas is not recommended (while the "Replication" section of the FAQ by gostev clearly states otherwise), I have to admit that I expected a little deeper investigation on an issue that raises doubts on replication integrity. We will re-replicate and we will implement Sure Replica jobs but we have to know what in our environment caused it - and fix it.
I want to state that we were using Veeam extensively for the last 3 years and never had one issue.

[Vitaliy S.]

Do you revert to the previous state after doing all the tests, or just commit all the changes to the vmdk/snapshot via vSphere Client?

[ori]

Just power off the machines and let Veeam continue running the replication jobs as usual.

[CaptainFred]

Just power off the machines and let Veeam continue running the replication jobs as usual.

That's what I do if I want to test something out on the DR VMs, because I thought Veeam automatically reverts the snapshot before it starts replicating any changes anyway so any changes I've made are lost at that point?

Btw, I had a similar issue a while back, do you use hotadd at the DR/replicated end? I don't know if that was the cause but support recommended I switch to NBD (network) mode and there isn't any performance difference anyway.

[ori]

Thanks captain, I use hot-add. Did support say why to stop using hot-add?
I just need to know what happened in my environment in order to avoid it in the future.

[Vitaliy S.]

That's what I do if I want to test something out on the DR VMs, because I thought Veeam automatically reverts the snapshot before it starts replicating any changes anyway so any changes I've made are lost at that point?

Yes, that's correct.

Btw, I had a similar issue a while back, do you use hotadd at the DR/replicated end? I don't know if that was the cause but support recommended I switch to NBD (network) mode and there isn't any performance difference anyway.

Both hotadd and NBD mode is fully supported to write data via target proxy, this should not cause any issues for target VM.

I just need to know what happened in my environment in order to avoid it in the future.

Keep working with our support team on this and let us know the outcome.

[ori]

The support team says we had the corruption because we didn't manually take a snapshot before powering on a replica, but if Veeam automatically reverts to the restore point why is it necessary to take another snapshot?

[ori]

Just to add on that, one of the VMs that had the issue (our PDC domain controller) was a fresh replica - it is a new VM that was never powered on before, this test was the first time it has been powered on.

[Vitaliy S.]

Taking snapshot manually is not required. Did you use application-aware image processing for the VM with DC role installed?

[ori]

yes

[Vitaliy S.]

Ok, keep working with our support team then and please update this topic once you find the resolution. Thanks!

[ori]

but the support team concluded with saying we should take a snapshot before testing, they don't further investigate

[VladV]

From what I now, manually testing replicas without using surereplica or the failover process within Veeam needs some steps taken to guarantee the integrity of the replica after testing.

Every step is described here including taking a snapshot at the beginning: http://www.veeam.com/kb1073

[ori]

So taking into account Vitaliy's reply and the FAQ - the KB is outdated / wrong

[Vitaliy S.]

Veeam backup server triggers revert to snapshot operation before each run of the replication job, so creating snapshot manually is not required. Yes, KB is out-dated and was relevant to version 6. Sorry for the confusion, I will ask our support team to make the corresponding updates to that article.