Discussions specific to the VMware vSphere hypervisor
Post Reply
ori
Enthusiast
Posts: 65
Liked: 1 time
Joined: Apr 28, 2012 9:51 pm
Full Name: Ori Besser
Contact:

Replicated VMs Start with bluescreen. v7 [Case # 00826751]

Post by ori »

Hi,

In our latest DR drill, we have found out that 7 of our replicated VMs (out of 162) got bluescreen at the very start of the boot process.
By using diskpart I saw that for all of these VMs the C partition was there but it had no NTFS structure - like it's not formatted. In addition, several more VMs were starting with the chkdsk process - a behavior we have never seen before.
Re-replicating incrementally did not help, only full replication fixed it.
It's interesting that all of these VMs are Server 2003, while we have a lot of other OSs too.

We are not using the replica failover process, but manually start the VMs at the DR site.
The support answer was that manually powering on replicas is not recommended (while the "Replication" section of the FAQ by gostev clearly states otherwise), I have to admit that I expected a little deeper investigation on an issue that raises doubts on replication integrity. We will re-replicate and we will implement Sure Replica jobs but we have to know what in our environment caused it - and fix it.
I want to state that we were using Veeam extensively for the last 3 years and never had one issue.

Vitaliy S.
Product Manager
Posts: 25386
Liked: 2242 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by Vitaliy S. »

Do you revert to the previous state after doing all the tests, or just commit all the changes to the vmdk/snapshot via vSphere Client?

ori
Enthusiast
Posts: 65
Liked: 1 time
Joined: Apr 28, 2012 9:51 pm
Full Name: Ori Besser
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by ori »

Just power off the machines and let Veeam continue running the replication jobs as usual.

CaptainFred
Enthusiast
Posts: 88
Liked: 2 times
Joined: Jul 31, 2013 12:05 pm
Full Name: Si
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by CaptainFred »

ori wrote:Just power off the machines and let Veeam continue running the replication jobs as usual.
That's what I do if I want to test something out on the DR VMs, because I thought Veeam automatically reverts the snapshot before it starts replicating any changes anyway so any changes I've made are lost at that point?

Btw, I had a similar issue a while back, do you use hotadd at the DR/replicated end? I don't know if that was the cause but support recommended I switch to NBD (network) mode and there isn't any performance difference anyway.

ori
Enthusiast
Posts: 65
Liked: 1 time
Joined: Apr 28, 2012 9:51 pm
Full Name: Ori Besser
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by ori »

Thanks captain, I use hot-add. Did support say why to stop using hot-add?
I just need to know what happened in my environment in order to avoid it in the future.

Vitaliy S.
Product Manager
Posts: 25386
Liked: 2242 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by Vitaliy S. »

CaptainFred wrote:That's what I do if I want to test something out on the DR VMs, because I thought Veeam automatically reverts the snapshot before it starts replicating any changes anyway so any changes I've made are lost at that point?
Yes, that's correct.
CaptainFred wrote:Btw, I had a similar issue a while back, do you use hotadd at the DR/replicated end? I don't know if that was the cause but support recommended I switch to NBD (network) mode and there isn't any performance difference anyway.
Both hotadd and NBD mode is fully supported to write data via target proxy, this should not cause any issues for target VM.
ori wrote:I just need to know what happened in my environment in order to avoid it in the future.
Keep working with our support team on this and let us know the outcome.

ori
Enthusiast
Posts: 65
Liked: 1 time
Joined: Apr 28, 2012 9:51 pm
Full Name: Ori Besser
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by ori »

The support team says we had the corruption because we didn't manually take a snapshot before powering on a replica, but if Veeam automatically reverts to the restore point why is it necessary to take another snapshot?

ori
Enthusiast
Posts: 65
Liked: 1 time
Joined: Apr 28, 2012 9:51 pm
Full Name: Ori Besser
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by ori »

Just to add on that, one of the VMs that had the issue (our PDC domain controller) was a fresh replica - it is a new VM that was never powered on before, this test was the first time it has been powered on.

Vitaliy S.
Product Manager
Posts: 25386
Liked: 2242 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by Vitaliy S. »

Taking snapshot manually is not required. Did you use application-aware image processing for the VM with DC role installed?

ori
Enthusiast
Posts: 65
Liked: 1 time
Joined: Apr 28, 2012 9:51 pm
Full Name: Ori Besser
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by ori »

yes

Vitaliy S.
Product Manager
Posts: 25386
Liked: 2242 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by Vitaliy S. »

Ok, keep working with our support team then and please update this topic once you find the resolution. Thanks!

ori
Enthusiast
Posts: 65
Liked: 1 time
Joined: Apr 28, 2012 9:51 pm
Full Name: Ori Besser
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by ori »

but the support team concluded with saying we should take a snapshot before testing, they don't further investigate

VladV
Expert
Posts: 223
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by VladV »

From what I now, manually testing replicas without using surereplica or the failover process within Veeam needs some steps taken to guarantee the integrity of the replica after testing.

Every step is described here including taking a snapshot at the beginning: http://www.veeam.com/kb1073

ori
Enthusiast
Posts: 65
Liked: 1 time
Joined: Apr 28, 2012 9:51 pm
Full Name: Ori Besser
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by ori »

So taking into account Vitaliy's reply and the FAQ - the KB is outdated / wrong

Vitaliy S.
Product Manager
Posts: 25386
Liked: 2242 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replicated VMs Start with bluescreen. v7 [Case # 0082675

Post by Vitaliy S. »

Veeam backup server triggers revert to snapshot operation before each run of the replication job, so creating snapshot manually is not required. Yes, KB is out-dated and was relevant to version 6. Sorry for the confusion, I will ask our support team to make the corresponding updates to that article.

Post Reply

Who is online

Users browsing this forum: Majestic-12 [Bot] and 29 guests