Backup Server Crashed on Failover "test"

[Michael_6835]

Hi I just recently upgraded to 6.1 from 5.x and I was in the process of "testing" the replication jobs. I deleted my legacy replica job and decided to create a new one.

I am replicating from an iscsi san to local storage on one of my hosts. I chose different ip addressing and also chose to only backup the "primary disk", choosing to exlclude a data drive. I was able to perform the replication without any issues.

Today I went to test the "failover" by choosing failover now and then after a few processes my rdp session to the server was disconnected. After a few minutes I was able to get back in, I originally thought some new feature in veamm was causing a freeze up and then it let me back in. But after logging in, it was clearly evident that the server blue screened.
This left Veeam thinking the replica was still live (while it wasn't on esx host) I tried to undo the failover, but that failed.
I contacted Veeam support case # 5202936, the tech had me reset the host services on esx host which allowed me to "undo" the failed failover.

I am looking as to what was the reason that the server crashed in the first place. I sent my logs to support and i was referred to http://www.resplendence.com/whocrashed

for me to figure it out as the logs didn't show anything.

I'm real disappointed at this point in the service I've received thus far from support. We pay for maintenance/support and through the whole call I felt like I was rushed off the phone. I'm all about self diagnosing and figuring stuff out, however I have to believe that there is a direct correlation between Veeam executing this task and my server failing.
I have been running version 5.x and never had this issue with this server doing this.
Has anyone seen this?

This is the last thing I saw.
[VddkDiskMountSession] Creating session for virtual disk 'XP Sys Admin_replica/XP Sys Admin-000002.vmdk'.
[05.07.2012 13:59:07] <01> Info Creating folder 'C:\Users\sqladmin\AppData\Local\Temp\02b6ccbc-4d75-4997-beec-b961796872f0'
[05.07.2012 13:59:07] <01> Info [VddkDiskMountSession] Creating temp folder 'C:\Users\sqladmin\AppData\Local\Temp\02b6ccbc-4d75-4997-beec-b961796872f0'.

Backup Server : 2008 x64 sp2 VBR 6.1 (latest release)
Backup storage is local to server (local disks)
2 esx 4.1 u1 hosts connected to hp msa iscsi san.

Thanks

[Michael_6835]

Just tried to re-run the job again this time choosing a different restore point.
the server blue screened again... It is clear to me that Veeam failover replication is causing this server to blue screen.
when the server comes back up Veeam Console shows replica as running, but vi client doesn't.

I was up in the data room and did this last test from the Console as I wanted to see what the message said on the stop error. The stop error occured shortly after "re-ip" gave the succesful check mark.

Now in version 5 i never had this re-ip option. The orignal vm i'm testing is on (just an xp machine) and the replica I telling it to re-ip to some non-existant network. Not sure if that even matters. I guess I will pull another replication with this turned off to just test again.

But I can't keep bsod my server! Can anyone help with this one?
Thank

[Michael_6835]

Does anyone have any theories on this one?

[tsightler]

You said you watched the console, what did the STOP message say? Honestly, it's almost impossible to predict what could be causing this. My first guess would be some type of antivirus or security agent running on the server. I've seen similar issues with some of these in the past when doing things like Surebackup where those types of services cause crashes with the VDDK which is what is being used to make the IP changes for Re-IP.

You might consider disabling the "Re-IP" function for the replica and see if the server still BSOD's. That would at least nail it down to the Re-IP feature which basically narrows it to something interfering with the operation of the VDDK.

[Michael_6835]

T,
your guess was correct. I was running CA on the server and that seemed to be the reason for the blue screen.

To test the theory, I created a separate replication job without the re-ip and that was able to failover correctly (with AV on)

So knowing the failover process itself was fine, I then disabled AV and reran the first replication with the Re-IP and this worked as well.

Level 2 support also suggested this behavior with AV, namely TrendMicro.

They have provided me with the File Exclusions in case anyone else has a similar issue or disable av when you do a failover.

File Exclusions

C:\Program Files\Veeam\Backup and Replication\vdk.exe

C:\Program Files\Veeam\Backup and Replication\vdk.sys

C:\Program Files\Veeam\Backup and Replication\VeeamAgent.exe

C:\Program Files\Veeam\Backup and Replication\VeeamFSR.sys


Thanks for the suggestion.