VEEAM Hyper-V CSV Backup - Blue Screen

[Gostev]

Hopefully there is a patch 4 that will address this?

We are not planning on releasing one. Patch 3 already includes a number of workarounds aimed to minimize the chance of this issue appearing, and in fact it helped 90% of customers who was having this issue (literally, we have closed over 30 support cases with that patch released). Unfortunately, we cannot really "fix" this issue from our side, because the issue sits in Windows 2008 Failover Clustering design. It can be reproduced without our product present, and is pretty common overall (otherwise, Microsoft would not have create the above blog post 2 years ago). However, we are determined to continue working with the remaining affected customers to see what else can we do from our side in terms of workarounds. Thanks!

[collinp]

smurteira,

On my backup jobs, I am only backing up specific vm's so I am adding vm's to my job on an individual basis. But I am drilling down through the vmm and cluster levels in order to add the specific vm's. I have tried backing up at the vmm and cluster level in the past. Not sure if that makes a difference in terms of how the job is processed. The other thing that may be different in my environment is my VMM server is virtualized.

[smurteira]

Got it. I had the cluster and vms into veeam by adding in the vmm, then I add each vm individually to the backup job. I'm just trying to figure out if we are doing something in common that could be causing these problems. My vmm is a physical box. One more question... are you doing off-host(agentless) backups or on-host(agent installed on host) backups?

[hopways]

Hi Gostev, if no patch update or work around is going to be released where does that leave the customer who has just purchased 14 x enterprise licenses ?
who cannot back up his hyper-v cluster

also i have looked further into the cluster csv issue
Micrsosoft state that if you use the software VSS provider then the csv is placed into redirected mode and that all i/o will go via SMB through the network to that host that owns the csv resource no matter how many VM's sit on that csv and i think that is why it 'crashes/hangs' to go from direct 4Gb fc to 1 1gb nic for i/o traffic would have any vm/cluster wobble can you confirm that this is probably the issue

when a VM is being backed up the csv stays in redirect mode until the backup is complete

if you can use a hardware vendor VSS Microsoft say it will use the SAN for dealing with snapshot creation and the csv will stay redirected for a very short time

look at the article below and it describes issues with backing up csv volumes

http://blogs.technet.com/b/askcore/arch ... e-csv.aspx

can anyone help !
i do have a case open with veeam at present but if it is by design by Microsoft can we get round it or do we just say Veeam cannot support CSV clusters !
at this point in time

[smurteira]

I've been running DPM2010 for a year and my hosts never got a BSOD during backup. This only started happening when I started using Veeam. My question to you then would be, what is Veeam doing differently than DPM? I was using software vss on DPM , same that I'm using on Veeam.

My other question is, do you guys have a buyback program for you licenses?

[Gostev]

Hi Gostev, if no patch update or work around is going to be released

Looks like you did not read my previous post entirely

do we just say Veeam cannot support CSV clusters !

That would not be true, as currently we are only talking about a few affected customers. Reading through my previous response, workarounds introduced in v6 patch #3 prevented the issue from surfacing for virtually all customers who ever had it. Some of you (see starting from the previous page) still report the issue w/patch 3 installed, and our support and R&D are currently researching the similarities in their Hyper-V configuration to understand what is in common between you. There are also immediate, Microsoft-blessed workarounds available which prevent this BSOD, I have linked them earlier in this topic.

My question to you then would be, what is Veeam doing differently than DPM?

As a long-standing DPM user, you probably know the answer... we are doing just about everything differently, and much better just need to nail down this thing. However, you gave me a very good idea. Do you still have DPM installed? If yes, would it be possible to start DPM backup and let our R&D observe CSV ownership change behavior when DPM triggers it?

The biggest problem for us is that the issue is not reproducible outside of a few environments, and thus we cannot really perform similar research in our own lab, where we have not seen this issue even with the RTM v6 code. I am aware that this situation requires a lot of patience from you, but we really need your help here...

By the way, there is also a small chance that some recent Microsoft hotfix possibly brought this issue, and now even DPM will start failing. So, this would be a very good test to perform indeed.

As for your licensing question, it is best to contact your Veeam sales rep for those, as I am completely not involved with sales.

[smurteira]

Yes, I would be ok with you observing a DPM backup in my environment. Let's schedule for monday at 1pm. You can get my contact information from case number 5174329

[Gostev]

Thank you! I've passed this onto R&D.

[collinp]

I am doing onhost agent backups. While I do have the Hyper-v proxy configured, I don't believe it is doing anything since I am using the software vss provider. I still see 98% source as the bottleneck.

For now, I am thinking that load may have something to do with this. I know that if I install a basic windows installation from template on a vm, I would have no problem with backing up this vm. It seems to be happening with one of the larger vm's in terms of processor usage. I think we could better isolate the problem if you found out if a particular vm is causing this problem. Then we can narrow our focus on similarities between these particular vm's (integration services version, etc). It is easy to do. Just filter the hyper-v host logs for event id 1001 and under source in the filter settings check the boxes that start with "WER". This will bring up the checkbug events that the Hyper-v host produces just after it comes back online after BSOD. Then go into your Veeam job history and see which job(s) were running when the Hyper-v host 1001 occured. I can't test anymore because my trial ran out. But I was going to put a large cpu load on a vm in our test environment which is completely different than our production environment. I have a calculation I can do in the windows calculator program that will spike the cpu. Then I was going to try to backup this vm to see if I can reproduce the problem.

Although I believe changing the cluster settings could band aid this, I think it would be in the best interest to fix the veeam software than to change default microsoft cluster settings that everyone is using. Yes, DPM is more stable, but it still isn't the answer - No dedup, proprietary file system with no drive letter that other backup systems can't back up to tape, need a separate backup job from vm image level backup job in order to be able to do granular file restores. Anyhow, here are some other things to compare with my environment to try to isolate this.

-Onhost proxy
-FibreChannel Storage
-Powerpath multipathing
-Software vss provider
-Hyper-v host is HP DL385 G7 with 128gb memory and dual 12 core amd procs. All hp server software is instaleld (agents, network config utility, system management homepage, etc)
-1 nic for vm network, 1 nic for host, 1 nic for csv, 1 nic for live migration
-23 vm's in a single veeam backup job
-Reverse incremental backup. Kept defaults for rest of backup settings
-No antivirus on hyper-v hosts at this time although there was at one time with exclusions based on Microsoft recommendations
-Windows datacenter edition 2008 r2 on each host sp1. Full installation (not running core)
-BSOD of Hyper-v host happens later in the job most of the time

[Gostev]

Collin, please feel free to email sales@veeam.com to request the extended trial license. You may reference this post in your request, PM me if you have any problems getting it.

[Gostev]

Quick update and some good news. At least one occurrence when the patch #3 did not help was only due to the fact that the Hyper-V storage was not listed as CSV storage in the Veeam B&R configuration database, so the newly introduced patch #3 logic around CSV ownership change handling was simply not leveraged by the job at all!

The storage connected to Hyper-V host is first detected when you register the host or cluster with the Veeam B&R, and is then automatically rescanned every 24 hours. On the support case in question, debug logs show that these storage rescans are failing due to some permission errors.

Alternatively, for example, adding Hyper-V host as standalone host first (no CSV storage is detected), but then re-adding as a part of the cluster may also potentially cause the same issue of storage not being correctly "marked" as CSV. And again, when B&R engine has no awareness of CSV presence, it simply does not use CSV-specific processing logic.

Please right-click each Hyper-V host in the Veeam B&R management tree, select Manage Volumes, and make sure you can see all your existing CSV volumes listed there with the type "CSV" instead of "Local". If this is not so, click Rescan in the same window. If rescan does correct the volume type, let our support know and they will troubleshoot that for you.

[dsouza]

Hello,

I had this issue with two of my Hyper-V clusters this weekend, and I had previously done successfully backups of these Hyper-V clusters, but they were NOT added as SCVMM servers, and last Friday I changed it from Hyper-V clusters to SCVMM managed servers, so I believe that this issue is related to SCVVM managed hosts only.
I will try a new backup without adding Hyper-V clusters as SCVMM servers and let you know the results.

BR
Diego.

[dsouza]

Hi,

Hyper-V Cluster servers added without using SCVMM did the trick.
Backup just executed successfully.

BR.
Diego.

[collinp]

Under "manage volumes" on the hyper-v hosts, mine show "local", even after a rescan. I emailed the Veeam support person who is managing my case with a screenprint.

[smurteira]

Veeam has been working with us on this issue over this past week. I'll report back with progress.

[collinp]

Yes, please let me know, thanks. My ticket has been open with level 3 for a week with no response so hopefully between the two of us we can get an answer.

[Gostev]

For those who is still experience BSOD after installing patch #3, we have confirmed that the issue is caused by 3rd party antivirus products affecting our change tracking logic. We were finally able to reproduce the issue in our own lab by installing McAfee. We also saw the same issue caused by another antivirus in the customer's environment.

We are looking at ways to work around this.

[sho]

Just to add that I'm seeing BSOD issues on my hosts with patch 3 installed and Sophos AV running. With all Sophos services disabled on my hosts, Veeam backups complete successfully.

Any news on how Veeam are progressing with this issue?

[Gostev]

Stuart, have you been in contact with out support? I believe, they have a patch now.

[sho]

I do have a support case open - id 5162204

Thanks

[Gostev]

Checked, and looks like the patch is still being tested internally, and piloted with a single customer who uses McAfee.

I've asked to provide you with the pilot fix as well. However, we cannot be sure if it will help with your antivirus. If it does not, please send us the crashdump.

We've seen different antiviruses affecting system behavior differently. Many of them are literally "braking" some OS functionality to make the life harder for viruses, however this obviously affects "normal" low-level applications like our change tracking as well. Best bet with any antivirus is to exclude CSV volumes from real-time antivirus protection, but unfortunately this may not disable that obfuscation functionality that causes us problems.

[smurteira]

I'm not sure if I'm that person, however Veeam has installed the patch on my Dev environment (with Mcafee) and so far it has been working great. Hoping this fixes it.

[gscherbel]

I am experiencing the same problem. Any update as to a fix?


Thanks

[Vitaliy S.]

Hi Gerry, based on the information from this ticket - 5162204, the hotfix did help to resolve this issue, so please reach our support guys directly to download it. Thanks!