Host-based backup of Microsoft Hyper-V VMs.
Gostev
Chief Product Officer
Posts: 31768
Liked: 7271 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

collinp wrote:Hopefully there is a patch 4 that will address this?
We are not planning on releasing one. Patch 3 already includes a number of workarounds aimed to minimize the chance of this issue appearing, and in fact it helped 90% of customers who was having this issue (literally, we have closed over 30 support cases with that patch released). Unfortunately, we cannot really "fix" this issue from our side, because the issue sits in Windows 2008 Failover Clustering design. It can be reproduced without our product present, and is pretty common overall (otherwise, Microsoft would not have create the above blog post 2 years ago). However, we are determined to continue working with the remaining affected customers to see what else can we do from our side in terms of workarounds. Thanks!
collinp
Expert
Posts: 239
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by collinp »

smurteira,

On my backup jobs, I am only backing up specific vm's so I am adding vm's to my job on an individual basis. But I am drilling down through the vmm and cluster levels in order to add the specific vm's. I have tried backing up at the vmm and cluster level in the past. Not sure if that makes a difference in terms of how the job is processed. The other thing that may be different in my environment is my VMM server is virtualized.
smurteira
Enthusiast
Posts: 26
Liked: never
Joined: Feb 22, 2012 1:45 pm
Full Name: Sergio Murteira
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by smurteira »

Got it. I had the cluster and vms into veeam by adding in the vmm, then I add each vm individually to the backup job. I'm just trying to figure out if we are doing something in common that could be causing these problems. My vmm is a physical box. One more question... are you doing off-host(agentless) backups or on-host(agent installed on host) backups?
hopways
Novice
Posts: 6
Liked: 4 times
Joined: Mar 07, 2012 3:36 pm
Full Name: john waugh
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by hopways »

Hi Gostev, if no patch update or work around is going to be released where does that leave the customer who has just purchased 14 x enterprise licenses ?
who cannot back up his hyper-v cluster

also i have looked further into the cluster csv issue
Micrsosoft state that if you use the software VSS provider then the csv is placed into redirected mode and that all i/o will go via SMB through the network to that host that owns the csv resource no matter how many VM's sit on that csv and i think that is why it 'crashes/hangs' to go from direct 4Gb fc to 1 1gb nic for i/o traffic would have any vm/cluster wobble can you confirm that this is probably the issue

when a VM is being backed up the csv stays in redirect mode until the backup is complete

if you can use a hardware vendor VSS Microsoft say it will use the SAN for dealing with snapshot creation and the csv will stay redirected for a very short time

look at the article below and it describes issues with backing up csv volumes

http://blogs.technet.com/b/askcore/arch ... e-csv.aspx

can anyone help !
i do have a case open with veeam at present but if it is by design by Microsoft can we get round it or do we just say Veeam cannot support CSV clusters !
at this point in time :!:
smurteira
Enthusiast
Posts: 26
Liked: never
Joined: Feb 22, 2012 1:45 pm
Full Name: Sergio Murteira
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by smurteira »

I've been running DPM2010 for a year and my hosts never got a BSOD during backup. This only started happening when I started using Veeam. My question to you then would be, what is Veeam doing differently than DPM? I was using software vss on DPM , same that I'm using on Veeam.

My other question is, do you guys have a buyback program for you licenses?
Gostev
Chief Product Officer
Posts: 31768
Liked: 7271 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

hopways wrote:Hi Gostev, if no patch update or work around is going to be released
Looks like you did not read my previous post entirely :wink:
hopways wrote:do we just say Veeam cannot support CSV clusters !
That would not be true, as currently we are only talking about a few affected customers. Reading through my previous response, workarounds introduced in v6 patch #3 prevented the issue from surfacing for virtually all customers who ever had it. Some of you (see starting from the previous page) still report the issue w/patch 3 installed, and our support and R&D are currently researching the similarities in their Hyper-V configuration to understand what is in common between you. There are also immediate, Microsoft-blessed workarounds available which prevent this BSOD, I have linked them earlier in this topic.
smurteira wrote:My question to you then would be, what is Veeam doing differently than DPM?
As a long-standing DPM user, you probably know the answer... we are doing just about everything differently, and much better ;) just need to nail down this thing. However, you gave me a very good idea. Do you still have DPM installed? If yes, would it be possible to start DPM backup and let our R&D observe CSV ownership change behavior when DPM triggers it?

The biggest problem for us is that the issue is not reproducible outside of a few environments, and thus we cannot really perform similar research in our own lab, where we have not seen this issue even with the RTM v6 code. I am aware that this situation requires a lot of patience from you, but we really need your help here...

By the way, there is also a small chance that some recent Microsoft hotfix possibly brought this issue, and now even DPM will start failing. So, this would be a very good test to perform indeed.

As for your licensing question, it is best to contact your Veeam sales rep for those, as I am completely not involved with sales.
smurteira
Enthusiast
Posts: 26
Liked: never
Joined: Feb 22, 2012 1:45 pm
Full Name: Sergio Murteira
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by smurteira »

Yes, I would be ok with you observing a DPM backup in my environment. Let's schedule for monday at 1pm. You can get my contact information from case number 5174329
Gostev
Chief Product Officer
Posts: 31768
Liked: 7271 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

Thank you! I've passed this onto R&D.
collinp
Expert
Posts: 239
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by collinp »

I am doing onhost agent backups. While I do have the Hyper-v proxy configured, I don't believe it is doing anything since I am using the software vss provider. I still see 98% source as the bottleneck.

For now, I am thinking that load may have something to do with this. I know that if I install a basic windows installation from template on a vm, I would have no problem with backing up this vm. It seems to be happening with one of the larger vm's in terms of processor usage. I think we could better isolate the problem if you found out if a particular vm is causing this problem. Then we can narrow our focus on similarities between these particular vm's (integration services version, etc). It is easy to do. Just filter the hyper-v host logs for event id 1001 and under source in the filter settings check the boxes that start with "WER". This will bring up the checkbug events that the Hyper-v host produces just after it comes back online after BSOD. Then go into your Veeam job history and see which job(s) were running when the Hyper-v host 1001 occured. I can't test anymore because my trial ran out. But I was going to put a large cpu load on a vm in our test environment which is completely different than our production environment. I have a calculation I can do in the windows calculator program that will spike the cpu. Then I was going to try to backup this vm to see if I can reproduce the problem.

Although I believe changing the cluster settings could band aid this, I think it would be in the best interest to fix the veeam software than to change default microsoft cluster settings that everyone is using. Yes, DPM is more stable, but it still isn't the answer - No dedup, proprietary file system with no drive letter that other backup systems can't back up to tape, need a separate backup job from vm image level backup job in order to be able to do granular file restores. Anyhow, here are some other things to compare with my environment to try to isolate this.

-Onhost proxy
-FibreChannel Storage
-Powerpath multipathing
-Software vss provider
-Hyper-v host is HP DL385 G7 with 128gb memory and dual 12 core amd procs. All hp server software is instaleld (agents, network config utility, system management homepage, etc)
-1 nic for vm network, 1 nic for host, 1 nic for csv, 1 nic for live migration
-23 vm's in a single veeam backup job
-Reverse incremental backup. Kept defaults for rest of backup settings
-No antivirus on hyper-v hosts at this time although there was at one time with exclusions based on Microsoft recommendations
-Windows datacenter edition 2008 r2 on each host sp1. Full installation (not running core)
-BSOD of Hyper-v host happens later in the job most of the time
Gostev
Chief Product Officer
Posts: 31768
Liked: 7271 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

Collin, please feel free to email sales@veeam.com to request the extended trial license. You may reference this post in your request, PM me if you have any problems getting it.
Gostev
Chief Product Officer
Posts: 31768
Liked: 7271 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

Quick update and some good news. At least one occurrence when the patch #3 did not help was only due to the fact that the Hyper-V storage was not listed as CSV storage in the Veeam B&R configuration database, so the newly introduced patch #3 logic around CSV ownership change handling was simply not leveraged by the job at all!

The storage connected to Hyper-V host is first detected when you register the host or cluster with the Veeam B&R, and is then automatically rescanned every 24 hours. On the support case in question, debug logs show that these storage rescans are failing due to some permission errors.

Alternatively, for example, adding Hyper-V host as standalone host first (no CSV storage is detected), but then re-adding as a part of the cluster may also potentially cause the same issue of storage not being correctly "marked" as CSV. And again, when B&R engine has no awareness of CSV presence, it simply does not use CSV-specific processing logic.

Please right-click each Hyper-V host in the Veeam B&R management tree, select Manage Volumes, and make sure you can see all your existing CSV volumes listed there with the type "CSV" instead of "Local". If this is not so, click Rescan in the same window. If rescan does correct the volume type, let our support know and they will troubleshoot that for you.
dsouza
Novice
Posts: 3
Liked: never
Joined: Mar 07, 2012 7:40 pm
Full Name: Diego Souza
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by dsouza »

Hello,

I had this issue with two of my Hyper-V clusters this weekend, and I had previously done successfully backups of these Hyper-V clusters, but they were NOT added as SCVMM servers, and last Friday I changed it from Hyper-V clusters to SCVMM managed servers, so I believe that this issue is related to SCVVM managed hosts only.
I will try a new backup without adding Hyper-V clusters as SCVMM servers and let you know the results.

BR
Diego.
dsouza
Novice
Posts: 3
Liked: never
Joined: Mar 07, 2012 7:40 pm
Full Name: Diego Souza
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by dsouza »

Hi,

Hyper-V Cluster servers added without using SCVMM did the trick.
Backup just executed successfully.

BR.
Diego.
collinp
Expert
Posts: 239
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by collinp »

Under "manage volumes" on the hyper-v hosts, mine show "local", even after a rescan. I emailed the Veeam support person who is managing my case with a screenprint.
smurteira
Enthusiast
Posts: 26
Liked: never
Joined: Feb 22, 2012 1:45 pm
Full Name: Sergio Murteira
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by smurteira »

Veeam has been working with us on this issue over this past week. I'll report back with progress.
collinp
Expert
Posts: 239
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by collinp »

Yes, please let me know, thanks. My ticket has been open with level 3 for a week with no response so hopefully between the two of us we can get an answer.
Gostev
Chief Product Officer
Posts: 31768
Liked: 7271 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

For those who is still experience BSOD after installing patch #3, we have confirmed that the issue is caused by 3rd party antivirus products affecting our change tracking logic. We were finally able to reproduce the issue in our own lab by installing McAfee. We also saw the same issue caused by another antivirus in the customer's environment.

We are looking at ways to work around this.
sho
Novice
Posts: 6
Liked: never
Joined: Mar 21, 2012 12:47 pm
Full Name: Stuart
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by sho »

Just to add that I'm seeing BSOD issues on my hosts with patch 3 installed and Sophos AV running. With all Sophos services disabled on my hosts, Veeam backups complete successfully.

Any news on how Veeam are progressing with this issue?
Gostev
Chief Product Officer
Posts: 31768
Liked: 7271 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

Stuart, have you been in contact with out support? I believe, they have a patch now.
sho
Novice
Posts: 6
Liked: never
Joined: Mar 21, 2012 12:47 pm
Full Name: Stuart
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by sho »

I do have a support case open - id 5162204

Thanks
Gostev
Chief Product Officer
Posts: 31768
Liked: 7271 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

Checked, and looks like the patch is still being tested internally, and piloted with a single customer who uses McAfee.

I've asked to provide you with the pilot fix as well. However, we cannot be sure if it will help with your antivirus. If it does not, please send us the crashdump.

We've seen different antiviruses affecting system behavior differently. Many of them are literally "braking" some OS functionality to make the life harder for viruses, however this obviously affects "normal" low-level applications like our change tracking as well. Best bet with any antivirus is to exclude CSV volumes from real-time antivirus protection, but unfortunately this may not disable that obfuscation functionality that causes us problems.
smurteira
Enthusiast
Posts: 26
Liked: never
Joined: Feb 22, 2012 1:45 pm
Full Name: Sergio Murteira
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by smurteira »

I'm not sure if I'm that person, however Veeam has installed the patch on my Dev environment (with Mcafee) and so far it has been working great. Hoping this fixes it.
gscherbel
Lurker
Posts: 1
Liked: never
Joined: May 23, 2012 4:49 pm
Full Name: Gerry Scherbel
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by gscherbel »

I am experiencing the same problem. Any update as to a fix?


Thanks
Vitaliy S.
VP, Product Management
Posts: 27357
Liked: 2788 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Vitaliy S. »

Hi Gerry, based on the information from this ticket - 5162204, the hotfix did help to resolve this issue, so please reach our support guys directly to download it. Thanks!
Post Reply

Who is online

Users browsing this forum: daniel.sironi and 16 guests