We are not planning on releasing one. Patch 3 already includes a number of workarounds aimed to minimize the chance of this issue appearing, and in fact it helped 90% of customers who was having this issue (literally, we have closed over 30 support cases with that patch released). Unfortunately, we cannot really "fix" this issue from our side, because the issue sits in Windows 2008 Failover Clustering design. It can be reproduced without our product present, and is pretty common overall (otherwise, Microsoft would not have create the above blog post 2 years ago). However, we are determined to continue working with the remaining affected customers to see what else can we do from our side in terms of workarounds. Thanks!collinp wrote:Hopefully there is a patch 4 that will address this?
-
- Chief Product Officer
- Posts: 31768
- Liked: 7271 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
-
- Expert
- Posts: 239
- Liked: 13 times
- Joined: Feb 14, 2012 8:56 pm
- Full Name: Collin P
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
smurteira,
On my backup jobs, I am only backing up specific vm's so I am adding vm's to my job on an individual basis. But I am drilling down through the vmm and cluster levels in order to add the specific vm's. I have tried backing up at the vmm and cluster level in the past. Not sure if that makes a difference in terms of how the job is processed. The other thing that may be different in my environment is my VMM server is virtualized.
On my backup jobs, I am only backing up specific vm's so I am adding vm's to my job on an individual basis. But I am drilling down through the vmm and cluster levels in order to add the specific vm's. I have tried backing up at the vmm and cluster level in the past. Not sure if that makes a difference in terms of how the job is processed. The other thing that may be different in my environment is my VMM server is virtualized.
-
- Enthusiast
- Posts: 26
- Liked: never
- Joined: Feb 22, 2012 1:45 pm
- Full Name: Sergio Murteira
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Got it. I had the cluster and vms into veeam by adding in the vmm, then I add each vm individually to the backup job. I'm just trying to figure out if we are doing something in common that could be causing these problems. My vmm is a physical box. One more question... are you doing off-host(agentless) backups or on-host(agent installed on host) backups?
-
- Novice
- Posts: 6
- Liked: 4 times
- Joined: Mar 07, 2012 3:36 pm
- Full Name: john waugh
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Hi Gostev, if no patch update or work around is going to be released where does that leave the customer who has just purchased 14 x enterprise licenses ?
who cannot back up his hyper-v cluster
also i have looked further into the cluster csv issue
Micrsosoft state that if you use the software VSS provider then the csv is placed into redirected mode and that all i/o will go via SMB through the network to that host that owns the csv resource no matter how many VM's sit on that csv and i think that is why it 'crashes/hangs' to go from direct 4Gb fc to 1 1gb nic for i/o traffic would have any vm/cluster wobble can you confirm that this is probably the issue
when a VM is being backed up the csv stays in redirect mode until the backup is complete
if you can use a hardware vendor VSS Microsoft say it will use the SAN for dealing with snapshot creation and the csv will stay redirected for a very short time
look at the article below and it describes issues with backing up csv volumes
http://blogs.technet.com/b/askcore/arch ... e-csv.aspx
can anyone help !
i do have a case open with veeam at present but if it is by design by Microsoft can we get round it or do we just say Veeam cannot support CSV clusters !
at this point in time
who cannot back up his hyper-v cluster
also i have looked further into the cluster csv issue
Micrsosoft state that if you use the software VSS provider then the csv is placed into redirected mode and that all i/o will go via SMB through the network to that host that owns the csv resource no matter how many VM's sit on that csv and i think that is why it 'crashes/hangs' to go from direct 4Gb fc to 1 1gb nic for i/o traffic would have any vm/cluster wobble can you confirm that this is probably the issue
when a VM is being backed up the csv stays in redirect mode until the backup is complete
if you can use a hardware vendor VSS Microsoft say it will use the SAN for dealing with snapshot creation and the csv will stay redirected for a very short time
look at the article below and it describes issues with backing up csv volumes
http://blogs.technet.com/b/askcore/arch ... e-csv.aspx
can anyone help !
i do have a case open with veeam at present but if it is by design by Microsoft can we get round it or do we just say Veeam cannot support CSV clusters !
at this point in time
-
- Enthusiast
- Posts: 26
- Liked: never
- Joined: Feb 22, 2012 1:45 pm
- Full Name: Sergio Murteira
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
I've been running DPM2010 for a year and my hosts never got a BSOD during backup. This only started happening when I started using Veeam. My question to you then would be, what is Veeam doing differently than DPM? I was using software vss on DPM , same that I'm using on Veeam.
My other question is, do you guys have a buyback program for you licenses?
My other question is, do you guys have a buyback program for you licenses?
-
- Chief Product Officer
- Posts: 31768
- Liked: 7271 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Looks like you did not read my previous post entirelyhopways wrote:Hi Gostev, if no patch update or work around is going to be released
That would not be true, as currently we are only talking about a few affected customers. Reading through my previous response, workarounds introduced in v6 patch #3 prevented the issue from surfacing for virtually all customers who ever had it. Some of you (see starting from the previous page) still report the issue w/patch 3 installed, and our support and R&D are currently researching the similarities in their Hyper-V configuration to understand what is in common between you. There are also immediate, Microsoft-blessed workarounds available which prevent this BSOD, I have linked them earlier in this topic.hopways wrote:do we just say Veeam cannot support CSV clusters !
As a long-standing DPM user, you probably know the answer... we are doing just about everything differently, and much better just need to nail down this thing. However, you gave me a very good idea. Do you still have DPM installed? If yes, would it be possible to start DPM backup and let our R&D observe CSV ownership change behavior when DPM triggers it?smurteira wrote:My question to you then would be, what is Veeam doing differently than DPM?
The biggest problem for us is that the issue is not reproducible outside of a few environments, and thus we cannot really perform similar research in our own lab, where we have not seen this issue even with the RTM v6 code. I am aware that this situation requires a lot of patience from you, but we really need your help here...
By the way, there is also a small chance that some recent Microsoft hotfix possibly brought this issue, and now even DPM will start failing. So, this would be a very good test to perform indeed.
As for your licensing question, it is best to contact your Veeam sales rep for those, as I am completely not involved with sales.
-
- Enthusiast
- Posts: 26
- Liked: never
- Joined: Feb 22, 2012 1:45 pm
- Full Name: Sergio Murteira
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Yes, I would be ok with you observing a DPM backup in my environment. Let's schedule for monday at 1pm. You can get my contact information from case number 5174329
-
- Chief Product Officer
- Posts: 31768
- Liked: 7271 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Thank you! I've passed this onto R&D.
-
- Expert
- Posts: 239
- Liked: 13 times
- Joined: Feb 14, 2012 8:56 pm
- Full Name: Collin P
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
I am doing onhost agent backups. While I do have the Hyper-v proxy configured, I don't believe it is doing anything since I am using the software vss provider. I still see 98% source as the bottleneck.
For now, I am thinking that load may have something to do with this. I know that if I install a basic windows installation from template on a vm, I would have no problem with backing up this vm. It seems to be happening with one of the larger vm's in terms of processor usage. I think we could better isolate the problem if you found out if a particular vm is causing this problem. Then we can narrow our focus on similarities between these particular vm's (integration services version, etc). It is easy to do. Just filter the hyper-v host logs for event id 1001 and under source in the filter settings check the boxes that start with "WER". This will bring up the checkbug events that the Hyper-v host produces just after it comes back online after BSOD. Then go into your Veeam job history and see which job(s) were running when the Hyper-v host 1001 occured. I can't test anymore because my trial ran out. But I was going to put a large cpu load on a vm in our test environment which is completely different than our production environment. I have a calculation I can do in the windows calculator program that will spike the cpu. Then I was going to try to backup this vm to see if I can reproduce the problem.
Although I believe changing the cluster settings could band aid this, I think it would be in the best interest to fix the veeam software than to change default microsoft cluster settings that everyone is using. Yes, DPM is more stable, but it still isn't the answer - No dedup, proprietary file system with no drive letter that other backup systems can't back up to tape, need a separate backup job from vm image level backup job in order to be able to do granular file restores. Anyhow, here are some other things to compare with my environment to try to isolate this.
-Onhost proxy
-FibreChannel Storage
-Powerpath multipathing
-Software vss provider
-Hyper-v host is HP DL385 G7 with 128gb memory and dual 12 core amd procs. All hp server software is instaleld (agents, network config utility, system management homepage, etc)
-1 nic for vm network, 1 nic for host, 1 nic for csv, 1 nic for live migration
-23 vm's in a single veeam backup job
-Reverse incremental backup. Kept defaults for rest of backup settings
-No antivirus on hyper-v hosts at this time although there was at one time with exclusions based on Microsoft recommendations
-Windows datacenter edition 2008 r2 on each host sp1. Full installation (not running core)
-BSOD of Hyper-v host happens later in the job most of the time
For now, I am thinking that load may have something to do with this. I know that if I install a basic windows installation from template on a vm, I would have no problem with backing up this vm. It seems to be happening with one of the larger vm's in terms of processor usage. I think we could better isolate the problem if you found out if a particular vm is causing this problem. Then we can narrow our focus on similarities between these particular vm's (integration services version, etc). It is easy to do. Just filter the hyper-v host logs for event id 1001 and under source in the filter settings check the boxes that start with "WER". This will bring up the checkbug events that the Hyper-v host produces just after it comes back online after BSOD. Then go into your Veeam job history and see which job(s) were running when the Hyper-v host 1001 occured. I can't test anymore because my trial ran out. But I was going to put a large cpu load on a vm in our test environment which is completely different than our production environment. I have a calculation I can do in the windows calculator program that will spike the cpu. Then I was going to try to backup this vm to see if I can reproduce the problem.
Although I believe changing the cluster settings could band aid this, I think it would be in the best interest to fix the veeam software than to change default microsoft cluster settings that everyone is using. Yes, DPM is more stable, but it still isn't the answer - No dedup, proprietary file system with no drive letter that other backup systems can't back up to tape, need a separate backup job from vm image level backup job in order to be able to do granular file restores. Anyhow, here are some other things to compare with my environment to try to isolate this.
-Onhost proxy
-FibreChannel Storage
-Powerpath multipathing
-Software vss provider
-Hyper-v host is HP DL385 G7 with 128gb memory and dual 12 core amd procs. All hp server software is instaleld (agents, network config utility, system management homepage, etc)
-1 nic for vm network, 1 nic for host, 1 nic for csv, 1 nic for live migration
-23 vm's in a single veeam backup job
-Reverse incremental backup. Kept defaults for rest of backup settings
-No antivirus on hyper-v hosts at this time although there was at one time with exclusions based on Microsoft recommendations
-Windows datacenter edition 2008 r2 on each host sp1. Full installation (not running core)
-BSOD of Hyper-v host happens later in the job most of the time
-
- Chief Product Officer
- Posts: 31768
- Liked: 7271 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Collin, please feel free to email sales@veeam.com to request the extended trial license. You may reference this post in your request, PM me if you have any problems getting it.
-
- Chief Product Officer
- Posts: 31768
- Liked: 7271 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Quick update and some good news. At least one occurrence when the patch #3 did not help was only due to the fact that the Hyper-V storage was not listed as CSV storage in the Veeam B&R configuration database, so the newly introduced patch #3 logic around CSV ownership change handling was simply not leveraged by the job at all!
The storage connected to Hyper-V host is first detected when you register the host or cluster with the Veeam B&R, and is then automatically rescanned every 24 hours. On the support case in question, debug logs show that these storage rescans are failing due to some permission errors.
Alternatively, for example, adding Hyper-V host as standalone host first (no CSV storage is detected), but then re-adding as a part of the cluster may also potentially cause the same issue of storage not being correctly "marked" as CSV. And again, when B&R engine has no awareness of CSV presence, it simply does not use CSV-specific processing logic.
Please right-click each Hyper-V host in the Veeam B&R management tree, select Manage Volumes, and make sure you can see all your existing CSV volumes listed there with the type "CSV" instead of "Local". If this is not so, click Rescan in the same window. If rescan does correct the volume type, let our support know and they will troubleshoot that for you.
The storage connected to Hyper-V host is first detected when you register the host or cluster with the Veeam B&R, and is then automatically rescanned every 24 hours. On the support case in question, debug logs show that these storage rescans are failing due to some permission errors.
Alternatively, for example, adding Hyper-V host as standalone host first (no CSV storage is detected), but then re-adding as a part of the cluster may also potentially cause the same issue of storage not being correctly "marked" as CSV. And again, when B&R engine has no awareness of CSV presence, it simply does not use CSV-specific processing logic.
Please right-click each Hyper-V host in the Veeam B&R management tree, select Manage Volumes, and make sure you can see all your existing CSV volumes listed there with the type "CSV" instead of "Local". If this is not so, click Rescan in the same window. If rescan does correct the volume type, let our support know and they will troubleshoot that for you.
-
- Novice
- Posts: 3
- Liked: never
- Joined: Mar 07, 2012 7:40 pm
- Full Name: Diego Souza
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Hello,
I had this issue with two of my Hyper-V clusters this weekend, and I had previously done successfully backups of these Hyper-V clusters, but they were NOT added as SCVMM servers, and last Friday I changed it from Hyper-V clusters to SCVMM managed servers, so I believe that this issue is related to SCVVM managed hosts only.
I will try a new backup without adding Hyper-V clusters as SCVMM servers and let you know the results.
BR
Diego.
I had this issue with two of my Hyper-V clusters this weekend, and I had previously done successfully backups of these Hyper-V clusters, but they were NOT added as SCVMM servers, and last Friday I changed it from Hyper-V clusters to SCVMM managed servers, so I believe that this issue is related to SCVVM managed hosts only.
I will try a new backup without adding Hyper-V clusters as SCVMM servers and let you know the results.
BR
Diego.
-
- Novice
- Posts: 3
- Liked: never
- Joined: Mar 07, 2012 7:40 pm
- Full Name: Diego Souza
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Hi,
Hyper-V Cluster servers added without using SCVMM did the trick.
Backup just executed successfully.
BR.
Diego.
Hyper-V Cluster servers added without using SCVMM did the trick.
Backup just executed successfully.
BR.
Diego.
-
- Expert
- Posts: 239
- Liked: 13 times
- Joined: Feb 14, 2012 8:56 pm
- Full Name: Collin P
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Under "manage volumes" on the hyper-v hosts, mine show "local", even after a rescan. I emailed the Veeam support person who is managing my case with a screenprint.
-
- Enthusiast
- Posts: 26
- Liked: never
- Joined: Feb 22, 2012 1:45 pm
- Full Name: Sergio Murteira
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Veeam has been working with us on this issue over this past week. I'll report back with progress.
-
- Expert
- Posts: 239
- Liked: 13 times
- Joined: Feb 14, 2012 8:56 pm
- Full Name: Collin P
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Yes, please let me know, thanks. My ticket has been open with level 3 for a week with no response so hopefully between the two of us we can get an answer.
-
- Chief Product Officer
- Posts: 31768
- Liked: 7271 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
For those who is still experience BSOD after installing patch #3, we have confirmed that the issue is caused by 3rd party antivirus products affecting our change tracking logic. We were finally able to reproduce the issue in our own lab by installing McAfee. We also saw the same issue caused by another antivirus in the customer's environment.
We are looking at ways to work around this.
We are looking at ways to work around this.
-
- Novice
- Posts: 6
- Liked: never
- Joined: Mar 21, 2012 12:47 pm
- Full Name: Stuart
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Just to add that I'm seeing BSOD issues on my hosts with patch 3 installed and Sophos AV running. With all Sophos services disabled on my hosts, Veeam backups complete successfully.
Any news on how Veeam are progressing with this issue?
Any news on how Veeam are progressing with this issue?
-
- Chief Product Officer
- Posts: 31768
- Liked: 7271 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Stuart, have you been in contact with out support? I believe, they have a patch now.
-
- Novice
- Posts: 6
- Liked: never
- Joined: Mar 21, 2012 12:47 pm
- Full Name: Stuart
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
I do have a support case open - id 5162204
Thanks
Thanks
-
- Chief Product Officer
- Posts: 31768
- Liked: 7271 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Checked, and looks like the patch is still being tested internally, and piloted with a single customer who uses McAfee.
I've asked to provide you with the pilot fix as well. However, we cannot be sure if it will help with your antivirus. If it does not, please send us the crashdump.
We've seen different antiviruses affecting system behavior differently. Many of them are literally "braking" some OS functionality to make the life harder for viruses, however this obviously affects "normal" low-level applications like our change tracking as well. Best bet with any antivirus is to exclude CSV volumes from real-time antivirus protection, but unfortunately this may not disable that obfuscation functionality that causes us problems.
I've asked to provide you with the pilot fix as well. However, we cannot be sure if it will help with your antivirus. If it does not, please send us the crashdump.
We've seen different antiviruses affecting system behavior differently. Many of them are literally "braking" some OS functionality to make the life harder for viruses, however this obviously affects "normal" low-level applications like our change tracking as well. Best bet with any antivirus is to exclude CSV volumes from real-time antivirus protection, but unfortunately this may not disable that obfuscation functionality that causes us problems.
-
- Enthusiast
- Posts: 26
- Liked: never
- Joined: Feb 22, 2012 1:45 pm
- Full Name: Sergio Murteira
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
I'm not sure if I'm that person, however Veeam has installed the patch on my Dev environment (with Mcafee) and so far it has been working great. Hoping this fixes it.
-
- Lurker
- Posts: 1
- Liked: never
- Joined: May 23, 2012 4:49 pm
- Full Name: Gerry Scherbel
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
I am experiencing the same problem. Any update as to a fix?
Thanks
Thanks
-
- VP, Product Management
- Posts: 27357
- Liked: 2788 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: VEEAM Hyper-V CSV Backup - Blue Screen
Hi Gerry, based on the information from this ticket - 5162204, the hotfix did help to resolve this issue, so please reach our support guys directly to download it. Thanks!
Who is online
Users browsing this forum: daniel.sironi and 16 guests