VEEAM Hyper-V CSV Backup - Blue Screen

Hyper-V specific discussions

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby Gostev » Fri Mar 09, 2012 5:14 pm

collinp wrote:Hopefully there is a patch 4 that will address this?

We are not planning on releasing one. Patch 3 already includes a number of workarounds aimed to minimize the chance of this issue appearing, and in fact it helped 90% of customers who was having this issue (literally, we have closed over 30 support cases with that patch released). Unfortunately, we cannot really "fix" this issue from our side, because the issue sits in Windows 2008 Failover Clustering design. It can be reproduced without our product present, and is pretty common overall (otherwise, Microsoft would not have create the above blog post 2 years ago). However, we are determined to continue working with the remaining affected customers to see what else can we do from our side in terms of workarounds. Thanks!
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby collinp » Fri Mar 09, 2012 6:28 pm

smurteira,

On my backup jobs, I am only backing up specific vm's so I am adding vm's to my job on an individual basis. But I am drilling down through the vmm and cluster levels in order to add the specific vm's. I have tried backing up at the vmm and cluster level in the past. Not sure if that makes a difference in terms of how the job is processed. The other thing that may be different in my environment is my VMM server is virtualized.
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby smurteira » Fri Mar 09, 2012 6:49 pm

Got it. I had the cluster and vms into veeam by adding in the vmm, then I add each vm individually to the backup job. I'm just trying to figure out if we are doing something in common that could be causing these problems. My vmm is a physical box. One more question... are you doing off-host(agentless) backups or on-host(agent installed on host) backups?
smurteira
Enthusiast
 
Posts: 26
Liked: never
Joined: Wed Feb 22, 2012 1:45 pm
Full Name: Sergio Murteira

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby hopways » Fri Mar 09, 2012 7:35 pm

Hi Gostev, if no patch update or work around is going to be released where does that leave the customer who has just purchased 14 x enterprise licenses ?
who cannot back up his hyper-v cluster

also i have looked further into the cluster csv issue
Micrsosoft state that if you use the software VSS provider then the csv is placed into redirected mode and that all i/o will go via SMB through the network to that host that owns the csv resource no matter how many VM's sit on that csv and i think that is why it 'crashes/hangs' to go from direct 4Gb fc to 1 1gb nic for i/o traffic would have any vm/cluster wobble can you confirm that this is probably the issue

when a VM is being backed up the csv stays in redirect mode until the backup is complete

if you can use a hardware vendor VSS Microsoft say it will use the SAN for dealing with snapshot creation and the csv will stay redirected for a very short time

look at the article below and it describes issues with backing up csv volumes

http://blogs.technet.com/b/askcore/arch ... e-csv.aspx

can anyone help !
i do have a case open with veeam at present but if it is by design by Microsoft can we get round it or do we just say Veeam cannot support CSV clusters !
at this point in time :!:
hopways
Novice
 
Posts: 6
Liked: 4 times
Joined: Wed Mar 07, 2012 3:36 pm
Full Name: john waugh

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby smurteira » Fri Mar 09, 2012 8:34 pm

I've been running DPM2010 for a year and my hosts never got a BSOD during backup. This only started happening when I started using Veeam. My question to you then would be, what is Veeam doing differently than DPM? I was using software vss on DPM , same that I'm using on Veeam.

My other question is, do you guys have a buyback program for you licenses?
smurteira
Enthusiast
 
Posts: 26
Liked: never
Joined: Wed Feb 22, 2012 1:45 pm
Full Name: Sergio Murteira

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby Gostev » Fri Mar 09, 2012 9:14 pm

hopways wrote:Hi Gostev, if no patch update or work around is going to be released

Looks like you did not read my previous post entirely :wink:

hopways wrote:do we just say Veeam cannot support CSV clusters !

That would not be true, as currently we are only talking about a few affected customers. Reading through my previous response, workarounds introduced in v6 patch #3 prevented the issue from surfacing for virtually all customers who ever had it. Some of you (see starting from the previous page) still report the issue w/patch 3 installed, and our support and R&D are currently researching the similarities in their Hyper-V configuration to understand what is in common between you. There are also immediate, Microsoft-blessed workarounds available which prevent this BSOD, I have linked them earlier in this topic.

smurteira wrote:My question to you then would be, what is Veeam doing differently than DPM?

As a long-standing DPM user, you probably know the answer... we are doing just about everything differently, and much better ;) just need to nail down this thing. However, you gave me a very good idea. Do you still have DPM installed? If yes, would it be possible to start DPM backup and let our R&D observe CSV ownership change behavior when DPM triggers it?

The biggest problem for us is that the issue is not reproducible outside of a few environments, and thus we cannot really perform similar research in our own lab, where we have not seen this issue even with the RTM v6 code. I am aware that this situation requires a lot of patience from you, but we really need your help here...

By the way, there is also a small chance that some recent Microsoft hotfix possibly brought this issue, and now even DPM will start failing. So, this would be a very good test to perform indeed.

As for your licensing question, it is best to contact your Veeam sales rep for those, as I am completely not involved with sales.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby smurteira » Fri Mar 09, 2012 9:43 pm

Yes, I would be ok with you observing a DPM backup in my environment. Let's schedule for monday at 1pm. You can get my contact information from case number 5174329
smurteira
Enthusiast
 
Posts: 26
Liked: never
Joined: Wed Feb 22, 2012 1:45 pm
Full Name: Sergio Murteira

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby Gostev » Fri Mar 09, 2012 11:49 pm

Thank you! I've passed this onto R&D.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby collinp » Sat Mar 10, 2012 2:35 am

I am doing onhost agent backups. While I do have the Hyper-v proxy configured, I don't believe it is doing anything since I am using the software vss provider. I still see 98% source as the bottleneck.

For now, I am thinking that load may have something to do with this. I know that if I install a basic windows installation from template on a vm, I would have no problem with backing up this vm. It seems to be happening with one of the larger vm's in terms of processor usage. I think we could better isolate the problem if you found out if a particular vm is causing this problem. Then we can narrow our focus on similarities between these particular vm's (integration services version, etc). It is easy to do. Just filter the hyper-v host logs for event id 1001 and under source in the filter settings check the boxes that start with "WER". This will bring up the checkbug events that the Hyper-v host produces just after it comes back online after BSOD. Then go into your Veeam job history and see which job(s) were running when the Hyper-v host 1001 occured. I can't test anymore because my trial ran out. But I was going to put a large cpu load on a vm in our test environment which is completely different than our production environment. I have a calculation I can do in the windows calculator program that will spike the cpu. Then I was going to try to backup this vm to see if I can reproduce the problem.

Although I believe changing the cluster settings could band aid this, I think it would be in the best interest to fix the veeam software than to change default microsoft cluster settings that everyone is using. Yes, DPM is more stable, but it still isn't the answer - No dedup, proprietary file system with no drive letter that other backup systems can't back up to tape, need a separate backup job from vm image level backup job in order to be able to do granular file restores. Anyhow, here are some other things to compare with my environment to try to isolate this.

-Onhost proxy
-FibreChannel Storage
-Powerpath multipathing
-Software vss provider
-Hyper-v host is HP DL385 G7 with 128gb memory and dual 12 core amd procs. All hp server software is instaleld (agents, network config utility, system management homepage, etc)
-1 nic for vm network, 1 nic for host, 1 nic for csv, 1 nic for live migration
-23 vm's in a single veeam backup job
-Reverse incremental backup. Kept defaults for rest of backup settings
-No antivirus on hyper-v hosts at this time although there was at one time with exclusions based on Microsoft recommendations
-Windows datacenter edition 2008 r2 on each host sp1. Full installation (not running core)
-BSOD of Hyper-v host happens later in the job most of the time
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby Gostev » Sat Mar 10, 2012 11:08 am

Collin, please feel free to email sales@veeam.com to request the extended trial license. You may reference this post in your request, PM me if you have any problems getting it.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby Gostev » Sun Mar 11, 2012 12:41 pm

Quick update and some good news. At least one occurrence when the patch #3 did not help was only due to the fact that the Hyper-V storage was not listed as CSV storage in the Veeam B&R configuration database, so the newly introduced patch #3 logic around CSV ownership change handling was simply not leveraged by the job at all!

The storage connected to Hyper-V host is first detected when you register the host or cluster with the Veeam B&R, and is then automatically rescanned every 24 hours. On the support case in question, debug logs show that these storage rescans are failing due to some permission errors.

Alternatively, for example, adding Hyper-V host as standalone host first (no CSV storage is detected), but then re-adding as a part of the cluster may also potentially cause the same issue of storage not being correctly "marked" as CSV. And again, when B&R engine has no awareness of CSV presence, it simply does not use CSV-specific processing logic.

Please right-click each Hyper-V host in the Veeam B&R management tree, select Manage Volumes, and make sure you can see all your existing CSV volumes listed there with the type "CSV" instead of "Local". If this is not so, click Rescan in the same window. If rescan does correct the volume type, let our support know and they will troubleshoot that for you.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby dsouza » Sun Mar 11, 2012 3:02 pm

Hello,

I had this issue with two of my Hyper-V clusters this weekend, and I had previously done successfully backups of these Hyper-V clusters, but they were NOT added as SCVMM servers, and last Friday I changed it from Hyper-V clusters to SCVMM managed servers, so I believe that this issue is related to SCVVM managed hosts only.
I will try a new backup without adding Hyper-V clusters as SCVMM servers and let you know the results.

BR
Diego.
dsouza
Novice
 
Posts: 3
Liked: never
Joined: Wed Mar 07, 2012 7:40 pm
Full Name: Diego Souza

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby dsouza » Mon Mar 12, 2012 10:18 am

Hi,

Hyper-V Cluster servers added without using SCVMM did the trick.
Backup just executed successfully.

BR.
Diego.
dsouza
Novice
 
Posts: 3
Liked: never
Joined: Wed Mar 07, 2012 7:40 pm
Full Name: Diego Souza

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby collinp » Mon Mar 12, 2012 4:06 pm

Under "manage volumes" on the hyper-v hosts, mine show "local", even after a rescan. I emailed the Veeam support person who is managing my case with a screenprint.
collinp
Expert
 
Posts: 115
Liked: 10 times
Joined: Tue Feb 14, 2012 8:56 pm
Full Name: Collin P

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby smurteira » Fri Mar 16, 2012 12:29 pm

Veeam has been working with us on this issue over this past week. I'll report back with progress.
smurteira
Enthusiast
 
Posts: 26
Liked: never
Joined: Wed Feb 22, 2012 1:45 pm
Full Name: Sergio Murteira

PreviousNext

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 11 guests