Hardware VSS

[collinp]

Is there an option in Veeam so the backups will never failover to Software VSS? I use onhost hardware vss but there are times when the backup fails over to the Microsoft software CSV provider. I would rather have the backup fail, than switch to software vss.

[veremin]

Hi, Collin. Unfortunately, it’s not possible at the moment. Though, thanks for the feedback; highly-appreciated.

[collinp]

Would there be any ill effects in terms of backups or Hyper-v snapshots if I were to remove the Microsoft CSV provider from HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers? It just seems like we are more susceptible to outages when backups failover to software VSS.

[Vitaliy S.]

Hi Collin,

Theoretically, there shouldn't be any issues, as this VSS provider will not just show up in the list of available providers. If you decide to try it, I would appreciate if you could update this topic with your findings.

Thanks!

[jake1110]

Colin - I'm just curious, what Hardware provider are you using and what version of Windows do you use?

This is off topic a bit, but we're fighting one heck of an issue with Dell Equallogic Hardware VSS and Server 2012, though I've heard of people having issues with 3par as well. Basically, when a snap occurs with the hardware provider, it can trigger a memory leak that drains the host of it's resources. If you migrate the CSV, the leak will actually follow to the next host eventually taking down the cluster. While this certainly isn't a Veeam issue, MS and Dell continually point the finger at each other and there are very long forum threads on technet debating on what to do.

Unfortunately, the only "fix" for everyone so far is to use the MS CSV Writer, which means our backups are much slower...

[collinp]

Here is MS Premier supports answer to our outages


Analysis:

Whenever we take initiate backup of VM, the snapshot of the underlying CSV needs to be taken. At the same time if another node starts the backup of another VM residing on the same CSV then either the Backup application\Provider should be able to serialize the job else it will fail to take snapshot of the CSV volume as there is already one in progress.

During this process there are chances that the second node might try to own the CSV resulting in complete disconnectivity of CSV volume.

[collinp]

BTW, Veeam Case# is Case # 00428789

Jake,

Microsoft told me the same thing in regards to using the MS CSV Writer instead of Hardware VSS due to the incompatibilities they have seen with Hardware VSS Providers. I am using the Netapp VSS provider with 2012. I think the Netapp VSS provider is compatible because I have never experienced this memory leak issue and Netapp makes their own backup product for Hyper-v called "SnapManager for Hyper-v" which backs up through the cluster stack similar to Veeam and DPM.

My issues seem to happen when using the Microsoft CSV provider. When using the Netapp VSS provider, I find that backups wait for CSV Snapshots to complete before proceeding to try to snapshot the same CSV from another job. Whereas the Microsoft CSV Provider seems to fire off indiscriminately whether there is a snapshot in place or not. But it sounds like multiple snapshots per CSV are acceptable.

For me the most reliable method is to use an offhost proxy and uncheck the "failover to onhost" box.

[jake1110]

I've actually scheduled my backups by CSV. So each job represents one of our CSV's and they more or less balance out. While it's no ideal, this was the easiest option for me.

I understand your situation though as ours behaves the same way in regards to snapping and I agree, it'd be nice to completely disable the MS CSV writer. Due to the ongoing memory leak woes, we've had to switch every job over to the MS CSV writers...hoping not for too long though! My processing speed was around 750MBps backing up 168 VM's on hardware VSS, now it's down to 372Mbps and likely to drop.

Equallogic has it's own auto snapshot manager that integrates as well. Strangely, we have two Server 2012 clusters and it only affects one of them making it more difficult for Dell/Microsoft to figure out. Hoping it's MS's issue though as we've sunk a considerable amount of money into premier support hours!

[collinp]

If you use offhost, does it offload the memory leak to the proxy? What is your disk utilization and latency on the Equallogic during the snapshot? We have SAS and SATA pools and we have had more issues with the SATA. Not sure that this would be correlated to the memory leak. The CSV's seem to be sensitive to latency during the snapshot depending on whether the snapshot technology is copy on first write versus using pointers.

I have installed SCOM agents on the Hyper-v hosts and I see memory paging on the hosts when doing onhost during the snapshot at times.

[rskinner]

I'm having a similar issue with the EqualLogic hardware VSS and Server 2012 R2. Each time a VSS snapshot is taken memory is leaked. I've opened a call with MS who've told me to use the software VSS. This is completely unworkable as many snapshots have timed out and the backups are taking hours longer than with the hardware VSS. I opened a call with Dell who say it's a known issue since Server 2012 and there's no resolution.

Has anyone managed to find a resolution for this issue? At the moment I think I'll have to switch back to hardware VSS and reboot each host on a schedule.

[Gostev]

Hi, Richard. This looks like a finger pointing on DELL side to me. I seriously doubt the issue is in Windows Server 2012 R2, because we have huge amount of customers using Hardware VSS assisted backup with other storage devices with great success. Including DELL's own storage (I have not heard the same complaints from DELL Compellent users). For example, here is a white paper from Hyper-V MVP who has done a really hard core testing of Veeam on his Compellents.

If this is a "known issue" with Windows Server, can DELL perhaps provide a link to the Microsoft support KB article? One thing Microsoft is really good at, is documenting known issues. And this will give me something to go to Microsoft Hyper-V team for clarification.

One thing we normally recommend is to disable ODX - not sure this is related to your issue, but it certainly helps to address a bunch of other issues with Hyper-V backups.

Thanks!

[rskinner]

I asked Dell for the KB, but all I got was "There is no Microsoft Kb that I’m aware off.
I deal with Microsoft Premier support quiet frequently and I’ve discussed off the record this issues with them in the past and it just becomes a table-tennis game, so redirecting you to Microsoft will lead to them redirecting you back to me."

I have already disabled ODX due to some instability issues it was causing. Dell have suggested turning off MPIO for snapshots, so I'll give that a go, but I'm not confident it will make any difference.

Thanks

[Gostev]

I think DELL should approach the issue more seriously, and involve R&D. Microsoft is very open to work on any issues with any vendors. For example, I am observing another Hardware VSS case development at the moment with HP, and it looks like night and day from what you are describing. No finger pointing at all.

Basically, one of our HP 3PAR customers experiences extended disk I/O stun at hardware snapshot creation. After we demonstrated the issue to be outside of our product, the customer talked to his HP rep and they almost immediately made 3PAR R&D engineers available for 3-way webex where they reproduced the issue with low-level debug logs enabled, obtained all technical information they needed from our R&D team, and went back to analyze. Few days later, I hear HP R&D is working on a special test tool on their side to localize the issue in the customer's environment. And honestly, I would expect all other storage vendors to approach critical serious issues this way.

There is NO way Microsoft will not admit and immediately fix a critical issue on their side if the storage vendor is able to demonstrate one, so I guess this simply has not been done by DELL in this case.

[rskinner]

It's been several months, but I have an update. Microsoft have released a patch for the issue I was seeing for Windows Server 2012 http://support2.microsoft.com/kb/3004098/en-us. I've been testing a pre-release version of the patch for Windows Server 2012 R2, which is due to be released as part of the December 2014 rollup on patch Tuesday. The good news is that the patch has resolved the memory leak.

[jake1110]

I'm absolutely stunned this actually came out. We spent well over 80 hours with Microsoft Premier support on this and came up with no answers. Lots of finger pointing with Dell as well. Last year, this crippled our cluster constantly as we studied the issue and watched our VM's crash. We had to change all our backups to use software providers because of it and it slowed down our backups significantly.

This announcement just made my day