Veeam B&R 8.0 : Backups of HyperV CSV slowed down to a crawl

[doalwa]

Hi,

we're running Veeam Backup & Replication 8.0 for HyperV at one of our customers sites backing up a two-node HyperV 2008 R2 cluster.

We're talking 10 virtual machines in total distributed across two Dell PowerEdge R510 servers.

The servers themselves are connected to two Dell PowerConnect Gigabit switches which in turn are connected to a Dell PowerVault MD3200i SAN using MPIO'd Gigabit ethernet connections.

A couple of weeks ago, the customer suffered a prolonged power outage which led to one of the older APC UPSs to hit the bucket. The second APC USV was able to keep the whole environment running, fortunately.

Problem is, ever since this incident we weren't able to successfully finish a single backup using Veeam V8.

We backup all 10 VMs to a physical Dell PowerEdge 2950 using local SATA disks in a RAID10 array. The backups are incremental.

Afterwards, those backup files are then exported to a locally-attached LTO5 SAS-tape drive.

We were using this solution to backup our HyperV environment since January of this year without much problems. There was the occasional hiccup, of course, but all in all everything was working fine.

Since the UPS incident, however, backups have slowed down to a crawl.

Previously we were seeing backups speeds of about 70 MB/s for our daily incremental backup jobs, which might not be earth-shattering fast but was more than enough to allow each backup run to be finished over night, including the backup to tape afterwards.

Now we're only seeing transfer rates of between 5 to 10 MB/s, which is nowhere near enough to finish the backups over night.

The other undesired side effect is that our ClusterSharedVolumes are switched to Redirected access for the duration of the backup job, which impacts overall performance severely.

We're having huge problems with our print and terminal servers, respectively and are forced to simply quit the backup, otherwise it's almost impossible to get any work done.

I've since involved Veeam support and was able to get hold of a helpful guy which first suggested installing several MS hot fixes on our Cluster nodes. None of the hot fixes were needed, either because they were already installed or were simply not applicable to our scenario.

I've since uploaded our job logs for the last month. First thing he pointed our was that backup encryption was activated for our backup job.

Since were using a subnet of 131.1.0.0/16 for our internal network, Veeam seems to be think we're on a public network which lead to the activation of this encryption feature.

I was able to deactivate the encryption by setting the appropriately called"DisablePublicIPTrafficEncryption" registry key at

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication

and

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Veeam

Didn't have any impact on overall backup performance, unfortunately.

Haven't heard anything backup from Veeam since two days, so I figured I'd try my luck here.

One solution I've read multiple times about concerned the driver responsible for generating the ChangedBlockTracking deltas located at %windir%\system32\drivers\veeamfct.sys and to restart said driver using sc stop veeamctf && sc start veeamctf .

Tried it, but didn't seem to have any impact, neither.

Is there anything else I could try? We haven't had a successful backup for the better part of 4 week now and I'm starting to get a little nervous here.

Thanks in advance,

Dominik

[Dima P.]

Hello Dominik,

Could you please provide us with the case ID so we could check the investigation results on the support’s end? Also I wonder what are the results of the bottleneck analysis for the affected jobs. Thanks.