Backup Slow, Bottleneck Target

[jochot]

Hey,
I have a Cluster which I am backing up via SAN and a backup Proxy.
Setup:
4x 20TB Drives on a DataCore storage presented to a Hyper-V Cluster attached to one VM (storage is ReFS formatted).
Proxy server which is also the backup repo and backup server (HP DL380 Gen9 with Windows Server 2012R2, 128GB and a HP D6020 connected with 16GBit SAS with around 120TB as repo storage, also formatted in ReFS).
They are all connected via at least 8GBit SAN.
Now I have installed the DataCore HW VSS on both the Cluster nodes and the Proxy Server.
There is no error whatsoever, but the Backup is running with only 160MB/s with every Disk being read at around 40MB/s.
The Bottleneck here is the Target with 95%, Proxy is at 14%, Network at 5%.
I guess Network here is the SAN, but I wonder how "Target" is measured, as Proxy and Target are the same Server in my case. I would guess Proxy is the CPU and throughput of the Proxy and the Target is how fast it is written to the Storage itself?
There are currently no other jobs running onto that target, also no copy jobs, just one job running in a different building (Hyper-V host -> Gateway Server -> Storage via LAN, the local repo is not used anyhow)

how can I find out why my backup is so slow?
The Proxy used is the "Hyper-V Off-Host Backup Proxy" which is installed by default, it has 16 Max Concurrent tasks and no network rules.
The Backup repository also has no Limit to read and write data and 8 concurrent tasks.
Storage Optimization is set to local target, as I didn't know yet if the Full backup files would be larger than 16TB and the Backup mode is Reverse incremental.

[jochot]

So changed from Reverse Incremental to Forever Forward, and it got to around 120MB/s per drive, 500MB/s total.
Still I thought that Reverse Incremental should be faster with ReFS?
As seen in this article: https://www.veeam.com/de/wp-microsoft-r ... arget.html

[jochot]

So the job is running, we're at 750GB of 19,5 TB.
The Speed is down to 86MB/s per drive and 347MB/s in total, Bottleneck is Target at 64%
The CPU is at around 30% usage on the Proxy/Target and the resource monitor shows me:

Code: Select all

Image                       Read (B/sec)      Write (B/sec)    Total (B/sec)
System                      347.283.712      348.485.097      695.768.809
System                      0                  296.533.713      296.533.713
VeeamAgent.exe       1.858.839           11.540.294       13.399.133
VeeamAgent.exe       238.313               7.054.057        7.292.369
VeeamAgent.exe       0                         6.953.146        6.953.146
VeeamAgent.exe       1.811.177            4.645.795        6.456.972
VeeamAgent.exe       7285.975             3.875.561        4.161.536

(Yes, this is the best formatting I could do, sorry :/ )
So I've got 2x System tasks that write a around 300MB/s simultaniously, additionally one of them is reading the same amount and some VeeamAgent.exe's which are also moving data on the drive.
In above mentioned Whitepaper (which unfortunately is in German, just saw it), I see for Forward Incremental without synthetic fulls: "Die gesamte IO-Last wird von einem Lese- und zwei Schreib-Vorgängen auf nur einen Schreib-Vorgang sowie eine ReFS 3.1 MetadatenOperation reduziert"
Which translates to "The whole IO is reduced from one read and two write-processes to one write and one ReFS 3.1 Metadata Operation"
So I don't really get why System is reading at the same speed as it is writing and who the 2nd System Process is and what it is writing.
Also What are all the VeeamAgent.exe Processes doing?

I am looking to improve the speed of that backup as we expect the storage to grow from 4x20TB Drives to even more, possibly over 200TB in total.
If you have any other idea to improve the speed, just comment please

[csydas]

Hi jochot,

Quick question, during the snapshot part of the job, do you see it using the HW provider actually or is it falling over to the Microsoft default providers? If I had a guess, for some reason Veeam isn't using the HW provider and getting those nice HW provider speeds. Likewise, it's staying on the off-host proxy in the job, right? Or is it failing back to the on-host?

That would explain to me anyways; the jobs will transparently fail-over to the Software Provider if I recall correctly with a green informational message. I'm thinking also that the storage might not be using ReFS to the fullest during merges; it's being seen as ReFS by veeam right? During the merge you see [fastclone] in the job notes during synthetic operations?

Basic, I know, but just seems to be the easiest culprits

[jochot]

DataCore HW VSS is used
06.03.2018 14:04:07 :: Creating snapshot DataCore VSS Hardware Provider on Hyper-V Off-Host Backup Proxy (mode: Crash consistent)
06.03.2018 14:05:53 :: Importing snapshot on offhost proxy Hyper-V Off-Host Backup Proxy
I have actually turned off that it uses Software VSS.
There was no Synthetic operation yet, and as far as I know I cannot force a synthetic full?

[jochot]

Hey,

apparently Veeam is not aware of it beeing ReFS:
10.03.2018 00:21:55 :: Creating synthetic full backup (58% done)
It is going on for over 60h now.

how can I make it aware of the fact that it is ReFS formatted?

[csydas]

Jochot,

Did you move the storages over to the ReFS server from an NTFS or were they made fresh on the ReFS server? I believe if I remember from our migration, we had to run an Active Full before Fastclone was possible, something about the block alignment.

Else, sounds like a support issue to figure out why it's not using the Fastclone.

[Mike Resseler]

Jochot,

Other question. That ReFS, is that on a ws2012? ReFS with blockcloning only works on server 2016 (previous version of ReFS doesn't have that blockcloning API)

[jochot]

Yep just talked to Support and Server 2012R2 is the issue here when it comes to ReFS :/