Slow Perfomance backing up from storage snapshot

[jgross]

Hi,

we have a VMware Cluster connected via NFS to a Netapp Storage. The Veeam server is running in a VM and using a CIFS-connected Synology as Backup Repository. We are using Storage Snapshots to back up most of the VMs (only a few test server are not located on the Netapp but on a Synology volume). Network is 1 GB.

The incremental backup jobs is running with a processing rate of 7 MB/s. Bottleneck is Source (56%).

When looking at the statistics of some VM, I find:
4/11/2015 2:33:56 PM :: Festplatte 2 (450.0 GB) 6.3 GB read at 1 MB/s [CBT]
4/11/2015 2:33:57 PM :: Festplatte 1 (80.0 GB) 14.3 GB read at 3 MB/s [CBT]
4/11/2015 4:09:45 PM :: Busy: Source 90% > Proxy 27% > Network 9% > Target 36%
4/11/2015 4:09:45 PM :: Primary bottleneck: Source

On the other hand, when I am copying some data from an NFS share on the Netapp to a Linux VM (through the same network the Veeam server is using) I am seeing transfer rates above 100MB/s.

I don't understand Source bottleneck really means in case of a storage snapshot, but more then 5h for reading 100 GB and transferring 35 GB seems far too long to me. Any recommendations what to check?

Regards,
Johannes

[jgross]

Case ID is #00876203.

[foggy]

Thanks for opening a case, please continue looking into this with our technical team. Bottleneck source means that reading the data from the source storage slows down the entire backup process.

[jgross]

Hi Alexander,

thanks for replying. Unfortunately, support didn't find any issues. We have to wait now for patch 2 to see if it brings any improvement.

I am wondering if there are any comparative figures on how long an incremental backup should take if there are 3TB to process (200 GB read, 30 GB transferred).

Regards,
Johannes

[foggy]

Typical performance should definitely be greater than you're seeing. As far as I know, there should not be any fixes regarding that in the upcoming update, so I recommend to escalate the case for further investigation.

[berrosch]

Hi Alexander, any progress in your case? I've experienced similar behavior (maximum 25-30MB/s for one VM/VMDK, maximum 100MB/s in parallel) with virtual veeam proxy with NFS backup from NetApp SnapShot. HotAdd flows 600-700MB/s
It is FlexPod with SSD accelerated FAS8040

Here is the statement from our support:

Unfortunately, this is behavior by design and it's related to the fact that a virtual proxy cannot have direct access to the storage snapshot itself. This means that it has to access it through the ESXi host and use VDDK commands for this(NBD mode) which always has much lower performance than reading directly from the storage snapshot itself.

We'll deploy physical proxy servers, the customer uses both NFS and FC/VMFS protocols, let you know the physical NFS proxy performance.
Case # 00966928
Martin

[vee789]

Is there a way to see in the Veeam backup logs if NBD or which method was used when using NetApp shapshots as source?
Setup: NFS datastore, ESXi 5.5, cDOT 8.3.1, FAS2552, Veeam v8 Enterprise Plus (physical proxy)

[foggy]

You can see the transport mode selected for processing if you click the VM in the job statistics window and look for the "Using backup proxy..." server record. It should be one of the following: san, hotadd, nbd, or storage snapshot.

That said, v9 should introduce significant improvements for backup from storage snapshots over NFS.

[vee789]

We see "storage snapshot" as transport mode but the backup takes longer without Storage Integration. That seems quite strange...
Am I right that the Veeam proxy uses a Veeam proprietary NFS client to access the snapshot directories on the NetApp directly to copy the VM files?

[foggy]

Am I right that the Veeam proxy uses a Veeam proprietary NFS client to access the snapshot directories on the NetApp directly to copy the VM files?

Yes, this is correct.

[zummy430]

Old thread- but was performance improved with physical proxy and backups from NetApp storage snapshot compared to that of virtual proxy?

[veremin]

It certainly should have, as Direct SAN proxy mode typically is faster than Hot Add one. Thanks!

[Adam.Bergh]

Hi,

If your question is whether a physical proxy is higher performing than a virtual proxy when comparing direct storage backup performance, the answer will always be "it depends". What you really want to think about are network connectivity speeds to the front and back end (repository) storage, how many CPU cores each proxy has and how much memory is in the proxy. A virtual proxy can absolutely saturate a 10GB link during backup given enough cores and memory for example. A physical proxy however is required for fiber channel connectivity to front end storage, as this is not available on a virtual proxy. Physical proxy tend to be higher performing in general due to dedicated resources (CPU/Mem/Network) vs virtual proxy when tend to be on shared infrastructure.

Regards,
Adam Bergh