Extremely slow backup.

[ivanildogalvao]

I configured a new server with V&B 10. Windows Server 2019, Dell R540, has 16GB RAM, two gigabit cards (nic teaming).
The virtual environment is VMware vSphere 6.0, with Dell/EMC, iSCSI storage.
I created a Linux virtual machine CentOS 8 to serve as a Backup Proxy, it has 2VCPU and 6GB of VRAM, it is hosted on the same Vmware cluster.
Well, when performing a test backup with a small 20GB VM, it takes more than 1 hour, the transfer rate is meager, 3Mb/s.
Well, best practices say that using a separate proxy (hot-add) is the best way to get a quick backup, but in practice this is not the case.
I opened a support ticket at Veeam, the Support Analyst did some tests and believes that the problem is in the storage, because when we performed a backup of a VM stored on the internal disk of one of the ESXi 6 hosts, it was fast, reaching 211MB/s.
But then I ask, what problem or configuration may be occurring in the SAN/iSCSI storage, which is making the backup so slow?

Any idea ?

[HannesK]

Hello,
please post the case number that we can check... otherwise the post will be deleted. veeam-backup-replication-f2/rules-of-po ... -t755.html

The conclusion the support engineer makes sense to me. I mean yes, your proxy is pretty small from a configuration perspective. But if the same proxy does 211MB/s from a local disk, then the storage (connection) is obviously the reason.

But then I ask, what problem or configuration may be occurring in the SAN/iSCSI storage, which is making the backup so slow?

this question should be asked to Dell / EMC (sure, we can guess anything from networking to controller cache settings, but not sure how useful that is) One quick thing you can test is to copy some large files from thestorage to the backup server.

Best regards,
Hannes

[ivanildogalvao]

Case #04389215

If I use the Veeam server's own proxy, the transfer rate goes up to 42Mb/s, even so it is very low.

Code: Select all

9/29/2020 9:19:50 AM :: Queued for processing at 9/29/2020 9:19:50 AM
9/29/2020 9:19:50 AM :: Required backup infrastructure resources have been assigned
9/29/2020 9:19:55 AM :: VM processing started at 9/29/2020 9:19:55 AM
9/29/2020 9:19:55 AM :: VM size: 20 GB
9/29/2020 9:19:58 AM :: Getting VM info from vSphere
9/29/2020 9:20:06 AM :: Creating VM snapshot
9/29/2020 9:20:13 AM :: Saving [DS_VDP] HCDNS/HCDNS.vmx
9/29/2020 9:20:14 AM :: Saving [DS_VDP] HCDNS/HCDNS.nvram
9/29/2020 9:20:14 AM :: Using backup proxy VMware Backup Proxy for disk Hard disk 1 [nbd]
9/29/2020 9:20:15 AM :: Hard disk 1 (20 GB) 20 GB read at 40 MB/s
9/29/2020 9:28:50 AM :: Removing VM snapshot
9/29/2020 9:28:54 AM :: Finalizing
9/29/2020 9:29:00 AM :: Busy: Source 99% > Proxy 2% > Network 0% > Target 0%
9/29/2020 9:29:00 AM :: Primary bottleneck: Source
9/29/2020 9:29:00 AM :: Network traffic verification detected no corrupted blocks
9/29/2020 9:29:00 AM :: Processing finished at 9/29/2020 9:29:00 AM

[PetrM]

Hello,

what problem or configuration may be occurring in the SAN/iSCSI storage, which is making the backup so slow?

You may take a look at ESXTOP statistics as long as we're talking about slow read in HotAdd mode. I believe it's worth trying HotAdd with Windows virtual proxy and to compare the results of tests.

Regarding Veeam server proxy:

Code: Select all

9/29/2020 9:20:14 AM :: Using backup proxy VMware Backup Proxy for disk Hard disk 1 [nbd]
….
9/29/2020 9:29:00 AM :: Busy: Source 99% > Proxy 2% > Network 0% > Target 0%

Based on this statistics we can say that Source Data Mover spends most time of its activity on reading data because of slow read in NBD mode. The main reasons are slow data transfer from ESXi to the proxy over network link or the source storage is not fast enough.

Thanks!

[ivanildogalvao]

Yes, I have already done tests using a Linux Proxy, which is supported on Veeam 10.
As I said before, using the Linux Proxy (hot-add) the transfer rate is even worse, I will paste the statistics here.

Code: Select all

Queued for processing at 9/28/2020 11:43:12 PM 	
Required backup infrastructure resources have been assigned 	
VM processing started at 9/28/2020 11:43:17 PM 	
VM size: 20 GB 	
Getting VM info from vSphere 	00:03
Creating VM snapshot 	00:02
Saving [DS_VDP] HCDNS/HCDNS.vmx 	00:00
Saving [DS_VDP] HCDNS/HCDNS.nvram 	00:00
Using backup proxy veeamproxy.domain.local for disk Hard disk 1 [hotadd] 	00:05
Hard disk 1 (20 GB) 20 GB read at 5 MB/s	01:12:39
Removing VM snapshot 	00:02
Finalizing 	00:01
Busy: Source 99% > Proxy 3% > Network 0% > Target 0% 	
Primary bottleneck: Source 	
Network traffic verification detected no corrupted blocks 	
Processing finished at 9/29/2020 12:56:46 AM

[PetrM]

The idea was to try Windows proxy working in HotAdd mode and to compare results with Linux VM in order to exclude an issue at the level of Linux VM itself from the list of potential bottlenecks.

Thanks!

[ivanildogalvao]

My friend, you got it right.
I created a VM with Windows Server 2016 to serve as a Proxy, with 80GB of disk, 4VCPU and 8GB VRAM, so I installed the Veeam proxy agents on it, then performed two backups, the transfer rate varied between 178Mb/s to 220Mb/s , it is not much, but it has improved considerably.
Therefore, it is not a good idea to use a Linux VM as a proxy.

Thanks for the tip !

[PetrM]

Glad to hear that it helped! Windows may give better performance due to the Advanced Data Fetcher feature that is not yet available for Linux proxies. However, the following Linux distros showed solid performance results during our internal tests: Ubuntu 19.x, Debian 10, openSUSE 15.1.

Thanks!

[Gostev]

Well, even if Advanced Data Fetcher may give a noticeable increase depending on the storage, the difference of almost 70x times is definitely due to something else. Perhaps some VM hardware or Linux OS settings are off. We'd appreciate it if you ask your support engineer to escalate your case to higher support tiers for further investigation. Thanks!

[ivanildogalvao]

Hello friends, I created a new VM with Ubuntu to use as a Veeam Proxy.
The performance is actually much better than using the VM with CentOS, the backup copy had a transfer rate of around 110Mb/s.

[basildane]

Hey Ivanildo,
Can you share how you configured your Ubuntu vm? Disk, cpu, ram, etc?
I'm having the same problem. 11 Mb/s transfer. I have a Windows 2016 Proxy on the same esxi, it's horrid. At this rate I cannot perform backups.

[PetrM]

Hi Phil,

Where is the "bottleneck" according to job statistics and which transport mode do you use? Also, we always recommend to open support requests for any kind of technical issues, please don't forget to paste case ID over here.

Thanks!

[soncscy]

Hey all,

Just my 2 cents, Debian/Ubuntu seem to be the way to go here. In our lab testing, CentOS and RHEL just didn't keep up like Debian/Ubuntu did. For most of my clients, we're sizing at 4 cores 16 GB, but I'm pretty sure we could increase the core count without changing memory. The VMs themselves are vanilla, fresh from the respective official mirrors.

Usually we've been seeing around 150+ MB/s

[Gostev]

You're not alone Harvey... here's my almost 1 year old post from the internal forums:

Team,

Here are some curious backup proxy stress testing results with the current v10 build, using different job and compression settings.

WS2019:
Most consistent performance overall regardless of proxy load.

Ubuntu 19, Debian 10:
By far top performance specifically under heavy load, worse than WS2019 under lighter loads.

openSUSE 15.1:
Very consistent and only slight worse performance than WS2019.

CentOS8:
Most inconsistent performance (all over the place depending on load).

CentOS7:
Most terrible performance overall, not recommended.

Thanks!

[basildane]

Bottleneck is SOURCE or NETWORK, depending on which mode i tried. Nothing got usable results.
Case #04431163

My source is Windows 2019 ReFS volume, during the backup all the servers were idle, no load, no interference from anything else.

Hi Phil,

Where is the "bottleneck" according to job statistics and which transport mode do you use? Also, we always recommend to open support requests for any kind of technical issues, please don't forget to paste case ID over here.

Thanks!

[PetrM]

Hi Phil,

Thanks for reply! I'd suppose that "Network" is the bottleneck when you run backup job in HotAdd mode and is shifted to "Source" in NBD mode? If my assumption was correct, then backup would be slowed down due to some network performance issues in a vSphere environment. I'd start by checking suggestions from this KB article however I believe that our support engineers will be able to determine the root cause after debug logs examination.

Thanks!

[basildane]

Thanks Petr, I'm reading that.
I do have another question. is it possible to get the proxy to use my higher performance network rather than the management network?

[PetrM]

It depends on transport mode:
1) In HotAdd mode source Data Mover reads data from the directly attached disk and it makes sense to try the preferred network to route traffic between source (proxy) and target (repository) Data Movers over higher performance network.

2) If I remember correctly, in NBD mode the choice of network is transparent for Veeam and depends on how ESXi hostnames are resolved on the proxy server.

Thanks!

[Gostev]

My source is Windows 2019 ReFS volume

How can this be? The source is the VMFS volume where protected VMs reside.

[basildane]

yeah, i mean when doing network backup directly from the vm.

[Gostev]

You mean your VM is Windows 2019 with a ReFS volume? This does not matter at all, because backup is image-based, so it's done by pulling blocks of the VMDK file hosted on a VMFS volume. Transport mode (NBD or hot add) is what affects backup performance the most, and then primary/backup storage performance.