Backup Incredible Slow - Processing at about 200kb/s

[beistrich]

Hallo!

I set up Veeam Backup&Replication as usual.
- Veeam is installed on a virtual server (and is used as Proxy)
- Backup repository is a NAS
- VSS, indexing and cbt enabled

Never had any issues before. But now i have enormous performance problems - running backups with about 200kb/s. It is a new environment with fresh Veeam Installation. (Its a single vSphere-host deployment)

I already tried disabling different Options (vss, indexing cbt...) - nothing helped.

Has anybody an idea what i can try?

[Vitaliy S.]

Hello Christoph,

According to the screenshot above, source might be the problem, but I would still ask for full bottleneck stats for this job. Also what kind of VM is it? Can you please try to backup another VM and confirm that you have the same performance issues?

Thank you!

[veremin]

As to CBT, disabling it won't increase the processing speed. Instead, it will increase the job time, as without CBT the whole VM image needs to be read by VB&R in order to find changes.

Thanks.

[foggy]

In addition to full bottleneck stats,I also wonder what transport mode is being utilized to retrieve VM data from the source? (You can select the particular VM in the list to the left and see the transport mode tag next to the proxy server name.)

[beistrich]

Yes, i see that the bottleneck is the source - But i don't know why? I have not recognized any performance issues of other applications...

[Vitaliy S.]

Can you please clarify other information, so we could assist you further?

[beistrich]

THe following shows up for backup Proxy:

Using backup proxy VMware Backup Proxy for disk Hard disk 1 [hotadd]

VMware Backup Proxy is "this Server" with Transport mode set to "Automatic"

[foggy]

What if you force the proxy to use network mode?

[veremin]

What about CBT information? Do you see the corresponding mark ([CBT]) nearby virtual disk name? Also, as mentioned above, can you try to backup different VM and see whether the speed rates are the same? This should give us a better understanding of the scope of the issue.

Thanks.

[beistrich]

Hi!

I have now force "Network" on Transport mode

It doubled the "speed" to 400 kb/s

[beistrich]

I also want to mention that it is VM Independent. Also other vms backup at same slow speed.
Could it be the disk array? I doubt that, becaus overall server performance is very good...

[foggy]

What kind of storage is it? Bottleneck source means that your production VM storage cannot provide data any faster.

[beistrich]

Its a ESXi 5.5 host with local storage (6 disks with RAID 5)
Write Cache enabled on RAID Adapter

[Vitaliy S.]

What download speed do you have if you try to download any big file from the datastore via vSphere Client -> Datastore Browser?

[beistrich]

WOW, ok, i think i have found the problem

Copying TO the Datastore ist quick, but also not very fast.
Copying FROM the Datastore ist incredible slow! (20 minutes for a 230 MB file)

BUT copying via SMB share at full Gigabit speed!

Anyone knows about issues about using local storage with ESXi 5.5?

This is very strange, because i have already deployed various servers with local storage, not having such issues!

[sbbots]

I am using a similar setup at two remote sites; Single ESXi host, Veeam installed on a VM, backup repository on a Synology NAS (CIFS share).

I noticed slow speeds below 1Mb when going through a Cisco POE Gigabit switch. I attached the NAS directly to an extra NIC on the host server, then passed that NIC to the backup VM using Directpath. All speed issues went away immediately and backup transfer speed now completely saturates the GbE connection.

[beistrich]

Yes, would be an option (which i will try)
But i don't think thats really the problem, because the copying from the datastore i run from a VM ON the ESXi host. Does this traffic passes the physical switch?

[sbbots]

I also run Veeam on a VM on the host, so your setup is similar. Once you connect the NAS directly to the host the traffic bypasses the switch.

[Vitaliy S.]

Christoph, just to isolate the issue, can you please try to perform a backup of any test VM to the local storage (virtual disk of your backup proxy). What performance do you have in this case?

[beistrich]

Hi!
Now i tried a backup to the backup-Proxy. Also slow at 300kbit/s...

I found an VMware article adressing such Problem:
http://kb.vmware.com/selfservice/micros ... Id=1006602
I have already configured this...

[beistrich]

I now did a test with "esxtop"

I see that the value of CMDS/s and READ/WRITES is significantly different!

This is the result:
Write to Datastore
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/
naa.600605b006e979901a583d9b1ad8fb71 - 128 - 0 0 0 0.00 108.02 1.00 107.01 0.03 16.27 0.16 0


READ from Datastore
DEVICE PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD LOAD CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/
naa.600605b006e979901a583d9b1ad8fb71 - 128 - 0 0 0 0.00 11.83 1.91 9.92 0.24 0.25 0.21 0

[Vitaliy S.]

Now i tried a backup to the backup-Proxy. Also slow at 300kbit/s...

Was it a hotadd or a network processing mode? Do you have any other ESXi host in the same environment to compare job performance rates?

[beistrich]

Hi!
I want to give an update to this...
On the weekend i tried different settings of the raid controller (write back, write back with bbu, write throuch, no read ahead...)
Nothing worked...
I really don't know what to try next, anybody has further ideas?

[veremin]

Then, you can open a ticket with our support team and let them confirm your environment, as some of the environmental changes that decrease performance might have slipped out of your attention. Thanks.

[Vitaliy S.]

If it is a fresh environment, then using a different RAID level (for example, 10) might also help.

[beistrich]

I know its a long time.

but the problem was the Switch/ ESXi Nic configuration! It was set to autodetect, but had 100Mbit non-duplex! - This means VERY slow...
We manually set the link speed and everything worked fine!

[foggy]

Christoph, thanks for coming back after a long time and finally updating the thread with the resolution, much appreciated!