Veeam B&R 6 slow on ESXi 4.1

[Heiko]

Good Day

This forum is to find out how the community experience is when running Veeam B&R version 6 Patch3 to protect a vSphere 4.1 environment.
Veeam support ref ID#5173604

Configuration:
iSCSI SAN
ESXi 4.1 U2
Veeam B&R server is Virtual

Problem: After upgrading to Veeam 6 backup performance are very very slow. 1MB/s to local attached storage. (Attached are VMDK files which stores the backup)

What has been done:
Upgrade to from Veeam B&R 5.0.2 to Veeam B&R 6.0 Patch 3.
Uninstalled and fresh installed Veeam B&R 6.0 Patch3 and same performance problems. We even created proxy servers and ran a single 50GB Virtual Machine job.
After uninstalling version 6 and install version 5 all is back to normal.

Explanation on the problem: If it works fine in v5 probably it is issue with VDDK version. VDDK is VMware libraries that allow is get access to the VMFS datastore. We are using VDDK 1.2.1 in v5. In v6 we are using VDDK 5.0 I suspect VDDK 5.0 may have some performance issue with reading data from older VMFS version.

Any ideas suggestions?

Thank you

[Gostev]

VDDK 5.0 has no known performance issues. What are the 4 numbers from bottleneck stats in v6? Most likely, your v5 jobs are working in the different processing mode than with v5, that's the only thing I can think of...

[Heiko]

Says the source.
Did take a snapshot of the Veeam server before uninstalling all. Easy now to revert back and test.
FYI - tried Virtual mode etc and no differance

[Heiko]

Hi

For Feedback purposes.
All backups are running 100% fine on Veeam 5. We even tried all various settings in Veeam 6 without any success.
Veeam support is investigating the problem further. The call has been escalated to a higher tier.

[paps40]

Heiko,
Did you ever find a solution to get version 6 to work correctly? I am also experiencing performance issues after I upgraded ESX from 4.0 to 4.1 and upgraded to veeam version 6 patch 3.

Just Curious.

Thanks.

[foggy]

Peter, as far as I can tell from the Heiko's support case, it's still under investigation (by VMware at this time).

What are your bottleneck stats numbers and what backup mode is actually used by your jobs?

[tomuk5]

Hi

im also having the same issues now, just started a week ago out of the blue.

hosts running 4.1, separate physical veeam box with local storage running veeam 6 patch 3

the bottleneck stats are: Load: Source 99% > Proxy 9% > Network 0% > Target 0%

the backup mode is network.

[paps40]

Alexander,
Sorry, I had forgot about this post. My issues have been resolved. My issues were related to a Dual Port Intel NIC and a failing switch. I'm not quite sure why but when using a dual port nic when you set one ethernet port to jumbo frames and the other one to not use jumbo frames your backup speed decreases. Once I used a different nic (the Dell Server Motherboard NIC) my speed increased. Once I replaced the failing switch it increased even further.

We are using backup network mode and backing up to our old SAN MD3000i. Our bottleneck stats are: Source 93% > Proxy 93% > Network 28% > Target 18%. On full vm backups over the weekend we are getting speeds of around 14 MB/s. Our reverse incremental speeds during the week are anywhere between 70 MB/s and 150 MB/s. We are also running veeam 6 patch 3. We we do full backups to local disk 15K SAS on the Physical Veeam Server our Full VM backups speeds jump to 48 MB/s.

[foggy]

Peter, glad to hear that your issues are now resolved. Thanks for coming back to update the topic!

[Heiko]

Hi

Apology for the late replay.
The problem has been resolved after it has been escalated to VMware.
Cause: We saw in the logs each 1-3ms connection loss to the iSCSI target.
Resolution: Upgraded to the latest SAN software revision and all problems have been resolved. We do get about 40-50MB/s iSCSI backups speed in production times (Virtual Appliance mode) to an CIFS target.

Need to mention how good VMware is. Even with connection drops we did not experience any problems during production times. Well done VMware.

Thank you for all your support. If there are any queries let me know.

Heiko

[OWTech]

Alexander, fyi, I don't know if it's switch specific, but for my procurve if you set jumbo frames on a port, all the devices and ports in that vlan must also be set to jumbo frames.

[Heiko]

OWTech ur 100% correct but if a port is set to 1500MTU it will be able to communicate with a port that set to 9000MTU. But thats a reason why he might experience performance problems.
My experience if you do not use Jumbo frames from start to the end you will have performance problems and its always a good idea to make sure all hardware is updated to the latest firmware.

[dellock6]

You always have to have jumbo frames enable along the whole chain, from start to end. If you do not do so, you will have for sure performance problems, more than not using jumbo at all, since for every packet there is always a check to see if jumbo can be used, so before using 1500mtu there are some re-transmits.