My backups are consistently slow.

[tim5700]

My backups are routinely taking 13+ hours to backup 565GB. I have VI foundation running with two ESX servers. I am backing up to USB 2.0 hard drives. I understand the initial backup taking that long, but I thought when the incrementals kicked in the process would speed up.

[Gostev]

Hello, what types of hosts (ESX/ESXi) and backup mode (VCB/network) do you use?

[tim5700]

Hello, what types of hosts (ESX/ESXi) and backup mode (VCB/network) do you use?

Network.

[ssimakov]

tim5700, what is the average number of changes in VMs between backup runs? Is compression enabled?

[Gostev]

tim5700, in case of network backup the type of host (ESX or ESXi) really matters for incremental backup speed. Can you please clarify what type of hosts are you using?

[tim5700]

I am using ESX 3.5 Foundation. The compression level is optimal. How can I find the number of changes?

[Gostev]

Can you verify that you are using service console agent based backup? Just right-click the server in the Backup management tree and see if it has service console connection enabled and credentials provided. Backup mode is also reflected on Session statistics for each VM - please verify if it says "agent-based" or "agentless"?

[tim5700]

I am using agent-less mode. When I performed my first backup I received a warning suggesting that I use agentless mode. On both ESX servers I have "force agentless mode"

[Ben Milligan]

You will see an improvement of the incremental backup speed if you enabled the Service Console Connection for these ESX hosts instead of forcing agentless mode. You can check the box under the SSH connection tab of the properties of the hosts inside of Veeam Backup.

[tim5700]

OK. So why did the software tell me to force agentless mode?

[Gostev]

I think you have probably mis-read the warning you have received from Veeam Backup. This warning actually recommends that you use service console agent mode for best results. It also states that you can switch to "Force agentless" to stop receiving this warning (otherwise you would be getting this warning after every backup cycle).

Service console agent mode is what you get out of box with the default settings, when network backup mode is set to "Automatic". You have most likely received the warning initially because you had the default "Automatic" mode selected for network backup, but did not specify service console connection settings and credentials in your ESX hosts' settings, so Veeam Backup had to leverage "agentless" mode to ensure that you are still able to perform backup, and this is why it warned you about using non-optimal engine at the end of backup cycle.

Hope this helps.

[Ben Milligan]

This could also be due to necessity as it is possible that the SSH by remote was disabled on your ESX host, and had to revert to "agentless" mode since no SSH connection could be made to the ESX host. You can configure a normal account on the ESX host to be elevated to root privileges without enabling remote SSH by root.

[tim5700]

The check box for SSH was already checked on both. Does that mean I should go ahead and select "force service console agent mode"?

[tim5700]

OK so now that I have confirmed ssh credentials should I force console agent?

[Ben Milligan]

Yes, please check the box "Use Service Console Connection to this server" on the SSH connection tab on the properties of the ESX host. When we are leveraging the Service Console, your backup speeds should increase quite a bit. Please let us know if you have difficulties with the jobs after making these adjustments.

[Gostev]

Yes, you can set this mode for now while you are still setting up the product.

After everything starts working normally, it would be best to reset the mode to default ("Automatic"), so that Veeam Backup is able to failover to agentless mode if needed. This way your backups will not start failing in future if Veeam Backup is not able to leverage service console connection for some reason (due to accidental SSH settings change or other operational issues) - and you will still get your backups done, albeit a little slower.

[tim5700]

OK Thanks.

[tim5700]

I am now using "back up mode: service console agent" per the realtime statistics. The backups are no faster. The backups started at 9:00 PM and are still running as up 6:55 AM only 67% complete. One VMwas 175GB and took 4.5 hours. My 3rd VM is 245 GB and started backing up at 3:10 AM, as of 6:55 AM it still has 3+ hours left.

[Gostev]

That makes it about 10MB/s speed - this is very slow - should be at least 3 times faster even in worst case scenario on 1Gb network. Is your network 1Gb or 100Mb?

Please perform the following testing when all backups finish:
1. Using file management capabilities Veeam Backup provides, copy some VMDK file (a few gigs) of non-running VM from your production ESX storage to Veeam Backup console hard disk. Simply expand ESX and browse to vmfs folder, then copy the file and paste it into "My Computer" local hard drive disk. Note the copy speed.
2. Now, do copy the same copy but select your USB hard drive as destination under "My Computer". Note the copy speed.

Let us know the speed you've observed during the two tests - this should allows to determine a bottleneck.

[tim5700]

No improvement on the backup speeds. For example one VM is 175 GB and it started at 9:00 PM and did not complete until 2:30 AM. Another (245 GB) started at 3:30 AM and did not complete until 8:45 AM. The backup mode on all the backups is service console agent.

[tim5700]

Yes, please check the box "Use Service Console Connection to this server" on the SSH connection tab on the properties of the ESX host. When we are leveraging the Service Console, your backup speeds should increase quite a bit. Please let us know if you have difficulties with the jobs after making these adjustments.

Sorry disregard that message for the time being. My browser cached, it so I didn't see my post or yours. I will try your steps in a moment.

[tim5700]

Copied 2 GB from storage to local disk on Veeam server = 9MB/s
Copied 2 GB from storage to USB drive on Veeam server = 9MB/s

[Gostev]

This is clearly 100Mb LAN metrics... please check your NICs on computer with Veeam Backup installed, on source ESX host, and all ports of all switches/routers in between. We are very often seeing a situation (even had this internally) when mis-behabing 1Gb NIC or router auto-negotiates for 100Mb connection for no reason, and slows done everything.

[tim5700]

Everything is set to (and displays) gigabit. The ESX servers and the Veeam/vCenter server.

[Gostev]

Then the next step would be to test actual network throughtput between source ESX and some Linux-based server, as well as ESX storage speed. If you don't know how to do this, it would be best that you contact our support and they will provide you with instructions. By the way - I should have asked earlier, but what storage your ESX is using?

[tim5700]

I did a physical spot check of the switch. One of my ESX servers' ethernet cable was moved to another switch, so that traffic was not on the same switch as Veeam, and one of the switchports had moved to 100 Mbps. I hard coded it to 1000 Mbps.

I will let you know how it goes.

[tim5700]

That got it. Backups are now completing between 4 - 5 hours.

[Gostev]

OK, thanks for the update