New user, need help optimizing backups (very slow backup)

[buck]

I have a small vSphere Essentials install (3 hosts, 15 VM's). I use shared storage for most VM's (Buffalo 2TB NAS device), with some local storage as well. I recently bought Veeam Essentials, and have installed another Buffalo 2TB NAS device to store Veeam backups only. The Veeam software is installed on a Windows Server 2008 VM (which I'm not currently backing up). The back-up NAS device is accessed as a network drive through Windows in order to allow the Veeam backup job to 'see' the volume.

I started backing up last week with Veeam, but I don't think I've set it up right, as the performance is pretty bad. I'm backing up just 6 VM's nightly, but the backup job is taking more than 12 hours to complete. 3 of the VM's are Windows XP boxes, 2 are Windows Server 2003 VM's, and one is a Windows 2008 Server with SQL 2008 installed. The SQL server is the largest VM I have, at about 280 GB used space. The rest are 80 GB or smaller (as low as 8GB for a couple).

My Backup Job is set to "Virtual Appliance" processing mode with "Reversed Incremental" backup mode selected. Additionally, I've enabled "Application-aware image processing" and "guest file system indexing." Otherwise, I stuck with the defaults everywhere.

I thought after a few backup jobs, the time needed to complete the task would drop, as I would only be backing up VM changes, but so far, they're still taking way too long.

Have I set something up above wrong, or is there something else I can try to improve the performance?

Thanks in advance for any advice!

[Gostev]

You need to identify your bottleneck first, before we can suggest on how to improve the performance. Hard to guess remotely what is the bottleneck in your case...

[buck]

And how can I do that? Not sure where to look first...

I'm new at this, sort of. I've been using Backup Exec to backup physical & virtual machines for a while now, but have decided to switch to Veeam for all VM's, and so it's a bit new to me in that respect. If it worked as expected out of the box, I think I'd be fine, but troubleshooting is not my strong suit.

[Gostev]

Basically, the bottleneck can either be source data retrieval speed, your backup server performance, or your backup target performance.

Usually, the performance troubleshooting pattern is as follows.

1. First thing to look at is CPU load on backup server while your jobs are running. If it is capped at 100%, then you have already found your bottleneck

2. If the CPU load is under 100%, then the next common suspect is backup target. Try creating 2 test jobs backing up the same server, one backing up to your main target, and another one backing up to a local disk on backup server. Perform daily backups with both jobs (not concurrently), and compare the run times.

3. If there are no differences in performance in the above test, then the next suspect is source data retrieval speed. Create a test job with a single shutdown VM, and disable changed block tracking in the advanced job settings. After the first backup is created, start the backup job again for the incremental run. Since there will be no changes in the image (test VM is still shutdown), nothing will be written to target (pretty much). However, because CBT is disabled, the backup job would have to read and hash the whole source image to determine that there are no changed blocks. So, just divide VM size (only used space in case of thin disk) by the time your incremental run took to get your max source data retrieval speed. The source data access throughput may be quite limited when there are some issues in the environment, and this can be your bottleneck.

Of course, make sure you do all the testing in off-hours and before starting your main backups. Otherwise production or backup storage may be busy with something else, and your test results will not pinpoint the issue correctly.

Thanks!

[buck]

Great, I will start with those suggestions, and will reply when I have some results to share!

[buck]

CPU load is the first culprit. The VM was set with only 1 vCPU. I changed that to 4, and last night's backup was completed in 4 hours!

I will continue to monitor, and will try your other suggestions if it starts to stumble again.

Thanks!

[Gostev]

Sounds good, thanks.

By the way, did not our product setup warn you that 1 vCPU is not enough?
Or you was not the one setting up the product on the server?

[buck]

I'm the one doing it all, and no, I never got any warning when installing it.

[Gostev]

Hmm. Are you running Veeam v5, or previous versions? If v5, then out of curiosity, could you please try starting setup program on another 1 vCPU VM, and see if you get this warning? You should get a warning message box right on the first step of the setup wizard. And no need to actually install the product, just cancel out immediately.

[buck]

Using the latest, 5.0.2 or whatever it is.

Just tried it on another VM with 1 vCPU, and it threw up the warning that I needed a minimum of 2. I must have missed that when I did my original install. Sorry about that!

[Gostev]

Thanks for the confirmation. Just wanted to double-check.

[fahdp]

[merged]

HI,

Im trying to evaluate VBR for our environment but at the moment its not looking so good. I created a backup job to do 167GB VM and am getting speeds of 8mb/s ( env is a 10GB network ). 8mb/s!!! That is awful. I have VBR installed on a physical server HP BL685c G5 with 32Gb RAM ( its the only spare server I could find for now ). I have on this server LUNS presented via the Qlogic cards so in essence its like a CIFS. The back end storage is an EMC Celera.

The backup job is set as VA, compression optimal, optimize for local target. 1 restore point.

Anyone with a similar setup and can share what speeds they get? Any ideas of why its slow?

[Gostev]

Hi, typical performance troubleshooting steps are outlined above. Please note that you cannot really use VA (virtual appliance) mode when your backup server is physical. What happens is that it fails over to network mode with your job setup. However, with 10GB Ethernet network backup should fly. Thanks.