Extremely slow backup

[ADM]

Hi guys,
Having an issue with Veeam 5.0.1 back up speeds, having searched through the forum I see some of you have had similar issues at some point or another.

Brief system background...

Single Host running VMWare vSphere 4.1
VMWare running from Internal USB
Fujitsu TX300 S4
RAID 5 Datastore (4 x 300 3.5 inch SAS 15k drives)
8 Guest VMs (mixture of Server 2003 / Server 2008 / Win 7)
vCenter running as VM Guest on Server 2008 x64 SP2
Veeam Server running as VM Guest on Server 2008 x64 SP2 (4 vCPU, 2GB RAM)
USB 2.0 Tandberg RDX drive

A Full backup of the VM's (around 750GB) takes 18 and a half hours backing up at 11MB/sec to the RDX which is mounted direct to the Veeam Server using USB passthrough.
The backup job is using Virtual Appliance Mode and Reverse Incremental but otherwise default settings.
The job also fails to backup the vCenter Server with the following error: Freezing guest operating system VSSControl: Failed to freeze guest, wait timeout

Might be worth mentioning that when the Veeam server backed itself up, throughput increased to 50MB/sec but otherwise never rose above 14MB/sec on the other guests.

Surely the backup should be capable of much more than 11MB/sec ?
Has anyone had similar issues and been able to solve them?

Chris

[jbeunel]

Hi

We use Veeam since 2008 and the maximum speed i saw was 15MB/s.
By moment you can see 30MB or 50MB (using network mode, SAN mode is faster) but it's a processing speed not a real network speed.
If you look the network bandwitch in the task manager or with a snifer the speed is 4 or 5 MB/s.
In my case a backup of my lotus domino (full backup 300GB) with a daily incremental of 3GB run during 9hours.
It's faster to use vmware converter to convert 250GB (it take 40minutes) than a veeam backup of 3GB, find the error....
I saw in the forum that the vmware api has a limited speed...
And someone said use san mode is really faster...
If you find something i'm interrested.
ps:10MB/ with gigabit lan and CPU used a 15% when the job is running with network mode or appliance...
Bye

[Vitaliy S.]

Chris,

I believe the main bottleneck of your performance issues is USB Passthrough, we've heard the same feedback from our existing customers regarding poor performance of USB passthrough, please check out this topic for more details: vSphere 4.1 and USB Passthrough

As regards vCenter Server backup failure, please review existing topic which describes a proper procedure to be used for vCenter Server backup jobs: Can I backup the vCenter server?

Finally, If I were you would simply use local drives or any networks shares as a destination target. By the way, what is the CPU load on the backup server when you run your jobs?

[Gostev]

the main bottleneck of your performance issues is USB Passthrough

Ditto. Fast backup requires fast backup target and vSphere 4.1 USB passthrough is slow as hell.

10MB/ with gigabit lan and CPU used a 15% when the job is running with network mode or appliance

Same here, if source link and CPU are barely loaded, then the only thing you can point finger at is your backup target, really.

[ADM]

CPU load is around 50-55% while a backup is in progress, although the physical memory usage is quite high. Is 2GB RAM enough??

I think you may be right with the Write Speed theory. I have run the bottleneck tests that Tech Support suggested and the Incremental was significantly faster.
Backing up a Windows 2003 Server with a 50GB Disk:

First Pass - 01:26:16 - 10MB/sec
Second Pass - 00:17:14 - 49MB/sec

After digging around on the VMWare forums I found a suggestion to remove the 'Supports vMotion' option from the USB device which I have done and am now re-running the bottleneck tests to see if there are any improvements.

Unfortunately local disks aren't really an option for us.

Thanks for the replys.

[ADM]

Ok, so after unchecking the 'Supports vMotion' option on the RDX drive I re-ran the backup tests.

Backed up the sam 50GB Windows Server 2003 Guest:

First Pass - 00:55:01 - 16MB/sec
Second Pass - 00:10:10 - 84MB/sec

While these speeds are still not ideal, they are an obvious improvement and it just goes to show the kind of overhead that the vMotion option creates.

I still need to try the sDelete defrag to see if this improves things further.

[Gostev]

Do you have a spare physical box, worst possible specs are OK?
If yes then I may have a better suggestion for you

[ADM]

We do have a spare physical box, however the MD is trying to get to an 'all in one box' scenario so unfortunately I'm stuck with what I have for now, although obviously I'm open to suggestion!!

Seth from Support suggested that I defrag the VM Guests and run sDelete to zero the free space so I tried that on my test server and then let a Full backup of all VM Guests run on Friday night.
The VM that I have been using for testing backed up (following the defrag) at 26MB/sec and only took 00:32:22 to backup. In fact, with the Supports vMotion option turned off, all 8 VM Guests (750GB) took 12 hours instead of the 18 it was taking before. Once a defrag and sDelete has been run on each guest, I'm confident I can get this down even further.

As long as we can get a full backup finished overnight, I'm happy!!

As an aside, does the registry hack still work on 5.0.1 so that I can run a full backup on a fresh RDX Cartridge every night, and what happens after the weekly cycle when Mondays cartridge is put back in?? Does the backup get overwritten with a new Full Backup, or does it then do an incremental?

Cheers for the help!

[Gostev]

You may want to try splitting these 8 VMs into 2 separate jobs, this may improve performance as well (make sure CPU on your backup server is not a bottleneck).

I would assume it leaves existing backup intact and creates new full, but this needs to be verified. I am not sure if this key was tested with v5 and it may not even work for new backup modes.

[ADM]

Last nights incremental took 3:32:33 at 60MB/sec.
One of the VM Guests backed up at nearly 217MB/sec!!
Still yet to run the defrag and sDelete though!

Anton, I thought that backing up groups of similar VM's would improve performance as you should get better deduplication?
CPU is hardly working during the backup however RAM usage is quite high (2GB). Unfortunately I have no spare resource RAM wise to throw at it.

[Gostev]

You are 100% right, when target is a bottleneck, it is best to use single job for better deduplication, which translates in less data written to the slowest component in the picture (backup target). Which is the case with using vSphere 4.1 USB passthrough targets.

RAM does not really matter for Veeam B&R, so no worries... 2 GB is fine!