Veeam backup slow but VCB is fast. Why

[NWP_Tom]

Ok, I'm currently using Symantec BE 12.5 with the AVVI module. We are trying to speed up our backups and are testing Veeam but I'm not impressed so hopefully it's just a tweak that will fix our issue.

I performed a test backup with the Veeam software and got a whopping 11 MB/s according to the backup report. Following the guidelines laid out by Gostev in this thread http://www.veeam.com/forums/viewtopic.php?f=2&t=685 I tested the VCB framework to ensure that it's not VCB slowing things down. I was able to run the same VCBMounter command I located in the Veeam log files and it calculates to be about 65 MB/s.

Veeam 11 MB/s
VCB 65 MB/s

Why such the huge discrepancy in speeds? What do I need to look at?


VMWare environment:
Dell 2950 servers
MD3000i iSCSI san

Backup Server:
Dell R200
Dual core 3ghz with 4 gb ram
1 gb link to iSCSI san
Internal storage (where I backup to) is 2x 750GB SAS drives in Raid-0 (striped)

[Gostev]

Tom, when doing your testing did you point VCB to store file in the same location as Veeam Backup job? Where are you writing your backups too? Only thing I can suspect based on your provided metrics is target storage which is used for Veeam job.

65MB/s is exactly what I would expect from typical iSCSI storage; so you really should be getting similar (actually, faster) speed with Veeam when pointing both to the same target storage.

I think best would be to setup webex and let our devs take a look at this in live.

Really, VCB data retrieval speed is the most important thing which matters to be able to get fast backups with Veeam, and everything is just fine with this in your case. Other issues are easy to resolve (in most cases, I/O controller on target storage is responsible for the backup slowness when VCB is fast).

[NWP_Tom]

Tom, when doing your testing did you point VCB to store file in the same location as Veeam Backup job? Where are you writing your backups too? Only thing I can suspect based on your provided metrics is target storage which is used for Veeam job.

Yes, I pointed the VCB to store the file on the same set of disks. The backups are being written to 2 750gb SAS drives that are setup in RAID 0 on a Dell R200 server. The raid controller is the SAS 6ir that is built into the motherboard.

[Gostev]

I should have asked about this before, but are you using drive exclusion feature in Veeam job by any chance?

[NWP_Tom]

I should have asked about this before, but are you using drive exclusion feature in Veeam job by any chance?

I dont know what that feature is but I'm going to venture a guess and say that it allows me to exclude a particular drive in a VM from being backed up. If that is the case then no I am not using that feature.

I setup Veeam to talk to my vCenter server, then selected a particular VM to backup, put in the path to where I want the backup file stored and told it to do it's job. It backup up the whole VM.

[Gostev]

OK, then it would be to setup webex and let our devs take a look at this in live...
Please shoot email to support@veeam.com and include the link to this topic.

[NWP_Tom]

OK, then it would be to setup webex and let our devs take a look at this in live...
Please shoot email to support@veeam.com and include the link to this topic.

You've got email

[schedlbauerm]

I have nearly the same problem.

I use Veeam Backup for about 3 weeks. At the beginning the backup speed was about 40 MB/s according to the backup report.

Since the first full backup the backup speed is constantly decreasing and is now at about 17 MB/s. I thought that the incremental backups should be faster and not slower.


I use the VCB framework with Veaambackup.
To ensure that it's not VCB that is slowing things down. I was able to run the same VCBMounter command I located in the Veeam log files and it calculates to be about 50 MB/s.

Veeam incremental 17 MB/s
VCB 50 MB/s

Why is there such a huge discrepancy in speeds? What do I need to look at?


VMWare environment:
HP DL380G6 servers with 32gb memory

Backup Server:
HP DL380 G4 server
Dual core 3ghz with 8 gb ram
Windows Server 2003 R2 64bit

2 gb fc-hba connected to a Datacore SDS-Server

Internal storage (where I backup to) is 11x 750GB SATA drives in Raid-5 (HP-MSA60)

[Gostev]

Martin, please also contact our support for troubleshooting, based on your numbers I can only suspect target backup storage.

[fgw]

something else to take into account:

first of all, i don't know how speedcalulations are done when using vcbmounter, but if net backuptime is used, you cant compare this to the speed indication coming from veeam.

same if you are simply copy files between esx servers or the vcb proxy and measuring elapsed time.

veeam calculates the backupspeed by using the total time it takes to backup a vm. so if it takes a reasonable amount of time to create and remove the snapshot, this time runs into the backupspeed calculation resulting in lower figures.

the best way to get some more accurate indications would be to compare the file creation and file modification times of your backupfiles and use this together with the actual filesize to calculate backupspeeds.

i know this won't make your backups run faster, but at least you are going to compare apples to apples.

let us know what figures you end up with ...

would be a good idea for veeam to add this figure to the veeam gui too. eg. total backup speed (as it is displayed now) and net backup speed.

[Gostev]

Ah, I've been asking dev.lead about such counter for ages - but something prevents from easily adding it... so this feature constantly being pushed out.

You are totally correct above, our speed counter is "processing speed" - VM size divided by total time it took to backup. So snapshot operation time (VSS freeze alone can take up to 10-15 minutes for loaded Exchange servers) and VI API interaction through SOAP (connecting to vCenter, enumerating VM data etc.) can also take up to 1 minute) does affect the counter. Still, this is actually more useful and "fair" counter, as it gives actual processing speed and so allows approximate backup activities much better than throughtput counter.

However, this situation is not "normal" anyway - speed difference is too large. I wish I knew support case numbers for these issues so I can check on how is this going.

[schedlbauerm]

Here is more data:

This are the VCBStart- and VCBEnd-Times of one of our Veeam-backup-Jobs. The VM is a Windows Server 2003 SP2 server (32-bit) with Oracle 9.2 on it. During the Veeamackup the Oracle-Services are stopped and nobody is connected to the VM.

[20.09.2009 22:11:07] <15> Info VCB Started
[20.09.2009 23:10:36] <15> Info VCB Ended
[22.09.2009 01:38:13] <15> Info VCB Started
[22.09.2009 02:43:17] <15> Info VCB Ended
[23.09.2009 01:38:08] <15> Info VCB Started
[23.09.2009 03:05:32] <15> Info VCB Ended

The remaining VCB Started time is always between 01:38:00-01:38:30
[24.09.2009 03:17:18] <15> Info VCB Ended
[25.09.2009 03:31:56] <15> Info VCB Ended
[26.09.2009 03:31:50] <15> Info VCB Ended
[29.09.2009 03:52:15] <15> Info VCB Ended
[30.09.2009 03:44:10] <15> Info VCB Ended
[01.10.2009 03:57:06] <15> Info VCB Ended
[02.10.2009 03:52:16] <15> Info VCB Ended
[03.10.2009 03:43:36] <15> Info VCB Ended
[06.10.2009 04:11:29] <15> Info VCB Ended
[07.10.2009 04:05:40] <15> Info VCB Ended
[08.10.2009 04:07:00] <15> Info VCB Ended
[09.10.2009 04:16:02] <15> Info VCB Ended

As you can see the Backuptimes are 'constantly' increasing.

The size of the vbk-file is 141 GB and the size of the vrb-files is between 30-32 gb. I don't understand why backupspeed is constantly decreasing.
This problem concerns all of our vms. Backupspeed is now at about 50% of the speed of the initial backup.

Here are the outputs of the Veeam Console concerning this job:

16 of 16 files processed

Total VM size: 147,00 GB
Processed size: 147,00 GB
Avg. performance rate: 41 MB/s
Backup mode: VCB SAN

Start time: 20.09.2009 22:10:55
End time: 20.09.2009 23:11:52

-> Speed 41 MB/s

16 of 16 files processed

Total VM size: 147,00 GB
Processed size: 147,00 GB
Avg. performance rate: 16 MB/s
Backup mode: VCB SAN

Start time: 09.10.2009 01:37:54
End time: 09.10.2009 04:17:42

-> Speed 16 MB/s

@ gostev:
My support number is [Ticket#513578]

This is the answer of the Veeamware-Support concerning my problem:

Hello Martin,
Even if there were no any software or hardware changes, during the VM is running there should be some transactions on that VM. While you are performing incremental backup Veeam scans file system for changes in blocks between the source and current state. The time needed to perform this depends on the hardware performance, disk fragmentation and last but not the least the amount of transactions comparing with last backup.
Hope I've answered your question.
Thank you for contacting Veeam Support.

I' don't think that this is the whole explanation for my problem. As the size of the vrb-files show, the amount of transactions comparing with last backup stays nearly the same (always 30-32 gb), but the backupspeed has dropped to less than 50 %.

I also think that the hardwareperformance of the Backup-Server should be constant, because there are no overlapping backup jobs and no other tasks are performed on the backupserver during the VeeamBackup. The cpu-usage is about 20-30% during the backups. The disk is about 10% fragmented and this number didn’t change much during the last backups.

Another important point is that this problem doesn’t only occur on 1 or 2 VMs, but concerns all our VMs. For all our VMs the backupspeed has dropped to under 50% compared with the initial backups.

I hope someone can point me in the right direction or VeeamBackup is getting useless for me.

Greetings Martin