How to improve backup speed: VCB performance in SAN mode

[drbarker]

Hi guys,

I've been playing with veeam/vcb a while - I thought it would be helpful to share my experiences...

- VCB is fussy about HBA driver performance. Upgrading to the latest qlogic drivers improved performance from 20MB/s -> 30MB/s
- VCB is also extremely fussy about multipathing. You have to disable all paths bar 1 on your SAN. e.g. on an EMC Clariion CX3 , there will be 4 paths by default - disable 3 out of 4. If VCB sees the extra paths performance drops off. After disabling SAN multipathing, we jumped from 30Mb/s to 140MB/s !!

[Gostev]

Wow, drbarker you cannot imagine how much I appreciate you posting these findings here.

This explains so much... I was always totally lost when some customers with awesome SAN hardware were complaining about bad VCB backup performance - actually we had a few about topics about this lately. And all I could say in return was "well, I've seen Veeam Backup doing 150+ MB/s in the DELL lab on FC4, and we also have tons of customers reporting 120+ MB/s speed - so not sure why your performance is so low".

This topic definitely deserves being stickied!

[Suderman]

Hello,

We have a similar situation with EMC Clariion CX300 and CX3-10. Speeds are 30-40 MB/s

What do You mean by disabling all paths bar 1 on your SAN ?
Did You do it from PowerPath ?

Could You give some hints how to do it ?

Thanks.

[kristianaasen]

Suderman: Dunno how the CX works, but on EVA's, you can set NO LOAD BALANCING, and just run with a fixed path that changes if the link does down.

drbarker: if you ever stop by Norway, I got a cold beer waiting.

My speed went from 25MB to 140MB!!

[drbarker]

Suderman:

There are two ways of doing it - from the host & from the array. Full info is in emc's knowledgebase article emc155381, but the short version is...

Uninstall powerpath, then:
1. Log on to your VCB proxy as a user with administrative privileges.

2. Click Start. Right-click My Computer in the Start menu and select Manage. The Computer Management window opens.

3. In the Computer Management window, select Storage > Disk Management.

If the Initialize and Convert Disk Wizard opens, close it.

The lower-right portion of the Computer Management window displays a list of all disks visible to the system.

Inactive paths to disks are indicated by a No Entry icon placed over the disk.

4. To disable an inactive path, right-click its icon and select Properties from the context menu. The Disk Drive Properties dialog box opens.

5. On the General tab, change the value for Device usage to "Do not use this device (disable)." This removes the disk from the list of devices presented by Disk Management.

6. Repeat Step 4 and Step 5 for all inactive paths.

[beckhamk]

I would like to know are the 30MBs number comign from a VCB log somewhere or in the job sessions for VB?

I would then like to know if your 120MBs is without compression or low job compression?

We are looking at the VB job sessions and various jobs run at different MBs some 30MBs, 46MBs, 57MBs, 74MBs.

I had always thought the amount of jobs running and compression and san disk usage affected the MBs shown in VB?

[Suderman]

Hello,

Thanks drbaker for Your reply. I have followed Your instructions.
It seems that disabling inactive paths didn't helped me a lot. My speeds are quite similar like before.

Well .. my speeds are not so bad. It's a 60 MB/s for backup and even less for replication.
I was just wondering if I could squeeze out more from my configuration (EMC CX300 FC 2GB).

To know if I could do more than this I'd like to ask You few short questions about Your scenario.
How did You achieve that 140 MB/s ?

- Did You backup or replicate VM's ?
- What speed of FC do You use ? 2 GB or 4 GB ?
- If You replicate did You do it between arrays for example EMC CX300 to CX3-10 ?
- Did You use compression recommended in Veeam Backup or none ?

Thanks.

[Gostev]

Kyle, with 3.0 fastest speed should be achieved with Optimal compression (which is the default setting).

[beckhamk]

Gostev - I just wanted to point out to everyone and correct me if i am wrong about this with regards to VB), but the MBs will never show you the true MBs wire speed as there are many factors that affect the MBs number you see in the VB session report correct? IE: how busy the san is, vcb wire speed/lan speed, how busy the disks you are writting to and the total time for datadedupe?

[Gostev]

That is correct Kyle, avg. performance rate counter is calculated by dividing the total source data size by the total time it took to backup the VM. Other affecting activities include creating and deleting snapshot, performing VSS freeze etc. - it all takes time and so affects this counter.

[donikatz]

I'll have to agree with other comments that while the advice here is definitely important, it is certainly not the magic answer for SAN speed issues. I've always had multipathing disabled in Windows, but still achieve only 30-50 MBps from my CX300. I would argue the most-important factor remains SAN config: heavy disk I/O, not enough spindles, no alignment, etc. The stock answer should never be "Others are seeing 4-5x your speeds, you should be able to as well." In many configs, despite hardware similarities, you should not be able to as well.

Note that VMW will be releasing some major enhancements in ESX4 that will greatly reduce backup times. So we can hopefully look forward to that sometime mid-year.

[Gostev]

The stock answer should never be "Others are seeing 4-5x your speeds, you should be able to as well."

I never said that everyone should be able to see similar high speeds. Absolutely, SAN speed does depend on specific hardware type and mode... even FC alone comes in 1/2/4Gbps speed modes, so obviously this will provide extremely different VCB SAN backup performance.

What was always very strange to me about this VCB performance problem though, is that customers having exactly same makes and model of SAN and HBA, and both performing backup in off-hours when hardware load is equally minimal, were obtaining extremely different results. Thanks to original poster, now I know what else to check for.

Anyway, the 2 reasons mentioned by original poster might not be the complete list - someone might find out other reasons too - and I would really appreciate if you let all of us know this. By the way, as someone has already suggested, it would probably be beneficial if all speed reports were accompanied with exact make/model/mode of SAN hardware used.

[donikatz]

By the way, as someone has already suggested, it would probably be beneficial if all speed reports were accompanied with exact make/model/mode of SAN hardware used.

I appreciate your position, and of course you are correct. The point I'm trying to emphasize, however, is even with the same make/model/mode hardware there are many other factors, the most-important of which being how you are using that hardware. Two people with the exact same physical config can have completely different logical configs. Even during off-peak hours (which itself is a very relative term), the same exact hardware can give very different results for different logical environments.

This is just the inherent nature of a shared resource environment, such as a SAN and/or ESX. Without accounting for logical and operational differences, the only apples-to-apples comparison would be on isolated paths (FC bus/SP/spindle/array/LUN/source/destination). Otherwise it's no surprise to find some apples are oranges, even with they don't seem to be at first glance.

So I agree: folks should definitely provide their make/model/mode info. And just as importantly, they should provide other environmental info, including details of their SAN and VI3 logical config, as needed.

[gvinpin]

I have 3 VMware environments:
1. 4GB SAN, CX4-120, x3650's as hosts.
Best Performance:
Total VM size: 240.00 GB
Processed size: 240.00 GB
Avg. performance rate: 277 MB/s
Backup mode: VCB SAN
Start time: 25/02/2009 11:22:09 PM
End time: 25/02/2009 11:36:56 PM
Average performance is ~60 MB/s

2. 4GB SAN, CX3-10, x3850's as hosts.
Best Performance:
Total VM size: 68.26 GB
Processed size: 68.26 GB
Avg. performance rate: 182 MB/s
Backup mode: VCB SAN
Start time: 25/02/2009 1:35:45 AM
End time: 25/02/2009 1:42:09 AM
Average performance is ~45 MB/s

3. 2GB SAN, CX3-10, x3650's
Best performance:
Total VM size: 40.00 GB
Processed size: 40.00 GB
Avg. performance rate: 82 MB/s
Backup mode: VCB SAN
Start time: 25/02/2009 9:51:51 AM
End time: 25/02/2009 10:00:11 AM
Average performance is ~30 MB/s

I have 3 different VCB proxy servers with Veeam 3.0 on them. They have 2 HBA's each (all Emulex) One is mapped to VMware LUN's and the other to Backup LUN's. Backup LUN's are on different Clariion's then VMware LUN's
Multipath on VCB proxy never worked for me without disabling inactive LUN's. PowerPath does not work. That is why I have one HBA for for VMware LUN's.

[donikatz]

great info, thanks! how do those speeds compare when only using VCB CLI, not Veeam?

[Edited by Gostev] gvinpin, if you wish to perform such test you can get the VCB CLI command line directly from the Veeam Backup job logs. To do this, create new test VCB job and run the full backup, then go to Help | Support information, look for the latest job log, open it and look for "vcbmounter", copy that command line, substitute asterisks with the actual password, change backup path to real folder (located on your backup destination storage), and run the resulting command line.

[gvinpin]

great info, thanks! how do those speeds compare when only using VCB CLI, not Veeam?

[Edited by Gostev] gvinpin, if you wish to perform such test you can get the VCB CLI command line directly from the Veeam Backup job logs. To do this, create new test VCB job and run the full backup, then go to Help | Support information, look for the latest job log, open it and look for "vcbmounter", copy that command line, substitute asterisks with the actual password, change backup path to real folder on your VCB proxy, and run the resulting command line.

I will see if I can squeeze it this week. I still have old scripts I have used before veeam.

[Gostev]

If you use old scripts, make sure that they are set to produce flat disk output (I believe that default VCB output is sparse) before comparing performance head to head. Because if you configure VCB sparse output, it may work a little faster - but with this mode VCB will not output bit-identical copy of source VM disks. Also, file level restore functionality requires that there are flat disks in the backup file.

[gvinpin]

Thanks Mr. Gostev, I will use scripts from Veeam logs.

[gvinpin]

Stupid question - how do I know what is the performance of VCB scripted backup?

[Gostev]

Just run the command, taking the notes of start and end time.
Then compare the time it takes to process the same VM with Veeam Backup job.

[holografique]

great post. the numbers look really nice. however, im not seeing any of these types of numbers in our lab.

the biggest question for everyone here reporting numbers is: what are you backing up to? what is your backup target storage? it's obviously disk since Veeam doesn't backup to tape. But what disk are you backing up to when you see this type of throughput?

Are you backing up back to the same SAN/Array?

Are you backing up to a different array?

If you are backing up to a different array, what protocol are you using? iSCSI? FC? CIFS? NFS?

Are you multi-pathing to your backup disk target? ( I understand VCB doesn't support multi-pathing when reading from the VMFS datastores, but multi-pathing to your disk target should be ok)

If you are not backing up to a SAN/NAS, then are you backing up to DAS on the Veeam server?


so far, the best performance I've seen is when using a single Veeam server, doing network backups, and having each ESX host backing up directly to a common NFS share. I can kick off tons of multipe sessions from the single Veeam server, with anywhere from 2-4 streams/session per ESX host. As long as your NFS device can handle the load, you scale by having multiple ESX servers generating their own backup streams. So far, based on these results, network backups seems to scale better for larger environments than having to scale out multiple Veeam/VCB servers. This method also seems to eliminiate management issues when managing hundreds of VM''s. Now you can manage all backups from a single Veeam server, instead of multipe Veeam servers and having to remember which server has which backup job for a given VM.

thoughts?
-o

[holografique]

another question regarding people reporting 100+MB/s....do these numbers represent a single stream/backup session of a single VM? or a combined total of multiple sessions? If it's multiple sessions, are all sessions originating from the same Veeam server or one session per multiple Veeam servers?

thanks

-o

[Gostev]

That's definitely a single stream, combined streams are reported to be 500-600 MB/s. Most customers get these numbers directly from Veeam Backup performance report that reflects per-job performance (not combined).

[duhaas]

I was curious to know what kind of results folks are getting when using VCB and ISCSI. I have an EMC Celerra where all my esx vms reside, and see spotty performance when it comes to using veeam. I range from anywhere between 3 MB/s all the way to 66MB/s on my largest vm 100GB. I am using the ms iscsi initiator, am not using multipathing, and am scratching my head. The VMS reside on Fibre Channel disks, and the backup destination is SATA disks. I have two fully populated FIBRE Channel shelves, and one fully populated SATA shelf. I have three luns in which the VCB proxy looks @, each about 450GB in size, just scratching my head on what the problem could be.

[ssimakov]

I range from anywhere between 3 MB/s all the way to 66MB/s on my largest vm 100GB.

Could you please provide us with more details? Do you mean that resulting speeds in reports vary from 3 MB/s to 66 MB/s? Speeds reported in realtime do not necessarily reflect real speed due to the speed calculation algorithm. Also, MS software initiator relies on the OS network API layer so tweaking OS network settings may also help in increasing performance of iSCSI operations.

[duhaas]

The resulting speed in the report I get via email after the job finishes. Like I said, the slowest vm backup job will show about 2-3MB/Sec and the fastest up to 66. We are using HP hardware, have jumbo frames turned on the switch and the nic and in ESX. Being that we are evaluating the product just want to get an understanding of what the best practice settings should be when it comes to network settings or any other settings. Can you speak to the "teaking the OS network settings" and what the might be???

[tjestr]

We've got a performance problem too. The Backup speed varies from 25MB/s up to 69MB/s. We're using SAN mode over 4GB FC. We've got QLogic QLA2462 HBAs in the backupservers and in our Open-E storage servers. We've got 2 EMC 5100 FC switches. Everything is set to manual 4G (on target HBA and initiator HBA side and on the switch). Multipathing is setup correctly on each ESX Host (no path trashing). Multipathing on the backupservers is disabled. The throug-put out of a virtual machine is about 350MB/s read and 300MB/s write. The throug-put on the backupservers is 300MB/s read and 250MB/s write.

Each Open-E storage server has two RAID-10 sets each having 8 disks. (Read Ahead and Write Back enabled)(RAID Controller: Areca 1160)
Each RAID-10 set has 2 volumes/lun with a size of 920GB.
There is a maximum of 5 virtual machines on each volume/lun.

Each backupserver has one RAID-6 sets with 16 disks. (Read Ahead and Write Back enabled)(RAID Controller: Areca 1160)

Veeam options:
We've enabled VMware tools quiescence.
The Compression level is set to Optimal.
VSS integration is disabled.
Everything else is set to default.

diskpart is setup correctly on each backupserver.

Why is the Veeam backup performance so bad?

Here is an example of our backup statistics:
Each Jobs shows a Backup of one ESX host, each having a dedicated Open-E storage server.
The backup load is balanced over three backupservers. Two Jobs are running concurrently on a single backupserver.
The load of the backuped virtual machines is absolutely minimal. (No heavy I/O)



28 of 28 VMs processed (3 failed, 0 warnings)
Total size of VMs to backup: 1.258,12 GB
Processed size: 1.068,12 GB
Avg. performance rate: 31 MB/s
Start time: 17.04.2009 20:00:34
End time: 18.04.2009 05:52:02
---
28 of 28 VMs processed (1 failed, 0 warnings)
Total size of VMs to backup: 929,01 GB
Processed size: 869,01 GB
Avg. performance rate: 44 MB/s
Start time: 17.04.2009 20:00:34
End time: 18.04.2009 01:38:35
---
10 of 10 VMs processed (1 failed, 0 warnings)
Total size of VMs to backup: 815,01 GB
Processed size: 395,01 GB
Avg. performance rate: 50 MB/s
Start time: 17.04.2009 20:00:29
End time: 17.04.2009 22:15:33
---
21 of 21 VMs processed (4 failed, 0 warnings)
Total size of VMs to backup: 699,98 GB
Processed size: 620,00 GB
Avg. performance rate: 42 MB/s
Start time: 17.04.2009 20:00:19
End time: 18.04.2009 00:14:21
---
15 of 15 VMs processed (0 failed, 0 warnings)
Total size of VMs to backup: 1.209,99 GB
Processed size: 1.209,99 GB
Avg. performance rate: 53 MB/s
Start time: 17.04.2009 20:00:19
End time: 18.04.2009 02:28:54

[Gostev]

I would say the issue is likely due to target storage perfromance (for instance, this could be due to cache setting like in this case). It would be best if you run VCB perfromance test I have described above to pinpoint the performance problem.

[tjestr]

The target performance is as described very good (read 300MB/s and write 250MB/s both sequential).
This can not be the bottleneck...

[Gostev]

Have you had a chance to perform VCB performance test?

Also, just to clarify: are you saying these results above are from jobs running concurrently? In that case they are not too bad, since combined speed for the 2 jobs is almost 100MB/s... did you try to run jobs sequantially and see if it helps with overall performance? Incremental backup passes are pretty heavy on target storage I/O, so running two jobs in parallel may reduce the speed depending on how well storage treats random writes.