How to improve backup speed: VCB performance in SAN mode

[Gostev]

Darhl, you cannot use VCB SAN mode in a VM since it requires the direct connection into SAN fabric via hardware HBA (unless you want to use iSCSI shared storage and software initiator, which will result in slow performance anyway).

Overall, there are absolutely no pros of installing Veeam Backup in VM for environments larger than a few ESX hosts. The only thing worth virtualizing (if possible) is backend SQL - this will allow you to easily restore your Veeam Backup installation in case of any issues. Physical Veeam Backup server on powerful multi-core will give you tons better backup performance and smaller backups due to being able to leverage Best compression without any performance hit.

But for small environments, it does not really matter because there are very few VMs to backup and you do will not need great speed to fit the backup window.

[withanh]

We currently have 4 hosts and about 25 VMs. Part of our storage is integrated/local to the BladeCenter Chassis, part is iSCSI. Would we still be better off with a physical backup server vs. a virtual backup server? It sounds like you're saying we should be fine with a virtual backup server.

[Gostev]

Well, I do not know what are your backup window requirements and VM sizes, compression level, as well as storage access speed which can vary quite a lot depending on storage device, backup mode and even ESX version. You should try installing in VM first and perform test back up, if it takes longer then your backup window then go with physical server instead.

[withanh]

With the exception of the Exchange backup that we're discussing in http://www.veeam.com/forums/viewtopic.php?f=2&t=1667, my backup times are acceptable. I just created a separate job for Exchange so it doesn't run outside the window, that will run the first time tonight, we'll see how it does

[Gostev]

OK. If your backup times are acceptable and can handle 20-25% environment increase (plus 1 more ESX host and 6 VMs in your case), then you can safely stay with virtualized server since this should be good for your for at least 1 year.

[withanh]

Perfect, thanks for the advice Anton!

[Manfriday]

Hi,

I am running Backup & Replication 3.1.1 and getting slower VCB backup speeds than I was expecting.
I am seeing speeds ranging from 12MB/sec - 34MB/sec.
I have tried running the backup jobs in series, as well as in parallel with up to 4 jobs running at once.
The total throughput never seems to exceed more than 38MB/sec or so.

I am currently running the backups from our backup server and writing them to an openfiler box via SMB.
I thought this was the speed bottleneck, so I did a test backup, writing the file to the backup servers local SATA raid.
The speed did increase, but only by about 3-4 MB/sec.

I ran IOMETER in a VM to get an idea about my EqualLogic san's performance and got the following results:

##################################################################################
TEST NAME-------------------Av. Resp. Time ms------Av. IOs/sek-------Av. MB/sek------
##################################################################################

Max Throughput-100%Read….….15.025.……….…3915.89….....…..122.37

RealLife-60%Rand-65%Read……12.20..…..…..….3324.92.…..…….25.97

Max Throughput-50%Read….……13.18..………….4460.97….…..….139.40

Random-8k-70%Read….….….…..13.40….………..3033.14….…..…..23.69

Here is a run-down of my other hardware

ESX:

3 x Sun Sunfire X4150 w/ 48 GB ram, 12 nics (4 dedicated to iscsi)
All 3 machines are running ESXi version 4
vCenter is also version 4.

SAN:
2 x EqualLogic PS6000E w/ SATA disks, each in a RAID 50 config (14 disks 2 hot spares)


Backup Server:

Sun Sunfire X4250 with 8 GB ram
Running windows 2003 X64
1 nice for iSCSI using microsoft iscsi initiator & Dell's MPIO driver for the equallogic.
VCB 1.5 installed, along with both Veeam 3.1.1, VMPlayer and Symantecs backupexec software (symantec is obviously not used for these backups)

NAS/SMB box for storing my backups:
OpenFiler box attached to an older FC RAID box filled with 16 1 TB drives in a RAID 5 config.
The FC is 1 Gb i think.
I am using SMB to stash the backups on the openfiler box.

So, with the above hardware, should I realistically expect to get better performance than the 38MB max that I am seeing now?
Any ideas what I can try in order to get better performance?

Sorry this got so long. I was trying to be thorough. Odds are I still left something important out.


Oh yes, I did read the notes the original poster made about multipathing, and I am not sure they apply in the case of the Equallogic boxes.
The backup server itself only has the 1 IP address for the group I created on the equallogic boxes.
My understanding is that the Equallogics handle the multipathing.

[tsightler]

I would think you should be able to do better than 38MB/sec. I have a couple of Equallogic arrays with a backup server that's actually less powerful than yours and, while I can only get a single backup to 30-40MB/sec I can run 6-8 jobs in parallel and easily push 180MB/sec with spiked to well over 200MB/sec (that's with dual gigabit links to the source stroage and a dedicated gigabit link to the backup target).

Even our old Veeam test server, a Dell 2650, can push 80-100MB/sec from these Equallogic arrays when running 4 jobs in parallel although only with optimal compression (not highest compression).

I've not been able to tell a performance difference with the Equallogic MPIO driver loaded or not. We leave it loaded, but set to "failover" mode.

Are you sure that your target storage can support fast speed? Some of those old storage arrays are pretty slow with writes. I'd run some performance tests on the target. Also, I think Veeam may have had some performance issues when writing to some Samba based servers. Openfiler uses Samba so that could be a culprit as well.

[Gostev]

I suspect OpenFiler too... try writing back files locally to Veeam Backup server for the sake of experiment.
Changing CIFS to NFS may also help, if this is possible with OpenFiler.

[tsightler]

Another option that I've had good success with for Openfiler, it's just a linux box, so it can be a direct target for Veeam backups. Just enable SSH on the Openfiler, create an account if you like, and add the system as a Linux box to Veeam. Then you can direct your backups to the Openfiler without using CIFS or NFS. Veeam backup will push an agent to the Openfiler and send the backups directly there (appears to use Veeam FastSCP for transfer to my untrained eye). The only real disadvantage to this is that it needs a good bit of CPU on the Openfiler, if it's underpowered, it may not help much.

Good luck!

[Gostev]

Tom, you are correct. Backup compression will be offloaded to target server in this scenario. But, CPU is not a concern with v3 "Optimal" compression setting, as it is designed to be used inside ESX service console (which is very CPU limited). However, "Best" compression level does require powerful CPU on target.

[Manfriday]

Thanks for the insight, guys.

I tried out Tom's suggestion of using the OpenFiler box as a linux target...
I backed up a 27GB VM.

Using the linux target method I got a performance rate of 28MB/s

Then I tried doing two VM's at once. I did a 20GB VM along with a 30 GB VM.
I made sure the VM's were on different ESXi hosts, and on different luns.
I got 17MB/s on one and 23MB/s on the other.

Then I tried writing the backup file to the backup servers local raid (12 x 1 TB SATA disks in raid 5)
The rate I got was 31 MB/s.

Still not great, even going to the local disks..

[Manfriday]

oh yeah, I am using "optimal" compression settings..

[Gostev]

Jason, what are you using Microsoft iSCSI initiator for, it is not clear from description?

Also, what are the results of this VCB test (see the bottom post)? Based on your testing of different backup targets above, the issue is most likely caused by slow VCB data retrieval speed. One thing I would do first is remove multipathing software completely.

[tsightler]

Then I tried writing the backup file to the backup servers local raid (12 x 1 TB SATA disks in raid 5)
The rate I got was 31 MB/s.
Still not great, even going to the local disks..

So you're saying this was a local test straight from the Openfiler? That's pretty slow. I'd suspect there's something wrong here. We use a linux box (not Openfiler, just RHEL with Samba/NFS) connected via 4 GigE iSCSI links (load balanced) to a cheap Enahance Technology RS16i and we can push 360-400MB/sec with 16 1TB SATA disks in RAID 6. Are these 1TB SATA disk connected directly to the Openfiler or via some other type of connection (FC or iSCSI)?

[sthlawyers]

I am running in Veeam in VCB NDB mode and I am only getting 8-9 MBs. I am backing up to a Dell MD3000i. Any suggestions or follow up questions? Any help would be appreciated.

[Gostev]

Sean, VCB NBD mode is unrelated to this discussion (direct storage access mode tweaks are discussed in this thread). But I suggest that you use Veeam "Network" mode instead. VCB NBD is the slowest possible backup mode.

[kristal1]

How would you get the proxy server to use all 4 fiber paths without multipathing software?

[Gostev]

As far as I know, Windows Server has native MPIO? This feature is not enabled by default though.