slow backup on new eval of veeam br 4.1

[joergr]

dear board,

we are planning to buy veeam b+r for a datacenter and today we tested the eval license. installed a brand new w2008r2x64 virtual machine, 4 gig, 4 vcpu´s (as suggested by veeam). created backups. checked network mode, very poor performance, only 20% of a gig connection being used. We checked the transfer to a shared network location (which is very highspeed accessible).

next, and that is more important we did the same job with the hotadd mode. luns rely on equallogic iscsi storage, fast as hell with gig, if i did a vcb in the past or a direct transfer i got about 99% of network utilized and transfer rates at about 98mb/s. now the veeam takes it to about 50mb/s. not bad for the beginning but not enough. i checked the task manager in the vm with veeam (latest version, btw) installed and noticed that all 4 vcpu´s were really busy, each at about 50%.

now my question would be: where do i have to adjust to get more power out of this product? 50mb/s is too slow. might it help if i would add additional vcpu´s (i could raise it to eight)?

best regards
joerg

[Gostev]

Hello Joerg, 50MB/s processing rate is perfect for full backup on EQL iSCSI judging on previous feedback from other users with EQL. Incremental backup speed will be about 10x faster assuming you are on ESX4. You can also search for existing threads on this forum about Equallogic where customers have shared some ways to improve performance with these boxes (jumboframes is one of them, if I recall correctly), but again you results are perfect for EQL.

Comparing throughtput rates with direct transfers and VCB are not really valid, as those do not do any post processing (dedupe and compression).

Customers with EQL usually choose to run a few jobs in parallel, and if you have CPU only 50% loaded then you can run 2 jobs in parallel.

Thanks!

[joergr]

So you tell me if i would turn off compression i would have my "old" transfer rates? I will try that tomorrow.

[Gostev]

No, I did not say this.

Disabling compression will not give you "plain" VCB transfer rates because there is additional post-processing besides compression:
1. We do not store backups in plain file, but in Veeam synthetic backup storage - which is similar to transactional database with SIS capabilities and some level of redundancy. Thus, there are additional I/O operations on target storage comparing to writing into "plain" file.
2. Incoming data blocks hashing cannot be disabled, because the backup storage requires it.

There is definitely some performance fee comparing to backing up plain files with VCB, but on the other hand our solution requires 10x or more less storage due to deduplication and compression even for full backups. And when you take retention policy and incrementals into account (something that VCB cannot provide), and the fact that Veeam uses synthetic backup, storage savings become simply incredible.

BTW, I should have mentioned this before, but processing rate in Veeam jobs is not the same as VCB throughtput. Our processing rate is calculated by dividing VM size by the total processing time, and so it is affected by time it takes to communicate with vCenter, perform VSS freeze, perform snapshot management operations and so on.

Thanks!

[joergr]

Gostev,

thanks for the clarification.

best regards,
Joerg

[joergr]

okidoki, got it boosted to a permanent speed of about 90mb/s, that looks better, had to do some tweaking with the autotuning tcp feature uf w2008r2. can i append a picture here somehow? or could you get me your mail address so i can send it to you (veeam guys)?

bets regards,
Joerg

[Gostev]

Interesting! I have PMed you my email address (see private messages on this forum). I will attach the screenshot to your post once I received it. I know for sure many other people will be intereste to know how you did that, because Equallogic is arguably one of the most common SAN among our customers.

[joergr]

It depends and differs from system to system. what you need to know is that microsoft (i guess since vista and server 2008) adjusts the tcp window scaling dynamically with tcp autotuning feature. where this is mostly a good one it can become a bad one on some systems. ive explicitly seen very bad experience with especially high-performance access and rdp access. to get rid of this autotuning feature you have to manually disable it (netsh, interface, tcp, set global autotuninglevel=disable). but then again, sometimes you may want to only set it to highlyrestricted. and again sometimes you may want to set it even more aggressive than microsoft does per default. as i said before, it depends on your system, your nic, your driver, your switches & cabling. You have to find an ideal solution for your special environment.

best regards
Joerg

[joergr]

...and, for heavens sake, if you use Equallogic storage, PLEASE enable flow control on your switches and PLEASE use good ones (for example the dell 6248 is a very good switch for doing great stuff with equallogic).

[Gostev]

Thanks, I think I need to copy this to separate thread and sticky it, so that other customers can try the same thing.

[twlytle]

joergr - Using MS iSCSI you have to specify a target and lun/luns you are connecting to. Are you just connecting to the ISCSI network or are you connecting to all the different luns you are backing up with veeam?

[joergr]

twlytle: i bind my software iscsi initiator to a dedicated nic and then connect to multiple equallogic portals carrying each multiple luns.
hth. if you got any further questions don´t hesitate contacting me.

best regards,
Joerg

[bhwong]

Is the bottleneck due to iSCSI connectivity rather than CPU? Possible to improve the transfer rate if I delicate 2 NICs teamed up for iSCSI connection?

[Vitaliy S.]

It depends, anyway what are you current bottleneck statistics for the backup job? Based on these figures it's really easy to determine the "weakest" part of your setup that should be improved, so you could get higher job performance rates.