Performance of Backups on Clusters with different SAN

[FBraendle]

HI Guys


I just wanted to get your opinion about the backup speeds of 2 esx clusters we have:


cluster1
Dell R900 Hosts (>180 VMs)
Equallogic ISCSI SAN
Physical Backup Server (Dell R710) with LACP 2GBit/s
Local SATA Shelf as backup target

VMs are partly misaligned
Pagefile resides on C, most of the time
some VMs are ridiculously fragmented...

When doing initial full we get 25-30MB/s
When doing incs we get somewhere between 10-250MBs...
(depending on storage pool / vm setup)

Im fine with those speeds, since the slow jobs contain the VMs with the lets say "suboptimal" configuration...

Cluster2
Dell R910 Hosts
VMAX FC SAN
Physical Backup Server Dell R710 connected via FC


VMs are aligned
Pagefile resides on its own disk, as do the log files etc...
no fragmentation since its all brand new

Initial full 25-30MB/s
Incrementals run with up to 950MB/s (the actual backup of the machines takes less than half a min. (pre/post operations account for 1min)

needless to say, the inc speed is a jaw dropper

BUT: Do you have some special information from Vmware when this annoying 30MB/s limit of the Storage API will increase/ fall away entirely? or is there a way to adjust it?

I can understand this with a highly consolidated cluster like cluster1 where over 150VMs are running on the Equallogic(don't want to crush it...)

But with 32Core Hosts that have a utilization of maybe 5% (since brand new) and are wired via FC to a VMAX with 256GB Cache and the data residing on FC Disks, there is really no need for this limitation....
Not that it would really matter since the jobs themself are not that big(we exclude the DBs) , so the backup window weould be even big enough to take fulls everyday

Greets
Felix

[TrevorBell]

Hi,

Which backup mode are you using ?

[FBraendle]

Hi Trevor

Direct San with network failover(v5)
Its definitly using the SAN since i dont see any network activity...

Job Options:
Reversed Incremental
Optimal Compression
Optimize for WAN target (since i want to have small incs...)
inline dedup activated


Greets
Felix

Backup Server:
newest HBA driver/firmware (qlogic 2500 series)
Windows 2003 R2 SP2 x64
The target(cx4 via 4GB FC) is actually "quite slow", its a Lun on a RAID5 4+1 Raid group with 2TB SATA disks.
But Veeam is the only Application writing to the RAID Group at that times
(the other Backup SWs are configured to run at different times)

[joergr]

Hi,

20-30MB/s is too slow for initial full backup with a killer san how you describe it.

you are 100% sure it uses SAN mode? Check the ESX/i utilization during backup? Do you see a massive CPU rise?
what ESX/i Version?
What HBA? Software/Hardware? If Hardware, please specify. EXACT Driver Version please, too.
Production FC SAN details? How many spindles? SAS/SATA? Vendor? Firmware?
FC Switch type, Vendor and Firmware?

Oh and one final question, just for interest, because i saw it on some R910s these days: Do you have the subjective feeling, when booting and using your DELL R910 that it somehow behaves slow, let´s say compared to a R710 or R900?

best regards,
Joerg

[FBraendle]

Hi Joergr

ESXi:4.1.0

Its definitly in SAN Mode and CPU is almost not utilized...
HBAs are qlogic 2500 series with the 2.16 firmware
Queue depth is set to 256
Driver Version: 9.1.8.25

46 600GB FC Disks (Concat, wide striped)
Switches are brocade

with the dell servers the only thing is, that it takes ages to boot up the r910s since they have 256GB Memory each...

Actually i might have just found the root cause:
since we're backing up the system disks only(plus some smaller disks)
the speed per vm is by no means accurat, since it takes much of the time for pre/post actions and veeam seems to start slow and then speed up.
but since the system disks are "only" 40GB it does not have time to speed up, resulting in the calculated 25-30MB/s

I did some tests with 200GB drives with 100GB actual data and im getting between 65-90MB/s (even with the RAID5 target)
So this seems to be the actual cause...
(my bad, i should have thought of that earlier :/ )
so NOW I'm REALLY impressed hehe

Greets
Felix

[joergr]

OK, 65-90 MB/s is definitely OK. Glad you figured out the main reason by yourself

best regards,
Joerg

[Gostev]

Yep, this is decent result. Can be even faster if backup target speed and backup server CPU allow. But you are right, processing rate is not throughtput.