New Backup Speed record for me

[dkvello]

From a job running at a new customer:

Veeam B&R 5 Standard Edition
vSphere 4.1, ESXi 4.1



Average 1 GB/s
High 8 GB/s


Backupserver, ESXi Servers and the IBM DS3524 (96*2.5" 300GB 10k spindles) all use 8GB FC connections.

[Gostev]

Nice hardware, big VM and little changes since last pass

I really, really want details on amount of changed data on source added to this report instead of Processed size. I am missing this myself all the time. Devs did not get enough time to add this in v5, but I keep pushing!

[dkvello]

Udo (The second last one) had around 1TB of changes (That's why it "only" did 939 MB/s) and "blackie" was around 200GB of changes...

And Yes, actual amount changed would be nice on the report.
5.0.1 ?

[Gostev]

Are you sure, or is this a typo? Because Udo itself is 1TB, are you saying every single block was changed?
Also for Blackie, did you possibly mean 200 MB instead of 200 GB?

Also, what is your backup server looks like (hardware)?
And where are you backing up to, what storage?
I am really interested in these details!

Concerning the feature, I doubt QC will agreem, maintenance updates = granular fixes only, no new functionality. So most likely 5.1 or 5.5 (whatever it will be).

[dkvello]

Yupp, for this run them are the numbers. Udo's DB's are changed close to 100% every night.

[ctchang]

96*2.5" 300GB 10k spindles ....that's scary.

Btw, I think you are using SAN Offload mode right? And with 1 8Gbps Fibre NIC, you are really pushing it's limit, may I know how many Veeam backup servers are you using? and what kind of hardware configurations ? (CPU, Ram), what's the average loading on the CPU?

Finally, 1GB/s is about 8Gbps, so you must have saturated your fibre link for 2 hours?

[tsightler]

Finally, 1GB/s is about 8Gbps, so you must have saturated your fibre link for 2 hours?

1GB/s is the processing rate, not the transfer rate, I'm sure his 8Gbps fiber was not not saturated. Veeam's "processing rate" is basically "Total VMDK size/time" so it includes skipped blocks such a unused blocks, and blocks not changed since the last backup.

That being said, no doubt that his speeds are impressive for a total backup, but to give you an idea, our nightly backups of 7 VM's totalling 2.5TB regularly report 700-800MB/s, and we're backing up using a 1Gb MetroEthernet link to a facility 8 miles away. It's quite obvious that Veeam is not somehow breaking the laws of physics and trasferring 800MB/s via a 1Gb link.

[lewislampkin]

From a job running at a new customer:

Veeam B&R 5 Standard Edition
vSphere 4.1, ESXi 4.1

Average 1 GB/s
High 8 GB/s


Backupserver, ESXi Servers and the IBM DS3524 (96*2.5" 300GB 10k spindles) all use 8GB FC connections.

That is a nice setup . I'm jealous.

We also have the DS3524, but nowhere near the 96 drives (which is the max-configurable for this unit) (which I would have liked, for performance reasons). We're at 36 drives, and we use the 600 gb 15k sas drives x 24, and then the 2 TB 7.2 SATA x 12


Just to confirm, you are using Raid 10, correct? Or is that too much to ask what your RAID level is? I noticed you obscured the names ...


It's just that I would like some comparisons of what other people are using.

Right now, our backup target is RAID 5, though I would prefer RAID 10, what are you using?

My first backup took 4.5 hours, to back up 7 TB across 18 VMs, using default job settings.

[Gostev]

Yupp, for this run them are the numbers. Udo's DB's are changed close to 100% every night.

In that case, even I am totally impressed with the numbers above can you tell what is your backup server hardware, and where are you backing up to?

[dkvello]

Just to confirm the config.

The DS3524 (with three loaded EXP3524) runs one Array (all 96 drives) in RAID10 carved up in 1TB Logical Drives with 64k blocks and 8MB VMFS Sectors
It's not the Turbo-edition and it only sports the standard Cache (2GB pr. controller)
Write-through caching and Dynamic Pre-fetch off (just ordinary Read-cache).
Cache-mirroring also turned off since we don't use write-back cacheing.

Each of the two controllers int the DS3524 run 4 * 8GBs FC (total of Eight) connections to two Brocade 8GB SAN Switches
ESXi Hosts run two Dual 8GBs FC HBA's each

All Paths are RoundRobin and balances nicely between all available paths to the DS3524
There is no VAAI support on this storage controller yet, but I'm hoping IBM/LSI will deliver in Q1/2011

The Veeam Server is nothing special.
A three year old, recycled HP DL380G5 with two Quad core Intel E5405 cpu's and 32GB RAM
Local disk (for backup) is an IBM EXP3000 with two ESM's (Dual, balanced path with 12 * 2TB SATA disks in RAID5 +HS) attached to an internal HP Smart Array P411 controller.
It attaches to the SAN with one Dual 8GBs FC (Emulex)

Now, going for the 2.5 Inch drives limits the available space (max 146GB at 15k, 300GB disks at 10k, 500GB at 7.2k), but it only uses 8U of rackspace, barely uses any power and hardly generates heat.
The many spindles compensates for the lack of 15k drives so far and IBM has promised they'll allow for at least twice (192) the drives in Q2/2011.
Hopefully there'll be larger 2.5 inch 10k/15k drives by then.

The speed is wonderfull though, considering this is sold as a vSMB, low-end system. It beats all of the Mid-Range systems I've tried (IBM DS5300/DS5020, EMC CLARiiON CX4, HP EVA4400/EVA6400/EVA8400, Dell EQL PS4000/PS6000) so far.

[Gostev]

Thank you very much Dag for this detailed response! As I suspected, Veeam Backup server is pretty powerful. Only one last question - is it still possible for you top open backup properties (select Backups node, right-click the corresponding backup, select Properties), and look up some details on this run (increment size and dedupe/compress ratios). You can simply post screenshot here as well. I will really appreaciate this, since this is a very good real-world data attached to very thorough configuration description. Thanks a lot in advance!

[ctchang]

Anton,

I would like to add my parts here.

Status Success Start time 11/4/2010 2:00:00 AM Details

Total VMs 6 End time 11/4/2010 2:04:31 AM
Processed VMs 6 Duration 0:04:31
Successful VMs 6 Total size 308.00 GB
Failed VMs 0 Processed size 308.00 GB
VMs in progress 0 Processing rate 1 GB/s

Status Start time End time Total size Processed size Processing rate Duration
Success 11/4/2010 2:03:20 AM 11/4/2010 2:03:56 AM 100.00 GB 100.00 GB 3 GB/s 0:00:36


Ha...I am happy now, I am getting 3GB/s everyday as well, but Anton I would like to point out this information is actually "Mis-Leading" as it doesn't mean much really as the VM content or block Didn't Change much between the current and previous backup window.

You can easily see how this works, another example for backing up a 5GB VM, it's only 184MB/s comparing to 3GB/s in the above.

Status Start time End time Total size Processed size Processing rate Duration
Success 11/4/2010 2:01:04 AM 11/4/2010 2:01:32 AM 5.00 GB 5.00 GB 184 MB/s 0:00:27

So this simply means I could potentially get a supersonic speed with no change in block in my VM, so getting 10GB/s or 20GB/s is possible, but this doesn't mean much really as it's not the real number of your backup speed.

That's why I've suggested coming Veeam v5.x to add other meaningful/useful stats than the Processing Rate (simple size/time).

Please kindy consider.

[Bunce]

We've requested these stat updates for a while now. Pretty sure Anton is working on it (and definitely sick of hearing it from me! )

To his credit - he often adds the qualification to his posts when discussion performance numbers.

[Gostev]

it's not the real number of your backup speed

Correct, that is exactly why we are not calling it "backup speed", but rather "processing rate". This comes back from no changed tracking block times, back from version 1.0 (it was implemented this way before I joined Veeam). I never liked it much myself tbh, but there always were more important features pending...