6.1 Patch 1 = HUGE Improvement!

[lobo519]

WOW! My backup jobs run time have been cut in half (Reverse Incremental) with patch 1! Craziness!

[davidb1234]

whats your setup like? we use reverse incremental too and i just installed the patches today. we'll see what happens. all FC SAN, direct attached here.

[lobo519]

All my storage is iSCSI. I have a job that backups locally and a second job that backs up to a co-location. Both literally are finishing in half the time.

[foggy]

WOW! My backup jobs run time have been cut in half (Reverse Incremental) with patch 1! Craziness!

Thanks for sharing this! I know, our R&D did work really hard to gain this improvement.

[homerjnick]

Hmmm installed the patch and everything is fine but haven't had time to check timings...good point I will do...

[pendragon8062]

I must concur. Our synthetic full and transform (previously a 22hr job on 6.1) took 10 hrs after applying patch 1.

Many thanks to the Veeam team!

Chris

[chimera]

question is, what caused the supposed poor performance in the first place? bug fix or utilisation of an "undocumented vmware feature?" after patch 1, our backups are more stable than before, most vm's backup performance is no different (10GbE iscsi, direct san, reversed incrementals) but i do see that some vm's are significantly quicker (2.5 times faster). I assume the variation is due to the amount of data changed (or not) on each vm.

We do Veeam backups to local storage ( 6 x 2TB SATA's in RAID 0 ) then off to tape afterwards. I see Veeam is noting that in half the jobs, the target is the bottleneck, so am going to upgrade write cache in the RAID controller see how much improvement this makes.

[VER]

Same here. I went from a duration of almost 8 hours to a duration of 3:15.
I'm very happy with this update.

Our physical backup server has iscsi connections to the SAN, backups are stores on local disks.

[Gostev]

question is, what caused the supposed poor performance in the first place? bug fix or utilisation of an "undocumented vmware feature?"

As you may already know - under the hood, our backup files have pretty complex architecture. They implement single instancing for block-level deduplication, B-tree indexes for high-performance block lookup (important for instant restores), as well as transactional updates (to ensure data is not lost due to crash or power off) implemented through some data redundancy.

Transactional updates require knowing for sure that certain data is committed to the storage. This can only be done by issuing low-level flush command to the storage device. Previously, we would flush data too often - and our engine was spending most of the time waiting for flushes to complete. Now, flush happens more rarely, which in turn improves performance significantly. Updates to the backup storage are still transactional, just happen with bigger chunks of data. The only drawback is very slight (2GB on average) increase of VBK size (more space is required to maintain data redundancy with more rare update cycles).

One other performance enhancement relates to using direct SAN access mode in large deployments with many LUNs, this one is best classified as optimization. Originally, we simply did not realize certain operations can take so much time in large environments, and did not pay much attention to optimizing certain parts of our code.

Version 6.5 will have additional enhancements around direct SAN access that we have recently identified thanks to this post and willingness of this customer to work closely with us. I can already say that in 6.5, you can expect a few more minutes cut down from each virtual disk processed in direct SAN access mode, resulting in significant overall backup performance improvement. We never stop improving

[Gostev]

i do see that some vm's are significantly quicker (2.5 times faster). I assume the variation is due to the amount of data changed (or not) on each vm

Correct.

[deduplicat3d]

Thanks for the technical details... Got my CS degree at Cal, so I appreciate the small things.

[babyjuale]

Anyone here with synthetic backups configured in their jobs and can speak to the improvements they are seeing with Patch 1?


Thanks.

[Starman]

My backups are also. my Veeam vm has 8 cpu's assigned to it and now vCenter keeps flinging me warnings about CPU utilization. Nothing wrong with that but its clearly hitting CPU a lot harder!

The VMworld Veeam party was great! Saw Gostev in the Veeam session but seeing him get down on the dance floor with his girl friend / wife to techno was priceless!!

[Gostev]

Hmm, actually she is former Veeam employee, and she was with her husband at the party (who is also my friend) but yes, she is the best dancer I've ever met. Wow, this is worst offtopic ever, haha!

[deduplicat3d]

hahaha, I missed more than just vSphere!

[zoltank]

Hmm, actually she is former Veeam employee, and she was with her husband at the party (who is also my friend) but yes, she is the best dancer I've ever met. Wow, this is worst offtopic ever, haha!

I nominate this post for Post Of The Year.