Slow Exchange 2003 incremental backup

[pops106]

Hello,

I have had a look through the forum and found other users with the same issue but cant find any resolution. We have around 60>70 VM's which are running on average between 30>60MB's each night without any issues. We are using VCB and everything is Hitachi 4gbit FC attached including the backup target.

We do have a lot of changed data a night which reduces the overall 30>60MB speeds but those speeds are fine for us. The Exchange server which is only about 110GB with around 70GB of mail is taking between 2>4MB's.

I have had a look at the logs and there are no errors etc no large pauses in the backup just very very slow.

If I remove the old backup and take a new full it comes in around 43MB's then as soon as we do an incremental we are back down to 2>4MB

We run a NTBackup job well before the Veeam job to clear up the log's ready for Veeam to take its backup, I know there is obviously going to be loads and loads of changed blocks but is there anyway we can increase performance?

Cheers

[Gostev]

Jody, Exchange backups can be a bit slower than regular VMs because it takes long time to quiesce the server with VSS until the snapshot can be created (up to 15 minutes), and this time is accounted for when calculating average VM processing speed.

However, there should not be difference between full and incremental backups of your Exchange server, because under the hood there are almost the same set of operations performed. So I cannot imagine what can lead to such speed difference even theoretically. I suggest that you open support case and let our support investigate all logs - we should be able to compare full and incremental runs from the logs and see what operations take much longer in case of incremental cycle.

Thanks!

[pops106]

A little further testing, I have just switched off VSS but using VM quiesce and it appears to be much much quicker but there hasn't been much change as it is the weekend.

I will do a little more testing and then send the logs through.

Cheers

[pops106]

It appears the C: drive backs up really quickly around 48MB/s then hits the e: and f: which is where the exchange DB's live. Once it hits those disks it drops really quickly at the moment it is down to 10MB/s.

All VMDK's are all on the same LUN which is FC. Once this is finished I will try a fresh full backup and see what happens.

Cheers

[Gostev]

In case of full backup with VMDKs located on the same LUN, the only reason why backup of one drive can be faster than other is due to VMDK containing a lot of "white" space (zeroed blocks). Which is usually the case with system drives.

And in case of incremental backup system drive will be almost always much faster because it typically has very little changes compared to drives with application database and transaction log files.

Yes, VMware quiescence is nearly instant because it does not work on application-level (just file system level freeze) - unlike in case of Veeam VSS which instructs Exchanges to properly freeze itself... there's not much use from Exchange backup with corrupted mailbox databases inside But Exchange VSS freeze can take up to 10-15 minutes and this obviously affects overall VM backup processing speed counter.

[pops106]

It doesn't look like it is anything to do with the exchange VSS, with either using veeam VSS or just the VM quiesce it doesn't effect the performance at the beginning.

I tried to stop the job about 1hr or more ago and it is still in a "stopping" status.

All i can see at the moment is it is to do with the amount of data change, but this is performing really badly at the moment.

[tsightler]

In case of full backup with VMDKs located on the same LUN, the only reason why backup of one drive can be faster than other is due to VMDK containing a lot of "white" space (zeroed blocks). Which is usually the case with system drives.

Anton, I think you should reconsider this statement, there are OBVIOUSLY other reasons that can cause the one LUN to be much slower than the others. The issue with Exchange backups being much slower than other backup types is has been reported on this very forum by many users. We see this behavior as well. Actually, we see the behavior with 3 systems in total, one Exchange and two MSSQL systems. All of these systems have a large amount of block changes spread all across the disk. We backup almost 5TB of data every night, most LUN's give a rate around 30-40MB/sec. LUN's with a large amount of whitespace get 70-120MB/sec, but these three servers only get 10MB/sec or much less (the Exchange server, after a bust day, is maybe only 8MB/sec).

The "slow" server are stored on the same SAN storage as the other VM's, but because of their changes dispersed changes, they backup MUCH slower. Actually, a full backup of our Exchange server is about 3x faster than the incremental pass. I can only assume that the pattern of changes to many small, dispersed blocks is not an optimized case for Veeam. Perhaps this causes an undue IOP load on the target storage or something, but it's obvious that this particular case is not the best scenario for Veeam. We haven't worried about this too much, even though it takes about 12 hours a night to backup our Exchange server, because we figured Veeam 4/ESX4 with block change tracking will work much better in this case.

We see exactly the same behavior this user is posting, the system drive of our Exchange server backs up at 35MB/sec, but the data and log volumes at only 8-10MB/sec. This is not caused by "whitespace" as we have MANY large VM's with very little whitespace (for example a 1.1TB VM with approximately 950GB of compressed files) but this data only has a small percentage of change every day and we see 30-40MB/sec on these systems.

[Gostev]

Anton, I think you should reconsider this statement, there are OBVIOUSLY other reasons that can cause the one LUN to be much slower than the others.

Sure, but we are talking the same LUN above, not different LUNs.

I do believe in slower Exchange backups problem, but the only real explanation for this at the moment is long time required to perform VSS freeze affects the speed counter. As for the rest, the actual VMDK content and its changes simply do not matter for the current version, because as you know with VCB whole VM is pulled from storage for processing each time, during each incremental pass.

So, basically it completely does not matter:
- how "dispersed" the changes since last pass are (whole image is pulled anyway)
- actual content does not matter (100% blocks are processed when Veeam Backup is looking for changed blocks)

Again, it is image-level backup, image can have some application or even do not have OS installed (just random data, this is what we are using for some performance tests) - but this content completely does not matter for Veeam Backup and simply cannot define backup performance, because whole image is pulled and every single image blog is then processed similarly to determine whether it was changed since the last pass.

[Gostev]

Actually I do have one idea on what could be causing this slowness, although it is unrelated to Veeam Backup.

The LUN hosting Exchange/SQL VM files is pretty much constantly busy with I/O (Exchange/SQL do lots of I/O with random reads/writes), so underlying hard drives are very busy - and thus SAN can only give out data to VCB with fairly low speed from this LUN. It is like when you are trying to launch multiple application at the same time on non-SSD hard drive - they will all take ages to launch.

Does it make sense? I do not have much expertise around enterprise storage devices...

You can try using Veeam Monitor (full, with trial license) to check out the datastore load.

[tsightler]

Sure, but we are talking the same LUN above, not different LUNs.

I can reproduce this on the VERY SAME LUN or on DIFFERENT LUNS, it doesn't matter one bit!! Heck, we're lucky right now to have a 15KRPM iSCSI storage array in our lab currently on load for some beta testing. We can put our Exchange LUN, an array that is otherwise 100% quiet (our storage allows us to migrate LUN's between storage systems while still active).

I do believe in slower Exchange backups problem, but the only real explanation for this at the moment is long time required to perform VSS freeze affects the speed counter. As for the rest, the actual VMDK content and its changes simply do not matter for the current version, because as you know with VCB whole VM is pulled from storage for processing each time, during each incremental pass.

This is not the only real explanation for it. As a matter of fact, it doesn't even make sense. Our VSS freeze and removal together take about 45-60 minutes. That's on a 12 hour backup. So the difference with VSS is 11 hours or 12 hours. Not really a big deal since many other systems of the same size (or even larger) take only 2.5-3 hours. Heck, our 1.2TB VM only takes 12 hours. I can PROMISE you our Exchange performance issue is not caused VSS feeze, we can watch the performance of our storage array while the backup is running.

So, basically it completely does not matter:
- how "dispersed" the changes since last pass are (whole image is pulled anyway)
- actual content does not matter (100% blocks are processed when Veeam Backup is looking for changed blocks)"

I understand that it reads the entire disk no matter what, but there must be some overhead when there are lots of changed blocks that must be processed. Perhaps that's the actual issue. The VBK file for our Exchange server is approximately 450GB. A typical VBR file is between 70-90GB. That means the system has to read and write about 3x that much to make the backup (read the old blocks from existing VBK, write them to VBR, then write new blocks to VBK. This doesn't even count reading the blocks from the original storage. So to create my nightly Exchange backup, I'm reading my 450GB of VM files from the source storage, and then reading/writing 210-300GB of data on the target storage. My suspicion is that all of this I/O on the target causes the much slower performance when there are many changed blocks.

Again, it is image-level backup, image can have some application or even do not have OS installed (just random data, this is what we are using for some performance tests) - but this content completely does not matter for Veeam Backup and simply cannot define backup performance, because whole image is pulled and every single image blog is then processed similarly to determine whether it was changed since the last pass.

But if more data has changed, more I/O has to happen. In the Veeam case, a LOT more because of the read/read/write/write nature of creating rollback files. Based on my "uneducated guess" as to how Veeam backup works, it appears that the I/O pattern would be something like this:

1. Read block from VM disk and calculate checksum
2. Read checksum from VBK file for block.
3. If checksums are different then read block from VBK file
4. Write old data from VBK file to VBR file
5. Write new data from VM disk to VBK file

I could be wrong about the checksum, but if I'm right, that's actually 4 IOP operations on the target storage for every changed block in the VM, thus lots of changed blocks would have a major impact in performance of the backup.

[Gostev]

Tom, it is almost right, except that all VBK block size are already calculated and stored with VBK, so minus one read operation. Still, 3 times more I/O operations with target storage than in case of full backup... which - if multiplied by VRB size - is still much less than data that needs to be processed during full backup if we take numbers from your example.

May be also the fact that this is random reads/writes instead of sequential writes in case of full backup, so this may be reducing performance (v3 storage is not optimized for performance and likes to do a lot of small I/Os). But... can it make performance TIMES smaller? Not so sure...

It would be interesting to perform the same test with v4 in VCB mode, as new storage is much "smarter" around reads and writes - it does them much more rarely, and in "big chunks".

[tsightler]

Right, I'm thinking exactly the same thing, that the number of IOPS is much higher. IOPS are storage performance killers so the "randomness" of the I/O on the target may be the issue here. What I'm think is that, in the "normal" case, a VBR file is probably 10-20% of the full VBK size.

I can test v4 with the same server and see how it behaves. I'll work on that today (obviously can't get the incremental results until tomorrow).

[Gostev]

Thanks, it would be very interesting to compare.

[tsightler]

Just to keep it updated, full backup of 320GB Exchange server took 3.5 hours with VCB on Veeam 4. About 35 minutes were for the VSS freeze/thaw, snapshot creation, and snapshot removal. Total speed reported 27MB/sec. That's actually pretty slow right there, but too bad for our environment seeing that I ran it on a busy server during the middle of the business day. The snapshot grew to almost 4GB during that time and took a long time to consolidate because of ongoing Exchange activity at the peak of the business day.

I'd say this time is very close to our performance on v3. We've run a couple of fulls on our Exchange environment with Veeam 3 and we typically see speeds in the 30-35MB/sec range, but we don't normally run them during the peak of the day, so 27MB/sec is probably about right because of the extra load.

[Gostev]

Tom, thanks a lot for testing out this. I did not expect full backup speed to change... but I am really anxious to see incremental backup speed now!

The snapshot grew to almost 4GB during that time and took a long time to consolidate because of ongoing Exchange activity at the peak of the business day.

This exactly scenario when v4 safe snapshot removal is real savior.
Yes, comprehensive explanation of the process is still on my to do list!

[tsightler]

OK, so I ran an incremental today with Veeam Backup 4. Speed was 27MB/sec, almost exactly the same as a full. Actually, amazingly close, the full took 3:19:54, while the incremental took 3:18:44. That's another "middle of the day" backup and there was a good bit of activity going on. That's a huge improvement over the speed I see with Veeam Backup 3. The Veeam4 backup was directed to the same target that we use for Veeam3, so the target disk speed was the same.

Some things we noticed:

Both the v3 and v4 backups used "Optimal" compression, however, optimal is obviously quite different on v4. With v3, our VBK file is nearly 450GB. The source Exchange has 320GB of virtual disk, with about 250GB used, and an initial v3 backup with "Optimal" compression is almost 300GB. The v4 backup was only 146GB for the entire Exchange server even though it also used "Optimal" compression. The VBR file created by the v4 incremental pass was 61GB, while a typical v3 pass with 70-90GB. Don't really know what that means, could have just been a "light" 24 hours.

Anyway, don't really know what this means other than I didn't see a slowdown in incremental speed with v4 with my Exchange server backups and I've ALWAYS seen it with v3. I'm interested to see if vStorage API makes any performance difference, but these test were VCB just to keep things the same. I also can't wait to test vSphere 4 and block change tracking with Veeam Backup 4. Current plan calls for our first vSphere 4 upgrade to take place this weekend.

[Gostev]

Just for my records, with v3 incremental backup speed was about 12 hours for the same VM, so this basically makes it almost 4 times faster with v4?

For ESX 3.5, vStorage SAN and VCB SAN modes provide identical speed, so vStorage API could not affect these results.

The speed improvement you are seeing comes 100% from the new transactional backup storage. We've learned a lot based on v3 support cases and statistics, and designed the new storage in a way to make storage I/O pattern suit most storage devices and I/O controllers. As some customers proved before, you can fix v3 slowness by either changing I/O controller settings, or swapping controller all together for another one which is less picky to v3 storage access I/O pattern. It was a real pain to troubleshoot, and used to generate many support cases and forum posts...

And apparently (thanks to your testing), all these issues are indeed going away with v4, good news for everyone, especially our support! Sounds like we have delivered exactly what I promised you 4 months ago

The whole post is actually interesting to re-read now when 4 months passed, but here is the most relevant part:

Yes, Veeam Backup 4.0 enhanced storage has similar concepts, although we are not implementing these enhancements for the same reason alone. We've taken a complex approach because backup storage seems to affect just about anything: full and incremental backup performance, repair time, crash protection, integration with 3rd party storage device (aka DataDomain CIFS integration issue), etc. And many of these requirements are typically conflicting with each other (speed vs. reliability for instance). So now that we have collected all the intelligence, we have designed the "best of all worlds" implementation.

Indeed, it was actually a big problem for us to design complete VBK protection from backup server crashes without affecting backup performance heavily (frankly, I thought it would be impossible). And my biggest concern was that if product evaluator does not see good backup performance - they just go away and we lose the customer - he or she simply would not get to the next step, evaluating our awesome backup reliability. Anyway, our devs seem to have made the impossible... yet again. By the way, it would be awesome if you could find some time to participate in closed beta when it is available and help us verify that we got it all covered, since from what you said above you clearly have great experience with VBK crash-tests

[tsightler]

So I couldn't help myself and decided to run a vStorage API test as well. Talk about impressive. Admittedly there was less change (since I had just run a backup this morning), but it was still 30GB written to the VBR (about 4 hours of activity). The backup took 1:59:28 for a reported speed of 46MB/sec, significantly higher than the VCB reported speed of 27MB/sec, and especially impressive considering the hardware is an old Dell 2650 (a dual CPU, hyper-threaded system, certainly nothing to write home about). I truly cannot wait to run my first tests with vSphere 4 and block change tracking next week.

[Gostev]

Great increase over VCB, our implementation of vStorage API is expected to be faster than VCB based on our testing too... even though because you are using ESX 3.5, vStorage API still reads the whole VM in your case during incremental passes, just like VCB.

Anyhow, based on these results, there's little sense to use VCB with v4, even if you are not on ESX4.

[tsightler]

So this weekend we upgraded our infrastructure to vSphere 4, including all but two our ESX host (we have 12). This meant I was able to give vStorage API with changed block tracking a spin on our slow Exchange VM. Initial backup, as reported above, was 46MB/sec. An incremental was 70MB/sec, taking 1:18:33 with about 20 minutes for the VSS freeze/thaw, and final snapshot removal. Pretty impressive compared to the fact that our nightly Veeam Backup 3 incrementals take ~12 hours. that's a 10x increase for us, and our test backup server is slow. Looks like this will not be an issue for us with Veeam Backup 4. Can't wait to actually use it in production!