Exchange backup and replication files are huge

[st0623]

I'm a Technologist with the HyperIP team at NetEX.

Just wanted to throw this out there: HyperIP can accelerate a 3Mbs link with, conservatively, 1.5:1 compression and move 1.8GB per hour. 2:1 compression, 2.4GB per hour. HyperIP keeps the link full by fixing latency and packet loss issues, then compressing the data if it can.

Before you provision any more bandwidth, try our product and the bandwidth costs ROI should easily pay for HyperIP licenses. My email address is steve.thompson@netex.com and our eval URL for Veeam customers is: http://www.netex.com/products/hyperip/v ... ad-request. The eval key is wide open for 30 days to test.

Call me if you have any questions.

[bolnetworks]

We have a Veeam terminalserver backup job which is incremental. In the second run of the backup job on a 40GB HD, the next incremental after an hour is 500MB in size.
I think this is very big, esspecially because it is a terminalserver, so not much data is changing.

I'm running veeam 5.01 on VMware 3.5 update5.

Is this a normal behaviour?

I'm asking this because i have to copy all the changed incremental files once a day to an offsite NAS (nfs).

Arie Rietkerk
Bol Networks

[simeonh]

I have Exchange 2007 with a 130GB Store at the moment.
When i try and replicate the VM over a WAN Link it takes forever because the Online Maintenance of Exchange 2007 Defragments the Database and therefore changes a lot of blocks, which in turn means that when Veeam tries to replicate it, it sees all the changed blocks and replicates it in each restore point. We are getting around 50GB of Changed Data each each day, simply due to the defragmenting at night.

I have thought of turning off the Online maintenance but unfortunately this would mean the store would grow even quicker, because it's not being defragmented regularly.

Do you guys have any best practice solutions for replicating Exchange over a 10Mbps WAN link?
Is it plausible to replicate a 180GB VM (Exchange 2007, using thick disks, 22GB unused space, 158GB used space, 130GB Exchange Store), to another datacentre just using a 10Mbps Connection once per day?
Or even better yet, Near-CDP?

Do I have to upgrade my WAN Link? or can i use other solutions to optimize the WAN Link or maybe change settings within Veeam?

[trentl]

Hi Folks,
I'm after a bit of info
I have a support case open already ID#5205176

We've recently started using Veeam B&R 6.1 and have it backup up our Exchange server.

The server is running Exchange 2003 on 2003 R2 O/S
The server hosts roughly 200 mailboxes and the Exchange DB is roughly 70GB in size
The server is a P2V conversion.

Incremental backups are run every 30min and we are seeing incremental sizes from 2-5GB.

I've checked the backup job and it's excluding the swap file. Compression is set to high and dedup is set to WAN optimised.
I've moved the Exchange database to a new partition that is aligned to reduce excessive block changes.
The server currently has no AV and no scheduled defrag jobs.

I know it's comparing apples and oranges but we were originally using ShadowProtect and the incremental size for those jobs were up to 10x smaller. So it's really the only comparison I have.

So is anyone out there running incremental backups this frequently? and what sort of incremental files size are you experiencing?
Any info would be handy

Trent

[dellock6]

Hi,
it depends on the I/O created on the Exchang server. As we said in this thread and others, even few minutes after snapshots creation the Exchange server can have several changed blocks, leading to high CBT data and following long times for committing the snapshot while closing the backup.
Personally, I've seen some customers having backup time of about 15-20 minutes followed by almost an hour for deleting the snapshot.

So, it's not so uncommon to see such behaviour. As always, further investigation may be needed anyway.

Luca.

[J1mbo]

Also Exchange 2003 is heavy on disk compared to 2007 and especially compared to 2010. CBT block sizes are bigger than Exchange 2k3 page sizes, so there is potentially scope for significant write amplification there. The Exchange maintenance task can be changed to run only once a week provided it then runs long enough to complete one pass and the read/freed ratio stays within guidance (3000:1, IIRC) thereby making the daily deltas smaller, with a much larger delta at the weekend after the task has completed.

[Gostev]

Also Exchange 2003 is heavy on disk compared to 2007 and especially compared to 2010.

That is for sure. Huge difference with 2007, and especially with 2010.

[trentl]

Thanks for the info.
Might be a good time to transition to 2010.

[Gostev]

You should! I pushed for this transition very hard internally couple of years ago, because I wanted some new Exchange 2012 features like cool OWA but no amount of arguments helped me to convince our CIO to touch what's working. Luckily, we started to experience I/O capacity related issues with our Exchange 2007 at about the same time due to our exponential growth. I also got on hold of some technical Exchange 2010 benefits presentation with actual I/O metrics comparing to previous Exchange versions on the same workload... THAT was really convincing stuff, we moved immediately and never looked back.

[Axion00]

Hello,

I thought I would post this here and see if anyone has had similiar experiences with backing up Windows Server SBS 2008 with Veeam B&R.

I have a Nightly backup job that has about 7 VM's in it's set. The backup job is currently just being directed towards a single removable USB drive connected to the server that Veeam B&R is installed on. This drive is not rotated at this time.

I am finding that the majority of the servers in this set seem to take maybe an hour max with most of them taking less than 20 minutes a piece. The most data these jobs end up backing up is maybe 1-2GB of changes every day at the most.

The SBS server that is part of this backup job is the exception to this. It is regularly finding 30 GB of changes to backup nightly on a good night. That takes about 6-7 hours to complete that one VM. This is pushing our backup window outside of an acceptable time frame and then is affecting performance of this particular VM during production hours.

My problem is this. I can't figure out why this server is consistently identified by Veeam(really VMware) as having >30 GB of changes daily when my other servers aren't. The SBS server runs as a DC(obviously), Exchange and then has WSUS on it although the WSUS data is all on a seperate drive that is excluded from the job. So the only thing I can find as far as applications that might have data changing regularly is Exchange and AD. Also, the drive that seems to be real culprit for all of this changed data is the secondary DATA drive and not the SYSTEM partition that AD data is stored on. The rest of the data on this drive besdies the Exchange database is just user data like office documents.

I would like to know if there is any way I can identify what is causing Veeam to find all of these changes and instead of backing up maybe 3 or 4 GB of changes per day is finding 30 GB of changes. I was told by the technician that I spoke to on the phone that this was all just identified by VMware and so he wasn't sure what data might be causing VMware to identify this massive amount of changed data.

My Case Number is 00133173. I am running VMWare ESXi 5.0 and vCenter server 5.0. I am currently running Veeam 6.1

Thanks.

[dellock6]

Probably the huge amount of data is created by Exchange server, this software is known to heavily modify large blocks of data even for small changes in its database (2010 is even worst). You will find several threads in these forums about this issue.
Another concern is about the backup speed, 30 Gb in 6 hours means 1.42 MB per second, is really a poor number. What's the bottleneck as stated by the realtime statistics of the job?

Luca.

[bdawson]

Hello -

We have a few Exchange 2007/2010 mailbox servers that we would like to backup with Veeam (using DPM currently), but we must be missing something as using Veeam so far has been a bit of a struggle. Our main issue has been the size of the backups. We have just over 1TB data spread over 7 databases on 3 servers (2x2007, 1x2010). The first backup is large and expected, but the reverse incrementals are almost as big as the full backup so unless we allocate a 30TB storage platform for Veeam, we cannot get to our 30 day retention objective. My guess is that the online maintenance is causing the issue since we run it daily...

Am I missing something here or is this as good as it gets? DPM can backup all of our mailboxes with 30 day retention at just over 6TB space allocated. Our initial testing with Veeam required almost 5x as much space.

I would really like to use the Exchange AIR tool as this is a significant improvement over DPM, but I don't have a 30TB storage system for Veeam backups...

Any suggestions?

Thanks!

[fcandy]

Case # 00264474

I’m running Veeam backup and replication 6.5.0.144, Hyper-V Replication job and having an issue where by the job is reading and transferring large amounts of data even when the job has been re-run a few minutes apart. The VM is MS Windows 2008 R2 – Exchange 2007 server. The large amounts of data to transfer will prove problematic when I go to move my repository to another site in our WAN with a 2mbps link to the main host server. Currently it’s on same LAN.

Please note there is no AV on the server and I’ve tried different compression and optimisation settings on the job with no real difference. I’ve also tried the s-delete utility which made no change other than to make the job slower.

Below are some html screen shots of three concurrent runs of the job.

Please can anyone explain why in the space of a few minutes the data read and transferred is so large?

Hyper-V Replica job: Replication Job 1
1 of 1 VMs processed
23 July 2013 11:07:50
Success 1 Start time 11:07:50 Total size 200.0 GB Backup size 24.1 MB
Warning 0 End time 11:28:14 Data read 27.1 GB Dedupe 0.0x
Error 0 Duration 0:20:24 Transferred 5.2 GB Compression 1.0x
Details
Name Status Start time End time Size Read Transferred Duration Details
Mail Server Success 11:10:44 11:28:01 200.0 GB 27.1 GB 5.2 GB 0:17:17

This job below was run 2 mins after the job above

Hyper-V Replica job: Replication Job 1
1 of 1 VMs processed
23 July 2013 11:29:31
Success 1 Start time 11:29:31 Total size 200.0 GB Backup size 24.1 MB
Warning 0 End time 11:34:41 Data read 5.1 GB Dedupe 0.0x
Error 0 Duration 0:05:09 Transferred 443.9 MB Compression 1.0x
Details
Name Status Start time End time Size Read Transferred Duration Details
Mail Server Success 11:32:06 11:34:31 200.0 GB 5.1 GB 443.9 MB 0:02:24

The next job was run 15 mins after the job above.

Hyper-V Replica job: Replication Job 1
1 of 1 VMs processed
23 July 2013 11:50:50
Success 1 Start time 11:50:50 Total size 200.0 GB Backup size 24.1 MB
Warning 0 End time 11:57:33 Data read 7.2 GB Dedupe 0.0x
Error 0 Duration 0:06:43 Transferred 651.1 MB Compression 1.0x
Details
Name Status Start time End time Size Read Transferred Duration Details
Mail Server Success 11:54:13 11:57:20 200.0 GB 7.2 GB 651.1 MB 0:03:07

Thanks

[foggy]

Andrew, please look through the topic you've been merged into for explanation of the behavior you are observing. Basically, it's typical for the Exchange Server VM to generate large amount of changes due to heavy transaction logs utilization.

Should any additional questions arise, feel free to ask here. Thanks!

[kte]

I have 2 DB servers and 1 public folder server in 1 backup job of 3 TB ( VM's size is 4TB) when I do a full backup it takes 2,5h and create a backup file of 2,5TB, but all my incremental jobs have a filesize of 250GB to 300 GB. (Veeam read 450 GB to 50 GB) Is there anyway to put exchange in a mode that it changes lesser blocks? Or disable some mainteannce tasks in exchange?

K

[Vitaliy S.]

Disabling some maintenance tasks will not allow you to significantly reduce the size of the incremental files, however if you change storage optimization settings to WAN, that should reduce the size of the VRB/VIB files. Please keep in mind that after changing storage optimization settings in your backup job, you should run an active full job pass to apply these settings.

[kte]

why would this change so mutch to put in wan because the blocks are smaller?
what is the impact on backup and restore times?
can i do this per job or only backupstore?

k

[Vitaliy S.]

1. Correct. See this topic for further reading > Why "Storage optimizations" only for Target?
2. The impact is described in our on-line help center and in this existing thread.
3. This setting is per job, not per datastore.

Hope this helps!

[kte]

what are the max backup sizes because in the forum links they take 2 TB for WAN , 4TB for LAN is this changed in version7

[Gostev]

what are the max backup sizes because in the forum links they take 2 TB for WAN , 4TB for LAN is this changed in version7

v7 has no maximum backup size, meaning no matter of your settings, the backup should never fail with the out of memory error.

For best deduplication ratio, I recommend using 64-bit OS for backup repository, and not exceeding the following thresholds:

Code: Select all

WAN: 4TB
LAN: 8TB
Local: 16TB
Local 16TB+: 128TB

[kte]

is this for the full backup vbk files or ony for the incremenals, the recommanded sizes?

[kte]

if i copy the the backup job and change the dedup from local to wan, will the copy job still work?

[Gostev]

You will need to recreate a copy job after changing the block size. Backup Copy jobs do not support mix of block sizes.