Exchange backup and replication files are huge

[gopher_49]

On my backend exchange server and my file server my backup and replication differentials seems way too large. My backend exchange server files are showing 16 GB's of changed data per day and my file server is showing 20 GB's of changed data per day. That seems way too large... My email db is only 50 GB's in total size... This just seems too large. Any suggestions or ideas? Is there a way to sync my exchange OS with one job and my db with another? Maybe that will help?! Maybe do the Exchange infostore drive with commited files nightly and OS on weekend...

[tsightler]

The Exchange server would seem to be right in line with expectations based on current Veeam technology and block size. There are many threads about this issue.

You don't say how big your fileserver is, but I'll admit that seems pretty large as, even our 1.6TB fileserver rarely has more than 20-40GB of changes. Do you happen to have any defragmenting software running on the server? Is there any process that is constantly updating and deleting files?

[Gostev]

Tom had posted good and detailed explanation about Exchange in this topic yesterday. Thanks!

[gopher_49]

interesting for I have a few tickets in the past pertaining to large backup and replication file sizes... I mentioned large block sizes for I thought maybe my virtual disks had large block sizes enabled or something.. so.. Long story short.. When will your new release be out that addresses this?

[gopher_49]

Also,

My file server is also a SQL server... I guess this makes sense too... I'm very anxious to get this resolved for my remote replication project has been on hold due to the fact that I couldn't figure out a way to get this much data across the wire... I knew something was odd about this and I was blaming it on the block size of my target volumes or maybe even the source volumes... I guess I was kinna close.. hahahaha...

[tsightler]

Yep, SQL has the same problem. Are you already using the "WAN target" option? That will give you the best solution that Veeam, by itself, can deliver today. Then you could try HyperIP, which does help a lot since SQL and Exchange data have a tendency to be very compressible.

[gopher_49]

The WAN target helped but it's still insanely large.. Sending over my mission critical data on a 3mbps connection will never happen with this size.. I'll download HyperIP.. If it's real expensive I might hold off my replication project until VEEAM has this resolved. In regards to WAN replication I just don't see how VEEAM can be used in a SQL and Exchange environment... At least not at a cost that is affordable.. I can't even get a internet connectino where I'm at that's over 3 mbps.

[gopher_49]

I called VEEAM support and they knew nothing abouy HyperIP... They also said it wasn't free nor bundled, however, below is copied and pasted from Netex.com's website?! I guess this means I get it for free for one year.. hopefully VEEAM will have this resolve before then.

- HyperIP 2Mbs subscriptions are free to Veeam Customers
- Maximum of 2 free HyperIP virtual appliances are available per customer
- Free subscriptions are valid for one year
- Offer available through July 2011

[lobo519]

HyperIP isnt cheap - we bought a 20 MB license.

But I am using it for Exchange server replication and getting pretty good results. Takes anywhere from 5 - 20 minutes to replicate each server (2 servers - separate jobs ) depending on the load (15MB pipe shared - throttling to 15MB during the day).

However I seem to being having some trouble with the Backup job of an exchange server taking 8 hours to run. seems to just kinda crap out and not really push any data or does it at a snails pace. I can see in HyperIP almost no data going across. Still looking into it though...

[tsightler]

But I am using it for Exchange server replication and getting pretty good results. Takes anywhere from 5 - 20 minutes to replicate each server (2 servers - separate jobs ) depending on the load (15MB pipe shared - throttling to 15MB during the day).

Your link appears to be much higher than his, although it's a little difficult to understand since you said 15MB throttled to 15MB, which doesn't make much sense so I'm assuming one of those is a typo. Also, do you mean 15MB (120Mb) or 15Mb? The OP said he had only a 3Mb, which I'd guess would take him around 12 hours with Veeam and no other optimization based solely on his 50GB info store size (he hasn't really supplied any other information such as log generation per day, etc). I'd guess HyperIP would cut this in half, maybe a little better, and get it to the 4-6 hour range.

We are backing up a system that at least sounds similar to the OP's. It's an Exchange 2003 server with an infostore that's just under 50GB. The system has 50, relatively heavy users, with about 50% of them using unified messaging and Blackberry devices which significantly increase the traffic load. Active archiving also increases the generated logging. We backup this server with Veeam nightly, via a 2Mb link from our Europe offices, to one of our datacenters in the US. Without WAN acceleration this would take probably 15 hours or so. With WAN acceleration we're able to get this done in about 3-4 hours.

One thing you can do is be sure to disable the nightly info store online defrag runs. We configure our Exchange servers to run online defrags only during the weekend (when normal traffic is lighter) rather than for 4-6 hours every night which I think is the default. This online defrag run simply adds to the amount of changes that have to be replicated during the week.

[gopher_49]

I have not replicated my Exchange server via the WAN option yet... Currently the replicas are 13-19 GB's in size. We plan to replicate this over a 3mbps DSL connection via a site to site 3DES tunnel... I'm lucky to get 2.6 through the tunnel for the upstream of my dual T1's is right at 3mbps... so, at 2.6mbps we can send 1.17 GB/hr. I'll perform my first replica this evening using the WAN option. Then the following day it will run again at 1am. We'll see how large the files are then...

I just got off the phone with HyperIP. They claim their application will help in regards to the transfer of the data through the internet and will have SOME affect on the file size... They said it will mainly just speeds up the transfer. Most of the performance increase happens over the network... They suggest I give it a shot via a trial key.

My maintenance schedule was set to daily. I changed it to Saturday and Sunday only... We'll see how that in conjunction with the WAN target works.

The bottom line is that VEEAM needs to backup SQL and Exchange via a smaller block size, right? The current 256 KB block size versus the 8 KB block size for SQL and Exchange is using is insane...

[tsightler]

It's probably not realistic to expect Veeam to be able to replicate at a very small block sizes as this would potentially have negative impact on the performance of backups in general. Trying to account for a large number of blocks has very high overhead, and it's the ability of Veeam to compress, dedupe, and manipilate blocks very quickly that provides for many of it's innovative features like Instant Restore, and fast file level recovery.

In the end, the size on disk is relatively "non-important" because the cost of disk is "cheap". However, the cost of bandwidth is still expensive in many areas. For example, I can buy an enterprise capable iSCSI storage system, with 32TB's of storage, for ~$7,000. However, my 10Mb WAN link to my remote datacenter costs me ~$5000/mo (counting the costs of the circuit on both ends).

So, what I'm hoping for is for Veeam to decrease the amount of data that actually send over the wire. For example, if I have a 256K block, and I change 1 byte, I don't want Veeam to send the entire 256K block over the WAN. Instead, I want Veeam to basically read the new 256K block, read the existing 256K block on the target, and then only send the bytes that changed in that block using something like the rsync algorithm. This will not reduce the size of the rollbacks, but will reduce the size of the data that has to traverse the WAN for small block transaction workloads like Exchange and SQL.

I have no idea what solution Veeam will actually implement, but they understand the issue, and they're an innovative group of guys, so I'm sure they'll come up with something. If they figure out a way to use small block sizes, well, that would be awesome too, but I'm just not sure how harmful that would be to the performance of other features in the product. Simply decreasing the data that has to traverse the WAN is good enough for me.

[gopher_49]

Either way sounds like it will work.. I look at the replication feature seperate from the backups. I'm not performing restores or file level recovery with my replica jobs.. I simply want to replicate to another host for in the event of disaster I can turn it on and go live... I'm perfeclty fine with it hammering my processors during the replica job... We'll see what they come up with but the sooner the better!

I thought they already had technology that only sent the changed blocks... I thought it only sent the part of the changed block and not the entire block... I thought it was like rsync protocol where sections of the changed blocked were sent and not the entire block.. I guess not... We'll see... I'm about to be at the phase of my virtualization project where replication will be implemented... Hopefully it won't be too long.

I'll update the ticket once my second replica job runs...

[tsightler]

I thought they already had technology that only sent the changed blocks... I thought it only sent the part of the changed block and not the entire block... I thought it was like rsync protocol where sections of the changed blocked were sent and not the entire block.. I guess not... We'll see... I'm about to be at the phase of my virtualization project where replication will be implemented... Hopefully it won't be too long.

Well, they do seem to compress the data (assuming ESX, you don't even get that with ESXi). Actually, if you have ESX server you can replicate the server locally with Veeam, and then rsync the replicated VMDK between ESX servers (using the ESX console) faster than you can replicate with Veeam natively.

It is important to note however, that nothing Veeam can do will be magic. Replication of an entire Exchange VM with a 50GB info store via a 3Mb link is still likely to take 3-4 hours because it won't be uncommon for the server to create 2-3GB of logs, and there will be 1-2GB of changes in the info store. You've never mentioned how many GB's of logs your Exchange server generates a day, but that pretty much indicates the minimum amount of data that has to be transferred no matter the block size.

[Gostev]

It is important to note however, that nothing Veeam can do will be magic.

That's right, unless of course we can come up with something similar to MP3 or JPG for your transaction logs data hehe. Although I am sure most people here would still prefer their backups to be lossless APE

Replication of an entire Exchange VM with a 50GB info store via a 3Mb link is still likely to take 3-4 hours because it won't be uncommon for the server to create 2-3GB of logs, and there will be 1-2GB of changes in the info store. You've never mentioned how many GB's of logs your Exchange server generates a day, but that pretty much indicates the minimum amount of data that has to be transferred no matter the block size.

You should keep in mind compression that we do, too... and disk blocks backing those transaction logs files usually compress extremely well. But even without compression, we have some ideas on how we could go below this minimum amount in some cases - while still being lossless, of course

Let's just wait and see how it all works out.

[tsightler]

I don't want to wait!! I'm already bored with v5.

[Gostev]

Unfortunately, 9 women cannot give birth to a child in 1 month.
But we will most certainly have at least 9 beautiful children (features) in about 9 months!

For example, I can buy an enterprise capable iSCSI storage system, with 32TB's of storage, for ~$7,000.

Are you serious? What storage make and model is that?

[tsightler]

Are you serious? What storage are you thinking here?

Well, I'm not talking about "big boy" storage, but high performance storage with good performance excellent for a second tier like a backup target for Veeam. We're currently using Enhance Technology RS16-IP4, which is 4 1Gb iSCSI ports. We "bring our own" drives, which is a lot cheaper than buying it in a bundle with the chassis. Last time we priced the 2TB drives that are certified for this array they were around $225/each, and we were able to get the chassis for right around $3500 (it was actually a little less, but it was some type of promotion). So thats:

RS16-IP4 Chassis - $3500
16x2TB Drives ($225) - $3600

For a total of $7100 for 32TB of raw storage. You could bring cheaper, non-enterprise class drives and lower this price a good bit. These things are not "barn-burners", especially with random I/O, but are pretty good for streaming backups, and support simple snapshots and full web based administration. Heck, they managed to get us through our disaster when our Equallogic array died on us.

I'm not sure we would use these for tier-1 storage since they don't have redundant power and redundant controllers, but Enhance does sell arrays that have these features for only a few thousand more, including some models that support 10Gb. We've been seriously considering buying several of these and front-ending them with clustered OpenFiler for full, array-independent redundancy.

[gopher_49]

I run ESXi only.. Now... My NAS appliance has rsync built into it. (It's a Linux based appliance) I guess I could experiment with using it to replicate for now. I thought about that in the past.

I'm perfectly happy with 3-4 hours to sync my Exchange server. I actually have two Exchange servers with 50 GB DB's and it taking 8 hours to sync for both would be perfectly fine with me. When looking at other replication products this is what I was told also... So... I'm happy with that.

I also plan to replicate my Exchange system volumes too. I assume those will be small.. I'll make note of my log files this evening before their truncated but I think they grow to around 2GB's in size.

[gopher_49]

Also,

Since ESXi doesn't have rsync how can I replicate from my NAS appliance to a ESXi target? I guess I could place one of these NAS appliances at the remote site and store my VM's / replicas on it for the NAS appliances can replicate to each other and they can see the data volumes... But.. I'm trying to void purchasing another NAS appliance... I guess I need the ability for my Windows based rsync server to see my VMFS volumes... Any suggestions?

[gopher_49]

It's probably not realistic to expect Veeam to be able to replicate at a very small block sizes as this would potentially have negative impact on the performance of backups in general. Trying to account for a large number of blocks has very high overhead, and it's the ability of Veeam to compress, dedupe, and manipilate blocks very quickly that provides for many of it's innovative features like Instant Restore, and fast file level recovery.

In the end, the size on disk is relatively "non-important" because the cost of disk is "cheap". However, the cost of bandwidth is still expensive in many areas. For example, I can buy an enterprise capable iSCSI storage system, with 32TB's of storage, for ~$7,000. However, my 10Mb WAN link to my remote datacenter costs me ~$5000/mo (counting the costs of the circuit on both ends).

So, what I'm hoping for is for Veeam to decrease the amount of data that actually send over the wire. For example, if I have a 256K block, and I change 1 byte, I don't want Veeam to send the entire 256K block over the WAN. Instead, I want Veeam to basically read the new 256K block, read the existing 256K block on the target, and then only send the bytes that changed in that block using something like the rsync algorithm. This will not reduce the size of the rollbacks, but will reduce the size of the data that has to traverse the WAN for small block transaction workloads like Exchange and SQL.

I have no idea what solution Veeam will actually implement, but they understand the issue, and they're an innovative group of guys, so I'm sure they'll come up with something. If they figure out a way to use small block sizes, well, that would be awesome too, but I'm just not sure how harmful that would be to the performance of other features in the product. Simply decreasing the data that has to traverse the WAN is good enough for me.

In regards to storage... I have 2 x QNAP TS-859U-RP+ Turbo NAS' and they've been good for us.. The performance is poor but just enough for us... They have redundant power supplies and support low cost SATA drives.. I wish they supported RAID 1+0 though...

[tsightler]

Since ESXi doesn't have rsync how can I replicate from my NAS appliance to a ESXi target? I guess I could place one of these NAS appliances at the remote site and store my VM's / replicas on it for the NAS appliances can replicate to each other and they can see the data volumes... But.. I'm trying to void purchasing another NAS appliance... I guess I need the ability for my Windows based rsync server to see my VMFS volumes... Any suggestions?

You definitely don't want to rsync from VMFS to VMFS because VMFS is simply not designed for this type of access and performance is poor. We basically use Linux boxes as NAS. We take a pool of storage and present it to ESX as an NFS datastore. We then replicate our VM's to the NFS datastore. Since these datastores are on Linux we then use rsync to push it to the remote datastore, which can also be seen by our remote ESX hosts. We've imported the remote copies of the VM's into the remote ESX servers, and leave them sitting there, powered off. We use Linux snapshot to protect them against disasters during the rsync process so that, if a disaster hits during an rsync we simply revert the snapshot and use the previous copies.

It's a kludge, but works quite well actually, but we're a Linux heavy shop and have the skills to script this up, might be too much if you don't have those skills on staff.

[gopher_49]

I'm using NAS appliances (Linux based) as storage for both my Veeam backups and as datavolumes for my mounted VM's. I do exactly what you're doing. I take a pool of storage and present it to my ESXi host as a NFS datastore. I guess I can create local replication jobs and replicate to these mounted NFS based datastores. I can then deploy one of my NAS appliances to my remote site and use the built in rsync replication and simply replicate to the folder that the ESXi is looking at for it's replicated VM's. I also plan to replicate all other files in the VM/replicated folder so I can simply power then on like you're doing. Will this work? Can't I also simply roll back to a different version of my replica via VEEAM instead of implementing the Linux snapshot solution you mentioned in the event of rsync failure?

[tsightler]

Can't I also simply roll back to a different version of my replica via VEEAM instead of implementing the Linux snapshot solution you mentioned in the event of rsync failure?

Since you won't be using Veeam for the remote copy, then no, you can't simply rollback to a previous replica as VEEAM will have no ideas about the remote copy. Rolling back to a previous replica would be fine for the "local" copy. Of course, if you use rsync in it's default mode, this is really a non-issue because it actually doesn't modify the existing file, but rather creates a new, hidden file, and, only once the file is completely copied does it delete the old file and rename the new file. This makes it pretty safe.

The disadvantage of this is that you need as much free space as the size of your largest VMDK to be able to finish an rsync cycle. Because some of our VMDK's are huge, we instead choose to use the "--inplace" option which, as it's name implies, modifies the file inplace.

I don't know anything about the versions of rsync that are on your NAS appliance, nor the CPU and RAM, so you would need to test this carefully. Our boxes are pretty powerful and have lots of RAM, but they're custom built, while a lot of dedicated NAS devices are underpowered. It might work fine, I just don't know.

[gopher_49]

Thanks for the information. After letting my backend server replicate a few times with the new WAN target option it seems that its replica size is right at 3GB's which is way less than before. I also disabled the daily maintenance schedule on my Exchange server. I guess the replica after the weekend will be larger but at least my other replicas will be smaller. I'm assuming most of the decrease in size is due to the WAN option. I'm excited for I can easily send 3 GB's of data over a 3 mbps internet connection per night. My other backend server is only two mailboxes and it's now 20 GB's... It does get a lot of transcations for those two mailboxes get copies of most of the companies emails... We'll see how large it gets but in the past it was about half the size of the other backend server... So... It seems that my Exchange servers will be able to replicate on a nightly basis. I'll now test my file/SQL server...

I'll update once I get a few reverse incremental replicas created of my other backend server and my file/SQL server.

Thanks for your help.

[Gostev]

I'm assuming most of the decrease in size is due to the WAN option.

Just wanted to note that changing this setting on existing job only takes effect after full backup is done (as this feature requires creating and initializing backup storage differently). Because replication jobs have no such thing as periodic or manual "full backup" (they are forever incremental), you can only set this option when creating new job (editing it will have no effect).

Changing optimization from LAN to WAN typically results in 2x smaller incremental backup files, and so 2x faster replication performance for where WAN link speed is a primary bottleneck.

[tsightler]

I'd bet most of his decrease is from removing the nightly maintenance. Nightly maintenance performs online defrags the datastore which results in huge changes. The biggest thing to remember is to make up for the missed nightly maintenance on the weekend or eventually your Exchange performance will suffer. Normally Exchange is configured for 4 hours of maintenance each night (at least, I think it's 4 hours), if it doesn't finish, it picks up where it left off the next night. If it takes the full 4 hours then you need to make sure you have at least 28 hours of "maintenance" scheduled for the weekend, or at least close. If you were getting full passes in less than 4 hours then you can have a shorter "weekend" maintenance period.

[gopher_49]

It seems that I have 11 GB's of changes total for both of my Exchange DB's... This is much smaller... I created new jobs that had the WAN target enabled from the start. That's still alot to send over 3mbps per night considering I still have to replicate my file/SQL server. I'll increase my Exchange scheduled maintenance time intervals on the weeked maintenance jobs...

Also,

Should I have my VEEAM server local or remote when running these WAN replication jobs?

Below are the results from jobs I've ran with change block tracing disabled.

Below are the results… Keep in mind the replication server is still on my LAN and is not offsite yet.

full started at 1 MB/sec went up to 2 MB/sec at 20% stayed on 2 MB/sec

reverse incremental started at 3 MB/s got up to 4 MB/sec at 8% at 25% it sped up to 5 MB/sec and at 80% it went up to 6 MB/sec

So, it ended up being 3 times the throughput on the reverse incremental job. So, the write is the culprit, correct? So, if the bottle neck is the write then I shouldn’t give up my fast read access correct? Maybe I’m confused… Where should I place my replication server?

[albert]

Hi, i have a virtual machine with win 2008 & sql 2005, it's a test, when i replicate that VM the data size is big, 2 - 3 Gb aprox, i dont understand, why i have big delta??, if i dont have changes in the virtual machine ????

thnks.

[Vitaliy S.]

Alberto,

SQL is quite a highly transactional application with many block being relocated/changed daily, so the more changed blocks you have to process, the bigger the incremental rollback will be.

By default every changed block can cause 1 MB of changes, you can reduce that number by choosing WAN target in Advanced Job Settings - Storage, please choose the parameters that suit you best:

Local Disk -- 1MB blocks
LAN Target -- 512KB blocks
WAN Target -- 256KB blocks

Thanks!