Exchange backup and replication files are huge

[gopher_49]

On my backend exchange server and my file server my backup and replication differentials seems way too large. My backend exchange server files are showing 16 GB's of changed data per day and my file server is showing 20 GB's of changed data per day. That seems way too large... My email db is only 50 GB's in total size... This just seems too large. Any suggestions or ideas? Is there a way to sync my exchange OS with one job and my db with another? Maybe that will help?! Maybe do the Exchange infostore drive with commited files nightly and OS on weekend...

[tsightler]

The Exchange server would seem to be right in line with expectations based on current Veeam technology and block size. There are many threads about this issue.

You don't say how big your fileserver is, but I'll admit that seems pretty large as, even our 1.6TB fileserver rarely has more than 20-40GB of changes. Do you happen to have any defragmenting software running on the server? Is there any process that is constantly updating and deleting files?

[Gostev]

Tom had posted good and detailed explanation about Exchange in this topic yesterday. Thanks!

[gopher_49]

interesting for I have a few tickets in the past pertaining to large backup and replication file sizes... I mentioned large block sizes for I thought maybe my virtual disks had large block sizes enabled or something.. so.. Long story short.. When will your new release be out that addresses this?

[gopher_49]

Also,

My file server is also a SQL server... I guess this makes sense too... I'm very anxious to get this resolved for my remote replication project has been on hold due to the fact that I couldn't figure out a way to get this much data across the wire... I knew something was odd about this and I was blaming it on the block size of my target volumes or maybe even the source volumes... I guess I was kinna close.. hahahaha...

[tsightler]

Yep, SQL has the same problem. Are you already using the "WAN target" option? That will give you the best solution that Veeam, by itself, can deliver today. Then you could try HyperIP, which does help a lot since SQL and Exchange data have a tendency to be very compressible.

[gopher_49]

The WAN target helped but it's still insanely large.. Sending over my mission critical data on a 3mbps connection will never happen with this size.. I'll download HyperIP.. If it's real expensive I might hold off my replication project until VEEAM has this resolved. In regards to WAN replication I just don't see how VEEAM can be used in a SQL and Exchange environment... At least not at a cost that is affordable.. I can't even get a internet connectino where I'm at that's over 3 mbps.

[gopher_49]

I called VEEAM support and they knew nothing abouy HyperIP... They also said it wasn't free nor bundled, however, below is copied and pasted from Netex.com's website?! I guess this means I get it for free for one year.. hopefully VEEAM will have this resolve before then.

- HyperIP 2Mbs subscriptions are free to Veeam Customers
- Maximum of 2 free HyperIP virtual appliances are available per customer
- Free subscriptions are valid for one year
- Offer available through July 2011

[lobo519]

HyperIP isnt cheap - we bought a 20 MB license.

But I am using it for Exchange server replication and getting pretty good results. Takes anywhere from 5 - 20 minutes to replicate each server (2 servers - separate jobs ) depending on the load (15MB pipe shared - throttling to 15MB during the day).

However I seem to being having some trouble with the Backup job of an exchange server taking 8 hours to run. seems to just kinda crap out and not really push any data or does it at a snails pace. I can see in HyperIP almost no data going across. Still looking into it though...

[tsightler]

But I am using it for Exchange server replication and getting pretty good results. Takes anywhere from 5 - 20 minutes to replicate each server (2 servers - separate jobs ) depending on the load (15MB pipe shared - throttling to 15MB during the day).

Your link appears to be much higher than his, although it's a little difficult to understand since you said 15MB throttled to 15MB, which doesn't make much sense so I'm assuming one of those is a typo. Also, do you mean 15MB (120Mb) or 15Mb? The OP said he had only a 3Mb, which I'd guess would take him around 12 hours with Veeam and no other optimization based solely on his 50GB info store size (he hasn't really supplied any other information such as log generation per day, etc). I'd guess HyperIP would cut this in half, maybe a little better, and get it to the 4-6 hour range.

We are backing up a system that at least sounds similar to the OP's. It's an Exchange 2003 server with an infostore that's just under 50GB. The system has 50, relatively heavy users, with about 50% of them using unified messaging and Blackberry devices which significantly increase the traffic load. Active archiving also increases the generated logging. We backup this server with Veeam nightly, via a 2Mb link from our Europe offices, to one of our datacenters in the US. Without WAN acceleration this would take probably 15 hours or so. With WAN acceleration we're able to get this done in about 3-4 hours.

One thing you can do is be sure to disable the nightly info store online defrag runs. We configure our Exchange servers to run online defrags only during the weekend (when normal traffic is lighter) rather than for 4-6 hours every night which I think is the default. This online defrag run simply adds to the amount of changes that have to be replicated during the week.

[gopher_49]

I have not replicated my Exchange server via the WAN option yet... Currently the replicas are 13-19 GB's in size. We plan to replicate this over a 3mbps DSL connection via a site to site 3DES tunnel... I'm lucky to get 2.6 through the tunnel for the upstream of my dual T1's is right at 3mbps... so, at 2.6mbps we can send 1.17 GB/hr. I'll perform my first replica this evening using the WAN option. Then the following day it will run again at 1am. We'll see how large the files are then...

I just got off the phone with HyperIP. They claim their application will help in regards to the transfer of the data through the internet and will have SOME affect on the file size... They said it will mainly just speeds up the transfer. Most of the performance increase happens over the network... They suggest I give it a shot via a trial key.

My maintenance schedule was set to daily. I changed it to Saturday and Sunday only... We'll see how that in conjunction with the WAN target works.

The bottom line is that VEEAM needs to backup SQL and Exchange via a smaller block size, right? The current 256 KB block size versus the 8 KB block size for SQL and Exchange is using is insane...

[tsightler]

It's probably not realistic to expect Veeam to be able to replicate at a very small block sizes as this would potentially have negative impact on the performance of backups in general. Trying to account for a large number of blocks has very high overhead, and it's the ability of Veeam to compress, dedupe, and manipilate blocks very quickly that provides for many of it's innovative features like Instant Restore, and fast file level recovery.

In the end, the size on disk is relatively "non-important" because the cost of disk is "cheap". However, the cost of bandwidth is still expensive in many areas. For example, I can buy an enterprise capable iSCSI storage system, with 32TB's of storage, for ~$7,000. However, my 10Mb WAN link to my remote datacenter costs me ~$5000/mo (counting the costs of the circuit on both ends).

So, what I'm hoping for is for Veeam to decrease the amount of data that actually send over the wire. For example, if I have a 256K block, and I change 1 byte, I don't want Veeam to send the entire 256K block over the WAN. Instead, I want Veeam to basically read the new 256K block, read the existing 256K block on the target, and then only send the bytes that changed in that block using something like the rsync algorithm. This will not reduce the size of the rollbacks, but will reduce the size of the data that has to traverse the WAN for small block transaction workloads like Exchange and SQL.

I have no idea what solution Veeam will actually implement, but they understand the issue, and they're an innovative group of guys, so I'm sure they'll come up with something. If they figure out a way to use small block sizes, well, that would be awesome too, but I'm just not sure how harmful that would be to the performance of other features in the product. Simply decreasing the data that has to traverse the WAN is good enough for me.

[gopher_49]

Either way sounds like it will work.. I look at the replication feature seperate from the backups. I'm not performing restores or file level recovery with my replica jobs.. I simply want to replicate to another host for in the event of disaster I can turn it on and go live... I'm perfeclty fine with it hammering my processors during the replica job... We'll see what they come up with but the sooner the better!

I thought they already had technology that only sent the changed blocks... I thought it only sent the part of the changed block and not the entire block... I thought it was like rsync protocol where sections of the changed blocked were sent and not the entire block.. I guess not... We'll see... I'm about to be at the phase of my virtualization project where replication will be implemented... Hopefully it won't be too long.

I'll update the ticket once my second replica job runs...

[tsightler]

I thought they already had technology that only sent the changed blocks... I thought it only sent the part of the changed block and not the entire block... I thought it was like rsync protocol where sections of the changed blocked were sent and not the entire block.. I guess not... We'll see... I'm about to be at the phase of my virtualization project where replication will be implemented... Hopefully it won't be too long.

Well, they do seem to compress the data (assuming ESX, you don't even get that with ESXi). Actually, if you have ESX server you can replicate the server locally with Veeam, and then rsync the replicated VMDK between ESX servers (using the ESX console) faster than you can replicate with Veeam natively.

It is important to note however, that nothing Veeam can do will be magic. Replication of an entire Exchange VM with a 50GB info store via a 3Mb link is still likely to take 3-4 hours because it won't be uncommon for the server to create 2-3GB of logs, and there will be 1-2GB of changes in the info store. You've never mentioned how many GB's of logs your Exchange server generates a day, but that pretty much indicates the minimum amount of data that has to be transferred no matter the block size.

[Gostev]

It is important to note however, that nothing Veeam can do will be magic.

That's right, unless of course we can come up with something similar to MP3 or JPG for your transaction logs data hehe. Although I am sure most people here would still prefer their backups to be lossless APE

Replication of an entire Exchange VM with a 50GB info store via a 3Mb link is still likely to take 3-4 hours because it won't be uncommon for the server to create 2-3GB of logs, and there will be 1-2GB of changes in the info store. You've never mentioned how many GB's of logs your Exchange server generates a day, but that pretty much indicates the minimum amount of data that has to be transferred no matter the block size.

You should keep in mind compression that we do, too... and disk blocks backing those transaction logs files usually compress extremely well. But even without compression, we have some ideas on how we could go below this minimum amount in some cases - while still being lossless, of course

Let's just wait and see how it all works out.