Availability for the Always-On Enterprise
gopher_49
Enthusiast
Posts: 39
Liked: never
Joined: Jun 26, 2010 2:27 pm
Full Name: chris h
Contact:

Exchange backup and replication files are huge

Post by gopher_49 » Dec 16, 2010 8:56 pm

On my backend exchange server and my file server my backup and replication differentials seems way too large. My backend exchange server files are showing 16 GB's of changed data per day and my file server is showing 20 GB's of changed data per day. That seems way too large... My email db is only 50 GB's in total size... This just seems too large. Any suggestions or ideas? Is there a way to sync my exchange OS with one job and my db with another? Maybe that will help?! Maybe do the Exchange infostore drive with commited files nightly and OS on weekend...

tsightler
Veeam Software
Posts: 5203
Liked: 2086 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: backup and replication files are huge

Post by tsightler » Dec 16, 2010 9:06 pm

The Exchange server would seem to be right in line with expectations based on current Veeam technology and block size. There are many threads about this issue.

You don't say how big your fileserver is, but I'll admit that seems pretty large as, even our 1.6TB fileserver rarely has more than 20-40GB of changes. Do you happen to have any defragmenting software running on the server? Is there any process that is constantly updating and deleting files?

Gostev
Veeam Software
Posts: 23189
Liked: 2955 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: backup and replication files are huge

Post by Gostev » Dec 16, 2010 9:24 pm

Tom had posted good and detailed explanation about Exchange in this topic yesterday. Thanks!

gopher_49
Enthusiast
Posts: 39
Liked: never
Joined: Jun 26, 2010 2:27 pm
Full Name: chris h
Contact:

Re: backup and replication files are huge

Post by gopher_49 » Dec 16, 2010 10:10 pm

interesting for I have a few tickets in the past pertaining to large backup and replication file sizes... I mentioned large block sizes for I thought maybe my virtual disks had large block sizes enabled or something.. so.. Long story short.. When will your new release be out that addresses this?

gopher_49
Enthusiast
Posts: 39
Liked: never
Joined: Jun 26, 2010 2:27 pm
Full Name: chris h
Contact:

Re: backup and replication files are huge

Post by gopher_49 » Dec 16, 2010 10:52 pm

Also,

My file server is also a SQL server... I guess this makes sense too... I'm very anxious to get this resolved for my remote replication project has been on hold due to the fact that I couldn't figure out a way to get this much data across the wire... I knew something was odd about this and I was blaming it on the block size of my target volumes or maybe even the source volumes... I guess I was kinna close.. hahahaha...

tsightler
Veeam Software
Posts: 5203
Liked: 2086 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: backup and replication files are huge

Post by tsightler » Dec 16, 2010 11:18 pm

Yep, SQL has the same problem. Are you already using the "WAN target" option? That will give you the best solution that Veeam, by itself, can deliver today. Then you could try HyperIP, which does help a lot since SQL and Exchange data have a tendency to be very compressible.

gopher_49
Enthusiast
Posts: 39
Liked: never
Joined: Jun 26, 2010 2:27 pm
Full Name: chris h
Contact:

Re: backup and replication files are huge

Post by gopher_49 » Dec 20, 2010 7:09 pm

The WAN target helped but it's still insanely large.. Sending over my mission critical data on a 3mbps connection will never happen with this size.. I'll download HyperIP.. If it's real expensive I might hold off my replication project until VEEAM has this resolved. In regards to WAN replication I just don't see how VEEAM can be used in a SQL and Exchange environment... At least not at a cost that is affordable.. I can't even get a internet connectino where I'm at that's over 3 mbps.

gopher_49
Enthusiast
Posts: 39
Liked: never
Joined: Jun 26, 2010 2:27 pm
Full Name: chris h
Contact:

Re: backup and replication files are huge

Post by gopher_49 » Dec 20, 2010 7:11 pm

I called VEEAM support and they knew nothing abouy HyperIP... They also said it wasn't free nor bundled, however, below is copied and pasted from Netex.com's website?! I guess this means I get it for free for one year.. hopefully VEEAM will have this resolve before then.

- HyperIP 2Mbs subscriptions are free to Veeam Customers
- Maximum of 2 free HyperIP virtual appliances are available per customer
- Free subscriptions are valid for one year
- Offer available through July 2011

lobo519
Expert
Posts: 297
Liked: 34 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: backup and replication files are huge

Post by lobo519 » Dec 20, 2010 7:30 pm

HyperIP isnt cheap - we bought a 20 MB license.

But I am using it for Exchange server replication and getting pretty good results. Takes anywhere from 5 - 20 minutes to replicate each server (2 servers - separate jobs ) depending on the load (15MB pipe shared - throttling to 15MB during the day).

However I seem to being having some trouble with the Backup job of an exchange server taking 8 hours to run. seems to just kinda crap out and not really push any data or does it at a snails pace. I can see in HyperIP almost no data going across. Still looking into it though...

tsightler
Veeam Software
Posts: 5203
Liked: 2086 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: backup and replication files are huge

Post by tsightler » Dec 20, 2010 9:07 pm

lobo519 wrote:But I am using it for Exchange server replication and getting pretty good results. Takes anywhere from 5 - 20 minutes to replicate each server (2 servers - separate jobs ) depending on the load (15MB pipe shared - throttling to 15MB during the day).
Your link appears to be much higher than his, although it's a little difficult to understand since you said 15MB throttled to 15MB, which doesn't make much sense so I'm assuming one of those is a typo. Also, do you mean 15MB (120Mb) or 15Mb? The OP said he had only a 3Mb, which I'd guess would take him around 12 hours with Veeam and no other optimization based solely on his 50GB info store size (he hasn't really supplied any other information such as log generation per day, etc). I'd guess HyperIP would cut this in half, maybe a little better, and get it to the 4-6 hour range.

We are backing up a system that at least sounds similar to the OP's. It's an Exchange 2003 server with an infostore that's just under 50GB. The system has 50, relatively heavy users, with about 50% of them using unified messaging and Blackberry devices which significantly increase the traffic load. Active archiving also increases the generated logging. We backup this server with Veeam nightly, via a 2Mb link from our Europe offices, to one of our datacenters in the US. Without WAN acceleration this would take probably 15 hours or so. With WAN acceleration we're able to get this done in about 3-4 hours.

One thing you can do is be sure to disable the nightly info store online defrag runs. We configure our Exchange servers to run online defrags only during the weekend (when normal traffic is lighter) rather than for 4-6 hours every night which I think is the default. This online defrag run simply adds to the amount of changes that have to be replicated during the week.

gopher_49
Enthusiast
Posts: 39
Liked: never
Joined: Jun 26, 2010 2:27 pm
Full Name: chris h
Contact:

Re: backup and replication files are huge

Post by gopher_49 » Dec 20, 2010 9:13 pm

I have not replicated my Exchange server via the WAN option yet... Currently the replicas are 13-19 GB's in size. We plan to replicate this over a 3mbps DSL connection via a site to site 3DES tunnel... I'm lucky to get 2.6 through the tunnel for the upstream of my dual T1's is right at 3mbps... so, at 2.6mbps we can send 1.17 GB/hr. I'll perform my first replica this evening using the WAN option. Then the following day it will run again at 1am. We'll see how large the files are then...

I just got off the phone with HyperIP. They claim their application will help in regards to the transfer of the data through the internet and will have SOME affect on the file size... They said it will mainly just speeds up the transfer. Most of the performance increase happens over the network... They suggest I give it a shot via a trial key.

My maintenance schedule was set to daily. I changed it to Saturday and Sunday only... We'll see how that in conjunction with the WAN target works.

The bottom line is that VEEAM needs to backup SQL and Exchange via a smaller block size, right? The current 256 KB block size versus the 8 KB block size for SQL and Exchange is using is insane...

tsightler
Veeam Software
Posts: 5203
Liked: 2086 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: backup and replication files are huge

Post by tsightler » Dec 20, 2010 9:31 pm 1 person likes this post

It's probably not realistic to expect Veeam to be able to replicate at a very small block sizes as this would potentially have negative impact on the performance of backups in general. Trying to account for a large number of blocks has very high overhead, and it's the ability of Veeam to compress, dedupe, and manipilate blocks very quickly that provides for many of it's innovative features like Instant Restore, and fast file level recovery.

In the end, the size on disk is relatively "non-important" because the cost of disk is "cheap". However, the cost of bandwidth is still expensive in many areas. For example, I can buy an enterprise capable iSCSI storage system, with 32TB's of storage, for ~$7,000. However, my 10Mb WAN link to my remote datacenter costs me ~$5000/mo (counting the costs of the circuit on both ends).

So, what I'm hoping for is for Veeam to decrease the amount of data that actually send over the wire. For example, if I have a 256K block, and I change 1 byte, I don't want Veeam to send the entire 256K block over the WAN. Instead, I want Veeam to basically read the new 256K block, read the existing 256K block on the target, and then only send the bytes that changed in that block using something like the rsync algorithm. This will not reduce the size of the rollbacks, but will reduce the size of the data that has to traverse the WAN for small block transaction workloads like Exchange and SQL.

I have no idea what solution Veeam will actually implement, but they understand the issue, and they're an innovative group of guys, so I'm sure they'll come up with something. If they figure out a way to use small block sizes, well, that would be awesome too, but I'm just not sure how harmful that would be to the performance of other features in the product. Simply decreasing the data that has to traverse the WAN is good enough for me.

gopher_49
Enthusiast
Posts: 39
Liked: never
Joined: Jun 26, 2010 2:27 pm
Full Name: chris h
Contact:

Re: backup and replication files are huge

Post by gopher_49 » Dec 20, 2010 9:35 pm

Either way sounds like it will work.. I look at the replication feature seperate from the backups. I'm not performing restores or file level recovery with my replica jobs.. I simply want to replicate to another host for in the event of disaster I can turn it on and go live... I'm perfeclty fine with it hammering my processors during the replica job... We'll see what they come up with but the sooner the better!

I thought they already had technology that only sent the changed blocks... I thought it only sent the part of the changed block and not the entire block... I thought it was like rsync protocol where sections of the changed blocked were sent and not the entire block.. I guess not... We'll see... I'm about to be at the phase of my virtualization project where replication will be implemented... Hopefully it won't be too long.

I'll update the ticket once my second replica job runs...

tsightler
Veeam Software
Posts: 5203
Liked: 2086 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: backup and replication files are huge

Post by tsightler » Dec 20, 2010 9:47 pm

gopher_49 wrote:I thought they already had technology that only sent the changed blocks... I thought it only sent the part of the changed block and not the entire block... I thought it was like rsync protocol where sections of the changed blocked were sent and not the entire block.. I guess not... We'll see... I'm about to be at the phase of my virtualization project where replication will be implemented... Hopefully it won't be too long.
Well, they do seem to compress the data (assuming ESX, you don't even get that with ESXi). Actually, if you have ESX server you can replicate the server locally with Veeam, and then rsync the replicated VMDK between ESX servers (using the ESX console) faster than you can replicate with Veeam natively.

It is important to note however, that nothing Veeam can do will be magic. Replication of an entire Exchange VM with a 50GB info store via a 3Mb link is still likely to take 3-4 hours because it won't be uncommon for the server to create 2-3GB of logs, and there will be 1-2GB of changes in the info store. You've never mentioned how many GB's of logs your Exchange server generates a day, but that pretty much indicates the minimum amount of data that has to be transferred no matter the block size.

Gostev
Veeam Software
Posts: 23189
Liked: 2955 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: backup and replication files are huge

Post by Gostev » Dec 20, 2010 11:14 pm

tsightler wrote:It is important to note however, that nothing Veeam can do will be magic.
That's right, unless of course we can come up with something similar to MP3 or JPG for your transaction logs data hehe. Although I am sure most people here would still prefer their backups to be lossless APE ;)
tsightler wrote:Replication of an entire Exchange VM with a 50GB info store via a 3Mb link is still likely to take 3-4 hours because it won't be uncommon for the server to create 2-3GB of logs, and there will be 1-2GB of changes in the info store. You've never mentioned how many GB's of logs your Exchange server generates a day, but that pretty much indicates the minimum amount of data that has to be transferred no matter the block size.
You should keep in mind compression that we do, too... and disk blocks backing those transaction logs files usually compress extremely well. But even without compression, we have some ideas on how we could go below this minimum amount in some cases - while still being lossless, of course :D

Let's just wait and see how it all works out.

Post Reply

Who is online

Users browsing this forum: anthony_b, christian.kotze, Exabot [Bot] and 28 guests