Exchange/SQL Replication Speeds

[pmalinov]

I have the feeling that VEEAM hadles change blocks differently on VMs that have SQL instance or Exchange running.

I have:
- domain controller VM that is 50GB and that one takes 8-9 min to replicate over 10mbps pipe (Great)
- Application VM that is 350GB and i takes 30 minutes to complete ver 10mbps pipe (Great)

- SharePoint VM that is 60GB and that takes 1.40 hours to replicate over 10mbps pipe (NOT Good considering that our intranet does not get many hits and it is pretty static on content)
- Exchange VM that is 368GB total size, but actual used is about 100GB takes 8 hours ver 10mbps pipe (CRAZY NOT GOOD!)

Any thoughts on why SQl and Exchange VMs take so long?
How does VEEAM handle changed blocks on database servers like SQl & Exchange?
What would be a good replication setup (example) for those database VMs?

Currently all my jobs are separate for each VM, but I read that I may be better off by combining them. How do I combine them without breaking them or having to re-seed to USB drive. Some of my other VMs are pretty large and it will take days to re-seed.

Thank you!

[Vitaliy S.]

Hello Peter,

The processing rate may differ from one VM to another due to a number of empty blocks on the virtual disks etc. The reason why you see such a difference between your jobs is that for highly transactional server there is much more incremental data to process (not to transmit). This is caused by the fact that these applications use transaction logs which change a lot of disk blocks, and all these blocks have to be processed with an incremental pass.

By the way, do all VMs reside on the same datastore? Do you use the same destination host for replication?

Combining those jobs won't bring you significant benefits, as de-duplication and compression are not used for replication (doesn't relate to VRB files), as an original, unchanged .vmdk should be presented to target host. So I would stick to the job setup you have now.

Thank you!

[tsightler]

Well, Veeam handles them the same way, but unfortunately that way simply doesn't work as well for systems with lots of small random updates. By default Veeam uses 1MB blocks, which means that if a single 2K block is changed in a database, the entire 1MB block is replicated. If you choose "WAN target" you can shrink the block size to 256K, which is much better, but there's still a lot of wasted data (my experience with caching WAN accelerators suggests that 70+% of the data replicated when replicating Exchange or SQL is duplicated data)

Unfortunately WAN target is the best you can get for now. I'm hoping Veeam will implement improved replication technology in the future that either uses smaller block sizes, or at least eliminates sending duplicate data over and over. For now, the best solution that I'm aware of is to pair Veeam with WAN acceleration.

[lobo519]

I am replicating two Exchange servers in separate jobs running continuously.
Both are 200gb.
Not clearing the transaction logs which I found increased my processing rate.
Going to different hosts but the same datastore.
Over a 15MB pipe using Hyperip WAN acceleration
Normal usage it takes about 10-15 minutes to replicate. If its a heavy day - 20-30 minutes.

[pmalinov]

We have 2 ESX hosts with 3 datastores on each.
The replication jobs are spread over the hosts and the different datastores.

On the SAN at the company where the original (live) vms are located we have a Dell EQ array with multiple LUNs carved out.

[pmalinov]

I am replicating two Exchange servers in separate jobs running continuously.
Both are 200gb.
Not clearing the transaction logs which I found increased my processing rate.
Going to different hosts but the same datastore.
Over a 15MB pipe using Hyperip WAN acceleration
Normal usage it takes about 10-15 minutes to replicate. If its a heavy day - 20-30 minutes.

How do I setup the "Not clearing the transaction logs which I found increased my processing rate."?
Is that an option somewhere within VEEAM?
Will I have to run a complete backup after I adjust that option?

Thanks!

[lobo519]

assuming your running v5 - its in the job properties on the Replica Consistency screen. Click advanced, select the VM and click edit, "Do not truncate logs". Do you have another backup running or something else to truncate the exchange logs though?

You shouldn't have to do anything after change that option.

[pmalinov]

assuming your running v5 - its in the job properties on the Replica Consistency screen. Click advanced, select the VM and click edit, "Do not truncate logs". Do you have another backup running or something else to truncate the exchange logs though?

You shouldn't have to do anything after change that option.

I found the option but I am a little uncertain on how/when to select the "Do not truncate logs".

If I am replicating throughout the day and backing up those same VMs at night, will that cause any issues?

Example:
My exchange VM backup runs after office hours at night and that backup job has the "Truncate logs on successful backup only" selected.
Will it be OK to enable the "Do not truncate logs" option for my replication job for that VM (replication will run during the day starting after all my backups are completed)?

Thank you!

[lobo519]

This is exactly how I have mine setup. One of the product managers can probably comment on this but yes this should be fine.

I am using the replica's in a site DR scenario. In a recovery scenario (DB corruption) - I want to pull the DB from last nights backup and apply my transaction logs up until the point of DB corruption. If my replication job was truncating the logs all day - this becomes a problem. which is why my replica jobs do not truncate the logs. I only truncate the logs after a successful backup on the nightly.

[pmalinov]

This is exactly how I have mine setup. One of the product managers can probably comment on this but yes this should be fine.

I am using the replica's in a site DR scenario. In a recovery scenario (DB corruption) - I want to pull the DB from last nights backup and apply my transaction logs up until the point of DB corruption. If my replication job was truncating the logs all day - this becomes a problem. which is why my replica jobs do not truncate the logs. I only truncate the logs after a successful backup on the nightly.

Great! So, do I need to run a full replication/seed when I make that change or I should be able to make the change and continue with replications?

[lobo519]

should just be able to make the change and it just wont do the truncate function

[pmalinov]

I will try that and see how it changes (hopefully reduce) the replication time.
Thank you for all your suggestions

[Gostev]

This is exactly how I have mine setup. One of the product managers can probably comment on this but yes this should be fine.

Yes, I can confirm that this is absolutely correct implementation - processing transactions log during daily backups only, and have periodic replication jobs not touching those. Thanks!

[Bunce]

If using XGE 2010, I can't really see the need for replication jobs to DR sites (assuming you have a live link) - a DAG will do the job for you.

XGE Backups - certainly, and if you need offsite retention then you need a way of getting the backups offsite, but it seems the going concensus is to do you local backup jobs, and get them offsite using rSync or SAN replication, rather than replication jobs..

[pmalinov]

I tried replicating with the "Do not truncate logs" option checked and it did not seem to help if at all.
I noticed that my Exchange replication runs at a pretty good rate until it reaches the L:\ drive, which is only hosting logs. The logs total a size of 1.2GB.
It kind of gets stuck at about 50-60% of the replication for hours - could it be going through those log files?
Is there a way that I can exclude that drive from replicating?

[Gostev]

Yep, this is to be expected - initially replication goes through mostly unchanged disks since last replication (which is why it is very fast), but then it hits transaction logs volume, and those logs do make a lot of disk blocks dirty on every day (because a lot of logs is being written on active Exchange server), so now Veeam has to pick up and send over a lot of changed blocks. Sure, you can exclude specific virtual disks from replication with Veeam, but I am not quite sure it is okay to do in case of Exchange.

[pmalinov]

I tried replicating with the "Do not truncate logs" option checked and it did not seem to help if at all.
I noticed that my Exchange replication runs at a pretty good rate until it reaches the L:\ drive, which is only hosting logs. The logs total a size of 1.2GB.
It kind of gets stuck at about 50-60% of the replication for hours - could it be going through those log files?
Is there a way that I can exclude that drive from replicating?

I checked and those logs are from the same day, so my guess is that I cant really delete them
Any suggestions?
Is that "I noticed that my Exchange replication runs at a pretty good rate until it reaches the L:\ drive, which is only hosting logs. The logs total a size of 1.2GB.
It kind of gets stuck at about 50-60% of the replication for hours - could it be going through those log files?
" behavior normal for veeam?

[Gostev]

Yes, this is normal behavior for any image-level backup/replication. The more changed blocks you have to process, the longer the incremental pass will run.

[pmalinov]

so I guess it is not the size that slows down veeam replications, but the ammount fo log files?

[Gostev]

Yes, first it is amount of logs (so really, agent-based replication will not be much different if run on the same schedule). And secondly, how those logs are written - if the volume is severly fragmented, 1MB of logs can make 10MB of disk blocks "dirty". So, for the latter issue it may worth trying to defragment the volume, and see if it helps (just beware that first incremental after replication will be huge for obvious reasons).

[pmalinov]

Will give that a try and report back

[pmalinov]

Defrag did not change a lot except for the fact that it took a really long time to replicate afterwards (like you mentioned).

[pmalinov]

File servers are replicating with decent speeds, but we still struggle with Exchange & SQL.
I wonder if someone else out there can chime-in and share their experience with SQL & Exchange server replication.

I understand that there are a lot of factors that can play a role, but at this point I am simply looking for others experiences and try to figure out if our Exchange and SQL are messed up or if this is a VEEAM limitation.

Our set up is:
1) VEEAM 5.01
2) SQL Server 2005 on windows server 2003
3) Exchange 2003 on server 2003
4) Replicating over to a DR site that is 20 miles from our Office
5) Replication pipe is fiber 10 mbps
6) Exchange & SQL can take anywhere from 4 to 8 hours EACH to replicate
7) A pure file server takes 10-20 minutes over the same pipe (10 mbps)

If you can share your experience I would appreciate it!

VEEAM: Any chances that you will change the replication to see changed block differently? Maybe on the bit level?

[Gostev]

VEEAM: Any chances that you will change the replication to see changed block differently? Maybe on the bit level?

The fact that deduplication did not help you shows that current block size is already fine. If there is say 4GB of new transaction log files created by Exchange every day, it does not really matter what block size you will use to move this data. You will still need to pipe those 4GB through WAN, even with bit-level change processing. And may be 4.5 GB with 256KB block such as today so no big difference. But there is huge processing overhead going much lower than 256KB with our current engine.

[pmalinov]

The fact that deduplication did not help you shows that current block size is already fine. If there is say 4GB of new transaction log files created by Exchange every day, it does not really matter what block size you will use to move this data. You will still need to pipe those 4GB through WAN, even with bit-level change processing. And may be 4.5 GB with 256KB block such as today so no big difference. But there is huge processing overhead going much lower than 256KB with our current engine.

Gostev - my VRB files for exchange are pushing 20GBs
We only have 60 users and mailbox size is between 250-500MBs
My exchange logs at the end of the day are about 1 - 1.5GBs

I dont understand where the other 18GBs come from

[Gostev]

May be you should open a support case and let our technical staff review full logs... now when you say actual numbers, they do not make any sense for me either for this kind of Exchange server and transaction logs load (especially after defrag).

[tsightler]

The fact that deduplication did not help you shows that current block size is already fine. If there is say 4GB of new transaction log files created by Exchange every day, it does not really matter what block size you will use to move this data. You will still need to pipe those 4GB through WAN, even with bit-level change processing. And may be 4.5 GB with 256KB block such as today so no big difference. But there is huge processing overhead going much lower than 256KB with our current engine.

I don't know if this is possible, but I agree, and disagree with you all at the time time. Exchange really has two components, logs and edb files. For logs, assuming you're filesystem is relatively unfragmented, well, your right, you just have to push them and block size won't matter much, but my experience says that for every GB of logs, there's likely to be a GB of random changes to the EDB files. These are all small changes spread all over the place, simple attribute (like read mail) updates on objects, etc. Veeam balloons these changes like crazy.

For example, our Exchange server generates on average about 8GB of logs for a 24 hours period. Our EDB size is around 400GB. How much data does Veeam push during a typical backup or replication? Around 75GB (prior to the smaller block size it was 150GB). I can promise you there were not 75GB. So, take 75GB, subtract 8GB, that's 67GB of changes to the EBD file? Is there really anywhere near that much change? NO WAY!

Prior to Veeam we replicated our Exchange server with storage array level replication, one product used 256K block, and the other used 4K blocks. The difference between the two was dramatic. With the solution that used 4K blocks, we were able to replicate our Exchange over a 3Mb link and keep it near real time. With the 256K block solution we had to move to replicating only every hour and we increased our bandwidth to 10Mb. When we first implemented Veeam, it was saturating our 10Mb connection and we went to replicating only at night. We eventually implemented WAN acceleration, which typically makes about a 95% reduction in the WAN traffic for a Exchange replication with Veeam 4. Now that Veeam 5 has 256K blocks, combined with WAN acceleration, we're back to reasonable replication speeds, although still not great.

We're actually considering going back to storage array based replication because of this issue. Sure, it costs a lot, but compared to the monthly cost of adding bandwidth it should pay for itself fairly quickly.

[Gostev]

Oh, I did not know about EDB files being affected too (not just transaction logs). That explains it then.

[tsightler]

May be you should open a support case and let our technical staff review full logs... now when you say actual numbers, they do not make any sense for me either for this kind of Exchange server and transaction logs load (especially after defrag).

Those 18GB come from the small updates to the EDB files. Typical Exchange uses 8K blocks and those blocks are changed in a very random pattern spread across the EDB files as mail is read, deleted, calendars are updated, etc, etc. Even using 256K blocks with Veeam, the size of the changes will be huge (32x bigger than the actual on disk changes). This is why WAN acceleration helps so much, because the WAN accelerator only sends the actual changed data, while Veeam takes a 256K block and sends all of it again, even if only a single byte changes in the block (say the "read" flag of an email). The WAN accelerator caching compensates for this by sending on the bytes that changed over the WAN.

We see this issue across the board with Veeam and "database" style loads. SQL, Oracle, Exchange, it doesn't matter, Veeam's large blocks are simply too big. If Veeam would transfer only the changed data in the block, that's the solution to the problem. The higher CPU overhead in this scenario doesn't matter because, since the WAN is the limiting factor, there's CPU cycles to burn.

Veeam has to address this issue to become a real player in the replication space.

[Gostev]

Yes, I definitely understand the issue here. And the technology we have implemented for the next release should address it.