Exchange/SQL Replication Speeds

[tsightler]

I dont understand where the other 18GBs come from

Well, my formula for calculating Exchange replication bandwidth used by Veeam is really pretty simple:

Local Disk (1MB blocks) -- EDB size/2
LAN Target (512KB blocks) -- EDB size/3
WAN Target (256KB blocks) -- EDB size/4

Now, you've provided a little bit of conflicting information on the size of your Exchange as, in your first post you stated that your Exchange server had ~100GB used), while more recently you posted that you had 60 users with 250-500MB mailboxes, which is only ~30GB of EDB max.

If your Exchange is really using 100GB, and your using WAN target, then 20GB is just a little under my "rule of thumb" and that's about the best Veeam is going to do. You can decrease this about 90% with a WAN acceleration box that supports caching, maybe 30-50% if it supports just TCP optimization and advanced compression.

[mplep]

Yes, I definitely understand the issue here. And the technology we have implemented for the next release should address it.

Will this solution still apply when the Veeam server is offsite at the other end of the link and replicating ESXi to ESXi?

Thanks

[Gostev]

Yes.

[pmalinov]

Now, you've provided a little bit of conflicting information on the size of your Exchange as, in your first post you stated that your Exchange server had ~100GB used), while more recently you posted that you had 60 users with 250-500MB mailboxes, which is only ~30GB of EDB max.

If your Exchange is really using 100GB, and your using WAN target, then 20GB is just a little under my "rule of thumb" and that's about the best Veeam is going to do. You can decrease this about 90% with a WAN acceleration box that supports caching, maybe 30-50% if it supports just TCP optimization and advanced compression.

My Exchange Database is 30GB, but there are other files in that VM (OS, Logs etc...).
Based on my calculations, VEEAM should not take more than 3-4 hours to replicate my entire Exchange 2003 VM.
My VRB files vary (depending on use) from 10-17GB, so my replication over a 10 Mbps pipe should be between 2:30min to 4:30 min.

In my case the replication takes between 8 and 12 hours!!!

[pmalinov]

Yes, I definitely understand the issue here. And the technology we have implemented for the next release should address it.

When is the next release going to be released?

[pmalinov]

I dont know if this helps, but I noticed that VEEAM replication kind of stalls when it hits the vmdk that contains my exchange database. Yes, I have separated logs, database, swap file and OS into 4 different vmdk's so I can troubleshoot easier.

VEEAM replication hits the vmdk file containing my exchange database at about 66% and then it case hours (maybe 3-4) until it finishes the replication for that drive (about 29.5 GB exchange database).

Is that normal for VEEAM and Exchange 2003????
Do I need to run ESEUTIL & ISINTEG????
Is it possible that I have a corruption in the database or this is normal for VEEAM replication???

VEEAM support is clueless and asked me to reach out to the forums...

I am really out of options here, and do not want to buy more bandwith if it is not going to make much of a difference.

[Gostev]

This topic has in-depth explanation of this, and should answer all your questions > backup and replication files are huge

[pmalinov]

This topic has in-depth explanation of this, and should answer all your questions > backup and replication files are huge

I read that thread but that does not explain my concerns for corruption nor tells me if it is worth running ESEUTIL and ISINTEG on Exchange and be offline for 1-12 hours.

I am suspicious because of my really long replication times, especially when VEEAM hits the vmdk that contains my exchange databases. I understand that exchange makes a lot of small changes, and that VEEAM is not optimal in reading those, but even then my calculations show that it should NOT take twice as long at the calculated time for replication.

If my VRB file is 17 GB it should NOT take 12 hours to replicate over 10Mbps link.

[tsightler]

If my VRB file is 17 GB it should NOT take 12 hours to replicate over 10Mbps link.

Assuming optimal speed, that's true, however, there are many factors that might be involved. What is the latency of the link? What is your replication target (ESX or ESXi)? Are you doing push or pull replication?

[pmalinov]

Assuming optimal speed, that's true, however, there are many factors that might be involved. What is the latency of the link? What is your replication target (ESX or ESXi)? Are you doing push or pull replication?

I am not sure what the latency is, but I know that is is a brand new dedicated fiber link.
I am replicating to ESX.
Not sure what you mean by push or pull replication.

[tsightler]

I am not sure what the latency is, but I know that is is a brand new dedicated fiber link.
I am replicating to ESX.
Not sure what you mean by push or pull replication.

Doesn't really matter how new it is, it might matter what the latency is. I'd assume it's a local fiber the latency would be pretty low (our 1Gb MAN link to our facility 8 miles away sub ms latency), but if they had to backhaul to a major POP, it might still have 12-20ms of latency. Latency can have a significant impact on performance when a lot of blocks have to be changed. We replicate a few servers over a 10Mb link that links to one of our facilities 700 miles away. The latency is around 30ms and we see performance similar to what you are seeing (~4-5GB/hr).

So it's full ESX? Have you entered SSH passwords for the ESX host so that it can push the agent?

Push or Pull replication refers to wherer the Veeam server is located. If the Veeam server is on the source side, then it is pushing data to the replica, if it's on the target side, then it is pulling data from the source. In general best performance is achieved by using push replication to ESX with SSH credentials. Have you monitored the bandwidth being used during replication to see if it's topping out?

[pmalinov]

Doesn't really matter how new it is, it might matter what the latency is. I'd assume it's a local fiber the latency would be pretty low (our 1Gb MAN link to our facility 8 miles away sub ms latency), but if they had to backhaul to a major POP, it might still have 12-20ms of latency. Latency can have a significant impact on performance when a lot of blocks have to be changed. We replicate a few servers over a 10Mb link that links to one of our facilities 700 miles away. The latency is around 30ms and we see performance similar to what you are seeing (~4-5GB/hr).

So it's full ESX? Have you entered SSH passwords for the ESX host so that it can push the agent?

Push or Pull replication refers to wherer the Veeam server is located. If the Veeam server is on the source side, then it is pushing data to the replica, if it's on the target side, then it is pulling data from the source. In general best performance is achieved by using push replication to ESX with SSH credentials. Have you monitored the bandwidth being used during replication to see if it's topping out?

The ISP we use for that fiber link is getting on my nerves. I have requested access to monitor my pipe 4 weeks ago and still have no response...
From what you are saying, then we use PUSH. Our VEEAM server is on the main office and it is pushing to the replica site. I am not sure what / where do I need to enter these credentials. Is that within VEEAM?

[Vitaliy S.]

I am not sure what / where do I need to enter these credentials. Is that within VEEAM?

Yes, you can configure this in the connection settings of ESX host's properties, just expand your vCenter Server on the left pane and drill down to the host level.

[pmalinov]

Yes, you can configure this in the connection settings of ESX host's properties, just expand your vCenter Server on the left pane and drill down to the host level.

I am not really sure what exactly I need to do and what password I need to set up. I thought that once VEEAM is setup and can take backups and replications it is communicating correctly with vCenter and the hosts. Am I not right?
Could you send me or post some ste by step instructions and provide a little more clarification?

Thank you!

[pmalinov]

OK, I think I know what you are referring to:
- In VEEAM I expanded my "Servers" on the left hand
- Right clicked my hosts and checked the "Properties"
- Under "SSH Connection" Tab I do have the username and password setup there and saved.
- I noticed that the username is different for my on-office hosts and my replica site hosts (I hope that is not a big deal)
- Under "Data Transfer" Tab I have "Automatic Selection (recommended)" selected for the transfer engine.

Please correct me if I am wrong, but it looks like I have it set the correct way, right?

Thanks

[Vitaliy S.]

Yes, that's right. In case you have service console settings specified then you should already be using a small agent on the target ESX host during the replication process.

[pmalinov]

I never received any input on the replication "stall" when it reaches 65% (when it hits the vmdk file that contains the exchange DB).
Is this a normal behavior for VEEAM when it starts replicating exchange databases?
If it is not normal, then do you suggest that I should run the ESEUTIL and ISINTEG agains my exchange DB?
Thanks

[tsightler]

My opinion is I think it is "normal". You might get some improvement from running ESEUTIL, but I don't think it will be significant. Anything with changes to lots of random blocks slows down a good bit. We see this behavior with Exchange, as well as with busy transactional database servers, both SQL and Oracle. Incremental (as opposed to reverse incrementals) helps a lot because there's not so much block movement on the target, but it's still slower.

[Gostev]

The more disk changes specific VMDK has, the longer the incremental run will be processing given VMDK, so processing rate will drop noticeably if previous disks in job were system disks with little to no changes. So yes, this is normal, and not specific to Exchange - but driven solely by the amount of changed blocks that incremental run has to deal with. Thanks!

P.S. Tom keeps beating me lately!

[pmalinov]

I do realize that the amount of changes will determine the speed/rate of replication, but in my case VEEAM looks like it freezes and it does not update the time remaining or the speed/rate. For example: VEEAM will be at 66% and display 2.5 hours for the next 4 hours. Does it make sense?

[Gostev]

OK, this is definitely not "normal" and needs to be investigated with our support.

[pmalinov]

OK, this is definitely not "normal" and needs to be investigated with our support.

That is why I thought I may need to run ESEUTIL on exchange server.
BTW I have had a support ticket open with support for over a month now, and no resolution.