Do I need WAN Accelerator, more bandwidth or both?

[Alan_ORiordan]

Hi

I have started a continuous replication schedule on one of two database VM's to the WAN DR site. When little is changing the job completes quickly (a few minutes) transferring only 20MB or so. When it's busier it can transfer 2-3GB taking 1-3 hrs. This morning at 8am the job started and it is only a third of the way through and could take all day to finish, should a disaster happen I would only have the restore point previous to 8am. This won't be enough to satisfy the clients RPO.

Duration 3:54:01
Processing rate 855 KB/s (Network speed is 10Mbit)
Bottleneck: Network

Data: Processed 108.4GB (36%)
Read: 11.3GB
Transferred 10.9GB (1x)

Start time: 08:00:26


If I am correct in my assumption then either I need to make what is transferred over the WAN more efficient/less or do it faster. I know the Enterprise plus license has WAN accelerator technology built in and that it needs a VM at each site to cache the data. Do you think that will improve the times enough to get the replication window down to say a 15 minute window or is there just not enough bandwidth?

I still need to add a 400GB SQL VM as well as both are required for the protected financial application to be protected. Currently they are protected by log shipping.

[Vitaliy S.]

What is your current bandwidth between the sites?

[Alan_ORiordan]

10Mbit - 1MB/sec (Up and Down)

[Vitaliy S.]

In this case, I would suggest giving WAN accelerators a try. Not only it should reduce you the amount of traffic sent to the offsite location, but your jobs will sustain network connection drops (if that link is unstable).

This topic might be a good read for you as well > testing out the WAN accelerator. In addition to this, consider doing initial seeding for that 400 GBs VM, so you could start transferring incremental changes only.

[Alan_ORiordan]

Vitaliy

Is there a way to try the WAN Accelerator without buying enterprise plus licenses? I would hate to ask the customer to speed £1300 + and not improve situation enough. The In the link you sent the poster had a 150MBit link!

I will do an seed for the VM even though the backup is from last week it's better than nothing.

This morning's 8am replication sync required over 30GB transferred with little to no dedup. It took 5.5 hrs. Could I expect WAN accelerators to bring this down to under an hour on the same link?

Thanks again I appreciate it

[Vitaliy S.]

Yes, you can try that of course, just contact your sales rep and ask for a trial license key of the required edition of Veeam B&R. It's hard to say what you should expect just based on the amount of data, the content/type of the data also matters here.

I would appreciate of you could post your results of testing WAN accelerators with this backup copy job.

[tsightler]

Data: Processed 108.4GB (36%)
Read: 11.3GB
Transferred 10.9GB (1x)

I'm a little surprised that you didn't see better compression. Do you have compression enabled on the job? Normally SQL data compresses fairly well so this just kind of stood out as something unusual.

[Alan_ORiordan]

No compression set, just default settings in that regard, however this machine is not the SQL VM it is a 300GB Linux\Oracle DB server if that makes a difference?

[veremin]

Typically, by using applicable level of compression you can achieve as much as 50% of data reduction, meaning half of the current data size will be sent through existing link; thus, the a job time will be reduced.

[Alan_ORiordan]

All jobs are set to optimal (recommended). When I said none above I must have been thinking of backup jobs, should I try High?

I have installed trial key and setup a WAN accelerators on each site on the the backup repos/proxy server (All-in-one Veeam physical servers). When I try to populate the cache at the DR site it fails to populate OSW2003x86 data. It is accessing it's local repository using local storage but the backup files are from last week (imported from live).

I seem to be successfully populating the cache at the live site so far however. Do I need to populate both successfully before using the WAN Accelerators in a replication job or will it just build up over time?

[veremin]

When I said none above I must have been thinking of backup jobs, should I try High?

No need, as you're going to utilize WAN Accelerators anyway.

Do I need to populate both successfully before using the WAN Accelerators in a replication job or will it just build up over time?

You can proceed as is, the global cache will be populated over a time.

Thanks.

[Alan_ORiordan]

OK, test results so far, not very positive at the moment unfortunately:

Having not seen much improvement in replication speeds over yesterday I have discovered two likely causes:

I see this in the logs for each replication job the below message but only in jobs AFTER selecting WAN acceleration:

05/03/2015 02:01:45 :: CBT data is invalid, failing over to legacy incremental backup. No action is required, next job run should start using CBT again. If CBT data remains invalid, follow KB1113 to perform CBT reset. Usual cause is power loss

I understand what the message is telling me and that may explain why the jobs are slower. However why did it only appear after using WAN acceleration? We did suffer a power loss twice recently once on 27/2/15 and on 2/3/15 but these messages did not show up until I started using WAN acceleration.

The KB article is very descriptive I just haven't done it before and will require out of hours work unfortunately.

The second point is:

While the Transferred compression is now about 5 x higher after WAN acceleration 24x Vs 5x the job is only taking minimal data from the cache resulting in little improvement in replication time

05/03/2015 08:34:19 :: 7.6 MB transferred over network, 12.2 KB obtained from WAN Accelerator cache

This is from a small 120GB Windows VM however not the original 300GB Linux VM that I originally posted about.

Alan

[Vitaliy S.]

I understand what the message is telling me and that may explain why the jobs are slower. However why did it only appear after using WAN acceleration? We did suffer a power loss twice recently once on 27/2/15 and on 2/3/15 but these messages did not show up until I started using WAN acceleration

Maybe there was something else involved that have caused this? You can ask our technical team for assistance with logs review before and after using WAN Accelerators.

While the Transferred compression is now about 5 x higher after WAN acceleration 24x Vs 5x the job is only taking minimal data from the cache resulting in little improvement in replication time

Are these stats from the incremental job pass or full backup? Incrementals usually have unique data like any other changes on the VMs.

[veremin]

05/03/2015 08:34:19 :: 7.6 MB transferred over network, 12.2 KB obtained from WAN Accelerator cache

That global cache is likely to be populated after a while, thus, becoming more useful. Also, be aware that WAN Acceleration uses there sources for deduplication and global cache is just one of them.

[Alan_ORiordan]

Vitaliy

The jobs are incremental replication jobs.

I have opened a case Ref 00823944 to see what is going on with the invalid CBT.

Roughly, after how many sync jobs does the global cache normally provide a beneficial amount of data reduction. Today's example is the 300GB linux machine:

05/03/2015 16:13:54 :: 14.8 GB transferred over network, 55.8 MB obtained from WAN Accelerator cache

Job time 7hrs 49 mins

I have stopped this VM from sync'ing until we work through the logs

Alan

[Gostev]

05/03/2015 16:13:54 :: 14.8 GB transferred over network, 55.8 MB obtained from WAN Accelerator cache

Alan, please don't refer to this event as the statistics is not currently correct (it reflects only 1 out of 5 counters, and actually the one that usually represents no more than a few percent of savings). Change request already submitted for Patch #3 to improve this event to include data from all data reduction engines, as this has been a common support issue.

For full data reduction statistics by WAN accelerator, you may refer to the section of debug logs that will look something like this:

Code: Select all

[27.11.2014 07:00:54] <  1928> wan| ___________________________________________________________________
[27.11.2014 07:00:54] <  1928> wan| Dedup statistics:
[27.11.2014 07:00:54] <  1928> wan| Total handled size       21.0938 GB 100%
[27.11.2014 07:00:54] <  1928> wan| New data size            0.3651 GB  1%  
[27.11.2014 07:00:54] <  1928> wan| Cur disk dedup           0.0241 GB  0%  
[27.11.2014 07:00:54] <  1928> wan| Prev disk dedup          0.0659 GB  0%  
[27.11.2014 07:00:54] <  1928> wan| Prev disk no write dedup 20.5437 GB 97% 
[27.11.2014 07:00:54] <  1928> wan| Global disk dedup        0.0950 GB  0%  
[27.11.2014 07:00:54] <  1928> wan| Zero data size           0.0000 GB  0%  
[27.11.2014 07:00:54] <  1928> wan| ___________________________________________________________________

Global disk dedup counter is the only one that is currently used to show savings in the event in questions, as you can see it's impact is always minimal comparing to other counters.

[Alan_ORiordan]

Gostev

Thanks for the additional info about the debug logs, what is their default path so I can have a look?

[veremin]

Take a look Svc.VeeamWANSvc.log located in one of the following folders:

Code: Select all

    Windows 2003, XP – C:\Documents and Settings\All Users\Application Data\Veeam\Backup\

    Windows Vista, 7 - C:\Users\All Users\Veeam\Backup\

    Windows 2008/2008 R2/2012 - C:\ProgramData\Veeam\Backup\

    Linux - /var/log/VeeamBackup/

Thanks.

[Alan_ORiordan]

Here is one of last night's daily replication jobs:

Code: Select all

[05.03.2015 21:26:57] <  6292> wan| Total handled size       10.0938 GB 100%
[05.03.2015 21:26:57] <  6292> wan| New data size            0.3932 GB  3%  
[05.03.2015 21:26:57] <  6292> wan| Cur disk dedup           0.0831 GB  0%  
[05.03.2015 21:26:57] <  6292> wan| Prev disk dedup          0.0415 GB  0%  
[05.03.2015 21:26:57] <  6292> wan| Prev disk no write dedup 9.4600 GB  93% 
[05.03.2015 21:26:57] <  6292> wan| Global disk dedup        0.1159 GB  1%  
[05.03.2015 21:26:57] <  6292> wan| Zero data size           0.0000 GB  0%  

This job contains six fairly simple Windows VM's. Without WAN Acelerator's would the replication job not even compare the prev disk dedup?

[Vitaliy S.]

Without WAN Acelerator's would the replication job not even compare the prev disk dedup?

Correct. See this link for further reading. Thanks!

[Gostev]

Regular jobs are optimized for performance, all this additional processing will slow them down significantly. On the other hand, when going over WAN, we have all the time in the world to spend squeezing some extra bytes out.

[Alan_ORiordan]

So this returns me to my original query. What do I need to do to bring down a continuous replication job that may take over five hours and reserve use of the WAN accelerator (preventing other jobs using it) while in use. I need a linux/oracle VM and a windows SQL VM to be up to date every 15 minutes at all times of day as a disaster could happen anytime.

Can this be achieved with a 10Mbit connection, do you have other customers that have managed this? Or will I have to examine the reasons for the big changes/slow replication by moving database backups within the VM to another disk which is then excluded from replication?

[Vitaliy S.]

It might be possible, but as you've correctly stated the answer will depend on the amount of changes you need to transfer during each replication job run.

Or will I have to examine the reasons for the big changes/slow replication by moving database backups within the VM to another disk which is then excluded from replication?

[Alan_ORiordan]

Hmm

My trial key expired and I could not get an extended trial in time so I had to revert to my Enterprise license with no WAN accelerators:

After doing so the now direct transfer jobs DO NOT complain of invalid CBT whereas with WAN accelerator one VM was reporting invalid CBT? Why would that be?

Two VM's now sync in parallel due to not having to wait on WAN accelerator availability.

So far I have to say I prefer the environment I have without WAN accelerators, given my experience in this log. I still have the WAN accelerator service running as I was anticipating the new trial key. Will it do any harm to leave the system as is?

Will the WAN accelerator functionality just re-activate if I choose to upgrade to Enterprise plus in the future?

Thanks

[foggy]

After doing so the now direct transfer jobs DO NOT complain of invalid CBT whereas with WAN accelerator one VM was reporting invalid CBT? Why would that be?

Did it report it constantly? CBT could get repaired during previous run, so this might be just a coincidence.

So far I have to say I prefer the environment I have without WAN accelerators, given my experience in this log. I still have the WAN accelerator service running as I was anticipating the new trial key. Will it do any harm to leave the system as is?

No harm.

Will the WAN accelerator functionality just re-activate if I choose to upgrade to Enterprise plus in the future?

As far as I recall, they will need to re-calculate digests.

[Alan_ORiordan]

Yes, it was reporting on every replication (continuous schedule) for the last few days but not before that. I had previously reset cbt on the other vm's reporting it at the weekend following a power cut last month. Not sure weather to still carry out the reset?

[foggy]

You can try and contact support if it does not help.

[BradJensen]

I tried to privately message Vitaliy S. about this because I don't mean to be spamming this board, but the forum says I can't do that till I post more.

Another possibility that might work better than a WAN accelerator is to use a Windows Deduplication volume for the repository, and then replicate it with a third party product called Replacador. (I designed Replacador, so I will start ducking now.)

Replacador replicates a Windows Deduplication volume to an identical remote volume without reflating the data.

You can also replicate to an external drive, or to several destinations at once.

Basically the idea is to be able to use Windows Deduplication the same way you would use a deduplication appliance like Data Domain.

I've been testing it with Veeam Enterprise Plus and it works well.

[veremin]

Hi, Brad,

I believe that Alan has been talking more about using WAN Accelerators for replication jobs, not backup copy ones. So, the described approach of copying backup data from one dedupe repository to another would not help him a lot.

Anyway, thank you for sharing your experience with the said solution.

Thanks.