Availability for the Always-On Enterprise
Post Reply
andreash
Enthusiast
Posts: 39
Liked: 5 times
Joined: Dec 04, 2013 8:13 am
Full Name: Andreas Holzhammer
Contact:

WAN Accelerator RAM requirements

Post by andreash » Feb 27, 2017 9:37 am

Hi,

I'm having a hard time sizing my source WAN-Accelerator. It eats up all RAM (32GB currently) and then starts paging like mad.

This is my setup:
8 vSphere VMs backed up in three jobs, 7.5TB in total.
virtual Veeam B&R 9.5 backup Server on Windows 2012R2 with Wan Accelerator Source (4 Cores, 32GB, 300GB Disk on a FC Array with 10k spinning disks and 4GB Write Back Cache)
Backup Proxy is on a virtual Windows 7 OS, 24GB and 4 Cores
Currently the backup server and proxy are connected with a 1GBit/s LAN, this will change later to a 50MBit/s WAN link.

My largest Job is a ~3TB Fileserver with ~300GB change rate per day. I have just upgraded the backup Server from 16GB to 32GB and watched the behaviour when the Backup Copy Job for this Fileserver kicked in:
1. all available RAM is being filled up to 100%
2. Disk usage on C: jumps to 100%, caused by pagefile.sys (starting with writing ~75MB/s, it has now settled down to read 8MB/s, write 2MB/s)
3. Backup Copy Job has startet transferring data. Processing rate is ~300KB/s, the disk is read at 15MB/s from Daily Backup source. This is an iSCSI array with 0% active time

I don't think the job will ever finish at this rate.
How much ram is needed for the source WAN Accelerator? Currently the System drive is constantly at 100% utilisation and no data is being passed to the Target. Ressource Monitor Shows ~100 page faults per second for VeeamWANSvc.exe.

I also already had to resize the WAN Cache disk twice to accomodate the Cache size on the source side, C:\VeeamWAN is now at 200GB in Digests. This seems lot to me.

Kind Regards,
Andreas

Shestakov
Veeam Software
Posts: 5970
Liked: 521 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: WAN Accelerator RAM requirements

Post by Shestakov » Feb 27, 2017 10:57 am

Hi Andreas,

General recommendation is to provide at least 8GB RAM for the accelerators.

Source WAN Accelerator requires 20 GB per 1 TB of source data to store digests of data blocks of source VM disks. Disk space consumption is dynamic and changes as unique VMs are added to (or removed from) to jobs with WAN Acceleration enabled.
Target WAN Accelerator requires global cache size as defined by user (fixed amount). Disk space is reserved immediately upon selecting WAN Accelerators as a target one in any job.

Did the job finish its first run?
First job run takes longer since digests are calculated to populate WAN cache.
Thanks!

v.Eremin
Veeam Software
Posts: 15142
Liked: 1141 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: WAN Accelerator RAM requirements

Post by v.Eremin » Feb 27, 2017 12:19 pm

This section of our Reference Architecture guide should be also useful. Thanks.

andreash
Enthusiast
Posts: 39
Liked: 5 times
Joined: Dec 04, 2013 8:13 am
Full Name: Andreas Holzhammer
Contact:

Re: WAN Accelerator RAM requirements

Post by andreash » Feb 27, 2017 2:24 pm

Hi Vladimir and Nikita,

thank you for your quick reply!

The job has now finished, at some point disk usage because of paging went down. The smaller jobs did not require as much ram as the 3TB fileserver.
I'll check at the next run with the fileserver on RAM usage.

I'm always impressed with the quality of responses on this forum!

Regards,
Andreas

Shestakov
Veeam Software
Posts: 5970
Liked: 521 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: WAN Accelerator RAM requirements

Post by Shestakov » Feb 28, 2017 9:31 pm

Nice to know, Andreas, thank you!
Next job runs should take significantly less time and give traffic savings.
You could also pre-populate cache to minimize the first job runtime.

andreash
Enthusiast
Posts: 39
Liked: 5 times
Joined: Dec 04, 2013 8:13 am
Full Name: Andreas Holzhammer
Contact:

Re: WAN Accelerator RAM requirements

Post by andreash » Mar 24, 2017 1:02 pm

Unfortunaltely the same issue again.

The Copy Job over WAN Accelerator is waiting since 2 hours to start transferring data for one of the server disks ("Hard disk 2 (1,1 TB) 0,0 B read at 0 KB/s - 2:00:15").
The Backup Job had read 466GB during the backup run.

Looking at Ressource Monitor the Server (32GB RAM) is busy with paging:
VeeamWANsvc.exe - 1MB/s read from pagefile.sys
VeeamWANsvc.exe - Page Faults: 127 - commit: 34GB - Working Set- 13GB and growing

Any idea where to look at what the server is actually doing? This is Windows 2012R2 and NTFS, if that makes any difference.

Edit:
Reconfigured WAN-Accelerator to use 5 streams instead of 10, which triggered a restart of the VeeamWAN-Service -> Disk Transfer started immediately.

Shestakov
Veeam Software
Posts: 5970
Liked: 521 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: WAN Accelerator RAM requirements

Post by Shestakov » Mar 25, 2017 9:03 pm

Thanks for the update, Andreas!
Did the first run finish or you stopped it and restarted the job?
Have you done prepopulation of cache?

DaveWatkins
Expert
Posts: 320
Liked: 85 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: WAN Accelerator RAM requirements

Post by DaveWatkins » Mar 25, 2017 11:37 pm

Is this perhaps an aspect of running it over 1Gb/s connections? Is this part of why it's not recommended to use a WAN accelerator on anything more than 100Mb/s?

Gostev
Veeam Software
Posts: 22995
Liked: 2890 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: WAN Accelerator RAM requirements

Post by Gostev » Mar 26, 2017 12:54 pm

Actually there's no such recommendation and no, connection speed is not the issue here as logic is the same regardless of available bandwidth. Something simply does not work as it should, the job should not be hanging for 2 hours - needs to be investigated with support.

StoneFritz
Service Provider
Posts: 11
Liked: 4 times
Joined: Feb 14, 2017 12:47 pm
Contact:

Re: WAN Accelerator RAM requirements

Post by StoneFritz » Dec 12, 2017 1:22 pm

Same problem here.
What is the current status?

foggy
Veeam Software
Posts: 16837
Liked: 1361 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: WAN Accelerator RAM requirements

Post by foggy » Dec 12, 2017 2:56 pm

What particular problem do you mean? Have you reviewed the links provided above?

StoneFritz
Service Provider
Posts: 11
Liked: 4 times
Joined: Feb 14, 2017 12:47 pm
Contact:

Re: WAN Accelerator RAM requirements

Post by StoneFritz » Dec 12, 2017 3:28 pm

The source WAN Accelerator RAM is growing over 20 GB.
My backup server has 32 GB RAM, often it grow until 100%.
In the job copy I got this error: "Error: Source WAN accelerator error: bad allocation"

foggy
Veeam Software
Posts: 16837
Liked: 1361 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: WAN Accelerator RAM requirements

Post by foggy » Dec 12, 2017 3:57 pm

I'd contact support for a closer look. Cannot check the OP's case, since he hasn't shared his case ID.

Post Reply

Who is online

Users browsing this forum: aldo02, Bing [Bot] and 66 guests