Host-based backup of VMware vSphere VMs.
emk23
Influencer
Posts: 11
Liked: never
Joined: Jul 23, 2012 1:30 am
Full Name: Eric

WAN accel not adhering to cache size limit

Post by emk23 »

Noticing our target WAN accelerator is creating an ever expanding .tmp file in VeeamWAN\Send\Genfiles\guid\

Why is it doing that?

What is the point of having a configurable WAN accel cache size maximum if the server is going to create these other files? It chewed up all the space that we defined for the parameter, now it is using up all the rest of the space.

Frustrated because we have been pounding our heads against the table trying to get a 2TB remote site backed up to our DR site with copy-job+WAN Accelerator.

We have literally been spending weeks sending USB disks back and forth, and when we finally think we are "there" we encounter a out of control temp file consuming all of the disk space on our target WAN accelerator. Job has been calculating digests for 10+ hours and before it finishes that I am sure the disk will run out, and we will be back to square one - tearing our hair out trying to get our remote sites backed up. :evil:
emk23
Influencer
Posts: 11
Liked: never
Joined: Jul 23, 2012 1:30 am
Full Name: Eric

Re: WAN accel not adhering to cache size limit

Post by emk23 »

Well, the job chewed up 40GB of disk space for no reason with these temp files and then failed at 99% with a "failed to create disk digests" error message.

Venting right now, but I am generally PO'ed with how difficult it is to get large source site VM's to my DR site using Veeam. Everything else about the product works well except copy-job+wan-accelerator.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: WAN accel not adhering to cache size limit

Post by veremin »

What is the size of the source VM data? Have you taken into account 20 GB overhead required for each 1TB of source VM data (Wan Accelerator System Requirements)? Thanks.
emk23
Influencer
Posts: 11
Liked: never
Joined: Jul 23, 2012 1:30 am
Full Name: Eric

Re: WAN accel not adhering to cache size limit

Post by emk23 »

The source site (single job) contains approximately 3TB of VM data.

Apparently we did not see the "Disk Space: Global cache size as defined by user, plus 20GB per each 1TB of source VM data."

It would be nice if this was more at the forefront of the configuration screen. :|


The real question is - how do I force the job to restart the creation of the digest data?

The target wan accel host became unstable due to lack of disk space and I had to stop the copy job.

Now that I have resolved the disk space problem and restarted the job - it seems that the job is not recreating that digest data or using the data that we seeded to the target site.
emk23
Influencer
Posts: 11
Liked: never
Joined: Jul 23, 2012 1:30 am
Full Name: Eric

Re: WAN accel not adhering to cache size limit

Post by emk23 »

I was able to trick the job to recalculate digests by setting up a temporary repository on a tertiary machine we were using to copy the seed file from USB disk to the target repository - then pull out the big vbk to a different folder, purge the folder and rescan repository on the target repository, the move the vbk back in and bring only the vbm off of the seed disk.

I then pointed the copy job back at the target repository and it is again now recalculating digests.

It would be nice if Veeam could figure a way to make it "resume" digest calculation in the event of a problem. I reviewed the logs and it stated that the changed extents were not available and it was doing a full.


Last question - if I change a copy job settings from one target WAN accelerators to a different one, both in the target data center (and the target repository does not change) is a digest recalculation done? Is the digest information stored on the WAN accelerator or the target repository?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: WAN accel not adhering to cache size limit

Post by foggy »

emk23 wrote:Last question - if I change a copy job settings from one target WAN accelerators to a different one, both in the target data center (and the target repository does not change) is a digest recalculation done? Is the digest information stored on the WAN accelerator or the target repository?
Digests are stored on the source WAN accelerator, so changing the target one will not cause their re-calculation (however target WAN cache will have to be populated using data stored in the target repository).
krismason
Influencer
Posts: 13
Liked: never
Joined: Feb 16, 2014 9:16 am
Full Name: Kris Mason
Contact:

Sizing WAN Accelerator Cache

Post by krismason »

Hi There,

Is there somewhere that provides any more information on the sizing of WAN Accelerator cache? 10GB per OS a mentioned on the window and the information about multi (branch/parent office) locations isnt really that helpful?

Thanks

Kris
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Sizing WAN Accelerator Cache

Post by Shestakov »

Hi, Kris,

You can find more info in the User Guide`s section about WAN Cache.
Here on the forum we have some relevant topics: one, two.

Also, please review system requirements and mention that additional 20GB of space is required for each 1TB of source VM data.

Do you have any specific question?

Thank you.
EdwardsCP
Influencer
Posts: 16
Liked: 2 times
Joined: May 20, 2014 8:12 pm
Full Name: Colin Edwards
Contact:

Re: WAN accel not adhering to cache size limit

Post by EdwardsCP »

Bumping an old thread, but this is relevant to an issue I'm having with WAN Cache sizing. Can I have some clarity on the best way to determine how much disk space the WAN Cache requires? At one point I was told cache should be 100 GB plus 10 GB per each different OS that's backed up. Now I've found the "plus 20 GB per each TB of VM data".

Let's say, hypothetically, you have 6 different OS's being backed up and a total of about 2.5 TB.
100 GB base-requirement for cache
plus 60 GB for OS's
plus 50 GB overhead (20 GB per TB of data)
210 GB total?

Is that an accurate way to size your WAN Accelerator disks?


Edit: this should probably be right in the B&R forum...not necessarily the vSphere subforum, but I just found the thread from a search.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: WAN accel not adhering to cache size limit

Post by veremin »

EdwardsCP wrote:Is that an accurate way to size your WAN Accelerator disks?
The correct approach is mentioned in the system requirements section of our User Guide. The OS reference has most likely to do with initial sizing (when you're planing to define global cache size in GUI), rather than with the overhead.

Thanks.
EdwardsCP
Influencer
Posts: 16
Liked: 2 times
Joined: May 20, 2014 8:12 pm
Full Name: Colin Edwards
Contact:

Re: WAN accel not adhering to cache size limit

Post by EdwardsCP »

Sorry, but I doubt I'm the only one that finds those requirements to be vague. That's why I was asking for clarification.

We followed a recommendation from support (case# 645925) and sized our WAN Cache disks for 100 GB plus 10 GB per OS. A couple of weeks later, we have run into the situation again where the Backup Copy Job can't run because there is no disk space left.

Walking through the hypothetical scenario that I gave and correcting it if I have erred in my understanding of the recommendations would have been more helpful than pointing me at a document that I already read.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: WAN accel not adhering to cache size limit

Post by foggy »

Do you have those 20 GB of free space per 1 TB of source VM data on top of the specified cache size? The recommendation is to have, for example, 100 GB cache plus additional 20 GB per 1 TB of data overhead in the cache folder (not included in the specified cache size).
randy
Influencer
Posts: 12
Liked: 1 time
Joined: Jun 08, 2011 2:05 pm
Full Name: Randy Smith
Contact:

[MERGED] WAN Cache 100GB not enough?

Post by randy »

We have 2 Backup Copy jobs that go from site A to B using WAN accelerators. Both jobs are for 3-4 VMs running a mix of 64-bit Win2008r2 and Win2012r2. We have 100GB WAN cache on each accelerator. But we are getting "Target refused global dedupe session because of growth failing. Disable global dedupe." I can allocate more cache but from the documentation it seems like 100 GB should be more than enough at 10GB per OS. Should I just increase cache to 200GB or is there another problem/solution?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: WAN accel not adhering to cache size limit

Post by foggy »

Randy, please review above for the WAN cache size recommendations.
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

[MERGED] WAN Accelerator. Sizing.

Post by nunciate »

I know there is info out there on this but it is a bit confusing.
I have a backup storage server with about 63Tb of backup data on it (VBK files total doesn't include incremental backups also on the storage).
There are about 230 VMs and I want to backup copy all of them to my DR site.

I currently have the server to be placed in DR connected here next to my primary server.
There is a single Gigbit interface connected to the DR server as that is what we will have as our WAN connection from production to DR.
I have also set throttling on the servers to not exceed 1Gbps

Now I already have all of my backup copies done using direct mode and they have been syncing up nicely over the last month or so.
Before I move the server to DR I would like to turn on WAN Acceleration.

One of my file servers has multiple 2TB drives totaling about 10Tb of storage. Maybe 8Tb used.
I turned on WAN Acceleration for the backup copy job on this server and it filled my cache and failed after a very long time. I didn't write down the times.

Both servers have a 450Gb high performance SSD drive to use for WAN Cache.
I have each WAN Accelerator set to 400Gb but apparently that isn't' enough.

Can someone simplify this for me or tell me how much cache I really need. Honestly I don't know that I even need to use acceleration since I have a gigabit connection to my DR site but every little bit helps as we will have other things going over that pipe as well.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: WAN accel not adhering to cache size limit

Post by foggy »

Alan, you definitely do not need WAN acceleration on such a fast link. It is designed for slow and busy links with little bandwidth available (when network is the real bottleneck, typically under 100 Mbps).

As to the cache size recommendations, you can find those above (though they are for v7, but still apply).
verre
Enthusiast
Posts: 67
Liked: never
Joined: Jun 30, 2014 1:51 pm
Full Name: Valentino Erre
Contact:

[MERGED] Wan acceleration and SSD cache

Post by verre »

Hi everybody, I am setting up a wan acceleration between two sites in order to accelerate the backup copy job. I know that it is necessary to have an ssd datastore in the target site in order to cache the data. I wanted to know if there are any prerequisites or best practices about the sizing and the location...is there anywhere a calculator? And moreover I need to know if it is necessary to have also an SSD cache in the primary site and if it must have the same size of the target cache. Thank you.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Wan acceleration and SSD cache

Post by Shestakov » 1 person likes this post

Hi Valentino,
verre wrote:I wanted to know if there are any prerequisites or best practices about the sizing and the location...is there anywhere a calculator?
The more space you allocate, the more repeating data blocks will be written to the global cache and the more efficient WAN acceleration will be. By default, the size of the global cache is 100 GB and it is recommended to allocate 20GB per 1TB of source data, but not less than 40 GB.
verre wrote:And moreover I need to know if it is necessary to have also an SSD cache in the primary site and if it must have the same size of the target cache.
The global cache size is specified per source WAN accelerator. That is, if you plan to use one target WAN accelerator with several source WAN accelerators, the specified amount of space will be allocated for every source WAN accelerator that will be working with the target WAN accelerator and the size of the global cache will increase proportionally.
Thanks.
verre
Enthusiast
Posts: 67
Liked: never
Joined: Jun 30, 2014 1:51 pm
Full Name: Valentino Erre
Contact:

Re: Wan acceleration and SSD cache

Post by verre »

Tnahk you.... 100G plus 20G every 1T? Is it correct?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Sizing WAN Accelerator Cache

Post by Shestakov » 1 person likes this post

Yes, 100GB + additional 20GB per 1 TB
Murigar
Enthusiast
Posts: 37
Liked: 4 times
Joined: Nov 05, 2014 4:19 pm
Contact:

[MERGED] WAN Accelerator Drive Full

Post by Murigar »

I am using a one to one WAN Accelerator. I have the backup infrastructure setup at both sites. Each site lists its self and the other.

They are configured to use 150GB on a 200GB drive.
On the target end I am seeing the GlobalCache using the 150GB. But, I have Send using 36GB.
On the source side the GlobalCache is small, ~3Gb but the Digests are 209GB.

I'm sure there are many variables but do these metrics sound correct?
Should I set this down to 100GB instead of 150?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: WAN accel not adhering to cache size limit

Post by foggy »

Please review the thread above for the cache size recommendations. For the source side please also consider additional 20 GB of free space per 1 TB of source VM data for digests.
rpedrazzini
Novice
Posts: 3
Liked: never
Joined: May 22, 2015 6:18 am
Full Name: Ruggero Pedrazzini
Contact:

[MERGED] WAN Accelerator Configuring

Post by rpedrazzini »

Good morning

My name is Ruggero Pedrazzini and work in the Municipality of Pavia, Italy.

I wanted to ask some questions about the setup of the Local and Global Cache for WAN Accelerator Veeam 8 Update 2.
Our infrastructure is will constitute to 1 cluster with 3 Host Esxi 5.5Up2 and 30 VM (Windows XP and Seven - Windows Server 2003,2008,2012 -
Linux: Debian, Centos, Ubuntu).
To determine the disk space to be allocated to the CACHE Wan Accelerator i have been using this methodology:

20 GB x 8 S.O. (5 Microsoft + 3 Linux) = 160 GB
20 GB x 12 TB (space occupied by Backup) = 240 GB

Tot Space CACHE 400 GB

Questions:

1) The space of 400 GB is to be assigned to both the LOCAL CACHE that the GLOBAL CACHE?

2) During configuration of CACHE SIZE, specifying a value 400 GB with disk space of 450 GB (vmdk), you receive the following message:
"The selected volume may have insufficient free disk space for storing digests of Transferred date.
Make sure you are meeting system requirement for free disk space available to WAN accelerator. "

How can I fix?

Thank you.

Ruggero
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: WAN accel not adhering to cache size limit

Post by Shestakov »

Hello Ruggero,
1) It`s a global cahce. Could you clarify what you mean by the local one?
2) Recommendation is to assign 100 GB + 20 more GB per 1 TB of source VM data for digests. Looks like you have 20 TB of source data. Is it correct?
Please also review the posts above. Thanks
rpedrazzini
Novice
Posts: 3
Liked: never
Joined: May 22, 2015 6:18 am
Full Name: Ruggero Pedrazzini
Contact:

Re: WAN accel not adhering to cache size limit

Post by rpedrazzini »

Hello,

GLOBAL CACHE is WAN Accelerator TARGET (WAN Accelerator DR)

LOCAL CACHE is WAN Accelerator SOURCE

The 20 TB match the disk space used by backups, while the 30 VM occupy a space of 7 TB

Thank you
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: WAN accel not adhering to cache size limit

Post by Shestakov »

Thanks for the answer, Ruggero.
We usually use terms Source WAN accelerator and Target WAN accelerator global cache.
In your case the recommended cache size is 7*20+100=240 GB
rpedrazzini
Novice
Posts: 3
Liked: never
Joined: May 22, 2015 6:18 am
Full Name: Ruggero Pedrazzini
Contact:

Re: WAN accel not adhering to cache size limit

Post by rpedrazzini »

Thanks for the answer, but i a further question


In tests I have often found the following message:
"The selected volume may have insufficient free disk space for storing digests of Transferred date.
Make sure you are meeting system requirement for free disk space available to WAN accelerator. "


I solved by configuring CACHE SIZE to 250 GB with a disk space of 500 GB VMDK thus leaving 250 GB Free.

But I can not understand why.

You could explain it?

Thank you.

Ruggero
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: WAN accel not adhering to cache size limit

Post by Shestakov »

Ruggero,

The warning appeared because the disk was running out of free space.
I guess your jobs worked well, you just had the warning to be aware of.

Thanks!
Scol
Lurker
Posts: 2
Liked: 1 time
Joined: Jun 04, 2015 1:55 pm

Re: WAN accel not adhering to cache size limit

Post by Scol »

Hi everyone,

I am have the same question as Ruggero, and there are no answers yet.
rpedrazzini wrote: In tests I have often found the following message:
"The selected volume may have insufficient free disk space for storing digests of Transferred date.
Make sure you are meeting system requirement for free disk space available to WAN accelerator. "
This message appears when selecting the cache size while configuring the WAN Accelerator.
What is the ratio used here ? Is this a false alarm ?

Case #1 If I use the following settings : WAN disk size 100GB, Global Cache size 75GB, Free Space 25GB, I don't have this message.
Case #2 If I use the following settings : WAN disk size 175GB, Global Cache size 110GB, Free Space 65GB I have this message.

In case #1 I have 25% of free space, in case #2 37%.
I just don't get it, how much free space is needed ?

Also if I understand everything well, in case I only have one source and one target, both cache size should be the same (on source and target) ?

Regards,

Raphael
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: WAN accel not adhering to cache size limit

Post by Shestakov » 1 person likes this post

Hi Raphael,

The initial logic is very straightforward: GUI should warn if the free space after WAN cache allocation is less than 20GB, no linkage to % of free space.

However the warning might work incorrectly when editing the accelerator. In your cases provided, have you just created the accelerator in the 1st case and edited in the case #2? If my guess is correct, you will have a warning if try to edit the 1st accelerator, even keeping the same numbers. I`ve sent the issue to QA team for extra-testing.
One way or the other it`s just a warning. Ignore it if you have more than 20GB of free space.

Thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 82 guests