WAN accel not adhering to cache size limit

[emk23]

Noticing our target WAN accelerator is creating an ever expanding .tmp file in VeeamWAN\Send\Genfiles\guid\

Why is it doing that?

What is the point of having a configurable WAN accel cache size maximum if the server is going to create these other files? It chewed up all the space that we defined for the parameter, now it is using up all the rest of the space.

Frustrated because we have been pounding our heads against the table trying to get a 2TB remote site backed up to our DR site with copy-job+WAN Accelerator.

We have literally been spending weeks sending USB disks back and forth, and when we finally think we are "there" we encounter a out of control temp file consuming all of the disk space on our target WAN accelerator. Job has been calculating digests for 10+ hours and before it finishes that I am sure the disk will run out, and we will be back to square one - tearing our hair out trying to get our remote sites backed up.

[emk23]

Well, the job chewed up 40GB of disk space for no reason with these temp files and then failed at 99% with a "failed to create disk digests" error message.

Venting right now, but I am generally PO'ed with how difficult it is to get large source site VM's to my DR site using Veeam. Everything else about the product works well except copy-job+wan-accelerator.

[veremin]

What is the size of the source VM data? Have you taken into account 20 GB overhead required for each 1TB of source VM data (Wan Accelerator System Requirements)? Thanks.

[emk23]

The source site (single job) contains approximately 3TB of VM data.

Apparently we did not see the "Disk Space: Global cache size as defined by user, plus 20GB per each 1TB of source VM data."

It would be nice if this was more at the forefront of the configuration screen.


The real question is - how do I force the job to restart the creation of the digest data?

The target wan accel host became unstable due to lack of disk space and I had to stop the copy job.

Now that I have resolved the disk space problem and restarted the job - it seems that the job is not recreating that digest data or using the data that we seeded to the target site.

[emk23]

I was able to trick the job to recalculate digests by setting up a temporary repository on a tertiary machine we were using to copy the seed file from USB disk to the target repository - then pull out the big vbk to a different folder, purge the folder and rescan repository on the target repository, the move the vbk back in and bring only the vbm off of the seed disk.

I then pointed the copy job back at the target repository and it is again now recalculating digests.

It would be nice if Veeam could figure a way to make it "resume" digest calculation in the event of a problem. I reviewed the logs and it stated that the changed extents were not available and it was doing a full.


Last question - if I change a copy job settings from one target WAN accelerators to a different one, both in the target data center (and the target repository does not change) is a digest recalculation done? Is the digest information stored on the WAN accelerator or the target repository?

[foggy]

Last question - if I change a copy job settings from one target WAN accelerators to a different one, both in the target data center (and the target repository does not change) is a digest recalculation done? Is the digest information stored on the WAN accelerator or the target repository?

Digests are stored on the source WAN accelerator, so changing the target one will not cause their re-calculation (however target WAN cache will have to be populated using data stored in the target repository).

[krismason]

Hi There,

Is there somewhere that provides any more information on the sizing of WAN Accelerator cache? 10GB per OS a mentioned on the window and the information about multi (branch/parent office) locations isnt really that helpful?

Thanks

Kris

[Shestakov]

Hi, Kris,

You can find more info in the User Guide`s section about WAN Cache.
Here on the forum we have some relevant topics: one, two.

Also, please review system requirements and mention that additional 20GB of space is required for each 1TB of source VM data.

Do you have any specific question?

Thank you.

[EdwardsCP]

Bumping an old thread, but this is relevant to an issue I'm having with WAN Cache sizing. Can I have some clarity on the best way to determine how much disk space the WAN Cache requires? At one point I was told cache should be 100 GB plus 10 GB per each different OS that's backed up. Now I've found the "plus 20 GB per each TB of VM data".

Let's say, hypothetically, you have 6 different OS's being backed up and a total of about 2.5 TB.
100 GB base-requirement for cache
plus 60 GB for OS's
plus 50 GB overhead (20 GB per TB of data)
210 GB total?

Is that an accurate way to size your WAN Accelerator disks?


Edit: this should probably be right in the B&R forum...not necessarily the vSphere subforum, but I just found the thread from a search.

[veremin]

Is that an accurate way to size your WAN Accelerator disks?

The correct approach is mentioned in the system requirements section of our User Guide. The OS reference has most likely to do with initial sizing (when you're planing to define global cache size in GUI), rather than with the overhead.

Thanks.

[EdwardsCP]

Sorry, but I doubt I'm the only one that finds those requirements to be vague. That's why I was asking for clarification.

We followed a recommendation from support (case# 645925) and sized our WAN Cache disks for 100 GB plus 10 GB per OS. A couple of weeks later, we have run into the situation again where the Backup Copy Job can't run because there is no disk space left.

Walking through the hypothetical scenario that I gave and correcting it if I have erred in my understanding of the recommendations would have been more helpful than pointing me at a document that I already read.

[foggy]

Do you have those 20 GB of free space per 1 TB of source VM data on top of the specified cache size? The recommendation is to have, for example, 100 GB cache plus additional 20 GB per 1 TB of data overhead in the cache folder (not included in the specified cache size).

[randy]

We have 2 Backup Copy jobs that go from site A to B using WAN accelerators. Both jobs are for 3-4 VMs running a mix of 64-bit Win2008r2 and Win2012r2. We have 100GB WAN cache on each accelerator. But we are getting "Target refused global dedupe session because of growth failing. Disable global dedupe." I can allocate more cache but from the documentation it seems like 100 GB should be more than enough at 10GB per OS. Should I just increase cache to 200GB or is there another problem/solution?

[foggy]

Randy, please review above for the WAN cache size recommendations.

[nunciate]

I know there is info out there on this but it is a bit confusing.
I have a backup storage server with about 63Tb of backup data on it (VBK files total doesn't include incremental backups also on the storage).
There are about 230 VMs and I want to backup copy all of them to my DR site.

I currently have the server to be placed in DR connected here next to my primary server.
There is a single Gigbit interface connected to the DR server as that is what we will have as our WAN connection from production to DR.
I have also set throttling on the servers to not exceed 1Gbps

Now I already have all of my backup copies done using direct mode and they have been syncing up nicely over the last month or so.
Before I move the server to DR I would like to turn on WAN Acceleration.

One of my file servers has multiple 2TB drives totaling about 10Tb of storage. Maybe 8Tb used.
I turned on WAN Acceleration for the backup copy job on this server and it filled my cache and failed after a very long time. I didn't write down the times.

Both servers have a 450Gb high performance SSD drive to use for WAN Cache.
I have each WAN Accelerator set to 400Gb but apparently that isn't' enough.

Can someone simplify this for me or tell me how much cache I really need. Honestly I don't know that I even need to use acceleration since I have a gigabit connection to my DR site but every little bit helps as we will have other things going over that pipe as well.

[foggy]

Alan, you definitely do not need WAN acceleration on such a fast link. It is designed for slow and busy links with little bandwidth available (when network is the real bottleneck, typically under 100 Mbps).

As to the cache size recommendations, you can find those above (though they are for v7, but still apply).

[verre]

Hi everybody, I am setting up a wan acceleration between two sites in order to accelerate the backup copy job. I know that it is necessary to have an ssd datastore in the target site in order to cache the data. I wanted to know if there are any prerequisites or best practices about the sizing and the location...is there anywhere a calculator? And moreover I need to know if it is necessary to have also an SSD cache in the primary site and if it must have the same size of the target cache. Thank you.

[Shestakov]

Hi Valentino,

I wanted to know if there are any prerequisites or best practices about the sizing and the location...is there anywhere a calculator?

The more space you allocate, the more repeating data blocks will be written to the global cache and the more efficient WAN acceleration will be. By default, the size of the global cache is 100 GB and it is recommended to allocate 20GB per 1TB of source data, but not less than 40 GB.

And moreover I need to know if it is necessary to have also an SSD cache in the primary site and if it must have the same size of the target cache.

The global cache size is specified per source WAN accelerator. That is, if you plan to use one target WAN accelerator with several source WAN accelerators, the specified amount of space will be allocated for every source WAN accelerator that will be working with the target WAN accelerator and the size of the global cache will increase proportionally.
Thanks.

[verre]

Tnahk you.... 100G plus 20G every 1T? Is it correct?

[Shestakov]

Yes, 100GB + additional 20GB per 1 TB

[Murigar]

I am using a one to one WAN Accelerator. I have the backup infrastructure setup at both sites. Each site lists its self and the other.

They are configured to use 150GB on a 200GB drive.
On the target end I am seeing the GlobalCache using the 150GB. But, I have Send using 36GB.
On the source side the GlobalCache is small, ~3Gb but the Digests are 209GB.

I'm sure there are many variables but do these metrics sound correct?
Should I set this down to 100GB instead of 150?

[foggy]

Please review the thread above for the cache size recommendations. For the source side please also consider additional 20 GB of free space per 1 TB of source VM data for digests.

[rpedrazzini]

Good morning

My name is Ruggero Pedrazzini and work in the Municipality of Pavia, Italy.

I wanted to ask some questions about the setup of the Local and Global Cache for WAN Accelerator Veeam 8 Update 2.
Our infrastructure is will constitute to 1 cluster with 3 Host Esxi 5.5Up2 and 30 VM (Windows XP and Seven - Windows Server 2003,2008,2012 -
Linux: Debian, Centos, Ubuntu).
To determine the disk space to be allocated to the CACHE Wan Accelerator i have been using this methodology:

20 GB x 8 S.O. (5 Microsoft + 3 Linux) = 160 GB
20 GB x 12 TB (space occupied by Backup) = 240 GB

Tot Space CACHE 400 GB

Questions:

1) The space of 400 GB is to be assigned to both the LOCAL CACHE that the GLOBAL CACHE?

2) During configuration of CACHE SIZE, specifying a value 400 GB with disk space of 450 GB (vmdk), you receive the following message:
"The selected volume may have insufficient free disk space for storing digests of Transferred date.
Make sure you are meeting system requirement for free disk space available to WAN accelerator. "

How can I fix?

Thank you.

Ruggero

[Shestakov]

Hello Ruggero,
1) It`s a global cahce. Could you clarify what you mean by the local one?
2) Recommendation is to assign 100 GB + 20 more GB per 1 TB of source VM data for digests. Looks like you have 20 TB of source data. Is it correct?
Please also review the posts above. Thanks

[rpedrazzini]

Hello,

GLOBAL CACHE is WAN Accelerator TARGET (WAN Accelerator DR)

LOCAL CACHE is WAN Accelerator SOURCE

The 20 TB match the disk space used by backups, while the 30 VM occupy a space of 7 TB

Thank you

[Shestakov]

Thanks for the answer, Ruggero.
We usually use terms Source WAN accelerator and Target WAN accelerator global cache.
In your case the recommended cache size is 7*20+100=240 GB

[rpedrazzini]

Thanks for the answer, but i a further question


In tests I have often found the following message:
"The selected volume may have insufficient free disk space for storing digests of Transferred date.
Make sure you are meeting system requirement for free disk space available to WAN accelerator. "


I solved by configuring CACHE SIZE to 250 GB with a disk space of 500 GB VMDK thus leaving 250 GB Free.

But I can not understand why.

You could explain it?

Thank you.

Ruggero

[Shestakov]

Ruggero,

The warning appeared because the disk was running out of free space.
I guess your jobs worked well, you just had the warning to be aware of.

Thanks!

[Scol]

Hi everyone,

I am have the same question as Ruggero, and there are no answers yet.

In tests I have often found the following message:
"The selected volume may have insufficient free disk space for storing digests of Transferred date.
Make sure you are meeting system requirement for free disk space available to WAN accelerator. "

This message appears when selecting the cache size while configuring the WAN Accelerator.
What is the ratio used here ? Is this a false alarm ?

Case #1 If I use the following settings : WAN disk size 100GB, Global Cache size 75GB, Free Space 25GB, I don't have this message.
Case #2 If I use the following settings : WAN disk size 175GB, Global Cache size 110GB, Free Space 65GB I have this message.

In case #1 I have 25% of free space, in case #2 37%.
I just don't get it, how much free space is needed ?

Also if I understand everything well, in case I only have one source and one target, both cache size should be the same (on source and target) ?

Regards,

Raphael

[Shestakov]

Hi Raphael,

The initial logic is very straightforward: GUI should warn if the free space after WAN cache allocation is less than 20GB, no linkage to % of free space.

However the warning might work incorrectly when editing the accelerator. In your cases provided, have you just created the accelerator in the 1st case and edited in the case #2? If my guess is correct, you will have a warning if try to edit the 1st accelerator, even keeping the same numbers. I`ve sent the issue to QA team for extra-testing.
One way or the other it`s just a warning. Ignore it if you have more than 20GB of free space.

Thanks!