Comprehensive data protection for all workloads
Post Reply
Kas_Tigar
Influencer
Posts: 13
Liked: 2 times
Joined: Jan 29, 2015 9:18 am
Contact:

Calculate virtual memory needed to compact backup

Post by Kas_Tigar »

Hello,

we run into issue by compacting large full backup file (more than 20TB). Compact task failed with error like "Unable to allocate memory for storage metadata bank". The only one event in windows log at this time is the warning about virtual memory getting low. Virtual memory is controlled by the OS. So it should actually be auto increased. First I thought that OS increase virtual memory and only the space on drive C is the limitation of such increase.
But then I found the article with the hardcode limitation being 3 x RAM or 4 GB, whichever is larger
https://support.microsoft.com/en-us/hel ... of-windows

From this perspective I think If repository has 16GB RAM, virtual memory is increased during the compact job up to 48GB. If compact job is still not finished, OS is unable to increase the virtual memory and task is failed with the error "Unable to allocate memory for storage metadata bank".

The obvious workaround for this problem - assigning more RAM or manual configure virtual memory limits. But the question I have - how large should be virtual memory to compact 20TB of data? I wound experiment with size because such job takes days. So if someone has experience or know how I can calculate virtual memory needed for such task, would be great.
Thanks
JoshuaPostSAMC
Expert
Posts: 124
Liked: 22 times
Joined: Jul 30, 2015 7:32 pm
Contact:

Re: Calculate virtual memmory needed to compact backup

Post by JoshuaPostSAMC » 1 person likes this post

I don't have a specific formula, but I had the same errors when doing a compact with multiple jobs with 2-3 TB files each. I went from 16 GB to 32 GB and all of those went away. Now that I'm doing Copy Jobs as well, I just increased to 48 GB and it still got into the 90% used range. More is better
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Calculate virtual memory needed to compact backup

Post by foggy »

Such memory consumption doesn't look expected, indeed. The process performing compact requires ~150 bytes per block, which even with the smallest block possible (256 KB) gives 12 GB of RAM for 20 TB backup file. What block size is used for this job? I also recommend opening a case with our technical support for a closer investigation. Thanks.
Post Reply

Who is online

Users browsing this forum: Semrush [Bot] and 195 guests