Saturated memory on my physical servers during backup

[matizmat]

Hi,

My case id : 00166126
My environment : - Veeam Backup 6.5 / - 4 Physical server HP DL380 G7, Windows 2012 Std, CSV hyper-v 2012 / SAN EMC VNX 5300.

My problem is that when the backup is running, the memory used of my physical servers is saturated, and swap. When i take control on my server, i can see the percentage used (99%) but i can't view the real process that use all my memory, very strange. The problem appears on the physical server that owns the disk in CSV. It's not really the proxy server.
the virtual machines are very impacted by this performance issue, and i can see some I/O errors in my CSV events logs. Two times, 3 VMs were shutdown
if I'm not the only one, do not hesitate to reply to me !!!

[foggy]

Actually, you are not the only one - there were a couple of similar cases, which finally resulted in opening a case with Microsoft (we could reproduce this behavior internally without Veeam B&R being involved - while simply copying the VHDX file from the mounted CSV snapshot). Please continue working with support to confirm that this is the same case and to track its state. Thanks!

[Gostev]

i can't view the real process that use all my memory, very strange

Check memory usage by the system cache...

[matizmat]

Check memory usage by the system cache...

I try to reproduce the problem in my test environment, I see that the cache is used, but not to saturate my physical memory (but have only one VM on my physical server)
In my production environment, physical memory becomes full (99%).

I see some forums with these problems but i don't know the final solution... i have found your forum and this topic (it's exactly the same problem) http://forums.veeam.com/viewtopic.php?f ... 9&start=15 but i'm lost.

Here is a screenshot of my test environment (during a full backup)


Do you have any idea to work around this problem

[tsightler]

You test system doesn't appear to show any problem. Perhaps you should post a screenshot of your production system when the problem is occurring. Do you have dynamic memory configured on your VMs? If so, what percentage of memory buffer do you have configured?

[matizmat]

You test system doesn't appear to show any problem. Perhaps you should post a screenshot of your production system when the problem is occurring. Do you have dynamic memory configured on your VMs? If so, what percentage of memory buffer do you have configured?

Yes i have some vm's with the dynamic memory, the buffer is configured by default (20%)
i have ran a backup with the dynamic memory configured on my test environment but no saturated memory... the only difference is that i have only one vm on my server... i need to duplicate ten time my vm to force the same environment ! (take long time ...)

[tsightler]

Your test environment is unlikely to provide the same results. The problem is, when you backup lots of data, and the target that you are writing to is slowing than the processing rate, Windows will slowly use more and more cache. My guess is that, the memory usage builds until such time that it forces swapping of your VMs because they are configured to require only 20% of their configured memory. How much memory does you host have and how much memory is allocated to your VMs? How much host memory is available prior to the backup starting?

You might want to consider an off-host backup proxy.