Saturated memory on my physical servers during backup

Hyper-V specific discussions

Saturated memory on my physical servers during backup

Veeam Logoby matizmat » Thu Dec 13, 2012 9:27 am

Hi,

My case id : 00166126
My environment : - Veeam Backup 6.5 / - 4 Physical server HP DL380 G7, Windows 2012 Std, CSV hyper-v 2012 / SAN EMC VNX 5300.

My problem is that when the backup is running, the memory used of my physical servers is saturated, and swap. When i take control on my server, i can see the percentage used (99%) but i can't view the real process that use all my memory, very strange. The problem appears on the physical server that owns the disk in CSV. It's not really the proxy server.
the virtual machines are very impacted by this performance issue, and i can see some I/O errors in my CSV events logs. Two times, 3 VMs were shutdown :(
if I'm not the only one, do not hesitate to reply to me !!!
matizmat
Novice
 
Posts: 8
Liked: never
Joined: Mon Nov 26, 2012 8:53 am

Re: Saturated memory on my physical servers during backup

Veeam Logoby foggy » Thu Dec 13, 2012 1:01 pm

Actually, you are not the only one - there were a couple of similar cases, which finally resulted in opening a case with Microsoft (we could reproduce this behavior internally without Veeam B&R being involved - while simply copying the VHDX file from the mounted CSV snapshot). Please continue working with support to confirm that this is the same case and to track its state. Thanks!
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Saturated memory on my physical servers during backup

Veeam Logoby Gostev » Thu Dec 13, 2012 1:25 pm

matizmat wrote:i can't view the real process that use all my memory, very strange

Check memory usage by the system cache...
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Saturated memory on my physical servers during backup

Veeam Logoby matizmat » Mon Jan 07, 2013 3:18 pm

Gostev wrote:
matizmat wrote:i can't view the real process that use all my memory, very strange

Check memory usage by the system cache...


I try to reproduce the problem in my test environment, I see that the cache is used, but not to saturate my physical memory (but have only one VM on my physical server)
In my production environment, physical memory becomes full (99%).

I see some forums with these problems but i don't know the final solution... i have found your forum and this topic (it's exactly the same problem) http://forums.veeam.com/viewtopic.php?f=2&t=8999&start=15 but i'm lost.

Here is a screenshot of my test environment (during a full backup)
Image

Do you have any idea to work around this problem
matizmat
Novice
 
Posts: 8
Liked: never
Joined: Mon Nov 26, 2012 8:53 am

Re: Saturated memory on my physical servers during backup

Veeam Logoby tsightler » Mon Jan 07, 2013 4:18 pm

You test system doesn't appear to show any problem. Perhaps you should post a screenshot of your production system when the problem is occurring. Do you have dynamic memory configured on your VMs? If so, what percentage of memory buffer do you have configured?
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Saturated memory on my physical servers during backup

Veeam Logoby matizmat » Mon Jan 07, 2013 4:45 pm

tsightler wrote:You test system doesn't appear to show any problem. Perhaps you should post a screenshot of your production system when the problem is occurring. Do you have dynamic memory configured on your VMs? If so, what percentage of memory buffer do you have configured?


Yes i have some vm's with the dynamic memory, the buffer is configured by default (20%)
i have ran a backup with the dynamic memory configured on my test environment but no saturated memory... the only difference is that i have only one vm on my server... i need to duplicate ten time my vm to force the same environment ! (take long time ...)
matizmat
Novice
 
Posts: 8
Liked: never
Joined: Mon Nov 26, 2012 8:53 am

Re: Saturated memory on my physical servers during backup

Veeam Logoby tsightler » Mon Jan 07, 2013 4:54 pm

Your test environment is unlikely to provide the same results. The problem is, when you backup lots of data, and the target that you are writing to is slowing than the processing rate, Windows will slowly use more and more cache. My guess is that, the memory usage builds until such time that it forces swapping of your VMs because they are configured to require only 20% of their configured memory. How much memory does you host have and how much memory is allocated to your VMs? How much host memory is available prior to the backup starting?

You might want to consider an off-host backup proxy.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler


Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 11 guests