Host-based backup of Microsoft Hyper-V VMs.
Post Reply
mbroaders
Service Provider
Posts: 120
Liked: 8 times
Joined: May 15, 2012 9:06 am
Full Name: Martin Broaders
Contact:

High Memory Usage On Restore/Replication

Post by mbroaders »

Case ID 00454092
Server 2012 hyper-v cluster environment. Production environment on Cisco UCS C240 servers with EMC VNXe Storage. DR Environment on Dell R710 servers with a Dell Powervault MD3200i.
Issue occurs in Veeam 6.5 and Veeam 7 (with update 1).
Server 2012 has all hyper-v and failover clustering hotfixes installed linked from http://blogs.technet.com/b/askcore/arch ... fixes.aspx .
Veeam 7 is installed on a 2012 VM on both the production environment and in the DR environment.
Issue is when restoring a virtual machine the memory usage goes up by approximately 20GB which causes the production hosts to max out and make vms unresponsive. Can reproduce 100% of the time, issue occurs on both production and dr servers although as the DR servers are not as loaded memory doesnt go to 100% but still see up to 20GB increase in RAM usage. When performing a backup there is no noticeable problem on the hosts. When permorming a replication from production to DR i see memory usage on the DR side go up again.

Support have told me it is a known issue with 2012. I can see known issues with backups of CSVs but not restores. Would anyone be able to confirm what bug this is and if it is reproducible on their systems?
nefes
Veeam Software
Posts: 643
Liked: 162 times
Joined: Dec 10, 2012 8:44 am
Full Name: Nikita Efes
Contact:

Re: High Memory Usage On Restore/Replication

Post by nefes »

Martin, do you see what consumes that memory?

As far as I know, Win2012 use it's memory to hold files it copies in the system cache, so when it comes to copying large VHDs or VHDXs, they are stored in host's RAM if it has some spare space, leading to behavior you see.

You can try to reproduce it by copying some big file with Windows itself and see what happens to your host.
mbroaders
Service Provider
Posts: 120
Liked: 8 times
Joined: May 15, 2012 9:06 am
Full Name: Martin Broaders
Contact:

Re: High Memory Usage On Restore/Replication

Post by mbroaders »

Thanks for the reply. I followed the below procedure to test :

1. Copied a 60GB backup file from the backup location (an iomega nas) to the local file system of a 2012 host. Memory increased by just 2% during the restore
2. Repeated this test but used a CSV for the destination. Same results. 2% increase in memory.
3. Ran a full vm restore to the same host and same csv as 2. Memory increased by 45%.

Have been asked for log files which i am uploading at present.
nshoobridge
Lurker
Posts: 2
Liked: never
Joined: Aug 02, 2013 4:28 am
Full Name: Neils
Contact:

Re: High Memory Usage On Restore/Replication

Post by nshoobridge »

Hi,

i have exactly the same issuse. My host is using all memory when making backup. I also have the issue when I copie big files. So I think it is related to the caching mechanism of Mircrosoft. Is there a way to disable that function?
The only workaround when copying files I have to use xcopy. XCOPY with option /J (Copies files without buffering) the system uses no memory and copies the files like a charm.

But what to do when making backupjobs?

Cheers,
Neils
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: High Memory Usage On Restore/Replication

Post by foggy »

First of all, it should be verified, what processes consume memory during Veeam B&R activity (using Task Manager, for example).
nshoobridge wrote:So I think it is related to the caching mechanism of Mircrosoft. Is there a way to disable that function?
HOW TO: Manually Turn Disk Write Caching On or Off

Though the instruction is for W2008, it still applies to W2012. Note, that altering this setting is not recommended.
nshoobridge
Lurker
Posts: 2
Liked: never
Joined: Aug 02, 2013 4:28 am
Full Name: Neils
Contact:

Re: High Memory Usage On Restore/Replication

Post by nshoobridge »

Hi,

thanks for your mail. I missed to say that source an destination are iSCSI devices and not local drive. So I'm not able to disable write cache in that way.
/C
doobie
Lurker
Posts: 1
Liked: never
Joined: Jul 10, 2014 8:06 am
Full Name: Darren Stokes
Contact:

Re: High Memory Usage On Restore/Replication

Post by doobie »

Did you ever find a resolution to this issue. I have the same issue and have had a ticket outstanding with Veeam for over a month. It only does it when Veeam preallocates the space for the vhd. Doesn't happen when large files are copied.

Only difference is I'm running with direct attached storage rather than ISCSI, but need to get this sorted otherwise Veeam is no good for me.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: High Memory Usage On Restore/Replication

Post by Vitaliy S. »

Darren, can you please post your case ID? Also can you please clarify with the Task Manager what process utilizes most memory on the host?
nublet
Novice
Posts: 5
Liked: never
Joined: Jul 03, 2013 1:59 am
Full Name: Nublet
Contact:

Re: High Memory Usage On Restore/Replication

Post by nublet »

Has anyone able to fix this problem on restore high mem usage on server 2008 r2? Thanks.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: High Memory Usage On Restore/Replication

Post by foggy »

What Veeam B&R version are you using and what processes take the most memory on the host? Have you already contacted technical support?
nublet
Novice
Posts: 5
Liked: never
Joined: Jul 03, 2013 1:59 am
Full Name: Nublet
Contact:

Re: High Memory Usage On Restore/Replication

Post by nublet »

We are using Veeam 8.0
Yeah I have contacted technical support, they mention its a caching problem on the server, but not able to fix it.
Do you know if there's anything about this?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: High Memory Usage On Restore/Replication

Post by foggy »

Could you please provide your support case ID?
iroal
Lurker
Posts: 1
Liked: never
Joined: Apr 25, 2016 8:31 am
Contact:

[MERGED]: High Cache Memory Veeam HyperV

Post by iroal »

Hi.

I'm testing Veeam Backup for HyperV.

I've a problem with the Cached Memory, It grows to much and collapse my Server.

is there any option to limit the cache in Veeam Backup?

I tried the AgentReadOnlyCache set to 0, but the problem continue.

Thanks.

Image
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: High Memory Usage On Restore/Replication

Post by PTide »

Hi,

Your post has been merged into an existing topic. Please review it and make sure that it's VBR what causes high memory consumption. If that is the case then please contact support team and post your case ID here.

Thank you.
stephc_msft
Technology Partner
Posts: 24
Liked: 15 times
Joined: May 04, 2016 12:35 pm
Full Name: Stephen Cole
Contact:

Re: High Memory Usage On Restore/Replication

Post by stephc_msft »

Please confirm if you are running on Server 2012 or 2012R2 as the hyper-V host
as 2012R2 should not be having the issues to the same extent
Post Reply

Who is online

Users browsing this forum: No registered users and 26 guests