High Memory Usage On Restore/Replication

[mbroaders]

Case ID 00454092
Server 2012 hyper-v cluster environment. Production environment on Cisco UCS C240 servers with EMC VNXe Storage. DR Environment on Dell R710 servers with a Dell Powervault MD3200i.
Issue occurs in Veeam 6.5 and Veeam 7 (with update 1).
Server 2012 has all hyper-v and failover clustering hotfixes installed linked from http://blogs.technet.com/b/askcore/arch ... fixes.aspx .
Veeam 7 is installed on a 2012 VM on both the production environment and in the DR environment.
Issue is when restoring a virtual machine the memory usage goes up by approximately 20GB which causes the production hosts to max out and make vms unresponsive. Can reproduce 100% of the time, issue occurs on both production and dr servers although as the DR servers are not as loaded memory doesnt go to 100% but still see up to 20GB increase in RAM usage. When performing a backup there is no noticeable problem on the hosts. When permorming a replication from production to DR i see memory usage on the DR side go up again.

Support have told me it is a known issue with 2012. I can see known issues with backups of CSVs but not restores. Would anyone be able to confirm what bug this is and if it is reproducible on their systems?

[nefes]

Martin, do you see what consumes that memory?

As far as I know, Win2012 use it's memory to hold files it copies in the system cache, so when it comes to copying large VHDs or VHDXs, they are stored in host's RAM if it has some spare space, leading to behavior you see.

You can try to reproduce it by copying some big file with Windows itself and see what happens to your host.

[mbroaders]

Thanks for the reply. I followed the below procedure to test :

1. Copied a 60GB backup file from the backup location (an iomega nas) to the local file system of a 2012 host. Memory increased by just 2% during the restore
2. Repeated this test but used a CSV for the destination. Same results. 2% increase in memory.
3. Ran a full vm restore to the same host and same csv as 2. Memory increased by 45%.

Have been asked for log files which i am uploading at present.

[nshoobridge]

Hi,

i have exactly the same issuse. My host is using all memory when making backup. I also have the issue when I copie big files. So I think it is related to the caching mechanism of Mircrosoft. Is there a way to disable that function?
The only workaround when copying files I have to use xcopy. XCOPY with option /J (Copies files without buffering) the system uses no memory and copies the files like a charm.

But what to do when making backupjobs?

Cheers,
Neils

[foggy]

First of all, it should be verified, what processes consume memory during Veeam B&R activity (using Task Manager, for example).

So I think it is related to the caching mechanism of Mircrosoft. Is there a way to disable that function?

HOW TO: Manually Turn Disk Write Caching On or Off

Though the instruction is for W2008, it still applies to W2012. Note, that altering this setting is not recommended.

[nshoobridge]

Hi,

thanks for your mail. I missed to say that source an destination are iSCSI devices and not local drive. So I'm not able to disable write cache in that way.
/C

[doobie]

Did you ever find a resolution to this issue. I have the same issue and have had a ticket outstanding with Veeam for over a month. It only does it when Veeam preallocates the space for the vhd. Doesn't happen when large files are copied.

Only difference is I'm running with direct attached storage rather than ISCSI, but need to get this sorted otherwise Veeam is no good for me.

[Vitaliy S.]

Darren, can you please post your case ID? Also can you please clarify with the Task Manager what process utilizes most memory on the host?

[nublet]

Has anyone able to fix this problem on restore high mem usage on server 2008 r2? Thanks.

[foggy]

What Veeam B&R version are you using and what processes take the most memory on the host? Have you already contacted technical support?

[nublet]

We are using Veeam 8.0
Yeah I have contacted technical support, they mention its a caching problem on the server, but not able to fix it.
Do you know if there's anything about this?

[foggy]

Could you please provide your support case ID?

[iroal]

Hi.

I'm testing Veeam Backup for HyperV.

I've a problem with the Cached Memory, It grows to much and collapse my Server.

is there any option to limit the cache in Veeam Backup?

I tried the AgentReadOnlyCache set to 0, but the problem continue.

Thanks.

[PTide]

Hi,

Your post has been merged into an existing topic. Please review it and make sure that it's VBR what causes high memory consumption. If that is the case then please contact support team and post your case ID here.

Thank you.

[stephc_msft]

Please confirm if you are running on Server 2012 or 2012R2 as the hyper-V host
as 2012R2 should not be having the issues to the same extent