-
- Service Provider
- Posts: 128
- Liked: 11 times
- Joined: May 15, 2012 9:06 am
- Full Name: Martin Broaders
- Contact:
High Memory Usage On Restore/Replication
Case ID 00454092
Server 2012 hyper-v cluster environment. Production environment on Cisco UCS C240 servers with EMC VNXe Storage. DR Environment on Dell R710 servers with a Dell Powervault MD3200i.
Issue occurs in Veeam 6.5 and Veeam 7 (with update 1).
Server 2012 has all hyper-v and failover clustering hotfixes installed linked from http://blogs.technet.com/b/askcore/arch ... fixes.aspx .
Veeam 7 is installed on a 2012 VM on both the production environment and in the DR environment.
Issue is when restoring a virtual machine the memory usage goes up by approximately 20GB which causes the production hosts to max out and make vms unresponsive. Can reproduce 100% of the time, issue occurs on both production and dr servers although as the DR servers are not as loaded memory doesnt go to 100% but still see up to 20GB increase in RAM usage. When performing a backup there is no noticeable problem on the hosts. When permorming a replication from production to DR i see memory usage on the DR side go up again.
Support have told me it is a known issue with 2012. I can see known issues with backups of CSVs but not restores. Would anyone be able to confirm what bug this is and if it is reproducible on their systems?
Server 2012 hyper-v cluster environment. Production environment on Cisco UCS C240 servers with EMC VNXe Storage. DR Environment on Dell R710 servers with a Dell Powervault MD3200i.
Issue occurs in Veeam 6.5 and Veeam 7 (with update 1).
Server 2012 has all hyper-v and failover clustering hotfixes installed linked from http://blogs.technet.com/b/askcore/arch ... fixes.aspx .
Veeam 7 is installed on a 2012 VM on both the production environment and in the DR environment.
Issue is when restoring a virtual machine the memory usage goes up by approximately 20GB which causes the production hosts to max out and make vms unresponsive. Can reproduce 100% of the time, issue occurs on both production and dr servers although as the DR servers are not as loaded memory doesnt go to 100% but still see up to 20GB increase in RAM usage. When performing a backup there is no noticeable problem on the hosts. When permorming a replication from production to DR i see memory usage on the DR side go up again.
Support have told me it is a known issue with 2012. I can see known issues with backups of CSVs but not restores. Would anyone be able to confirm what bug this is and if it is reproducible on their systems?
-
- Veeam Software
- Posts: 649
- Liked: 170 times
- Joined: Dec 10, 2012 8:44 am
- Full Name: Nikita Efes
- Contact:
Re: High Memory Usage On Restore/Replication
Martin, do you see what consumes that memory?
As far as I know, Win2012 use it's memory to hold files it copies in the system cache, so when it comes to copying large VHDs or VHDXs, they are stored in host's RAM if it has some spare space, leading to behavior you see.
You can try to reproduce it by copying some big file with Windows itself and see what happens to your host.
As far as I know, Win2012 use it's memory to hold files it copies in the system cache, so when it comes to copying large VHDs or VHDXs, they are stored in host's RAM if it has some spare space, leading to behavior you see.
You can try to reproduce it by copying some big file with Windows itself and see what happens to your host.
-
- Service Provider
- Posts: 128
- Liked: 11 times
- Joined: May 15, 2012 9:06 am
- Full Name: Martin Broaders
- Contact:
Re: High Memory Usage On Restore/Replication
Thanks for the reply. I followed the below procedure to test :
1. Copied a 60GB backup file from the backup location (an iomega nas) to the local file system of a 2012 host. Memory increased by just 2% during the restore
2. Repeated this test but used a CSV for the destination. Same results. 2% increase in memory.
3. Ran a full vm restore to the same host and same csv as 2. Memory increased by 45%.
Have been asked for log files which i am uploading at present.
1. Copied a 60GB backup file from the backup location (an iomega nas) to the local file system of a 2012 host. Memory increased by just 2% during the restore
2. Repeated this test but used a CSV for the destination. Same results. 2% increase in memory.
3. Ran a full vm restore to the same host and same csv as 2. Memory increased by 45%.
Have been asked for log files which i am uploading at present.
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Aug 02, 2013 4:28 am
- Full Name: Neils
- Contact:
Re: High Memory Usage On Restore/Replication
Hi,
i have exactly the same issuse. My host is using all memory when making backup. I also have the issue when I copie big files. So I think it is related to the caching mechanism of Mircrosoft. Is there a way to disable that function?
The only workaround when copying files I have to use xcopy. XCOPY with option /J (Copies files without buffering) the system uses no memory and copies the files like a charm.
But what to do when making backupjobs?
Cheers,
Neils
i have exactly the same issuse. My host is using all memory when making backup. I also have the issue when I copie big files. So I think it is related to the caching mechanism of Mircrosoft. Is there a way to disable that function?
The only workaround when copying files I have to use xcopy. XCOPY with option /J (Copies files without buffering) the system uses no memory and copies the files like a charm.
But what to do when making backupjobs?
Cheers,
Neils
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: High Memory Usage On Restore/Replication
First of all, it should be verified, what processes consume memory during Veeam B&R activity (using Task Manager, for example).
Though the instruction is for W2008, it still applies to W2012. Note, that altering this setting is not recommended.
HOW TO: Manually Turn Disk Write Caching On or Offnshoobridge wrote:So I think it is related to the caching mechanism of Mircrosoft. Is there a way to disable that function?
Though the instruction is for W2008, it still applies to W2012. Note, that altering this setting is not recommended.
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Aug 02, 2013 4:28 am
- Full Name: Neils
- Contact:
Re: High Memory Usage On Restore/Replication
Hi,
thanks for your mail. I missed to say that source an destination are iSCSI devices and not local drive. So I'm not able to disable write cache in that way.
/C
thanks for your mail. I missed to say that source an destination are iSCSI devices and not local drive. So I'm not able to disable write cache in that way.
/C
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Jul 10, 2014 8:06 am
- Full Name: Darren Stokes
- Contact:
Re: High Memory Usage On Restore/Replication
Did you ever find a resolution to this issue. I have the same issue and have had a ticket outstanding with Veeam for over a month. It only does it when Veeam preallocates the space for the vhd. Doesn't happen when large files are copied.
Only difference is I'm running with direct attached storage rather than ISCSI, but need to get this sorted otherwise Veeam is no good for me.
Only difference is I'm running with direct attached storage rather than ISCSI, but need to get this sorted otherwise Veeam is no good for me.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: High Memory Usage On Restore/Replication
Darren, can you please post your case ID? Also can you please clarify with the Task Manager what process utilizes most memory on the host?
-
- Novice
- Posts: 5
- Liked: never
- Joined: Jul 03, 2013 1:59 am
- Full Name: Nublet
- Contact:
Re: High Memory Usage On Restore/Replication
Has anyone able to fix this problem on restore high mem usage on server 2008 r2? Thanks.
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: High Memory Usage On Restore/Replication
What Veeam B&R version are you using and what processes take the most memory on the host? Have you already contacted technical support?
-
- Novice
- Posts: 5
- Liked: never
- Joined: Jul 03, 2013 1:59 am
- Full Name: Nublet
- Contact:
Re: High Memory Usage On Restore/Replication
We are using Veeam 8.0
Yeah I have contacted technical support, they mention its a caching problem on the server, but not able to fix it.
Do you know if there's anything about this?
Yeah I have contacted technical support, they mention its a caching problem on the server, but not able to fix it.
Do you know if there's anything about this?
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: High Memory Usage On Restore/Replication
Could you please provide your support case ID?
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Apr 25, 2016 8:31 am
- Contact:
-
- Product Manager
- Posts: 6551
- Liked: 765 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: High Memory Usage On Restore/Replication
Hi,
Your post has been merged into an existing topic. Please review it and make sure that it's VBR what causes high memory consumption. If that is the case then please contact support team and post your case ID here.
Thank you.
Your post has been merged into an existing topic. Please review it and make sure that it's VBR what causes high memory consumption. If that is the case then please contact support team and post your case ID here.
Thank you.
-
- Technology Partner
- Posts: 30
- Liked: 26 times
- Joined: May 04, 2016 12:35 pm
- Full Name: Stephen Cole
- Contact:
Re: High Memory Usage On Restore/Replication
Please confirm if you are running on Server 2012 or 2012R2 as the hyper-V host
as 2012R2 should not be having the issues to the same extent
as 2012R2 should not be having the issues to the same extent
Who is online
Users browsing this forum: No registered users and 13 guests