Comprehensive data protection for all workloads
udesai99
Lurker
Posts: 1
Liked: never
Joined: Sep 23, 2011 8:54 am
Full Name: Umang Desai
Contact:

Re: Memory performance

Post by udesai99 »

Hi,

i am having the exact same issue i orginally had 6gb ram i added 12 gb and it eats up all the memory. I have 3 file servers that are around 2TB per vm and it maxes out the memory when backups runs which causes to fail the jobs and replicas fail or helper files gets created. How do i fix this?

thanks

osin_1
Service Provider
Posts: 6
Liked: never
Joined: Nov 02, 2009 8:28 pm
Full Name: Bruiser
Contact:

Re: Memory performance

Post by osin_1 »

withanh, You can use RamMap to see what's using the memory within Windows easily enough. We have a physical Veeam proxy server that has a large amount of local disk storage to accommodate the backups. When we run Veeam backups to this local storage the memory utilisation of the server goes up to approx 95% until the Veeam backup finishes - which is much the same as what you experience.

As others have stated the Veeam processes themselves dont utilise an extreme amount of memory but the file creation/copy/transfer within Windows is what uses as much memory as possible. When a Veeam job is running, open RamMap and the key field that you will see on the main screen of RamMap is Mapped File - check the Active column. This is where your memory is being used. If you then click on the File Summary tab and sort by Total, you will see the file name of the file thats currently consuming the memory, which will most probably be "your_backup_file_name.vbk".

I hope this helps a bit and you can show your manager where and why the memory is being used.

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Memory performance

Post by withanh »

OK, I'm pretty well convinced that Veeam itself isn't using the memory, it's the cache that's doing it. I'll pursue the avenues that Tom suggested and get the MS hotfixes in place to hopefully prevent it from eating up all the RAM.

Latest graph screenshots of the memory metrics are below:
Image
Image
Image

I've been running a forced full backup on one job that I started this morning around 11. You can see how it affected the cache in the graphs. Not entirely sure exactly what it means, but there it is.
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: Memory performance

Post by withanh »

osin_1 wrote:When a Veeam job is running, open RamMap and the key field that you will see on the main screen of RamMap is Mapped File - check the Active column. This is where your memory is being used. If you then click on the File Summary tab and sort by Total, you will see the file name of the file thats currently consuming the memory, which will most probably be "your_backup_file_name.vbk".
This is great, and shows exactly what you described. my_backup.vbk is using the memory. I love those SysInternals tools, Mark Russinovich is a very smart man!
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Gostev
SVP, Product Management
Posts: 26701
Liked: 4276 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Memory performance

Post by Gostev »

Oh yes, he is... and very fun guy IRL, too. Never miss his sessions on TechEd and other conferences, they are fantastic. Also, he had published a few great books on Windows internals in the past years - those are must have for any admin!

lobo519
Expert
Posts: 315
Liked: 37 times
Joined: Sep 29, 2010 3:37 pm
Contact:

[MERGED] Backup Performance

Post by lobo519 »

I run two backup jobs in parallel (local, remote). When my jobs start there is a huge increase in memory usage - I had to allocate an additional 8gb to satisfy but it still uses almost all of it. The strange thing is that I dont see as much CPU usage as i would image. I have 4 vCPUs and I have all my jobs set to best compression level and I still only seem spikes to 50% but usually hanging between 15- 30. Does that seem normal?

My replication jobs seem to be about the same if not slower compared to v5 using a local and remote proxy both with 4vcpu

I have not submitted a support case yet - just looking for any ideas - I know support is probably swamped

Vitaliy S.
Product Manager
Posts: 24256
Liked: 1863 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Memory performance

Post by Vitaliy S. »

lobo519 wrote:I have 4 vCPUs and I have all my jobs set to best compression level and I still only seem spikes to 50% but usually hanging between 15- 30. Does that seem normal?
Yes, that means that proxy CPU resources is not a bottleneck for your replication jobs, as they are not fully saturated as might be expected with best compression level. Something else is the bottleneck - can you lookup what does the real-time statistics window reports as a bottleneck?

As for the exccesive memory consumption, please review this thread for additional details.

pizang
Enthusiast
Posts: 61
Liked: never
Joined: Aug 02, 2009 7:33 pm
Contact:

Re: Memory performance

Post by pizang »

I guess you are using 64bit system for this server. I had similar issue. It was because system cache (for disk access) tried to allocate more and more memory until it completely dries it up. Please see another great tool from sysinternals.
http://technet.microsoft.com/en-us/sysi ... s/bb897561

Set working set max to logical value and you will be fine.

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

[MERGED] Memory usage with different backup methods

Post by cffit »

I have a physical server setup for our Veeam Backup server. It has two quad core Intel CPUs and 16GB of memory. Just to start out with, I setup a local repository and did a network backup of one of our larger database servers. I set the job to do the highest compression. About 40% of the way through the backup, the server shows no more free memory.

My curiousities are..

1. Is this normal?
2. Is this a result of using the best compression option?
3. Should I add even more memory in this case, or would it not make a difference?

My processing rate is around 40MB/s. I know there are many factors and methods that can affect this rate, but what is an average rate that I should expect to get or shoot for? I have no idea if 40MB/s is great or horribe. My
CPU runs at about 35% and my network connection is around 50%. Eventually I plan to do direct SAN backups to an iSCSI SAN, and I assume that will improve my rates of transfer.

dellock6
Veeam Software
Posts: 5929
Liked: 1746 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Memory usage on backup proxy server

Post by dellock6 »

You can start some tests, first beeing to reduce compression to default and see if it runs better; compression and deduplication puts a load on server memory to handle data before it's compressed/dedupe, so the more compression you ask, more load you put on the server.
Do this test and compare to the highest compression.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2020
Veeam VMCE #1

Gostev
SVP, Product Management
Posts: 26701
Liked: 4276 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Memory usage on backup proxy server

Post by Gostev »

Compression level does not affect memory usage, don't worry about testing this. The real reason for no memory is described above in this topic.

jawaidbazyar
Lurker
Posts: 1
Liked: never
Joined: Dec 31, 2012 7:00 pm
Full Name: Jawaid Bazyar
Contact:

Re: Memory usage on backup proxy server

Post by jawaidbazyar »

I wanted to post and say that the SetSystemFileCacheSize setting did the trick. Before doing this, my proxy/repository servers would often go crazy, running out of RAM, generating lots of failed backups due to resource (RAM) exhaustion.

I set the max file cache size 4GB smaller than my physical RAM (64-bit windows 2008), to leave room for Windows, the proxy software, etc.

This seems to have resolved the routine backup failures.

As SetSystemFileCacheSize is a Windows API call, should Veeam not be setting this to something reasonable?

I used this tool:

http://www.uwe-sieber.de/ntcacheset_e.html

Gostev
SVP, Product Management
Posts: 26701
Liked: 4276 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Memory usage on backup proxy server

Post by Gostev »

Yes, we started doing this ourselves in one of the recent product releases. Long ago enough for me to forget which one exactly. So unless this functionality broke, it's there...

Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 33 guests