Memory usage on backup proxy server

[udesai99]

Hi,

i am having the exact same issue i orginally had 6gb ram i added 12 gb and it eats up all the memory. I have 3 file servers that are around 2TB per vm and it maxes out the memory when backups runs which causes to fail the jobs and replicas fail or helper files gets created. How do i fix this?

thanks

[osin_1]

withanh, You can use RamMap to see what's using the memory within Windows easily enough. We have a physical Veeam proxy server that has a large amount of local disk storage to accommodate the backups. When we run Veeam backups to this local storage the memory utilisation of the server goes up to approx 95% until the Veeam backup finishes - which is much the same as what you experience.

As others have stated the Veeam processes themselves dont utilise an extreme amount of memory but the file creation/copy/transfer within Windows is what uses as much memory as possible. When a Veeam job is running, open RamMap and the key field that you will see on the main screen of RamMap is Mapped File - check the Active column. This is where your memory is being used. If you then click on the File Summary tab and sort by Total, you will see the file name of the file thats currently consuming the memory, which will most probably be "your_backup_file_name.vbk".

I hope this helps a bit and you can show your manager where and why the memory is being used.

[withanh]

OK, I'm pretty well convinced that Veeam itself isn't using the memory, it's the cache that's doing it. I'll pursue the avenues that Tom suggested and get the MS hotfixes in place to hopefully prevent it from eating up all the RAM.

Latest graph screenshots of the memory metrics are below:




I've been running a forced full backup on one job that I started this morning around 11. You can see how it affected the cache in the graphs. Not entirely sure exactly what it means, but there it is.

[withanh]

When a Veeam job is running, open RamMap and the key field that you will see on the main screen of RamMap is Mapped File - check the Active column. This is where your memory is being used. If you then click on the File Summary tab and sort by Total, you will see the file name of the file thats currently consuming the memory, which will most probably be "your_backup_file_name.vbk".

This is great, and shows exactly what you described. my_backup.vbk is using the memory. I love those SysInternals tools, Mark Russinovich is a very smart man!

[Gostev]

Oh yes, he is... and very fun guy IRL, too. Never miss his sessions on TechEd and other conferences, they are fantastic. Also, he had published a few great books on Windows internals in the past years - those are must have for any admin!

[lobo519]

I run two backup jobs in parallel (local, remote). When my jobs start there is a huge increase in memory usage - I had to allocate an additional 8gb to satisfy but it still uses almost all of it. The strange thing is that I dont see as much CPU usage as i would image. I have 4 vCPUs and I have all my jobs set to best compression level and I still only seem spikes to 50% but usually hanging between 15- 30. Does that seem normal?

My replication jobs seem to be about the same if not slower compared to v5 using a local and remote proxy both with 4vcpu

I have not submitted a support case yet - just looking for any ideas - I know support is probably swamped

[Vitaliy S.]

I have 4 vCPUs and I have all my jobs set to best compression level and I still only seem spikes to 50% but usually hanging between 15- 30. Does that seem normal?

Yes, that means that proxy CPU resources is not a bottleneck for your replication jobs, as they are not fully saturated as might be expected with best compression level. Something else is the bottleneck - can you lookup what does the real-time statistics window reports as a bottleneck?

As for the exccesive memory consumption, please review this thread for additional details.

[pizang]

I guess you are using 64bit system for this server. I had similar issue. It was because system cache (for disk access) tried to allocate more and more memory until it completely dries it up. Please see another great tool from sysinternals.
http://technet.microsoft.com/en-us/sysi ... s/bb897561

Set working set max to logical value and you will be fine.

[cffit]

I have a physical server setup for our Veeam Backup server. It has two quad core Intel CPUs and 16GB of memory. Just to start out with, I setup a local repository and did a network backup of one of our larger database servers. I set the job to do the highest compression. About 40% of the way through the backup, the server shows no more free memory.

My curiousities are..

1. Is this normal?
2. Is this a result of using the best compression option?
3. Should I add even more memory in this case, or would it not make a difference?

My processing rate is around 40MB/s. I know there are many factors and methods that can affect this rate, but what is an average rate that I should expect to get or shoot for? I have no idea if 40MB/s is great or horribe. My
CPU runs at about 35% and my network connection is around 50%. Eventually I plan to do direct SAN backups to an iSCSI SAN, and I assume that will improve my rates of transfer.

[dellock6]

You can start some tests, first beeing to reduce compression to default and see if it runs better; compression and deduplication puts a load on server memory to handle data before it's compressed/dedupe, so the more compression you ask, more load you put on the server.
Do this test and compare to the highest compression.

[Gostev]

Compression level does not affect memory usage, don't worry about testing this. The real reason for no memory is described above in this topic.

[jawaidbazyar]

I wanted to post and say that the SetSystemFileCacheSize setting did the trick. Before doing this, my proxy/repository servers would often go crazy, running out of RAM, generating lots of failed backups due to resource (RAM) exhaustion.

I set the max file cache size 4GB smaller than my physical RAM (64-bit windows 2008), to leave room for Windows, the proxy software, etc.

This seems to have resolved the routine backup failures.

As SetSystemFileCacheSize is a Windows API call, should Veeam not be setting this to something reasonable?

I used this tool:

http://www.uwe-sieber.de/ntcacheset_e.html

[Gostev]

Yes, we started doing this ourselves in one of the recent product releases. Long ago enough for me to forget which one exactly. So unless this functionality broke, it's there...