Memory management and San based backup performance

[Peejay62]

Hi,

I have seen more discussions about proxies concerning performance and related to that things like network behaviour, san connections, memory usage and sizing aspects.
I start a seperate thread anyway,don’t know in which existing one it would fit best.

In our environment I use all three kinds op transportmodes for backup. Most of it is still networkbased but of course san-based is the preferred one for known reasons, so I started implementing it by connecting datastores to the proxy.
For the proxy for san based backup I have some serious horsepower available (a Six-core AMD Opteron 8431 2.41 ghz (4processors) Dl380 with 128GB internal memory running windows 2008 Enterprise R2) and for the backup repositorypart a dedicated IBM v7000 storage array. All connected @8gbit.
What I notice is that over time, the san based backup seems to be going slower and slower or there is a lot of fluctuation in the performance.

So I started some backuptesting. My testsetup in short: three datastores of each 1TB connected to the proxy, 2 esxhosts hosting appx 30 machines.
The repository is san based connected to the proxy also. I defined a job to read 12 vms,equally distributed over the datastores. Cleared everything, so it will be a fresh job (full). Data to read is appx 600GB.
Backup runs fine i might state @1.3GB/s througphut, processing rate states appx 800 MB/sec. Ran a second job, somehow indentical of the first with the same infra, different vms,fresh full backup. Appx 25% pct lower throughput/processing rate.
I monitored that the proxy is working hard, all cpu’s very busy (normally I suspect) but the memory is only used @8-10GB (@start it was appx 4,5). The memory cache is filling though. Eventually the cache fills up.
Checking what is in cache it is a big part of the backupfile created in the first job and partially also the second backup. So I did various testing with the cache memory in mind.
What i notice :After a backup the backupfile is kept in cache, subsequent backups seem to suffer of this in matter of performance. If you remember my first test, the second backup was 25% slower.
I did the exact same backup but flushed the cache first, it ran at 1.3GB/sec instead of 0.9GB/sec. Flushing the cache is fairly easy, either use rammap tool or just delete backup from disk within veeam.
Probably the fact that san backups tend to slow down over time is caused by this cache memory management? Does this make sense and any suggestions?
Seems useless that a newly created backupfile remains cached in the proxy memory after it is processed.
I also wonder if the 24 parallel processes I am running do only need that few a memory. It’s only consuming 4-5 GB internal memory while running.

Thanks, Peter

[Gostev]

Hi, Peter. Your observations are correct, in our research we also saw Windows cache memory management to be the problem. But from what I remember, the later Windows version fix the issue, so try upgrading.

The memory consumption looks correct for source data movers. As per System Requirements for Backup Proxy server, we require 200 MB of RAM per concurrent task.

Thanks!

[Peejay62]

Hi Anton,

thanks, I will take a look at the Windows system to see it needs an update.
As far as the proxy is concerned. This server is acting as a proxy and as a repository server in one. As per system requirements mentioned i would need another2 GB per task for the repository role. That was were I was thinking about, I would expect it to consume another 48GB when running at full usage, 24 tasks... As the biggest usage of my memory doesn't come over 10GB I am a bit puzzled.

Thanks, Peter

[Gostev]

It's 2GB per incoming job, not per task.