Veeam Copy (and backup). each job has a big process

[barchas]

I have discovered an issue that seems to be a deal breaker for actually using Veeam for environments with client separation required.

Both job types, Backup and Copy exhibit the following behavior.

Each job, when it is running has a process named Veeam.Backup.Manager.exe associated with it.
In our environment each one of those is 120meg to 150meg. This is not an issue in itself, however, it seems to cause issues, maybe because of too many handles or something. But regardless, the all run at the same time no matter what the scheduling.

We have 12 copy jobs and 42 backup jobs. The copy jobs just sit there, even when idle, with these processes loaded. Even though they are not doing anything at all, except maybe keeping an eye out for a new restore point to copy. However, this seems a bit silly, since I would have assumed any number of jobs could exist, in continuous mode, and a single watchdog process could activate them when a new point is available, then start one job, wait until it is done, then move on.

The same thing applies for backup jobs. Which means if you're not careful you can torch your server needing a hard reboot. However, the backup jobs don't load up until the trigger time, except i am guessing if you had it set to continuous.

Am I the only one who has a need for chained (and controlled) backup and copy jobs? I would love to set all jobs to the same time of day, and have veeam just load up the first one when the start time is reached, then when done, start the next job, etc...

On top of it, setting scheduling for Copy jobs has no effect at all upon the process, which sits there whether it is in the time window or not.

The only real option I see is running backups from powershell, which is what I had to resort to in the past and thought v7 fixed that with limiting connections to storage. However, I can see no way to accomplish this for Copy jobs since they cannot be run if they are disabled...

Any advice on how to deal with this since for me it is only going to get worse in the next few months.

Thanks
d

[Dima P.]

Hello d,

Have you tried to specify time interval when copy job could transfer data (Copy Job - Schedule step)? As far as I can see the process does not eat so much resources, in my lab it was even closed pass some time. Thank you.

[veremin]

Also, the best option to control the number of simultaneous backup jobs is to configure the number of concurrent tasks the corresponding proxy/repository server can handle. Thanks.

[barchas]

Thanks for the feedback so far guys.

@popov I do specify the following:
Under job: Copy every 1 day at 6am
Under schedule: every day between 6am and 4pm for example.
However, it seems no matter what I set under schedule, even if I only enable sunday from 6a-7a the job process shows all day every day and uses 120-150mb ram. so 12 processes uses 1.5gig about.
But the bigger issue is that it seems that ram is not the problem. The system locks up at around 15 processes. Not CPU or Ram, but maybe threads?

@Eremin the concurrent tasks is set, but the problem is that all the tasks que up, still loading those managers. They don't run the copy, but they do run in general taking up resources.

[barchas]

To add, here is a screenshot of running 6 copy jobs at the same time. Though only 1 is actually executing a transfer while the other 5 are sitting idle.
You can see the usage of resources.

[Gostev]

If the processes do indeed sit idle, then they do not use any resources. Even the memory allocated by idle processes is automatically moved to swap, should the system need more physical memory for active processes. The beauty of modern OS memory management techniques!

Which means if you're not careful you can torch your server needing a hard reboot.

What makes you think so? I've seen customers scheduling hundreds of the jobs at the backup window start without any issues to the server. So, if you see system lockups at just 15 processes, then something is seriously wrong with the system, and it should be investigated.

The truth is, you do not want to micro-manage job schedules and concurrency in large deployments, and this cannot be done reliably in the first place. Sure, daisy chaining jobs (functionality that we actually DO provide for the backup jobs) is great on paper. But should one job in the chain take 3 times longer on one day due to multiple retries, or huge incremental data size caused by some Exchange defrag, this will completely screw up all the remaining jobs in the chain.

Talk to any Veeam Solution Architect, and they will always recommend against daisy chaining the jobs. And these guys do not work with customers smaller than 1000 VMs at all...

[barchas]

Hi Gostev, Thanks the feedback. Nothing makes me "think" this happens. It DOES happen, at least to us. I contacted tech support about this on Jan 14, and the response was to combine jobs so there were less active copy jobs, which of course sounded silly to me, but I took it for what it was worth. So, that is why I am posting here to try a different channel for troubleshooting, because frankly, it would be absurd to require combining jobs, and from your reaction, my guess is correct.

So what I gather is that this is bad info I got, and instead something different is wrong.

But the facts on my end are this. 12 copy jobs (not talking about the backup jobs). If all are running but idle, the server becomes unresponsive. Even if I mess with scheduling.

Any thoughts on what is going on?
I obviously really appreciate the help.
Dave

[Gostev]

I recommend simply re-opening your support case and referencing me and this topic, as the suggestion is obviously silly. Unless of course the suggestion was caused by the fact that your backup server simply does not meet the system requirements for your amount of ACTIVE (non-idle) concurrent jobs it processes.

For example, if your backup server runs data mover agents due to also carrying backup proxy and backup repository roles (in addition to management role), then system requirements for those must also be added up to overall server spec, as they are pretty resource intensive when actively processing the data (see the System Requirements section for those). Data movers need both lots of CPU and physical RAM, and multiple data movers certainly CAN lock up the system bad if it does not meet minimum requirements.

Typically, for larger environments we recommend that management server does not carry any other roles (meaning, you just disable default repository and default proxy), as the load from the local SQL Server alone can get pretty high. That is, unless you have purpose-built your server to be the complete solution in a single box, and be able to carry all roles, which is also not uncommon (although this requires monster spec with powerful CPU and lots of RAM, basically sum of requirements for all individual components).

Bottom line, 15 started jobs, with even less jobs actually processing the data and just idling, is nothing for v7. We've had 10 times larger deployments even with v6, which was much less optimized in terms of memory consumption (the same processes you are looking at consumed about 7 times more RAM), and no issues whatsoever with the System Requirements met. You can be sure at that, as one of the first v6 complaints reported by multiple customers was unintended 64 concurrent jobs limit per backup server, something we have removed in one of the first v6 patches.

[barchas]

Thanks Gostev. I will do that.

The machine is also the data mover, no proxies, but we limit both the local proxy and storage to 1 active connection, So that "should" limit it to only ever being 1 running at a time, correct?

It is a virtual under VMware, with 12 cores and 16gb ram.

I am going to try adding a proxy or two and remove the data mover related roles from the manager, and try that first before getting back to support.

Thanks!
Dave

[Gostev]

As far as RAM consumption, backup proxy data mover is not so hungry for RAM (just a few hundred MB per data mover), it is backup repository data mover that can potentially consume a lot of RAM (easily a few GB per data mover). Backup proxy can only hurt CPU resources, but this still should not cause system lockups, as we run data mover processes at lower priority specifically to prevent this kind of impact. And, of course, both proxy and repository roles can flood the network bad, but that should not cause OS lockups either.

Your spec looks good as fas as the management server (as per system requirements, you need at least 10 GB with 12 active jobs) , but you certainly want more RAM if your backup server carries any other roles.

If you are saying that local proxy and repository where NOT limited to 1 active concurrent task before, then there is your problem right there. I would just bump RAM to 32GB first thing, and see if that helps.

[tsightler]

Certainly just having the management server processes should not be a problem, however, when a backup copy job starts, it starts a repository agent as well. Are you saying that this server is both the proxy and the repository and the target repository for the copy? That could easily be a lot of memory, especially if all Copy Jobs are running on the same server. It sounds like you'll definitely want to start the process of scale out using additional proxy and repository servers if this is the case rather than trying to run everything on the single VM.

I can certainly see why support might make the suggestion to have less copy jobs as copy jobs are designed to be able to have a many to one relationship with backup jobs, i.e. multiple backup jobs can easily be "linked" to a single backup copy job. Of course, this may not always be ideal due to repository size or other requirements to keep the data separate.

[barchas]

You Know what. The way Tsightler just said it, made me think of something...

I just realized that this became an issue when we switched to a "local" iscsi mount. This would mean the repo is technically local to the veeam server right? the repo is actually an iSCSI target which is mounted as a NTFS folder under the C drive. Is my understanding correct?

[veremin]

Is my understanding correct?

Yep, in this case, "target" (repository) data movers are executed on the same server with the "source" (proxy) data movers. Thanks.

[barchas]

bah ok Thanks Eremin.

So back to the copy jobs then. How do I run dozens of copy jobs? Or is this the same problem? And basically I need create a couple proxies?

Thanks

[foggy]

Proxies are not involved in the backup copy process. Data is transferred directly between data movers installed on source and target repositories (on the same computer, in your case).

[tsightler]

I would suggest that you need to separate the repository and proxies to their own servers. In any scenario where you need more than a couple of jobs running at the same time the primary Veeam server should be dedicated to the task of managing jobs. Then you can begin to scale out with proxies and repositories. Ideally the source and target repositories of a Backup Copy should be two different repository servers, but it is possible to create two repositories on the same server. This will make it easier to plan for the required memory and to limit tasks to keep memory in check since repositories have their own tasks limits which can be set.

[AlexL]

We seem to have the exact same problem as the topic starter:

physical B&R server (8 cores, 32gb ram, using san mode) and 6 virtual proxies on vmware

all 90 backup jobs are started in a staggered way between 01:00 and 06:00
all copy jobs are started in a staggered way between 08:00 and 09:00
copy job window is set from 09:00 to 00:00

with veeam 6.5 and 90 backup jobs: no problems
with veeam 7.0 and 90 backup jobs: no problems (except some occasional errors with parallel processing enabled, not fully diagnosed yet, disabled pp for now)
with veeam 7.0 and 90 backup jobs and 45 copy jobs: between 10 and 20 backups jobs would give errors, either starting or only some vm's, job do mostly not retry, copy jobs run fine
with veeam 7.0 and 90 backup jobs and 90 copy jobs: 30 or more backup jobs are failing and NOT doing an automatic retry, copy jobs run fine

miscellaneous errors, when doing manual retries getting strange errors like "no process for this job" or sometimes "out of memory", but looking at task manager there is only 20gb of the 32gb in use.

With 15+ years system admin experience (and using veeam for 4 years now) my "feeling" after struggling with this config for over a week now is that the server, or service(s), are overloaded in some way, the concurrency seems simply to high, there are no clear signs of overloading, cpu is between 20 and 60%, memory 20 to 5gb free, network not fully used but still the product simply cannot keep up.

Especially the continuesly running copy jobs seem to have a big impact. Any insight or solution to this matter would be highly appreciated.

[tsightler]

physical B&R server (8 cores, 32gb ram, using san mode) and 6 virtual proxies on vmware

all 90 backup jobs are started in a staggered way between 01:00 and 06:00
all copy jobs are started in a staggered way between 08:00 and 09:00
copy job window is set from 09:00 to 00:00

with veeam 6.5 and 90 backup jobs: no problems
with veeam 7.0 and 90 backup jobs: no problems (except some occasional errors with parallel processing enabled, not fully diagnosed yet, disabled pp for now)
with veeam 7.0 and 90 backup jobs and 45 copy jobs: between 10 and 20 backups jobs would give errors, either starting or only some vm's, job do mostly not retry, copy jobs run fine
with veeam 7.0 and 90 backup jobs and 90 copy jobs: 30 or more backup jobs are failing and NOT doing an automatic retry, copy jobs run fine

Backup copy jobs were not really built to scale 1-to-1 with backup jobs, that's why it's possible to link one backup copy to mulitple backup jobs, although I understand there are cases where this is not ideal since doing so also consolidates the data into a single job on the target, so if keeping the data separate is important you might have to go 1-to-1 like you're doing.

It's important to remember that, unlike backup jobs, backup copies run "continuously" which manes there is a Veeam manager process using resources at all times. For a backup job, there's only a manager process when the job is actually actively processing VMs, so this is a major difference in resource usage. Since the base recommendation is to have 256MB for each active manager process you'd need about 24GB of RAM just to have the 90 copy jobs running at all times, leaving only 8GB of your 32GB total for the rest of the processing.

I'd strongly suggest increasing the memory in this box if you want to have 90 copy jobs active. I'm not sure if that's the only constraint you're running up against, but it's almost certainly one of them. Yes, it may not seem like the box is using all of the memory, but as Manager processes start, they generally peak at around 750MB to 1GB of RAM, before settling back down at a lower number, my guess is there are periods of time where there simply isn't enough memory for the job to start. You can look at the peak sizes for the manager processes by using task manager and adding the column for "Peak working set" to the view.

[AlexL]

Your answer does confirm my 'feeling' and observations over the past week. Unfortionately, combining multiple backup jobs in a few copy jobs is not practical for us as a hoster/service provider, in essence we have a backup and backup copy job per customer to clearly separate data for compliancy reason amongst other reasons.

Also if I'd combine lets say 90 backup jobs in 4 copy jobs I'd be looking at very large jobs (at least 100+ vm's) and backup copy files (easily 5 to 10GB for the full backup) and I can only image the time it would take to do a monthly check or compact on these large backups.

The backup copy job is (almost) the next best thing to sliced bread for doing offsite backups but if veeam development could optimize stuff so running a large number of jobs on a single server than it would really excel for larger environments and/or service providers.

[tsightler]

I definitely have some ideas around how to significantly improve Backup Copy for this use case so I'll push those up the chain.

For now though, you could probably get around this with some creative Powershell, or perhaps just add some more memory to the box and see if that solves the problem.

For the Powershell idea the thought is that since you don't allow the Backup Copy jobs to overlap you backup jobs anyway, just use a script via Task Manager to disable the Backup Copy jobs at 00:00, and then enable them again at 09:00, something like:

To Disable:

Code: Select all

Get-VBRJob | ? {$_.JobType -eq "BackupSync"} | Disable-VBRJob

To Enable:

Code: Select all

Get-VBRJob | ? {$_.JobType -eq "BackupSync"} | Enable-VBRJob

Admittedly you might need to add a little loop with a short delay to enable all of those 90 backup copy jobs, perhaps 5 seconds per job. I have not tested this, but it was just a stray thought that I think might work OK for the interim.

[AlexL]

Hmm, that might work, will try it, adding ram is not a short term solution since it is fairly old hardware and all memory slots are occupied, new hardware with higher specs will arive in 2 or 3 months