Can you run multiple simultaneous backup jobs?

[TaylorB]

Simple question, can I run multiple backup jobs at the same time from the same Veeam server, assuming they don't point to the same VM's, etc.

Thanks.

[Gostev]

Sure, this is exactly what most of our customers are doing today.

[smithjamesp55]

I've tried to schedule mine throughout the week, but get some overlap sometimes. Are there any guidelines about how many jobs can run simultaneously based on the server's specs?

[Gostev]

With modern CPU, I recommend 1 job for every 2 CPU cores.

[mooreka777]

I run 4-5 jobs run at a time with the veeam virtual machine taking 6 CPU's and 12 GB of RAM. The VM goes critical on CPU for about an hour at night. Fine with me.

[Gostev]

12GB of RAM?! What process exactly consume all this RAM? Cannot be Veeam Backup processes for sure.

[mooreka777]

12GB of RAM?! What process exactly consume all this RAM? Cannot be Veeam Backup processes for sure.

Yes, it is. When the backups run it takes up all 12Gb. I could probably give it more and it would take it. That's the good thing about vmware and drs. When the VM needs the resources it takes them. When done, it gives them back.

[TrevorBell]

My R710 has 32 gig ram anton

[Gostev]

Yes, it is. When the backups run it takes up all 12Gb. I could probably give it more and it would take it.

What processes exactly consumes this much RAM according to the Task Manager on Veeam server? I am just not seeing this in my lab. Thanks.

[TaylorB]

What processes exactly consumes this much RAM according to the Task Manager on Veeam server? I am just not seeing this in my lab. Thanks.

I can use 8GB with just one job!

The Veeam Backup service is about 4-5GB, the SQL Express service about 2GB and the rest of windows is 1-2GB.

[Gostev]

This is some sort of issue that needs investigating. Something causes memory leak, for example antivirus? Veeam Backup service should have around 26MB working set under normal circumstances, it does not really do much. In fact, Trevor had sent me a screenshot of his production system earlier today, with two parallel jobs. All Veeam Backup processes collectively were consuming way under 2GB RAM, with 2 main "working" agents sitting at 400-500MB. Whole system was indeed consuming 12GB of RAM, but this had nothing to deal with Veeam processes.

[brupnick]

I just opened a ticket with Veeam because I have noticed similar resource issues and actually received some "Client error: Cannot allocate memory for an array. Array size: [4198400]." errors over the weekend.

[pschneider]

I just opened a ticket with Veeam because I have noticed similar resource issues and actually received some "Client error: Cannot allocate memory for an array. Array size: [4198400]." errors over the weekend.

I have the same error since a few days. Do you have any updates from support?

[brupnick]

I have the same error since a few days. Do you have any updates from support?

It's somewhat strange because I built two brand new 2008R2 SP1 machines at the same time from the same VM template to handle my VBR jobs. One server runs fine (#2), but the other (#1) has the performance issues reported above. After talking with Justin from Veeam last week, he confirmed that, if possible, I should rebuild #1 from scratch. I did that on Monday, but due to some production problems, probably won't be able to run jobs on it until tomorrow (Friday). I'll let you know how it goes on Monday.

I know that if it works, this isn't an ideal solution for some people, but since we're a VMware shop, new builds only take a few minutes.

[brupnick]

I'm not trying to steal TaylorB's post here and will gladly start a new thread, but I just wanted to provide an update. After rebuilding my VBR server last week and doing a clean install of v5, my fulls ran over the weekend. I have 3 jobs: one for production, one for test, and one for development. Production calls test (when finished), then test calls development, so that if I have issues, production gets done first.

My production job has 66 VMs for a total of 10.53 TB of data. The first 54 VMs processed as expected, but starting with VM #55, I received the following message for that VM and all subsequent servers:

Code: Select all

Backing up object "[[DATASTORE NAME] [SERVER NAME]/[SERVER NAME].vmdk"
BackupDiskFull failed 
Client error: Cannot allocate memory for an array. Array size: [2097152]. 

Server error: End of file 

When the job retried for those failed VMs, I received a different error:

Code: Select all

Publishing catalog session
CheckStorage failed 
Client error: bad allocation

I have a ticket open for this, so I just sent an email to support (Justin) to see what he thinks. I'm curious to know if anyone in the community has a job that produces a backup file(s) that is larger than 2.02 TB. The reason I ask is because this seems to be where this job encounters issues when I run a full. This job was writing to an empty folder, so the only files in there after this job are a .vib and .vbk which together are 2.02 TB. I know that VMware reaches its limit around this point, so I'm curious if maybe there is a VBR ceiling around there as well.

Any ideas would be greatly appreciated.

[Gostev]

The size of backup files absolutely cannot be an issues. I've seen many customers backing up to a few times larger backup files.

[brupnick]

The size of backup files absolutely cannot be an issues. I've seen many customers backing up to a few times larger backup files.

The impression that I got from support seemed to indicate otherwise. I have split my production VMs into two separate jobs and they are running now, so we'll see how that works. @Gostev: How much larger are we talking?

[Gostev]

Well, 4-8 TB is pretty common size I think... there was a bug with "WAN target" block size optimization right after v5 release (metadata size overflow because of number of blocks). The issue would only trigger on backups larger that 4 TB, and we had quite a few support cases opened on this issue until the bug was fixed in version 5.0.1. So clearly, large backup files are very common.

[brupnick]

I did end up splitting my one production job in half, with each job backing up just over 4 TB of data. Both jobs ran without issue (both fulls and incrementals), so it looks like I'm going to have to keep it as two jobs. This is too bad since I can't get the dedup benefits of having all my production VMs together, but having them all back up is more important than dedupe.

I am using WAN target for these jobs (since I replicate them to a DR site), but I'm running 5.0.2.230, so I should not be affected by the bug that you mentioned.

[linux4guru]

I read somewhere that there is a limit on how many jobs can be run against an ESX host. I think it is 4 and this is VMware limitation...

[brupnick]

I read somewhere that there is a limit on how many jobs can be run against an ESX host. I think it is 4 and this is VMware limitation...

Does this mean 4 separate VBR jobs that are each backing up 1 VM, but each of those VMs are on the same host?

[Gostev]

There is no such limit (or anything like that).

[WouterS]

I am facing the exact same issue, anything else suggested then spliting up jobs?

[WouterS]

From VEEAM support I get the following response:

>I suggest that you contact Microsoft support for help to resolve this issue as this is a windows error.

Anyone an idea if this is indeed a Windows related issue?

[Vitaliy S.]

Could you please tell me your ticket number, so I could have a look at the issue you're having?

[WouterS]

HI Vitaliy,

It is #5144868

Thanks.

[Vitaliy S.]

Hm...looks like you have a similar issue to what we have seen here: Veeam on windows 2008

Not sure it would help but you may want to apply all the latest Microsoft patches to see if this resolves the problem or not.

[WouterS]

Thanks!

We have added 8GB of ram to the server and all is running fine for now.

[Vitaliy S.]

Thank you for update! Sounds like increasing memory limits for 2008 windows server is a way to go with this issue.

[zak2011]

Also got the same errors Client error: Cannot allocate memory for an array.
Exception from server: Cannot allocate memory for an array. for backup jobs and backup copy jobs.
All latest updates have been applied and server has 20GB memory.
Any idea why this error comes up?