Comprehensive data protection for all workloads
TaylorB
Enthusiast
Posts: 54
Liked: 4 times
Joined: Jan 28, 2011 4:40 pm
Full Name: Taylor B.
Contact:

Can you run multiple simultaneous backup jobs?

Post by TaylorB »

Simple question, can I run multiple backup jobs at the same time from the same Veeam server, assuming they don't point to the same VM's, etc.

Thanks.

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by Gostev »

Sure, this is exactly what most of our customers are doing today.

smithjamesp55
Influencer
Posts: 15
Liked: never
Joined: Jan 27, 2011 11:09 pm
Full Name: James Smith
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by smithjamesp55 »

I've tried to schedule mine throughout the week, but get some overlap sometimes. Are there any guidelines about how many jobs can run simultaneously based on the server's specs?

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by Gostev »

With modern CPU, I recommend 1 job for every 2 CPU cores.

mooreka777
Influencer
Posts: 18
Liked: 1 time
Joined: Jan 20, 2011 1:05 pm
Full Name: K Moore
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by mooreka777 »

I run 4-5 jobs run at a time with the veeam virtual machine taking 6 CPU's and 12 GB of RAM. The VM goes critical on CPU for about an hour at night. Fine with me.

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by Gostev »

12GB of RAM?! What process exactly consume all this RAM? Cannot be Veeam Backup processes for sure.

mooreka777
Influencer
Posts: 18
Liked: 1 time
Joined: Jan 20, 2011 1:05 pm
Full Name: K Moore
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by mooreka777 »

Gostev wrote:12GB of RAM?! What process exactly consume all this RAM? Cannot be Veeam Backup processes for sure.
Yes, it is. When the backups run it takes up all 12Gb. I could probably give it more and it would take it. That's the good thing about vmware and drs. When the VM needs the resources it takes them. When done, it gives them back.

TrevorBell
Expert
Posts: 356
Liked: 13 times
Joined: Feb 13, 2009 10:13 am
Full Name: Trevor Bell
Location: Worcester UK
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by TrevorBell »

My R710 has 32 gig ram anton :D

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by Gostev »

mooreka777 wrote:Yes, it is. When the backups run it takes up all 12Gb. I could probably give it more and it would take it.
What processes exactly consumes this much RAM according to the Task Manager on Veeam server? I am just not seeing this in my lab. Thanks.

TaylorB
Enthusiast
Posts: 54
Liked: 4 times
Joined: Jan 28, 2011 4:40 pm
Full Name: Taylor B.
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by TaylorB »

Gostev wrote: What processes exactly consumes this much RAM according to the Task Manager on Veeam server? I am just not seeing this in my lab. Thanks.
I can use 8GB with just one job! :)

The Veeam Backup service is about 4-5GB, the SQL Express service about 2GB and the rest of windows is 1-2GB.

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by Gostev »

This is some sort of issue that needs investigating. Something causes memory leak, for example antivirus? Veeam Backup service should have around 26MB working set under normal circumstances, it does not really do much. In fact, Trevor had sent me a screenshot of his production system earlier today, with two parallel jobs. All Veeam Backup processes collectively were consuming way under 2GB RAM, with 2 main "working" agents sitting at 400-500MB. Whole system was indeed consuming 12GB of RAM, but this had nothing to deal with Veeam processes.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by brupnick »

I just opened a ticket with Veeam because I have noticed similar resource issues and actually received some "Client error: Cannot allocate memory for an array. Array size: [4198400]." errors over the weekend.

pschneider
Lurker
Posts: 1
Liked: never
Joined: Aug 23, 2011 6:35 am
Full Name: Peter Schneider
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by pschneider »

brupnick wrote:I just opened a ticket with Veeam because I have noticed similar resource issues and actually received some "Client error: Cannot allocate memory for an array. Array size: [4198400]." errors over the weekend.
I have the same error since a few days. Do you have any updates from support?

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by brupnick »

pschneider wrote: I have the same error since a few days. Do you have any updates from support?
It's somewhat strange because I built two brand new 2008R2 SP1 machines at the same time from the same VM template to handle my VBR jobs. One server runs fine (#2), but the other (#1) has the performance issues reported above. After talking with Justin from Veeam last week, he confirmed that, if possible, I should rebuild #1 from scratch. I did that on Monday, but due to some production problems, probably won't be able to run jobs on it until tomorrow (Friday). I'll let you know how it goes on Monday.

I know that if it works, this isn't an ideal solution for some people, but since we're a VMware shop, new builds only take a few minutes.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by brupnick »

I'm not trying to steal TaylorB's post here and will gladly start a new thread, but I just wanted to provide an update. After rebuilding my VBR server last week and doing a clean install of v5, my fulls ran over the weekend. I have 3 jobs: one for production, one for test, and one for development. Production calls test (when finished), then test calls development, so that if I have issues, production gets done first.

My production job has 66 VMs for a total of 10.53 TB of data. The first 54 VMs processed as expected, but starting with VM #55, I received the following message for that VM and all subsequent servers:

Code: Select all

Backing up object "[[DATASTORE NAME] [SERVER NAME]/[SERVER NAME].vmdk"
BackupDiskFull failed 
Client error: Cannot allocate memory for an array. Array size: [2097152]. 

Server error: End of file 
When the job retried for those failed VMs, I received a different error:

Code: Select all

Publishing catalog session
CheckStorage failed 
Client error: bad allocation
I have a ticket open for this, so I just sent an email to support (Justin) to see what he thinks. I'm curious to know if anyone in the community has a job that produces a backup file(s) that is larger than 2.02 TB. The reason I ask is because this seems to be where this job encounters issues when I run a full. This job was writing to an empty folder, so the only files in there after this job are a .vib and .vbk which together are 2.02 TB. I know that VMware reaches its limit around this point, so I'm curious if maybe there is a VBR ceiling around there as well.

Any ideas would be greatly appreciated.

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by Gostev »

The size of backup files absolutely cannot be an issues. I've seen many customers backing up to a few times larger backup files.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by brupnick »

Gostev wrote:The size of backup files absolutely cannot be an issues. I've seen many customers backing up to a few times larger backup files.
The impression that I got from support seemed to indicate otherwise. I have split my production VMs into two separate jobs and they are running now, so we'll see how that works. @Gostev: How much larger are we talking?

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by Gostev »

Well, 4-8 TB is pretty common size I think... there was a bug with "WAN target" block size optimization right after v5 release (metadata size overflow because of number of blocks). The issue would only trigger on backups larger that 4 TB, and we had quite a few support cases opened on this issue until the bug was fixed in version 5.0.1. So clearly, large backup files are very common.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by brupnick »

I did end up splitting my one production job in half, with each job backing up just over 4 TB of data. Both jobs ran without issue (both fulls and incrementals), so it looks like I'm going to have to keep it as two jobs. This is too bad since I can't get the dedup benefits of having all my production VMs together, but having them all back up is more important than dedupe.

I am using WAN target for these jobs (since I replicate them to a DR site), but I'm running 5.0.2.230, so I should not be affected by the bug that you mentioned.

linux4guru
Enthusiast
Posts: 40
Liked: never
Joined: Aug 24, 2011 8:26 pm
Full Name: Thomas Lam
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by linux4guru »

I read somewhere that there is a limit on how many jobs can be run against an ESX host. I think it is 4 and this is VMware limitation...

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by brupnick »

linux4guru wrote:I read somewhere that there is a limit on how many jobs can be run against an ESX host. I think it is 4 and this is VMware limitation...
Does this mean 4 separate VBR jobs that are each backing up 1 VM, but each of those VMs are on the same host?

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by Gostev »

There is no such limit (or anything like that).

WouterS
Novice
Posts: 6
Liked: never
Joined: Mar 07, 2009 11:53 am
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by WouterS »

I am facing the exact same issue, anything else suggested then spliting up jobs?

WouterS
Novice
Posts: 6
Liked: never
Joined: Mar 07, 2009 11:53 am
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by WouterS »

From VEEAM support I get the following response:

>I suggest that you contact Microsoft support for help to resolve this issue as this is a windows error.

Anyone an idea if this is indeed a Windows related issue?

Vitaliy S.
Product Manager
Posts: 24243
Liked: 1859 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by Vitaliy S. »

Could you please tell me your ticket number, so I could have a look at the issue you're having?

WouterS
Novice
Posts: 6
Liked: never
Joined: Mar 07, 2009 11:53 am
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by WouterS »

HI Vitaliy,

It is #5144868

Thanks.

Vitaliy S.
Product Manager
Posts: 24243
Liked: 1859 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by Vitaliy S. »

Hm...looks like you have a similar issue to what we have seen here: Veeam on windows 2008

Not sure it would help but you may want to apply all the latest Microsoft patches to see if this resolves the problem or not.

WouterS
Novice
Posts: 6
Liked: never
Joined: Mar 07, 2009 11:53 am
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by WouterS »

Thanks!

We have added 8GB of ram to the server and all is running fine for now.

Vitaliy S.
Product Manager
Posts: 24243
Liked: 1859 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by Vitaliy S. »

Thank you for update! Sounds like increasing memory limits for 2008 windows server is a way to go with this issue.

zak2011
Expert
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Can you run multiple simultaneous backup jobs?

Post by zak2011 »

Also got the same errors Client error: Cannot allocate memory for an array.
Exception from server: Cannot allocate memory for an array. for backup jobs and backup copy jobs.
All latest updates have been applied and server has 20GB memory.
Any idea why this error comes up?

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 29 guests