Comprehensive data protection for all workloads
Post Reply
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

B&R v9 oddities (maybe bugs)

Post by DaveWatkins » 1 person likes this post

I've just upgraded from 8 to 9 and the upgrade itself was fine.

I then wanted combine two repos into a scale out repo but everytime I tried I got an error saying it couldn't access a folder on one of the repos. The folder was from an old backup job that had been removed and deleted a long while ago. I'm assuming this was associated with an orphan record in the database about that job that wasn't removed properly. I ended up completely removing that repo from B&R and re-adding it. After that I could build the scale out repo with it.

The other issue I've seen so far is that an Exchange job that was running perfectly fine in 8 gave me this error on one of the Exchange servers

"Error: BlobCall: Undefined function id: 54"

and failed. That's not one I've seen before and if it persists I'll log a support job for it (the job is still running).

Also, config backups don't appear to be able to go to scale out repos, which means you'll need to keep a separate repo for them if you were planning on running just one big scale out repo. Still adding a 1GB repo on a proxy somewhere isn't a huge issue

Other than that, things have gone well. Per VM backup chains are on and working. EMC SAN's are added and having the console locally is very nice (although I'm not sure about the flat look and the "blandness")
nielsengelen
Product Manager
Posts: 5618
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by nielsengelen » 2 people like this post

Do keep in mind that you could create a folder one of the 2 repositories and add that one to VBR again to save your configuration.

For example on a Windows server:
E:\BACKUPS (this will be part of SOBR)
E:\CONFBACKUP (this won't be part of SOBR)
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by Dima P. »

DaveWatkins,
"Error: BlobCall: Undefined function id: 54"
This one needs to be investigated with support team for sure. If the issue persists, kindly, open a support case and post the case ID this thread.
config backups don't appear to be able to go to scale out repos
That’s by design but the workaround Niels suggested sound good.
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by DaveWatkins »

Yeah, the config backup one wasn't really anything to worry about for me, just thought I'd put it here so others saw it. Plenty of ways to work around it easily.

The Scale out issue was interesting, but I'm guessing was more related to v8 not cleanly removing the DB records about that folder than anything to do with v9. Still waiting for that exchange job to finish so I can retry it and see f the error persists
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by Delo123 » 1 person likes this post

Dave,

just updated to V9 and after reading your post i ran an incremental backup of our Exchange 2010 DAG Cluster, all fine here...
Dima V.
Veeam Software
Posts: 50
Liked: 12 times
Joined: Oct 21, 2010 8:54 am
Full Name: Dmitry Vedyakov
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by Dima V. »

DaveWatkins,
DaveWatkins wrote: I then wanted combine two repos into a scale out repo but everytime I tried I got an error saying it couldn't access a folder on one of the repos. The folder was from an old backup job that had been removed and deleted a long while ago. I'm assuming this was associated with an orphan record in the database about that job that wasn't removed properly. I ended up completely removing that repo from B&R and re-adding it. After that I could build the scale out repo with it.
Can you please clarify. It may help us to improve our code.
If I understand right you had orphaned job which was pointed to to that Repository, had some runs in the past, but there wasn't any registered backups (even failed) neither under Disk->Backup nor under Disk->Imported?
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by DaveWatkins »

That's correct, there was no record of the old job anywhere in the UI. All the Repo backups had been removed from the GUI and the job had been deleted months ago.

I've just stumbled upon a new issue but this one will likely need to go via support. My Scale out repo consists of 2 repos, one had ~500GB free and the other has 9TB. I've created a new job with 12 VM's abd about 350GB of data in the VM's total.

When running the job the first 3 VM's start backing up fine, but the remaining all fail immediately with:

" Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file. "

There is another backup in progess to this scale out repo going but there is plenty of free space available and even if the other job is "locking" a repo even the server with 500Gb free would be enough to store the initial backups of these VM's

Retrying the job after it finished and then the next 3 VM's start backing up fine and the remaining 6 fail again. I'll keep retrying until I have all 12 VM's with backup files on the repo and then launch a normal backup and see what happens
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by foggy »

This last one indeed looks unexpected, however can be caused by the space estimation algorithm used during backup to scale-out repository. So please contact technical support with this for confirmation.
I.Makaryev
Veeam Software
Posts: 15
Liked: 5 times
Joined: Dec 21, 2012 8:53 am
Full Name: Igor Makaryev
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by I.Makaryev »

DaveWatkins wrote:
Retrying the job after it finished and then the next 3 VM's start backing up fine and the remaining 6 fail again. I'll keep retrying until I have all 12 VM's with backup files on the repo and then launch a normal backup and see what happens
Hello, Dave!
Do you use VMware?
Two questions more - do your VMs have thick or thin provisioned disks? How much "used storage" space is VMware reporting for that VMs?
Thanks.
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by DaveWatkins »

Yes, these are all running on VMWare 6U1 and the VM's are thick provisioned. Used storage space is a mix but in all cases the full size of the provisioned disks (between 25GB and a couple of hundred GB depending on the VM in question)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by foggy »

Large thick disks with small amount of actual data can easily confuse backup repository space estimation. Job logs will tell whether this is the cause of this behavior.
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by DaveWatkins »

It isn't/wasn't. The disks didn't have a lot of free space in fact most of them are at about 80% full. Also if that were the case surely retrying the job wound't have fixed it (or at least allowed it to start 3 more backups).

To reiterate, none of the VM's by themselves would have filled any of the underlying repo's, In fact the total size of all the VM's in the job would not have filled up one of the underlying repos even if all the disks were at 100% usage
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by foggy »

Anyway, we're interested in further investigation of this behavior, so contacting support is recommended.
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by SyNtAxx »

foggy wrote:Anyway, we're interested in further investigation of this behavior, so contacting support is recommended.
I'm interested also, as I seem to be seeing similar behavior(s).

'Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file. '

There is currently 38.5TB free of 64TB.

-Nick
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by DaveWatkins »

It didn't ever open a ticket for this as we have some other stuff going on taking my time and I didn't have the time to dedicate to it (new SAN, wireless upgrade etc), and I could work around it for now. Given the ease with which I can replicate it I figured it could wait. Nick, you're welcome to open a ticket yourself
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by SyNtAxx »

DaveWatkins wrote:It didn't ever open a ticket for this as we have some other stuff going on taking my time and I didn't have the time to dedicate to it (new SAN, wireless upgrade etc), and I could work around it for now. Given the ease with which I can replicate it I figured it could wait. Nick, you're welcome to open a ticket yourself
I'll have to open a case. I just started using the scale out repos and I need to be able to trust them. They work great and let me load balance, but an awful lot of free space available to fail a job over.

Nick
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by SyNtAxx » 1 person likes this post

Veeam Support - Case # 01691757
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by SyNtAxx »

New Errors for Scale Out Repo(s) on my build: about a dozen of these: Error: Failed to check extent 'Dcspwvbs002_v400_vol07' heartbeat
jl@fire-eater.dk
Lurker
Posts: 1
Liked: never
Joined: Feb 14, 2016 9:52 am
Full Name: Jesper Løwe
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by jl@fire-eater.dk »

was this ever resovelved

i get Error: BlobCall: Undefined function id: 54 on a 2003 server and on a 2008 server after upgrading to veeam 9
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by SyNtAxx »

Veeam Support - Case # 01691757

'Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file. '

There is currently 38.5TB free of 64TB.

-Nick

I still continue to be plagued by this problem. So much so, that I cant run automated backups anymore. I basically have to babysit my backups 1 by one. This issue needs a resolve. I will have to migrate all my backups off SOBR(s) since the placement logic seems to get overloaded and fail out almost 100% of my jobs pointed at it. The SOBR is now at 16TB free of a total f 80TB. The less space in the SOBR the more frequent the issues.

-NIck
nielsengelen
Product Manager
Posts: 5618
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by nielsengelen »

jl@fire-eater.dk wrote:was this ever resovelved

i get Error: BlobCall: Undefined function id: 54 on a 2003 server and on a 2008 server after upgrading to veeam 9
Windows 2003 server support was dropped (as can be found in the release notes) so you can't use them anymore for your Veeam infrastructure such as proxy, repo,... Please consider upgrading the server to 2008/2012 as Microsoft also dropped 2003 support.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by foggy »

You can still backup VMs running Windows 2003 though, this is what Jesper is doing, I suppose. Anyway, with any technical issue it is always recommended to contact technical support first.
ITP-Stan
Service Provider
Posts: 201
Liked: 55 times
Joined: Feb 18, 2013 10:45 am
Full Name: Stan (IF-IT4U)
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by ITP-Stan »

BlobCall: Undefined function id: 54
On 2 VM's in the job, the rest (10+/-) ran fine.
Windows 2012 R2 repo, b&r server & proxy all-in-one.

I'll have to open a support case by the looks of it.
BartP
Veeam Software
Posts: 230
Liked: 62 times
Joined: Aug 31, 2015 8:24 am
Full Name: Bart Pellegrino
Location: Netherlands
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by BartP »

This error i see more and more lately.
Gut feeling is that it related to the performance/pending time of the current Job. Especially Full Backups seem to be more prone to the issue.
Whenever a job takes very long to finish and other machines are pending a while, they tend to stop with Function Call ID 54.
When you manually start the same job afterwards, it runs fine.

I am currently testing with a couple setups to see if i can reproduce the error and more importantly see if i can locate a cause.
Last training i had all students fail a job... in hindsight my storage had a defective array which could be the cause.
Bart Pellegrino,
Technical Account Manager - EMEA
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by foggy »

Hi Bart, contacting support will also be a step forward in troubleshooting the issue.
Zew
Veteran
Posts: 365
Liked: 80 times
Joined: Mar 17, 2015 9:50 pm
Full Name: Aemilianus Kehler
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by Zew » 2 people like this post

For those experiencing undefined function call id 54.

https://www.veeam.com/kb1813

I didn't catch this KB until after I had a Veeam support tech tell me to do these steps.

On the VM that gets this error; Open services.msc, stop VeeamVssSupport service. open cmd (as admin); sc delete VeeamVssSupport; rerun the job.
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by pirx »

SyNtAxx wrote: I still continue to be plagued by this problem. So much so, that I cant run automated backups anymore. I basically have to babysit my backups 1 by one. This issue needs a resolve. I will have to migrate all my backups off SOBR(s) since the placement logic seems to get overloaded and fail out almost 100% of my jobs pointed at it. The SOBR is now at 16TB free of a total f 80TB. The less space in the SOBR the more frequent the issues.

-NIck
We just started our backups with veeam (9.5 U1) and see this error too. We currently use 2 SOBR, each 100TB. The the beginning everything was working as expected, I started a couple of jobs and they successfully wrote to the SOBR (which has only one extend at the moment). Yesterday I started all ~20 jobs so that the initial full can run over weekend. There were already a couple of jobs running at that time. These jobs are still running but most of the others failed with

Polling scale-out backup repository extents: Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file.
03/02/2017 21:36:41 :: Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file.

Both SOBR have 50+ TB free at the moment. There are some large jobs but with dedup and compression all backups should fit int the SOBR.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: B&R v9 oddities (maybe bugs)

Post by Gostev »

Probably a different issue, as this thread is almost 1 year old now. Please contact support for investigation.
Post Reply

Who is online

Users browsing this forum: Baidu [Spider], Semrush [Bot] and 168 guests