B&R v9 oddities (maybe bugs)

[DaveWatkins]

I've just upgraded from 8 to 9 and the upgrade itself was fine.

I then wanted combine two repos into a scale out repo but everytime I tried I got an error saying it couldn't access a folder on one of the repos. The folder was from an old backup job that had been removed and deleted a long while ago. I'm assuming this was associated with an orphan record in the database about that job that wasn't removed properly. I ended up completely removing that repo from B&R and re-adding it. After that I could build the scale out repo with it.

The other issue I've seen so far is that an Exchange job that was running perfectly fine in 8 gave me this error on one of the Exchange servers

"Error: BlobCall: Undefined function id: 54"

and failed. That's not one I've seen before and if it persists I'll log a support job for it (the job is still running).

Also, config backups don't appear to be able to go to scale out repos, which means you'll need to keep a separate repo for them if you were planning on running just one big scale out repo. Still adding a 1GB repo on a proxy somewhere isn't a huge issue

Other than that, things have gone well. Per VM backup chains are on and working. EMC SAN's are added and having the console locally is very nice (although I'm not sure about the flat look and the "blandness")

[nielsengelen]

Do keep in mind that you could create a folder one of the 2 repositories and add that one to VBR again to save your configuration.

For example on a Windows server:
E:\BACKUPS (this will be part of SOBR)
E:\CONFBACKUP (this won't be part of SOBR)

[Dima P.]

DaveWatkins,

"Error: BlobCall: Undefined function id: 54"

This one needs to be investigated with support team for sure. If the issue persists, kindly, open a support case and post the case ID this thread.

config backups don't appear to be able to go to scale out repos

That’s by design but the workaround Niels suggested sound good.

[DaveWatkins]

Yeah, the config backup one wasn't really anything to worry about for me, just thought I'd put it here so others saw it. Plenty of ways to work around it easily.

The Scale out issue was interesting, but I'm guessing was more related to v8 not cleanly removing the DB records about that folder than anything to do with v9. Still waiting for that exchange job to finish so I can retry it and see f the error persists

[Delo123]

Dave,

just updated to V9 and after reading your post i ran an incremental backup of our Exchange 2010 DAG Cluster, all fine here...

[Dima V.]

DaveWatkins,

I then wanted combine two repos into a scale out repo but everytime I tried I got an error saying it couldn't access a folder on one of the repos. The folder was from an old backup job that had been removed and deleted a long while ago. I'm assuming this was associated with an orphan record in the database about that job that wasn't removed properly. I ended up completely removing that repo from B&R and re-adding it. After that I could build the scale out repo with it.

Can you please clarify. It may help us to improve our code.
If I understand right you had orphaned job which was pointed to to that Repository, had some runs in the past, but there wasn't any registered backups (even failed) neither under Disk->Backup nor under Disk->Imported?

[DaveWatkins]

That's correct, there was no record of the old job anywhere in the UI. All the Repo backups had been removed from the GUI and the job had been deleted months ago.

I've just stumbled upon a new issue but this one will likely need to go via support. My Scale out repo consists of 2 repos, one had ~500GB free and the other has 9TB. I've created a new job with 12 VM's abd about 350GB of data in the VM's total.

When running the job the first 3 VM's start backing up fine, but the remaining all fail immediately with:

" Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file. "

There is another backup in progess to this scale out repo going but there is plenty of free space available and even if the other job is "locking" a repo even the server with 500Gb free would be enough to store the initial backups of these VM's

Retrying the job after it finished and then the next 3 VM's start backing up fine and the remaining 6 fail again. I'll keep retrying until I have all 12 VM's with backup files on the repo and then launch a normal backup and see what happens

[foggy]

This last one indeed looks unexpected, however can be caused by the space estimation algorithm used during backup to scale-out repository. So please contact technical support with this for confirmation.

[I.Makaryev]

Retrying the job after it finished and then the next 3 VM's start backing up fine and the remaining 6 fail again. I'll keep retrying until I have all 12 VM's with backup files on the repo and then launch a normal backup and see what happens

Hello, Dave!
Do you use VMware?
Two questions more - do your VMs have thick or thin provisioned disks? How much "used storage" space is VMware reporting for that VMs?
Thanks.

[DaveWatkins]

Yes, these are all running on VMWare 6U1 and the VM's are thick provisioned. Used storage space is a mix but in all cases the full size of the provisioned disks (between 25GB and a couple of hundred GB depending on the VM in question)

[foggy]

Large thick disks with small amount of actual data can easily confuse backup repository space estimation. Job logs will tell whether this is the cause of this behavior.

[DaveWatkins]

It isn't/wasn't. The disks didn't have a lot of free space in fact most of them are at about 80% full. Also if that were the case surely retrying the job wound't have fixed it (or at least allowed it to start 3 more backups).

To reiterate, none of the VM's by themselves would have filled any of the underlying repo's, In fact the total size of all the VM's in the job would not have filled up one of the underlying repos even if all the disks were at 100% usage

[foggy]

Anyway, we're interested in further investigation of this behavior, so contacting support is recommended.

[SyNtAxx]

Anyway, we're interested in further investigation of this behavior, so contacting support is recommended.

I'm interested also, as I seem to be seeing similar behavior(s).

'Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file. '

There is currently 38.5TB free of 64TB.

-Nick

[DaveWatkins]

It didn't ever open a ticket for this as we have some other stuff going on taking my time and I didn't have the time to dedicate to it (new SAN, wireless upgrade etc), and I could work around it for now. Given the ease with which I can replicate it I figured it could wait. Nick, you're welcome to open a ticket yourself

[SyNtAxx]

It didn't ever open a ticket for this as we have some other stuff going on taking my time and I didn't have the time to dedicate to it (new SAN, wireless upgrade etc), and I could work around it for now. Given the ease with which I can replicate it I figured it could wait. Nick, you're welcome to open a ticket yourself

I'll have to open a case. I just started using the scale out repos and I need to be able to trust them. They work great and let me load balance, but an awful lot of free space available to fail a job over.

Nick

[SyNtAxx]

Veeam Support - Case # 01691757

[SyNtAxx]

New Errors for Scale Out Repo(s) on my build: about a dozen of these: Error: Failed to check extent 'Dcspwvbs002_v400_vol07' heartbeat

[jl@fire-eater.dk]

was this ever resovelved

i get Error: BlobCall: Undefined function id: 54 on a 2003 server and on a 2008 server after upgrading to veeam 9

[SyNtAxx]

Veeam Support - Case # 01691757

'Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file. '

There is currently 38.5TB free of 64TB.

-Nick

I still continue to be plagued by this problem. So much so, that I cant run automated backups anymore. I basically have to babysit my backups 1 by one. This issue needs a resolve. I will have to migrate all my backups off SOBR(s) since the placement logic seems to get overloaded and fail out almost 100% of my jobs pointed at it. The SOBR is now at 16TB free of a total f 80TB. The less space in the SOBR the more frequent the issues.

-NIck

[nielsengelen]

was this ever resovelved

i get Error: BlobCall: Undefined function id: 54 on a 2003 server and on a 2008 server after upgrading to veeam 9

Windows 2003 server support was dropped (as can be found in the release notes) so you can't use them anymore for your Veeam infrastructure such as proxy, repo,... Please consider upgrading the server to 2008/2012 as Microsoft also dropped 2003 support.

[foggy]

You can still backup VMs running Windows 2003 though, this is what Jesper is doing, I suppose. Anyway, with any technical issue it is always recommended to contact technical support first.

[ITP-Stan]

BlobCall: Undefined function id: 54

On 2 VM's in the job, the rest (10+/-) ran fine.
Windows 2012 R2 repo, b&r server & proxy all-in-one.

I'll have to open a support case by the looks of it.

[BartP]

This error i see more and more lately.
Gut feeling is that it related to the performance/pending time of the current Job. Especially Full Backups seem to be more prone to the issue.
Whenever a job takes very long to finish and other machines are pending a while, they tend to stop with Function Call ID 54.
When you manually start the same job afterwards, it runs fine.

I am currently testing with a couple setups to see if i can reproduce the error and more importantly see if i can locate a cause.
Last training i had all students fail a job... in hindsight my storage had a defective array which could be the cause.

[foggy]

Hi Bart, contacting support will also be a step forward in troubleshooting the issue.

[Zew]

For those experiencing undefined function call id 54.

https://www.veeam.com/kb1813

I didn't catch this KB until after I had a Veeam support tech tell me to do these steps.

On the VM that gets this error; Open services.msc, stop VeeamVssSupport service. open cmd (as admin); sc delete VeeamVssSupport; rerun the job.

[pirx]

I still continue to be plagued by this problem. So much so, that I cant run automated backups anymore. I basically have to babysit my backups 1 by one. This issue needs a resolve. I will have to migrate all my backups off SOBR(s) since the placement logic seems to get overloaded and fail out almost 100% of my jobs pointed at it. The SOBR is now at 16TB free of a total f 80TB. The less space in the SOBR the more frequent the issues.

-NIck

We just started our backups with veeam (9.5 U1) and see this error too. We currently use 2 SOBR, each 100TB. The the beginning everything was working as expected, I started a couple of jobs and they successfully wrote to the SOBR (which has only one extend at the moment). Yesterday I started all ~20 jobs so that the initial full can run over weekend. There were already a couple of jobs running at that time. These jobs are still running but most of the others failed with

Polling scale-out backup repository extents: Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file.
03/02/2017 21:36:41 :: Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file.

Both SOBR have 50+ TB free at the moment. There are some large jobs but with dedup and compression all backups should fit int the SOBR.

[Gostev]

Probably a different issue, as this thread is almost 1 year old now. Please contact support for investigation.