Failed to create processing task for VM

[mrhcarlin]

Being a little cheeky here as I have raised a support case regarding this [ID#00691261], but thought it might be useful to post on the forum as I've been unable to find anyone having the same or similar issue.

Our problem stems from another error on 20-11-14 where one of our servers started complaining about corrupted and unreadable files (a VRB). The exact error was:

Code: Select all

Error: The file or directory is corrupted or unreadable. Failed to open [<.vrb path>] in readonly mode.

At the time, only this VM and its job were failing. Happened three times last week.

Over the weekend, however, when our Active Full was scheduled (last Saturday of the month), three jobs completed successfully, but then the following three errored with this:

Code: Select all

Failed to create processing task for VM <VM name> Error: Disk '<long string - looks like an ID of some sort>' not found.

Whenever I try to re-run the job, I get the same error and job failure.


Thanks in advance for any help.


EDIT: We're running 8.0.0.817.

[foggy]

Were there any changes performed to the said VM between successful and failed job runs?

[mrhcarlin]

The original failure we had (the corruption/unreadable error) occurred immediately after a chkdsk was run on the server. The fact that the error alludes to corruption is disconcerting because we've never had problems with this server in the past. It was rebooted (unscheduled) and the chkdsk started automatically. The backup that took place a few hours later, at night, failed with that error and has done each night ever since.

As for the other jobs failing...no changes have been made to the VMs. We have multiple jobs with 2-3 VMs per job. Ignoring the anomalous chkdsk VM for a second, all other VMs are fine so I don't see why they would fail. The jobs themselves have also been unchanged.

It's mainly the second error in my OP that I'd like to fix, as having 1 failed VM backup out of 15 fail isn't as mission-critical as having all 15 fail.

[foggy]

Fair enough. Let's wait for what our engineers could say after reviewing the log files.

[b.vanhaastrecht]

A customer of mine is experiencing the same after upgrading to v8. Jobs run fine for some days, and now suddenly the same error as above. Swithing from forced SAN transport to NDB temporarely solves the problem. LUN's are presented to the Veeam server, nothing changed on that side.

[veremin]

Hi, Bastiaan, if there have been any changes introduced in the environment, kindly, make your customer open a ticket with our support and provide them with the require logs. Having logs at hand should allow us to find the root cause as soon as possible. Thanks.

[mrhcarlin]

Just to update...

Ksenia responded to my ticket advising the following: Backup Infrastructure > Backup Proxies, then, in properties for each backup proxy, change 'Connected datastores' to 'Manual selection', choosing all of the datastores you are backing up from. This resolved our issue instantly. We've had an active full and an incremental go through successfully, without warning or error, for all of our backup jobs since Monday evening.

Would be interesting to know if this is bug or a change in v8, as we've had 'Automatic selection' set since we deployed Veeam.

Anyway, case closed at least!

[b.vanhaastrecht]

Multiple customers of ours with fixed SAN transport are experiancing the exact same error. Mostly after the Veeam server has rebooted or the Veeam services restart. It has nothing to do with change on storage/presentation. Setting the datastores to fixed on the proxy solves the problem.

This must be a known bug and should be placed on known issue list.

[zmedgyesi]

Some problem with v8. Bakcups working for a while, then the same error message for all jobs. Changing proxy settings solved it, even without restarting any server or the server itself. Thank you.

[foggy]

The issue is confirmed from our side, shows up after upgrade and is planned to be fixed in the patch.

[Gostev]

According to the build notes, the fix was included in today's Patch #1 build.

[b.vanhaastrecht]

According to the build notes, the fix was included in today's Patch #1 build.

Hi, great! Where can we find this patch you are refering to?

[foggy]

Anton will mention in the weekly digest, once the patch is released (or you will be able to see the corresponding message in the sticky topic here on forums). It is not released yet.

[lillebi]

Hi, I have the same problem after i upgraded from Veeam v7 to v8.
Any news about the fix?

[foggy]

See my previous post.

[b.vanhaastrecht]

We have installed the patch "veeam_backup_8.0.0.917.patch1_setup.exe", rebooted the servers out of precaution, but the error is not resolved. Tested on multiple setups where this bug is active.

Code: Select all

24-12-2014 10:40:00 :: Failed to create processing task for VM xxxxxx Error: Disk '298ee4c5-a9d1-40f3-9f44-fb44baf704b0' not found. 

(note, this bug it not listed on the v8 known issue list while every SAN setup we manage has this bug )

[foggy]

Bastiaan, please let your engineer know about that. You may also try to re-add the proxy server in Veeam B&R console on one of the setups to check whether this clears the things up.

[Grodomin]

After hours of investigations I found this post -_-

Same problem here but only with one customer, yet:
Error: Failed to create processing task for VM [NAME] Error: Disk '056467c6-9a82-4a49-b3a5-1a2f0ef062df' not found.
Version: was v8, now patched to v8u1
Affectet Jobs: all jobs (backup & replica)
Affectet VMs: all VMs.

Updating did not solve the problem.
Re-adding proxy did not solve the problem.

We opened a support case but the logs are still uploading.
Hope to get a hotfix asap... :-/

[Vitaliy S.]

Don't forget to update your post with a case ID for future readers. Thanks!

[Grodomin]

Sorry, I forgot to post the ID and it seems I'm not allowed to edit my post.
ID#00716232

[Grodomin]

Sorry for the double-post, the support just solved the problem by manipulating the database.
The problem was that somehow the disk-ids changed and Veeam got stuck and only changed it in half of the databases...

Here is what he had done:

1. open the db in SQL-Mgt-Studio
2. create backup of Veeam database
3. edit the following tables:
- dbo.Disks
- dbo.PhysHost_Disk
- maybe there was a thrid one
4. compare the IDs between the databases and the error in Veeam
5. delete the wrong rows from dbo.PhysHost_Disk

After that the job seems to start, but I will wait until it is finished

But I would suggest that you don't try that by yourself - let the support do it

Thanks Veeam!

EDIT: new Problem, now it is stuck in trying to use HotAdd

[06Manu]

Problem fixed by :
Veeam 8 Patch #1 => not enough
+ backup proxy specified manually (backup jobs)
+ Transport mode forced to Virtual Appliance (backup proxies)
+ Manual selection of Connected Datastore (backup proxies)

[Grodomin]

@06Manu that didn't work out for me

at least i could see what is going wrong:
Veaam is trying to hotadd the disk of the first VM in the job...
...but not to the backup-server, it hotadds it to its own replica an gets stuck?!

[Grodomin]

Again a doublepost:

Support called me again in my homeoffice and we fixed it together:

Problem fixed by :
Veeam 8 Patch #1 => not enough
+ backup proxy specified manually (backup jobs)
+ Transport mode forced to Virtual Appliance (backup proxies)
+ Manual selection of Connected Datastore (backup proxies)

+ create new proxy for "127.0.0.1"
+ edit all jobs to force them to use new proxy
+ delete proxy "This server"
+ uninstall "Veeam Backup Transport"
+ create proxy "This server" ("Veeam Backup Transport" is reinstalled)
+ switch jobs back to "This server" Proxy
+ delete proxy "127.0.0.1"
+ switch jobs back to automatic selection of proxy (if used)

Don't know what exactly fixed it, but for the moment the job is running - lets hope it stays like that...

I will give you a feedback next week; have a happy new year!

[chrisBrindley]

HEADS UP Veeam.
Ran into this issue Wednesday, searched found post, applied Hotfix one, error still exists, had to manually add datasotres to all proxy servers, that fixed my issue.

[b.vanhaastrecht]

The suggested solutions are all work arounds, and Patch #1 does not solve it. We are working with Veeam to get this issue solved permenantly. The easiest way to get around this error is to set the datastores selection from automatic to fixed on the desired proxy.

[Grodomin]

Another engineer contacted me 10 minutes ago, it seems as if 2-level took over.

[rkovhaev]

After installing patch #1, you need to perform a rescan operation on your virtual infrastructure, and rescan will fix inconsistency in the configuration. If that doesn't help, feel free to open up a support case with us.

[Gostev]

+ Manual selection of Connected Datastore (backup proxies)

The easiest way to get around this error is to set the datastores selection from automatic to fixed on the desired proxy.

Sounds like rescan would have helped here. Please let me know if it does not.

[Grodomin]

no a normal rescan did not help after patching to v8u1 (22.12.2014)