V6 Direct SAN Mode: Unable to allocate processing resources

[pinkerton]

Hi,

we just moved from VBR 5 to 6. Backing up particular VMs from one particular datastore using Direct SAN access fails with the message:

"Unable to allocate processing resources. Error: No backup proxy is able to backup this VM. Check processing mode settings on proxies."

Other VMs on this datastore are backed up just fine. We also had no problems with VBR 5. When moving affected VMs to a different datastore, they are backed up fine as well. Backing them up using NBD works fine too.

- VBR is installed on a physical W2K8R2 SP1 server. It's a one-server installation.
- We removed 5.0 and did a clean install of 6.0, including SQL DB.
- We set the Backup Proxy to use Direct SAN access without failback and performed a manual selection of connected datastores.
- The VMware environment consists of vCenter Server 5 and three ESXi 5 hosts.
- Datastore LUNs are zoned to the VBR server. Nothing was changed prior the move to VBR 6.0.

Do you have any idea what could cause the Direct SAN backup for some VMs fail with the mentioned message?

I've also created a support ticket and sent logfiles - [ID#5157744].

/edit: Just saw the post about reported problems in V6: Maybe I am facing the issue that is marked as "ISSUE 1". Since if failover to NBD is enabled, there's no error message.

Thanks
Michael

[doms]

Hi.. Veeam Support does this mean we should hold off on upgrading until the new build is out? Seems like lots of people are getting this san issue?

[Gostev]

These are not support forums, so you won't get a response from Veeam support here however, I can help.

If you are using direct SAN access mode on your job, there is a chance that you may get affected. The cause is not yet known, we could not confirm the issue on any SAN storage we have in possession. At least, it would be best to wait until the research is done.

If you use any other mode, then the issue above does not matter.

[pinkerton]

Just a quick update for everyone who's interested/affected: After having the daily backup running last night I found out that this is totally random. In my post above I wrote that only VMs from one datastore are affected - that's not true, other VMs from different datastores are affected as well.

[Gostev]

Please check out the workaround I have just posted in the sticky on v6 issues.

[pinkerton]

Thanks, however, as written in my first post we have manually selected the datastores. The automatic selection doesn't work at all, even when doing a manual rescan before.

[pinkerton]

Had a webex session with support today - it's a different issue with us! Not a big problem since NBD works fine and now the backup is only slower. We'll see!

[joost1981]

Hi Michael,

Looks we are also experiencing the same issue. Please let me know if there is a fix?

Cheers,
Joost Poulissen

[pinkerton]

Hello Joost,

it's best to get in touch with Support as they need as much information as possible. What kind of SAN are you using? Possible HP EVA as well?

Thanks
Michael

[joost1981]

Michael,

I will do that. We are using EMC CX4-240.

Thank you for your quick answer,
Joost

[EdwardPoll]

We have got 2 x HP EVA8400 and 2 x EMC NS480.

Veeam is failing to allocate proxy resources to 'random' VMs across luns from all 4 devices. Only have one proxy/backup server and have tried set the proxy to 'manual' for the storage luns it has access too....made no difference in this environment. This an upgrade from v5.

Anyone had any luck yet on any other workarounds or potential fixes?

[pinkerton]

Veeam Support is still working on the issue, just got out of a phone call with them. It's best to get in touch with them as well since I guess they are happy about any information about this they can get.

[EdwardPoll]

Already got a ticket open and they are trawling the logs.

Just examining something I've noticed in the logs...I'll let you know my findings..

[EdwardPoll]

Gostev

Issue 1
~20 support cases are reporting issue with direct SAN access mode failing over to network mode with certain SAN makes and models for some VMs (job completes successfully, with the corresponding warning). If you are not using direct SAN access mode, ignore this issue.
[UPDATE] Workaround: under the Backup Infrastructure node, right-click the proxy server to open its properties, click Choose next to connected datastores, select Manual, click Add and pick all datastores which are supposed to be directly accessible by the given proxy.

This workaround doesn't work for us, is there something else we need to do once i switch the datastores to manual?

[pinkerton]

There currently is no solution to the problem, as I said Veeam is working on the issue. The workaround doesn't help either, in fact the automatic selectio doesn't work at all for us, the manual selections works but with the mentioned problems.

[Gostev]

Edward, the issue discussed in this thread is different than ISSUE 1, so that workaround is not applicable.

[EdwardPoll]

Is it? Having spoken to Pinkerton directly it's not....stopping the faildown to NBD on the proxy causes the error as it THINKS it can't see the luns/datastore. Turning the NBD faildown on again just results in it stepping down according. Not sure about setting it to manual detection as that hasn't worked in our case which is affecting datastores on 4 different arrays (2 different manufacturers).

Could you point out the finer details of both the different issues? I have a ticket with support and they say it is a known issue (singular) and said "just wait for a fix, Gostev will post it on the forums". If there are 2 issues that appear to be the same to the user how can I tell which I have, and why didnt support explain that to me (ID#5158692).

Thanks

Ed

[pinkerton]

As I said, "Automatic Seletion" doesn't work at all for us. This is different than the issue described in the sticky thread, where Automatic Selection does work but not with all VMs. If Automtic Selection doesn't work for you at all either, I think you have the same issue like we have.

[EdwardPoll]

Well on 'automatic' it looks like its just failing straight down to NBD.

Network Team are having kittens (put politely)...but the monitor graphs look good....well I've certainly never seen them that high...

[Gostev]

I am hearing that the fix addressing all issues around direct SAN access is almost ready, so hopefully you all will get it tomorrow through support. Thanks.

[EdwardPoll]

Great news....that's for the update.

[pinkerton]

I got a private hotfix which has solved the problem for me. It is now being tested with other customers and once it is confirmed that it really solves everything, there will be a new build of the software.

[EdwardPoll]

Great, at least one of has a working installation then!

Hopefully they will release the new build/hotfix soon, as with 300+ VMs NBD is not a nice way to be running...

[pinkerton]

I think we are isolated cases, most of the users don't seem to have problems. And it's always a little risky to move new major releases to production a few days after release, you actually sound like being a little to hard with Veeam here Anyway, it's indeed good that it is working now.

BTW in my post above I meant that it is NOW being tested, not NOT

[EdwardPoll]

Dont have this issue on our test system, surprisingly enough everything worked fine there. Must be the right kind of datastores on the right kind of luns on the right disk arrays.............oh hang on they are the same in both environments!!

Set the new record for bandwidth utilisation last night though, topped out around 5.5Gbps....

[aweiss]

Hi All
I have the same issue and workaround doen't help
But when I vmotion the VM to another ESX, it gets backed up with no problem
If i vmotion the VM back to the original ESX, the problem is back and the VM cannot be backed up.
ESX (with console) 4.1 on 4 hosts. Backup only runs well on 4th ESX. When vMotion is done to ESX 1,2 or 3 VM cannot be backed up anymore. I confirm, for me, it's not a problem of datastore but more a problem in the communication between Veeam/vCenter/and ESX

Rgds
Case opened today 5159136

Alex

[EdwardPoll]

Not in our case it's not, we've checked that. Well I'm pretty sure it's not anyways.

Gostev, any sign of that patch you mentioned yet?

Ed

[Gostev]

Our support will reach everyone affected (those with support case open) as soon as the patch is ready. Last I heard a few hours ago, the patch was in testing. Thanks!

[aweiss]

Hope today is the patch day ! It's my last day at customer site to implement patch. After i'll have to do it remotely

[Gostev]

It looks like the patch did not work for 2 pilot customers, R&D is investigating. Sounds like SAN make and model related.