Discussions specific to the VMware vSphere hypervisor
pinkerton
Enthusiast
Posts: 78
Liked: 4 times
Joined: Sep 29, 2011 9:57 am
Contact:

V6 Direct SAN Mode: Unable to allocate processing resources

Post by pinkerton » Dec 01, 2011 5:41 pm

Hi,

we just moved from VBR 5 to 6. Backing up particular VMs from one particular datastore using Direct SAN access fails with the message:

"Unable to allocate processing resources. Error: No backup proxy is able to backup this VM. Check processing mode settings on proxies."

Other VMs on this datastore are backed up just fine. We also had no problems with VBR 5. When moving affected VMs to a different datastore, they are backed up fine as well. Backing them up using NBD works fine too.

- VBR is installed on a physical W2K8R2 SP1 server. It's a one-server installation.
- We removed 5.0 and did a clean install of 6.0, including SQL DB.
- We set the Backup Proxy to use Direct SAN access without failback and performed a manual selection of connected datastores.
- The VMware environment consists of vCenter Server 5 and three ESXi 5 hosts.
- Datastore LUNs are zoned to the VBR server. Nothing was changed prior the move to VBR 6.0.

Do you have any idea what could cause the Direct SAN backup for some VMs fail with the mentioned message?

I've also created a support ticket and sent logfiles - [ID#5157744].

/edit: Just saw the post about reported problems in V6: Maybe I am facing the issue that is marked as "ISSUE 1". Since if failover to NBD is enabled, there's no error message.

Thanks
Michael

doms
Lurker
Posts: 1
Liked: never
Joined: Dec 01, 2011 6:00 pm
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by doms » Dec 01, 2011 6:04 pm

Hi.. Veeam Support does this mean we should hold off on upgrading until the new build is out? Seems like lots of people are getting this san issue?

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by Gostev » Dec 01, 2011 6:20 pm

These are not support forums, so you won't get a response from Veeam support here :wink: however, I can help.

If you are using direct SAN access mode on your job, there is a chance that you may get affected. The cause is not yet known, we could not confirm the issue on any SAN storage we have in possession. At least, it would be best to wait until the research is done.

If you use any other mode, then the issue above does not matter.

pinkerton
Enthusiast
Posts: 78
Liked: 4 times
Joined: Sep 29, 2011 9:57 am
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by pinkerton » Dec 02, 2011 8:21 am

Just a quick update for everyone who's interested/affected: After having the daily backup running last night I found out that this is totally random. In my post above I wrote that only VMs from one datastore are affected - that's not true, other VMs from different datastores are affected as well.

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by Gostev » Dec 02, 2011 10:06 am

Please check out the workaround I have just posted in the sticky on v6 issues.

pinkerton
Enthusiast
Posts: 78
Liked: 4 times
Joined: Sep 29, 2011 9:57 am
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by pinkerton » Dec 02, 2011 10:16 am

Thanks, however, as written in my first post we have manually selected the datastores. The automatic selection doesn't work at all, even when doing a manual rescan before.

pinkerton
Enthusiast
Posts: 78
Liked: 4 times
Joined: Sep 29, 2011 9:57 am
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by pinkerton » Dec 02, 2011 2:47 pm

Had a webex session with support today - it's a different issue with us! Not a big problem since NBD works fine and now the backup is only slower. We'll see!

joost1981
Influencer
Posts: 16
Liked: never
Joined: Apr 22, 2009 10:42 am
Full Name: Joost Poulissen
Location: Roermond, The Netherlands
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by joost1981 » Dec 02, 2011 7:06 pm

Hi Michael,

Looks we are also experiencing the same issue. Please let me know if there is a fix?

Cheers,
Joost Poulissen

pinkerton
Enthusiast
Posts: 78
Liked: 4 times
Joined: Sep 29, 2011 9:57 am
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by pinkerton » Dec 02, 2011 7:09 pm

Hello Joost,

it's best to get in touch with Support as they need as much information as possible. What kind of SAN are you using? Possible HP EVA as well?

Thanks
Michael

joost1981
Influencer
Posts: 16
Liked: never
Joined: Apr 22, 2009 10:42 am
Full Name: Joost Poulissen
Location: Roermond, The Netherlands
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by joost1981 » Dec 02, 2011 7:17 pm

Michael,

I will do that. We are using EMC CX4-240.

Thank you for your quick answer,
Joost

EdwardPoll
Influencer
Posts: 17
Liked: never
Joined: Feb 09, 2011 9:03 am
Full Name: Edward Poll
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by EdwardPoll » Dec 06, 2011 11:55 am

We have got 2 x HP EVA8400 and 2 x EMC NS480.

Veeam is failing to allocate proxy resources to 'random' VMs across luns from all 4 devices. Only have one proxy/backup server and have tried set the proxy to 'manual' for the storage luns it has access too....made no difference in this environment. This an upgrade from v5.

Anyone had any luck yet on any other workarounds or potential fixes?

pinkerton
Enthusiast
Posts: 78
Liked: 4 times
Joined: Sep 29, 2011 9:57 am
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by pinkerton » Dec 06, 2011 12:04 pm

Veeam Support is still working on the issue, just got out of a phone call with them. It's best to get in touch with them as well since I guess they are happy about any information about this they can get.

EdwardPoll
Influencer
Posts: 17
Liked: never
Joined: Feb 09, 2011 9:03 am
Full Name: Edward Poll
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by EdwardPoll » Dec 06, 2011 12:29 pm

Already got a ticket open and they are trawling the logs.

Just examining something I've noticed in the logs...I'll let you know my findings..

EdwardPoll
Influencer
Posts: 17
Liked: never
Joined: Feb 09, 2011 9:03 am
Full Name: Edward Poll
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by EdwardPoll » Dec 06, 2011 12:53 pm

Gostev

Issue 1
~20 support cases are reporting issue with direct SAN access mode failing over to network mode with certain SAN makes and models for some VMs (job completes successfully, with the corresponding warning). If you are not using direct SAN access mode, ignore this issue.
[UPDATE] Workaround: under the Backup Infrastructure node, right-click the proxy server to open its properties, click Choose next to connected datastores, select Manual, click Add and pick all datastores which are supposed to be directly accessible by the given proxy.

This workaround doesn't work for us, is there something else we need to do once i switch the datastores to manual?

pinkerton
Enthusiast
Posts: 78
Liked: 4 times
Joined: Sep 29, 2011 9:57 am
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by pinkerton » Dec 06, 2011 12:58 pm

There currently is no solution to the problem, as I said Veeam is working on the issue. The workaround doesn't help either, in fact the automatic selectio doesn't work at all for us, the manual selections works but with the mentioned problems.

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by Gostev » Dec 06, 2011 1:58 pm

Edward, the issue discussed in this thread is different than ISSUE 1, so that workaround is not applicable.

EdwardPoll
Influencer
Posts: 17
Liked: never
Joined: Feb 09, 2011 9:03 am
Full Name: Edward Poll
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by EdwardPoll » Dec 06, 2011 3:24 pm

Is it? Having spoken to Pinkerton directly it's not....stopping the faildown to NBD on the proxy causes the error as it THINKS it can't see the luns/datastore. Turning the NBD faildown on again just results in it stepping down according. Not sure about setting it to manual detection as that hasn't worked in our case which is affecting datastores on 4 different arrays (2 different manufacturers).

Could you point out the finer details of both the different issues? I have a ticket with support and they say it is a known issue (singular) and said "just wait for a fix, Gostev will post it on the forums". If there are 2 issues that appear to be the same to the user how can I tell which I have, and why didnt support explain that to me (ID#5158692).

Thanks

Ed

pinkerton
Enthusiast
Posts: 78
Liked: 4 times
Joined: Sep 29, 2011 9:57 am
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by pinkerton » Dec 06, 2011 3:41 pm

As I said, "Automatic Seletion" doesn't work at all for us. This is different than the issue described in the sticky thread, where Automatic Selection does work but not with all VMs. If Automtic Selection doesn't work for you at all either, I think you have the same issue like we have.

EdwardPoll
Influencer
Posts: 17
Liked: never
Joined: Feb 09, 2011 9:03 am
Full Name: Edward Poll
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by EdwardPoll » Dec 06, 2011 4:07 pm

Well on 'automatic' it looks like its just failing straight down to NBD.

Network Team are having kittens (put politely)...but the monitor graphs look good....well I've certainly never seen them that high...

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by Gostev » Dec 06, 2011 5:10 pm

I am hearing that the fix addressing all issues around direct SAN access is almost ready, so hopefully you all will get it tomorrow through support. Thanks.

EdwardPoll
Influencer
Posts: 17
Liked: never
Joined: Feb 09, 2011 9:03 am
Full Name: Edward Poll
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by EdwardPoll » Dec 07, 2011 8:15 am

Great news....that's for the update.

pinkerton
Enthusiast
Posts: 78
Liked: 4 times
Joined: Sep 29, 2011 9:57 am
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by pinkerton » Dec 07, 2011 11:12 am

I got a private hotfix which has solved the problem for me. It is now being tested with other customers and once it is confirmed that it really solves everything, there will be a new build of the software.

EdwardPoll
Influencer
Posts: 17
Liked: never
Joined: Feb 09, 2011 9:03 am
Full Name: Edward Poll
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by EdwardPoll » Dec 07, 2011 11:22 am

Great, at least one of has a working installation then!

Hopefully they will release the new build/hotfix soon, as with 300+ VMs NBD is not a nice way to be running...

pinkerton
Enthusiast
Posts: 78
Liked: 4 times
Joined: Sep 29, 2011 9:57 am
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by pinkerton » Dec 07, 2011 11:38 am

I think we are isolated cases, most of the users don't seem to have problems. And it's always a little risky to move new major releases to production a few days after release, you actually sound like being a little to hard with Veeam here ;-) Anyway, it's indeed good that it is working now.

BTW in my post above I meant that it is NOW being tested, not NOT :-)

EdwardPoll
Influencer
Posts: 17
Liked: never
Joined: Feb 09, 2011 9:03 am
Full Name: Edward Poll
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by EdwardPoll » Dec 07, 2011 11:53 am

Dont have this issue on our test system, surprisingly enough everything worked fine there. Must be the right kind of datastores on the right kind of luns on the right disk arrays.............oh hang on they are the same in both environments!!

Set the new record for bandwidth utilisation last night though, topped out around 5.5Gbps....

aweiss
Enthusiast
Posts: 33
Liked: never
Joined: Jul 26, 2011 6:22 am
Full Name: Alexandre Weiss
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by aweiss » Dec 07, 2011 3:44 pm

Hi All
I have the same issue and workaround doen't help
But when I vmotion the VM to another ESX, it gets backed up with no problem
If i vmotion the VM back to the original ESX, the problem is back and the VM cannot be backed up.
ESX (with console) 4.1 on 4 hosts. Backup only runs well on 4th ESX. When vMotion is done to ESX 1,2 or 3 VM cannot be backed up anymore. I confirm, for me, it's not a problem of datastore but more a problem in the communication between Veeam/vCenter/and ESX

Rgds
Case opened today 5159136

Alex

EdwardPoll
Influencer
Posts: 17
Liked: never
Joined: Feb 09, 2011 9:03 am
Full Name: Edward Poll
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by EdwardPoll » Dec 07, 2011 3:47 pm

Not in our case it's not, we've checked that. Well I'm pretty sure it's not anyways.

Gostev, any sign of that patch you mentioned yet?

Ed

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by Gostev » Dec 07, 2011 9:03 pm

Our support will reach everyone affected (those with support case open) as soon as the patch is ready. Last I heard a few hours ago, the patch was in testing. Thanks!

aweiss
Enthusiast
Posts: 33
Liked: never
Joined: Jul 26, 2011 6:22 am
Full Name: Alexandre Weiss
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by aweiss » Dec 08, 2011 7:34 am

Hope today is the patch day ! It's my last day at customer site :) to implement patch. After i'll have to do it remotely

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V6 Direct SAN Mode: Unable to allocate processing resour

Post by Gostev » Dec 08, 2011 1:56 pm

It looks like the patch did not work for 2 pilot customers, R&D is investigating. Sounds like SAN make and model related.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 16 guests