Failed to obtain storage quota

[erbr]

Hi

We have recently had some problems with our Backup Copy Jobs that use Cloud Connect the job stops half way trough and i get the following error

Code: Select all

2/8/2016 1:00:04 AM :: Processing *** Error: Failed to obtain storage quota. Waiting has timed out.
Failed to download disk.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.

What does this actually mean i have tried to find out the error message but it seems there is no description of it anyware or at least im not finding it.

What i would like to know if the error is at our end our theirs.

We have another Cloud Provider and they work fine.

Support Case ID# 01688876

Regards

[veremin]

Have you already communicated with the Service Provider? Have them check the"Known Issues" topic in the service providers subforum.

This particular issue might be related to usage of specific "Region and Language" Windows settings.

Thanks.

[chavron]

erbr, what have you found out? I'm seeing the same error to my cloud provider, but only during overnight backups. The backups run fine and are successful if I perform them during the day when presumably most people aren't backing up to any cloud repository due to it being during production hours.

My setup:
- 11 VMWare Backups of Windows servers via Backup Jobs (not backup copy jos) to 4 TB repository - approx 800gb of data total for full, single backups of those machines
- 14 restore points (only currently at anywhere from 3-7 restore points due to continued failures)
- Each job is Per-VM and chained one after the other
- 1GBPS upload speed
- Veeam 9 both on the desktop and the cloud provider's repository is Veeam 9-enabled as well

Usually, less than a gigabyte of data is transferred, then the transfer hangs and generally times out, moves on to the next chained task, which also times out. Sometimes, if it's a VM with not much changed data (less than 500mb), the backup will be successful. During the day, I can get around 800-900 megabytes of data transferred per minute due to the speed of our connection.

This is the error I see:
Error: Failed to obtain storage quota. Waiting has timed out. Failed to download disk. Reconnectable protocol device was closed. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

Veeam Support reviewed my logs, and found the following:

<46> Info [AP] (4bcc) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
<42> Info [AP] (dd18) output: --asyncNtf:--busy: Target backup repository is overloaded.

Based on that and my admittedly-limited knowledge of Veeam errors, it seems that the cloud repository has run out of resources and is closing connections. I'm currently working with the cloud repository provider and Veeam support to try to get this resolved, but I just wanted to know if anything has been resolved here regarding erbr's situation.

Support case is 01697447

[veremin]

As far as I can see, your case has been just escalated. So, let's wait and see what support engineer of higher tier says about the problem. Thanks.

[JBEGON]

Hello,

I encounter the same problem.
Do you have news VEEAM support ?

Thank you

[foggy]

Joel, there's a hotfix for this issue available from support, so please contact them directly to verify the issue and get the hotfix, if applicable.

[chavron]

Was this hotfix included in the 9.1 release of Backup and Recovery? If so, it didn't solve my problem. My provider also updated their end with the latest release on Friday, and the problem persists. They've also tried connecting me to a completely separate datacenter (physically) with a clean repository. That did not solve the problem. Both Veeam support and the cloud provider have extensively looked over my Veeam configuration and haven't found anything that could be causing this.

If the problem was consistent per-VM, or failed at the same point every time, I could see that there could be something configured wrong. The fact that it's very intermittent (it will fail one day or one hour, but be successful on the next run) makes this very difficult to troubleshoot.

[foggy]

Hi Chris, yes, the update should include the mentioned hotfix, so if the issue persists after installing it, please contact support for verification.

[adapterer]

One of our clients has this problem too, even after we have both applied the hotfix.

However, it seems to work during the day??? Just not at night...

[cffit]

Same problem here. We are on v9 with the update 1, and our provider I know is up to date. We still get intermittent and random errors about being unable to determine storage quota. I believe our cloud provider worked with VEEAM on this a few months ago and thought they had the fix in place now. I'm on the phone to open a support case on it. It's definitely not a configuration issue as the timing, job and frequency of it is all very random and never the same.

[veremin]

I'd appreciate, guys, if you provide me with your ticket numbers. This way I will be able to follow the cases and QA team for assistance, should need be. Thanks.

[adapterer]

Hi Vladimir, 01767172

Still waiting on my client to do some testing though :/

[Gostev]

I just talked to lead dev and QC folks for Veeam Cloud Connect functionality earlier today to get a status update. This issue appears to be tenant specific.

For example, at one of our largest Veeam Cloud Connect providers with a few hundreds of tenants, there is only 1 tenant having this issue. The issue is quite persistent for this particular tenant, and yet never ever happened for hundreds (sic!) of other tenants sharing the same Cloud Connect infrastructure. So, I am now suspecting tenant's networking equipment or something along those lines... hopefully, we will be able to find some similarities between all affected tenants (across multiple service providers).

Thanks!

[bbuchan]

We also have a tenant experiencing this same issue. Veeam 9 Update 1.

[foggy]

Hi Bryan, please open a case with our technical support. Thanks.

[Ghost59]

Hi all,
same problem.. version 9.0.0.1491

[foggy]

Hi Philippe, same recommendation, please contact technical support for a closer look.

[hyvokar]

Has anyone yet resolved this? I have atleast two tenants with this problem.

[nielsengelen]

As mentioned in this thread best would be to contact support for assistance with the issue.

[veremin]

Has anyone yet resolved this?

The previous case has been closed due to no response received from a customer. Thus, as Niels's recommended, you'd better open your own ticket. Thanks.

[hyvokar]

Opened a ticket and got a patch. However I'm not sure should I install the patch on cloud service or the source veeam server. # 01846614

[veremin]

Cloud Connect server (one cloud service is running on). Thanks.

[chavron]

I have an update which will probably discourage some. Cliffs notes are that Veeam needs to figure this out.

Refer to my previous posts on Page 1 for background.

After approximately 4 months of problems, Veeam finally got around to issuing a patch to my Cloud Connect provider: OffsiteDataSync. This patch was to fix the problems I and others have mentioned in this thread. However this did not fix the problem.

Fast Forward another couple months, Veeam finally issues a patch on June 8th which fixes my problem!! Everything worked great for a week, and then I started getting a new error:
Error: Cloud repository (repo name) is unavailable due to the service provider restriction.

The owner of the company called me, and informed what happened. The Veeam patch, while it fixed my problem, caused multiple errors and failures for most of their other customers, so they had to revert their environment back to the previous state.

I can't say enough nice things about OffsiteDataSync. Their team has worked diligently in trying to take care of this, but in the end, ODS has offered to refund me money for the months I've paid and haven't been able to use the service without problems, and I'm left with Veeam Cloud Connect backups in limbo since Veeam can't seem to issue a proper fix for this problem. As is evident in this thread, the problem is not limited to one or two users.

Veeam, the ball is in your court. What are you going to do in order to take care of this?

I can't afford to waste further time on this, so I've been exploring alternative options for cloud backups of my VM environment. From what I've experienced, Veeam Cloud Connect is far from being ready for use in a production environment.

I'll be happy to speak to whomever would like to call me. Send me a private message for contact info.

I'd say this needs to be escalated to another level, but as it's already been escalated to the Head of North American Support, I'm not sure where it can go from there.

[veremin]

Hi, Chris,

Let me clarify the situation internally, I will update the topic, as soon as I get more information.

Thanks.

[ober72]

Hi folks,

I have seen this as well but it is intermittent and seems to cure itself. The next time I see it I will put in a case.

cheers

[Chris-DE]

Hi all,

Same Thing.
Provider and Tenant Version Veaam 9 U1.

Problem:
Failed to obtain storage quota. Waiting has timed out. Failed to download disk. Reconnectable protocol device was closed. Failed to upload disk. Agent failed to process method. {DataTransfer.SyncDisk}.
on several VMs during Cloud Backup (all others suceed).

Support ID:
01851581

Will update here if theres a Resolution in sight.
After all the Replys here i am a bit confused, if this is "Tenant-Specific", then why have there been Hotfixes for Cloud Providers in this?

[veremin]

Veeam, the ball is in your court. What are you going to do in order to take care of this?

Originally “Failed to obtain storage quota” error had been attributed to another issue that was fixed in Update 1.

As it turned out, there wasn’t a single cause of “Failed to obtain storage quota” error. Because of that and because of its fickle nature, the problem has been under investigation for several months.

We finally released the hot fix addressing the “Failed to obtain storage quota” on June, 1st.

At the same time the hot fix was applied, ODS experienced excessive slow-downs and sometimes a complete lack of throughput. During that period their users might have seen jobs failing with “Cloud repository is unavailable due to the SP restriction” error.

ODS have been running fine, since the infrastructure was reverted back to pre-fix state, and currently do not have any further problems.

In the meantime, we’re investigating the case closely (with lead dev and top tier support engineer assigned to the ticket) to find out whether there is a correlation between the experienced slow-downs and newly-issued hotfix or the problem has been caused by side matter.

Conclusion cannot be made, until both logs and process dumps are thoroughly analyzed.

Thank you for your understanding; I'll keep you updated on the results of our findings.

[hyvokar]

Cloud Connect server (one cloud service is running on). Thanks.

Applied the patch and so far almost everything seems promising..

[veremin]

Kindly, keep me updated and either post here or reach me directly (via PM) in case the issue re-occurs. Thanks.

[jsalinas]

Quick question.... How do I get the mentioned patch?

We are in the middle of deploying a new Cloud Connect Infrastructure so this patch may come handy for us.

Thank you