Veeam v9.5: Error: Full storage not found

[SyNtAxx]

Case # 01984571

I recently upgraded to version 9.5 from v9u2 and everything seemed to work fine after the fact except for one job failed with this error: Error: Full storage not found. I use SOBR for targets and what i noticed was the .vbm file seemed to be missing from one of the extents. Copying the .vbm into the extent did nothing as it was removed again once the job was retried. Running an active full on that Job created a new chain and job folder.

Fast forward to last night, eight (8) more jobs that all of the sudden fail and display the same error. The same symptoms appear to be present, missing .vbm in one of the SOBR extents. Something is afoot here and this time active full doesn't do the trick. I cant just simply create new jobs and let them rip, the issue is impacting ~25TB worth of vms and data and will clobber the back end array. We rotate fulls and schedules to keep everything happy.

What I also notice is that the job/data no longer appears under "Backups Disk" as if they job never existed but the data is clearly on the disk target. Rescanning the repositories does nothing to improve the problem.

Anybody else see this? I know it had been a known issue with previous releases and there was a hotfix or SQL fix for it.

Thanks.

[Mike Resseler]

syntaxx,

I see you have created a case so that is good. Please let us know the outcome here also. And I hope this is resolved quickly

Mike

[SyNtAxx]

Well, last night 97% of my jobs ran as fulls. .vbm files got nuked, even on jobs that had been recreated and run for 5 days. Something is really wrong. All my backups are MIA as far as veeam is concerned. All new chains, again.

[Mike Resseler]

That is not good news. Is support still working on the case?

[SyNtAxx]

Hoping for escalation to tier 2/3 for development and resolve.

-Nick

[ferrus]

Not sure if it's related to the problem above (the error is similar) - but I've had a recurring problem in both v9 and 9.5 - 'Full point not found'
Case #01970878

An Active Full resolved it, but the error has returned on other jobs.

[Gostev]

The fact of life is that the bug will not be escalated to R&D, investigated and fixed until you keep agreeing with your support engineer to just do an Active Full every time you have this issue.

[layte]

I had the exact same thing happen to every single one of my copy jobs after upgrading to 9.5

Veeam support were of little help and even tried to make out the jobs were disable for a number of days as the root of the issue. Not only were the jobs not disabled, why would them being disabled for a few days cause Veeam to trash them all and start again. In the end I had to create new storage repositories on the Data Domain and recreate all the copy jobs.

I am left being very disappointed in Veeam, and will be making some tough decisions when the support renewal is due.

[foggy]

Have you provided this feedback to support management (there's an option for this right on the customer support portal)? Otherwise how would we be able to react.

[ferrus]

OK, speaking personally, it's quite infrequent and for us, too easily remedied - to consider entering into a prolonged investigation with support.
That's why I just mentioned it here, to support what has already been posted above (the other poster's issues, seem more serious than our own).

However, the next time it happens - I can certainly put a call in, and request a deeper investigation.

[applc]

I had the exact same issue twice now. I have a brand new install of 9.5, just installed and configured over a week ago. This is a completely new green-field setup. It had been running great for several days. Then yesterday a job failed with Error: Full storage not found. Since we aren't using the system fully, I deleted the jobs, repositories, and all data. Then I recreated everything and now that job is running fine. Then today I had the exact same thing happen to another job. For now I am going to reactivate the backups on my old system, and call Veeam support tomorrow to troubleshoot further. In my case the backup disk data simply disappears. I rescan the repository and it completes saying 1 skipped. But no data appears. Since the storage is an Exagrid appliance, I can't easily view the file contents in the repository. We will see what support says.

[applc]

I opened a ticket with Veeam support. Case # 02007660. The Veeam engineer suspected it's possible the Exagrid unit may have changed the modified date on the .vbk file. But he admitted he wasn't entirely sure this was the cause. He asked for more logs, and left the ticket open. Nevertheless I will be working with Exagrid to investigate the possibility that it may have somehow modified the Veeam backup files. Moving forward, the Veeam engineer had me create a new job and start a whole new chain of backups for the failed job. He also showed me the original chain, and all the backup files, were still accessible if need be for restores.

[hyvokar]

Hi,

Did anyone got this problem resolved? Run in to same error today with 9.5 update4.

[foggy]

I've checked all the cases mentioned in this thread and all of them were closed after running active fulls. So no actual reason of the issue, plus they are from over 2 years ago and different releases. So, I recommend you to contact support for an investigation. Thanks!

[hyvokar]

Did contact, but it really didnt take me anywhere.
I believe the problem was that I changed the storage type to per-vm backup chains, but cannot tell for sure since the gui didn't give me enough info about what was missing.

[johnlong]

Was this resolved?

[foggy]

Nothing new in this regard, please contact technical support directly.

[Ci2Group]

we have a similar issue after upgrade from 9.5b to 10. We are a service provider and a client is receiving this error. The data set is several TBs in size so an active full is less than desirable.
Active sev 1 ticket 04133261 . We are having issues with many of our clients with different errors but this error is now the error we get after upgrading .

[atsrl]

we a have a similar issue on a brand new v10 cloud connect environment. Active sev2 ticket 04225838.
Did you have any workaround?

[foggy]

I've checked the latest case mentioned above and it was closed due to no response from the customer. Let's see if our engineers could track the issue down within your case.

[RubinCompServ]

Has anyone had any success on this? One of our customers has started to experience this on their VCC jobs. As they currently are using 18.5 TB of their 20 TB quota, they don't have the luxury of running new Active Fulls (not to mention the required bandwidth). I've opened case #04473901 as a Sev 1.

[PetrM]

Hi David,

Please continue to work with our support team as the error is generic and can have different root causes and workarounds depending on the specific case.

Thanks!

[clovischalegre]

I have the same problem with B&R v10. Several days have passed and no solution in the ticket I opened. # 04693610

[foggy]

Hi Clovis, the case is in Portuguese so I cannot make much sense of it but I do see that our team is investigating the log files, so please allow them some time.

[fojtp]

this should help:
- disable affected backup job
- edit job + enable GFS, save job
- again edit job (without starting job) + disable GFS, save job
- start job

[PetrM]

Hi Pavel,

Not sure it can help in every single case as the error "Full storage not found" is a generic one which means that a record about backup file that contains full point is not found at the level of our configuration database. The only correct way of working with the issue is to open a support case and let our engineers to investigate debug logs.

Thanks!

[EcoboostPerformance]

This has been a persistent issue. For us on versions 11, & 11a. It causes backups to fail. I really hope this bug gets squashed.

[Gostev]

There are currently no known issues like that in V11, so most likely it's not a bug but rather some environment-specific condition.

[EcoboostPerformance]

There may be no known issues but we have submitted 2 support issues with this on v11.

[PetrM]

Hi Ryan,

Basically, the error can be thrown when the job cannot find a starting point of the incremental chain when the job validates backup chain structure in Veeam configuration database. The root cause can be different in every specific case so it must be investigated by our support team. You may share a support case ID with us so that we can check that the investigation moves into the right direction.

Thanks!