Concerns with Tape backups - Restores don't always work

[danc]

Support Case ID: 02055724

I have been running Veeam Backups to disk each night to disk and on the weekend, this is where I run my Full Weekly backups to disk and then this is backed up to Tape on using Veeam BR v9.5.

I check my Veeam notifications and it all backups come up as Successful. This is all fine, but when I come and try to do a restore from Tape, this is where I run into problems. The restore from tape is ok, but when I try to import the backup, this is where it does not import. This is a very big concern as where I work, we need these Tape Backups and there are times where the business will require a restore from Tape from say 6 months ago. I will be in trouble when I cannot restore a backup from Tape.

I have logged cases with Veeam support, but cannot get to the bottom of this. We have used VeeamR2 and tried to open a restored backup and it says that "All instatnces of the storage metadata are corrupted"

I am not confident that the Veeam backups to Tape 100% work as I have resorted to testing Tape restores each Monday after the weekend to check if they are successful. It is a hit and miss, sometimes a backup job restore works, and another week it does not.

Had a request last week to do a file restore of files from our fileshare from 6 months back, and what do you know, the Veeam Tape backup did not restore! Luckily, I still had backups from another application before we fully switched to Veeam which saved me this time! Anyone else have these issues with Tape Backup restores? I am not confident with this as the backup notifications say it was successful, but when doing a test it was not!

[Shestakov]

Hello Dan,
Unfortunately "All instatnces of the storage metadata are corrupted" error means that backup is not restorable and doesn`t point out the exact reason of failure. Usually it caused by hardware corruption.

This is a very big concern as where I work, we need these Tape Backups and there are times where the business will require a restore from Tape from say 6 months ago. I will be in trouble when I cannot restore a backup from Tape.

Could you remember if you upgraded the product since the backup date? If yes, what was the initial version?

By the way, do you use recoverability techniques such as Surebackup for disk backups?
Thanks!

[danc]

I'm running 9.5 Update 1, but have been having tape restore issues as far as I know. The problem is that it does not have a pattern. It is very random, as in 1 week a tape backup job will restore and import fine, and the next week it does not import. The tape backups always comes up as successful after the backup to tape is run which gives me the impression that all is well. However, when I do a 'test' restore from the tape backup, it will be successful in the restore phase. It is when I click on Import Backup that nothing happens. I kind of have to hold my breath when I click on Import Backup and hope that it imports the backup to Veeam. It is a 'hit and miss' scenario at the moment.

I have been working with Veeam support, but have not been able to find out why this is happening. I use VeeamR to check its status and when they do not import I get the following error:


Anyone else experiencing this issue? Any help to resolve this would be much appreciated!

[veremin]

I'm wondering whether you've tried to validate the source backup files by running either restore or Surebackup activity. It seems that there might be a problem with backup repository and already corrupted files got copied to tapes. Isn't that a case by any chance? Thanks.

[danc]

I haven't tried to validate the source backup files with Surebackup yet.

However, the backups to disk are able to be restored without issues. It's the tape backups that when I try to restore is a 'hit and miss' prospect which is a worry.

[veremin]

Just to be on the safe side I'd run Surebackup with storage level corruption verification enabled against backups in question and see how well it goes.

Anyway, has support engineer found already the root cause of the experienced behaviour? Your issue looks quite severe (failed restores), so, feel free to escalate a ticket whenever you think it's necessary.

Thanks.

[danc]

I'll look into running Surebackup - may need to get Veeam Support to check that I've set it up correctly. I have been asked byt Veeam Supportto use VeemR after I do the tape restore, before I import it to Veeam to see if comes up with any errors.
I have finished my weekend (Month end) backups from the weekend and will start testing a few of the tape restores and see if they work or not.

The support engineer has not found the root cause of the issue. It's been going on for a while now (Support Case ID: 02055724 with no resolution as yet. I'm a little stumped at the moment and a bit concerned as if my tape restores don't work, it may come and bite me later on when the business needs to do a tape restore and the restore doesn't work.

[veremin]

The original error suggests that most likely backup file is corrupted. In other words, either source repository or target tape device misbehaves, which leads to backup corruption.

The problem appears to be quite serious, so, case escalation might be a good idea.

Thanks.

[danc]

I'm still having issues with Tape Restores. It is still a hit and miss when I test the restores.

I find that when I commence the tape restore when I open up the Restore - Tape - Restore Files, I get the following error:

An item with the same key has already been added. at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at Veeam.Tape.UI.RestoreSourceSearchResultDialog.GetDirectoryFullName(Int64 id)
at Veeam.Tape.UI.RestoreSourceSearchResultDialog.GetFileListViewItem(Int32 index)
at Veeam.Tape.UI.RestoreSourceSearchResultDialog.itemsListView_RetrieveVirtualItem(Object sender, RetrieveVirtualItemEventArgs e)
at System.Windows.Forms.ListView.OnRetrieveVirtualItem(RetrieveVirtualItemEventArgs e)
at System.Windows.Forms.ListView.WmReflectNotify(Message& m)
at System.Windows.Forms.ListView.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Even if I close Veeam console and reopen it, and run the tape restore, that tape restore will not import to Veeam. I would test with VeeamR and would get the following:


Still stumped with this issue and still concerned with my tape backups as I can't be 100% sure if they will restore.
Still not solution or what might be causing this from Veeam Support (Support Case ID: 02055724)
Getting a bit worried with the tape backups as this issue is still ongoing and concerned as the business has asked how our restores to tapes are like at the moment using Veeam.
Will get Surebackup set up, but from looking at setting up Surebackup, the process is quite involved so not a quick easy set up, so will need a few things to get it to set up and test that it works properly.

Any help to get to resolve this would be much appreciated as I'm a little stumped at the moment!

[veremin]

I've already contacted our support team. Will update the topic, once I have more information.

[danc]

Thank you for doing this, much appreciated!

I have now set up Surebackup, but I have a feeing that won't reveal where the problem lies, as Surebackup only check from backup job to disk, but the issue is from disk to tape.
I have tested a few of my tape backup restores and still a hit and miss in regards to which ones don't import after the restore process in Veeam.

[Dima P.]

Dan,

Thanks for the heads up. I see that the case was escalated, so let wait for the log analysis.

[Dima P.]

Dan,

Had a conversation with support team today. Can you please try to restore the affected backup files to regular windows repository instead of deduplication repo and share the results? Thanks.

[danc]

Dan,

Had a conversation with support team today. Can you please try to restore the affected backup files to regular windows repository instead of deduplication repo and share the results? Thanks.

Hi Dima P.
I have done a Tape restore to the HP StoreOnce and it failed to import to Veeam.

I then did the same Tape restore to a regular Windows Repository and it still failed with the same results as before:


Problem still occuring with a hit and miss rate with Tape restores. And also still trying to get SureBackup to work, working with Veeam Support on this as well.

[reaperhammer]

I may have found a solution to this as I had a similar problem, the error was:
3/07/2018 4:29:32 p.m. Error Failed to save backup meta to 'E:|Restores|Backup - CRM + AXWeb on Tape|Backup - CRM + AXWeb on Tape.vbm'

I was doing a restore test of a restore point that contained 1full and 1 incremental backup, the VBK and VIB files are there but as expected no .vbm file.

I can import the VBK file manually using import backup, then it automatically imported the VIB file also.

This was with 9.5u3.