File-level recovery from VM-data copied to tape

[Zep]

Hello,

this is the setup I have tried:

- VM-Backups of a Windows VM with guest filesystem indexing enabled.
Immediately after each VM-Backup a copy "Backup to Tape"-Job selecting the Backup Job as source is run.

Number of restore points for this test limited to 3 in the job definition.
Backups are run in this order: FULL, INC, INC, FULL, INC, INC

So at the end of the 6th backup, the first 3 backups are removed from the disk-repository and the repository shows 3 restore points.

All 6 Backups are still stored on tape and 6 restore points are shown, as expected.
All 6 restore points are shown in the Veeam Backup Enterprise Manager and all filenames can be searched for.

Then I have tried to restore a file via Enterprise Manager that is stored inside a backup on tape only.
I do understand that this is a block-based backup and it cannot be restored from tape directly.

From the log it looks like the restore process just tries to address the vkb/vib-file in the repository and fails,
but as explained above, it has been expired due to the number of restore points.

It looks the process does not attempt to retrieve the vbk/vib-files from tape and it does neither help if I manually restore the vbk/vib-files, re-read the catalog and then attempt to do the FLR again.


So, what is the correct way to do a file-level recovery using the search from Enterprise Manager when the data is on tape only?

Thank you for your thoughts.
Best regards, Zep

[Mildur]

You cannot do a OS Guest Item Restore from a „backup to Tape“ Job. Not possible with vbr, not possible with Enterprise Manager.

First, You need to do a „Backup Restore from Tape to Repository“. This needs to be done manually on the vbr server.
https://helpcenter.veeam.com/docs/backu ... ml?ver=110

After a rescan of the repo, restore the OS Guest Files with vbr and em will work.

[Zep]

Thank you,
I do understand that it is not possible to extract a file from a block-based backup on tape.
I used to work with other backup products like ex-HP's (now MicroFocus') Data Protector and I have been implementing storage solutions for 20+ years so I am well aware of the differences between file & block storage


But I am surprised that:
1. the software cannot trigger a restore of the backup data from tape to disk and then run the file recovery.

2. even if I put the backup back into the backup repository at the same location, for some reason the Enterprise Manager restore still fails and claims it cannot locate the backup.

3. Enterprise Manager appears to keep all catalog entries from old backups, even is the data is no longer in the Backup & Replication database.
No amount of synchronization seems to fix that.


It looks like you need 2 or more university degrees to recover a file that is stored inside a backup on tape

I must be missing something, but to me it appears a lot of work to manually find out which data to restore to disk first and then start the FLR and locate the file(s) again inside the VBR browser.


Thanks for your input & best regards,
Zep

PS: Switzerland... one of my favorite countries for travel

[Mildur]

3. Enterprise Manager appears to keep all catalog entries from old backups, even is the data is no longer in the Backup & Replication database.
No amount of synchronization seems to fix that.

Are you talking about the file index?
There are separated from the vbr database. You can configure the retention Setting here:

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

A little bit more information about this „historical index data“:
https://helpcenter.veeam.com/docs/backu ... ml?ver=110

PS: Switzerland... one of my favorite countries for travel

Thank you

[Dima P.]

1. the software cannot trigger a restore of the backup data from tape to disk and then run the file recovery.

Noted as improvement request. Thank you for sharing!

2. even if I put the backup back into the backup repository at the same location, for some reason the Enterprise Manager restore still fails and claims it cannot locate the backup.

Check that after backup file is restored from tape it is imported to Veeam B&R server added to Enterprise Manager. Such backup file should be displayed aside from tape and it should be possible to restore files via Enterprise Manager.

[Zep]

OK, I have run this test: FULL1, INC1.1, INC1.2, (set the max. number of restore points = 2), FULL2, INC2.1

After every Backup, I let run a chained automatic Copy Backups to Tape / Backup Jobs...

I have made a manual file copy of the repository files after INC1.2

----

For the restore, i went to Backups > Tape and ran "Restore backup from tape to repository..." where I chose the restore point of INC1.2

The files ended up in folder "JobName on Tape" and the restore session's log contained:
"Importing repository" with a green check mark.

----

Now, in the Veeam Backup Enterprise Manager I even tried the search filter with/ without "Search imported backups only".

It did not work when trying to restore when the files were in folder "JobName on Tape" and it did not work, when I moved toe files to folder just "JobName".

The restore session always says: "File does not exit."
Unfortunately it only shows the filename and not the folder in which it expects the file to restore from.

---

I've checked logfile: C:\ProgramData\Veeam\Backup\FLRSessions\Windows\FLR__'VMname'_\FLR__'VMname'_.log

For a successfull restore I see:

Code: Select all

::
[09.09.2021 08:33:07.207] <  8672> stg      |     Restore consolidation config:
[09.09.2021 08:33:07.207] <  8672> stg      |     Folders:
[09.09.2021 08:33:07.207] <  8672> stg      |     {
[09.09.2021 08:33:07.207] <  8672> stg      |       HostFS://E:\Backup\BCKVM_VMname\BCKVM_VMnameD2021-09-08T171814_D3D0.vbk[17742b2b-b25a-47c5-8b3e-787190f879ef (vm-60125)]
::

The unsuccessful entry looks like this:

Code: Select all

::
[09.09.2021 14:42:23.286] < 10900> stg      |     Restore consolidation config:
[09.09.2021 14:42:23.286] < 10900> stg      |     Folders:
[09.09.2021 14:42:23.286] < 10900> stg      |     {
[09.09.2021 14:42:23.286] < 10900> stg      |       HostFS://BCKVM_VMnameD2021-09-09T110517_B088.vib[17742b2b-b25a-47c5-8b3e-787190f879ef (vm-60125)]
::

As you can see, the folder path is missing in "HostFS://path_goes_here/BCKVM"

Ooops. What to do next?
I am using an NFR License to learn about Veaam.

Thanks for any suggestion.
Best Regards, Zep

[Zep]

Another test I thought I could do:
Used "Remove from configuration" to remove the metadata from VBR, but keep the files in the repository.
Then used "Import Backup..." with "Import guest file index" and made sure that the VBEM had synchronized catalog and database.

Result: file not found

[Dima P.]

Hello Zep,

I am sorry for the delay and thank you for updating the thread! Can you please clarify:

1. Do you see the missing file when starting the File Level Recovery from Veeam B&R and not via Enterprise Manager?
2. Did I get it right - you are running Veeam B&R v11?

Thank you in advance!

[Zep]

Hello Dima,

1. yes, the backup files (VBK,VIB) could be accessed from VBR

2. yes, this is VBR V11

Best regards, Zep

[Dima P.]

Thanks Zep! Can you raise a support ticket via customer portal and share case ID with me? If I am not mistaken it should be possible with NFR license.

[Zep]

Yes, will do so. I've noticed that VBR V11a is out so I think it is better to try it with that version.
Just need some time to run the tests again.

[Dima P.]

Let us know how v11a installation goes and if your notice any issues described in this posted in new latest version. Thanks!

[Zep]

Ok, I rebuild the whole system and did a fresh installation of V11a.

Then I ran Full, Inc1, Inc2, Inc3 backups with guest filesystem indexing.

After each backup, the backup data was immediately copied to tape (Tape Jobs - Backups to Tape - selection from Backup Job).


In the VBEM console I searched for a test-file made for Inc1 and I was able to restore it to the VM.

Then I removed the on-disk backup data from the repository (right-click - remove from configuration).

In VBEM I ran a synchronization and then tried a search for the same file. It was successfull.

When I started the restore I pretty quick got the message:
"Failed to validate file restore credentials: Oib info was not found"

--

I removed all old backups from the repository and moved the tape medium to the free pool.

For the next test I set the retention policy to 3 restore points to let Veeam remove the older files from the repository.

Then I ran Ful1 Inc1.1 Inc1.2 Inc1.3 Ful2 Inc2.1 Inc2.2

Before each backup I created a test-file on the client VM.

Then I made sure that catalog and database replication had run after the last backup.

A restore from the last 3 backups which is still in the on-disk repository is possible.

The attempt to restore a file from Inc1.1 which was on tape, but not in the repository did fail again with the same error message:
"Failed to validate file restore credentials: Oib info was not found"

--

Then I restored the data on tape back to the repository using the desired restore point (as expected, the full backup was restored as well).
The restore job said: "Importing repository" - fine.

Made sure that the database and catalog are replicated.

Searched for the file and tried to restore. . .

"Failed to validate file restore credentials: Oib info was not found"

[Dima P.]

Hello Zep,

Thank you for the update! can you please raise a support case and share it with me? We need to check the debug logs to identify the root cause.

[Zep]

Case # 05152161
screenshot and logs from the restore attempt provided in the call, thanks

[Dima P.]

Shared with RnD folks. Thank you!

[Zep]

This morning I got the message that the case is closed due to high Support Team load.

[Dima P.]

Hello Zep,

I am sorry for the delay, I've asked again RnD team to review this thread as soon as they go time, unfortunately last week was extremely busy due to the latest cumulative patch release. Please stay tuned, I'll update this thread once I hear back anything.

[Zep]

Hello Dima,
thank you. I will look in from time to time.
Best wishes for 2022 to everybody.