Inaccessible backup

[Dr.Gerry]

Warning - this post contains information that will probably lay bare my complete lack of knowledge!!

I am notionally the Veeam administrator in my organisation. My experience with Veeam has been to run the install (many years ago) by clicking Next, Next, Next, set up our backup jobs (many years ago) and then luxuriate in my blissful unawareness as I got my emails every day telling me that the backup jobs have run successfully. I know the backups are working because we have been able to restore individual files from backups on occasion, but I have no level of 'expertise' by any means.

Anyhoo.....today I was checking whether a certain folder on a drive on one of our VMs had been backed up. According to the logs from last night's run of the backup job, 22Gb was read from the drive, and Veeam reported success, so I started a 'restore guest files' job just so I could peruse the backup. I got a bit of a shock when I saw that this particular drive appeared to have NOT been backed up - it was not listed as a drive in the 'restore' window. All the other drives from that server WERE available. I then spent the next few minutes changing my trousers (this is an EXTREMELY important drive), and then I started looking a bit more closely.

Long story short......in the Event Viewer of our Veeam server there was an error "A corruption was discovered in the file system structure on volume C:\VeeamFLR\SERVERNAME_1eca8162\Volume4." Once again I changed my trousers (lucky I carry lots of spares!) and started searching the forum. I noticed that there was another one or two mentions of this sort of problem. On one of them, the final post was "has the upgrade to 9.5 solved the issue", but there was no response.

Does anyone know whether an upgrade to 9.5 will fix this issue? We are currently running 9.0, but at my boss's request, we have stayed with that version. I know it is unsupported now, but as I said previously, I am by no means a Veeam expert (you would probably be being extremely polite if you even called me a 'low level user'!), and the thought of upgrading fills me with trepidation. We are in a situation of rapid and rather large change at my organisation (4 separate Windows domains from 4 separate organisations being merged into one), and the last thing I want to do is screw up our backups.

So I suppose my question(s) are - 1) I am guessing that all this means that the disk in question IS being backed up, but is inaccessible to restore jobs? 2) does anyone know definitively if an upgrade to 9.5 will solve this issue ? 3) Is there any way to fix this without an upgrade? (I know an upgrade is the best option, and I am guessing that it will probably be a case of Next, Next, Next-ing again....in which case it is prefectly suited to my level of expertise) 4) Does anyone know any good trouser vendors - I am starting to run out!

Any and all assistance gratefully accepted!

[Andreas Neufert]

I know... not funny right?
It is a bit hard to understand what happened. There are multiple ways you can restore data with Veeam. As this is not a support forum I strongly suggest that you open a Veeam Support ticket (you can use Severity 1 if you are in a hurry restoring the data). They can analyse logs and help you.

[Dr.Gerry]

Thanks for your reply. I don't think I can open a ticket for an unsupported version, can I?

[Andreas Neufert]

9.0 went out of support in January 2019...
But before you update this environment to get support, I would say open a support ticket and follow what they request to avoid that the update will cause additional issues with the backup file.

[foggy]

Gerry, please be aware that you can only open support cases if you have active maintenance. Also, what you could do to check if the data is actually there is trying to perform a full VM restore.

[Andreas Neufert]

Maybe full VM restore or Instant VM restore (to other VM name) and boot up the OS might help to get access to the data.

[Mildur]

Had a similar error like you.

in the Event Viewer of our Veeam server there was an error "A corruption was discovered in the file system structure on volume C:\VeeamFLR\SERVERNAME_1eca8162\Volume4."

In my case, the Source Partition before the Backup in the vm was corrupted.
A chkdsk inside the windows vm has shown me that on the partition multiple blocks was corrupted.

As soon Veeam FLR mounts the restorepoint, the corrupted partition is mounted in the path C:\VeeamFLR\********\volume

[Dr.Gerry]

Thanks everyone for all the assistance. I tried lodging a ticket initially, but when it told me that 9 is unsupported I figured I wouldn't be able to proceed.

I'm not sure about running chkdsk…...the server in question is rather large and it will probably take days to run. Also, it is a critical server. We have an application team that looks after the application it runs, but they will wash their hands of the issue, and I don't want to risk messing things up, because they won't give me any assistance if anything goes wrong.

Instant VM restore probably isn't an option either - we are running right on the margins, space-wise, and we really don't have any storage to which I could run the restore (yes, as you can gather, my workplace is pretty messed up at the moment...)

[Andreas Neufert]

Instant VM Recovery do not really need space... the idea is to boot the server without network from the backup and do not migrate it to production... Then you can see if you can access the files as needed or run chkdsk there. When you can access the data you can copy it to a new added vmdk and mount the vmdk later to your production VM.

There are as well other options that depend on your situation like you can use instant VM recovery and repair the disk with chkdsk and then remount the disk to your original VM and then use storage vmotion to migrate the data. (you need space when you want to keep the original vmdk disk.

[Dotmax]

Hi to all, just my 2 cents..
While you are investigating and fixing the root cause of the problem, maybe you can install and configure a veeam agent for windows backup (if your OS is supported), just to be sure that the critical data are in a safe location.

If you have space, in the guest OS you can:
- add a newly created disk
- replicate the data to the new disk
- run a test backup job
- ensure that the new disk is correctly backed up
- change the shares to the new disk, (and/or swap the local drive letters if you need fixed local paths)
After that you can remove (or leave it unused) the old drive.

Hope it helps.

[YouGotServered]

Dr. Gerry,
What kind of VMDK are you backing up, and what version of Windows is your backup server (I believe we really only need to know the Mount server, but I'm assuming that you have a simple deployment, where the Mount server is the VBR server)?

For example, if you are backing up a 2012 R2 server and the disk you are trying to browse is using 2012 R2 dedupe, then you need to be using a mount server to browse those backup files that also has that dedupe role installed (on the same version of Windows server as well). Same thing goes if your guest VM is a 2016 / 2019 ReFS volume, or other Windows Server dedupe volume.