Host-based backup of Microsoft Hyper-V VMs.
Post Reply
DBoose
Novice
Posts: 3
Liked: never
Joined: Jul 27, 2016 12:10 pm
Full Name: David Boose
Contact:

possible corrupted backup chain, sparce replies from support

Post by DBoose »

Hello,

case #01887963, opened on Wednesday 8/24 @ 9:10am CST.

I've always had great experiences with veeam support, but for whatever reason this time.. the experience has been less than satisfactory. I'm not a Veeam pro by any means and am still learning the ins and outs of the system.

I created a new backup job for some 'secure' vms a few weeks ago. at this time, there is only one VM in the job, but I was planning on adding more to this job later. im looking through the back up statuses and veeam is showing 'incomplete' for the initial full back up but then went almost 2 weeks showing successful incrementals until earlier this week. im assuming a merge needed to happen and this is where it failed. the error was saying I may have a corrupted backup chain. testing a file level restore for the affected vm confirmed that something is indeed wrong with the chain.

"cannot mount result. only partial restore point for FIB [guid] can be created. cannot process remote mount commands. read: end of file. cannot open a remote backup for the file level restore. agent failed to process method {mount.genericmoount}"

@ 10am the same day; the support rep inquired if I had had any updates pending. ironically at the time I had staged windows updates("installed" within windows) then scheduled a reboot for after hours like I've always done. that window of time was less than 8 hours. the backup that threw the chain corruption error ran in between that and the reboot. then @ 4pm-ish on Wednesday they replied, "If you think about it, it really is kind of risky to let a server run for any length of time between installing updates and rebooting. I advise you to force an active full backup so that you have a good backup chain." I've never had any issues with the other 30 servers im backing up when im patching them in this manner. Advising a client to only do an active back up would then leave the old chain (wasting space) in my repo, doesn't address cleaning up the bad chain, and doesn't address what the root of the issue was. this part kind of rubbed me the wrong way. oh well. I just want to fix this and get back to the other projects happening here.

I replied yesterday at 7:22 am CST on Thursday and requested instructions on how to create a new back up chain, and then how to delete the bad one so im not wasting space on my repository with a bad backup chain. I haven't heard anything back from the support rep in over 36 hours. id really really like to get this done today before the weekend. I have had to remove a bad chain once before, but would like a refresher on how to do it again so I can make sure im not missing any steps..

I'm hoping to learn why veeam let the incrementals go for this long if the initial full backup was incomplete/not viable, and what may have caused the 1st full backup to fail like this in the first place. and if it was a misconfig or something on my part.


thanks in advance for any assistance, and sorry if I am coming across a little harshly over this issue.
DBoose
Novice
Posts: 3
Liked: never
Joined: Jul 27, 2016 12:10 pm
Full Name: David Boose
Contact:

Re: possible corrupted backup chain, sparce replies from sup

Post by DBoose »

the irony is killing me! lol

as soon as I sent this in, got a reply!

"You are in kind of a Catch-22 situation with updates. It's never easy to find time to reboot in a production environment. Apparently, there is a known issue where there is some risk of corrupted data if a backup job runs during that time after updates have been installed and the reboot. It can cause VSS writers to act erratically, for example. I suppose it makes a difference what exactly is updated.

Here is a link to our documentation for managing backups and replicas: https://helpcenter.veeam.com/backup/hyp ... ckups.html

Basically, what likely happened is that the update possibly cause the VSS writers to act erratically during a merge or some other part of the backup operation that involves the full. It depends on how your retention is configured.

I hope this answers your questions."

this unfortunately, does not answer my questions at all... is this really a known issue in veeam? I highly doubt veeam would let that issue become a caveat that cant be addressed. veeam has been rock solid for us and like I said before, we have always patches our servers in this way. if this "known issue" was the case, then I assume we would see this issue every single month when we apply MS patches to 30+ servers..

anyways, I've gone ahead and removed the bad backup job, the chain, and removed from config DB rescanned the repo. am now in the process of recreating the job. im hoping I didn't miss anything....
DBoose
Novice
Posts: 3
Liked: never
Joined: Jul 27, 2016 12:10 pm
Full Name: David Boose
Contact:

Re: possible corrupted backup chain, sparce replies from sup

Post by DBoose »

appears I have resolved the issue myself by manually removing the backup job from the config DB, manually remove the backup files from the repo, rescan repo, then recreate the job.

thanks
Post Reply

Who is online

Users browsing this forum: No registered users and 26 guests