Corrupted backup copy

Availability for the Always-On Enterprise

Corrupted backup copy

Veeam Logoby mkretzer » Wed Nov 29, 2017 5:51 am

Hello,

Case 02404420

yesterday we had a outage of the connection to our remote backup copy target. The outage happened as some copy jobs copied data and some merged. Also, we had a defective disk (of a RAID 6) the day before in one of the remote locations storages.
There was no outage of power and connection between the copy target host and its storage system. Just the WAN link.

Today the two health checks that ran both showed corrupted backups.

How likely is it that the corruption comes from the WAN link failure?

Markus
mkretzer
Expert
 
Posts: 367
Liked: 76 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Corrupted backup copy

Veeam Logoby v.Eremin » Wed Nov 29, 2017 11:13 am

Abrupt stop of merge operation might leave backups in an inconsistent state. Support team should be able to provide the precise root cause after debug log investigation. Thanks.
v.Eremin
Veeam Software
 
Posts: 13882
Liked: 1038 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Corrupted backup copy

Veeam Logoby foggy » Wed Nov 29, 2017 11:24 am 1 person likes this post

Next job run will repair corrupt restore points and finalize the merge correctly, this will happen automatically.
foggy
Veeam Software
 
Posts: 15394
Liked: 1142 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Corrupted backup copy

Veeam Logoby mkretzer » Wed Nov 29, 2017 8:48 pm

@foggy:
Thats what i thought. In the last 4 years we had crashes, disable of copy jobs in the middle of HC / Merge / Transfer but never had our monthly health check show an inconsistency.
We checked more backups and every backup file on the copy storage target shows inconsistencies for some VMs - even after merges.

Right now we are health-checking another backup copy on the same target repo server but a different (also different brand) backend storage.
mkretzer
Expert
 
Posts: 367
Liked: 76 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Corrupted backup copy

Veeam Logoby foggy » Thu Nov 30, 2017 10:43 am

Then please check with support.
foggy
Veeam Software
 
Posts: 15394
Liked: 1142 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Corrupted backup copy

Veeam Logoby mkretzer » Thu Nov 30, 2017 5:46 pm

"whether it was caused by network issues - No, it wasn't"

Sounds like the backend storage is the problem - that would be nice that way we can continue to trust Veeam backup copies...
mkretzer
Expert
 
Posts: 367
Liked: 76 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Corrupted backup copy

Veeam Logoby Gostev » Sun Dec 03, 2017 11:29 pm 1 person likes this post

Inconsistent restore point due to failed/aborted merge is not a problem as it will be repaired automatically by the next run (and previous restore points will not be affected regardless). As long as the backup storage itself is solid and not causing the corruption, there's nothing to worry about. This is why it is important to determine the root cause with our support. Thanks!
Gostev
Veeam Software
 
Posts: 21648
Liked: 2433 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Corrupted backup copy

Veeam Logoby mkretzer » Mon Dec 04, 2017 5:36 am

Will do. We also suspect the storage now.

Problem is it is a copy target and support still has not provided us information how we can run the Validator without a full Veeam installation...
mkretzer
Expert
 
Posts: 367
Liked: 76 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Corrupted backup copy

Veeam Logoby Eamonn Deering » Mon Dec 04, 2017 7:52 am

If you suspect the storage then let us know what storage are you using and firmware level? Someone out there may be able to through some light on the matter.
Eamonn Deering
Service Provider
 
Posts: 26
Liked: 1 time
Joined: Wed Feb 29, 2012 1:42 pm
Location: Dublin, Ireland
Full Name: EamonnD

Re: Corrupted backup copy

Veeam Logoby mkretzer » Tue Dec 05, 2017 8:32 am

It was a MD3600f from Dell. But that storage has no data corruption protection. Now we migrated to a HUS110 from Hitachi which has all the protections. Still, the same backup shows as corrupted (we re-seeded it from the main site).

Main site backup was also checked and shows no error.
mkretzer
Expert
 
Posts: 367
Liked: 76 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Corrupted backup copy

Veeam Logoby foggy » Tue Dec 05, 2017 1:01 pm

So you've copied the verified backups from source, use them to seed the job. and they are shown as corrupt on target?
foggy
Veeam Software
 
Posts: 15394
Liked: 1142 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Corrupted backup copy

Veeam Logoby mkretzer » Tue Dec 05, 2017 2:02 pm

No, not immediately. The first check was ok.

Here is what happened exactly:

- Backup copy job mirroring backup from the primary site (last active full ~ 4 weeks ago, health check yesterday) to a remote site. Remote site had a Dell MD3600f (RAID 6 config) as backup storage.
- Last week this MD3600f had an strange warning that a disk in in per-failure state.
- My co-worker sadly replaced the wrong disk from the same RAID 6. We waited for the raid rebuild to finish before we replaced the real defective disk. My theory is that the first rebuild was done with the data from the partly dead disk
- After all the rebuilds we saw corruption on nearly all health checks we did on the remote site
- Since we had a free Hitachi HUS110 on the main site we used temporary backup copy jobs to create a new backup seed on that storage
- We then brought that storage system to the remote site, created & rescanned the repo, disabled the copy jobs, removed the old, corrupt backups copied from configuration, targeted the copy jobs to the new repo and mapped the backups
- All copy jobs did an initial health check without errors
- After two days (and the first merge) another health check showed a corruption in the same VM file again as on the old storage
- A heath check on the primary site showed no issues with the primary site backups

The only thing i can think of if that Veeam tried to "heal" the corrupt blocks on the remote site after the backup target was replaced (we kept the same jobs).
mkretzer
Expert
 
Posts: 367
Liked: 76 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Corrupted backup copy

Veeam Logoby foggy » Tue Dec 05, 2017 4:36 pm

Looks strange, please ask support to investigate.
foggy
Veeam Software
 
Posts: 15394
Liked: 1142 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Corrupted backup copy

Veeam Logoby mkretzer » Tue Dec 05, 2017 4:40 pm 1 person likes this post

They are on it. Case 02409456 now.
mkretzer
Expert
 
Posts: 367
Liked: 76 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Corrupted backup copy

Veeam Logoby mkretzer » Wed Dec 06, 2017 7:25 am

One more thing: How should the "auto-healing" of the copy job data work? I do not have the feeling it is doing its job!
mkretzer
Expert
 
Posts: 367
Liked: 76 times
Joined: Thu Dec 17, 2015 7:17 am

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot], NightBird, Yahoo [Bot] and 1 guest