Backup agent for Linux servers and workstations on-premises or in the public cloud
jgiacone
Novice
Posts: 8
Liked: 3 times
Joined: Jun 16, 2016 6:53 pm
Full Name: Joseph Giacone
Contact:

All instances of the storage metadata are corrupted

Post by jgiacone » Aug 31, 2016 2:29 pm

****
[error] All instances of the storage metadata are corrupted.
****

Has anyone else ran into this error during incremental backups? We have several Ubuntu workstations that we are testing the Veeam Agent for Linux on, and this error happens on only one of them. We are taking a full backup of the machine. The first backup runs fine. However, every subsequent backup, returns the above error after the volume snapshot is created, and the backup fails. If we created a new job, everything works fine, until the next backup, then the same error pops up. An interesting note, if we do a file level backup (instead of the entire system), and select just certain folders, this does not occur.

Another thing to note, the full backup that actually succeeds, cannot be restored to another machine. The machine reports the same metadata error during the restore. I've dug around, and can't find much about this, other than a Veeam KB article explaining what metadata is and how Veeam uses it, and apparently how you are screwed if it gets corrupted. Any help or insight is appreciated.

vmniels
Veeam Software
Posts: 2064
Liked: 458 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: All instances of the storage metadata are corrupted

Post by vmniels » Aug 31, 2016 3:12 pm

Could you try installing BETA2 which is available now and see if this still happens?
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

jgiacone
Novice
Posts: 8
Liked: 3 times
Joined: Jun 16, 2016 6:53 pm
Full Name: Joseph Giacone
Contact:

Re: All instances of the storage metadata are corrupted

Post by jgiacone » Sep 07, 2016 2:14 pm

Thanks, I installed BETA2 on all of our test machines. All are working fine, except the machine that had the 'metadata' issue. That machine no longer gets a the metadata issue, but now has a different issue:

Code: Select all

2016-09-07 06:00:02 Job started at 2016-09-07 06:00:02
     2016-09-07 06:00:02 Starting backup
     2016-09-07 06:00:10 Creating volume snapshot                                                                                                         00:00:02
     2016-09-07 06:00:23 [error] Retrieved less bytes from the storage [0] than required [5632]. Offset: [1457563136]. File: [/tmp/veeam/...
     2016-09-07 06:00:23 [error] Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://RestoreText_{defdb0e0-3b68-47...
     2016-09-07 06:00:23 [error] Agent failed to process method {DataTransfer.RestoreText}.
     2016-09-07 06:00:23 [error] Failed to perform backup
And this is in the logs:

*************************

Code: Select all

ERR |Job has failed.
[07.09.2016 06:00:23] <140363314071296> lpbcore| >>  |Retrieved less bytes from the storage [0] than required [5632]. Offset: [1457563136]. File: [/tmp/veeam/.../BackupJob1_2016-09-03T060014.vib].
[07.09.2016 06:00:23] <140363314071296> lpbcore| >>  |--tr:Failed to retrieve V4 data block. Block offset: [1457563136]. Compressed size: [5173]. Original size: [16739].
[07.09.2016 06:00:23] <140363314071296> lpbcore| >>  |--tr:Failed to retrieve restore block [0] for file [summary.xml].
[07.09.2016 06:00:23] <140363314071296> lpbcore| >>  |--tr:Failed to read block [0] from restore point
[07.09.2016 06:00:23] <140363314071296> lpbcore| >>  |--tr:Next asynchronous read request cannot be processed.
[07.09.2016 06:00:23] <140363314071296> lpbcore| >>  |--tr:Asynchronous data reader has failed.
[07.09.2016 06:00:23] <140363314071296> lpbcore| >>  |--tr:Failed to process conveyored task.
[07.09.2016 06:00:23] <140363314071296> lpbcore| >>  |Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://RestoreText_{defdb0e0-3b68-4703-990b-65424a1203af}]. CHMOD mask: [32528].
*************************

Any ideas? Again, this is only happening on one machine, but it also happens to be the same machine that had the corrupted metadata issue.

Thanks again,
Joe

PTide
Veeam Software
Posts: 4550
Liked: 374 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: All instances of the storage metadata are corrupted

Post by PTide » Sep 07, 2016 2:25 pm

Hi,

Please describe the machine that is failing the backup job, here are things we need to know:

- disks configuration (lsblk -af output)
- dmesg -T output
- Is any RAID present in the system
- Backup mode and target (shared folder/USB/Local)
- archived /var/log/veeam directory

Could you please upload everything to the dropbox?

Thank you

premeau
Novice
Posts: 3
Liked: 1 time
Joined: Aug 26, 2016 4:30 pm
Full Name: SPremeau
Contact:

Re: All instances of the storage metadata are corrupted

Post by premeau » Sep 07, 2016 2:34 pm

I am seeing this on a CentOS 6 x86 box backing up to a Windows 2012R2 CIFS share.

My Agent.Repair.Log:

Code: Select all

[07.09.2016 07:00:03] <3040869232> stg    | Checking metadata of the storage [/tmp/veeam/<desthost>Linux/<localhost> OmegaBackup/OmegaBackup_2016-09-06T203535.vbk]
[07.09.2016 07:00:03] <3040869232> stg    |   Opening storage [/tmp/veeam/<desthost>Linux/<localhost> OmegaBackup/OmegaBackup_2016-09-06T203535.vbk] in read-only mode.
[07.09.2016 07:00:03] <3040869232> stg    |     Opening storage file [HostFS:///tmp/veeam/<desthost>Linux/<localhost> OmegaBackup/OmegaBackup_2016-09-06T203535.vbk].
[07.09.2016 07:00:03] <3040869232> stg    |       Applying shared lock on file [/tmp/veeam/<desthost>Linux/<localhost> OmegaBackup/OmegaBackup_2016-09-06T203535.vbk].
[07.09.2016 07:00:03] <3040869232> stg    |     Creating size-quoted storage file. Current file size: [4646961152].
[07.09.2016 07:00:03] <3040869232> stg    |     Standard block size: "1048576".
[07.09.2016 07:00:03] <3040869232> stg    |     Loading metastore.
[07.09.2016 07:00:03] <3040869232> stg    |       Enumerating instances of metadata stored on disk.
[07.09.2016 07:00:03] <3040869232> stg    |         Version of the metadata stored in slot [4096]: [12].
[07.09.2016 07:00:03] <3040869232> stg    |         Version of the metadata stored in slot [53248]: [12].
[07.09.2016 07:00:03] <3040869232> stg    |       Storage current dedupe limit: [4194304].
[07.09.2016 07:00:03] <3040869232> stg    |       Initialized bank cache.
[07.09.2016 07:00:03] <3040869232> stg    |       Bank cache limits: 
[07.09.2016 07:00:03] <3040869232> stg    |         Dirty: [138]
[07.09.2016 07:00:03] <3040869232> stg    |         Clean: [162]
[07.09.2016 07:00:03] <3040869232> stg    |         Overall: [194]
[07.09.2016 07:00:03] <3040869232> stg    |       Dedupe disabled: [false].
[07.09.2016 07:00:03] <3040869232> stg    |       Prepared dynamic load bank factory, using bank cache.
[07.09.2016 07:00:03] <3040869232> stg    |       Loading snapshot from slot [4096].
[07.09.2016 07:00:19] <3040869232> stg    |       Loading snapshot from slot [4096]. Failed.
[07.09.2016 07:00:19] <3040869232> stg    |     Loading metastore. Failed.
[07.09.2016 07:00:19] <3040869232> stg    |   Opening storage [/tmp/veeam/<desthost>Linux/<localhost> OmegaBackup/OmegaBackup_2016-09-06T203535.vbk] in read-only mode. Failed.
[07.09.2016 07:00:19] <3040869232> stg    |   Closing storage file [HostFS:///tmp/veeam/<desthost>Linux/<localhost> OmegaBackup/OmegaBackup_2016-09-06T203535.vbk].
[07.09.2016 07:00:19] <3040869232> stg    |     Removing lock on file [/tmp/veeam/<desthost>Linux/<localhost> OmegaBackup/OmegaBackup_2016-09-06T203535.vbk].
[07.09.2016 07:00:19] <3040869232> stg    | Checking metadata of the storage [/tmp/veeam/<desthost>Linux/<localhost> OmegaBackup/OmegaBackup_2016-09-06T203535.vbk] Failed.
[07.09.2016 07:00:19] <3040869232> cli    | ERR |Failed to process method {Stg.CheckMetadataCorrupt}
[07.09.2016 07:00:19] <3040869232> cli    | >>  |Retrieved less bytes from the storage [0] than required [5246976]. Offset: [4662856704]. File: [/tmp/veeam/<desthost>Linux/<localhost> OmegaBackup/OmegaBackup_2016-09-06T203535.vbk].
[07.09.2016 07:00:19] <3040869232> cli    | >>  |--tr:Failed to parse metadata stored in slot [4096]
[07.09.2016 07:00:19] <3040869232> cli    | >>  |--tr:Unable to load metadata snapshot. Slot: [4096].
[07.09.2016 07:00:19] <3040869232> cli    | >>  |--tr:Failed to load metastore
[07.09.2016 07:00:19] <3040869232> cli    | >>  |--tr:Failed to load metadata partition.
[07.09.2016 07:00:19] <3040869232> cli    | >>  |--tr:Failed to open storage for read access. Storage: [/tmp/veeam/<desthost>Linux/<localhost> OmegaBackup/OmegaBackup_2016-09-06T203535.vbk].
[07.09.2016 07:00:19] <3040869232> cli    | >>  |--tr:Failed to check metadata of the storage [/tmp/veeam/<desthost>Linux/<localhost> OmegaBackup/OmegaBackup_2016-09-06T203535.vbk].
[07.09.2016 07:00:19] <3040869232> cli    | >>  |--tr:Storage check failed.
[07.09.2016 07:00:19] <3040869232> cli    | >>  |An exception was thrown from thread [3040869232].
And my lsblk output -- no RAID, just a single simple partitioned disk:

Code: Select all

NAME   FSTYPE LABEL UUID                                 MOUNTPOINT
loop0                                                    
loop1                                                    
loop2                                                    
loop3                                                    
loop4                                                    
loop5                                                    
loop6                                                    
loop7                                                    
sr0                                                      
sda                                                      
├─sda1 ext4         a4d3c8f0-2430-4515-99d3-c3747adea3e9 /boot
├─sda2 swap         8388d48a-d81a-4a03-be62-7fa47ef5b017 [SWAP]
└─sda3 ext4         9b5f9e0a-922c-4c20-9c95-d8d36b5f4e80 /

PTide
Veeam Software
Posts: 4550
Liked: 374 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: All instances of the storage metadata are corrupted

Post by PTide » Sep 08, 2016 12:36 pm

Hi SPremeau,

Could you please try to backup to some other destination? For example you can use USB drive or NFS share instead of CIFS.

Thank you.

jgiacone
Novice
Posts: 8
Liked: 3 times
Joined: Jun 16, 2016 6:53 pm
Full Name: Joseph Giacone
Contact:

Re: All instances of the storage metadata are corrupted

Post by jgiacone » Sep 08, 2016 1:46 pm

Hi, thanks for the response. The requested files have been uploaded here:

https://drive.google.com/folderview?id= ... sp=sharing

-There is no RAID present in the system
-Backup mode is 'Entire Machine' and the target is a CIFS share on an Isilon storage array

Thanks,
Joe

PTide
Veeam Software
Posts: 4550
Liked: 374 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: All instances of the storage metadata are corrupted

Post by PTide » Sep 08, 2016 3:05 pm

Backup mode is 'Entire Machine' and the target is a CIFS share on an Isilon storage array
I assume that all other machines backup to the very same CIFS share, under the same credentials, is that correct?

premeau
Novice
Posts: 3
Liked: 1 time
Joined: Aug 26, 2016 4:30 pm
Full Name: SPremeau
Contact:

Re: All instances of the storage metadata are corrupted

Post by premeau » Sep 08, 2016 8:24 pm 1 person likes this post

PTide wrote:Could you please try to backup to some other destination? For example you can use USB drive or NFS share instead of CIFS.
I was able to do a Full Backup and one incremental to an NFS share hosted by a OpenFiler VM.

Gostev
Veeam Software
Posts: 23215
Liked: 2970 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: All instances of the storage metadata are corrupted

Post by Gostev » Sep 08, 2016 11:46 pm

With B&R, this error would indicate an issue with backup storage, so as the very first troubleshooting test I would try backing up to a different target.

vmniels
Veeam Software
Posts: 2064
Liked: 458 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: All instances of the storage metadata are corrupted

Post by vmniels » Sep 09, 2016 6:48 am

If you mount the CIFS share on that server manually can you perform normal file copies to it (take some large folders) without a problem?

What about writing big files using 'dd' ? Any moment where an issue occurs like timeout or a like?
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

PTide
Veeam Software
Posts: 4550
Liked: 374 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: All instances of the storage metadata are corrupted

Post by PTide » Sep 09, 2016 1:58 pm

Joseph,

One more question - do other Ubuntu machines have the same OS version and backup to the same destination (CIFS on Isilon ) as the failing one?

Thanks

jgiacone
Novice
Posts: 8
Liked: 3 times
Joined: Jun 16, 2016 6:53 pm
Full Name: Joseph Giacone
Contact:

Re: All instances of the storage metadata are corrupted

Post by jgiacone » Sep 12, 2016 2:45 pm 1 person likes this post

Hi all,

Thank you for the replies. I will try an answer these in order:

1.) "I assume that all other machines backup to the very same CIFS share, under the same credentials, is that correct?"

-Partially. All of the other machines backup to the same CIFS share, but not all of them use the same credentials. The machine with the error does use the same credentials as two other machines, one with the exact same OS level (Ubuntu 16.04), and another Windows PC. Both of these other backups run without issue.

2.) "With B&R, this error would indicate an issue with backup storage, so as the very first troubleshooting test I would try backing up to a different target."

-OK, we can definitely try that. However wouldn't the other 10+ machines, several with the same OS, experience the same problem, if the target is the issue?

3.) "If you mount the CIFS share on that server manually can you perform normal file copies to it (take some large folders) without a problem?"

-Yes, I can copy very large files without issues to the CIFS share, if I mount it manually.

4.) "One more question - do other Ubuntu machines have the same OS version and backup to the same destination (CIFS on Isilon ) as the failing one?"

-Yes, I setup another Ubuntu machine, with the same OS level, that uses the same credentials, to better try and rout out this issue. That machine's backup has been configured identically and backs up flawlessly. A week of backups, no issue.

Again, thanks for all of the replies. Please let me know if any other information is needed.

-Joe

PTide
Veeam Software
Posts: 4550
Liked: 374 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: All instances of the storage metadata are corrupted

Post by PTide » Sep 12, 2016 3:34 pm

Yes, I setup another Ubuntu machine, with the same OS level, that uses the same credentials, to better try and rout out this issue. That machine's backup has been configured identically and backs up flawlessly. A week of backups, no issue.
The machine with the error does use the same credentials as two other machines, one with the exact same OS level (Ubuntu 16.04) <...> Both of these other backups run without issue
These two facts make me think that the reason might be somewhere inbetween the server and the shared storage. It might be a long shot but could you please swap network connections on two servers: the failing one and the other one that uses the same credentials and the same OS but does not fail backups?

Thanks

Gostev
Veeam Software
Posts: 23215
Liked: 2970 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: All instances of the storage metadata are corrupted

Post by Gostev » Sep 12, 2016 11:24 pm

jgiacone wrote:2.) "With B&R, this error would indicate an issue with backup storage, so as the very first troubleshooting test I would try backing up to a different target."

-OK, we can definitely try that. However wouldn't the other 10+ machines, several with the same OS, experience the same problem, if the target is the issue?
Yes, I would expect backups for all machines affected in the similar manner.

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests