Error: There is no FIB

VMware specific discussions

Re: Error: no FIB [xxx.vmdk] in the specified restore point

Veeam Logoby Bunce » Fri Aug 17, 2012 12:40 am

Gotchya. Ta.
Bunce
Expert
 
Posts: 256
Liked: 8 times
Joined: Fri Sep 18, 2009 9:56 am
Location: Adelaide, Australia
Full Name: Andrew

[MERGED] There is no FIB [summary.xml] in the specified rest

Veeam Logoby nreutemann » Mon Sep 17, 2012 5:02 pm

Hello, I got this error message when the job try to do the synthetic full:

"TextFromTar failed Client error: There is no FIB [summary.xml] in the specified restore point. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://Tar2Text]. CHMOD mask: [0]."

I got other 14 jobs without problems, this one got 26 VMs.

The server is W2008 R2 and the Veeam B&R version is 6.1.0.205 (I installed the update this weekend, when the error appears).

I´m trying to create an Support Case, but when I try to upload the logs file, the upload don´t finish (the file got 1.2Gb).

There is any other way to upload the logs files?

Thanks in advance.
nreutemann
Enthusiast
 
Posts: 47
Liked: 6 times
Joined: Tue Mar 06, 2012 11:45 pm
Full Name: Nicolas Reutemann

Re: There is no FIB [summary.xml] in the specified restore p

Veeam Logoby Vitaliy S. » Mon Sep 17, 2012 8:22 pm

Nicolas,

Please open the ticket and inform your support engineer about the size of the log files and you will receive a link to the FTP server which can be used to upload this amount of debug logs files.

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19960
Liked: 1145 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Error: no FIB [xxx.vmdk] in the specified restore point

Veeam Logoby nreutemann » Mon Sep 17, 2012 8:44 pm

Ok. Thanks.

Support Case ID: 5216333
nreutemann
Enthusiast
 
Posts: 47
Liked: 6 times
Joined: Tue Mar 06, 2012 11:45 pm
Full Name: Nicolas Reutemann

Re: Error: no FIB [xxx.vmdk] in the specified restore point

Veeam Logoby gwilmington » Tue Oct 02, 2012 8:15 pm

I'm getting the same error myself.

SR - 5128407
gwilmington
Influencer
 
Posts: 10
Liked: 1 time
Joined: Tue Aug 14, 2012 2:05 pm

Re: Error: no FIB [xxx.vmdk] in the specified restore point

Veeam Logoby Gostev » Tue Oct 02, 2012 9:10 pm

Hi, what product version and build number are you using?
Gostev
Veeam Software
 
Posts: 21611
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Error: no FIB [xxx.vmdk] in the specified restore point

Veeam Logoby gwilmington » Wed Oct 03, 2012 1:49 pm

B&R version 6.1.0.205 (64bit)

My setup:

vSphere 5.0 U1
1 main backup server and 2 proxies all Win2k8R2 with in-guest iSCSI initiator on the iSCSI network
Netapp 2240-2 serving a 5TB LUN of SATA disk via iSCSI
5TB NTFS volume formatted with GPT
All VM LUNs connected in offline mode to the main Veeam server and proxies
Reversed incremental backups with monthly archived fulls

Setup seems to work fine for a few days then all of a sudden backups start to fail. Only thing that works is running Active Full to start a new chain. All previous restore points are unusable.

Was asked to move the repository to another disk array, but that's not possible as this is our main storage. Technician is escalating to another tier. I'm thinking about presenting the disk to the VM via SnapDrive instead of natively to see if that helps.

Edit: I found an event log entry showing the iSCSI attached LUN is corrupted and to run chkdsk. However 1 of 3 jobs completed when this happened. Seems like a NetApp issue but that doesn't make much sense that it worked for 1 of the jobs if the volume is having an issue. I'm wondering if there are issues with running NetApp VSC snapshots on a VM with an iSCSI guest and it could be causing a problem with the attached LUN. I removed this VM and the proxies from VSC snapshotting. This seems to be a perfect storm of small problems.
gwilmington
Influencer
 
Posts: 10
Liked: 1 time
Joined: Tue Aug 14, 2012 2:05 pm

Re: Error: no FIB [xxx.vmdk] in the specified restore point

Veeam Logoby Gostev » Wed Oct 03, 2012 2:43 pm

If your job was created at the time when the product had this bug (versions 6.0 or 6.1), then the issue may surface even with the latest version of the code. In this case, performing Active Full once will fix it once and for all.

However, if this job was created on 6.1 patch 1 code, then this would need to be closely investigated by our support.
Gostev
Veeam Software
 
Posts: 21611
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Error: no FIB [xxx.vmdk] in the specified restore point

Veeam Logoby gwilmington » Wed Oct 03, 2012 2:50 pm

Yeah I had the previous version to 205 and the tech had me upgrade. I removed all the jobs and recreated them and it ran fine for a few days and then corruption again. I'm going to remove them all again and try again. I also disabled snapshots on the VM using the NetApp VSC. I can't usually make it more than 3-4 days so we'll see. The good thing is this is a brand new environment and I'm doing testing so I have time.
gwilmington
Influencer
 
Posts: 10
Liked: 1 time
Joined: Tue Aug 14, 2012 2:05 pm

Re: Error: no FIB [xxx.vmdk] in the specified restore point

Veeam Logoby gwilmington » Fri Oct 05, 2012 5:23 pm 1 person likes this post

http://kb.vmware.com/selfservice/micros ... Id=1009073

Following this KB article, snapping my Veeam backup server with an in-guest iSCSI attached is not supported and could be what is causing my iSCSI Disk corruption issues. I have removed the VM from NetApp VSC with VMware enabled VSS quiescing and I have not had an issue yet. If I make it more than a week then I'll feel much better.
gwilmington
Influencer
 
Posts: 10
Liked: 1 time
Joined: Tue Aug 14, 2012 2:05 pm

Re: Error: no FIB [xxx.vmdk] in the specified restore point

Veeam Logoby Gostev » Fri Oct 05, 2012 9:17 pm

Based on your issue description, I am not 100% convinced this is what was causing the issue. Was it the suggestion from our support? Please update us in a week.
Gostev
Veeam Software
 
Posts: 21611
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Error: no FIB [xxx.vmdk] in the specified restore point

Veeam Logoby gwilmington » Sat Oct 06, 2012 2:16 pm

It wasn't a suggestion from support no. My original issue was a metadata corruption issue. But after upgrading to 205 it's now become a FIB issue. The escalated technician didn't see FIB in the original log send because that wasn't my original problem so I had to send new logs to look at. I was asked to look at the filer logs but i dont see anything out of the ordinary related to iSCSI connections or resets or anything.

They continue to say it looks like an underlying storage issue but they havent reviewed the new logs just yet so we'll see. I haven't heard back so I'm trying a few things myself. Since its not a supported configuration in the first place to snap a VM with iSCSI in guest, I'm trying to rule that out. I have kept support apprised of any configuration changes or testing procedures that I'm trying as well. So far my backups have been running without this issue so far. I'll keep the thread updated.
gwilmington
Influencer
 
Posts: 10
Liked: 1 time
Joined: Tue Aug 14, 2012 2:05 pm

Re: Error: no FIB [xxx.vmdk] in the specified restore point

Veeam Logoby gwilmington » Tue Oct 09, 2012 1:58 pm

I had a new issue but I was able to recreate the FIB problem. This time I received these errors. Now I'm starting to wonder if it's because of controller giveback/takebacks. Yesterday afternoon I performed some testing of our NetApp controller failover and giveback. I’ve been doing some of that testing for about a month or so and although I don’t recall each time I did it, I definitely did it yesterday and now I’m having issues.

The current configuration –
4 Backup Jobs
•CH_APPLIANCE_BACKUP –
o3 VMs backed up, all Virtual Appliances, Reversed incremental, monthly active full on first Sunday of each month
o 10/9/2012 failed
 PrepStorageForWriteEx failed 'Z:\Backups\CH_APPLIANCE_BACKUP\CH_APPLIANCE_BACKUP2012-10-07T211541.vbk' Client error: The specified number of stored banks (1613797575) does not match to supported limit (744).
oAttempted resolution – Edit job and recalculate VMs to ensure they show up. Had to remove one VM and re-add to job. Job still fails with the above error.
•CH_EXCHANGE_OS_BACKUP
o1 VM backed up, Reversed incremental, monthly active full on first Sunday of each month
o10/9/2012 failed
Failed to save backup meta to 'Z:\Backups\CH_EXCHANGE_OS_BACKUP\CH_EXCHANGE_OS_BACKUP.vbm' [CHPVBR01] Failed to save file content. The file or directory is corrupted and unreadable. --tr:FC: Failed to create file. File path: [Z:\Backups\CH_EXCHANGE_OS_BACKUP\CH_EXCHANGE_OS_BACKUP.vbm]. Desired access: [1073741824]. Creation disposition: [0]. --tr:Failed to call DoRpc. CmdName: [FcCreateFile] inParam: [<InputArguments><FilePath value="Z:\Backups\CH_EXCHANGE_OS_BACKUP\CH_EXCHANGE_OS_BACKUP.vbm" /><Desired
Attempted resolution – Edit job and recalculate VM to ensure they show up. Noticed that folder 'Z:\Backups\CH_EXCHANGE_OS_BACKUP\ is completely corrupted and will not open any longer. I ran chkdsk on the volume and it corrected issues and now I'm able to open the folder. Re-running the job now gives me the FIB error. - yay!
•CH_MANAGEMENT_BACKUP
o6 VMs that include the Veeam B&R server, 5 proxy servers (3 offline). Reversed incremental, monthly active full on first Sunday of each month
o10/9/2012 failed
One of the proxy servers, it was offline at the time, was not found. Removing and re-adding to the job and then restarting job allowed it to complete successfully.
•CH_VCENTER_BACKUP
o2 VMs backed up using standalone host method to ensure vCenter/Veeam DB server can be backed up properly. Reversed incremental, monthly active full on first Sunday of each month
oJob completed successfully with no issues.

I still have not moved this volume to be controlled by SnapDrive yet. I do have SnapDrive configured for my new Exchange server that we're building and testing and I'm not noticing anything like the above on it's NTFS volumes. My next course of action would be to remove the LUN and pass it through via SnapDrive.
gwilmington
Influencer
 
Posts: 10
Liked: 1 time
Joined: Tue Aug 14, 2012 2:05 pm

Re: Error: no FIB [xxx.vmdk] in the specified restore point

Veeam Logoby gwilmington » Wed Oct 10, 2012 2:30 pm

Opened a case with NetApp yesterday. Also noticed a configuration issue on my controllers concerning how takeover/giveback is working between them (read: both controllers not configured exactly the same on each side of an HA pair). Once I corrected those commands on the controller presenting the Veeam LUN and then running a takeover/giveback, the corruption seems to have subsided. I'm not 100% that this is the issue, but I have been performing failover testing of the HA pair over the last month or so. When I performed that testing on Monday and then Monday evening experienced corruption on the volume, being the only thing I did to the environment all day, that's where I started looking. Matching those tests to when I experienced the volume corruption I'm still working on but this is the closest I have to a resolution. I'm sure Veeam would love to know that it was a customer configuration issue rather than a product problem and I'm man enough to admit when I forgot something. :) Still researching and I've updated my case with Veeam and will be doing the same with NetApp.
gwilmington
Influencer
 
Posts: 10
Liked: 1 time
Joined: Tue Aug 14, 2012 2:05 pm

Re: Error: no FIB [xxx.vmdk] in the specified restore point

Veeam Logoby Gostev » Wed Oct 17, 2012 1:35 pm

To provide an update for future readers - this issue came down to storage device not configured correctly, resulting in multiple storage controller failovers during backup, which in turn caused data corruptions in the file system. After the storage vendor helped to configure the storage correctly, the problem went away.
Gostev
Veeam Software
 
Posts: 21611
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: Bing [Bot] and 1 guest