Error: There is no FIB

[Bunce]

Gotchya. Ta.

[nreutemann]

Hello, I got this error message when the job try to do the synthetic full:

"TextFromTar failed Client error: There is no FIB [summary.xml] in the specified restore point. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://Tar2Text]. CHMOD mask: [0]."

I got other 14 jobs without problems, this one got 26 VMs.

The server is W2008 R2 and the Veeam B&R version is 6.1.0.205 (I installed the update this weekend, when the error appears).

I´m trying to create an Support Case, but when I try to upload the logs file, the upload don´t finish (the file got 1.2Gb).

There is any other way to upload the logs files?

Thanks in advance.

[Vitaliy S.]

Nicolas,

Please open the ticket and inform your support engineer about the size of the log files and you will receive a link to the FTP server which can be used to upload this amount of debug logs files.

Thanks!

[nreutemann]

Ok. Thanks.

Support Case ID: 5216333

[gwilmington]

I'm getting the same error myself.

SR - 5128407

[Gostev]

Hi, what product version and build number are you using?

[gwilmington]

B&R version 6.1.0.205 (64bit)

My setup:

vSphere 5.0 U1
1 main backup server and 2 proxies all Win2k8R2 with in-guest iSCSI initiator on the iSCSI network
Netapp 2240-2 serving a 5TB LUN of SATA disk via iSCSI
5TB NTFS volume formatted with GPT
All VM LUNs connected in offline mode to the main Veeam server and proxies
Reversed incremental backups with monthly archived fulls

Setup seems to work fine for a few days then all of a sudden backups start to fail. Only thing that works is running Active Full to start a new chain. All previous restore points are unusable.

Was asked to move the repository to another disk array, but that's not possible as this is our main storage. Technician is escalating to another tier. I'm thinking about presenting the disk to the VM via SnapDrive instead of natively to see if that helps.

Edit: I found an event log entry showing the iSCSI attached LUN is corrupted and to run chkdsk. However 1 of 3 jobs completed when this happened. Seems like a NetApp issue but that doesn't make much sense that it worked for 1 of the jobs if the volume is having an issue. I'm wondering if there are issues with running NetApp VSC snapshots on a VM with an iSCSI guest and it could be causing a problem with the attached LUN. I removed this VM and the proxies from VSC snapshotting. This seems to be a perfect storm of small problems.

[Gostev]

If your job was created at the time when the product had this bug (versions 6.0 or 6.1), then the issue may surface even with the latest version of the code. In this case, performing Active Full once will fix it once and for all.

However, if this job was created on 6.1 patch 1 code, then this would need to be closely investigated by our support.

[gwilmington]

Yeah I had the previous version to 205 and the tech had me upgrade. I removed all the jobs and recreated them and it ran fine for a few days and then corruption again. I'm going to remove them all again and try again. I also disabled snapshots on the VM using the NetApp VSC. I can't usually make it more than 3-4 days so we'll see. The good thing is this is a brand new environment and I'm doing testing so I have time.

[gwilmington]

http://kb.vmware.com/selfservice/micros ... Id=1009073

Following this KB article, snapping my Veeam backup server with an in-guest iSCSI attached is not supported and could be what is causing my iSCSI Disk corruption issues. I have removed the VM from NetApp VSC with VMware enabled VSS quiescing and I have not had an issue yet. If I make it more than a week then I'll feel much better.

[Gostev]

Based on your issue description, I am not 100% convinced this is what was causing the issue. Was it the suggestion from our support? Please update us in a week.

[gwilmington]

It wasn't a suggestion from support no. My original issue was a metadata corruption issue. But after upgrading to 205 it's now become a FIB issue. The escalated technician didn't see FIB in the original log send because that wasn't my original problem so I had to send new logs to look at. I was asked to look at the filer logs but i dont see anything out of the ordinary related to iSCSI connections or resets or anything.

They continue to say it looks like an underlying storage issue but they havent reviewed the new logs just yet so we'll see. I haven't heard back so I'm trying a few things myself. Since its not a supported configuration in the first place to snap a VM with iSCSI in guest, I'm trying to rule that out. I have kept support apprised of any configuration changes or testing procedures that I'm trying as well. So far my backups have been running without this issue so far. I'll keep the thread updated.

[gwilmington]

I had a new issue but I was able to recreate the FIB problem. This time I received these errors. Now I'm starting to wonder if it's because of controller giveback/takebacks. Yesterday afternoon I performed some testing of our NetApp controller failover and giveback. I’ve been doing some of that testing for about a month or so and although I don’t recall each time I did it, I definitely did it yesterday and now I’m having issues.

The current configuration –
4 Backup Jobs
• CH_APPLIANCE_BACKUP –
o 3 VMs backed up, all Virtual Appliances, Reversed incremental, monthly active full on first Sunday of each month
o 10/9/2012 failed
 PrepStorageForWriteEx failed 'Z:\Backups\CH_APPLIANCE_BACKUP\CH_APPLIANCE_BACKUP2012-10-07T211541.vbk' Client error: The specified number of stored banks (1613797575) does not match to supported limit (744).
o Attempted resolution – Edit job and recalculate VMs to ensure they show up. Had to remove one VM and re-add to job. Job still fails with the above error.
• CH_EXCHANGE_OS_BACKUP
o 1 VM backed up, Reversed incremental, monthly active full on first Sunday of each month
o 10/9/2012 failed
 Failed to save backup meta to 'Z:\Backups\CH_EXCHANGE_OS_BACKUP\CH_EXCHANGE_OS_BACKUP.vbm' [CHPVBR01] Failed to save file content. The file or directory is corrupted and unreadable. --tr:FC: Failed to create file. File path: [Z:\Backups\CH_EXCHANGE_OS_BACKUP\CH_EXCHANGE_OS_BACKUP.vbm]. Desired access: [1073741824]. Creation disposition: [0]. --tr:Failed to call DoRpc. CmdName: [FcCreateFile] inParam: [<InputArguments><FilePath value="Z:\Backups\CH_EXCHANGE_OS_BACKUP\CH_EXCHANGE_OS_BACKUP.vbm" /><Desired
 Attempted resolution – Edit job and recalculate VM to ensure they show up. Noticed that folder 'Z:\Backups\CH_EXCHANGE_OS_BACKUP\ is completely corrupted and will not open any longer. I ran chkdsk on the volume and it corrected issues and now I'm able to open the folder. Re-running the job now gives me the FIB error. - yay!
• CH_MANAGEMENT_BACKUP
o 6 VMs that include the Veeam B&R server, 5 proxy servers (3 offline). Reversed incremental, monthly active full on first Sunday of each month
o 10/9/2012 failed
 One of the proxy servers, it was offline at the time, was not found. Removing and re-adding to the job and then restarting job allowed it to complete successfully.
• CH_VCENTER_BACKUP
o 2 VMs backed up using standalone host method to ensure vCenter/Veeam DB server can be backed up properly. Reversed incremental, monthly active full on first Sunday of each month
o Job completed successfully with no issues.

I still have not moved this volume to be controlled by SnapDrive yet. I do have SnapDrive configured for my new Exchange server that we're building and testing and I'm not noticing anything like the above on it's NTFS volumes. My next course of action would be to remove the LUN and pass it through via SnapDrive.

[gwilmington]

Opened a case with NetApp yesterday. Also noticed a configuration issue on my controllers concerning how takeover/giveback is working between them (read: both controllers not configured exactly the same on each side of an HA pair). Once I corrected those commands on the controller presenting the Veeam LUN and then running a takeover/giveback, the corruption seems to have subsided. I'm not 100% that this is the issue, but I have been performing failover testing of the HA pair over the last month or so. When I performed that testing on Monday and then Monday evening experienced corruption on the volume, being the only thing I did to the environment all day, that's where I started looking. Matching those tests to when I experienced the volume corruption I'm still working on but this is the closest I have to a resolution. I'm sure Veeam would love to know that it was a customer configuration issue rather than a product problem and I'm man enough to admit when I forgot something. Still researching and I've updated my case with Veeam and will be doing the same with NetApp.

[Gostev]

To provide an update for future readers - this issue came down to storage device not configured correctly, resulting in multiple storage controller failovers during backup, which in turn caused data corruptions in the file system. After the storage vendor helped to configure the storage correctly, the problem went away.

[Tibo]

I have the same issue on Veeam 6.5 with two very different setups : One is a virtual server running Veeam 6.5 on Windows 2008 R2 backuping on a Quantum NAS, the other is a physical Veeam 6.5 on Windows 2003 32 Bits backuping on local disks).

Both setups are configured to use incremental + synthetic on saturday + transform previous backups into rollback.

Both are working great but fail to do the synthetic on saturday (a retry gives the same error).

I'm collecting the logs right now to open a support case.

[nreutemann]

I got (again) this error.

There is no error on storage system, so the corruption file don´t come from that side.

I need a solution very quick, I don´t have free space to create a new full backup so, that is not a choice to solution the problem.

Any clue? (Veeam B&R 6.5 installed, physical server, w2008 R2)

[Vitaliy S.]

To get immediate assistance on the technical issues, it's always better to contact our support team directly. This is not a support forum.

[nreutemann]

Yes, I know Vitaly. Simply, I post here to publicate my situation.

This is my support case: 00155035

Thanks.

[MaartenK]

I had the same problem.

first I received the message
TextFromTar failed Client error: There is no FIB [summary.xml] in the specified restore point. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://Tar2Text]. CHMOD mask: [0].

Ok no problem I think? But the next day The backup failed again

with the message
Failed to delete oib '[vmname=servername:creation_time=16-11-2012 0:20:03:alg=Transform:id=4d717669-b3ab-476c-a041-78fa67811b41:point_id=32cbcc8a-012d-495d-b6b8-2001312244cc:storage_id=2a8868c7-a403-44e5-89af-a8c0b263fe07]' from '\\nas-2\Backup Prod12012-11-15T193116.vbk', redo '\\nas-2\Backup Prod12012-11-15T193116.vrb' Undir failed Client error: The process cannot access the file because............also strange the line stops by 'because'??

After that I created a Full backup. This went successful (as mentioned in this topic)

The Full Backup went ok and started a rescan of the repository. I was able to restore A full VM and files older then the last Full Backup.

[Tobias_Elfstrom]

I generally see this type of errors if the process of the Veeam job suddenly stops for one reason or another. It may be storage problem, Veeam server or proxy problems etc.
All it says seems to be that there is some kind of problem with the backup files in question and doing a full backup of that job have always solved it for me. Now this will not solve the problem nor find the root cause but makes a pretty good workaround.

[aevenot]

Last night my backup failed with the error "there is no fib"

According to some messages on the forum, doing a full backup is the solution. But I'm doing offsite backup so is not a solution to me.

I already have a case opened but I will not have workaround until monday, so I'm asking for help.

[Gostev]

Hello, upgrading B&R to the latest version and doing full backup is the only solution I am aware of, unfortunately.

[bernardw]

I wanted to say I also am getting this error on a fully updated Veeam install. I have 5 other jobs running on the same hardware without any issues.

I will run a new full backup, but will there be any way to have support check the log files on Monday to determin why it is failing?

I should say that it looks like the incrementals are being created successfully. It is just the synthetic full with rollback chains that is not working. I think that it created one roll back since the previous full but the rest are still forward incrementals.

[aevenot]

you're right the problem happened right after the synthetic, I think the rollback chain has not been treated correctly.

I'm doing offsite backup for my 9 sites, the backup in error is around 1To so it's verry complicated to me to redo a full backup.

I prefer losing all backup for thids week instead of doing a full backup

I sincerely hope the support will find a workaround on monday.

[aevenot]

if that helps, I have a vib and a vb with the same name in my backup repo.

and now My job failed wth the error "failed to repair transformed iob"

[veremin]

The backup in error is around 1To so it's verry complicated to me to redo a full backup.

Then, i’m wondering whether it’s possible to perform full backup to local repository first and transfer it afterwards to remote location using external device.

Thanks.

[aevenot]

I've already done that to perform the first initial backup so it's possible but It's very time consuming.

On my central site, I have to put all the files (vib and vmb) on a usb drive, then send the drive to the remote site to do a full backup, then after resend the drive and commit all the files to my b&r server.

For now I have only one site but at term we will have 9 sites located in different country. So as I tell, It will be very time consuming if this problem happens another time.

[aevenot]

The support asks me to do a full backup so I'm doing it right now.

[TimeKnight]

Anyone know if this has been corrected with patch 3? I got this error again over the weekend. I know opening a case will just result in them asking me to run a full backup, which will band-aid the problem but does not really correct it. Running the active full will force me to deal with an extra 1TB backup for 3 weeks until it has rolled off with my retention policy since active fulls are not transformed when a synthetic backup is preformed.

To me this is a serious problem. If this has not been addressed in 6.5 patch 3 it should defiantly be corrected in 7.0. I wish Veeam would spend more time making the core of the product rock solid instead of adding features.