Hello all,
I have a disk to disk reverse increment backup job that was running fine for the last 2 weeks (14 restore points) that just recently failed with error.
First error message was "Error: Exception of type 'Veeam.Backup.AgentProvider.AgentClosedException' was thrown."
After the multiple attempts at retry: "Error: There is no FIB [GuestIndexData.zip] in the specified restore point. Failed to process [srcCopyLocal] command. Reconnectable protocol device was closed. Failed to process [srcCopyLocal] command."
Coincidentally, as I was doing some troubleshooting myself, I came across this in the job setting. The day the backup job failed was the day when the Storage-level corruption guard - Perform backup files health check (detects and auto-heals corruption) was set to run. Has anyone encountered an issue like this?
Any ideas are welcome. TIA
Support case is #02082681
Edit: Forgot to mention I'm running Windows Server 2012 HyperV environment and Dell Powervault as ReFS storage
Please continue to work with our support department. It might be indeed that an issue is created because they were trying or actually running at the same time. It could be a coincidence and there is some corruption in the backup chain or some other reasons. Our engineers will look through your logs and hopefully find a good solution.
Please keep us informed when you have more information
Just to clarify, it was a setting within the same job to run a health check on the file, not separate job that does it. From what I've been gathering, seems like running the Active full job is the only solution but I wanted to see if there was a way to retain the backup chain.
I will keep you updated if I find anything else
Regards,
Error: There is no FIB GuestIndexData.zip is an indicator that there are some missing instructions from the backup file that were expected to be there. In your case: File 'GuestIndexData.zip'
This happens when the job fails due to some other reasons and then later on Veeam will look at the backup files and notifies you that there is a missing backup that didn;t get to complete but was expected -By looking back in the job history you may find what was happening that led to the issue with the chain.
From the job report (right-click on the job and choose report), you can see that there is an error of an agent failure that took place on February 24, 2017 8:30:03 PM.
Error: Exception of type 'Veeam.Backup.AgentProvider.AgentClosedException' was thrown.
Even before that I see some other issues with the Veeam DB not being available mostly while backing up VM $server
In particular to this issue:
How VM Guest OS Indexing Works
When you run a backup job with the file indexing option enabled, Veeam Backup & Replication performs the following operations:
When the backup job starts, Veeam Backup & Replication connects to the VM whose file system must be indexed and deploys a runtime process inside this VM. The runtime process is responsible for coordinating indexing activities inside the VM.
The runtime process starts indexing the VM file system. The indexing procedure is carried out in parallel with the backup procedure. If indexing takes long, Veeam Backup & Replication will not wait for the indexing procedure to complete. It will start copying VM data and continue file indexing inside the VM. If you have enabled application-aware processing for the VM, Veeam Backup & Replication performs indexing using the VSS snapshot, not the VM guest OS itself. As a result, the created file index exactly reflects the state of the backed up VM.
When file indexing is complete, the runtime process collects indexing data and writes it to the GuestIndexData.zip file. The GuestIndexData.zip file is stored to a temporary folder on the backup server.
When the backup job completes, Veeam Backup & Replication notifies the local Veeam Guest Catalog Service, and the service saves indexing data in the Veeam Catalog folder on the backup server.
During the next catalog replication session, the global Veeam Guest Catalog Service replicates data from the backup server to the Veeam Catalog folder on the Veeam Backup Enterprise Manager server.
The only way to prevent the missing FIB error is to always fix the errors with the job before the chain realizes that one of the expected backups missed some needed files that were supposed to be part of it due to another failure. The failure can vary.
In your case the agent failure on one of the servers was the root cause. Agent failure is something you don't see alot unless there is a problem with the infrastructure itself: low infrastructure capacity, cpu, memory, AV, etc....
If you see this error more than often then the infrastructure will need to be looked upon otherwise it may just be a proxy that went off line at some point or the Veeam server DB that was overwhelmed due to processing more that it has the power to handle at some point during backups.
Error: Exception of type 'Veeam.Backup.AgentProvider.AgentClosedException' was thrown.
We just bought brand new hardware for the storage and hosts have more than enough resources to hold 3 times the amount of VMs we currently have. According to the support, my guess is that the connectivity hiccup on the proxy was the cause. Fingers crossed it won't happen again!
but what has caused the actual issue?