Comprehensive data protection for all workloads
FrancWest
Veteran
Posts: 488
Liked: 93 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

.vib files are left behind

Post by FrancWest » 1 person likes this post

Hi,

Case #02302437, it appears that for yet unknown reasons some .vib files are being left behind on the file system after creating a .vbk from them (using fast clone on ReFS). The file entries are being removed from the database, but they aren’t from the file system. The logs show that veeam can’t delete them because it’s gets a file in use error. I’ve configured all AV exceptions but that doesn’t help. Since veeam is not retrying the deletion, the unneeded .vib files are occupying precious disk space (I had an instance of 5TB being ‘wasted’). Manual deleting the .vib file is a bit dangerous, sine you don’t know exactly which files are redundant. You first have to rename the files you suspect to be redundant and then do a repository rescan. If after that a button ‘forget’ appears in the backup properties window, then you know that you are trying to remove the wrong file. It’s a bit trial and error, but can save several TB’s of disk space, especially since veeam doesn’t give any warning or error that it wasn’t able to delete the .vib file during the job run. A script or tool that compares the file entries in the database and on disk would be very helpful.
DonZoomik
Service Provider
Posts: 368
Liked: 120 times
Joined: Nov 25, 2016 1:56 pm
Full Name: Mihkel Soomere
Contact:

Re: .vib files are left behind

Post by DonZoomik »

It just happens for some reason. Also .temp file (instead of .vbk) also just tends to happen...
I wrote some script a while ago that compares files to db but only on a single system. I think there was something here in forums as well, but I can't currently find it.

My snippet from about 2 years ago - I don't know if it even works anymore as I no longer have any all-in-one deployments.

Code: Select all

$vbrrp = Get-VBRRestorePoint
$vbrrpstorage = $vbrrp.findstorage()
#local only works, remote smb/ssh todo
$repos = $(
    Get-VBRBackupRepository
    ((Get-VBRBackupRepository -ScaleOut)|Get-VBRRepositoryExtent).repository
)
$files = gci $repos.path -file -recurse -include @('*.temp','*.vbk','*.vib')

#find orphans
$badfiles=$files|?{$vbrrpstorage.PartialPath.elements -NotContains $_.name}
$badfiles|ft fullname
FrancWest
Veteran
Posts: 488
Liked: 93 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: .vib files are left behind

Post by FrancWest »

unfortunately the script doesn't work. It reports all files present on disk instead of the redundant ones.
DonZoomik
Service Provider
Posts: 368
Liked: 120 times
Joined: Nov 25, 2016 1:56 pm
Full Name: Mihkel Soomere
Contact:

Re: .vib files are left behind

Post by DonZoomik »

I remembered that my homelab Veeam Community is an all-in-one deployment.
Ran script on it but it outputed zero files, there don't seem to be any orphaned files if checking manually was well so the script seems to be working fine...
FrancWest
Veteran
Posts: 488
Liked: 93 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: .vib files are left behind

Post by FrancWest »

Ok, I’ll check again. Just thinking, perhaps the files reported are the imported backups we have from an old repository. I suspect they aren’t part of any job. I’ll modify the script to only check the primary repo. The backup copies repo don’t have the issue.
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: .vib files are left behind

Post by pirx »

FrancWest wrote: Feb 18, 2021 7:04 am Hi,

Case #02302437, it appears that for yet unknown reasons some .vib files are being left behind on the file system after creating a .vbk from them (using fast clone on ReFS). The file entries are being removed from the database, but they aren’t from the file system. The logs show that veeam can’t delete them because it’s gets a file in use error. I’ve configured all AV exceptions but that doesn’t help. Since veeam is not retrying the deletion, the unneeded .vib files are occupying precious disk space (I had an instance of 5TB being ‘wasted’). Manual deleting the .vib file is a bit dangerous, sine you don’t know exactly which files are redundant. You first have to rename the files you suspect to be redundant and then do a repository rescan. If after that a button ‘forget’ appears in the backup properties window, then you know that you are trying to remove the wrong file. It’s a bit trial and error, but can save several TB’s of disk space, especially since veeam doesn’t give any warning or error that it wasn’t able to delete the .vib file during the job run. A script or tool that compares the file entries in the database and on disk would be very helpful.
Are you using offload or copy feature? If yes, then this might be due to an bug which keeps a lock on a file. But then you should also see errors in logs of Veeam Console.
FrancWest
Veteran
Posts: 488
Liked: 93 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: .vib files are left behind

Post by FrancWest »

We are using these backups as the source for copy jobs, but no errors in the veeam console. Only in the logs.
FrancWest
Veteran
Posts: 488
Liked: 93 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: .vib files are left behind

Post by FrancWest »

DonZoomik wrote: Feb 20, 2021 5:54 pm Ran script on it but it outputed zero files, there don't seem to be any orphaned files if checking manually was well so the script seems to be working fine...
I modified the script a bit so that it only reports on regular backups and windows agent backups:

Code: Select all

$backup = get-vbrbackup | Where-Object { ($_.jobtype -eq "Backup") -or ($_.jobtype -eq "EpAgentManagement")}
$vbrrp = Get-VBRRestorePoint -Backup $backup
and in the repository where these jobs are stored (hard-coded) and it's working now. I can save up to 1.2TB of data by removing these redundant backups.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: .vib files are left behind

Post by foggy »

Turned out to be a known issue (keeping a file lock longer than required) addressed in v11.
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: .vib files are left behind

Post by pirx »

pirx wrote: Feb 21, 2021 8:39 am Are you using offload or copy feature? If yes, then this might be due to an bug which keeps a lock on a file. But then you should also see errors in logs of Veeam Console.
I guess it's the same I hit.
FrancWest
Veteran
Posts: 488
Liked: 93 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: .vib files are left behind

Post by FrancWest » 2 people like this post

After convincing support it’s a bug (they first assumed it was an issue with the job) I finally received an hotfix for V10 for this issue and I haven’t had any redundant files since.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: .vib files are left behind

Post by mkretzer »

Can someone from Veeam verify that this is fixed in V11? We had to delete ~30 TB of left-behind vib files (some are > 1 year old) manually 2 weeks ago. Support had no idee where this could come from and how to find these files automatically. Case 04629666.
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: .vib files are left behind

Post by pirx »

Ask support for Case 04608093 or 04594802 I'm not sure which one covered the root cause. I got feedback that this should be fixed in v11 but also received an hotfix a couple of days ago.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: .vib files are left behind

Post by mkretzer »

Hotfix for v11 or only for v10?
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: .vib files are left behind

Post by pirx »

Hotfix for V10
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: .vib files are left behind

Post by foggy »

mkretzer wrote: Mar 14, 2021 9:51 pm Can someone from Veeam verify that this is fixed in V11? We had to delete ~30 TB of left-behind vib files (some are > 1 year old) manually 2 weeks ago. Support had no idee where this could come from and how to find these files automatically. Case 04629666.
I can confirm that the OP's issue is addressed by v11.
FrancWest
Veteran
Posts: 488
Liked: 93 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: .vib files are left behind

Post by FrancWest »

the hotfix for V10 solves the above issue, but it introduces a new one. You can't view the properties of a backproxy anymore. Everytime you open the backup proxy screen or click on a proxy you receive a error pop-up about a 'method not found'. It looks like the DLL for the hotfix is older than the one included in the latest build which causes the issue.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: .vib files are left behind

Post by foggy »

You could be provided with the hotfix created for an earlier update - for example, if you're already running v10a and obtained the DLL for v10, could this be the case? Granted this is the issue, you can request the hotfix for your version - it is also available.
FrancWest
Veteran
Posts: 488
Liked: 93 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: .vib files are left behind

Post by FrancWest » 1 person likes this post

That was indeed the case. I received the fix 2 times. Both times the dll was older than the one currently installed.

Well, support requested the logs again for the last 5 days and the case was escalated to ‘superiors support’. Engineer stated the fix had to be created for the version we are running, which is build 10.0.1.4854 P20201202.

New case number #04735694
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: .vib files are left behind

Post by pirx »

I received a new hotfix Fix_307232_6f30225612.zip for this (locks on files) today, and I also have installed a fix before. I guess I'll wait what support tells you.
FrancWest
Veteran
Posts: 488
Liked: 93 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: .vib files are left behind

Post by FrancWest »

You aren’t getting the method not found error with the fix from today installed?
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: .vib files are left behind

Post by pirx »

I did not install it yet, now that I read this, I'll wait.
spiritie
Service Provider
Posts: 189
Liked: 40 times
Joined: Mar 01, 2016 10:16 am
Full Name: Gert
Location: Denmark
Contact:

Re: .vib files are left behind

Post by spiritie »

If was not able to get the PowerShell script working, even with hardcoding in the Repository path (this is a all in 1 server)

The script just outputs all backup files on the repository.

Could anyone link the script they got working?
FrancWest
Veteran
Posts: 488
Liked: 93 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: .vib files are left behind

Post by FrancWest »

This is the modified script I'm using which works, replace '<drive letter>' with yours. I'm using a scaleout repository, hence the parameter -ScaleOut

Code: Select all

$backup = get-vbrbackup | Where-Object { ($_.jobtype -eq "Backup") -or ($_.jobtype -eq "EpAgentManagement")}

$vbrrp = Get-VBRRestorePoint -Backup $backup

$vbrrpstorage = $vbrrp.findstorage()
#local only works, remote smb/ssh todo
$repos = $(
    Get-VBRBackupRepository
    ((Get-VBRBackupRepository -ScaleOut)|Get-VBRRepositoryExtent).repository
)

$files = get-childitem "<drive letter>:\Backups\" -file -recurse -include @('*.temp','*.vbk','*.vib')

#find orphans
$badfiles=$files|?{$vbrrpstorage.PartialPath.elements -NotContains $_.name}
$badfiles|ft fullname

$badfilesize = Get-ChildItem $badfiles | Measure-Object -property length -sum
$totalsizeGB = $badfilesize.sum/1GB
$totalsizeTB = $badfilesize.sum/1TB
write-host "Total size of redundant files: "$totalsizeGB.ToString("#.##")"GB ("$totalsizeTB.ToString("#.##")"TB)"
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: .vib files are left behind

Post by pirx »

I've to try to modify this for our SMB shares.
spiritie
Service Provider
Posts: 189
Liked: 40 times
Joined: Mar 01, 2016 10:16 am
Full Name: Gert
Location: Denmark
Contact:

Re: .vib files are left behind

Post by spiritie »

@FrancWest thanks a lot of the code, it works :)

I've tested this for a VBR Repo that is not local, and it works. Please note this takes quite some time compared to being local, and it requires that remote powershelling is allowed:

So the only change is to remove this:

Code: Select all

$files = Get-ChildItem "D:\Backups\" -file -recurse -include @('*.temp','*.vbk','*.vib') 
With this:

Code: Select all

 $files = Invoke-Command -Computername <IP/FQDN of Repo server hosting the files> -Scriptblock {
 return $remotefiles = Get-Childitem "D:\Backups\" -File -Recurse -Include @('*.temp','*.vbk','*.vib') } 
spiritie
Service Provider
Posts: 189
Liked: 40 times
Joined: Mar 01, 2016 10:16 am
Full Name: Gert
Location: Denmark
Contact:

Re: .vib files are left behind

Post by spiritie »

FYI, I've run this on a server that have v11 installed since RTM and the script have correctly found around 10 .vib files that no longer is part of Veeam's config.

So this bug is still present in v11, was this fixed in v11 RTM or first later. in GA or the cumulative patches?
FrancWest
Veteran
Posts: 488
Liked: 93 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: .vib files are left behind

Post by FrancWest »

you're welcome! Was this a completely new install of V11, or did the repository with the redundant files already exist before upgrading to V10?

According to the information I got this should be fixed in V11 RTM (and thus also GA).
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: .vib files are left behind

Post by pirx »

FrancWest wrote: Apr 07, 2021 5:10 pm You aren’t getting the method not found error with the fix from today installed?
Anything new? I'm about to install this hotfix in the next couple of days when I get a downtime.
FrancWest
Veteran
Posts: 488
Liked: 93 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: .vib files are left behind

Post by FrancWest »

Nope, case is still with higher support level..
Post Reply

Who is online

Users browsing this forum: ante_704, Google [Bot], Ivan239, Semrush [Bot], ybarrap2003 and 170 guests