.vib files are left behind

[FrancWest]

Hi,

Case #02302437, it appears that for yet unknown reasons some .vib files are being left behind on the file system after creating a .vbk from them (using fast clone on ReFS). The file entries are being removed from the database, but they aren’t from the file system. The logs show that veeam can’t delete them because it’s gets a file in use error. I’ve configured all AV exceptions but that doesn’t help. Since veeam is not retrying the deletion, the unneeded .vib files are occupying precious disk space (I had an instance of 5TB being ‘wasted’). Manual deleting the .vib file is a bit dangerous, sine you don’t know exactly which files are redundant. You first have to rename the files you suspect to be redundant and then do a repository rescan. If after that a button ‘forget’ appears in the backup properties window, then you know that you are trying to remove the wrong file. It’s a bit trial and error, but can save several TB’s of disk space, especially since veeam doesn’t give any warning or error that it wasn’t able to delete the .vib file during the job run. A script or tool that compares the file entries in the database and on disk would be very helpful.

[DonZoomik]

It just happens for some reason. Also .temp file (instead of .vbk) also just tends to happen...
I wrote some script a while ago that compares files to db but only on a single system. I think there was something here in forums as well, but I can't currently find it.

My snippet from about 2 years ago - I don't know if it even works anymore as I no longer have any all-in-one deployments.

Code: Select all

$vbrrp = Get-VBRRestorePoint
$vbrrpstorage = $vbrrp.findstorage()
#local only works, remote smb/ssh todo
$repos = $(
    Get-VBRBackupRepository
    ((Get-VBRBackupRepository -ScaleOut)|Get-VBRRepositoryExtent).repository
)
$files = gci $repos.path -file -recurse -include @('*.temp','*.vbk','*.vib')

#find orphans
$badfiles=$files|?{$vbrrpstorage.PartialPath.elements -NotContains $_.name}
$badfiles|ft fullname

[FrancWest]

unfortunately the script doesn't work. It reports all files present on disk instead of the redundant ones.

[DonZoomik]

I remembered that my homelab Veeam Community is an all-in-one deployment.
Ran script on it but it outputed zero files, there don't seem to be any orphaned files if checking manually was well so the script seems to be working fine...

[FrancWest]

Ok, I’ll check again. Just thinking, perhaps the files reported are the imported backups we have from an old repository. I suspect they aren’t part of any job. I’ll modify the script to only check the primary repo. The backup copies repo don’t have the issue.

[pirx]

Hi,

Case #02302437, it appears that for yet unknown reasons some .vib files are being left behind on the file system after creating a .vbk from them (using fast clone on ReFS). The file entries are being removed from the database, but they aren’t from the file system. The logs show that veeam can’t delete them because it’s gets a file in use error. I’ve configured all AV exceptions but that doesn’t help. Since veeam is not retrying the deletion, the unneeded .vib files are occupying precious disk space (I had an instance of 5TB being ‘wasted’). Manual deleting the .vib file is a bit dangerous, sine you don’t know exactly which files are redundant. You first have to rename the files you suspect to be redundant and then do a repository rescan. If after that a button ‘forget’ appears in the backup properties window, then you know that you are trying to remove the wrong file. It’s a bit trial and error, but can save several TB’s of disk space, especially since veeam doesn’t give any warning or error that it wasn’t able to delete the .vib file during the job run. A script or tool that compares the file entries in the database and on disk would be very helpful.

Are you using offload or copy feature? If yes, then this might be due to an bug which keeps a lock on a file. But then you should also see errors in logs of Veeam Console.

[FrancWest]

We are using these backups as the source for copy jobs, but no errors in the veeam console. Only in the logs.

[FrancWest]

Ran script on it but it outputed zero files, there don't seem to be any orphaned files if checking manually was well so the script seems to be working fine...

I modified the script a bit so that it only reports on regular backups and windows agent backups:

Code: Select all

$backup = get-vbrbackup | Where-Object { ($_.jobtype -eq "Backup") -or ($_.jobtype -eq "EpAgentManagement")}
$vbrrp = Get-VBRRestorePoint -Backup $backup

and in the repository where these jobs are stored (hard-coded) and it's working now. I can save up to 1.2TB of data by removing these redundant backups.

[foggy]

Turned out to be a known issue (keeping a file lock longer than required) addressed in v11.

[pirx]

Are you using offload or copy feature? If yes, then this might be due to an bug which keeps a lock on a file. But then you should also see errors in logs of Veeam Console.

I guess it's the same I hit.

[FrancWest]

After convincing support it’s a bug (they first assumed it was an issue with the job) I finally received an hotfix for V10 for this issue and I haven’t had any redundant files since.

[mkretzer]

Can someone from Veeam verify that this is fixed in V11? We had to delete ~30 TB of left-behind vib files (some are > 1 year old) manually 2 weeks ago. Support had no idee where this could come from and how to find these files automatically. Case 04629666.

[pirx]

Ask support for Case 04608093 or 04594802 I'm not sure which one covered the root cause. I got feedback that this should be fixed in v11 but also received an hotfix a couple of days ago.

[mkretzer]

Hotfix for v11 or only for v10?

[pirx]

Hotfix for V10

[foggy]

Can someone from Veeam verify that this is fixed in V11? We had to delete ~30 TB of left-behind vib files (some are > 1 year old) manually 2 weeks ago. Support had no idee where this could come from and how to find these files automatically. Case 04629666.

I can confirm that the OP's issue is addressed by v11.

[FrancWest]

the hotfix for V10 solves the above issue, but it introduces a new one. You can't view the properties of a backproxy anymore. Everytime you open the backup proxy screen or click on a proxy you receive a error pop-up about a 'method not found'. It looks like the DLL for the hotfix is older than the one included in the latest build which causes the issue.

[foggy]

You could be provided with the hotfix created for an earlier update - for example, if you're already running v10a and obtained the DLL for v10, could this be the case? Granted this is the issue, you can request the hotfix for your version - it is also available.

[FrancWest]

That was indeed the case. I received the fix 2 times. Both times the dll was older than the one currently installed.

Well, support requested the logs again for the last 5 days and the case was escalated to ‘superiors support’. Engineer stated the fix had to be created for the version we are running, which is build 10.0.1.4854 P20201202.

New case number #04735694

[pirx]

I received a new hotfix Fix_307232_6f30225612.zip for this (locks on files) today, and I also have installed a fix before. I guess I'll wait what support tells you.

[FrancWest]

You aren’t getting the method not found error with the fix from today installed?

[pirx]

I did not install it yet, now that I read this, I'll wait.

[spiritie]

If was not able to get the PowerShell script working, even with hardcoding in the Repository path (this is a all in 1 server)

The script just outputs all backup files on the repository.

Could anyone link the script they got working?

[FrancWest]

This is the modified script I'm using which works, replace '<drive letter>' with yours. I'm using a scaleout repository, hence the parameter -ScaleOut

Code: Select all

$backup = get-vbrbackup | Where-Object { ($_.jobtype -eq "Backup") -or ($_.jobtype -eq "EpAgentManagement")}

$vbrrp = Get-VBRRestorePoint -Backup $backup

$vbrrpstorage = $vbrrp.findstorage()
#local only works, remote smb/ssh todo
$repos = $(
    Get-VBRBackupRepository
    ((Get-VBRBackupRepository -ScaleOut)|Get-VBRRepositoryExtent).repository
)

$files = get-childitem "<drive letter>:\Backups\" -file -recurse -include @('*.temp','*.vbk','*.vib')

#find orphans
$badfiles=$files|?{$vbrrpstorage.PartialPath.elements -NotContains $_.name}
$badfiles|ft fullname

$badfilesize = Get-ChildItem $badfiles | Measure-Object -property length -sum
$totalsizeGB = $badfilesize.sum/1GB
$totalsizeTB = $badfilesize.sum/1TB
write-host "Total size of redundant files: "$totalsizeGB.ToString("#.##")"GB ("$totalsizeTB.ToString("#.##")"TB)"

[pirx]

I've to try to modify this for our SMB shares.

[spiritie]

@FrancWest thanks a lot of the code, it works

I've tested this for a VBR Repo that is not local, and it works. Please note this takes quite some time compared to being local, and it requires that remote powershelling is allowed:

So the only change is to remove this:

Code: Select all

$files = Get-ChildItem "D:\Backups\" -file -recurse -include @('*.temp','*.vbk','*.vib') 

With this:

Code: Select all

 $files = Invoke-Command -Computername <IP/FQDN of Repo server hosting the files> -Scriptblock {
 return $remotefiles = Get-Childitem "D:\Backups\" -File -Recurse -Include @('*.temp','*.vbk','*.vib') } 

[spiritie]

FYI, I've run this on a server that have v11 installed since RTM and the script have correctly found around 10 .vib files that no longer is part of Veeam's config.

So this bug is still present in v11, was this fixed in v11 RTM or first later. in GA or the cumulative patches?

[FrancWest]

you're welcome! Was this a completely new install of V11, or did the repository with the redundant files already exist before upgrading to V10?

According to the information I got this should be fixed in V11 RTM (and thus also GA).

[pirx]

You aren’t getting the method not found error with the fix from today installed?

Anything new? I'm about to install this hotfix in the next couple of days when I get a downtime.

[FrancWest]

Nope, case is still with higher support level..