Powershell File Level Restore script hangs

[DaKernel]

We have a powershell script that we use to check for a specific file in the backup files for our production vm's (roughly 200 vm's). We have an issue where the file level restore hangs in the GUI and according to logging we have in the powershell script the backup job file does not mount. Eventually after 6 hours the hung job does timeout in the GUI and is marked as failed. This does occur from time to time, but not every day. I opened a case with Veeam Support and they looked at the FLR log and they did not find any errors in the log regarding backup file mounting. We have had this script running for the last 3 months or so. Has anyone had this issue in the past? I can share my code if needed.

[veremin]

Do I understand correctly that the issue only occurs, if FLR is executed via PowerShell? Yep, sharing the script might be helpful. Thanks!

[DaKernel]

Yes, the only time this occurs is when a FLR is executed via powershell and this does NOT happen on a daily basis. Below is the bulk of the code, I did not include all variables and removed all of the logging we have.

The text file that is read from the script is in this format. Please note that the vm's can have multiple partitions: BackupJobName,VMName,PartitionLetter,PartitionLetter . So an example would look like this, BACKUPJOB1,VMNAME1,C,M,N,L.

Here is the code:

Code: Select all

#CMDLET to hook into Veeam powershell calls
asnp VeeamPSSnapin

#Need to connect to Veeam B/R Server
Connect-VBRServer -Server $strVBRServer -port 9392

#Need to loop thru text file to gather data to be used
foreach($strline in $strFileName)
    {

    #Need to split up line and parse out data
    $strSplitTemp = $strline.Split(",")
    $strJobName = $strSplitTemp[0]
    $strVMName = $strSplitTemp[1]
    write-host $strVMName

    #Now we are going to use the split data to build commands to do restore
    #We need to start the loop at item 2 (the partitions) so we do not use job name or vmname in loop
    for($i=2; $i -le $strSplitTemp.GetUpperBound(0); $i++)
        {

            $strRestoreFileName = "Veeam_Restore_Test_File.txt"

            #Need to build file name to look for with parsed out Partition Letter
            $strRestoreFileName = $strSplitTemp[$i]+":\$strRestoreFileName"
            $strPartition = $strSplitTemp[$i]

            #Start the FLR process
            #Build the restore command
            $strResult = Get-VBRBackup -Name $strJobName | Get-VBRRestorePoint -Name $strVMName | sort CreationTime -Descending | select -First 1 | Start-VBRWindowsFileRestore
            
            #Find the FLR mount point for the specific drive letter
            #Need to loop thru all mount points in backup job to find one that matches
            #Partition Letter, this is where the magic happens
            $flrmountpoint = ($strResult.MountSession.MountedDevices | ? {$_.DriveLetter -eq (Split-Path -Qualifier $strRestoreFileName)}).MountPoint

            #If flrmountpoint is empty stop restore job
            #saw this occur when job name was wrong
            If(!$flrmountpoint)
                {

                    #Stop the FLR process
                    Stop-VBRWindowsFileRestore $strResult
                    Clear-Variable -Name "strResult"
                    #Need to break out the loop so we dont continue to run due to error
                    Break
                }

            #Build the path to the file via the FLR mount point
            $file = $flrmountpoint + (Split-Path -NoQualifier $strRestoreFileName)


            # Copy/Restore the file
            Try
                {
                    #If all goes well and the partition was backed and the restore verification file is there
                    #we can copyt the file and all is good!
                    Copy $file $strCopyPath -ErrorAction Stop
                }
            Catch
                {
                    #Boooo something is wrong
                    #Stop the FLR process
                    Stop-VBRWindowsFileRestore $strResult
                }
            #Lets tidy up some variables
            Clear-Variable -Name "file"
            Clear-Variable -Name "flrmountpoint"
            Clear-Variable -Name "strRestoreFileName"

            #Stop the FLR process
            Stop-VBRWindowsFileRestore $strResult

        }
            #Additional Backup Job data
            $strResultVmName = $strResult.VmName
            $strResultSize = $strResult.Size
            $strResultCreationTime = $strResult.CreationTime
            $strResultDrives = $strResult.Drives
    }
    

Please let me know what else you might need.

[DaKernel]

hoping someone can help me out. We have upgraded to v11 a few months and I am trying to tackle the issue where the script is no longer using the veeam b/r server as the mount server by default. Now it is using the mount server of the backup repository that the job that the vm is on. Ok, saw that there is a -host variable when calling the Start-VBRWindowsFileRestore commandlet. I added the veeam b/r host and still refuses to use it as the mount server.

Here are the 2 lines I added to the script to add the mount server option:

Code: Select all

$strMountServer = Get-VBRServer -Name $strVBRServer
$strResult = Get-VBRBackup -Name $strJobName | Get-VBRRestorePoint -Name $strVMName | Sort-Object –Property CreationTime ` 
–Descending | Select-Object -First 1 | Start-VBRWindowsFileRestore -Host $strMountServer

Thank you in advance

[oleg.feoktistov]

Hi Larry,

Correct, this is how it works since v10.
Confirmed in my lab with v11 that this parameter is being ignored, even though 'Mount to Console' option in File Browser works fine. I'll take some time to duly check it and discuss with QA the upcoming week and keep you posted.

Thanks,
Oleg

[DaKernel]

Thank you Oleg, was scratching my head for awhile there!

[oleg.feoktistov]

QA have been quite busy, so no precise response yet. However, as soon as I have the answer, I'll update the thread. Thanks!

[DaKernel]

Thank you again Oleg for updating the thread on the staus!

[oleg.feoktistov]

Hi Larry,

Confirmed with QA that this is a bug, indeed. -Host parameter should allow you to specify different mount server. As of now, the fix is planned for vNext.

Thanks,
Oleg

[DaKernel]

Ok thanks for that Oleg.

[DaKernel]

Hi Oleg, hope all is well.

Is the fix for this making it into v11a by chance? If not, do you know what version it might be fixed in?

[oleg.feoktistov]

Hi Larry,

Recently we rechecked this case with QA and found out that the logic to mount volumes to a different host is available only when launching FLR from SAN and is related to choosing ESXi host. For other scenarios it is not implemented in Windows FLR core part. But we decided to go forward with it anyways and start with adding "Mount to a Console" feature in Start-VBRWindowsFileRestore cmdlet. It won't make it to v11a, so, as of now, planned for the next major release. The second step will be to allow mounting windows backup to any different host, which we also plan to add to our core part, but no ETA for that.

Thanks,
Oleg

[DaKernel]

So what you are saying is that still cant use the powershell script to verify we have good backups via automation? We will need to wait until v12 for the mount to a console feature to be available?

Thank you.

[oleg.feoktistov]

We will need to wait until v12 for the mount to a console feature to be available?

Yes, correct. As a workaround you can setup backup server or console as a mount server for the repository your backup is residing on for the duration of FLR. Example:

Code: Select all

$mountServer = Get-VBRServer -Name 'Backup Server'
$repository = Get-VBRBackupRepository -Name 'Default Backup Repository'
$mountHostId = $repository.MountHostId
Set-VBRBackupRepository -Repository $repository -MountServer $mountServer 

# Restore process here

# Change mount server back to the old one
$oldMountServer = Get-VBRServer | where {$_.Id -eq $mountHostId}
Set-VBRBackupRepository -Repository $repository -MountServer $oldMountServer

Thanks!