Guest OS Starting up is very slow on Hyper-V 2016

[harry89359]

Hi @wishr

i have got the below reply which i am implementing but the engineer is mentioning that it is not going to reduce the size of vmcx and VMRS files. is there any way to reduce the size.

Regards
Harvinder

The case was internally discussed and we can confirm that by adding the below Registry Value on the Veeam Server will remove all the Reference Points (including natively generated RP by Hyper-V) from the vmcx file but will not shrunk it (You will need to run the Backup Job again to remove the RP). Reference Points can be created by Veeam or natively by Hyper-V. The size of the file will be modified only if you will use a Powershell script. Unfortunately , we are unable to offer support for this method.

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ Name:HyperVReferencePointCleanup
Type: REG_DWORD
Value: 1

Also, a possible solution will be to create a "new" VM and import the disks from the current VM.

In regards with Veeam v10, the RP for V&R job will be removed automatically, according to the job retention setting, unless the VM has a VHDS disk.

[wishr]

Hi Harvinder,

Yes, we do not shrink VM files after ref. points removal. The purpose of the new capability is eliminating the unusual boot times by removing unnecessary ref. points.

Thanks

[harry89359]

Hi @wishr

The problem is sorted using the Reg. key as now VM's starting withing few minutes of the Hyper-V host restarts. Thanks

Regards
Harvinder

[Eluich]

Hi

While waiting for the possible release of a fix, Microsoft asked me to execute the 2 scripts below

Hoping that this can help you

We deleted the RCT reference points manually to fix the issue temporarily using below script:

Script to count the reference points:

Code: Select all

# Pass the Host names of the respective VMs.
$VmNames = Get-VM -ComputerName HOST1, HOST2, HOST3
Foreach ($VmName2 in $VmNames)
{
$VmName = $VmName2.VMName

    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Write the number of refPoints to the console
    Write-Host $VmName $allrefPoints.Count
}

Script to Delete the Reference points:

(Note: It will only delete reference points older than 7 days.This relies on the format of the reference point name including the date, so it may not work if there are any reference points other than Veeam (or Hyper-V Replica) )

Code: Select all

# Pass the Host names of the respective VMs.

$VmNames = Get-VM -ComputerName HOST1, HOST2, HOST3
Foreach ($VmName2 in $VmNames)
{
$VmName = $VmName2.VMName
filter ProcessWMIJob
{
param
(
[WMI]$WmiClass = $null,
[string]$MethodName = $null
)
$errorCode = 0
$returnObject = $_
if ($_.ReturnValue -eq 4096)
{
$Job = [WMI]$_.Job
$returnObject = $Job
while ($Job.JobState -eq 4)
{
Write-Progress -Activity $Job.Caption -Status ($Job.JobStatus + " - " + $Job.PercentComplete + "%") -PercentComplete $Job.PercentComplete
Start-Sleep -seconds 1
$Job.PSBase.Get()
}
if ($Job.JobState -ne 7)
{
  if ($Job.ErrorDescription -ne "")
  {
Write-Error $Job.ErrorDescription
Throw $Job.ErrorDescription
  }
  else
  {
$errorCode = $Job.ErrorCode
  }
}
Write-Progress -Activity $Job.Caption -Status $Job.JobStatus -PercentComplete 100 -Completed:$true
}
elseif($_.ReturnValue -ne 0)
{
 $errorCode = $_.ReturnValue
}
 
if ($errorCode -ne 0)
{
Write-Error "Hyper-V WMI Job Failed!"
if ($WmiClass -and $MethodName)
{
$psWmiClass = [WmiClass]("\\" + $WmiClass.__SERVER + "\" + $WmiClass.__NAMESPACE + ":" + $WmiClass.__CLASS)
$psWmiClass.PSBase.Options.UseAmendedQualifiers = $TRUE
$MethodQualifierValues = ($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["Values"]
$indexOfError = [System.Array]::IndexOf(($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["ValueMap"].Value, [string]$errorCode)
if (($indexOfError -ne "-1") -and $MethodQualifierValues)
{
Throw "ReturnCode: ", $errorCode, " ErrorMessage: '", $MethodQualifierValues.Value[$indexOfError], "' - when calling $MethodName"
}
else
{
Throw "ReturnCode: ", $errorCode, " ErrorMessage: 'MessageNotFound' - when calling $MethodName"
}
}
else
{
Throw "ReturnCode: ", $errorCode, "When calling $MethodName - for rich error messages provide classpath and method name."
}
}
return $returnObject
}
    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
$name = ([WMI]$enum.Current.Dependent).ElementName
# Skip Hyper-V Replica Reference points
If ($name -Notlike "_HVR*")
{
$regex = [regex]"\((.*)\)"
$date = [DateTime]::Parse([regex]::match($name, $regex).Groups[1].ToString().Replace(" - "," "))
$timespan = New-TimeSpan -End $date
# Keep any reference points from the last 7 days
if($timespan.TotalDays -lt -7)
{
Write-Host "Deleting " $name $timespan -ForegroundColor Yellow
$Msvm_VirtualSystemReferencePointService = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_VirtualSystemReferencePointService
# Removes the virtual machine reference, this method returns a job object.
$job = $Msvm_VirtualSystemReferencePointService.DestroyReferencePoint([WMI] $enum.Current.Dependent)
# Waits for the job to complete and processes any errors.
$job | ProcessWMIJob -WmiClass $Msvm_VirtualSystemReferencePointService -MethodName "DestroyReferencePoint" | Out-Null
}
else
{
Write-Host "Skipping " $name $time -ForegroundColor Green
}
     
}    
    }
}

Permanent fix:

The above functionality is already a part of the VEEAM v10 release. Now reference points created by a B&R job will be removed automatically.

[wishr]

Hi Eluich,

I've moved your post to the appropriate thread as the latest discussions in that thread are about a different issue. Also, added a little formatting to make it easier to read.

Generally speaking, the "reference points accumulation issue" is already addressed in VBR v10 (please, see my posts in this particular thread).

Thanks

[rjwolf82]

We are also seeing the same problem for at least three RDS server VM's running Server 2016, we already used the script to delete Reference Points and when we check now we see 0, so there are no Reference Points.
We still have the issue that the RDS servers need 10-15 minutes to boot to Windows, while waiting we only see Hyper-V logo with a spinning circle.
When Veeam creates a backup or replica, the Checkpoint creation is also hanging at 9% for about 15 minutes and then it will start the actual backup or replica.

Somehow it seems related, but even without any Reference Points will still see this slow startup.

[wishr]

Hi Robert,

Sounds strange. Maybe the root cause is different in your case, so I would suggest investigating the issue with the help of our support engineers. It would be also beneficial to raise a case with Microsoft in parallel.

Thanks

[Onward]

FYI, still seeing this issue on V11.

VMs on 2016 take ages to boot up.
5 guest VMs.
15,000 + Reference points on each guest VM.
I've been dealing with this issue for 2 years thinking it was a NIC issue.
How is this still a problem.....

[wishr]

Hi Edmundo,

Starting with v10 VBR automatically removes reference points created by it, so it should not be an issue anymore. See my posts above for more information.

Thanks

[Onward]

@wishr

While this maybe true, this statement is somewhat deceiving. v10 may automatically remove CURRENT reference points, however it does not remove HISTORICAL reference points created by prior versions of Backup & Replication. This is key point that should be noted.

I am at V11 and have spent 24 hour removing reference points from 5 different environments with 4-7 guest VM's and over 15k-20K RCT's for each VM. My VMCX files are over 15-20MB. What is this tool everyone keeps talking about that support only has access too to compact the VMCX files?

Case # 04832628

[wishr]

Yes, you are correct, and that was specifically mentioned on the previous pages of this thread.

We are not technically able to delete reference points created by VBR in the pre-v10 versions because there was no specific flag within RPs in these versions allowing us to differentiate our RPs from RPs created by 3rd-party solutions. So the workarounds are: delete all RPs including those created by 3rd-party solutions and Hyper-V using our official registry key, just keep them, or use scripts specified in this thread to delete those RPs that are not required (the latter is not officially supported).

Not sure what tool you are talking about.

Thanks

[soncscy]

With some quick searching, looks like they might be talking about the following topics:

https://social.technet.microsoft.com/Fo ... rverhyperv

post315882.html#p315882

At least one looks to have had a case? Hard to say

[wishr]

Ah, a tool for shrinking the files using WMI queries. Thanks for sharing, Harvey. I was not aware of it.

Does anyone know if these queries can be performed against a powered-on VM? Personally, I doubt it is technically possible, so a situation when you need to shrink VMCX implies a maintenance downtime of a VM.

Thanks

[BBloke]

Thank you for the script. It has worked a treat at cleaning up the reference points on my VM's so the VMCX files are pretty small. Saw some RP from 2018 in the list!

I've also added the Reg Key so hopefully not more bloated files.

[chuck614]

This script was very good, I have had problems to backup shared disks on my S2D cluster, but now it’s working again.
But I have an Issue with memory usage on the VMMS process.
When Veeam creates a Snapshot, the VM config is saved to C:\ProgramData\microsoft\Windows\Hyper-V\Snapshots Cache\
And when the Cluster Core Service is running on the same server as the VM this file is deleted when the backup is finished.
But if the VM is running on another server the .VMCX file is left there, and memory usage for the VMMS process grows.

Does anyone know if it is possible to clean up the VMCS files under C:\ProgramData\microsoft\Windows\Hyper-V\Snapshots Cache\
I don’t just want to delete them, because they are referred in C:\ProgramData\microsoft\Windows\Hyper-V\data.vmcx

Did you ever find out if you can clean out the Snapshots Cache folder? We can see this same thing happening every time Veeam runs a backup. It is causing OS upgrades on the hosts to take hours to days because of every file is processed in this folder during an OS upgrade.
Thanks.