Discussions specific to the Microsoft Hyper-V hypervisor
harry89359
Novice
Posts: 3
Liked: 1 time
Joined: Apr 15, 2020 8:48 am
Full Name: Harvinder Singh
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by harry89359 »

Hi @wishr

i have got the below reply which i am implementing but the engineer is mentioning that it is not going to reduce the size of vmcx and VMRS files. is there any way to reduce the size.

Regards
Harvinder

The case was internally discussed and we can confirm that by adding the below Registry Value on the Veeam Server will remove all the Reference Points (including natively generated RP by Hyper-V) from the vmcx file but will not shrunk it (You will need to run the Backup Job again to remove the RP). Reference Points can be created by Veeam or natively by Hyper-V. The size of the file will be modified only if you will use a Powershell script. Unfortunately , we are unable to offer support for this method.

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ Name:HyperVReferencePointCleanup
Type: REG_DWORD
Value: 1

Also, a possible solution will be to create a "new" VM and import the disks from the current VM.

In regards with Veeam v10, the RP for V&R job will be removed automatically, according to the job retention setting, unless the VM has a VHDS disk.

wishr
Veeam Software
Posts: 2493
Liked: 325 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Hi Harvinder,

Yes, we do not shrink VM files after ref. points removal. The purpose of the new capability is eliminating the unusual boot times by removing unnecessary ref. points.

Thanks

harry89359
Novice
Posts: 3
Liked: 1 time
Joined: Apr 15, 2020 8:48 am
Full Name: Harvinder Singh
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by harry89359 » 1 person likes this post

Hi @wishr

The problem is sorted using the Reg. key as now VM's starting withing few minutes of the Hyper-V host restarts. Thanks

Regards
Harvinder

Eluich
Influencer
Posts: 13
Liked: 15 times
Joined: Jul 06, 2020 2:31 pm
Contact:

[MERGED] Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich »

Hi

While waiting for the possible release of a fix, Microsoft asked me to execute the 2 scripts below

Hoping that this can help you

We deleted the RCT reference points manually to fix the issue temporarily using below script:

Script to count the reference points:

Code: Select all

# Pass the Host names of the respective VMs.
$VmNames = Get-VM -ComputerName HOST1, HOST2, HOST3
Foreach ($VmName2 in $VmNames)
{
$VmName = $VmName2.VMName

    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Write the number of refPoints to the console
    Write-Host $VmName $allrefPoints.Count
}

Script to Delete the Reference points:

(Note: It will only delete reference points older than 7 days.This relies on the format of the reference point name including the date, so it may not work if there are any reference points other than Veeam (or Hyper-V Replica) )

Code: Select all

# Pass the Host names of the respective VMs.

$VmNames = Get-VM -ComputerName HOST1, HOST2, HOST3
Foreach ($VmName2 in $VmNames)
{
$VmName = $VmName2.VMName
filter ProcessWMIJob
{
param
(
[WMI]$WmiClass = $null,
[string]$MethodName = $null
)
$errorCode = 0
$returnObject = $_
if ($_.ReturnValue -eq 4096)
{
$Job = [WMI]$_.Job
$returnObject = $Job
while ($Job.JobState -eq 4)
{
Write-Progress -Activity $Job.Caption -Status ($Job.JobStatus + " - " + $Job.PercentComplete + "%") -PercentComplete $Job.PercentComplete
Start-Sleep -seconds 1
$Job.PSBase.Get()
}
if ($Job.JobState -ne 7)
{
  if ($Job.ErrorDescription -ne "")
  {
Write-Error $Job.ErrorDescription
Throw $Job.ErrorDescription
  }
  else
  {
$errorCode = $Job.ErrorCode
  }
}
Write-Progress -Activity $Job.Caption -Status $Job.JobStatus -PercentComplete 100 -Completed:$true
}
elseif($_.ReturnValue -ne 0)
{
 $errorCode = $_.ReturnValue
}
 
if ($errorCode -ne 0)
{
Write-Error "Hyper-V WMI Job Failed!"
if ($WmiClass -and $MethodName)
{
$psWmiClass = [WmiClass]("\\" + $WmiClass.__SERVER + "\" + $WmiClass.__NAMESPACE + ":" + $WmiClass.__CLASS)
$psWmiClass.PSBase.Options.UseAmendedQualifiers = $TRUE
$MethodQualifierValues = ($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["Values"]
$indexOfError = [System.Array]::IndexOf(($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["ValueMap"].Value, [string]$errorCode)
if (($indexOfError -ne "-1") -and $MethodQualifierValues)
{
Throw "ReturnCode: ", $errorCode, " ErrorMessage: '", $MethodQualifierValues.Value[$indexOfError], "' - when calling $MethodName"
}
else
{
Throw "ReturnCode: ", $errorCode, " ErrorMessage: 'MessageNotFound' - when calling $MethodName"
}
}
else
{
Throw "ReturnCode: ", $errorCode, "When calling $MethodName - for rich error messages provide classpath and method name."
}
}
return $returnObject
}
    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
$name = ([WMI]$enum.Current.Dependent).ElementName
# Skip Hyper-V Replica Reference points
If ($name -Notlike "_HVR*")
{
$regex = [regex]"\((.*)\)"
$date = [DateTime]::Parse([regex]::match($name, $regex).Groups[1].ToString().Replace(" - "," "))
$timespan = New-TimeSpan -End $date
# Keep any reference points from the last 7 days
if($timespan.TotalDays -lt -7)
{
Write-Host "Deleting " $name $timespan -ForegroundColor Yellow
$Msvm_VirtualSystemReferencePointService = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_VirtualSystemReferencePointService
# Removes the virtual machine reference, this method returns a job object.
$job = $Msvm_VirtualSystemReferencePointService.DestroyReferencePoint([WMI] $enum.Current.Dependent)
# Waits for the job to complete and processes any errors.
$job | ProcessWMIJob -WmiClass $Msvm_VirtualSystemReferencePointService -MethodName "DestroyReferencePoint" | Out-Null
}
else
{
Write-Host "Skipping " $name $time -ForegroundColor Green
}
     
}    
    }
}
Permanent fix:

The above functionality is already a part of the VEEAM v10 release. Now reference points created by a B&R job will be removed automatically.

wishr
Veeam Software
Posts: 2493
Liked: 325 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by wishr »

Hi Eluich,

I've moved your post to the appropriate thread as the latest discussions in that thread are about a different issue. Also, added a little formatting to make it easier to read.

Generally speaking, the "reference points accumulation issue" is already addressed in VBR v10 (please, see my posts in this particular thread).

Thanks

rjwolf82
Service Provider
Posts: 9
Liked: 1 time
Joined: Mar 21, 2019 2:07 pm
Full Name: Robert-Jan
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by rjwolf82 »

We are also seeing the same problem for at least three RDS server VM's running Server 2016, we already used the script to delete Reference Points and when we check now we see 0, so there are no Reference Points.
We still have the issue that the RDS servers need 10-15 minutes to boot to Windows, while waiting we only see Hyper-V logo with a spinning circle.
When Veeam creates a backup or replica, the Checkpoint creation is also hanging at 9% for about 15 minutes and then it will start the actual backup or replica.

Somehow it seems related, but even without any Reference Points will still see this slow startup.

wishr
Veeam Software
Posts: 2493
Liked: 325 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Hi Robert,

Sounds strange. Maybe the root cause is different in your case, so I would suggest investigating the issue with the help of our support engineers. It would be also beneficial to raise a case with Microsoft in parallel.

Thanks

Onward
Lurker
Posts: 2
Liked: never
Joined: Jul 27, 2018 11:07 pm
Full Name: Edmundo Pena
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by Onward »

FYI, still seeing this issue on V11.

VMs on 2016 take ages to boot up.
5 guest VMs.
15,000 + Reference points on each guest VM.
I've been dealing with this issue for 2 years thinking it was a NIC issue.
How is this still a problem.....

wishr
Veeam Software
Posts: 2493
Liked: 325 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Hi Edmundo,

Starting with v10 VBR automatically removes reference points created by it, so it should not be an issue anymore. See my posts above for more information.

Thanks

Onward
Lurker
Posts: 2
Liked: never
Joined: Jul 27, 2018 11:07 pm
Full Name: Edmundo Pena
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by Onward »

@wishr

While this maybe true, this statement is somewhat deceiving. v10 may automatically remove CURRENT reference points, however it does not remove HISTORICAL reference points created by prior versions of Backup & Replication. This is key point that should be noted.

I am at V11 and have spent 24 hour removing reference points from 5 different environments with 4-7 guest VM's and over 15k-20K RCT's for each VM. My VMCX files are over 15-20MB. What is this tool everyone keeps talking about that support only has access too to compact the VMCX files?

Case # 04832628

wishr
Veeam Software
Posts: 2493
Liked: 325 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Yes, you are correct, and that was specifically mentioned on the previous pages of this thread.

We are not technically able to delete reference points created by VBR in the pre-v10 versions because there was no specific flag within RPs in these versions allowing us to differentiate our RPs from RPs created by 3rd-party solutions. So the workarounds are: delete all RPs including those created by 3rd-party solutions and Hyper-V using our official registry key, just keep them, or use scripts specified in this thread to delete those RPs that are not required (the latter is not officially supported).

Not sure what tool you are talking about.

Thanks

soncscy
Veeam Legend
Posts: 408
Liked: 198 times
Joined: Aug 04, 2019 2:57 pm
Full Name: Harvey Carel
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by soncscy » 1 person likes this post

With some quick searching, looks like they might be talking about the following topics:

https://social.technet.microsoft.com/Fo ... rverhyperv

post315882.html#p315882

At least one looks to have had a case? Hard to say

wishr
Veeam Software
Posts: 2493
Liked: 325 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Ah, a tool for shrinking the files using WMI queries. Thanks for sharing, Harvey. I was not aware of it.

Does anyone know if these queries can be performed against a powered-on VM? Personally, I doubt it is technically possible, so a situation when you need to shrink VMCX implies a maintenance downtime of a VM.

Thanks

BBloke
Lurker
Posts: 1
Liked: 1 time
Joined: Feb 10, 2017 2:13 pm
Full Name: Chris Hunton
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by BBloke » 1 person likes this post

Thank you for the script. It has worked a treat at cleaning up the reference points on my VM's so the VMCX files are pretty small. Saw some RP from 2018 in the list!

I've also added the Reg Key so hopefully not more bloated files.

Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests