Discussions specific to the Microsoft Hyper-V hypervisor
wishr
Veeam Software
Posts: 1239
Liked: 122 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr » Jun 14, 2019 10:40 am

Hi Sebastien,

You can refer to my previous comment. There is an official solution that can be obtained through our technical support. It is built into the product, but requires "activation" :). It's still not publicly available because we'd like the root cause to be confirmed by out engineers first - simply not to break your backups if something goes unexpected way! Hope it makes sense.

Thanks,
Fedor

chuck614
Novice
Posts: 4
Liked: 1 time
Joined: Jun 20, 2019 9:39 pm
Full Name: Kurt
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by chuck614 » Jun 24, 2019 4:35 pm

Is there a way for the WMI cleanup script to leave a certain amount, say the last 100 reference points? We found a registry key from Veeam support that will cleanup up all reference points after an incremental backup runs, but that breaks anything else that created a reference point, like Hyper-V replica.

wishr
Veeam Software
Posts: 1239
Liked: 122 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr » Jun 25, 2019 11:29 am

Hi Kurt,

No, there is no functionality allowing you to control ref. points deletion so precisely. The underlying mechanism deletes reference points once a task is finished, thus it is required to run a job after you have set the registry key. As of now, from Veeam perspective, you may choose between just two options: delete all the existing ref. points or keep them.

Thanks

dcampregher
Service Provider
Posts: 137
Liked: 6 times
Joined: Mar 23, 2016 5:57 pm
Full Name: Diogo Campregher
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by dcampregher » Jul 01, 2019 1:40 am 1 person likes this post

Hi! I'm also facing this issue. In my case, the customer VM have 25TB of disks and run the 01 backup job for each VHDX. The vmcx file is with 68MB and the VM take long time to boot and the live migration doesn't work with time out errors. We will open a support case and comment here.

dcampregher
Service Provider
Posts: 137
Liked: 6 times
Joined: Mar 23, 2016 5:57 pm
Full Name: Diogo Campregher
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by dcampregher » Jul 03, 2019 5:16 pm

Hello!

Are there any Veeam offical documentation about the changes in the registry:

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ Name:HyperVReferencePointCleanup
Type: REG_DWORD
Value: 1

Can we apply this change without any other impact ?

wishr
Veeam Software
Posts: 1239
Liked: 122 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr » Jul 04, 2019 8:01 am

Hi Diogo,

No, there is no documentation about this reg. key. Please use it ONLY if this has been advised by our support team! It was not supposed to become publicly available.

Please check with our support engineers if it's applicable for your situation.

Thanks

chuck614
Novice
Posts: 4
Liked: 1 time
Joined: Jun 20, 2019 9:39 pm
Full Name: Kurt
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by chuck614 » Jul 10, 2019 7:47 pm

We got the usual response of "this is a Microsoft problem" Which is funny, since all Microsoft products are able to clean up their reference points without impacting Veeam. Not sure why Veeam can't do the same. For VMs to become unusable in live migration after Veeam backs them up enough times is a bug. Hopefully this registry key is not the permanent fix.

squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier » Jul 11, 2019 7:28 am 1 person likes this post

Hi,

I improved my script to exclude the reference point used by Hyper-V replica, the name is different from the ones created by Veeam. I didn't find a way to sort the reference points, so I delete all of them, except the Hyper-V replica one, and never had a problem with Veeam for CBT.

If somebody is interested, even at Veeam, I can share my experience.

Kind regards,

Sebastien

wishr
Veeam Software
Posts: 1239
Liked: 122 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr » Jul 11, 2019 8:04 am

Hello gentlemen,

We'll explore what could be potentially improved in the ref. points deletion logic for future B&R versions. No ETA yet!

@Squestier, sure, feel free to share it with the community, but could you please also make a big red note "please use it at your own risk!" since it's a custom one.

Thanks

squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier » Jul 11, 2019 11:44 am 2 people like this post

Hi,

So, here is my improved script, and yes, use it at your own risk, as advised by Veeam ;-)

You can now deal with multiple servers at once if they are remotely manageable of course (so replace XXXX, YYYY and ZZZZ accordingly).

The important part is the exclusion of the reference point for Hyper-V Replica, starting with "_HVR" : there is only one for each vm, so, it's quite easy !!

Please report if you find it useful until Veeam fixes the problem natively.

Kind regards,

Sebastien

Delete-Reference-Points-all-vms.ps1

Code: Select all

$VmNames = Get-VM -ComputerName XXXX, YYYY, ZZZZ

Foreach ($VmName2 in $VmNames)
{

$VmName = $VmName2.VMName

filter ProcessWMIJob 
{ 
    param 
    ( 
        [WMI]$WmiClass = $null,
        [string]$MethodName = $null
    )
 $errorCode = 0
    $returnObject = $_
    if ($_.ReturnValue -eq 4096) 
    { 
        $Job = [WMI]$_.Job 
        $returnObject = $Job
        while ($Job.JobState -eq 4) 
        { 
            Write-Progress -Activity $Job.Caption -Status ($Job.JobStatus + " - " + $Job.PercentComplete + "%") -PercentComplete $Job.PercentComplete
            Start-Sleep -seconds 1
            $Job.PSBase.Get()
        } 
        if ($Job.JobState -ne 7) 
        { 
   if ($Job.ErrorDescription -ne "")
   {
             Write-Error $Job.ErrorDescription 
             Throw $Job.ErrorDescription 
   }
   else
   {
    $errorCode = $Job.ErrorCode
   }
        } 
        Write-Progress -Activity $Job.Caption -Status $Job.JobStatus -PercentComplete 100 -Completed:$true
        
    }
 elseif($_.ReturnValue -ne 0)
 {
  $errorCode = $_.ReturnValue
 }
 
 if ($errorCode -ne 0) 
    { 
        Write-Error "Hyper-V WMI Job Failed!" 
        if ($WmiClass -and $MethodName)
        {
            $psWmiClass = [WmiClass]("\\" + $WmiClass.__SERVER + "\" + $WmiClass.__NAMESPACE + ":" + $WmiClass.__CLASS)
            $psWmiClass.PSBase.Options.UseAmendedQualifiers = $TRUE
            $MethodQualifierValues = ($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["Values"]
            $indexOfError = [System.Array]::IndexOf(($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["ValueMap"].Value, [string]$errorCode)
            if (($indexOfError -ne "-1") -and $MethodQualifierValues)
            {
                Throw "ReturnCode: ", $errorCode, " ErrorMessage: '", $MethodQualifierValues.Value[$indexOfError], "' - when calling $MethodName"
            }
            else
            {
                Throw "ReturnCode: ", $errorCode, " ErrorMessage: 'MessageNotFound' - when calling $MethodName"
            }
        }
        else
        {
            Throw "ReturnCode: ", $errorCode, "When calling $MethodName - for rich error messages provide classpath and method name."
        }
    } 
 return $returnObject
}

    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
        
         $Msvm_VirtualSystemReferencePointService = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_VirtualSystemReferencePointService
    # Removes the virtual machine reference, this method returns a job object.
    If (([WMI] $enum.Current.Dependent).ElementName -Notlike "_HVR*")
     {
       $job = $Msvm_VirtualSystemReferencePointService.DestroyReferencePoint([WMI] $enum.Current.Dependent)
       # Waits for the job to complete and processes any errors.
       $job | ProcessWMIJob -WmiClass $Msvm_VirtualSystemReferencePointService -MethodName "DestroyReferencePoint" | Out-Null
     
     }    
    }
    
}

kbystrak
Lurker
Posts: 2
Liked: 2 times
Joined: Jul 30, 2019 10:53 pm
Full Name: Karl Bystrak
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by kbystrak » Jul 30, 2019 11:20 pm 2 people like this post

I am so glad i found this post. I am a few months into using Veeam for my replication and my HV Cluster performance has been slowly deteriorating each day. Tonight after waiting 20 minutes for a basic VM to start i started searching the web and found this post which has opened my eyes. I did some searching on my CSVs and found several VMCX files over 10MB. I found 12 VMs with 2000+ reference points. :shock:

I modified Squestier's script to produce a list of my VMs and their reference points.

Here is the code. Use at your own risk just like the original:

Code: Select all

$VmNames = Get-VM -ComputerName HOST1, HOST2, HOST3, HOST4

Foreach ($VmName2 in $VmNames)
{

$VmName = $VmName2.VMName


    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
        
         $virtualSystemRefPoint += ([WMI] $enum.Current.Dependent)
    # Removes the virtual machine reference, this method returns a job object.
    
    }
 Write-Host $VmName $virtualSystemRefPoint.Count   
}
Be patient. On my 4 node cluster it took about 20 minutes to produce a list of my 43 VMs.

I am going to open a case and get support's blessing to try their fix. I will post a follow up message with the results.

stormman
Influencer
Posts: 10
Liked: never
Joined: Apr 26, 2019 11:17 am
Full Name: Nathan Rashleigh
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by stormman » Aug 05, 2019 12:35 pm

I have exactly the same problem, I found this thread and logged a call with support on July 10th, I am still waiting for a fix which is supposed to be forthcoming, to date I have been given no ETA. This is using Veeam B+R 9.5 Update 3a. I cannot upgrade to update 4b as we have an HP Storeonce which is not supported on this version.

I have ~40 VM's replicated on a 24/7 hour basis some have over 20,000 reference points, these will no longer live migrate ! I have used the code kindly provided by squestier and it will remove the reference points. I have only tested on a few small VM's as Veeam have stated using the code at your own risk but at the moment I m still waiting for the fix. I will update as I get anything back from support.

wishr
Veeam Software
Posts: 1239
Liked: 122 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr » Aug 05, 2019 12:40 pm

Hi Nathan,

Thank you for the feedback. Btw, what's your support case ID?

Thanks

kbystrak
Lurker
Posts: 2
Liked: 2 times
Joined: Jul 30, 2019 10:53 pm
Full Name: Karl Bystrak
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by kbystrak » Aug 05, 2019 2:46 pm

I wanted to provide an update on my experience.

It took 4 days to get a response from support upon opening my case. I described my situation and provided a list of my VMs and their reference point count using the modified script i posted earlier. I also provided a link to this forum topic and mentioned that if support had questions they should try and contact wishr for more information on the topic. They gave me the OK to use the registry entry fix and i implemented it immediately.

Their instructions said to start and stop the Veeam Backup Service after adjusting the registry. I found that it as soon as the entry was there it had an effect. Before i could even restart the service i noticed that my replication jobs were taking 20-30 minutes instead of 5 minutes. I checked with the script and those replication jobs taking a long time was due to the removal of the reference points. Once a few of them completed I stopped and started the service and allowed the weekend to pass which got me past all the scheduled jobs.

I checked this morning and I have 0 reference points on all of my VMs now. Also replication jobs are moving a little faster than before. I live migrated a VM between servers several times that previously would not move at all. It migrated immediately like it should. I also stopped and started a few VMs and they started instantly like they should.

The only remaining item that is outstanding still is the VMCX files. They did not change size with the removal of the reference points. Their size doesn't seem to have an effect on the migration or the starting and stopping so I am going to leave them alone for now. I would like to see a solution to fix the file size that doesn't involve rebuilding the VM in such a way that breaks replication. I have too much data to restart my replication for something that is not critical.

stormman
Influencer
Posts: 10
Liked: never
Joined: Apr 26, 2019 11:17 am
Full Name: Nathan Rashleigh
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by stormman » Aug 05, 2019 3:01 pm

Thanks for providing your feedback "Kbystrak" very useful and seems to be a similar experience to ours, albeit you have a registry entry fix, I'm unsure if we are able to use that particular fix as we are on 9.5 Update 3a and not Update 4. Nonetheless your post and code have proved very useful, thank you

Hi "wishr" our support case ID is 03658096

aalstad
Novice
Posts: 3
Liked: never
Joined: Jun 23, 2017 8:59 am
Full Name: Anders Alstad
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by aalstad » Aug 19, 2019 3:44 pm

This script was very good, I have had problems to backup shared disks on my S2D cluster, but now it’s working again.
But I have an Issue with memory usage on the VMMS process.
When Veeam creates a Snapshot, the VM config is saved to C:\ProgramData\microsoft\Windows\Hyper-V\Snapshots Cache\
And when the Cluster Core Service is running on the same server as the VM this file is deleted when the backup is finished.
But if the VM is running on another server the .VMCX file is left there, and memory usage for the VMMS process grows.

Does anyone know if it is possible to clean up the VMCS files under C:\ProgramData\microsoft\Windows\Hyper-V\Snapshots Cache\
I don’t just want to delete them, because they are referred in C:\ProgramData\microsoft\Windows\Hyper-V\data.vmcx

nmdange
Expert
Posts: 472
Liked: 114 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by nmdange » Aug 19, 2019 8:28 pm 1 person likes this post

Here's a faster version of the script to get the count of reference points. You don't need to enumerate the reference points to get the count, which speeds up the script considerable

Code: Select all

$VmNames = Get-VM -ComputerName HOST1, HOST2, HOST3, HOST4

Foreach ($VmName2 in $VmNames)
{

$VmName = $VmName2.VMName


    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Write the number of refPoints to the console
    Write-Host $VmName $allrefPoints.Count
}


nmdange
Expert
Posts: 472
Liked: 114 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by nmdange » Aug 20, 2019 4:23 pm 2 people like this post

And here's an updated version of the delete script that only deletes reference points older than 7 days. I would think deleting the most recent reference point means Veeam loses the changes since the last backup and needs to run a full backup, so I didn't want to risk it :)

This relies on the format of the reference point name including the date, so it may not work if there are any reference points other than Veeam (or Hyper-V Replica)

Code: Select all

$VmNames = Get-VM -ComputerName XXXX, YYYY, ZZZZ

Foreach ($VmName2 in $VmNames)
{

$VmName = $VmName2.VMName

	filter ProcessWMIJob 
	{ 
		param 
		( 
			[WMI]$WmiClass = $null,
			[string]$MethodName = $null
		)

		$errorCode = 0
		$returnObject = $_
		if ($_.ReturnValue -eq 4096) 
		{ 
			$Job = [WMI]$_.Job 
			$returnObject = $Job
			while ($Job.JobState -eq 4) 
			{ 
				Write-Progress -Activity $Job.Caption -Status ($Job.JobStatus + " - " + $Job.PercentComplete + "%") -PercentComplete $Job.PercentComplete
				Start-Sleep -seconds 1
				$Job.PSBase.Get()
			} 
			if ($Job.JobState -ne 7) 
			{ 
			   if ($Job.ErrorDescription -ne "")
			   {
						 Write-Error $Job.ErrorDescription 
						 Throw $Job.ErrorDescription 
			   }
			   else
			   {
				$errorCode = $Job.ErrorCode
			   }
			} 
			Write-Progress -Activity $Job.Caption -Status $Job.JobStatus -PercentComplete 100 -Completed:$true
		}
		 elseif($_.ReturnValue -ne 0)
		 {
		  $errorCode = $_.ReturnValue
		 }
 
		if ($errorCode -ne 0) 
		{ 
			Write-Error "Hyper-V WMI Job Failed!" 
			if ($WmiClass -and $MethodName)
			{
				$psWmiClass = [WmiClass]("\\" + $WmiClass.__SERVER + "\" + $WmiClass.__NAMESPACE + ":" + $WmiClass.__CLASS)
				$psWmiClass.PSBase.Options.UseAmendedQualifiers = $TRUE
				$MethodQualifierValues = ($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["Values"]
				$indexOfError = [System.Array]::IndexOf(($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["ValueMap"].Value, [string]$errorCode)
				if (($indexOfError -ne "-1") -and $MethodQualifierValues)
				{
					Throw "ReturnCode: ", $errorCode, " ErrorMessage: '", $MethodQualifierValues.Value[$indexOfError], "' - when calling $MethodName"
				}
				else
				{
					Throw "ReturnCode: ", $errorCode, " ErrorMessage: 'MessageNotFound' - when calling $MethodName"
				}
			}
			else
			{
				Throw "ReturnCode: ", $errorCode, "When calling $MethodName - for rich error messages provide classpath and method name."
			}
		} 
		return $returnObject
	}

    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
		$name = ([WMI]$enum.Current.Dependent).ElementName
		# Skip Hyper-V Replica Reference points
		If ($name -Notlike "_HVR*")
		{
			$regex = [regex]"\((.*)\)"
			$date = [DateTime]::Parse([regex]::match($name, $regex).Groups[1].ToString().Replace(" - "," "))
			$timespan = New-TimeSpan -End $date
			# Keep any reference points from the last 7 days
			if($timespan.TotalDays -lt -7)
			{
				Write-Host "Deleting " $name $timespan -ForegroundColor Yellow
				$Msvm_VirtualSystemReferencePointService = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_VirtualSystemReferencePointService
				# Removes the virtual machine reference, this method returns a job object.
				$job = $Msvm_VirtualSystemReferencePointService.DestroyReferencePoint([WMI] $enum.Current.Dependent)
				# Waits for the job to complete and processes any errors.
				$job | ProcessWMIJob -WmiClass $Msvm_VirtualSystemReferencePointService -MethodName "DestroyReferencePoint" | Out-Null
			}
			else
			{
				Write-Host "Skipping " $name $time -ForegroundColor Green
			}
     
		}    
    }
}



wishr
Veeam Software
Posts: 1239
Liked: 122 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr » Aug 21, 2019 9:08 am

Hello gentlemen,

What if:
1. B&R would be able to distinguish reference points created by its jobs from any other reference points (not just Hyper-V replica);
2. Would be capable of deleting only those reference points that have been created exactly by its jobs.

Would this satisfy your requirements?

@nmdange, thank you for sharing those scripts.

Regards,
Fedor

nmdange
Expert
Posts: 472
Liked: 114 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by nmdange » Aug 21, 2019 2:01 pm 1 person likes this post

That would be ideal. I found this on CommVault's site so it should be the same idea for Veeam. A reference point for the last backup is retained until the next backup runs for that job. If you have both a backup job and a replication job on the same VM, each job should only delete the reference points for itself, not the other job. The only other situation would be if a backup job is deleted, you might want a way to cleanup reference points that refer to a job that doesn't exist, or just reference points over a certain age (that were created by Veeam).
For virtual machines running in a Windows Server 2016 Hyper-V cluster, the use of checkpoints for backups simplifies the backup process. For each virtual machine, the backup takes a checkpoint that includes only that virtual machine. Once the backup is completed, the checkpoint is converted to a reference point with a unique ID and saved with the backup. Changed block tracking uses the reference point to identify changes since the last backup. The reference point is retained until the next backup is completed and a new checkpoint is generated.

Post Reply

Who is online

Users browsing this forum: Michael R and 10 guests