Host-based backup of Microsoft Hyper-V VMs.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Hi Sebastien,

You can refer to my previous comment. There is an official solution that can be obtained through our technical support. It is built into the product, but requires "activation" :). It's still not publicly available because we'd like the root cause to be confirmed by out engineers first - simply not to break your backups if something goes unexpected way! Hope it makes sense.

Thanks,
Fedor
chuck614
Service Provider
Posts: 5
Liked: 1 time
Joined: Jun 20, 2019 9:39 pm
Full Name: Kurt
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by chuck614 »

Is there a way for the WMI cleanup script to leave a certain amount, say the last 100 reference points? We found a registry key from Veeam support that will cleanup up all reference points after an incremental backup runs, but that breaks anything else that created a reference point, like Hyper-V replica.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Hi Kurt,

No, there is no functionality allowing you to control ref. points deletion so precisely. The underlying mechanism deletes reference points once a task is finished, thus it is required to run a job after you have set the registry key. As of now, from Veeam perspective, you may choose between just two options: delete all the existing ref. points or keep them.

Thanks
dcampregher
Service Provider
Posts: 238
Liked: 18 times
Joined: Mar 23, 2016 5:57 pm
Full Name: Diogo Campregher
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by dcampregher » 1 person likes this post

Hi! I'm also facing this issue. In my case, the customer VM have 25TB of disks and run the 01 backup job for each VHDX. The vmcx file is with 68MB and the VM take long time to boot and the live migration doesn't work with time out errors. We will open a support case and comment here.
dcampregher
Service Provider
Posts: 238
Liked: 18 times
Joined: Mar 23, 2016 5:57 pm
Full Name: Diogo Campregher
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by dcampregher »

Hello!

Are there any Veeam offical documentation about the changes in the registry:

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ Name:HyperVReferencePointCleanup
Type: REG_DWORD
Value: 1

Can we apply this change without any other impact ?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Hi Diogo,

No, there is no documentation about this reg. key. Please use it ONLY if this has been advised by our support team! It was not supposed to become publicly available.

Please check with our support engineers if it's applicable for your situation.

Thanks
chuck614
Service Provider
Posts: 5
Liked: 1 time
Joined: Jun 20, 2019 9:39 pm
Full Name: Kurt
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by chuck614 »

We got the usual response of "this is a Microsoft problem" Which is funny, since all Microsoft products are able to clean up their reference points without impacting Veeam. Not sure why Veeam can't do the same. For VMs to become unusable in live migration after Veeam backs them up enough times is a bug. Hopefully this registry key is not the permanent fix.
squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier » 1 person likes this post

Hi,

I improved my script to exclude the reference point used by Hyper-V replica, the name is different from the ones created by Veeam. I didn't find a way to sort the reference points, so I delete all of them, except the Hyper-V replica one, and never had a problem with Veeam for CBT.

If somebody is interested, even at Veeam, I can share my experience.

Kind regards,

Sebastien
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Hello gentlemen,

We'll explore what could be potentially improved in the ref. points deletion logic for future B&R versions. No ETA yet!

@Squestier, sure, feel free to share it with the community, but could you please also make a big red note "please use it at your own risk!" since it's a custom one.

Thanks
squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier » 2 people like this post

Hi,

So, here is my improved script, and yes, use it at your own risk, as advised by Veeam ;-)

You can now deal with multiple servers at once if they are remotely manageable of course (so replace XXXX, YYYY and ZZZZ accordingly).

The important part is the exclusion of the reference point for Hyper-V Replica, starting with "_HVR" : there is only one for each vm, so, it's quite easy !!

Please report if you find it useful until Veeam fixes the problem natively.

Kind regards,

Sebastien

Delete-Reference-Points-all-vms.ps1

Code: Select all

$VmNames = Get-VM -ComputerName XXXX, YYYY, ZZZZ

Foreach ($VmName2 in $VmNames)
{

$VmName = $VmName2.VMName

filter ProcessWMIJob 
{ 
    param 
    ( 
        [WMI]$WmiClass = $null,
        [string]$MethodName = $null
    )
 $errorCode = 0
    $returnObject = $_
    if ($_.ReturnValue -eq 4096) 
    { 
        $Job = [WMI]$_.Job 
        $returnObject = $Job
        while ($Job.JobState -eq 4) 
        { 
            Write-Progress -Activity $Job.Caption -Status ($Job.JobStatus + " - " + $Job.PercentComplete + "%") -PercentComplete $Job.PercentComplete
            Start-Sleep -seconds 1
            $Job.PSBase.Get()
        } 
        if ($Job.JobState -ne 7) 
        { 
   if ($Job.ErrorDescription -ne "")
   {
             Write-Error $Job.ErrorDescription 
             Throw $Job.ErrorDescription 
   }
   else
   {
    $errorCode = $Job.ErrorCode
   }
        } 
        Write-Progress -Activity $Job.Caption -Status $Job.JobStatus -PercentComplete 100 -Completed:$true
        
    }
 elseif($_.ReturnValue -ne 0)
 {
  $errorCode = $_.ReturnValue
 }
 
 if ($errorCode -ne 0) 
    { 
        Write-Error "Hyper-V WMI Job Failed!" 
        if ($WmiClass -and $MethodName)
        {
            $psWmiClass = [WmiClass]("\\" + $WmiClass.__SERVER + "\" + $WmiClass.__NAMESPACE + ":" + $WmiClass.__CLASS)
            $psWmiClass.PSBase.Options.UseAmendedQualifiers = $TRUE
            $MethodQualifierValues = ($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["Values"]
            $indexOfError = [System.Array]::IndexOf(($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["ValueMap"].Value, [string]$errorCode)
            if (($indexOfError -ne "-1") -and $MethodQualifierValues)
            {
                Throw "ReturnCode: ", $errorCode, " ErrorMessage: '", $MethodQualifierValues.Value[$indexOfError], "' - when calling $MethodName"
            }
            else
            {
                Throw "ReturnCode: ", $errorCode, " ErrorMessage: 'MessageNotFound' - when calling $MethodName"
            }
        }
        else
        {
            Throw "ReturnCode: ", $errorCode, "When calling $MethodName - for rich error messages provide classpath and method name."
        }
    } 
 return $returnObject
}

    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
        
         $Msvm_VirtualSystemReferencePointService = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_VirtualSystemReferencePointService
    # Removes the virtual machine reference, this method returns a job object.
    If (([WMI] $enum.Current.Dependent).ElementName -Notlike "_HVR*")
     {
       $job = $Msvm_VirtualSystemReferencePointService.DestroyReferencePoint([WMI] $enum.Current.Dependent)
       # Waits for the job to complete and processes any errors.
       $job | ProcessWMIJob -WmiClass $Msvm_VirtualSystemReferencePointService -MethodName "DestroyReferencePoint" | Out-Null
     
     }    
    }
    
}
kbystrak
Novice
Posts: 5
Liked: 3 times
Joined: Jul 30, 2019 10:53 pm
Full Name: Karl Bystrak
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by kbystrak » 3 people like this post

I am so glad i found this post. I am a few months into using Veeam for my replication and my HV Cluster performance has been slowly deteriorating each day. Tonight after waiting 20 minutes for a basic VM to start i started searching the web and found this post which has opened my eyes. I did some searching on my CSVs and found several VMCX files over 10MB. I found 12 VMs with 2000+ reference points. :shock:

I modified Squestier's script to produce a list of my VMs and their reference points.

Here is the code. Use at your own risk just like the original:

Code: Select all

$VmNames = Get-VM -ComputerName HOST1, HOST2, HOST3, HOST4

Foreach ($VmName2 in $VmNames)
{

$VmName = $VmName2.VMName


    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
        
         $virtualSystemRefPoint += ([WMI] $enum.Current.Dependent)
    # Removes the virtual machine reference, this method returns a job object.
    
    }
 Write-Host $VmName $virtualSystemRefPoint.Count   
}
Be patient. On my 4 node cluster it took about 20 minutes to produce a list of my 43 VMs.

I am going to open a case and get support's blessing to try their fix. I will post a follow up message with the results.
stormman
Influencer
Posts: 10
Liked: never
Joined: Apr 26, 2019 11:17 am
Full Name: Nathan Rashleigh
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by stormman »

I have exactly the same problem, I found this thread and logged a call with support on July 10th, I am still waiting for a fix which is supposed to be forthcoming, to date I have been given no ETA. This is using Veeam B+R 9.5 Update 3a. I cannot upgrade to update 4b as we have an HP Storeonce which is not supported on this version.

I have ~40 VM's replicated on a 24/7 hour basis some have over 20,000 reference points, these will no longer live migrate ! I have used the code kindly provided by squestier and it will remove the reference points. I have only tested on a few small VM's as Veeam have stated using the code at your own risk but at the moment I m still waiting for the fix. I will update as I get anything back from support.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Hi Nathan,

Thank you for the feedback. Btw, what's your support case ID?

Thanks
kbystrak
Novice
Posts: 5
Liked: 3 times
Joined: Jul 30, 2019 10:53 pm
Full Name: Karl Bystrak
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by kbystrak »

I wanted to provide an update on my experience.

It took 4 days to get a response from support upon opening my case. I described my situation and provided a list of my VMs and their reference point count using the modified script i posted earlier. I also provided a link to this forum topic and mentioned that if support had questions they should try and contact wishr for more information on the topic. They gave me the OK to use the registry entry fix and i implemented it immediately.

Their instructions said to start and stop the Veeam Backup Service after adjusting the registry. I found that it as soon as the entry was there it had an effect. Before i could even restart the service i noticed that my replication jobs were taking 20-30 minutes instead of 5 minutes. I checked with the script and those replication jobs taking a long time was due to the removal of the reference points. Once a few of them completed I stopped and started the service and allowed the weekend to pass which got me past all the scheduled jobs.

I checked this morning and I have 0 reference points on all of my VMs now. Also replication jobs are moving a little faster than before. I live migrated a VM between servers several times that previously would not move at all. It migrated immediately like it should. I also stopped and started a few VMs and they started instantly like they should.

The only remaining item that is outstanding still is the VMCX files. They did not change size with the removal of the reference points. Their size doesn't seem to have an effect on the migration or the starting and stopping so I am going to leave them alone for now. I would like to see a solution to fix the file size that doesn't involve rebuilding the VM in such a way that breaks replication. I have too much data to restart my replication for something that is not critical.
stormman
Influencer
Posts: 10
Liked: never
Joined: Apr 26, 2019 11:17 am
Full Name: Nathan Rashleigh
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by stormman »

Thanks for providing your feedback "Kbystrak" very useful and seems to be a similar experience to ours, albeit you have a registry entry fix, I'm unsure if we are able to use that particular fix as we are on 9.5 Update 3a and not Update 4. Nonetheless your post and code have proved very useful, thank you

Hi "wishr" our support case ID is 03658096
aalstad
Novice
Posts: 3
Liked: never
Joined: Jun 23, 2017 8:59 am
Full Name: Anders Alstad
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by aalstad »

This script was very good, I have had problems to backup shared disks on my S2D cluster, but now it’s working again.
But I have an Issue with memory usage on the VMMS process.
When Veeam creates a Snapshot, the VM config is saved to C:\ProgramData\microsoft\Windows\Hyper-V\Snapshots Cache\
And when the Cluster Core Service is running on the same server as the VM this file is deleted when the backup is finished.
But if the VM is running on another server the .VMCX file is left there, and memory usage for the VMMS process grows.

Does anyone know if it is possible to clean up the VMCS files under C:\ProgramData\microsoft\Windows\Hyper-V\Snapshots Cache\
I don’t just want to delete them, because they are referred in C:\ProgramData\microsoft\Windows\Hyper-V\data.vmcx
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by nmdange » 1 person likes this post

Here's a faster version of the script to get the count of reference points. You don't need to enumerate the reference points to get the count, which speeds up the script considerable

Code: Select all

$VmNames = Get-VM -ComputerName HOST1, HOST2, HOST3, HOST4

Foreach ($VmName2 in $VmNames)
{

$VmName = $VmName2.VMName


    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Write the number of refPoints to the console
    Write-Host $VmName $allrefPoints.Count
}

nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by nmdange » 3 people like this post

And here's an updated version of the delete script that only deletes reference points older than 7 days. I would think deleting the most recent reference point means Veeam loses the changes since the last backup and needs to run a full backup, so I didn't want to risk it :)

This relies on the format of the reference point name including the date, so it may not work if there are any reference points other than Veeam (or Hyper-V Replica)

Code: Select all

$VmNames = Get-VM -ComputerName XXXX, YYYY, ZZZZ

Foreach ($VmName2 in $VmNames)
{

$VmName = $VmName2.VMName

	filter ProcessWMIJob 
	{ 
		param 
		( 
			[WMI]$WmiClass = $null,
			[string]$MethodName = $null
		)

		$errorCode = 0
		$returnObject = $_
		if ($_.ReturnValue -eq 4096) 
		{ 
			$Job = [WMI]$_.Job 
			$returnObject = $Job
			while ($Job.JobState -eq 4) 
			{ 
				Write-Progress -Activity $Job.Caption -Status ($Job.JobStatus + " - " + $Job.PercentComplete + "%") -PercentComplete $Job.PercentComplete
				Start-Sleep -seconds 1
				$Job.PSBase.Get()
			} 
			if ($Job.JobState -ne 7) 
			{ 
			   if ($Job.ErrorDescription -ne "")
			   {
						 Write-Error $Job.ErrorDescription 
						 Throw $Job.ErrorDescription 
			   }
			   else
			   {
				$errorCode = $Job.ErrorCode
			   }
			} 
			Write-Progress -Activity $Job.Caption -Status $Job.JobStatus -PercentComplete 100 -Completed:$true
		}
		 elseif($_.ReturnValue -ne 0)
		 {
		  $errorCode = $_.ReturnValue
		 }
 
		if ($errorCode -ne 0) 
		{ 
			Write-Error "Hyper-V WMI Job Failed!" 
			if ($WmiClass -and $MethodName)
			{
				$psWmiClass = [WmiClass]("\\" + $WmiClass.__SERVER + "\" + $WmiClass.__NAMESPACE + ":" + $WmiClass.__CLASS)
				$psWmiClass.PSBase.Options.UseAmendedQualifiers = $TRUE
				$MethodQualifierValues = ($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["Values"]
				$indexOfError = [System.Array]::IndexOf(($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["ValueMap"].Value, [string]$errorCode)
				if (($indexOfError -ne "-1") -and $MethodQualifierValues)
				{
					Throw "ReturnCode: ", $errorCode, " ErrorMessage: '", $MethodQualifierValues.Value[$indexOfError], "' - when calling $MethodName"
				}
				else
				{
					Throw "ReturnCode: ", $errorCode, " ErrorMessage: 'MessageNotFound' - when calling $MethodName"
				}
			}
			else
			{
				Throw "ReturnCode: ", $errorCode, "When calling $MethodName - for rich error messages provide classpath and method name."
			}
		} 
		return $returnObject
	}

    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
		$name = ([WMI]$enum.Current.Dependent).ElementName
		# Skip Hyper-V Replica Reference points
		If ($name -Notlike "_HVR*")
		{
			$regex = [regex]"\((.*)\)"
			$date = [DateTime]::Parse([regex]::match($name, $regex).Groups[1].ToString().Replace(" - "," "))
			$timespan = New-TimeSpan -End $date
			# Keep any reference points from the last 7 days
			if($timespan.TotalDays -lt -7)
			{
				Write-Host "Deleting " $name $timespan -ForegroundColor Yellow
				$Msvm_VirtualSystemReferencePointService = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_VirtualSystemReferencePointService
				# Removes the virtual machine reference, this method returns a job object.
				$job = $Msvm_VirtualSystemReferencePointService.DestroyReferencePoint([WMI] $enum.Current.Dependent)
				# Waits for the job to complete and processes any errors.
				$job | ProcessWMIJob -WmiClass $Msvm_VirtualSystemReferencePointService -MethodName "DestroyReferencePoint" | Out-Null
			}
			else
			{
				Write-Host "Skipping " $name $time -ForegroundColor Green
			}
     
		}    
    }
}


wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Hello gentlemen,

What if:
1. B&R would be able to distinguish reference points created by its jobs from any other reference points (not just Hyper-V replica);
2. Would be capable of deleting only those reference points that have been created exactly by its jobs.

Would this satisfy your requirements?

@nmdange, thank you for sharing those scripts.

Regards,
Fedor
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by nmdange » 2 people like this post

That would be ideal. I found this on CommVault's site so it should be the same idea for Veeam. A reference point for the last backup is retained until the next backup runs for that job. If you have both a backup job and a replication job on the same VM, each job should only delete the reference points for itself, not the other job. The only other situation would be if a backup job is deleted, you might want a way to cleanup reference points that refer to a job that doesn't exist, or just reference points over a certain age (that were created by Veeam).
For virtual machines running in a Windows Server 2016 Hyper-V cluster, the use of checkpoints for backups simplifies the backup process. For each virtual machine, the backup takes a checkpoint that includes only that virtual machine. Once the backup is completed, the checkpoint is converted to a reference point with a unique ID and saved with the backup. Changed block tracking uses the reference point to identify changes since the last backup. The reference point is retained until the next backup is completed and a new checkpoint is generated.
jnturner
Lurker
Posts: 1
Liked: 1 time
Joined: Dec 10, 2019 10:52 pm
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by jnturner » 1 person likes this post

@wishr: Yes that would be exactly what we need.
@nmdange: Thank you for your scripts, that has solved the issue we've been experiencing since February
flcap
Lurker
Posts: 2
Liked: never
Joined: Feb 06, 2020 9:25 am
Full Name: Alex Petersen
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by flcap »

Thank you very much nmdange! I've a customer where VM's on their source Hyper-V takes up to 4 hours to get past 'Starting...'

Please fix this Veeam!
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Hi everyone,

We are happy to say that this functionality is already a part of the v10 release ;) Now reference points created by a B&R job will be removed automatically, according to the job retention setting, unless the VM has a VHDS disk.

We are looking forward to hearing your feedback on this feature once you have a chance to try it out in your environments.

Cheers!
Luboss
Lurker
Posts: 1
Liked: never
Joined: Mar 22, 2020 12:33 pm
Full Name: Lubomír
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by Luboss »

Thank very much to nmdange and kbystrak for working powershell scripts
:-)
mamosorre84
Veeam Legend
Posts: 336
Liked: 34 times
Joined: Oct 24, 2016 3:56 pm
Full Name: Marco Sorrentino
Location: Ancona - Italy
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by mamosorre84 »

Hi all,

I'm getting in this issue: last week I rebooted my HV 2016 host and the 2 VM guest took about 30-40 before starting up!

I've run the script, I have 6841 and 10826 reference points!

VMCX files are 16 MB and 40 MB..also RTC files are 20-30-70 MB.

We use VBR 9.5 U3, replica jobs every 30 minutes.

Veeam support suggests me to upgrade the VBR server, but it is impossible due to an old vCloud Director compatibility.

is it not possible to manually add the registry key in our backup server and/or manually run the reference points deletion script?

Case number: 04102675

Thanks

Marco S.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Hi Marco,

You may use the scripts specified in this thread (use them at your own risk) or ask technical support for alternatives if you are not willing to upgrade to v10 which has introduced the aforementioned full-auto- and partially-auto- ref. points cleaning mechanisms.

Keep in mind that even if you upgrade to v10, B&R will not be able to automatically clean up the restore points that have been created before the upgrade, however, some manual manipulations with the registry keys can enforce the removal of all existing reference points including those created by non-Veeam products.

I would suggest working closely with engineers to let them help you in choosing the most appropriate solution.

Thanks
mamosorre84
Veeam Legend
Posts: 336
Liked: 34 times
Joined: Oct 24, 2016 3:56 pm
Full Name: Marco Sorrentino
Location: Ancona - Italy
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by mamosorre84 »

Hi @wishr,

I don't understand the "use at your own risk"..

What is the "official" Veeam alternative in my case?

How can the support could help me if I can't upgrade?

Marco S.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

If you use 9.5 Update 3 the only feasible official solution is to upgrade to 9.5 Update 4 or v10 and utilize the official ref. points cleanup mechanisms implemented in these versions. Alternatively, you may give any of the mentioned in this thread solutions a try but they are not officially supported.

Thanks
harry89359
Novice
Posts: 3
Liked: 1 time
Joined: Apr 15, 2020 8:48 am
Full Name: Harvinder Singh

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by harry89359 »

Hi @wishr

i have got the same issue and your engineer is not helpful at all and try to blame the Microsoft even though i am using Veeam B&R from day 1 the hyper-v 2016 host installed. my machines are taking 3 hours and 30 minutes to start after Hyperv host reboot. i ran the script to find out the reference point and out of 4 VM's i have got 7488, 7584, 22456 and for the exchange server the script is still running after 24 hours. the size of the exchange VM is .vmcx as 101MB and 36GB for VMRS. i am on Veeam Backup & Replication 9.5 U4a (9.5.4.2753) version. i really need help to sort this out.
Regards
Harry
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Hi Harvinder,

Welcome to Veeam Community Forums and thank you for the feedback.

v9.5 U4a has a registry key (see above in this thread) that activates the removal of all possible reference points, including those created by 3rd-party solutions.

Next, I would definitely recommend upgrading to v10 CP1 since v10 is capable of cleaning up the ref. points created by B&R v10 automatically. Note that ref. points created "before v10" will not be cleaned and to get it sorted you should utilize the aforementioned reg. key.

Please, feel free to escalate the case for assistance with the process - I definitely recommend performing all operations with reference points with the assistance of our engineers.

Thanks
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 35 guests