-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hi Sebastien,
You can refer to my previous comment. There is an official solution that can be obtained through our technical support. It is built into the product, but requires "activation" . It's still not publicly available because we'd like the root cause to be confirmed by out engineers first - simply not to break your backups if something goes unexpected way! Hope it makes sense.
Thanks,
Fedor
You can refer to my previous comment. There is an official solution that can be obtained through our technical support. It is built into the product, but requires "activation" . It's still not publicly available because we'd like the root cause to be confirmed by out engineers first - simply not to break your backups if something goes unexpected way! Hope it makes sense.
Thanks,
Fedor
-
- Service Provider
- Posts: 5
- Liked: 1 time
- Joined: Jun 20, 2019 9:39 pm
- Full Name: Kurt
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Is there a way for the WMI cleanup script to leave a certain amount, say the last 100 reference points? We found a registry key from Veeam support that will cleanup up all reference points after an incremental backup runs, but that breaks anything else that created a reference point, like Hyper-V replica.
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hi Kurt,
No, there is no functionality allowing you to control ref. points deletion so precisely. The underlying mechanism deletes reference points once a task is finished, thus it is required to run a job after you have set the registry key. As of now, from Veeam perspective, you may choose between just two options: delete all the existing ref. points or keep them.
Thanks
No, there is no functionality allowing you to control ref. points deletion so precisely. The underlying mechanism deletes reference points once a task is finished, thus it is required to run a job after you have set the registry key. As of now, from Veeam perspective, you may choose between just two options: delete all the existing ref. points or keep them.
Thanks
-
- Service Provider
- Posts: 239
- Liked: 18 times
- Joined: Mar 23, 2016 5:57 pm
- Full Name: Diogo Campregher
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hi! I'm also facing this issue. In my case, the customer VM have 25TB of disks and run the 01 backup job for each VHDX. The vmcx file is with 68MB and the VM take long time to boot and the live migration doesn't work with time out errors. We will open a support case and comment here.
-
- Service Provider
- Posts: 239
- Liked: 18 times
- Joined: Mar 23, 2016 5:57 pm
- Full Name: Diogo Campregher
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hello!
Are there any Veeam offical documentation about the changes in the registry:
Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ Name:HyperVReferencePointCleanup
Type: REG_DWORD
Value: 1
Can we apply this change without any other impact ?
Are there any Veeam offical documentation about the changes in the registry:
Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ Name:HyperVReferencePointCleanup
Type: REG_DWORD
Value: 1
Can we apply this change without any other impact ?
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hi Diogo,
No, there is no documentation about this reg. key. Please use it ONLY if this has been advised by our support team! It was not supposed to become publicly available.
Please check with our support engineers if it's applicable for your situation.
Thanks
No, there is no documentation about this reg. key. Please use it ONLY if this has been advised by our support team! It was not supposed to become publicly available.
Please check with our support engineers if it's applicable for your situation.
Thanks
-
- Service Provider
- Posts: 5
- Liked: 1 time
- Joined: Jun 20, 2019 9:39 pm
- Full Name: Kurt
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
We got the usual response of "this is a Microsoft problem" Which is funny, since all Microsoft products are able to clean up their reference points without impacting Veeam. Not sure why Veeam can't do the same. For VMs to become unusable in live migration after Veeam backs them up enough times is a bug. Hopefully this registry key is not the permanent fix.
-
- Influencer
- Posts: 13
- Liked: 5 times
- Joined: Apr 12, 2012 9:35 am
- Full Name: Sebastien QUESTIER
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hi,
I improved my script to exclude the reference point used by Hyper-V replica, the name is different from the ones created by Veeam. I didn't find a way to sort the reference points, so I delete all of them, except the Hyper-V replica one, and never had a problem with Veeam for CBT.
If somebody is interested, even at Veeam, I can share my experience.
Kind regards,
Sebastien
I improved my script to exclude the reference point used by Hyper-V replica, the name is different from the ones created by Veeam. I didn't find a way to sort the reference points, so I delete all of them, except the Hyper-V replica one, and never had a problem with Veeam for CBT.
If somebody is interested, even at Veeam, I can share my experience.
Kind regards,
Sebastien
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hello gentlemen,
We'll explore what could be potentially improved in the ref. points deletion logic for future B&R versions. No ETA yet!
@Squestier, sure, feel free to share it with the community, but could you please also make a big red note "please use it at your own risk!" since it's a custom one.
Thanks
We'll explore what could be potentially improved in the ref. points deletion logic for future B&R versions. No ETA yet!
@Squestier, sure, feel free to share it with the community, but could you please also make a big red note "please use it at your own risk!" since it's a custom one.
Thanks
-
- Influencer
- Posts: 13
- Liked: 5 times
- Joined: Apr 12, 2012 9:35 am
- Full Name: Sebastien QUESTIER
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hi,
So, here is my improved script, and yes, use it at your own risk, as advised by Veeam
You can now deal with multiple servers at once if they are remotely manageable of course (so replace XXXX, YYYY and ZZZZ accordingly).
The important part is the exclusion of the reference point for Hyper-V Replica, starting with "_HVR" : there is only one for each vm, so, it's quite easy !!
Please report if you find it useful until Veeam fixes the problem natively.
Kind regards,
Sebastien
Delete-Reference-Points-all-vms.ps1
So, here is my improved script, and yes, use it at your own risk, as advised by Veeam
You can now deal with multiple servers at once if they are remotely manageable of course (so replace XXXX, YYYY and ZZZZ accordingly).
The important part is the exclusion of the reference point for Hyper-V Replica, starting with "_HVR" : there is only one for each vm, so, it's quite easy !!
Please report if you find it useful until Veeam fixes the problem natively.
Kind regards,
Sebastien
Delete-Reference-Points-all-vms.ps1
Code: Select all
$VmNames = Get-VM -ComputerName XXXX, YYYY, ZZZZ
Foreach ($VmName2 in $VmNames)
{
$VmName = $VmName2.VMName
filter ProcessWMIJob
{
param
(
[WMI]$WmiClass = $null,
[string]$MethodName = $null
)
$errorCode = 0
$returnObject = $_
if ($_.ReturnValue -eq 4096)
{
$Job = [WMI]$_.Job
$returnObject = $Job
while ($Job.JobState -eq 4)
{
Write-Progress -Activity $Job.Caption -Status ($Job.JobStatus + " - " + $Job.PercentComplete + "%") -PercentComplete $Job.PercentComplete
Start-Sleep -seconds 1
$Job.PSBase.Get()
}
if ($Job.JobState -ne 7)
{
if ($Job.ErrorDescription -ne "")
{
Write-Error $Job.ErrorDescription
Throw $Job.ErrorDescription
}
else
{
$errorCode = $Job.ErrorCode
}
}
Write-Progress -Activity $Job.Caption -Status $Job.JobStatus -PercentComplete 100 -Completed:$true
}
elseif($_.ReturnValue -ne 0)
{
$errorCode = $_.ReturnValue
}
if ($errorCode -ne 0)
{
Write-Error "Hyper-V WMI Job Failed!"
if ($WmiClass -and $MethodName)
{
$psWmiClass = [WmiClass]("\\" + $WmiClass.__SERVER + "\" + $WmiClass.__NAMESPACE + ":" + $WmiClass.__CLASS)
$psWmiClass.PSBase.Options.UseAmendedQualifiers = $TRUE
$MethodQualifierValues = ($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["Values"]
$indexOfError = [System.Array]::IndexOf(($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["ValueMap"].Value, [string]$errorCode)
if (($indexOfError -ne "-1") -and $MethodQualifierValues)
{
Throw "ReturnCode: ", $errorCode, " ErrorMessage: '", $MethodQualifierValues.Value[$indexOfError], "' - when calling $MethodName"
}
else
{
Throw "ReturnCode: ", $errorCode, " ErrorMessage: 'MessageNotFound' - when calling $MethodName"
}
}
else
{
Throw "ReturnCode: ", $errorCode, "When calling $MethodName - for rich error messages provide classpath and method name."
}
}
return $returnObject
}
# Retrieve an instance of the virtual machine computer system that contains reference points
$Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
# Retrieve all refrence associations of the virtual machine
$allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
# Enumerate across all of the instances and add all recovery points to an array
$virtualSystemRefPoint = @()
$enum = $allrefPoints.GetEnumerator()
$enum.Reset()
while($enum.MoveNext())
{
$Msvm_VirtualSystemReferencePointService = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_VirtualSystemReferencePointService
# Removes the virtual machine reference, this method returns a job object.
If (([WMI] $enum.Current.Dependent).ElementName -Notlike "_HVR*")
{
$job = $Msvm_VirtualSystemReferencePointService.DestroyReferencePoint([WMI] $enum.Current.Dependent)
# Waits for the job to complete and processes any errors.
$job | ProcessWMIJob -WmiClass $Msvm_VirtualSystemReferencePointService -MethodName "DestroyReferencePoint" | Out-Null
}
}
}
-
- Novice
- Posts: 5
- Liked: 3 times
- Joined: Jul 30, 2019 10:53 pm
- Full Name: Karl Bystrak
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
I am so glad i found this post. I am a few months into using Veeam for my replication and my HV Cluster performance has been slowly deteriorating each day. Tonight after waiting 20 minutes for a basic VM to start i started searching the web and found this post which has opened my eyes. I did some searching on my CSVs and found several VMCX files over 10MB. I found 12 VMs with 2000+ reference points.
I modified Squestier's script to produce a list of my VMs and their reference points.
Here is the code. Use at your own risk just like the original:
Be patient. On my 4 node cluster it took about 20 minutes to produce a list of my 43 VMs.
I am going to open a case and get support's blessing to try their fix. I will post a follow up message with the results.
I modified Squestier's script to produce a list of my VMs and their reference points.
Here is the code. Use at your own risk just like the original:
Code: Select all
$VmNames = Get-VM -ComputerName HOST1, HOST2, HOST3, HOST4
Foreach ($VmName2 in $VmNames)
{
$VmName = $VmName2.VMName
# Retrieve an instance of the virtual machine computer system that contains reference points
$Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
# Retrieve all refrence associations of the virtual machine
$allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
# Enumerate across all of the instances and add all recovery points to an array
$virtualSystemRefPoint = @()
$enum = $allrefPoints.GetEnumerator()
$enum.Reset()
while($enum.MoveNext())
{
$virtualSystemRefPoint += ([WMI] $enum.Current.Dependent)
# Removes the virtual machine reference, this method returns a job object.
}
Write-Host $VmName $virtualSystemRefPoint.Count
}
I am going to open a case and get support's blessing to try their fix. I will post a follow up message with the results.
-
- Influencer
- Posts: 10
- Liked: never
- Joined: Apr 26, 2019 11:17 am
- Full Name: Nathan Rashleigh
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
I have exactly the same problem, I found this thread and logged a call with support on July 10th, I am still waiting for a fix which is supposed to be forthcoming, to date I have been given no ETA. This is using Veeam B+R 9.5 Update 3a. I cannot upgrade to update 4b as we have an HP Storeonce which is not supported on this version.
I have ~40 VM's replicated on a 24/7 hour basis some have over 20,000 reference points, these will no longer live migrate ! I have used the code kindly provided by squestier and it will remove the reference points. I have only tested on a few small VM's as Veeam have stated using the code at your own risk but at the moment I m still waiting for the fix. I will update as I get anything back from support.
I have ~40 VM's replicated on a 24/7 hour basis some have over 20,000 reference points, these will no longer live migrate ! I have used the code kindly provided by squestier and it will remove the reference points. I have only tested on a few small VM's as Veeam have stated using the code at your own risk but at the moment I m still waiting for the fix. I will update as I get anything back from support.
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hi Nathan,
Thank you for the feedback. Btw, what's your support case ID?
Thanks
Thank you for the feedback. Btw, what's your support case ID?
Thanks
-
- Novice
- Posts: 5
- Liked: 3 times
- Joined: Jul 30, 2019 10:53 pm
- Full Name: Karl Bystrak
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
I wanted to provide an update on my experience.
It took 4 days to get a response from support upon opening my case. I described my situation and provided a list of my VMs and their reference point count using the modified script i posted earlier. I also provided a link to this forum topic and mentioned that if support had questions they should try and contact wishr for more information on the topic. They gave me the OK to use the registry entry fix and i implemented it immediately.
Their instructions said to start and stop the Veeam Backup Service after adjusting the registry. I found that it as soon as the entry was there it had an effect. Before i could even restart the service i noticed that my replication jobs were taking 20-30 minutes instead of 5 minutes. I checked with the script and those replication jobs taking a long time was due to the removal of the reference points. Once a few of them completed I stopped and started the service and allowed the weekend to pass which got me past all the scheduled jobs.
I checked this morning and I have 0 reference points on all of my VMs now. Also replication jobs are moving a little faster than before. I live migrated a VM between servers several times that previously would not move at all. It migrated immediately like it should. I also stopped and started a few VMs and they started instantly like they should.
The only remaining item that is outstanding still is the VMCX files. They did not change size with the removal of the reference points. Their size doesn't seem to have an effect on the migration or the starting and stopping so I am going to leave them alone for now. I would like to see a solution to fix the file size that doesn't involve rebuilding the VM in such a way that breaks replication. I have too much data to restart my replication for something that is not critical.
It took 4 days to get a response from support upon opening my case. I described my situation and provided a list of my VMs and their reference point count using the modified script i posted earlier. I also provided a link to this forum topic and mentioned that if support had questions they should try and contact wishr for more information on the topic. They gave me the OK to use the registry entry fix and i implemented it immediately.
Their instructions said to start and stop the Veeam Backup Service after adjusting the registry. I found that it as soon as the entry was there it had an effect. Before i could even restart the service i noticed that my replication jobs were taking 20-30 minutes instead of 5 minutes. I checked with the script and those replication jobs taking a long time was due to the removal of the reference points. Once a few of them completed I stopped and started the service and allowed the weekend to pass which got me past all the scheduled jobs.
I checked this morning and I have 0 reference points on all of my VMs now. Also replication jobs are moving a little faster than before. I live migrated a VM between servers several times that previously would not move at all. It migrated immediately like it should. I also stopped and started a few VMs and they started instantly like they should.
The only remaining item that is outstanding still is the VMCX files. They did not change size with the removal of the reference points. Their size doesn't seem to have an effect on the migration or the starting and stopping so I am going to leave them alone for now. I would like to see a solution to fix the file size that doesn't involve rebuilding the VM in such a way that breaks replication. I have too much data to restart my replication for something that is not critical.
-
- Influencer
- Posts: 10
- Liked: never
- Joined: Apr 26, 2019 11:17 am
- Full Name: Nathan Rashleigh
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Thanks for providing your feedback "Kbystrak" very useful and seems to be a similar experience to ours, albeit you have a registry entry fix, I'm unsure if we are able to use that particular fix as we are on 9.5 Update 3a and not Update 4. Nonetheless your post and code have proved very useful, thank you
Hi "wishr" our support case ID is 03658096
Hi "wishr" our support case ID is 03658096
-
- Novice
- Posts: 3
- Liked: never
- Joined: Jun 23, 2017 8:59 am
- Full Name: Anders Alstad
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
This script was very good, I have had problems to backup shared disks on my S2D cluster, but now it’s working again.
But I have an Issue with memory usage on the VMMS process.
When Veeam creates a Snapshot, the VM config is saved to C:\ProgramData\microsoft\Windows\Hyper-V\Snapshots Cache\
And when the Cluster Core Service is running on the same server as the VM this file is deleted when the backup is finished.
But if the VM is running on another server the .VMCX file is left there, and memory usage for the VMMS process grows.
Does anyone know if it is possible to clean up the VMCS files under C:\ProgramData\microsoft\Windows\Hyper-V\Snapshots Cache\
I don’t just want to delete them, because they are referred in C:\ProgramData\microsoft\Windows\Hyper-V\data.vmcx
But I have an Issue with memory usage on the VMMS process.
When Veeam creates a Snapshot, the VM config is saved to C:\ProgramData\microsoft\Windows\Hyper-V\Snapshots Cache\
And when the Cluster Core Service is running on the same server as the VM this file is deleted when the backup is finished.
But if the VM is running on another server the .VMCX file is left there, and memory usage for the VMMS process grows.
Does anyone know if it is possible to clean up the VMCS files under C:\ProgramData\microsoft\Windows\Hyper-V\Snapshots Cache\
I don’t just want to delete them, because they are referred in C:\ProgramData\microsoft\Windows\Hyper-V\data.vmcx
-
- Veteran
- Posts: 528
- Liked: 144 times
- Joined: Aug 20, 2015 9:30 pm
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Here's a faster version of the script to get the count of reference points. You don't need to enumerate the reference points to get the count, which speeds up the script considerable
Code: Select all
$VmNames = Get-VM -ComputerName HOST1, HOST2, HOST3, HOST4
Foreach ($VmName2 in $VmNames)
{
$VmName = $VmName2.VMName
# Retrieve an instance of the virtual machine computer system that contains reference points
$Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
# Retrieve all refrence associations of the virtual machine
$allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
# Write the number of refPoints to the console
Write-Host $VmName $allrefPoints.Count
}
-
- Veteran
- Posts: 528
- Liked: 144 times
- Joined: Aug 20, 2015 9:30 pm
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
And here's an updated version of the delete script that only deletes reference points older than 7 days. I would think deleting the most recent reference point means Veeam loses the changes since the last backup and needs to run a full backup, so I didn't want to risk it
This relies on the format of the reference point name including the date, so it may not work if there are any reference points other than Veeam (or Hyper-V Replica)
This relies on the format of the reference point name including the date, so it may not work if there are any reference points other than Veeam (or Hyper-V Replica)
Code: Select all
$VmNames = Get-VM -ComputerName XXXX, YYYY, ZZZZ
Foreach ($VmName2 in $VmNames)
{
$VmName = $VmName2.VMName
filter ProcessWMIJob
{
param
(
[WMI]$WmiClass = $null,
[string]$MethodName = $null
)
$errorCode = 0
$returnObject = $_
if ($_.ReturnValue -eq 4096)
{
$Job = [WMI]$_.Job
$returnObject = $Job
while ($Job.JobState -eq 4)
{
Write-Progress -Activity $Job.Caption -Status ($Job.JobStatus + " - " + $Job.PercentComplete + "%") -PercentComplete $Job.PercentComplete
Start-Sleep -seconds 1
$Job.PSBase.Get()
}
if ($Job.JobState -ne 7)
{
if ($Job.ErrorDescription -ne "")
{
Write-Error $Job.ErrorDescription
Throw $Job.ErrorDescription
}
else
{
$errorCode = $Job.ErrorCode
}
}
Write-Progress -Activity $Job.Caption -Status $Job.JobStatus -PercentComplete 100 -Completed:$true
}
elseif($_.ReturnValue -ne 0)
{
$errorCode = $_.ReturnValue
}
if ($errorCode -ne 0)
{
Write-Error "Hyper-V WMI Job Failed!"
if ($WmiClass -and $MethodName)
{
$psWmiClass = [WmiClass]("\\" + $WmiClass.__SERVER + "\" + $WmiClass.__NAMESPACE + ":" + $WmiClass.__CLASS)
$psWmiClass.PSBase.Options.UseAmendedQualifiers = $TRUE
$MethodQualifierValues = ($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["Values"]
$indexOfError = [System.Array]::IndexOf(($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["ValueMap"].Value, [string]$errorCode)
if (($indexOfError -ne "-1") -and $MethodQualifierValues)
{
Throw "ReturnCode: ", $errorCode, " ErrorMessage: '", $MethodQualifierValues.Value[$indexOfError], "' - when calling $MethodName"
}
else
{
Throw "ReturnCode: ", $errorCode, " ErrorMessage: 'MessageNotFound' - when calling $MethodName"
}
}
else
{
Throw "ReturnCode: ", $errorCode, "When calling $MethodName - for rich error messages provide classpath and method name."
}
}
return $returnObject
}
# Retrieve an instance of the virtual machine computer system that contains reference points
$Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
# Retrieve all refrence associations of the virtual machine
$allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
# Enumerate across all of the instances and add all recovery points to an array
$virtualSystemRefPoint = @()
$enum = $allrefPoints.GetEnumerator()
$enum.Reset()
while($enum.MoveNext())
{
$name = ([WMI]$enum.Current.Dependent).ElementName
# Skip Hyper-V Replica Reference points
If ($name -Notlike "_HVR*")
{
$regex = [regex]"\((.*)\)"
$date = [DateTime]::Parse([regex]::match($name, $regex).Groups[1].ToString().Replace(" - "," "))
$timespan = New-TimeSpan -End $date
# Keep any reference points from the last 7 days
if($timespan.TotalDays -lt -7)
{
Write-Host "Deleting " $name $timespan -ForegroundColor Yellow
$Msvm_VirtualSystemReferencePointService = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_VirtualSystemReferencePointService
# Removes the virtual machine reference, this method returns a job object.
$job = $Msvm_VirtualSystemReferencePointService.DestroyReferencePoint([WMI] $enum.Current.Dependent)
# Waits for the job to complete and processes any errors.
$job | ProcessWMIJob -WmiClass $Msvm_VirtualSystemReferencePointService -MethodName "DestroyReferencePoint" | Out-Null
}
else
{
Write-Host "Skipping " $name $time -ForegroundColor Green
}
}
}
}
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hello gentlemen,
What if:
1. B&R would be able to distinguish reference points created by its jobs from any other reference points (not just Hyper-V replica);
2. Would be capable of deleting only those reference points that have been created exactly by its jobs.
Would this satisfy your requirements?
@nmdange, thank you for sharing those scripts.
Regards,
Fedor
What if:
1. B&R would be able to distinguish reference points created by its jobs from any other reference points (not just Hyper-V replica);
2. Would be capable of deleting only those reference points that have been created exactly by its jobs.
Would this satisfy your requirements?
@nmdange, thank you for sharing those scripts.
Regards,
Fedor
-
- Veteran
- Posts: 528
- Liked: 144 times
- Joined: Aug 20, 2015 9:30 pm
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
That would be ideal. I found this on CommVault's site so it should be the same idea for Veeam. A reference point for the last backup is retained until the next backup runs for that job. If you have both a backup job and a replication job on the same VM, each job should only delete the reference points for itself, not the other job. The only other situation would be if a backup job is deleted, you might want a way to cleanup reference points that refer to a job that doesn't exist, or just reference points over a certain age (that were created by Veeam).
For virtual machines running in a Windows Server 2016 Hyper-V cluster, the use of checkpoints for backups simplifies the backup process. For each virtual machine, the backup takes a checkpoint that includes only that virtual machine. Once the backup is completed, the checkpoint is converted to a reference point with a unique ID and saved with the backup. Changed block tracking uses the reference point to identify changes since the last backup. The reference point is retained until the next backup is completed and a new checkpoint is generated.
-
- Lurker
- Posts: 1
- Liked: 1 time
- Joined: Dec 10, 2019 10:52 pm
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
@wishr: Yes that would be exactly what we need.
@nmdange: Thank you for your scripts, that has solved the issue we've been experiencing since February
@nmdange: Thank you for your scripts, that has solved the issue we've been experiencing since February
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Feb 06, 2020 9:25 am
- Full Name: Alex Petersen
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Thank you very much nmdange! I've a customer where VM's on their source Hyper-V takes up to 4 hours to get past 'Starting...'
Please fix this Veeam!
Please fix this Veeam!
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hi everyone,
We are happy to say that this functionality is already a part of the v10 release Now reference points created by a B&R job will be removed automatically, according to the job retention setting, unless the VM has a VHDS disk.
We are looking forward to hearing your feedback on this feature once you have a chance to try it out in your environments.
Cheers!
We are happy to say that this functionality is already a part of the v10 release Now reference points created by a B&R job will be removed automatically, according to the job retention setting, unless the VM has a VHDS disk.
We are looking forward to hearing your feedback on this feature once you have a chance to try it out in your environments.
Cheers!
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Mar 22, 2020 12:33 pm
- Full Name: Lubomír
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Thank very much to nmdange and kbystrak for working powershell scripts
-
- Veeam Legend
- Posts: 351
- Liked: 36 times
- Joined: Oct 24, 2016 3:56 pm
- Full Name: Marco Sorrentino
- Location: Ancona - Italy
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hi all,
I'm getting in this issue: last week I rebooted my HV 2016 host and the 2 VM guest took about 30-40 before starting up!
I've run the script, I have 6841 and 10826 reference points!
VMCX files are 16 MB and 40 MB..also RTC files are 20-30-70 MB.
We use VBR 9.5 U3, replica jobs every 30 minutes.
Veeam support suggests me to upgrade the VBR server, but it is impossible due to an old vCloud Director compatibility.
is it not possible to manually add the registry key in our backup server and/or manually run the reference points deletion script?
Case number: 04102675
Thanks
Marco S.
I'm getting in this issue: last week I rebooted my HV 2016 host and the 2 VM guest took about 30-40 before starting up!
I've run the script, I have 6841 and 10826 reference points!
VMCX files are 16 MB and 40 MB..also RTC files are 20-30-70 MB.
We use VBR 9.5 U3, replica jobs every 30 minutes.
Veeam support suggests me to upgrade the VBR server, but it is impossible due to an old vCloud Director compatibility.
is it not possible to manually add the registry key in our backup server and/or manually run the reference points deletion script?
Case number: 04102675
Thanks
Marco S.
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hi Marco,
You may use the scripts specified in this thread (use them at your own risk) or ask technical support for alternatives if you are not willing to upgrade to v10 which has introduced the aforementioned full-auto- and partially-auto- ref. points cleaning mechanisms.
Keep in mind that even if you upgrade to v10, B&R will not be able to automatically clean up the restore points that have been created before the upgrade, however, some manual manipulations with the registry keys can enforce the removal of all existing reference points including those created by non-Veeam products.
I would suggest working closely with engineers to let them help you in choosing the most appropriate solution.
Thanks
You may use the scripts specified in this thread (use them at your own risk) or ask technical support for alternatives if you are not willing to upgrade to v10 which has introduced the aforementioned full-auto- and partially-auto- ref. points cleaning mechanisms.
Keep in mind that even if you upgrade to v10, B&R will not be able to automatically clean up the restore points that have been created before the upgrade, however, some manual manipulations with the registry keys can enforce the removal of all existing reference points including those created by non-Veeam products.
I would suggest working closely with engineers to let them help you in choosing the most appropriate solution.
Thanks
-
- Veeam Legend
- Posts: 351
- Liked: 36 times
- Joined: Oct 24, 2016 3:56 pm
- Full Name: Marco Sorrentino
- Location: Ancona - Italy
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hi @wishr,
I don't understand the "use at your own risk"..
What is the "official" Veeam alternative in my case?
How can the support could help me if I can't upgrade?
Marco S.
I don't understand the "use at your own risk"..
What is the "official" Veeam alternative in my case?
How can the support could help me if I can't upgrade?
Marco S.
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
If you use 9.5 Update 3 the only feasible official solution is to upgrade to 9.5 Update 4 or v10 and utilize the official ref. points cleanup mechanisms implemented in these versions. Alternatively, you may give any of the mentioned in this thread solutions a try but they are not officially supported.
Thanks
Thanks
-
- Novice
- Posts: 3
- Liked: 1 time
- Joined: Apr 15, 2020 8:48 am
- Full Name: Harvinder Singh
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hi @wishr
i have got the same issue and your engineer is not helpful at all and try to blame the Microsoft even though i am using Veeam B&R from day 1 the hyper-v 2016 host installed. my machines are taking 3 hours and 30 minutes to start after Hyperv host reboot. i ran the script to find out the reference point and out of 4 VM's i have got 7488, 7584, 22456 and for the exchange server the script is still running after 24 hours. the size of the exchange VM is .vmcx as 101MB and 36GB for VMRS. i am on Veeam Backup & Replication 9.5 U4a (9.5.4.2753) version. i really need help to sort this out.
Regards
Harry
i have got the same issue and your engineer is not helpful at all and try to blame the Microsoft even though i am using Veeam B&R from day 1 the hyper-v 2016 host installed. my machines are taking 3 hours and 30 minutes to start after Hyperv host reboot. i ran the script to find out the reference point and out of 4 VM's i have got 7488, 7584, 22456 and for the exchange server the script is still running after 24 hours. the size of the exchange VM is .vmcx as 101MB and 36GB for VMRS. i am on Veeam Backup & Replication 9.5 U4a (9.5.4.2753) version. i really need help to sort this out.
Regards
Harry
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Guest OS Starting up is very slow on Hyper-V 2016
Hi Harvinder,
Welcome to Veeam Community Forums and thank you for the feedback.
v9.5 U4a has a registry key (see above in this thread) that activates the removal of all possible reference points, including those created by 3rd-party solutions.
Next, I would definitely recommend upgrading to v10 CP1 since v10 is capable of cleaning up the ref. points created by B&R v10 automatically. Note that ref. points created "before v10" will not be cleaned and to get it sorted you should utilize the aforementioned reg. key.
Please, feel free to escalate the case for assistance with the process - I definitely recommend performing all operations with reference points with the assistance of our engineers.
Thanks
Welcome to Veeam Community Forums and thank you for the feedback.
v9.5 U4a has a registry key (see above in this thread) that activates the removal of all possible reference points, including those created by 3rd-party solutions.
Next, I would definitely recommend upgrading to v10 CP1 since v10 is capable of cleaning up the ref. points created by B&R v10 automatically. Note that ref. points created "before v10" will not be cleaned and to get it sorted you should utilize the aforementioned reg. key.
Please, feel free to escalate the case for assistance with the process - I definitely recommend performing all operations with reference points with the assistance of our engineers.
Thanks
Who is online
Users browsing this forum: No registered users and 12 guests