Discussions specific to the Microsoft Hyper-V hypervisor
wishr
Veeam Software
Posts: 962
Liked: 100 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr » Jun 14, 2019 10:40 am

Hi Sebastien,

You can refer to my previous comment. There is an official solution that can be obtained through our technical support. It is built into the product, but requires "activation" :). It's still not publicly available because we'd like the root cause to be confirmed by out engineers first - simply not to break your backups if something goes unexpected way! Hope it makes sense.

Thanks,
Fedor

chuck614
Novice
Posts: 4
Liked: never
Joined: Jun 20, 2019 9:39 pm
Full Name: Kurt
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by chuck614 » Jun 24, 2019 4:35 pm

Is there a way for the WMI cleanup script to leave a certain amount, say the last 100 reference points? We found a registry key from Veeam support that will cleanup up all reference points after an incremental backup runs, but that breaks anything else that created a reference point, like Hyper-V replica.

wishr
Veeam Software
Posts: 962
Liked: 100 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr » Jun 25, 2019 11:29 am

Hi Kurt,

No, there is no functionality allowing you to control ref. points deletion so precisely. The underlying mechanism deletes reference points once a task is finished, thus it is required to run a job after you have set the registry key. As of now, from Veeam perspective, you may choose between just two options: delete all the existing ref. points or keep them.

Thanks

dcampregher
Service Provider
Posts: 125
Liked: 6 times
Joined: Mar 23, 2016 5:57 pm
Full Name: Diogo Campregher
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by dcampregher » Jul 01, 2019 1:40 am 1 person likes this post

Hi! I'm also facing this issue. In my case, the customer VM have 25TB of disks and run the 01 backup job for each VHDX. The vmcx file is with 68MB and the VM take long time to boot and the live migration doesn't work with time out errors. We will open a support case and comment here.

dcampregher
Service Provider
Posts: 125
Liked: 6 times
Joined: Mar 23, 2016 5:57 pm
Full Name: Diogo Campregher
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by dcampregher » Jul 03, 2019 5:16 pm

Hello!

Are there any Veeam offical documentation about the changes in the registry:

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ Name:HyperVReferencePointCleanup
Type: REG_DWORD
Value: 1

Can we apply this change without any other impact ?

wishr
Veeam Software
Posts: 962
Liked: 100 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr » Jul 04, 2019 8:01 am

Hi Diogo,

No, there is no documentation about this reg. key. Please use it ONLY if this has been advised by our support team! It was not supposed to become publicly available.

Please check with our support engineers if it's applicable for your situation.

Thanks

chuck614
Novice
Posts: 4
Liked: never
Joined: Jun 20, 2019 9:39 pm
Full Name: Kurt
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by chuck614 » Jul 10, 2019 7:47 pm

We got the usual response of "this is a Microsoft problem" Which is funny, since all Microsoft products are able to clean up their reference points without impacting Veeam. Not sure why Veeam can't do the same. For VMs to become unusable in live migration after Veeam backs them up enough times is a bug. Hopefully this registry key is not the permanent fix.

squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier » Jul 11, 2019 7:28 am 1 person likes this post

Hi,

I improved my script to exclude the reference point used by Hyper-V replica, the name is different from the ones created by Veeam. I didn't find a way to sort the reference points, so I delete all of them, except the Hyper-V replica one, and never had a problem with Veeam for CBT.

If somebody is interested, even at Veeam, I can share my experience.

Kind regards,

Sebastien

wishr
Veeam Software
Posts: 962
Liked: 100 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr » Jul 11, 2019 8:04 am

Hello gentlemen,

We'll explore what could be potentially improved in the ref. points deletion logic for future B&R versions. No ETA yet!

@Squestier, sure, feel free to share it with the community, but could you please also make a big red note "please use it at your own risk!" since it's a custom one.

Thanks

squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier » Jul 11, 2019 11:44 am 2 people like this post

Hi,

So, here is my improved script, and yes, use it at your own risk, as advised by Veeam ;-)

You can now deal with multiple servers at once if they are remotely manageable of course (so replace XXXX, YYYY and ZZZZ accordingly).

The important part is the exclusion of the reference point for Hyper-V Replica, starting with "_HVR" : there is only one for each vm, so, it's quite easy !!

Please report if you find it useful until Veeam fixes the problem natively.

Kind regards,

Sebastien

Delete-Reference-Points-all-vms.ps1

Code: Select all

$VmNames = Get-VM -ComputerName XXXX, YYYY, ZZZZ

Foreach ($VmName2 in $VmNames)
{

$VmName = $VmName2.VMName

filter ProcessWMIJob 
{ 
    param 
    ( 
        [WMI]$WmiClass = $null,
        [string]$MethodName = $null
    )
 $errorCode = 0
    $returnObject = $_
    if ($_.ReturnValue -eq 4096) 
    { 
        $Job = [WMI]$_.Job 
        $returnObject = $Job
        while ($Job.JobState -eq 4) 
        { 
            Write-Progress -Activity $Job.Caption -Status ($Job.JobStatus + " - " + $Job.PercentComplete + "%") -PercentComplete $Job.PercentComplete
            Start-Sleep -seconds 1
            $Job.PSBase.Get()
        } 
        if ($Job.JobState -ne 7) 
        { 
   if ($Job.ErrorDescription -ne "")
   {
             Write-Error $Job.ErrorDescription 
             Throw $Job.ErrorDescription 
   }
   else
   {
    $errorCode = $Job.ErrorCode
   }
        } 
        Write-Progress -Activity $Job.Caption -Status $Job.JobStatus -PercentComplete 100 -Completed:$true
        
    }
 elseif($_.ReturnValue -ne 0)
 {
  $errorCode = $_.ReturnValue
 }
 
 if ($errorCode -ne 0) 
    { 
        Write-Error "Hyper-V WMI Job Failed!" 
        if ($WmiClass -and $MethodName)
        {
            $psWmiClass = [WmiClass]("\\" + $WmiClass.__SERVER + "\" + $WmiClass.__NAMESPACE + ":" + $WmiClass.__CLASS)
            $psWmiClass.PSBase.Options.UseAmendedQualifiers = $TRUE
            $MethodQualifierValues = ($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["Values"]
            $indexOfError = [System.Array]::IndexOf(($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["ValueMap"].Value, [string]$errorCode)
            if (($indexOfError -ne "-1") -and $MethodQualifierValues)
            {
                Throw "ReturnCode: ", $errorCode, " ErrorMessage: '", $MethodQualifierValues.Value[$indexOfError], "' - when calling $MethodName"
            }
            else
            {
                Throw "ReturnCode: ", $errorCode, " ErrorMessage: 'MessageNotFound' - when calling $MethodName"
            }
        }
        else
        {
            Throw "ReturnCode: ", $errorCode, "When calling $MethodName - for rich error messages provide classpath and method name."
        }
    } 
 return $returnObject
}

    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
        
         $Msvm_VirtualSystemReferencePointService = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_VirtualSystemReferencePointService
    # Removes the virtual machine reference, this method returns a job object.
    If (([WMI] $enum.Current.Dependent).ElementName -Notlike "_HVR*")
     {
       $job = $Msvm_VirtualSystemReferencePointService.DestroyReferencePoint([WMI] $enum.Current.Dependent)
       # Waits for the job to complete and processes any errors.
       $job | ProcessWMIJob -WmiClass $Msvm_VirtualSystemReferencePointService -MethodName "DestroyReferencePoint" | Out-Null
     
     }    
    }
    
}

kbystrak
Lurker
Posts: 2
Liked: 2 times
Joined: Jul 30, 2019 10:53 pm
Full Name: Karl Bystrak
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by kbystrak » Jul 30, 2019 11:20 pm 2 people like this post

I am so glad i found this post. I am a few months into using Veeam for my replication and my HV Cluster performance has been slowly deteriorating each day. Tonight after waiting 20 minutes for a basic VM to start i started searching the web and found this post which has opened my eyes. I did some searching on my CSVs and found several VMCX files over 10MB. I found 12 VMs with 2000+ reference points. :shock:

I modified Squestier's script to produce a list of my VMs and their reference points.

Here is the code. Use at your own risk just like the original:

Code: Select all

$VmNames = Get-VM -ComputerName HOST1, HOST2, HOST3, HOST4

Foreach ($VmName2 in $VmNames)
{

$VmName = $VmName2.VMName


    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -ComputerName $VmName2.ComputerName -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
        
         $virtualSystemRefPoint += ([WMI] $enum.Current.Dependent)
    # Removes the virtual machine reference, this method returns a job object.
    
    }
 Write-Host $VmName $virtualSystemRefPoint.Count   
}
Be patient. On my 4 node cluster it took about 20 minutes to produce a list of my 43 VMs.

I am going to open a case and get support's blessing to try their fix. I will post a follow up message with the results.

stormman
Influencer
Posts: 10
Liked: never
Joined: Apr 26, 2019 11:17 am
Full Name: Nathan Rashleigh
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by stormman » Aug 05, 2019 12:35 pm

I have exactly the same problem, I found this thread and logged a call with support on July 10th, I am still waiting for a fix which is supposed to be forthcoming, to date I have been given no ETA. This is using Veeam B+R 9.5 Update 3a. I cannot upgrade to update 4b as we have an HP Storeonce which is not supported on this version.

I have ~40 VM's replicated on a 24/7 hour basis some have over 20,000 reference points, these will no longer live migrate ! I have used the code kindly provided by squestier and it will remove the reference points. I have only tested on a few small VM's as Veeam have stated using the code at your own risk but at the moment I m still waiting for the fix. I will update as I get anything back from support.

wishr
Veeam Software
Posts: 962
Liked: 100 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr » Aug 05, 2019 12:40 pm

Hi Nathan,

Thank you for the feedback. Btw, what's your support case ID?

Thanks

kbystrak
Lurker
Posts: 2
Liked: 2 times
Joined: Jul 30, 2019 10:53 pm
Full Name: Karl Bystrak
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by kbystrak » Aug 05, 2019 2:46 pm

I wanted to provide an update on my experience.

It took 4 days to get a response from support upon opening my case. I described my situation and provided a list of my VMs and their reference point count using the modified script i posted earlier. I also provided a link to this forum topic and mentioned that if support had questions they should try and contact wishr for more information on the topic. They gave me the OK to use the registry entry fix and i implemented it immediately.

Their instructions said to start and stop the Veeam Backup Service after adjusting the registry. I found that it as soon as the entry was there it had an effect. Before i could even restart the service i noticed that my replication jobs were taking 20-30 minutes instead of 5 minutes. I checked with the script and those replication jobs taking a long time was due to the removal of the reference points. Once a few of them completed I stopped and started the service and allowed the weekend to pass which got me past all the scheduled jobs.

I checked this morning and I have 0 reference points on all of my VMs now. Also replication jobs are moving a little faster than before. I live migrated a VM between servers several times that previously would not move at all. It migrated immediately like it should. I also stopped and started a few VMs and they started instantly like they should.

The only remaining item that is outstanding still is the VMCX files. They did not change size with the removal of the reference points. Their size doesn't seem to have an effect on the migration or the starting and stopping so I am going to leave them alone for now. I would like to see a solution to fix the file size that doesn't involve rebuilding the VM in such a way that breaks replication. I have too much data to restart my replication for something that is not critical.

stormman
Influencer
Posts: 10
Liked: never
Joined: Apr 26, 2019 11:17 am
Full Name: Nathan Rashleigh
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by stormman » Aug 05, 2019 3:01 pm

Thanks for providing your feedback "Kbystrak" very useful and seems to be a similar experience to ours, albeit you have a registry entry fix, I'm unsure if we are able to use that particular fix as we are on 9.5 Update 3a and not Update 4. Nonetheless your post and code have proved very useful, thank you

Hi "wishr" our support case ID is 03658096

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests