Guest OS Starting up is very slow on Hyper-V 2016

[nkoji]

Hi, Guys .

I'm in trouble that Hyper-V guest os starting up is very slow after I added guest os in "Edit Replication Job" window.
When I restart guest os restart , it takes about a hour to start up...
Please give me any information and advises to fix this situation.

My Configuration is followings

Veeam Backup & Replication 9.5 u3.

Host OS A : Windows Server 2016 (Veeam Backup & Replication 9.5 u3 components)
-> 2 Guest OS: Windows Server 2016 both
This is replication source.

Host OS B : Windows Server 2016 (Veeam Backup & Replication 9.5 u3)
This is replication target.

Best Regards !

[PTide]

Hi and welcome to the community!

Please check the following:

1. Does the guest have any checkpoints except those which were created by Veeam?
2. Do you perform reboot during replication session?

If there are no other checkpoints and the reboot is performed outside of replication windows, then please contact our support team so they can investigate.

Thanks

[squestier]

Hi guys,

Almost the same problem with many customers, nearly every host with native Hyper-V replica vms "waits" a very very long time before vms start up, after the host rebooted (so at least once a month with WindowsUpdates).

I just realize Hyper-V hosts without Veeam components are not affected.

Any idea or link ?

Kind regards,

Sebastien, France

[Mike Resseler]

Hey Sebastien,

Since the OP didn't leave a support case ID for us to look, I suggest you create one to see what might be the issue. Please post the case ID here. Afterwards, please post the outcome from the investigation by our engineers.

Thanks
Mike

[squestier]

Hi Mike,

Case ID is 03189568.

Hope we will find an explanation…

Kind regards,

Sebastien

[Mike Resseler]

Indeed. Keep us informed after the investigation

[hasoft]

Any progres on this issue. Wa are also affected. Problem is with live migration too (virtual guest is stopping-starting takes very long).

Thanks
Zdenek

[Mike Resseler]

Hi Zdenek,

First: Welcome to the forums!
Second: The OP has opened the case but never responded back to us after that. So we have no more information. Can you create a support case and post the case ID here?

Many thanks
Mike

[hasoft]

Hi Mike,

case id is 03218147.

Zdenek

[squestier]

Hi guys,

To everybody facing this issue : I discovered something regarding the size of the vmcx files.

When you first build a vm with Hyper-V, the config file is around 60-70 KB. All my buggy vms have much bigger vmcx config files, around 60MB for the biggest one.

For my case, THESE huge vmcx files ARE the problem when rebooting the physical server, sometimes almost 30-60mn waiting for the vm to power on !!

Could you check every vmcx config file size on your servers ?

Please report, I may have a solution.

Kind regards,

Sebastien

[PTide]

When you first build a vm with Hyper-V, the config file is around 60-70 KB. All my buggy vms have much bigger vmcx config files, around 60MB for the biggest one.

I wonder what occupies so much space...Did you check what was inside?

Thanks

[Mike Resseler]

Sebastien,

VMCX are binary files so it is not possible to read inside anymore (which used to be possible in previous hyper-v). If they are that large, it means Hyper-V needs to read out all the information which is inside there (config/ devices/ ...). 60 MB just seems too big and probably means there is a lot of information in there which shouldn't be there. I would suggest to contact MSFT support with this to fix those VMCX files.

[squestier]

Hi Mike,

Indeed, we can barely "read" some vmcx config info with PowerShell cmdlets like "compare-vm" but Nothing useful enough to pinpoint what's wrong.

I think I found a workaround : when you create a native Hyper-V replica, the replica vm is "clean", ie its vmcx file size is 60-70k only.

So, with this "yet empty" replica vm, I just remove the vhd and export the vm.

On the "master" Host, I delete the vm (it does not delete the vhd), rename the vm folder, copy the small exported vm (witout the vhd) from the "replica" host, import the vm again and reattach the full vhd and the new vm will then boot with no delay.

And just create the replication again if needed.

Quick and Dirty, but seems to work great.

I hope it will help some people.

Regards

Sebastien

[Mike Resseler]

Sebastien,

That means the problem might be with Hyper-V replica. And that also means it probably will resurface. Did you contacted Microsoft around this?

[squestier]

Hi Mike,

Not yet but I feel like it's also somewhere related to Veeam because of the snapshots Veeam creates for eack backup. All my customers use Veeam and I can't let critical replicated servers without the too precious Veeam protection, so it's not easy to diagnose.

I will try to reach Microsoft but I don't even know where to open a ticket for this kind of problem. Any advice ?

Regards,

Sebastien

PS : You're From Belgium, do you speak French ?

[hasoft]

Hi all,

problem is with ReferencePoints inside VM. Veeam developers know about this problem and I think they fix this bug in future version. In brief: after succesful backup Snapshot is converted to ReferencePoint. ReferencePoints help incremental backups to calculate changes between incremental backups. Accidentaly these reference points are not deleted. When VM is stopped, reference points are stored to vmcx file (make it grow and grow). Every backup create one reference point. Replicas run more often than regular (for example one day) backup and problem is more visible. Replicas simple create more RestorePoints.

Till developers repair this problem, you can delete restore points manually using VMI. I can post code if you wish.

Zdenek

[hasoft]

typo in last sentence:

Till developers repair this problem, you can delete referene points manually using VMI.

[Mike Resseler]

Hey Hasoft,

Is that a problem with Veeam replication? Because previous poster is talking about Hyper-V replication.

PS: @Sebastien: I am from the Flemish part but I do speak French

[hasoft]

It's common problem with snapshots and reference points on W2016. Can you access support id 03218147? That case contain all infos about problem.

[squestier]

Thanks Zdenek,

Almost 15.000 Reference Points Inside several config files !!!

Already started a good cleanup with WMI Method "DestroyReferencePoint". Vm start delay is now back to normal !!

Would be cool if Veeam could warn for these hidden Reference Points.

Kind regards,

Sebastien

[nmdange]

I wouldn't mind seeing the code to delete the reference points manually just in case I run into this problem.

[hasoft]

Already started a good cleanup with WMI Method "DestroyReferencePoint". Vm start delay is now back to normal !!

it's a good practice to leave recent ReferencePoints to Veeam Backup can use them. I do daily backup and 30 min replication so I delete Reference Points older than week (for sure).

[squestier]

Main source code :
https://blogs.msdn.microsoft.com/taylor ... l-scripts/

List-reference-points.ps1
Display how many Reference Points exist for one vm :

Code: Select all

cls
$VmName = 'XXXXXXXX'

    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
      $virtualSystemRefPoint += ([WMI] $enum.Current.Dependent)
    }
    # Return the array of recovery points

$virtualSystemRefPoint.Count

[squestier]

Second part

Delete-Reference-Points.ps1

Code: Select all

$VmName = 'XXXXXXXX'

filter ProcessWMIJob 
{ 
    param 
    ( 
        [WMI]$WmiClass = $null,
        [string]$MethodName = $null
    )
 $errorCode = 0
    $returnObject = $_
    if ($_.ReturnValue -eq 4096) 
    { 
        $Job = [WMI]$_.Job 
        $returnObject = $Job
        while ($Job.JobState -eq 4) 
        { 
            Write-Progress -Activity $Job.Caption -Status ($Job.JobStatus + " - " + $Job.PercentComplete + "%") -PercentComplete $Job.PercentComplete
            Start-Sleep -seconds 1
            $Job.PSBase.Get()
        } 
        if ($Job.JobState -ne 7) 
        { 
   if ($Job.ErrorDescription -ne "")
   {
             Write-Error $Job.ErrorDescription 
             Throw $Job.ErrorDescription 
   }
   else
   {
    $errorCode = $Job.ErrorCode
   }
        } 
        Write-Progress -Activity $Job.Caption -Status $Job.JobStatus -PercentComplete 100 -Completed:$true
        
    }
 elseif($_.ReturnValue -ne 0)
 {
  $errorCode = $_.ReturnValue
 }
 
 if ($errorCode -ne 0) 
    { 
        Write-Error "Hyper-V WMI Job Failed!" 
        if ($WmiClass -and $MethodName)
        {
            $psWmiClass = [WmiClass]("\\" + $WmiClass.__SERVER + "\" + $WmiClass.__NAMESPACE + ":" + $WmiClass.__CLASS)
            $psWmiClass.PSBase.Options.UseAmendedQualifiers = $TRUE
            $MethodQualifierValues = ($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["Values"]
            $indexOfError = [System.Array]::IndexOf(($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["ValueMap"].Value, [string]$errorCode)
            if (($indexOfError -ne "-1") -and $MethodQualifierValues)
            {
                Throw "ReturnCode: ", $errorCode, " ErrorMessage: '", $MethodQualifierValues.Value[$indexOfError], "' - when calling $MethodName"
            }
            else
            {
                Throw "ReturnCode: ", $errorCode, " ErrorMessage: 'MessageNotFound' - when calling $MethodName"
            }
        }
        else
        {
            Throw "ReturnCode: ", $errorCode, "When calling $MethodName - for rich error messages provide classpath and method name."
        }
    } 
 return $returnObject
}

    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
        
    $Msvm_VirtualSystemReferencePointService = Get-WmiObject -Namespace root\virtualization\v2 -Class Msvm_VirtualSystemReferencePointService
    # Removes the virtual machine reference, this method returns a job object.
    $job = $Msvm_VirtualSystemReferencePointService.DestroyReferencePoint([WMI] $enum.Current.Dependent)
    # Waits for the job to complete and processes any errors.
    ($job | ProcessWMIJob -WmiClass $Msvm_VirtualSystemReferencePointService -MethodName "DestroyReferencePoint") | Out-Null
         
    }
    # !!! Deletes all Reference Points !!!

[renevw]

Hello All,

Great updates, i have checked the scripts and also found out that the VMs starting verry slow are having around 12000 reference points.
I am running an combination of continuously and hourly replication jobs.

I was wondering if any of you is also an memory "leaking" problem on the HV host. I think it can be related. I experienced the slow starting problem because i had to reboot the HV host from time to time to free up memory. I found that the vmm service is using up to 12 GB of RAM. The Host it self has 96 GB of fysical RAM, i have about 72 GB of ram (static) allocated to the VMs. The OS uses some + the 12 GB for vmm i have about 5 GB of free memory. The used memory grows slowly over time. At this point (5 GB memory free) the back-up jobs start having problems because of resource.

The (slave) HV Host (identical) which holds te replica's and 2 active vms for Veeam and AD does run stable for months without any memory problems. No VMs are replicated from this HOST.

03218147

[Gerdes]

Hello,

I got the same issue here. I'm running 5 VMs. Backup and Replication ist only created with Veeam.

One VM has actually over 26.000 Reference Points!!

Whats the recommendation to solve this issue? Is there any update available? Or do I need to delete alle the Reference Point by the script posted by squestier?

Regards,
Stephan

[wishr]

Hi Stephan,

Sure, as of now we recommend you to reach out to our support team directly to get an approved solution to reference points accumulation.

Thanks

[tejko30]

Hi,

Since we are experiencing this issue as well... Do we have a publicly available solution to deal with this? What is Veeam's approved solution?

Thanks!

[Mike Resseler]

Hey Matej,

First: Welcome to the forums
Second: Please see the reply from Wishr above. I do believe this should be solved with update 4 and some settings so please contact support to get the approved solution
Post the case ID here for our reference and the outcome after the work with the engineers
Thanks
Mike

[squestier]

Hi Mike,

Almost 1 year old but this thread is still current for me.

I updated all my Veeam hosts with 9.5 4a and no reference point is ever listed or automatically deleted by Veeam.

Do you have some public statement for us regarding this lingering issue please ?

Kind regards,

Sébastien