Host-based backup of Microsoft Hyper-V VMs.
nkoji
Lurker
Posts: 1
Liked: never
Joined: May 19, 2018 6:58 am
Full Name: Naoya Kojima
Contact:

Guest OS Starting up is very slow on Hyper-V 2016

Post by nkoji »

Hi, Guys .

I'm in trouble that Hyper-V guest os starting up is very slow after I added guest os in "Edit Replication Job" window.
When I restart guest os restart , it takes about a hour to start up... :cry:
Please give me any information and advises to fix this situation.

My Configuration is followings

Veeam Backup & Replication 9.5 u3.

Host OS A : Windows Server 2016 (Veeam Backup & Replication 9.5 u3 components)
-> 2 Guest OS: Windows Server 2016 both
This is replication source.

Host OS B : Windows Server 2016 (Veeam Backup & Replication 9.5 u3)
This is replication target.

Best Regards !
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by PTide » 1 person likes this post

Hi and welcome to the community!

Please check the following:

1. Does the guest have any checkpoints except those which were created by Veeam?
2. Do you perform reboot during replication session?

If there are no other checkpoints and the reboot is performed outside of replication windows, then please contact our support team so they can investigate.

Thanks
squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier »

Hi guys,

Almost the same problem with many customers, nearly every host with native Hyper-V replica vms "waits" a very very long time before vms start up, after the host rebooted (so at least once a month with WindowsUpdates).

I just realize Hyper-V hosts without Veeam components are not affected.

Any idea or link ?

Kind regards,

Sebastien, France
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by Mike Resseler »

Hey Sebastien,

Since the OP didn't leave a support case ID for us to look, I suggest you create one to see what might be the issue. Please post the case ID here. Afterwards, please post the outcome from the investigation by our engineers.

Thanks
Mike
squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier »

Hi Mike,

Case ID is 03189568.

Hope we will find an explanation…

Kind regards,

Sebastien
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by Mike Resseler »

Indeed. Keep us informed after the investigation
hasoft
Novice
Posts: 8
Liked: 1 time
Joined: Sep 29, 2018 10:19 am
Full Name: Zdenek Vasku
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by hasoft »

Any progres on this issue. Wa are also affected. Problem is with live migration too (virtual guest is stopping-starting takes very long).

Thanks
Zdenek
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by Mike Resseler »

Hi Zdenek,

First: Welcome to the forums!
Second: The OP has opened the case but never responded back to us after that. So we have no more information. Can you create a support case and post the case ID here?

Many thanks
Mike
hasoft
Novice
Posts: 8
Liked: 1 time
Joined: Sep 29, 2018 10:19 am
Full Name: Zdenek Vasku
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by hasoft »

Hi Mike,

case id is 03218147.

Zdenek
squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier »

Hi guys,

To everybody facing this issue : I discovered something regarding the size of the vmcx files.

When you first build a vm with Hyper-V, the config file is around 60-70 KB. All my buggy vms have much bigger vmcx config files, around 60MB for the biggest one.

For my case, THESE huge vmcx files ARE the problem when rebooting the physical server, sometimes almost 30-60mn waiting for the vm to power on !!

Could you check every vmcx config file size on your servers ?

Please report, I may have a solution.

Kind regards,

Sebastien
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by PTide »

When you first build a vm with Hyper-V, the config file is around 60-70 KB. All my buggy vms have much bigger vmcx config files, around 60MB for the biggest one.
I wonder what occupies so much space...Did you check what was inside?

Thanks
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by Mike Resseler »

Sebastien,

VMCX are binary files so it is not possible to read inside anymore (which used to be possible in previous hyper-v). If they are that large, it means Hyper-V needs to read out all the information which is inside there (config/ devices/ ...). 60 MB just seems too big and probably means there is a lot of information in there which shouldn't be there. I would suggest to contact MSFT support with this to fix those VMCX files.
squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier »

Hi Mike,

Indeed, we can barely "read" some vmcx config info with PowerShell cmdlets like "compare-vm" but Nothing useful enough to pinpoint what's wrong.

I think I found a workaround : when you create a native Hyper-V replica, the replica vm is "clean", ie its vmcx file size is 60-70k only.

So, with this "yet empty" replica vm, I just remove the vhd and export the vm.

On the "master" Host, I delete the vm (it does not delete the vhd), rename the vm folder, copy the small exported vm (witout the vhd) from the "replica" host, import the vm again and reattach the full vhd and the new vm will then boot with no delay.

And just create the replication again if needed.

Quick and Dirty, but seems to work great.

I hope it will help some people.

Regards

Sebastien
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by Mike Resseler »

Sebastien,

That means the problem might be with Hyper-V replica. And that also means it probably will resurface. Did you contacted Microsoft around this?
squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier »

Hi Mike,

Not yet but I feel like it's also somewhere related to Veeam because of the snapshots Veeam creates for eack backup. All my customers use Veeam and I can't let critical replicated servers without the too precious Veeam protection, so it's not easy to diagnose.

I will try to reach Microsoft but I don't even know where to open a ticket for this kind of problem. Any advice ?

Regards,

Sebastien

PS : You're From Belgium, do you speak French ?
hasoft
Novice
Posts: 8
Liked: 1 time
Joined: Sep 29, 2018 10:19 am
Full Name: Zdenek Vasku
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by hasoft »

Hi all,

problem is with ReferencePoints inside VM. Veeam developers know about this problem and I think they fix this bug in future version. In brief: after succesful backup Snapshot is converted to ReferencePoint. ReferencePoints help incremental backups to calculate changes between incremental backups. Accidentaly these reference points are not deleted. When VM is stopped, reference points are stored to vmcx file (make it grow and grow). Every backup create one reference point. Replicas run more often than regular (for example one day) backup and problem is more visible. Replicas simple create more RestorePoints.

Till developers repair this problem, you can delete restore points manually using VMI. I can post code if you wish.

Zdenek
hasoft
Novice
Posts: 8
Liked: 1 time
Joined: Sep 29, 2018 10:19 am
Full Name: Zdenek Vasku
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by hasoft »

typo in last sentence:

Till developers repair this problem, you can delete referene points manually using VMI.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by Mike Resseler »

Hey Hasoft,

Is that a problem with Veeam replication? Because previous poster is talking about Hyper-V replication.

PS: @Sebastien: I am from the Flemish part but I do speak French ;-)
hasoft
Novice
Posts: 8
Liked: 1 time
Joined: Sep 29, 2018 10:19 am
Full Name: Zdenek Vasku
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by hasoft »

It's common problem with snapshots and reference points on W2016. Can you access support id 03218147? That case contain all infos about problem.
squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier »

Thanks Zdenek,

Almost 15.000 Reference Points Inside several config files !!!

Already started a good cleanup with WMI Method "DestroyReferencePoint". Vm start delay is now back to normal !!

Would be cool if Veeam could warn for these hidden Reference Points.

Kind regards,

Sebastien
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by nmdange »

I wouldn't mind seeing the code to delete the reference points manually just in case I run into this problem.
hasoft
Novice
Posts: 8
Liked: 1 time
Joined: Sep 29, 2018 10:19 am
Full Name: Zdenek Vasku
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by hasoft »

squestier wrote: Nov 12, 2018 4:26 pm Already started a good cleanup with WMI Method "DestroyReferencePoint". Vm start delay is now back to normal !!
it's a good practice to leave recent ReferencePoints to Veeam Backup can use them. I do daily backup and 30 min replication so I delete Reference Points older than week (for sure).
squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier » 2 people like this post

Main source code :
https://blogs.msdn.microsoft.com/taylor ... l-scripts/

List-reference-points.ps1
Display how many Reference Points exist for one vm :

Code: Select all

cls
$VmName = 'XXXXXXXX'

    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
      $virtualSystemRefPoint += ([WMI] $enum.Current.Dependent)
    }
    # Return the array of recovery points

$virtualSystemRefPoint.Count
squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier »

Second part

Delete-Reference-Points.ps1

Code: Select all


$VmName = 'XXXXXXXX'

filter ProcessWMIJob 
{ 
    param 
    ( 
        [WMI]$WmiClass = $null,
        [string]$MethodName = $null
    )
 $errorCode = 0
    $returnObject = $_
    if ($_.ReturnValue -eq 4096) 
    { 
        $Job = [WMI]$_.Job 
        $returnObject = $Job
        while ($Job.JobState -eq 4) 
        { 
            Write-Progress -Activity $Job.Caption -Status ($Job.JobStatus + " - " + $Job.PercentComplete + "%") -PercentComplete $Job.PercentComplete
            Start-Sleep -seconds 1
            $Job.PSBase.Get()
        } 
        if ($Job.JobState -ne 7) 
        { 
   if ($Job.ErrorDescription -ne "")
   {
             Write-Error $Job.ErrorDescription 
             Throw $Job.ErrorDescription 
   }
   else
   {
    $errorCode = $Job.ErrorCode
   }
        } 
        Write-Progress -Activity $Job.Caption -Status $Job.JobStatus -PercentComplete 100 -Completed:$true
        
    }
 elseif($_.ReturnValue -ne 0)
 {
  $errorCode = $_.ReturnValue
 }
 
 if ($errorCode -ne 0) 
    { 
        Write-Error "Hyper-V WMI Job Failed!" 
        if ($WmiClass -and $MethodName)
        {
            $psWmiClass = [WmiClass]("\\" + $WmiClass.__SERVER + "\" + $WmiClass.__NAMESPACE + ":" + $WmiClass.__CLASS)
            $psWmiClass.PSBase.Options.UseAmendedQualifiers = $TRUE
            $MethodQualifierValues = ($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["Values"]
            $indexOfError = [System.Array]::IndexOf(($psWmiClass.PSBase.Methods[$MethodName].Qualifiers)["ValueMap"].Value, [string]$errorCode)
            if (($indexOfError -ne "-1") -and $MethodQualifierValues)
            {
                Throw "ReturnCode: ", $errorCode, " ErrorMessage: '", $MethodQualifierValues.Value[$indexOfError], "' - when calling $MethodName"
            }
            else
            {
                Throw "ReturnCode: ", $errorCode, " ErrorMessage: 'MessageNotFound' - when calling $MethodName"
            }
        }
        else
        {
            Throw "ReturnCode: ", $errorCode, "When calling $MethodName - for rich error messages provide classpath and method name."
        }
    } 
 return $returnObject
}

    # Retrieve an instance of the virtual machine computer system that contains reference points
    $Msvm_ComputerSystem = Get-WmiObject -Namespace root\virtualization\v2 -Class Msvm_ComputerSystem -Filter "ElementName='$VmName'"
    # Retrieve all refrence associations of the virtual machine
    $allrefPoints = $Msvm_ComputerSystem.GetRelationships("Msvm_ReferencePointOfVirtualSystem")
    # Enumerate across all of the instances and add all recovery points to an array
    $virtualSystemRefPoint = @()
    $enum = $allrefPoints.GetEnumerator()
    $enum.Reset()
    while($enum.MoveNext())
    {
        
    $Msvm_VirtualSystemReferencePointService = Get-WmiObject -Namespace root\virtualization\v2 -Class Msvm_VirtualSystemReferencePointService
    # Removes the virtual machine reference, this method returns a job object.
    $job = $Msvm_VirtualSystemReferencePointService.DestroyReferencePoint([WMI] $enum.Current.Dependent)
    # Waits for the job to complete and processes any errors.
    ($job | ProcessWMIJob -WmiClass $Msvm_VirtualSystemReferencePointService -MethodName "DestroyReferencePoint") | Out-Null
         
    }
    # !!! Deletes all Reference Points !!!
renevw
Lurker
Posts: 1
Liked: never
Joined: Nov 22, 2018 7:05 am
Full Name: René van Wingerden
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by renevw »

Hello All,

Great updates, i have checked the scripts and also found out that the VMs starting verry slow are having around 12000 reference points.
I am running an combination of continuously and hourly replication jobs.

I was wondering if any of you is also an memory "leaking" problem on the HV host. I think it can be related. I experienced the slow starting problem because i had to reboot the HV host from time to time to free up memory. I found that the vmm service is using up to 12 GB of RAM. The Host it self has 96 GB of fysical RAM, i have about 72 GB of ram (static) allocated to the VMs. The OS uses some + the 12 GB for vmm i have about 5 GB of free memory. The used memory grows slowly over time. At this point (5 GB memory free) the back-up jobs start having problems because of resource.

The (slave) HV Host (identical) which holds te replica's and 2 active vms for Veeam and AD does run stable for months without any memory problems. No VMs are replicated from this HOST.

03218147
Gerdes
Lurker
Posts: 1
Liked: never
Joined: Jan 28, 2019 8:44 am
Full Name: Stephan Schuler
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by Gerdes »

Hello,

I got the same issue here. I'm running 5 VMs. Backup and Replication ist only created with Veeam.

One VM has actually over 26.000 Reference Points!!

Whats the recommendation to solve this issue? Is there any update available? Or do I need to delete alle the Reference Point by the script posted by squestier?

Regards,
Stephan
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by wishr »

Hi Stephan,

Sure, as of now we recommend you to reach out to our support team directly to get an approved solution to reference points accumulation.

Thanks
tejko30
Novice
Posts: 6
Liked: 1 time
Joined: Feb 18, 2019 9:52 am
Full Name: Matej
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by tejko30 »

Hi,

Since we are experiencing this issue as well... Do we have a publicly available solution to deal with this? What is Veeam's approved solution?

Thanks!
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by Mike Resseler »

Hey Matej,

First: Welcome to the forums
Second: Please see the reply from Wishr above. I do believe this should be solved with update 4 and some settings so please contact support to get the approved solution
Post the case ID here for our reference and the outcome after the work with the engineers
Thanks
Mike
squestier
Influencer
Posts: 13
Liked: 5 times
Joined: Apr 12, 2012 9:35 am
Full Name: Sebastien QUESTIER
Contact:

Re: Guest OS Starting up is very slow on Hyper-V 2016

Post by squestier »

Hi Mike,

Almost 1 year old but this thread is still current for me.

I updated all my Veeam hosts with 9.5 4a and no reference point is ever listed or automatically deleted by Veeam.

Do you have some public statement for us regarding this lingering issue please ?

Kind regards,

Sébastien
Post Reply

Who is online

Users browsing this forum: No registered users and 21 guests