Alerts: Hyper-V Integration Service(s)

Unleash the power of System Center for vSphere and Hyper-V | Veeam Task Manager for Hyper-V

Alerts: Hyper-V Integration Service(s)

Veeam Logoby frowiers » Fri Sep 02, 2016 12:40 pm

Hi

We have installed "Veeam MP for Hyper-V" in our environment, monitoring about 800 virtual machines.

On many of our Citrix VDA's (atm. number of alerts: 137) we have the following Alert in SCOM:
The following Hyper-V Integration Service(s) are not running or not responding in the guest OS:
Volume Shadow Copy Requestor Service


I have verified the service is running, and have even reset the alert in SCOM, but after 5 minutes (the threshold is 300 seconds) the Alert is back.

  • Why is this alert showing when service is running
  • Why is this alert only affecting (mostly) Citrix VDA's


Any tip how to solve this would be highly appreciated :)

Best Regards Frode @ Netscenario
Frode @ Netscenario
frowiers
Service Provider
 
Posts: 13
Liked: never
Joined: Thu Feb 11, 2016 6:13 pm
Full Name: Frode Åbø Wiersen

Re: Alerts: Hyper-V Integration Service(s)

Veeam Logoby sergey.g » Fri Sep 02, 2016 2:39 pm

Hi Frode,

Is it possible to run the followign powershell command on the Hyper-V servers which are runnign VMs affected by this issue?

Code: Select all
gwmi -namespace "root\virtualization\v2" -class "Msvm_VssComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions


We suspect that some Citrix integration with in-guest VSS services could cause it to switch into some non-healthy or semi-healthy state and output of the command above could help us detect what is going on.

We would highly appreciate your help with troubleshooting the issue.

Thanks.
sergey.g
Veeam Software
 
Posts: 453
Liked: 75 times
Joined: Wed May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko

Re: Alerts: Hyper-V Integration Service(s)

Veeam Logoby -=MACROS=- » Fri Sep 02, 2016 7:20 pm

Hi Frode,

On the settings of your vServer/vClient did you've something like Backup Feature? On the vServer on Hyper-V you can set special this options onto vServer options. Maybe its you goal?

Regards

Rodi


Gesendet von iPhone mit Tapatalk
-=MACROS=-
Influencer
 
Posts: 15
Liked: 1 time
Joined: Mon Jun 27, 2016 12:18 pm
Location: Germany, Nuremberg
Full Name: Rodion Fedin

Re: Alerts: Hyper-V Integration Service(s)

Veeam Logoby -=MACROS=- » Fri Sep 02, 2016 7:26 pm

-=MACROS=- wrote:Hi Frode,

On the settings of your vServer/vClient did you've something like Backup Feature? On the vServer on Hyper-V you can set special this options onto vServer options. Maybe its you goal?

http://www.askme4tech.com/sites/default ... ttings.png

Regards

Rodi


Gesendet von iPhone mit Tapatalk



Gesendet von iPhone mit Tapatalk
-=MACROS=-
Influencer
 
Posts: 15
Liked: 1 time
Joined: Mon Jun 27, 2016 12:18 pm
Location: Germany, Nuremberg
Full Name: Rodion Fedin

Re: Alerts: Hyper-V Integration Service(s)

Veeam Logoby frowiers » Mon Sep 05, 2016 12:00 am

sergey.g wrote:Hi Frode,

Is it possible to run the followign powershell command on the Hyper-V servers which are runnign VMs affected by this issue?

Code: Select all
gwmi -namespace "root\virtualization\v2" -class "Msvm_VssComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions


We suspect that some Citrix integration with in-guest VSS services could cause it to switch into some non-healthy or semi-healthy state and output of the command above could help us detect what is going on.

We would highly appreciate your help with troubleshooting the issue.

Thanks.


This is what is written out when I run the cmdlet you suggested:

Image
Frode @ Netscenario
frowiers
Service Provider
 
Posts: 13
Liked: never
Joined: Thu Feb 11, 2016 6:13 pm
Full Name: Frode Åbø Wiersen

Re: Alerts: Hyper-V Integration Service(s)

Veeam Logoby sergey.g » Mon Sep 05, 2016 4:03 pm

Hi Frode,

Thank you very much for the provided output. We can see that some VMs has VSS Integration Service in "No Contact" state, which according to Microsoft means "The guest service is not installed or has not yet been contacted."

Our monitor is tracking services which have been enabled in the VM properties, so for us it means that the service needs to be monitored and that's why you see an issue when state is not "OK" for some VMs.

This service is necessary for Backup solutions which are using VSS inside guest OSes, I assume that backup with enabled VSS integration would fail for these VMs with "No Contact" state, so clearly for us it's a critical error.

If you don't need VSS integration for some VMs, you can either disable it on the Hyper-V side in the VM properties (in this case a Monitor will not trigger for VSS service issues) or you can disable the entire monitor for a group of VDA VMs for example (although in this case other Integration services will not be monitored either)

Let me know if you have any other questions or suggestions for Hyper-V Integration Services Monitor architecture.
Thanks.
sergey.g
Veeam Software
 
Posts: 453
Liked: 75 times
Joined: Wed May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko

Re: Alerts: Hyper-V Integration Service(s)

Veeam Logoby frowiers » Tue Sep 06, 2016 3:23 pm

-=MACROS=- wrote:Hi Frode,

On the settings of your vServer/vClient did you've something like Backup Feature? On the vServer on Hyper-V you can set special this options onto vServer options. Maybe its you goal?

Regards

Rodi


Gesendet von iPhone mit Tapatalk


sergey.g wrote:Hi Frode,

Thank you very much for the provided output. We can see that some VMs has VSS Integration Service in "No Contact" state, which according to Microsoft means "The guest service is not installed or has not yet been contacted."

Our monitor is tracking services which have been enabled in the VM properties, so for us it means that the service needs to be monitored and that's why you see an issue when state is not "OK" for some VMs.

This service is necessary for Backup solutions which are using VSS inside guest OSes, I assume that backup with enabled VSS integration would fail for these VMs with "No Contact" state, so clearly for us it's a critical error.

If you don't need VSS integration for some VMs, you can either disable it on the Hyper-V side in the VM properties (in this case a Monitor will not trigger for VSS service issues) or you can disable the entire monitor for a group of VDA VMs for example (although in this case other Integration services will not be monitored either)

Let me know if you have any other questions or suggestions for Hyper-V Integration Services Monitor architecture.
Thanks.



Hi

Thanks for the replies! A combination of your replies actually gave me the answer I was looking for. None of these Citrix VDA's are being backed up using application aware backups (VSS), so the resolution were to disable the Backup Integration Component on all Citrix VDA's :)

I used the following script to do so:
Code: Select all
$VMMFolder = "CTX"

$VMMServer = "mgmt50.mgmt.local"

$vms = Get-SCVirtualMachine -VMMServer $VMMServer | where {$_.HostGroupPath -match $VMMFolder}
foreach ($vm in $vms){
        write-host $vm.Name
        Set-SCVirtualMachine $vm -EnableBackup $false
}


After waiting the threshold of the monitor in SCOM (300 seconds / 5 minutes) all my alerts related to the monitor "Veeam HyperV: VM Integration Services Status" went away :)
Frode @ Netscenario
frowiers
Service Provider
 
Posts: 13
Liked: never
Joined: Thu Feb 11, 2016 6:13 pm
Full Name: Frode Åbø Wiersen

Re: Alerts: Hyper-V Integration Service(s)

Veeam Logoby frowiers » Wed Sep 14, 2016 10:32 am

Hi again,
I am afraid this only solved parts of the errors.

I still have SCOM complaining about integration services not running on several virtual machines. This can be several of the services offered (vmicvss, vmickvpexchange, vmicshutdown).

Is there any way I can verify / test the integration service communication, to see where the problem could be?
Frode @ Netscenario
frowiers
Service Provider
 
Posts: 13
Liked: never
Joined: Thu Feb 11, 2016 6:13 pm
Full Name: Frode Åbø Wiersen

Re: Alerts: Hyper-V Integration Service(s)

Veeam Logoby sergey.g » Fri Sep 16, 2016 4:23 pm

Hi Frode,

Sorry for a delay, could you try the following commands on thy Hyper-V server?

Code: Select all
gwmi -namespace "root\virtualization\v2" -class "Msvm_KvpExchangeComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions
gwmi -namespace "root\virtualization\v2" -class "Msvm_ShutdownComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions


Also, do you see these errors for same VMs as with VSS/Backup issue?

Thanks
sergey.g
Veeam Software
 
Posts: 453
Liked: 75 times
Joined: Wed May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko

Re: Alerts: Hyper-V Integration Service(s)

Veeam Logoby frowiers » Wed Sep 21, 2016 7:19 am

sergey.g wrote:Hi Frode,

Sorry for a delay, could you try the following commands on thy Hyper-V server?

Code: Select all
gwmi -namespace "root\virtualization\v2" -class "Msvm_KvpExchangeComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions
gwmi -namespace "root\virtualization\v2" -class "Msvm_ShutdownComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions


Also, do you see these errors for same VMs as with VSS/Backup issue?

Thanks


Hi

Not all of these VMs are included in Application Aware backups, so it is hard to say; the backup jobs are filtered on Tags in SCVMM, so not sure how I can exclude monitoring for VMs with a curtain TAG inside SCOM.

I have run the cmdlets on one Hyper-V Host hosting a VM where SCOM reports a failure on a integration service.

This is the output from the cmdlet:

Code: Select all
PS C:\Users\fw> gwmi -namespace "root\virtualization\v2" -class "Msvm_KvpExchangeComponent" | select SystemName, Status
OperationalStatus,StatusDescriptions

SystemName                    Status                        OperationalStatus             StatusDescriptions
----------                    ------                        -----------------             ------------------
0044FBBC-58C7-419D-B5AD-A9...                               {2, 32775}                    {OK, The protocol version ...
03F28958-AFFE-44D0-BC58-AB...                               {2}                           {OK}
05771D72-5A12-4C51-9F92-41...                               {2}                           {OK}
087F749F-D381-45A3-A22A-D8...                               {2}                           {OK}
0AF488FF-3D24-474F-ADDF-2B...                               {2}                           {OK}
0AF5B7E3-623E-4040-9FED-28...                               {2, 32775}                    {OK, The protocol version ...
0FB659A8-3F6A-4F7B-9331-FE...                               {2, 32775}                    {OK, The protocol version ...
104DB6C7-35DD-4B49-B374-C4...                               {2}                           {OK}
15503CAC-C13D-41B2-9805-67...                               {2}                           {OK}
19732D3E-432F-4BDF-B3D7-B3...                               {2}                           {OK}
1995B55A-3FE4-4B10-BF4E-D1...                               {2}                           {OK}
19DF4191-2ED4-4A2C-9D27-C7...                               {2, 32775}                    {OK, The protocol version ...
22494038-87A6-4DA2-9DC4-6F...                               {2}                           {OK}
23B4F67B-939A-4941-A359-C7...                               {2}                           {OK}
36F59B64-28F0-4B83-8AF5-E3...                               {2, 32775}                    {OK, The protocol version ...
3FA35398-37F8-456B-AFD8-7F...                               {2}                           {OK}
40836797-48A2-4936-8801-FB...                               {2}                           {OK}
4108287C-04E4-4C9E-80CB-18...                               {12}                          {No Contact}
53DD4ABB-6A94-4205-B07C-98...                               {2, 32775}                    {OK, The protocol version ...
54747956-E170-443B-B3B5-E7...                               {2, 32775}                    {OK, The protocol version ...
5571E37D-4D1E-4E93-8BD0-48...                               {2}                           {OK}
579A1E6A-1799-46C3-9A87-9A...                               {2}                           {OK}
5A2BB795-8786-4FEC-A272-5E...                               {2}                           {OK}
6035A510-FE52-4B95-AE2F-9D...                               {2}                           {OK}
61074F82-32CA-4647-B743-5B...                               {2, 32775}                    {OK, The protocol version ...
615E438F-1028-46B8-937A-57...                               {2, 32775}                    {OK, The protocol version ...
69AC58C6-CE3F-40F7-9B6F-20...                               {2}                           {OK}
6AC6D22C-2E5F-47C7-A088-71...                               {2}                           {OK}
6AF770E1-B831-4E25-B96F-60...                               {2, 32775}                    {OK, The protocol version ...
770E1DEC-3F16-46FF-8443-BC...                               {2}                           {OK}
77879E58-733D-44DC-BEDF-41...                               {2, 32775}                    {OK, The protocol version ...
810DF37B-6189-4E2F-84ED-4C...                               {2, 32775}                    {OK, The protocol version ...
96B6C934-D7F1-48C4-BC5E-9D...                               {2}                           {OK}
976632CB-0281-494B-9287-C4...                               {2}                           {OK}
9C6A54A5-8B9C-4A61-A5D8-E8...                               {2}                           {OK}
9D099A04-A37A-4261-89AC-D6...                               {2, 32775}                    {OK, The protocol version ...
9EC7AECB-C0A0-4ADD-BE3B-F7...                               {2, 32775}                    {OK, The protocol version ...
9ECA410A-4BBF-4AC9-80CC-6D...                               {2}                           {OK}
9F4AE2AA-E2DA-4785-B57B-63...                               {2}                           {OK}
A25045C9-FEF3-41D7-8A66-B5...                               {2}                           {OK}
A35E384A-BEAB-4C7D-83A8-C9...                               {2}                           {OK}
A3B5B9B6-0980-41AE-AEA8-A6...                               {2}                           {OK}
A7162BD3-440E-4D52-9574-CB...                               {2, 32775}                    {OK, The protocol version ...
AA0B5C8E-DA30-4389-9CC5-09...                               {2}                           {OK}
B63B2E57-1250-4CA6-BA19-B5...                               {2, 32775}                    {OK, The protocol version ...
B7063626-AD99-462A-A6D7-AA...                               {2}                           {OK}
BD420633-9F4A-43E7-9B88-73...                               {2}                           {OK}
BD863CC8-005C-41A4-A802-EB...                               {2, 32775}                    {OK, The protocol version ...
C43DE05E-407C-40A1-B926-57...                               {2}                           {OK}
CA14F43D-4305-4131-A017-5F...                               {2}                           {OK}
CE250E40-4F4A-4206-89BC-E0...                               {2}                           {OK}
CEA71ED2-45A7-4C40-BA38-DB...                               {12}                          {No Contact}
CEDAF65F-B932-47A8-9DE5-C1...                               {2}                           {OK}
CEEF1039-75BE-4282-94FD-55...                               {2, 32775}                    {OK, The protocol version ...
D10D2D97-5F36-4951-B748-8D...                               {2}                           {OK}
D21FEC1A-EF5F-4312-9D17-FF...                               {2}                           {OK}
D4709582-1295-45E7-BCFE-1D...                               {2}                           {OK}
D7D02D8C-4E22-4C40-87AE-7A...                               {2}                           {OK}
D7FD9B0F-122C-4DAC-A039-70...                               {2}                           {OK}
D9A03BB5-AC16-4C12-8A47-23...                               {2}                           {OK}
D9E43449-2331-41C5-826E-A7...                               {2}                           {OK}
DDDE6914-97F1-4844-8876-68...                               {2}                           {OK}
E4D48857-A1BC-4902-B0D8-E4...                               {2, 32775}                    {OK, The protocol version ...
E502801F-B768-4DEA-BBA7-A8...                               {2, 32775}                    {OK, The protocol version ...
EB3EF232-5627-49D2-BC28-42...                               {2}                           {OK}
EB8297E2-70C8-4585-90BF-CF...                               {2}                           {OK}
F3E95B86-AE38-48BF-9632-52...                               {2, 32775}                    {OK, The protocol version ...
F65E621F-387B-416A-A5FC-E0...                               {2}                           {OK}
F9D8C0F0-9D7C-4980-976D-9D...                               {2}                           {OK}


PS C:\Users\fw> gwmi -namespace "root\virtualization\v2" -class "Msvm_ShutdownComponent" | select SystemName, Status,Op
rationalStatus,StatusDescriptions

SystemName                    Status                        OperationalStatus             StatusDescriptions
----------                    ------                        -----------------             ------------------
0044FBBC-58C7-419D-B5AD-A9...                               {2}                           {OK}
03F28958-AFFE-44D0-BC58-AB...                               {2}                           {OK}
05771D72-5A12-4C51-9F92-41...                               {2}                           {OK}
087F749F-D381-45A3-A22A-D8...                               {2}                           {OK}
0AF488FF-3D24-474F-ADDF-2B...                               {2}                           {OK}
0AF5B7E3-623E-4040-9FED-28...                               {2}                           {OK}
0FB659A8-3F6A-4F7B-9331-FE...                               {2}                           {OK}
104DB6C7-35DD-4B49-B374-C4...                               {2}                           {OK}
15503CAC-C13D-41B2-9805-67...                               {2}                           {OK}
19732D3E-432F-4BDF-B3D7-B3...                               {2}                           {OK}
1995B55A-3FE4-4B10-BF4E-D1...                               {2}                           {OK}
19DF4191-2ED4-4A2C-9D27-C7...                               {2}                           {OK}
22494038-87A6-4DA2-9DC4-6F...                               {2}                           {OK}
23B4F67B-939A-4941-A359-C7...                               {2}                           {OK}
36F59B64-28F0-4B83-8AF5-E3...                               {2}                           {OK}
3FA35398-37F8-456B-AFD8-7F...                               {2}                           {OK}
40836797-48A2-4936-8801-FB...                               {2}                           {OK}
4108287C-04E4-4C9E-80CB-18...                               {2}                           {OK}
53DD4ABB-6A94-4205-B07C-98...                               {2}                           {OK}
54747956-E170-443B-B3B5-E7...                               {2}                           {OK}
5571E37D-4D1E-4E93-8BD0-48...                               {2}                           {OK}
579A1E6A-1799-46C3-9A87-9A...                               {2}                           {OK}
5A2BB795-8786-4FEC-A272-5E...                               {2}                           {OK}
6035A510-FE52-4B95-AE2F-9D...                               {2}                           {OK}
61074F82-32CA-4647-B743-5B...                               {2}                           {OK}
615E438F-1028-46B8-937A-57...                               {2}                           {OK}
69AC58C6-CE3F-40F7-9B6F-20...                               {2}                           {OK}
6AC6D22C-2E5F-47C7-A088-71...                               {2}                           {OK}
6AF770E1-B831-4E25-B96F-60...                               {2}                           {OK}
770E1DEC-3F16-46FF-8443-BC...                               {2}                           {OK}
77879E58-733D-44DC-BEDF-41...                               {2}                           {OK}
810DF37B-6189-4E2F-84ED-4C...                               {2}                           {OK}
96B6C934-D7F1-48C4-BC5E-9D...                               {2}                           {OK}
976632CB-0281-494B-9287-C4...                               {2}                           {OK}
9C6A54A5-8B9C-4A61-A5D8-E8...                               {2}                           {OK}
9D099A04-A37A-4261-89AC-D6...                               {2}                           {OK}
9EC7AECB-C0A0-4ADD-BE3B-F7...                               {2}                           {OK}
9ECA410A-4BBF-4AC9-80CC-6D...                               {2}                           {OK}
9F4AE2AA-E2DA-4785-B57B-63...                               {2}                           {OK}
A25045C9-FEF3-41D7-8A66-B5...                               {2}                           {OK}
A35E384A-BEAB-4C7D-83A8-C9...                               {2}                           {OK}
A3B5B9B6-0980-41AE-AEA8-A6...                               {2}                           {OK}
A7162BD3-440E-4D52-9574-CB...                               {2}                           {OK}
AA0B5C8E-DA30-4389-9CC5-09...                               {2}                           {OK}
B63B2E57-1250-4CA6-BA19-B5...                               {2}                           {OK}
B7063626-AD99-462A-A6D7-AA...                               {2}                           {OK}
BD420633-9F4A-43E7-9B88-73...                               {2}                           {OK}
BD863CC8-005C-41A4-A802-EB...                               {2}                           {OK}
C43DE05E-407C-40A1-B926-57...                               {2}                           {OK}
CA14F43D-4305-4131-A017-5F...                               {2}                           {OK}
CE250E40-4F4A-4206-89BC-E0...                               {2}                           {OK}
CEA71ED2-45A7-4C40-BA38-DB...                               {2}                           {OK}
CEDAF65F-B932-47A8-9DE5-C1...                               {2}                           {OK}
CEEF1039-75BE-4282-94FD-55...                               {2}                           {OK}
D10D2D97-5F36-4951-B748-8D...                               {2}                           {OK}
D21FEC1A-EF5F-4312-9D17-FF...                               {2}                           {OK}
D4709582-1295-45E7-BCFE-1D...                               {2}                           {OK}
D7D02D8C-4E22-4C40-87AE-7A...                               {2}                           {OK}
D7FD9B0F-122C-4DAC-A039-70...                               {2}                           {OK}
D9A03BB5-AC16-4C12-8A47-23...                               {2}                           {OK}
D9E43449-2331-41C5-826E-A7...                               {2}                           {OK}
DDDE6914-97F1-4844-8876-68...                               {2}                           {OK}
E4D48857-A1BC-4902-B0D8-E4...                               {2}                           {OK}
E502801F-B768-4DEA-BBA7-A8...                               {2}                           {OK}
EB3EF232-5627-49D2-BC28-42...                               {2}                           {OK}
EB8297E2-70C8-4585-90BF-CF...                               {2}                           {OK}
F3E95B86-AE38-48BF-9632-52...                               {2}                           {OK}
F65E621F-387B-416A-A5FC-E0...                               {2}                           {OK}
F9D8C0F0-9D7C-4980-976D-9D...                               {2}                           {OK}


Sorry for the late reply, been out of office some days...
Frode @ Netscenario
frowiers
Service Provider
 
Posts: 13
Liked: never
Joined: Thu Feb 11, 2016 6:13 pm
Full Name: Frode Åbø Wiersen

Re: Alerts: Hyper-V Integration Service(s)

Veeam Logoby sergey.g » Wed Sep 21, 2016 11:22 am

Hi Frode,

So, from this server only two VMs should be alerting, the ones that have 12 - No Contact

Veeam MP takes the first operational status into account, so version mismatch is not considered a service failure. You can get names for those VMs using the following command:

Code: Select all
gwmi -namespace "root\virtualization\v2" -class "Msvm_ComputerSystem" | ? {$_.Name -like "4108287C-04E4-4C9E-80CB-18*" -or $_.Name -like "CEA71ED2-45A7-4C40-BA38-DB*"} | select elementname


Let me know if you have any other concerns
Thanks.
sergey.g
Veeam Software
 
Posts: 453
Liked: 75 times
Joined: Wed May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko


Return to Veeam Management Pack for Microsoft System Center



Who is online

Users browsing this forum: No registered users and 3 guests