Alerts: Hyper-V Integration Service(s)

[frowiers]

Hi

We have installed "Veeam MP for Hyper-V" in our environment, monitoring about 800 virtual machines.

On many of our Citrix VDA's (atm. number of alerts: 137) we have the following Alert in SCOM:

The following Hyper-V Integration Service(s) are not running or not responding in the guest OS:
Volume Shadow Copy Requestor Service

I have verified the service is running, and have even reset the alert in SCOM, but after 5 minutes (the threshold is 300 seconds) the Alert is back.

• Why is this alert showing when service is running
• Why is this alert only affecting (mostly) Citrix VDA's

Any tip how to solve this would be highly appreciated

Best Regards Frode @ Netscenario

[sergey.g]

Hi Frode,

Is it possible to run the followign powershell command on the Hyper-V servers which are runnign VMs affected by this issue?

Code: Select all

gwmi -namespace "root\virtualization\v2" -class "Msvm_VssComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions

We suspect that some Citrix integration with in-guest VSS services could cause it to switch into some non-healthy or semi-healthy state and output of the command above could help us detect what is going on.

We would highly appreciate your help with troubleshooting the issue.

Thanks.

[-=MACROS=-]

Hi Frode,

On the settings of your vServer/vClient did you've something like Backup Feature? On the vServer on Hyper-V you can set special this options onto vServer options. Maybe its you goal?

Regards

Rodi


Gesendet von iPhone mit Tapatalk

[-=MACROS=-]

Hi Frode,

On the settings of your vServer/vClient did you've something like Backup Feature? On the vServer on Hyper-V you can set special this options onto vServer options. Maybe its you goal?

http://www.askme4tech.com/sites/default ... ttings.png

Regards

Rodi


Gesendet von iPhone mit Tapatalk

Gesendet von iPhone mit Tapatalk

[frowiers]

Hi Frode,

Is it possible to run the followign powershell command on the Hyper-V servers which are runnign VMs affected by this issue?

Code: Select all

gwmi -namespace "root\virtualization\v2" -class "Msvm_VssComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions

We suspect that some Citrix integration with in-guest VSS services could cause it to switch into some non-healthy or semi-healthy state and output of the command above could help us detect what is going on.

We would highly appreciate your help with troubleshooting the issue.

Thanks.

This is what is written out when I run the cmdlet you suggested:

[sergey.g]

Hi Frode,

Thank you very much for the provided output. We can see that some VMs has VSS Integration Service in "No Contact" state, which according to Microsoft means "The guest service is not installed or has not yet been contacted."

Our monitor is tracking services which have been enabled in the VM properties, so for us it means that the service needs to be monitored and that's why you see an issue when state is not "OK" for some VMs.

This service is necessary for Backup solutions which are using VSS inside guest OSes, I assume that backup with enabled VSS integration would fail for these VMs with "No Contact" state, so clearly for us it's a critical error.

If you don't need VSS integration for some VMs, you can either disable it on the Hyper-V side in the VM properties (in this case a Monitor will not trigger for VSS service issues) or you can disable the entire monitor for a group of VDA VMs for example (although in this case other Integration services will not be monitored either)

Let me know if you have any other questions or suggestions for Hyper-V Integration Services Monitor architecture.
Thanks.

[frowiers]

Hi Frode,

On the settings of your vServer/vClient did you've something like Backup Feature? On the vServer on Hyper-V you can set special this options onto vServer options. Maybe its you goal?

Regards

Rodi


Gesendet von iPhone mit Tapatalk

Hi Frode,

Thank you very much for the provided output. We can see that some VMs has VSS Integration Service in "No Contact" state, which according to Microsoft means "The guest service is not installed or has not yet been contacted."

Our monitor is tracking services which have been enabled in the VM properties, so for us it means that the service needs to be monitored and that's why you see an issue when state is not "OK" for some VMs.

This service is necessary for Backup solutions which are using VSS inside guest OSes, I assume that backup with enabled VSS integration would fail for these VMs with "No Contact" state, so clearly for us it's a critical error.

If you don't need VSS integration for some VMs, you can either disable it on the Hyper-V side in the VM properties (in this case a Monitor will not trigger for VSS service issues) or you can disable the entire monitor for a group of VDA VMs for example (although in this case other Integration services will not be monitored either)

Let me know if you have any other questions or suggestions for Hyper-V Integration Services Monitor architecture.
Thanks.

Hi

Thanks for the replies! A combination of your replies actually gave me the answer I was looking for. None of these Citrix VDA's are being backed up using application aware backups (VSS), so the resolution were to disable the Backup Integration Component on all Citrix VDA's

I used the following script to do so:

Code: Select all

$VMMFolder = "CTX"

$VMMServer = "mgmt50.mgmt.local"

$vms = Get-SCVirtualMachine -VMMServer $VMMServer | where {$_.HostGroupPath -match $VMMFolder}
foreach ($vm in $vms){
        write-host $vm.Name
        Set-SCVirtualMachine $vm -EnableBackup $false
}

After waiting the threshold of the monitor in SCOM (300 seconds / 5 minutes) all my alerts related to the monitor "Veeam HyperV: VM Integration Services Status" went away

[frowiers]

Hi again,
I am afraid this only solved parts of the errors.

I still have SCOM complaining about integration services not running on several virtual machines. This can be several of the services offered (vmicvss, vmickvpexchange, vmicshutdown).

Is there any way I can verify / test the integration service communication, to see where the problem could be?

[sergey.g]

Hi Frode,

Sorry for a delay, could you try the following commands on thy Hyper-V server?

Code: Select all

gwmi -namespace "root\virtualization\v2" -class "Msvm_KvpExchangeComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions
gwmi -namespace "root\virtualization\v2" -class "Msvm_ShutdownComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions

Also, do you see these errors for same VMs as with VSS/Backup issue?

Thanks

[frowiers]

Hi Frode,

Sorry for a delay, could you try the following commands on thy Hyper-V server?

Code: Select all

gwmi -namespace "root\virtualization\v2" -class "Msvm_KvpExchangeComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions
gwmi -namespace "root\virtualization\v2" -class "Msvm_ShutdownComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions

Also, do you see these errors for same VMs as with VSS/Backup issue?

Thanks

Hi

Not all of these VMs are included in Application Aware backups, so it is hard to say; the backup jobs are filtered on Tags in SCVMM, so not sure how I can exclude monitoring for VMs with a curtain TAG inside SCOM.

I have run the cmdlets on one Hyper-V Host hosting a VM where SCOM reports a failure on a integration service.

This is the output from the cmdlet:

Code: Select all

PS C:\Users\fw> gwmi -namespace "root\virtualization\v2" -class "Msvm_KvpExchangeComponent" | select SystemName, Status
OperationalStatus,StatusDescriptions

SystemName                    Status                        OperationalStatus             StatusDescriptions
----------                    ------                        -----------------             ------------------
0044FBBC-58C7-419D-B5AD-A9...                               {2, 32775}                    {OK, The protocol version ...
03F28958-AFFE-44D0-BC58-AB...                               {2}                           {OK}
05771D72-5A12-4C51-9F92-41...                               {2}                           {OK}
087F749F-D381-45A3-A22A-D8...                               {2}                           {OK}
0AF488FF-3D24-474F-ADDF-2B...                               {2}                           {OK}
0AF5B7E3-623E-4040-9FED-28...                               {2, 32775}                    {OK, The protocol version ...
0FB659A8-3F6A-4F7B-9331-FE...                               {2, 32775}                    {OK, The protocol version ...
104DB6C7-35DD-4B49-B374-C4...                               {2}                           {OK}
15503CAC-C13D-41B2-9805-67...                               {2}                           {OK}
19732D3E-432F-4BDF-B3D7-B3...                               {2}                           {OK}
1995B55A-3FE4-4B10-BF4E-D1...                               {2}                           {OK}
19DF4191-2ED4-4A2C-9D27-C7...                               {2, 32775}                    {OK, The protocol version ...
22494038-87A6-4DA2-9DC4-6F...                               {2}                           {OK}
23B4F67B-939A-4941-A359-C7...                               {2}                           {OK}
36F59B64-28F0-4B83-8AF5-E3...                               {2, 32775}                    {OK, The protocol version ...
3FA35398-37F8-456B-AFD8-7F...                               {2}                           {OK}
40836797-48A2-4936-8801-FB...                               {2}                           {OK}
4108287C-04E4-4C9E-80CB-18...                               {12}                          {No Contact}
53DD4ABB-6A94-4205-B07C-98...                               {2, 32775}                    {OK, The protocol version ...
54747956-E170-443B-B3B5-E7...                               {2, 32775}                    {OK, The protocol version ...
5571E37D-4D1E-4E93-8BD0-48...                               {2}                           {OK}
579A1E6A-1799-46C3-9A87-9A...                               {2}                           {OK}
5A2BB795-8786-4FEC-A272-5E...                               {2}                           {OK}
6035A510-FE52-4B95-AE2F-9D...                               {2}                           {OK}
61074F82-32CA-4647-B743-5B...                               {2, 32775}                    {OK, The protocol version ...
615E438F-1028-46B8-937A-57...                               {2, 32775}                    {OK, The protocol version ...
69AC58C6-CE3F-40F7-9B6F-20...                               {2}                           {OK}
6AC6D22C-2E5F-47C7-A088-71...                               {2}                           {OK}
6AF770E1-B831-4E25-B96F-60...                               {2, 32775}                    {OK, The protocol version ...
770E1DEC-3F16-46FF-8443-BC...                               {2}                           {OK}
77879E58-733D-44DC-BEDF-41...                               {2, 32775}                    {OK, The protocol version ...
810DF37B-6189-4E2F-84ED-4C...                               {2, 32775}                    {OK, The protocol version ...
96B6C934-D7F1-48C4-BC5E-9D...                               {2}                           {OK}
976632CB-0281-494B-9287-C4...                               {2}                           {OK}
9C6A54A5-8B9C-4A61-A5D8-E8...                               {2}                           {OK}
9D099A04-A37A-4261-89AC-D6...                               {2, 32775}                    {OK, The protocol version ...
9EC7AECB-C0A0-4ADD-BE3B-F7...                               {2, 32775}                    {OK, The protocol version ...
9ECA410A-4BBF-4AC9-80CC-6D...                               {2}                           {OK}
9F4AE2AA-E2DA-4785-B57B-63...                               {2}                           {OK}
A25045C9-FEF3-41D7-8A66-B5...                               {2}                           {OK}
A35E384A-BEAB-4C7D-83A8-C9...                               {2}                           {OK}
A3B5B9B6-0980-41AE-AEA8-A6...                               {2}                           {OK}
A7162BD3-440E-4D52-9574-CB...                               {2, 32775}                    {OK, The protocol version ...
AA0B5C8E-DA30-4389-9CC5-09...                               {2}                           {OK}
B63B2E57-1250-4CA6-BA19-B5...                               {2, 32775}                    {OK, The protocol version ...
B7063626-AD99-462A-A6D7-AA...                               {2}                           {OK}
BD420633-9F4A-43E7-9B88-73...                               {2}                           {OK}
BD863CC8-005C-41A4-A802-EB...                               {2, 32775}                    {OK, The protocol version ...
C43DE05E-407C-40A1-B926-57...                               {2}                           {OK}
CA14F43D-4305-4131-A017-5F...                               {2}                           {OK}
CE250E40-4F4A-4206-89BC-E0...                               {2}                           {OK}
CEA71ED2-45A7-4C40-BA38-DB...                               {12}                          {No Contact}
CEDAF65F-B932-47A8-9DE5-C1...                               {2}                           {OK}
CEEF1039-75BE-4282-94FD-55...                               {2, 32775}                    {OK, The protocol version ...
D10D2D97-5F36-4951-B748-8D...                               {2}                           {OK}
D21FEC1A-EF5F-4312-9D17-FF...                               {2}                           {OK}
D4709582-1295-45E7-BCFE-1D...                               {2}                           {OK}
D7D02D8C-4E22-4C40-87AE-7A...                               {2}                           {OK}
D7FD9B0F-122C-4DAC-A039-70...                               {2}                           {OK}
D9A03BB5-AC16-4C12-8A47-23...                               {2}                           {OK}
D9E43449-2331-41C5-826E-A7...                               {2}                           {OK}
DDDE6914-97F1-4844-8876-68...                               {2}                           {OK}
E4D48857-A1BC-4902-B0D8-E4...                               {2, 32775}                    {OK, The protocol version ...
E502801F-B768-4DEA-BBA7-A8...                               {2, 32775}                    {OK, The protocol version ...
EB3EF232-5627-49D2-BC28-42...                               {2}                           {OK}
EB8297E2-70C8-4585-90BF-CF...                               {2}                           {OK}
F3E95B86-AE38-48BF-9632-52...                               {2, 32775}                    {OK, The protocol version ...
F65E621F-387B-416A-A5FC-E0...                               {2}                           {OK}
F9D8C0F0-9D7C-4980-976D-9D...                               {2}                           {OK}


PS C:\Users\fw> gwmi -namespace "root\virtualization\v2" -class "Msvm_ShutdownComponent" | select SystemName, Status,Op
rationalStatus,StatusDescriptions

SystemName                    Status                        OperationalStatus             StatusDescriptions
----------                    ------                        -----------------             ------------------
0044FBBC-58C7-419D-B5AD-A9...                               {2}                           {OK}
03F28958-AFFE-44D0-BC58-AB...                               {2}                           {OK}
05771D72-5A12-4C51-9F92-41...                               {2}                           {OK}
087F749F-D381-45A3-A22A-D8...                               {2}                           {OK}
0AF488FF-3D24-474F-ADDF-2B...                               {2}                           {OK}
0AF5B7E3-623E-4040-9FED-28...                               {2}                           {OK}
0FB659A8-3F6A-4F7B-9331-FE...                               {2}                           {OK}
104DB6C7-35DD-4B49-B374-C4...                               {2}                           {OK}
15503CAC-C13D-41B2-9805-67...                               {2}                           {OK}
19732D3E-432F-4BDF-B3D7-B3...                               {2}                           {OK}
1995B55A-3FE4-4B10-BF4E-D1...                               {2}                           {OK}
19DF4191-2ED4-4A2C-9D27-C7...                               {2}                           {OK}
22494038-87A6-4DA2-9DC4-6F...                               {2}                           {OK}
23B4F67B-939A-4941-A359-C7...                               {2}                           {OK}
36F59B64-28F0-4B83-8AF5-E3...                               {2}                           {OK}
3FA35398-37F8-456B-AFD8-7F...                               {2}                           {OK}
40836797-48A2-4936-8801-FB...                               {2}                           {OK}
4108287C-04E4-4C9E-80CB-18...                               {2}                           {OK}
53DD4ABB-6A94-4205-B07C-98...                               {2}                           {OK}
54747956-E170-443B-B3B5-E7...                               {2}                           {OK}
5571E37D-4D1E-4E93-8BD0-48...                               {2}                           {OK}
579A1E6A-1799-46C3-9A87-9A...                               {2}                           {OK}
5A2BB795-8786-4FEC-A272-5E...                               {2}                           {OK}
6035A510-FE52-4B95-AE2F-9D...                               {2}                           {OK}
61074F82-32CA-4647-B743-5B...                               {2}                           {OK}
615E438F-1028-46B8-937A-57...                               {2}                           {OK}
69AC58C6-CE3F-40F7-9B6F-20...                               {2}                           {OK}
6AC6D22C-2E5F-47C7-A088-71...                               {2}                           {OK}
6AF770E1-B831-4E25-B96F-60...                               {2}                           {OK}
770E1DEC-3F16-46FF-8443-BC...                               {2}                           {OK}
77879E58-733D-44DC-BEDF-41...                               {2}                           {OK}
810DF37B-6189-4E2F-84ED-4C...                               {2}                           {OK}
96B6C934-D7F1-48C4-BC5E-9D...                               {2}                           {OK}
976632CB-0281-494B-9287-C4...                               {2}                           {OK}
9C6A54A5-8B9C-4A61-A5D8-E8...                               {2}                           {OK}
9D099A04-A37A-4261-89AC-D6...                               {2}                           {OK}
9EC7AECB-C0A0-4ADD-BE3B-F7...                               {2}                           {OK}
9ECA410A-4BBF-4AC9-80CC-6D...                               {2}                           {OK}
9F4AE2AA-E2DA-4785-B57B-63...                               {2}                           {OK}
A25045C9-FEF3-41D7-8A66-B5...                               {2}                           {OK}
A35E384A-BEAB-4C7D-83A8-C9...                               {2}                           {OK}
A3B5B9B6-0980-41AE-AEA8-A6...                               {2}                           {OK}
A7162BD3-440E-4D52-9574-CB...                               {2}                           {OK}
AA0B5C8E-DA30-4389-9CC5-09...                               {2}                           {OK}
B63B2E57-1250-4CA6-BA19-B5...                               {2}                           {OK}
B7063626-AD99-462A-A6D7-AA...                               {2}                           {OK}
BD420633-9F4A-43E7-9B88-73...                               {2}                           {OK}
BD863CC8-005C-41A4-A802-EB...                               {2}                           {OK}
C43DE05E-407C-40A1-B926-57...                               {2}                           {OK}
CA14F43D-4305-4131-A017-5F...                               {2}                           {OK}
CE250E40-4F4A-4206-89BC-E0...                               {2}                           {OK}
CEA71ED2-45A7-4C40-BA38-DB...                               {2}                           {OK}
CEDAF65F-B932-47A8-9DE5-C1...                               {2}                           {OK}
CEEF1039-75BE-4282-94FD-55...                               {2}                           {OK}
D10D2D97-5F36-4951-B748-8D...                               {2}                           {OK}
D21FEC1A-EF5F-4312-9D17-FF...                               {2}                           {OK}
D4709582-1295-45E7-BCFE-1D...                               {2}                           {OK}
D7D02D8C-4E22-4C40-87AE-7A...                               {2}                           {OK}
D7FD9B0F-122C-4DAC-A039-70...                               {2}                           {OK}
D9A03BB5-AC16-4C12-8A47-23...                               {2}                           {OK}
D9E43449-2331-41C5-826E-A7...                               {2}                           {OK}
DDDE6914-97F1-4844-8876-68...                               {2}                           {OK}
E4D48857-A1BC-4902-B0D8-E4...                               {2}                           {OK}
E502801F-B768-4DEA-BBA7-A8...                               {2}                           {OK}
EB3EF232-5627-49D2-BC28-42...                               {2}                           {OK}
EB8297E2-70C8-4585-90BF-CF...                               {2}                           {OK}
F3E95B86-AE38-48BF-9632-52...                               {2}                           {OK}
F65E621F-387B-416A-A5FC-E0...                               {2}                           {OK}
F9D8C0F0-9D7C-4980-976D-9D...                               {2}                           {OK}

Sorry for the late reply, been out of office some days...

[sergey.g]

Hi Frode,

So, from this server only two VMs should be alerting, the ones that have 12 - No Contact

Veeam MP takes the first operational status into account, so version mismatch is not considered a service failure. You can get names for those VMs using the following command:

Code: Select all

gwmi -namespace "root\virtualization\v2" -class "Msvm_ComputerSystem" | ? {$_.Name -like "4108287C-04E4-4C9E-80CB-18*" -or $_.Name -like "CEA71ED2-45A7-4C40-BA38-DB*"} | select elementname

Let me know if you have any other concerns
Thanks.