Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V in a single System Center Operations Manager Console
Post Reply
frowiers
Service Provider
Posts: 13
Liked: never
Joined: Feb 11, 2016 6:13 pm
Full Name: Frode Åbø Wiersen
Contact:

Alerts: Hyper-V Integration Service(s)

Post by frowiers »

Hi

We have installed "Veeam MP for Hyper-V" in our environment, monitoring about 800 virtual machines.

On many of our Citrix VDA's (atm. number of alerts: 137) we have the following Alert in SCOM:
The following Hyper-V Integration Service(s) are not running or not responding in the guest OS:
Volume Shadow Copy Requestor Service
I have verified the service is running, and have even reset the alert in SCOM, but after 5 minutes (the threshold is 300 seconds) the Alert is back.
  • Why is this alert showing when service is running
  • Why is this alert only affecting (mostly) Citrix VDA's

Any tip how to solve this would be highly appreciated :)

Best Regards Frode @ Netscenario
Frode @ Netscenario
sergey.g
Veteran
Posts: 452
Liked: 76 times
Joined: May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko
Contact:

Re: Alerts: Hyper-V Integration Service(s)

Post by sergey.g »

Hi Frode,

Is it possible to run the followign powershell command on the Hyper-V servers which are runnign VMs affected by this issue?

Code: Select all

gwmi -namespace "root\virtualization\v2" -class "Msvm_VssComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions
We suspect that some Citrix integration with in-guest VSS services could cause it to switch into some non-healthy or semi-healthy state and output of the command above could help us detect what is going on.

We would highly appreciate your help with troubleshooting the issue.

Thanks.
-=MACROS=-
Influencer
Posts: 16
Liked: 2 times
Joined: Jun 27, 2016 12:18 pm
Full Name: Rodion Fedin
Location: Germany, Nuremberg
Contact:

Re: Alerts: Hyper-V Integration Service(s)

Post by -=MACROS=- »

Hi Frode,

On the settings of your vServer/vClient did you've something like Backup Feature? On the vServer on Hyper-V you can set special this options onto vServer options. Maybe its you goal?

Regards

Rodi


Gesendet von iPhone mit Tapatalk
-=MACROS=-
Influencer
Posts: 16
Liked: 2 times
Joined: Jun 27, 2016 12:18 pm
Full Name: Rodion Fedin
Location: Germany, Nuremberg
Contact:

Re: Alerts: Hyper-V Integration Service(s)

Post by -=MACROS=- »

-=MACROS=- wrote:Hi Frode,

On the settings of your vServer/vClient did you've something like Backup Feature? On the vServer on Hyper-V you can set special this options onto vServer options. Maybe its you goal?

http://www.askme4tech.com/sites/default ... ttings.png

Regards

Rodi


Gesendet von iPhone mit Tapatalk

Gesendet von iPhone mit Tapatalk
frowiers
Service Provider
Posts: 13
Liked: never
Joined: Feb 11, 2016 6:13 pm
Full Name: Frode Åbø Wiersen
Contact:

Re: Alerts: Hyper-V Integration Service(s)

Post by frowiers »

sergey.g wrote:Hi Frode,

Is it possible to run the followign powershell command on the Hyper-V servers which are runnign VMs affected by this issue?

Code: Select all

gwmi -namespace "root\virtualization\v2" -class "Msvm_VssComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions
We suspect that some Citrix integration with in-guest VSS services could cause it to switch into some non-healthy or semi-healthy state and output of the command above could help us detect what is going on.

We would highly appreciate your help with troubleshooting the issue.

Thanks.
This is what is written out when I run the cmdlet you suggested:

Image
Frode @ Netscenario
sergey.g
Veteran
Posts: 452
Liked: 76 times
Joined: May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko
Contact:

Re: Alerts: Hyper-V Integration Service(s)

Post by sergey.g »

Hi Frode,

Thank you very much for the provided output. We can see that some VMs has VSS Integration Service in "No Contact" state, which according to Microsoft means "The guest service is not installed or has not yet been contacted."

Our monitor is tracking services which have been enabled in the VM properties, so for us it means that the service needs to be monitored and that's why you see an issue when state is not "OK" for some VMs.

This service is necessary for Backup solutions which are using VSS inside guest OSes, I assume that backup with enabled VSS integration would fail for these VMs with "No Contact" state, so clearly for us it's a critical error.

If you don't need VSS integration for some VMs, you can either disable it on the Hyper-V side in the VM properties (in this case a Monitor will not trigger for VSS service issues) or you can disable the entire monitor for a group of VDA VMs for example (although in this case other Integration services will not be monitored either)

Let me know if you have any other questions or suggestions for Hyper-V Integration Services Monitor architecture.
Thanks.
frowiers
Service Provider
Posts: 13
Liked: never
Joined: Feb 11, 2016 6:13 pm
Full Name: Frode Åbø Wiersen
Contact:

Re: Alerts: Hyper-V Integration Service(s)

Post by frowiers »

-=MACROS=- wrote:Hi Frode,

On the settings of your vServer/vClient did you've something like Backup Feature? On the vServer on Hyper-V you can set special this options onto vServer options. Maybe its you goal?

Regards

Rodi


Gesendet von iPhone mit Tapatalk
sergey.g wrote:Hi Frode,

Thank you very much for the provided output. We can see that some VMs has VSS Integration Service in "No Contact" state, which according to Microsoft means "The guest service is not installed or has not yet been contacted."

Our monitor is tracking services which have been enabled in the VM properties, so for us it means that the service needs to be monitored and that's why you see an issue when state is not "OK" for some VMs.

This service is necessary for Backup solutions which are using VSS inside guest OSes, I assume that backup with enabled VSS integration would fail for these VMs with "No Contact" state, so clearly for us it's a critical error.

If you don't need VSS integration for some VMs, you can either disable it on the Hyper-V side in the VM properties (in this case a Monitor will not trigger for VSS service issues) or you can disable the entire monitor for a group of VDA VMs for example (although in this case other Integration services will not be monitored either)

Let me know if you have any other questions or suggestions for Hyper-V Integration Services Monitor architecture.
Thanks.

Hi

Thanks for the replies! A combination of your replies actually gave me the answer I was looking for. None of these Citrix VDA's are being backed up using application aware backups (VSS), so the resolution were to disable the Backup Integration Component on all Citrix VDA's :)

I used the following script to do so:

Code: Select all

$VMMFolder = "CTX"

$VMMServer = "mgmt50.mgmt.local"

$vms = Get-SCVirtualMachine -VMMServer $VMMServer | where {$_.HostGroupPath -match $VMMFolder}
foreach ($vm in $vms){
        write-host $vm.Name
        Set-SCVirtualMachine $vm -EnableBackup $false
}
After waiting the threshold of the monitor in SCOM (300 seconds / 5 minutes) all my alerts related to the monitor "Veeam HyperV: VM Integration Services Status" went away :)
Frode @ Netscenario
frowiers
Service Provider
Posts: 13
Liked: never
Joined: Feb 11, 2016 6:13 pm
Full Name: Frode Åbø Wiersen
Contact:

Re: Alerts: Hyper-V Integration Service(s)

Post by frowiers »

Hi again,
I am afraid this only solved parts of the errors.

I still have SCOM complaining about integration services not running on several virtual machines. This can be several of the services offered (vmicvss, vmickvpexchange, vmicshutdown).

Is there any way I can verify / test the integration service communication, to see where the problem could be?
Frode @ Netscenario
sergey.g
Veteran
Posts: 452
Liked: 76 times
Joined: May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko
Contact:

Re: Alerts: Hyper-V Integration Service(s)

Post by sergey.g »

Hi Frode,

Sorry for a delay, could you try the following commands on thy Hyper-V server?

Code: Select all

gwmi -namespace "root\virtualization\v2" -class "Msvm_KvpExchangeComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions
gwmi -namespace "root\virtualization\v2" -class "Msvm_ShutdownComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions
Also, do you see these errors for same VMs as with VSS/Backup issue?

Thanks
frowiers
Service Provider
Posts: 13
Liked: never
Joined: Feb 11, 2016 6:13 pm
Full Name: Frode Åbø Wiersen
Contact:

Re: Alerts: Hyper-V Integration Service(s)

Post by frowiers »

sergey.g wrote:Hi Frode,

Sorry for a delay, could you try the following commands on thy Hyper-V server?

Code: Select all

gwmi -namespace "root\virtualization\v2" -class "Msvm_KvpExchangeComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions
gwmi -namespace "root\virtualization\v2" -class "Msvm_ShutdownComponent" | select SystemName, Status,OperationalStatus,StatusDescriptions
Also, do you see these errors for same VMs as with VSS/Backup issue?

Thanks
Hi

Not all of these VMs are included in Application Aware backups, so it is hard to say; the backup jobs are filtered on Tags in SCVMM, so not sure how I can exclude monitoring for VMs with a curtain TAG inside SCOM.

I have run the cmdlets on one Hyper-V Host hosting a VM where SCOM reports a failure on a integration service.

This is the output from the cmdlet:

Code: Select all

PS C:\Users\fw> gwmi -namespace "root\virtualization\v2" -class "Msvm_KvpExchangeComponent" | select SystemName, Status
OperationalStatus,StatusDescriptions

SystemName                    Status                        OperationalStatus             StatusDescriptions
----------                    ------                        -----------------             ------------------
0044FBBC-58C7-419D-B5AD-A9...                               {2, 32775}                    {OK, The protocol version ...
03F28958-AFFE-44D0-BC58-AB...                               {2}                           {OK}
05771D72-5A12-4C51-9F92-41...                               {2}                           {OK}
087F749F-D381-45A3-A22A-D8...                               {2}                           {OK}
0AF488FF-3D24-474F-ADDF-2B...                               {2}                           {OK}
0AF5B7E3-623E-4040-9FED-28...                               {2, 32775}                    {OK, The protocol version ...
0FB659A8-3F6A-4F7B-9331-FE...                               {2, 32775}                    {OK, The protocol version ...
104DB6C7-35DD-4B49-B374-C4...                               {2}                           {OK}
15503CAC-C13D-41B2-9805-67...                               {2}                           {OK}
19732D3E-432F-4BDF-B3D7-B3...                               {2}                           {OK}
1995B55A-3FE4-4B10-BF4E-D1...                               {2}                           {OK}
19DF4191-2ED4-4A2C-9D27-C7...                               {2, 32775}                    {OK, The protocol version ...
22494038-87A6-4DA2-9DC4-6F...                               {2}                           {OK}
23B4F67B-939A-4941-A359-C7...                               {2}                           {OK}
36F59B64-28F0-4B83-8AF5-E3...                               {2, 32775}                    {OK, The protocol version ...
3FA35398-37F8-456B-AFD8-7F...                               {2}                           {OK}
40836797-48A2-4936-8801-FB...                               {2}                           {OK}
4108287C-04E4-4C9E-80CB-18...                               {12}                          {No Contact}
53DD4ABB-6A94-4205-B07C-98...                               {2, 32775}                    {OK, The protocol version ...
54747956-E170-443B-B3B5-E7...                               {2, 32775}                    {OK, The protocol version ...
5571E37D-4D1E-4E93-8BD0-48...                               {2}                           {OK}
579A1E6A-1799-46C3-9A87-9A...                               {2}                           {OK}
5A2BB795-8786-4FEC-A272-5E...                               {2}                           {OK}
6035A510-FE52-4B95-AE2F-9D...                               {2}                           {OK}
61074F82-32CA-4647-B743-5B...                               {2, 32775}                    {OK, The protocol version ...
615E438F-1028-46B8-937A-57...                               {2, 32775}                    {OK, The protocol version ...
69AC58C6-CE3F-40F7-9B6F-20...                               {2}                           {OK}
6AC6D22C-2E5F-47C7-A088-71...                               {2}                           {OK}
6AF770E1-B831-4E25-B96F-60...                               {2, 32775}                    {OK, The protocol version ...
770E1DEC-3F16-46FF-8443-BC...                               {2}                           {OK}
77879E58-733D-44DC-BEDF-41...                               {2, 32775}                    {OK, The protocol version ...
810DF37B-6189-4E2F-84ED-4C...                               {2, 32775}                    {OK, The protocol version ...
96B6C934-D7F1-48C4-BC5E-9D...                               {2}                           {OK}
976632CB-0281-494B-9287-C4...                               {2}                           {OK}
9C6A54A5-8B9C-4A61-A5D8-E8...                               {2}                           {OK}
9D099A04-A37A-4261-89AC-D6...                               {2, 32775}                    {OK, The protocol version ...
9EC7AECB-C0A0-4ADD-BE3B-F7...                               {2, 32775}                    {OK, The protocol version ...
9ECA410A-4BBF-4AC9-80CC-6D...                               {2}                           {OK}
9F4AE2AA-E2DA-4785-B57B-63...                               {2}                           {OK}
A25045C9-FEF3-41D7-8A66-B5...                               {2}                           {OK}
A35E384A-BEAB-4C7D-83A8-C9...                               {2}                           {OK}
A3B5B9B6-0980-41AE-AEA8-A6...                               {2}                           {OK}
A7162BD3-440E-4D52-9574-CB...                               {2, 32775}                    {OK, The protocol version ...
AA0B5C8E-DA30-4389-9CC5-09...                               {2}                           {OK}
B63B2E57-1250-4CA6-BA19-B5...                               {2, 32775}                    {OK, The protocol version ...
B7063626-AD99-462A-A6D7-AA...                               {2}                           {OK}
BD420633-9F4A-43E7-9B88-73...                               {2}                           {OK}
BD863CC8-005C-41A4-A802-EB...                               {2, 32775}                    {OK, The protocol version ...
C43DE05E-407C-40A1-B926-57...                               {2}                           {OK}
CA14F43D-4305-4131-A017-5F...                               {2}                           {OK}
CE250E40-4F4A-4206-89BC-E0...                               {2}                           {OK}
CEA71ED2-45A7-4C40-BA38-DB...                               {12}                          {No Contact}
CEDAF65F-B932-47A8-9DE5-C1...                               {2}                           {OK}
CEEF1039-75BE-4282-94FD-55...                               {2, 32775}                    {OK, The protocol version ...
D10D2D97-5F36-4951-B748-8D...                               {2}                           {OK}
D21FEC1A-EF5F-4312-9D17-FF...                               {2}                           {OK}
D4709582-1295-45E7-BCFE-1D...                               {2}                           {OK}
D7D02D8C-4E22-4C40-87AE-7A...                               {2}                           {OK}
D7FD9B0F-122C-4DAC-A039-70...                               {2}                           {OK}
D9A03BB5-AC16-4C12-8A47-23...                               {2}                           {OK}
D9E43449-2331-41C5-826E-A7...                               {2}                           {OK}
DDDE6914-97F1-4844-8876-68...                               {2}                           {OK}
E4D48857-A1BC-4902-B0D8-E4...                               {2, 32775}                    {OK, The protocol version ...
E502801F-B768-4DEA-BBA7-A8...                               {2, 32775}                    {OK, The protocol version ...
EB3EF232-5627-49D2-BC28-42...                               {2}                           {OK}
EB8297E2-70C8-4585-90BF-CF...                               {2}                           {OK}
F3E95B86-AE38-48BF-9632-52...                               {2, 32775}                    {OK, The protocol version ...
F65E621F-387B-416A-A5FC-E0...                               {2}                           {OK}
F9D8C0F0-9D7C-4980-976D-9D...                               {2}                           {OK}


PS C:\Users\fw> gwmi -namespace "root\virtualization\v2" -class "Msvm_ShutdownComponent" | select SystemName, Status,Op
rationalStatus,StatusDescriptions

SystemName                    Status                        OperationalStatus             StatusDescriptions
----------                    ------                        -----------------             ------------------
0044FBBC-58C7-419D-B5AD-A9...                               {2}                           {OK}
03F28958-AFFE-44D0-BC58-AB...                               {2}                           {OK}
05771D72-5A12-4C51-9F92-41...                               {2}                           {OK}
087F749F-D381-45A3-A22A-D8...                               {2}                           {OK}
0AF488FF-3D24-474F-ADDF-2B...                               {2}                           {OK}
0AF5B7E3-623E-4040-9FED-28...                               {2}                           {OK}
0FB659A8-3F6A-4F7B-9331-FE...                               {2}                           {OK}
104DB6C7-35DD-4B49-B374-C4...                               {2}                           {OK}
15503CAC-C13D-41B2-9805-67...                               {2}                           {OK}
19732D3E-432F-4BDF-B3D7-B3...                               {2}                           {OK}
1995B55A-3FE4-4B10-BF4E-D1...                               {2}                           {OK}
19DF4191-2ED4-4A2C-9D27-C7...                               {2}                           {OK}
22494038-87A6-4DA2-9DC4-6F...                               {2}                           {OK}
23B4F67B-939A-4941-A359-C7...                               {2}                           {OK}
36F59B64-28F0-4B83-8AF5-E3...                               {2}                           {OK}
3FA35398-37F8-456B-AFD8-7F...                               {2}                           {OK}
40836797-48A2-4936-8801-FB...                               {2}                           {OK}
4108287C-04E4-4C9E-80CB-18...                               {2}                           {OK}
53DD4ABB-6A94-4205-B07C-98...                               {2}                           {OK}
54747956-E170-443B-B3B5-E7...                               {2}                           {OK}
5571E37D-4D1E-4E93-8BD0-48...                               {2}                           {OK}
579A1E6A-1799-46C3-9A87-9A...                               {2}                           {OK}
5A2BB795-8786-4FEC-A272-5E...                               {2}                           {OK}
6035A510-FE52-4B95-AE2F-9D...                               {2}                           {OK}
61074F82-32CA-4647-B743-5B...                               {2}                           {OK}
615E438F-1028-46B8-937A-57...                               {2}                           {OK}
69AC58C6-CE3F-40F7-9B6F-20...                               {2}                           {OK}
6AC6D22C-2E5F-47C7-A088-71...                               {2}                           {OK}
6AF770E1-B831-4E25-B96F-60...                               {2}                           {OK}
770E1DEC-3F16-46FF-8443-BC...                               {2}                           {OK}
77879E58-733D-44DC-BEDF-41...                               {2}                           {OK}
810DF37B-6189-4E2F-84ED-4C...                               {2}                           {OK}
96B6C934-D7F1-48C4-BC5E-9D...                               {2}                           {OK}
976632CB-0281-494B-9287-C4...                               {2}                           {OK}
9C6A54A5-8B9C-4A61-A5D8-E8...                               {2}                           {OK}
9D099A04-A37A-4261-89AC-D6...                               {2}                           {OK}
9EC7AECB-C0A0-4ADD-BE3B-F7...                               {2}                           {OK}
9ECA410A-4BBF-4AC9-80CC-6D...                               {2}                           {OK}
9F4AE2AA-E2DA-4785-B57B-63...                               {2}                           {OK}
A25045C9-FEF3-41D7-8A66-B5...                               {2}                           {OK}
A35E384A-BEAB-4C7D-83A8-C9...                               {2}                           {OK}
A3B5B9B6-0980-41AE-AEA8-A6...                               {2}                           {OK}
A7162BD3-440E-4D52-9574-CB...                               {2}                           {OK}
AA0B5C8E-DA30-4389-9CC5-09...                               {2}                           {OK}
B63B2E57-1250-4CA6-BA19-B5...                               {2}                           {OK}
B7063626-AD99-462A-A6D7-AA...                               {2}                           {OK}
BD420633-9F4A-43E7-9B88-73...                               {2}                           {OK}
BD863CC8-005C-41A4-A802-EB...                               {2}                           {OK}
C43DE05E-407C-40A1-B926-57...                               {2}                           {OK}
CA14F43D-4305-4131-A017-5F...                               {2}                           {OK}
CE250E40-4F4A-4206-89BC-E0...                               {2}                           {OK}
CEA71ED2-45A7-4C40-BA38-DB...                               {2}                           {OK}
CEDAF65F-B932-47A8-9DE5-C1...                               {2}                           {OK}
CEEF1039-75BE-4282-94FD-55...                               {2}                           {OK}
D10D2D97-5F36-4951-B748-8D...                               {2}                           {OK}
D21FEC1A-EF5F-4312-9D17-FF...                               {2}                           {OK}
D4709582-1295-45E7-BCFE-1D...                               {2}                           {OK}
D7D02D8C-4E22-4C40-87AE-7A...                               {2}                           {OK}
D7FD9B0F-122C-4DAC-A039-70...                               {2}                           {OK}
D9A03BB5-AC16-4C12-8A47-23...                               {2}                           {OK}
D9E43449-2331-41C5-826E-A7...                               {2}                           {OK}
DDDE6914-97F1-4844-8876-68...                               {2}                           {OK}
E4D48857-A1BC-4902-B0D8-E4...                               {2}                           {OK}
E502801F-B768-4DEA-BBA7-A8...                               {2}                           {OK}
EB3EF232-5627-49D2-BC28-42...                               {2}                           {OK}
EB8297E2-70C8-4585-90BF-CF...                               {2}                           {OK}
F3E95B86-AE38-48BF-9632-52...                               {2}                           {OK}
F65E621F-387B-416A-A5FC-E0...                               {2}                           {OK}
F9D8C0F0-9D7C-4980-976D-9D...                               {2}                           {OK}
Sorry for the late reply, been out of office some days...
Frode @ Netscenario
sergey.g
Veteran
Posts: 452
Liked: 76 times
Joined: May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko
Contact:

Re: Alerts: Hyper-V Integration Service(s)

Post by sergey.g »

Hi Frode,

So, from this server only two VMs should be alerting, the ones that have 12 - No Contact

Veeam MP takes the first operational status into account, so version mismatch is not considered a service failure. You can get names for those VMs using the following command:

Code: Select all

gwmi -namespace "root\virtualization\v2" -class "Msvm_ComputerSystem" | ? {$_.Name -like "4108287C-04E4-4C9E-80CB-18*" -or $_.Name -like "CEA71ED2-45A7-4C40-BA38-DB*"} | select elementname
Let me know if you have any other concerns
Thanks.
Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests