Real-time performance monitoring and troubleshooting
Post Reply
pufferdude
Expert
Posts: 222
Liked: 15 times
Joined: Jul 02, 2009 8:26 pm
Full Name: Jim
Contact:

Plug-in backup data collection failure

Post by pufferdude »

Couldn't find any threads that talk about this, so thought I'd ask before opening a ticket in case it's something trivial.

Out of the blue, my Veeam B&R 11.0.1.1261 / VeeamOne 11.0.1.1880 server has started alerting me to "backup data collection failures" related to plug-ins that we don't even use or have configured.

The first alert was:

"Veeam ONE server has failed to collect Microsoft Azure data from Veeam Backup & Replication server running Microsoft Azure Plug-in for Veeam Backup & Replication"

which was automatically Reset/Resolved 1 minute later. Then 3hr later I got:

"Veeam ONE server has failed to collect Google data from Veeam Backup & Replication server running Google Plug-in for Veeam Backup & Replication"

which again resolved 1 minute later. This started out of the blue after patching the server to the January 2023 Win updates and rebooting (I have no idea if it's related and just a coincidence since the first error occurred like 12r after the reboot , but we've never seen this errors before so mentioning it.)

Anyone have an idea why we're getting these, and what to do about it, other than suppress the alert? We're not currently using these plugins in any way.
Vitaliy S.
VP, Product Management
Posts: 26983
Liked: 2699 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Plug-in backup data collection failure

Post by Vitaliy S. »

Hi Jim,

These plugins are automatically installed with Veeam B&R. Can you please check if corresponding services are up and running on the Veeam backup side?

Thanks!
pufferdude
Expert
Posts: 222
Liked: 15 times
Joined: Jul 02, 2009 8:26 pm
Full Name: Jim
Contact:

Re: Plug-in backup data collection failure

Post by pufferdude »

Ah, didn't realize that, sorry! But yes, I see Veeam AWS, Azure and GCP services all running.
Helmi
Lurker
Posts: 1
Liked: never
Joined: Jan 20, 2023 8:44 am
Contact:

Re: Plug-in backup data collection failure

Post by Helmi »

We have exact the same problem with the same version-numbers of Veeam One and Backup Replication. We also not using any kind of Azure or Google-backup(plugin). Windows Updates were installed on the same day the errors started, so perhaps its really related.
RomanK
Veeam Software
Posts: 607
Liked: 161 times
Joined: Nov 01, 2016 11:26 am
Contact:

Re: Plug-in backup data collection failure

Post by RomanK »

Hi all,

In general, there are three possible reasons for the false Plug-in backup data collection failure alarms:
  • The Azure, AWS, or Nutanix Plugin connection ports are closed (443, 6172, 9402, 20443)
  • The Azure, AWS, or Nutanix Plugin platform service is stopped
  • The Azure, AWS, or Nutanix Plugin REST does not present properly
Try to check the ports and the services. If everything is OK with them and there are still false positives our support should help.

Thanks
pufferdude
Expert
Posts: 222
Liked: 15 times
Joined: Jul 02, 2009 8:26 pm
Full Name: Jim
Contact:

Re: Plug-in backup data collection failure

Post by pufferdude »

Just to clarify, are you saying the ports you listed must be open *inbound* to the VB&R?VeeamONE server? If that's the case, only 9402 and 20443 seem to be open on my server (verified by telnettting to the ports).

It seems weird that it would be a firewall issue, because the "failure" error happens, then within a minute it is resolved (and how could it be resolved if the comms are still broken?) Then it happens again... seems to be occurring for every plugin, every couple of hours since I installed Jan (MS) updates. I think I'll just go ahead and open a ticket and see where it goes.

Thanks!
jorgedlcruz
Veeam Software
Posts: 1327
Liked: 608 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: Plug-in backup data collection failure

Post by jorgedlcruz »

Hello Jim,
I think your thinking is correct, if it is an intermittent problem, or it appeared suddenly it is better to discuss it with Support so that we can investigate the logs of the source cause.

Please share with us the ticket ID after you do so.

Thanks a lot!
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2023 / InfluxAce / Grafana Champion
RomanK
Veeam Software
Posts: 607
Liked: 161 times
Joined: Nov 01, 2016 11:26 am
Contact:

Re: Plug-in backup data collection failure

Post by RomanK »

Hello Jim,

I've just shared the most common causes of the problem addressed in support.
pufferdude wrote: Jan 20, 2023 5:25 pm Then it happens again...
There could be other reasons which would require additional troubleshooting. Please open a support case and provide the case ID in this thread so we could collect all the details and investigate it accordingly.

Thanks
pufferdude
Expert
Posts: 222
Liked: 15 times
Joined: Jul 02, 2009 8:26 pm
Full Name: Jim
Contact:

Re: Plug-in backup data collection failure

Post by pufferdude »

I opened case #05831640 So far they are wanting me to verify that the proper ports are open and that the services are running (duh).

So far I have verified that 443, 6172, 9402, and 20443 are all open by visually looking at the Windows firewall config AND telnetting to them via a host outside of the Veeam server. All ports respond to telnet EXCEPT 443, which doesn't seem to have anything "listening" on it even though it is open in the firewall. Maybe that alone is the source of this issue, but if so, it's a problem that the January 2023 montly updates caused, not the firewall.

While waiting to hear back from support I un-installed the January 2023 security update, just to see if the errors stop again.
RomanK
Veeam Software
Posts: 607
Liked: 161 times
Joined: Nov 01, 2016 11:26 am
Contact:

Re: Plug-in backup data collection failure

Post by RomanK »

Hello Jim

Thanks for the case number.
Please continue to work with our support engineers. I'm confident they will find the cause of this issue.

Thanks
pufferdude
Expert
Posts: 222
Liked: 15 times
Joined: Jul 02, 2009 8:26 pm
Full Name: Jim
Contact:

Re: Plug-in backup data collection failure

Post by pufferdude »

Just an update... support escalated my ticket and they remoted into the box and are stumped by the behavior they're seeing (plug-in error, then a minute later it's resolved). Reverting the January updates did NOT fix the issue, so that was a red herring (or it indirectly caused the problem that doesn't get fixed when you uninstall the updates). Support will continue to dig into what's happening, but it's not a simple "port not open" or "service not running" problem.
RomanK
Veeam Software
Posts: 607
Liked: 161 times
Joined: Nov 01, 2016 11:26 am
Contact:

Re: Plug-in backup data collection failure

Post by RomanK »

Hello Jim

Thanks for the update. As far as I can see the case is assigned to the high-tier support engineer. Indeed, it has become not a simple issue.

Thanks
srahim
Lurker
Posts: 1
Liked: never
Joined: Feb 10, 2023 8:39 am
Contact:

Re: Plug-in backup data collection failure

Post by srahim »

Hello, do you have any update on this case ? i have the same problem.
RomanK
Veeam Software
Posts: 607
Liked: 161 times
Joined: Nov 01, 2016 11:26 am
Contact:

Re: Plug-in backup data collection failure

Post by RomanK »

Hello srahim,

The case has been escalated to the highest level of support, and as far as I can see, it is still being investigated.

Thanks.
pufferdude
Expert
Posts: 222
Liked: 15 times
Joined: Jul 02, 2009 8:26 pm
Full Name: Jim
Contact:

Re: Plug-in backup data collection failure

Post by pufferdude »

Yeah... so far they are stumped. They even have a complete copy of my VeeamONE db ;-)
RomanK
Veeam Software
Posts: 607
Liked: 161 times
Joined: Nov 01, 2016 11:26 am
Contact:

Re: Plug-in backup data collection failure

Post by RomanK »

Hi all,

As far as I can see case #05831640 was closed. The issue was fixed after upgrading to v12.

Thanks
pufferdude
Expert
Posts: 222
Liked: 15 times
Joined: Jul 02, 2009 8:26 pm
Full Name: Jim
Contact:

Re: Plug-in backup data collection failure

Post by pufferdude »

Yes. I was told it was fixed in v12, but haven't upgraded to that yet, so can't verify.
tholyoak
Enthusiast
Posts: 29
Liked: never
Joined: Sep 12, 2012 5:12 pm
Contact:

Re: Plug-in backup data collection failure

Post by tholyoak »

This just started happening over the weekend on my Veeam server, as well. This was after a reboot like everyone else is reporting. (I also moved the server to a different host while I'm doing VMware upgrades, but everything else is working normally.) It appears to have started about 24 hours after the VM was moved/turned back on.

I am on build 11.0.1.1261 P20230227
jorgedlcruz
Veeam Software
Posts: 1327
Liked: 608 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: Plug-in backup data collection failure

Post by jorgedlcruz »

Hello,
Please create a support ticket if you can, but we do not have a Veeam ONE build that matches that number, it seems is a VBR with latest patch, which is good to see :)
https://www.veeam.com/kb4357

What version of Veeam ONE are you running today?
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2023 / InfluxAce / Grafana Champion
tholyoak
Enthusiast
Posts: 29
Liked: never
Joined: Sep 12, 2012 5:12 pm
Contact:

Re: Plug-in backup data collection failure

Post by tholyoak »

Oops, sorry I went to the wrong product to get the version number. The Veeam ONE build is 11.0.1.1880.
tholyoak
Enthusiast
Posts: 29
Liked: never
Joined: Sep 12, 2012 5:12 pm
Contact:

Re: Plug-in backup data collection failure

Post by tholyoak »

I've now upgraded to 12.0.1.2591 and still getting the error, though now not nearly as frequently as before.
RomanK
Veeam Software
Posts: 607
Liked: 161 times
Joined: Nov 01, 2016 11:26 am
Contact:

Re: Plug-in backup data collection failure

Post by RomanK »

Hello tholyoak,

It is pity to hear that, it would be great if you would open a new support case and we will continue troubleshooting. Might be there was another root cause than in 05831640.

Thanks
Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests