Plug-in backup data collection failure

[pufferdude]

Couldn't find any threads that talk about this, so thought I'd ask before opening a ticket in case it's something trivial.

Out of the blue, my Veeam B&R 11.0.1.1261 / VeeamOne 11.0.1.1880 server has started alerting me to "backup data collection failures" related to plug-ins that we don't even use or have configured.

The first alert was:

"Veeam ONE server has failed to collect Microsoft Azure data from Veeam Backup & Replication server running Microsoft Azure Plug-in for Veeam Backup & Replication"

which was automatically Reset/Resolved 1 minute later. Then 3hr later I got:

"Veeam ONE server has failed to collect Google data from Veeam Backup & Replication server running Google Plug-in for Veeam Backup & Replication"

which again resolved 1 minute later. This started out of the blue after patching the server to the January 2023 Win updates and rebooting (I have no idea if it's related and just a coincidence since the first error occurred like 12r after the reboot , but we've never seen this errors before so mentioning it.)

Anyone have an idea why we're getting these, and what to do about it, other than suppress the alert? We're not currently using these plugins in any way.

[Vitaliy S.]

Hi Jim,

These plugins are automatically installed with Veeam B&R. Can you please check if corresponding services are up and running on the Veeam backup side?

Thanks!

[pufferdude]

Ah, didn't realize that, sorry! But yes, I see Veeam AWS, Azure and GCP services all running.

[Helmi]

We have exact the same problem with the same version-numbers of Veeam One and Backup Replication. We also not using any kind of Azure or Google-backup(plugin). Windows Updates were installed on the same day the errors started, so perhaps its really related.

[RomanK]

Hi all,

In general, there are three possible reasons for the false Plug-in backup data collection failure alarms:

• The Azure, AWS, or Nutanix Plugin connection ports are closed (443, 6172, 9402, 20443)

• The Azure, AWS, or Nutanix Plugin platform service is stopped

• The Azure, AWS, or Nutanix Plugin REST does not present properly

Try to check the ports and the services. If everything is OK with them and there are still false positives our support should help.

Thanks

[pufferdude]

Just to clarify, are you saying the ports you listed must be open *inbound* to the VB&R?VeeamONE server? If that's the case, only 9402 and 20443 seem to be open on my server (verified by telnettting to the ports).

It seems weird that it would be a firewall issue, because the "failure" error happens, then within a minute it is resolved (and how could it be resolved if the comms are still broken?) Then it happens again... seems to be occurring for every plugin, every couple of hours since I installed Jan (MS) updates. I think I'll just go ahead and open a ticket and see where it goes.

Thanks!

[jorgedlcruz]

Hello Jim,
I think your thinking is correct, if it is an intermittent problem, or it appeared suddenly it is better to discuss it with Support so that we can investigate the logs of the source cause.

Please share with us the ticket ID after you do so.

Thanks a lot!

[RomanK]

Hello Jim,

I've just shared the most common causes of the problem addressed in support.

Then it happens again...

There could be other reasons which would require additional troubleshooting. Please open a support case and provide the case ID in this thread so we could collect all the details and investigate it accordingly.

Thanks

[pufferdude]

I opened case #05831640 So far they are wanting me to verify that the proper ports are open and that the services are running (duh).

So far I have verified that 443, 6172, 9402, and 20443 are all open by visually looking at the Windows firewall config AND telnetting to them via a host outside of the Veeam server. All ports respond to telnet EXCEPT 443, which doesn't seem to have anything "listening" on it even though it is open in the firewall. Maybe that alone is the source of this issue, but if so, it's a problem that the January 2023 montly updates caused, not the firewall.

While waiting to hear back from support I un-installed the January 2023 security update, just to see if the errors stop again.

[RomanK]

Hello Jim

Thanks for the case number.
Please continue to work with our support engineers. I'm confident they will find the cause of this issue.

Thanks

[pufferdude]

Just an update... support escalated my ticket and they remoted into the box and are stumped by the behavior they're seeing (plug-in error, then a minute later it's resolved). Reverting the January updates did NOT fix the issue, so that was a red herring (or it indirectly caused the problem that doesn't get fixed when you uninstall the updates). Support will continue to dig into what's happening, but it's not a simple "port not open" or "service not running" problem.

[RomanK]

Hello Jim

Thanks for the update. As far as I can see the case is assigned to the high-tier support engineer. Indeed, it has become not a simple issue.

Thanks

[srahim]

Hello, do you have any update on this case ? i have the same problem.

[RomanK]

Hello srahim,

The case has been escalated to the highest level of support, and as far as I can see, it is still being investigated.

Thanks.

[pufferdude]

Yeah... so far they are stumped. They even have a complete copy of my VeeamONE db

[RomanK]

Hi all,

As far as I can see case #05831640 was closed. The issue was fixed after upgrading to v12.

Thanks

[pufferdude]

Yes. I was told it was fixed in v12, but haven't upgraded to that yet, so can't verify.

[tholyoak]

This just started happening over the weekend on my Veeam server, as well. This was after a reboot like everyone else is reporting. (I also moved the server to a different host while I'm doing VMware upgrades, but everything else is working normally.) It appears to have started about 24 hours after the VM was moved/turned back on.

I am on build 11.0.1.1261 P20230227

[jorgedlcruz]

Hello,
Please create a support ticket if you can, but we do not have a Veeam ONE build that matches that number, it seems is a VBR with latest patch, which is good to see
https://www.veeam.com/kb4357

What version of Veeam ONE are you running today?

[tholyoak]

Oops, sorry I went to the wrong product to get the version number. The Veeam ONE build is 11.0.1.1880.

[tholyoak]

I've now upgraded to 12.0.1.2591 and still getting the error, though now not nearly as frequently as before.

[RomanK]

Hello tholyoak,

It is pity to hear that, it would be great if you would open a new support case and we will continue troubleshooting. Might be there was another root cause than in 05831640.

Thanks