Protected VMs reports shows outdated Last Backup (Replica) Date

[pirx]

Hi,

today I was checking our Protected VMs reports for the first time and I noticed that 90% of the VM's are shown as unprotected (RPO 24h). Report date is 02.02.2021 14:00:00.

Report Parameters

Code: Select all

Scope:
Virtual Infrastructure
RPO period:
24 hours (02.02.2021 14:00:00)
Exclusion mask:
 
Job type:
Backup
Business View objects:
BackupType: Veeam
Include VM templates in this report:
No
Job Exclusion list:   
-

Protected VMs

Code: Select all

Unprotected Time: Last backup more than 24 hours ago
SDET2865    Not defined    Not defined     59     24      01.02.2021 22:10:51

But if I check the same VM in Veeam Console or Veeam EBM, it shows that the same VM was backed up yesterday evening at 22:12.

Code: Select all

SDET2865 sdxxxx BJ-DE-WRB-Bau10   38 points   DE-WRB-B01   BJ-DE-WRB-Bau10    02.02.2021   22:12:56

Even Veeam One's VM Daily Protection Status shows success for this VM. Maybe I'm missing something, but if I let the report run today 14:00 and a VM was backed up yesterday at 22:12 and RPO is 24h hours, shouldn't this VM be shown as protected with data of latest backup?

[wishr]

Hi Pirx,

This VM should be shown in the report as protected unless the latest restore point from yesterday was created by a different job type (you have only the backup job type selected in the settings of the Protected VMs report). Does it remain the same if you select all job types?

If it's not the case, I would recommend opening a support case. Please, as usual, don't forget to share your case ID here, so I could take a look at how it is progressing.

Thanks

[pirx]

It stays the same when I choose all job types. I'll open a case.

[wishr]

Hello,

Could you please share your case ID with us? I'll take a look.

Thanks in advance.

[pirx]

It's Case #04626386

[pirx]

Now after ~8 months there is finally a result:

We were able to reproduce the same issue on busy environment with a big database.
The query to collect data for the report is:

SELECT * FROM [WmiServer.VmInBackupsView] WHERE ((usn > 1176901481) AND (usn <= 1177032553)) AND (usn >= -1615) ORDER BY usn DESC

Unfortunately the query is complex and require more time to complete on such environments. The Collector.WMI.Timeout was successful and it's the only workaround at the moment.
Developers are aware about this issue and will implement a solution in future releases.
So the current solution is to implement the timeout again and expand it when the issue appear again.

I cant remember how much time I've spent in searching for performance bottlenecks at our end as this was always the #1 answer to this problem.

[nikolaj]

HI pirx,

It's quite a thread in the corresponding ticket in the bug tracker.
Hopefully, the resolution will arrive much sooner. Thank you for sharing the workaround in this topic.

[wishr]

Hi Pirx,

Just a quick update.

The case got to the VBR team and they are looking for possible solutions. Issue ID 311139. No ETA for now.

Thanks

[pirx]

Even after 2 years this is still an issue. Cases: 07007689, 07002243, 07001050 and other in the last 2 years

We regularly see protection reports with unprotected VMs. But the VMs were backed up within last 24h. This happens often when Veeam servers were rebooted. But at this time no backup jobs are running, report is running at 07:30, backups are all done by ~04:00.

In current case, 6 VMs out of 43 in a job and out of total ~1600 VMs are shown as unprotected. All VMs were backed up, including the 6 reported.

Default answer from support:

I assume that we encountered issues with the Data Collector. The Collector component is in charge of collecting the data from the Veeam B&R and Veeam EM. I suggest restarting the Collector tasks - it should restart Veeam B&R data collection and help with retrieving the data. Please follow the steps below:

We received different sql scripts in the last years to get everything in sync again, some are pretty basic, some are more complex.

My point is: why isn't it possible to fix this in some other way than asking customer regularly to run sql scripts directly on sql database? I'd expect that this is fixed in SW or can be triggered in SW. I thought for a long time that only the collection task that can be triggered in VO Web GUI is responsible for collecting data. But it seems that some collection task is running directly in sql.

[RomanK]

Hello prix,

I have checked with the support team the cases you've provided.
07002243 - a new bug was found and resolved by T2 engineers
07001050 - сollector failure issue, also resolved by T2 engineers

As far as I know, the support asking to create or create on their behalf a separate case per new issue for better tracking and transparency. I'm really sorry, that you faced multiple issues.

The support case 07007689 is about outdated information in Veeam ONE protected VM report that is still on T1. Indeed it really needs additional attention and I pinged support managers just in case. Please note, if you are not satisfied with the quality or time of the engineer's response feel free to escalate the ticket (talk to manager button within the support portal).

I completely agree that from the customer's perspective running SQL scripts may seem like a continuous and pointless task. At the same time, I'm confident our support engineers will find the cause of this issue.

Thanks

[pirx]

I'm not unsatisfied with support engineer, I guess he provides the scripts he gets from R&D. But I'm not satisfied that R&D does not fix the situation. And looking at last reply from support, I don't think it will happen soon.