Console does not show job details for all tasks anymore

[pirx]

#07133672 and 07160609

After update to Veeam 12.x I notice the job details in the 24h and history view sometimes are missing. It can be that there are no details at all (see screenshot) or some tasks are missing. Opening the job view multiple times sometimes helps, in other cases individual tasks appear slowly after 10-20sec in the list. But most of the time, missing tasks are just missing. That is an issue if you have errors that are shown in the job overview on right side and can't check details of the task as it is missing left.

The example in the first 2 screenshots was an initial backup of a small location, it took multiple retries to display the details.





The offload job always just one task, no matter how often I open it.

[Mildur]

Hello Pirx

The case was just opened today.
Let's give our support team a chance to analyze the logs, no need to post it in the forum 30min after case creation .

As I can see in the case, support suspects performance of the SQL server.

Best,
Fabian

[kapple]

We have been struggling with this for some time, and have had two tickets open with Veeam support. We are currently running Veeam 12.0.0.1420 P20230718.

When opening the job statistics from the past 24 hours:
1) This list of running or completed jobs takes up to a minute to load.
2) The list of VMs that are part of any backup job take up to 10 minutes to populate the lower left pane of the B&R console, as shown in your first screen shot. Support thought this was a resource contention issue, but it occurs whether backups are running or not.

Case #02767882 has been closed. This case was originally created for a different issue, but we reported this B&R job statistics reporting latency issue when working on this support request. There were issues with the database, and we were given a fix for them. Even though the issue briefly improved before the support request had been closed, it has reared its head again.

Case #07117914 is currently open. We had been planning for some time to migrate our physical Veeam servers to new hardware, upgrading from Win2012R2 to Win2022. The fix for this issue is to complete this migration, which we have not yet completed.

Let us know how this is resolved for your Veeam environment.

[pirx]

Veeam support points to a resource/performance problem on SQL server. They also show errors that db is not available etc. But we did not have those issues before latest updates. And the console issues happen also when nearly nothing is running during daytime. The massive load is during night, that's also when I see CPU spikes on slq server. But not during daytime.

I had a similar case years ago where we spent months to chase down an issue. And it was the same, support pointed to sql performance, we doubled cpu/ram no change. Then they said it's disk io. We moved server to all flash. After months it were issues with indexes and not so optimal stored procedures. Currentl sql server has 8 vCPUs, 96 GB vRAM and run on all flash / PCIe vSAN that is able to deliver nearly a million of IOPS.

If this is not enough to show information in console while less then 5 jobs are running, something is fundamentally wrong in VBR.

[pirx]

update: I noticed that we have low resource events on VBR server itself. When this happens suddenly memory usage spikes to 100%. Surprisingly this does not happen when the backup server is under heavy load during night with backup/copy/offloads, it happens most of the time during office hours. Veeam.Backup.Manager.exe then uses 28 - 32 GB memory. I'm not sure if this is expected. The low resource events start to appear one day after Veeam 12.1.1 update. But I've to admin that I've reduced vRAM for VBR VM a while before that as Veeam One reported VBR VM as massively oversized. I then run the size estimator and resized to 16 GB (we have proxies and dedicated gateways in every location, overall 30-40 systems).


Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: Veeam.Backup.Manager.exe (12136) consumed 27867426816 bytes, Veeam.Backup.UIServer.exe (9300) consumed 1361203200 bytes, and Veeam.Backup.Manager.exe (5304) consumed 1239736320 bytes.

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: Veeam.Backup.Manager.exe (26784) consumed 28095815680 bytes, Veeam.Backup.UIServer.exe (9300) consumed 1229914112 bytes, and Veeam.Backup.Service.exe (5848) consumed 924766208 bytes.

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: Veeam.Backup.Manager.exe (26588) consumed 32373587968 bytes, Veeam.Backup.Manager.exe (21524) consumed 1028554752 bytes, and Veeam.Backup.RestAPIService.exe (11124) consumed 822685696 bytes.

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: Veeam.Backup.Manager.exe (29776) consumed 30251900928 bytes, Veeam.Backup.Service.exe (35432) consumed 1013354496 bytes, and Veeam.Backup.RestAPIService.exe (6972) consumed 832917504 bytes.

[Mildur]

Hello Pirx

I'm currently reviewing your case and will discuss it with QA as well.

Best,
Fabian

[pirx]

I'm not very happy with the support I'm currently getting. Basically I've to drive investigation and I've always to ask how to proceed.

[tyler.jurgens]

@pirx - I'm sure you've set this already, but if the SQL server is virtualized, check what controller is used on your VM. I found *significant* performance improvements changing the SCSI controller from LSI Logic SAS to a VMware Paravirtual.


I also ensured my Veeam DB was on a disk on its own SCSI controller, and the temp DBs on a disk on a different SCSI controller.

Essentially we were seeing so many queries that couldn't complete (perhaps improperly optimized Veeam queries) that our disk IO was through the roof, and even though our storage was great, it didn't matter and performance suffered. Switching the controllers to VMware Paravirtual made a HUGE difference, and after a few hours on the new controllers, the IO load dropped down to reasonable levels (the queries were able to complete and we were in good shape again). If you do switch, make sure you add a new VMware Paravirtual controller first to ensure your OS has the drivers, then you can change the other controllers after the fact.

As said, you've probably already done that, but I thought I'd throw that out there just in case.

[pirx]

I think we nailed it down to a memory issue on VBR itself. Still wondering why this sudden spike in memory usage happens. Lets wait if support can find out something or if this is just how it works.

[Mildur]

Hi @pirx

The case will escalated to our QA team. I arranged the escalation a few minutes ago.
Our RnD teams will take a look at the logs and information you have provided. I will keep an eye on the case.

Best,
Fabian

[pirx]

I still don't know why console is not showing detail sometimes, but it seems that I'm just generating too many or too large log bundles...

Code: Select all

Thank you for the provided logs. I have found the process that caused the spike, it was a log export that was started at 11:21 on 06.03 and ran until the system ran out of memory:

[06.03.2024 11:32:26.173]    <17>   Error (3)    Failed to zip folder 'D:\Log and Cases\2024-03-06T112157_VeeamBackupLogs' to file 'D:\Log and Cases\2024-03-06T112157_VeeamBackupLogs.zip' (System.Exception)
[06.03.2024 11:32:26.173]    <17>   Error (3)       at Veeam.Backup.Common.SMultiFileArchiver.ArchiveFolder(String folderForArchivePath, String zipFilePath, Boolean deleteUnpackedFiles, Int32 numberOfThreads)
[06.03.2024 11:32:26.173]    <17>   Error (3)       at Veeam.Backup.Core.CLogsArchiver.TryCompressLogsSafe()
[06.03.2024 11:32:26.173]    <17>   Error (3)    Exception of type 'System.OutOfMemoryException' was thrown. (System.OutOfMemoryException)

[Mildur]

We have a known issue with a new compression logic introduced in v12.1 for log collection.
This symptoms you have descripted to us may come from that known issue.

Our support engineer shared a reg key with you. The reg key will switch back the log collection to the "old compression logic". We want to make sure that you are not affected by this known issue before continuing with RnD.

Best,
Fabian

[pirx]

Time passes.... I created 07189398 but didn't get much feedback in the last 6 weeks. In the end the case was now auto closed as I did not respond to the auto reminders. I didn't even know that I must provide anything.

[Mildur]

Hi Ralf

I see an internal comment approximately 1 hour before the case was closed. The internal comment talks about what todo next in your case. So I believe it was a misconfiguration (wrong status set by the support engineer). I will reach out to support management.

Best,
Fabian

[Mildur]

Hi Ralf

I checked the case again. Additionally to discuss the case with RnD, our support engineer ask you to answer the following question on April 12th:
"I wanted to know how they concluded of missing indexes probably."

But we never got your answer for it. Unfortunately they case was set to "waiting for customers answer" instead of "doing research with RnD".
Please open a new case and reference to the old case number. The new support engineer will be able to see the instructions on what todo next from the old case 07189398.

Best,
Fabian