Large traffic spike every day at 10am

[galbitz]

We are having an odd issue post veeam 12 upgrade. I am curious if anyone else is seeing this. every day almost exactly at 10am we see a large amount of traffic from all sites targeting the veeam server. The sources are the proxy servers and the immutable repos. The only thing I am aware of scheduled at that time is the config backup job. I moved that to 3am and it still occurs each day. We don't have backup jobs or offload jobs occurring at any site except for one at that time. We do have veeam one but we have no reports scheduled around that time and the data collector is scheduled for 6am.

It is causing problems at one site because the traffic is impacting production. This is occurring at all sites at the same time, but it lasts longer at our larger sites.


the proxy's traffic are source port 2500-2501Tcp to the veeam server
the repo's traffic is source port 2506 TCP to the veeam server.

There are no active jobs at most sites, one site does have a wasabi offload running. i confirmed its using the local proxy to that site as the gateway.

ticket# 05910012

[Mildur]

Hi Grant

Thanks for the case number.
Could you also upload a log collection from your vbr server to the case?
That will be the first thing our support will ask for.
https://www.veeam.com/kb1832

Best,
Fabian

[galbitz]

hi fabian i attached the logs a few moments after opening the case.

[galbitz]

Just an update we found the cause of the traffic but still not entirely sure why this just started with the upgrade or how to fully resolve it.

The veeamone agent is requesting logs for all infrastructure components at 10am each day. This may be a veeamone issue at this point and this might need to be moved to the other forum.

I am waiting on a review from support, but the loganalyzer log in the veeamone agent folder on the veeam server shows this activity. Our data collection schedule is 6am each day, not sure why this would be occurring. we only have 4 reports and they run daily at 1:30pm. We have no report for restores, just generic infrastructure health reports.

The other thing that is interesting is I looked at the resulting files that are generated from this log pull. it is a csv file that shows restored files on one of our vm servers. We have no such report and never did. We realized today we did not reboot the veeam server after the upgrade, its possible it was some random thing that occurred and we are hoping a reboot fixes it. For tomorrow we have veeamone shut off and the agent disabled on the veeam server to at least validate we don't encounter the traffic problem.

the general query its running seems to be:

01.03.23 10:00:12 [INFO ] 6260 [__50] LogAnalyzer.Jobs.TransferLogs: -->CollectLocalLogs: JobConfig:AnalysisJobConfig OnMaster period: 2023-02-28 15:00:01Z(SrcOffset:-5:00:00 - 2023-03-01 15:00:07Z(SrcOffset:-5:00:00 timewindow: N/A - N/A ServerTypes:All
01.03.23 10:00:12 [INFO ] 6260 [__50] LogAnalyzer.LogCollector.LogsPack: Created temp directory 'redacted\AppData\Local\Temp\VID\31' for LogsPack
01.03.23 10:00:12 [INFO ] 6260 [__50] LogAnalyzer.LogCollector.CollectLogs: Downloading Veeam Backup logs from date: [2023-02-28 10:00:01Z] to [C:\Users\redacted\AppData\Local\Temp\VID\31]



the resulting csv ends up in: AppData\Local\Temp\VID\31\VeeamSupportLogs\2023-03-01T100042_VeeamBackupLogs\veeamservername\Backup\Audit\2023\2\27
the file is Restore-2023-02-27-17-50-38Z-veeamservername-veeamadminusername-1.csv
the csv contains a oneliner about a file that was restored. the headers for the file are:

Time, User, SID, Operation, Result, Object, Options (RestoreMode; RestoreUser)


just posting it incase anyone can ID what might cause that. we upgraded on the 28th to 12, that restore was performed on the 27th and was likely never torn down properly.

[Mildur]

Hi Grant

Thanks for the update.
I moved your topic to Veeam One forum.

Please let us know about the outcome of the review.

Best,
Fabian

[RomanK]

Hello Grant,

LogAnalyzer is the Veeam Intelligent Diagnostics feature. Veeam ONE Agent collects and sends logs for known issues analysis.

I will notify support, that they need to transfer this case to the proper team.

Thanks

[galbitz]

We were able to resolve this with support. Did something change going to v12 with this? It wasn't a noticeable problem until the upgrade.

[RomanK]

Hello Grant,

No changes were made to the subject in v12. However, we are planning some improvements regarding the VID data collection for future versions.

Thanks