Strange issue maybe storeOnce memory leak

[StevenMeier]

Hi All,

I have a strange issue which is proving difficult to track down.

we have two HPE Apollo 4510 Gen 10 Servers.....sole role in Life is Veeam repository and proxy Servers and gateway boxes to TWO new Hpe storeOnce 5250 Servers.

1 is located at each Data center.

Running Veeam 9.5 update 4a (just updated to 4a from 4)

Both run windows Server 2019, same build levels and updates for windows , same FW and HPE drives installed.

One box hangs and freezes only during the backups. This is Not all the time but for example its run fine for a month and then last night and night before it hangs.

Responds to ping , can see login screen but no keyboard reposnse. No hardware errors I in HPE ilo for anything.

Cannot RDP or connect to server shares.

Does this sound systematic of the Storeonce memory Leak?

I have to power off the Server and back on. This server does more work than the other and the other one I have only had to power off once in its life so far.

There are no errors in the windows event logs other than when powered off.

The veeam mgmt Server is running as a virtual so is separate to these Servers. obviously this box has no issues.

At a bit off a loss for root cause.

I updated to 4a purely based on the storeonce memory leak.

I have not logged a job as I cannot be sure Veeam is the issue but it only happens when backups are running at night.

[Andreas Neufert]

It is hard to guess what the issues is, but it do not look like any StoreOnce related issue.

My suggestion is to do a Memory check (usually this can be performed by the bios or by boot CDs). If memory is OK, update all firmwares of the server and windows.
You can install Veeam ONE (evalution licenses) on the other server an add this server as Veeam Backup environment. Veeam ONE monitors then CPU and RAM usage and you can see when the server goes offline and what RAM cosumption it had at this time.

Open a ticket with HPE as well regarding the Server.

[nitramd]

Steven,

Have you checked the problematic Windows server Event Viewer logs? Usually, there's some sort of indication of an issue although it may not be intelligible.

[StevenMeier]

Hardware has been updated with the latest firmware and software as of April 2019.
Both Servers have 128 GB of ram and 2 x 12 core, and 2 x 10 GB nic's.

I logged a job with HPE and they can not find no any Hardware faults

Windows event logs are empty except for when I power off the Server

[Andreas Neufert]

OK... so what did the montioring said arround CPU/RAM consumption when it freezes? (Please see my suggestion).