100% CPU Unresponsive Server Synthetic Full

[mihlfeld-it]

First off, this forum software is hot garbage. I spend over a half hour meticulously writing up a well detailed post (after having already logged into the forum) only to click "SUBMIT", the forum take me to a log in screen--again--and then for it to take me to a new post with all my typed up details lost. Clicking back did nothing to go back to what I had typed up. Needless to say, I'm extremely agitated and I will not be documenting things to the detail I did before. I will provide details as asked / requested.

I already have a case open that is going nowhere, ID# 03767129. Bottom line is we have one of our backup jobs that will peg our repository server at 100% CPU usage during the tail end of synthetic full creation and the only way to get the server back is to cold boot the server. It is a 8 VCPU / 8GB RAM repository server setup for 3 concurrent repository tasks only. There is no backup proxy enabled on this server. The only other task running during this time on this server is a 24/7 backup job of our SQL server that is set to take transaction logs every 15 minutes from our main production SQL database. Resource usage on this server is low / nominal during backups, etc. The backup repository is a ~55TB Synology NAS that's interfaced via Microsoft iSCSI Initiator from within the machine that Veeam B&R is running. Veeam is on the very latest patched version, and the operating system is Windows 2016 Standard patched up to the very latest version from Microsoft Update. The NAS is setup with ReFS and 64KB block size.

I can provide any other details as needed. Oh, it basically has nothing else installed on this server except what Veeam installed / needed as prerequisites. It was installed fresh just for this purpose a few weeks ago and there is no AV software on it. We also do not use any encryption on the backups. Thanks for any insight on what to dig into and look at. I'll provide any other details that are needed.

[foggy]

Just to clarify, the repository server is a separate VM (not the one where Veeam B&R server itself is installed), to which NAS is presented via iSCSI, is that correct?

[mihlfeld-it]

Actually, no. The main server where Veeam B&R is installed is also the repository server. Can this cause issues? We did remove it as a backup proxy and only have that on one other server (our only other server that has any Veeam function on it) We are a fairly small shop. We are only backing up around 30 VMs, though about 4 of those are north of 1TB (a couple file servers and a couple SQL databases servers in the 1.5 - 2.5TB range as far as size of what is being backed up). Our issue with the synthetic full is only with our largest VM backup, which is around a 2.5TB or so virtual file server.

EDIT: As an example of a synthetic full that had no issues (just started it about 20 minutes ago), I backed up one of our two main file servers that hasn't ever had this issue. It's ~1.8TB virtual machine. It took about 3m52s to backup and transfer 2.4GB read / 1.4GB transferred as an incremental from what was backed up last night, then it took 8m02s to create a new synthetic full. During this entire window it never used more than 15% of the total CPU of the server (as viewed from the monitor section of the ESXi console that hosts this virtual machine running Veeam B&R). Also for the record, the ESXi host that this Veeam B&R server runs on is only currently running this Veeam B&R server and no other guest machines.

[wishr]

I'd like to add that as far as I see from the case history there is an ongoing conversation. Regardless of this, I had a talk to support management and asked them to keep an eye on the case flow. Please keep working closely with our engineers to identify the root cause.

Regarding your setup, it's not recommended to have a repository at the same machine as B&R server as it may lead to a situation when all the backup data is lost because it violates 3-2-1 rule. However, it should not cause issues similar to what you are experiencing.

[foggy]

Have you tried adding resources to this VM? According to the system requirements, you seem to be close to (if not above) the limits at least in terms of RAM, due to the fact you're running several components on a single server. Depending on the amount of concurrent tasks CPU might also be capped.

[mihlfeld-it]

We are only a few weeks into our deployment of our Veeam solution deployment. While I will feel really exposed to lose all of our backups and re-do our setup, now is the time if we're going to do it. I guess I'm not familiar with this 3-2-1 rule, so I will do some more research to see best practice of how this needs to be done. I never ran across documents in my time working to put together our environment that indicated I should be splitting out where we have our repository attached from where we run our main Veeam B&R server, so I clearly need to research more.

[mihlfeld-it]

Regarding bolstering up our resources, I only have a total of 12GB RAM available max on the ESXi host and a single 6 core / 12 thread processor in it. I haven't tried yet to allocate the last 4GB RAM from the host, but I do need to keep some for the host. I guess I need to consider that I need to buy some more hardware resources for this server if 8GB RAM is not enough. We tried to just use / re-purpose an older Dell PowerEdge R610 that we had available to us.

[wishr]

Thank you for the info.

Yes, it might be related to the hardware resources, especially keeping in mind there are several components running at the same place. Our support engineers should be able to confirm it or blow out of the water.

Also, there is a nice tool for sizing a backup deployment.

Speaking about 3-2-1 rule, that thread is a good starting point.

Hope it helps!

[foggy]

Running all-in-one installation is quite popular, and is definitely not an issue provided you spec the server according to the system requirements.

[mihlfeld-it]

Given the specs and age of the Dell PowerEdge R610 server that's hosting all of this, I am just going to rule that out of the equation and move to a new host ESXi server. I was approved to get something newer and far more capable hardware-wise, so I'm going to put any pursuits of issues on hold until we get that new server in place and operational.