Since VC 7 Update - VC "hang" caused by Veeam Backups

[mkretzer]

Hello,

case 04590954

This might be a pure VMware issue - but i wonder if anyone else in the forum has this:

We upgraded from VC 6.7 to VC 7u1c end of december and now 7u1d. Since the upgrade about every week (but sometimes every 2 days) our VCenter process "hangs" during the backup. VCenter is still reachable via SSH and there is no problem with the VC VM itself or any other VMs in the environment. In the service status the "profile driven storage" service has issues. The problem solves itself, most of the time after 30 minutes, sometimes after 60 minutes. We also can cleanly reboot VC via SSH without issues if we need to have it fixed faster. Veeam gets "Unhandled exception was thrown during licensing process " in the backups, all retry after the problem have no issue.

This *only* happens in the evening when a backup of about ~2800 of our VMs is running (this takes about 6 1/2 hours). All these backups use NBD. It never happened in the night when we backup the rest of our machines via hot-add (~1000 VMs).

The only thing we changed was the VC version. VMware has ruled out most external problems (host problems of the hosts VC was running on and so on). We increased RAM allocation for different VC processes. We reduced backup concurrency. No luck so far.

Markus

[segfault]

We are also up to now 7U1d and are seeing the same thing. vCenter hangs and the admin page for vCenter points to the profile driven storage service.

We don't really use NBD too much, most is direct SAN backups.

I suspect a vCenter bug at this point. It seemed to start once we bumped to 7U1b and 7U1d made it worse (or so it seems).

[mkretzer]

Hello segfault,

thank you very much - so we are not alone!

Would you be willing to share your case number? I will sent you mine via private message!

Markus

[mkretzer]

Just to give an update: Since we switches from NBD to hotadd the problem got much better - but today we had the first crash even with hotadd. VMware wants us to increase logging significantly and we will try to do this.

Do any more VC 7U1d Users besides segfault have the same issue?

[mkretzer]

We now finally have the analysis from VMware after months of troubleshooting:

The analysis of our ENG department is concluded and boils down to the amount of sessions VEEAM is creating on average (90 per minute) further these sessions are not properly reused and just new sessions created. because of that we have a big amount of user handling and also group findings and permission handling which therefore ends up creating deadlocks on the system. This all than results ultimately on a non responsive vCenter

In other words: Veeam crashes the VC when doing backup of many VMs in paralell. New case: 04779922

The only other way other than fixing this in Veeam is to upgrade to 7.0U2 and hope the optimisations VMware did help us here...

[emanueleroserba]

VMware is retiring 7.0U2 because of a bug that could lead to ESXi boot-bank corruption, definitely better waiting for 7.0.3 or update the vCenter but not the hosts.

[mkretzer]

They told me that they just released a hotfix to fix the biggest issues (VC-7.0u2a)... Also, this is a VCenter problem only...

[dtang]

We now finally have the analysis from VMware after months of troubleshooting:

The analysis of our ENG department is concluded and boils down to the amount of sessions VEEAM is creating on average (90 per minute) further these sessions are not properly reused and just new sessions created. because of that we have a big amount of user handling and also group findings and permission handling which therefore ends up creating deadlocks on the system. This all than results ultimately on a non responsive vCenter

In other words: Veeam crashes the VC when doing backup of many VMs in paralell. New case: 04779922

The only other way other than fixing this in Veeam is to upgrade to 7.0U2 and hope the optimisations VMware did help us here...

How many sessions/requests are considered safe from their side and in what interval? The 90 requests per minute translates to roughly 1.5 requests per second, or did I understand wrong?

[tsightler]

Yeah, it didn't seem that high to me honestly, especially for a busy backup server with a lot of jobs, and even more so, it seems to mean that any authenticated user can apparently launch a denial of service against vCenter?

[mkretzer]

Yes! That is what they say. There were some changes in VC 7 so that a user can cause database deadlocks.

We backup about 2800 VMs in 4-5 hours using NBD unsing 3 proxies and 48 concurrent threads. I don't see why this is an issue.

[athome]

Hi!

We observe something similar to this on vcenter 7 build 16749670, but for us to upgrade to vcenter U2 is not an option(not supported by simplivity)
But I did notice that there has been released a later build of VBR than we are running, we are running 11.0.0.837 P20210401.
is there anything in the newer build that addressed this issue? or is upgrading vcenter only option?

[Mildur]

You will find changes for this small patches on this KB:
https://www.veeam.com/kb4126