V11: Stuck hotadd disks when using linux proxies

[mkretzer]

We recently installed V11 and immediately installed new linux hotadd proxies.

For testing we did active full on a few hundret VMs. For several VMs now we got "all stuck VM snapshot consolidation attempts have failed". It seems like some of the hot added disks were not disconnected from the proxy correctly - but we see no error in the job logs for those VMs.

Is there any known issue for that in the latest release?

Case 04815996

[HannesK]

Hello Markus,
I just looked at the known bugs list for the Linux proxy and there is nothing obvious. There is one thing I found that might be related. I asked support to verify.

As we talked about it earlier... Do you have the lvm2 package installed?

Best regards,
Hannes

[mkretzer]

No we went completely lvm-less
Kernel messages look very good - the partitions are logged, but nothing more:

kernel: [186944.643539] scsi 0:0:4:0: Direct-Access VMware Virtual disk 2.0 PQ: 0 ANSI: 6
kernel: [186944.643902] sd 0:0:4:0: [sde] 125829120 512-byte logical blocks: (64.4 GB/60.0 GiB)
kernel: [186944.643931] sd 0:0:4:0: [sde] Write Protect is off
kernel: [186944.643983] sd 0:0:4:0: [sde] Cache data unavailable
kernel: [186944.644759] sd 0:0:4:0: Attached scsi generic sg5 type 0
kernel: [186944.647364] sde: sde1

Interestingly we always had some connection errors in V10 (transport closed and so on) but *never* hanging disks on the proxy - only if VC crashed (which is normal). These issues are completely gone in V11 and we now have this new issue (no errors whatsoever but hanging disks).

[HannesK]

ok, then the one thing I found is not related. let's see what support finds out.

[mkretzer]

I got an answer. The logic that should prevent the proxy itself to be backed up seems to be faulty in V11 or for linux proxies.
So the backing up of the proxy VM causes the issue. At first support thought that there is no logic to prevent this but i was sure there is and now they have confirmed that there indeed is a logic that does not seem to work.

[Gostev]

BTW this could also be an issue with the backup proxy BIOS UUID itself, rather than the Veeam logic. I've seen invalid BIOD UUIDs before a few times, so worth checking.

[mkretzer]

We were again quite paranoid setting this up, so no clones, no duplicate IDs. What exactly should we check?

[Gostev]

Just that it looks normal, and there's nothing unexpected or suspicious like all numbers being nulls

[NickKulkarni]

Hate to bump an old post but I have V11 and Linux Proxies and similar problems.
Losing connection to Linux proxy during backups ( or shortly afterwards I think) and finding LVM references to DM-0 through DM-9 with I/O errors during updates to the Linux proxies.
Wasn't until I did a recent update to Ubuntu LTS 20.04.4 that I found the DM error messages.
The first sign something was wrong is when I saw disconnection warnings from Veeam One and backup failures.
Didn't find the LVM references until doing recent maintenance updates on the proxies.
Since then I have found other references to LVM and Veeam problems in the forums however at least two are specific to Linux Proxies
vmware-vsphere-f24/linux-backup-proxies ... 69068.html
vmware-vsphere-f24/recommendation-for-l ... 65637.html

the last one of these raises some very interesting issues about LVM and AutoMount but doesn't say much about dismounting them LOL.
I am going to try redeploying all my Linux Proxies without LVM to see if that has any effect and if not opening a support ticket.
As always any suggestions etcetera gratefully appreciated.

[HannesK]

Hello,

I am going to try redeploying all my Linux Proxies without LVM

please let us know, how it goes. It seems that LVM can cause side effects, but all cases I saw so far don't give a clear picture. Setting activation = 0 in the lvm.conf helped in some cases, but I did not see that for "disconnecting proxies".

Although LVM is not a problem in general, it might make sense to recommend against LVM with HotAdd proxies.

and if not opening a support ticket.

you can also open a ticket for the current issues, because that would allow comparison of the log files and answer the question about "correlation" and "causality".

Best regards,
Hannes

[NickKulkarni]

Support ID: #02680172

Hi Hannes, Correlation and Causality LOL very good reminder.
100% of murderers have eaten Bread at some time in their lives but that doesn't necessarily mean eating bread always causes people to commit murder.

FYI we are running VMware VMs as our Linux proxy not physical machines

Seems we think alike. Already been going back and checking a few things through my internal ticket system. I originally deployed 18.04 LTS originally and had no significant problems (other than my own stupidity when deploying the last of the proxies and forgetting to complete the Local Search Domains in the IPV4 setup DUH) . Issues became readily apparent shortly after I was offered an in place upgrade to 20.04.3 LTS during a regular patch session.

So I decided to go back to clean base line, pull all the proxies out of Veeam, remove from Managed Servers and then reinstall a fresh copy of 18.04 LTS.
Found that there is no option in the server ISO (unlike the Desktop version) to install or not install LVM and patching 18.04 to current latest shows LVM2 2.02.176-4.1 Ubuntu 18.04.3 as being installed.
So based on that it looks like LVM was installed in this version and was not causing any issues. Correlation seems to be with the in place upgrade to 20.04 LTS. Wondering if doing an in place upgrade with the Veeam server management installed might not be a good thing. Perhaps it may be better to remove this before doing major revision then putting it back once it has been done.
Experimenting now with three new 18.04 proxies will update this tomorrow after backups run tonight.
cheers
Nick

[NickKulkarni]

Update: Backups failed and I lost connection last night as well.

Found a bug in my Veeam Jobs though and not sure if its related or not.

Got an advisory from VMWare about ESXi booting from SD Card and possible corruption and a setting change to mitigate. I have been bitten before by this so I was pulling the weekend shift last Saturday (can't shut down servers in the working week) to do this. I had modified ESXi config as per earlier recommendations but this last fix came too late, had just edited and gone patch then reboot when the patch failed, card came up as corrupt on reboot so that's why it failed) Second one in two years and we reboot once in a blue moon so its not that causing wear.

Anyway back to the plot. Had to swap out card and reinstall ESXi then recover the VMs. This seems to do something to Veeam and maybe the UUID in the database or something but three of my VMs in a couple of the backup jobs did not back up. Veeam reported them as unavailable and failed the backup. During that backup we lost Proxy connection. All three are on the affected server and when I edited the job today the affected VMs show as 0 GB. Deleted them and added them back today and ran successful manual backups. Still waiting for someone to get back to me on the support ticket, had an initial contact within the SLA that simply asked me a couple of questions to which I replied. That was 17:33 last night. Nothing since. Will update when I have more information.