Frequent stuck consolidation errors?

[danswartz]

Everything is working fine with my V10 install and 1 Linux proxy. Except pretty much every daily backup run, I get a message that the Vcenter Server Appliance has a stuck snapshot, and all consolidation attempts failed. Investigation reveals that the Linux proxy still has one of the vdisks hot'added. Is this a known issue? Do you wish me to open a case?

[PetrM]

Hi Dan,

All disk attach or detach operations are performed at the level of vSphere, the only role of Veeam is to send a request.
I'd recommend to open a support ticket to investigate the root cause of the issue.

You may take a look at this KB to test HotAdd functionality manually and check if there are some issues with manual detach of disks (it's better to use some test non-production vm).

Thanks!

[danswartz]

I will open a ticket. Let me clarify something: no other guest ever seems to have this issue - only the VCSA, and it always seems to be the same disk (the 50GB one.) VCSA has 13 vdisks, and this is disk #12.

[danswartz]

Case 04071262 opened.

[foggy]

Hi Dan, thanks for opening a case. The fact that it is the same disk every time looks suspicious.

[danswartz]

That's what I thought as well. Every time the same VM, and every time the same disk. Hmmm...

[foggy]

We weren't able to reproduce it internally so far. I see that the case has been escalated, please continue working with our engineers.

[danswartz]

This may turn out to be nothing to do with veeam. I noticed several times lately, the VCSA has been afflicted with stuck CPU console messages (as well as inability to svmotion it from a local datastore to the NFS4.1 datastore.) I moved all guests off the NFS4.1 onto local storage, unmounted the NFS4.1 datastore, remounted it as NFS3, migrated everyone back, and re-ran the daily job. No errors and no stuck snapshots. By monday AM, there should have been 2 more daily and one weekly backup. If those all come back clean, I'll close the ticket. I have no idea if this NFS4.1 crap is vmware's fault, or CentOS7 (or some unfortunate interaction), and frankly I don't care at this point. The datastore is served on a private infiniband subnet, so I'm not worried about lack of security...

[foggy]

Thanks for the prompt update, much appreciated.

[danswartz]

No joy. Same stuck disk. I do note something potentially interesting. For whatever reason, vmware creates the VCSA with 13 different disks, and disk #12 is independent&non-persistent.

I read that old thread about veeam B&R and independent disks. This raises a question: if an independent disk is not to be backed up, why is it being hot-added to the linux proxy?

[danswartz]

On top of the current joy, nfs repositories seem to be wreaking havoc on my CentOS7 server. Because of how veeam is doing hot-add, I see a loopback connection from 192.168.3.34 (nfs server) to the same address, and the server starts timing out. I'm switching back to the windows proxy for now. I will close the ticket. Sigh...

[foggy]

No joy. Same stuck disk. I do note something potentially interesting. For whatever reason, vmware creates the VCSA with 13 different disks, and disk #12 is independent&non-persistent.

The strange thing is that all VCSA VM disks in your logs, including #12, are listed as persistent. If you happen to change our mind and investigate the issue further, feel free to re-open the case. Thanks!

[PetrM]

Hello!

Potentially, this behavior could be related to VC database which can be frozen during backup. You may try to test the workaround from this KB if you run VCSA on Windows VM.

There is another "How to" article which describes recommended methods of VCSA backup for two main cases: VCSA with embedded or external database.

Thanks!

[danswartz]

VCSA is the vcenter server appliance. It is a pre-build linux VM.

[danswartz]

foggy, that is truly strange. i just checked, and disk #12 is dependent, yes. i have no idea how it was the other way. hmm...

[foggy]

Dan, our R&D would like to get deeper into this - would you mind to re-open the case and provide more logs from this installation or have you completely decided to switch to Windows proxies and do not want to investigate the issue anymore?

[danswartz]

Sure. I'll re-enable the linux proxy and run a job containing only the VCSA and see.

[foggy]

Thank you, I will PM you the logs we would need.

[danswartz]

Of course, now it won't repro. I *think* I have everything set up like before...

[foggy]

Will you try to stick to Linux proxies then?