Snapshot removal issues of a large VM

[mdi02]

Hello foggy,
Thank you for your reply.
The Veeam Support asked us to deactivate the VMQ but we are not using Broadcom NetXtreme 1 gigabit adapter.
Through HyperV Manager I have deactivated the VMQ for one VM and run a new backup but same issue, the VM lost the connectivity.

I have tried to run the following command in the Guest VM on PowerShell but it did not return any result : Get-NetAdapterVmq (also tried with -Name "Lan_Prod")

Could you please tell me more on the faster storage option ?

Thank you for your help.

[soncscy]

Alexander means that basically this is happening because the storage your Oracle vm resides on is not fast enough too handle snapshot (creation/consolidation) quickly. The end result is that the VM loses connectivity during the creation/consolidation process.

For HyperV, it's a little different though and you might ask on the Hyperv forums, but I'm assuming it gets similar issues where the host just cannot handle this.

If you cannot move the VM to faster storage you might just consider doing Agent Based Backups (Veeam Agent for Windows).

[mdi02]

Hi,
Thanks for your reply.

I have double checked and yesterday we had a backup of one VM. It started at 01:00am and finished at 03:22am. The VM lost the connection at 01:06am. At 01:06am we had the following event in the Windows Event Viewer on the VM : Miniport NIC 'Microsoft Hyper-V Network Adapter' connected.
I am not in charge of Veeam or Hyper-V in my company but if my understanding is correct the issue discussed in this topic is that the VM lost the connection during the snapshot removal at the end of the backup. Looking at the times above and the time the network issue occured, it seems that this is not happening during the snapshot removal. Could you please give me your opinion guys ?

I will check with our system engineer regarding the possibility to consider the Agent Based Backups and the faster storage option.

[foggy]

Right, judging on the timestamps you're indeed seeing a different issue than the one discussed in this thread.

[mdi02]

I just manually created a checkpoint for one VM from Hyper-V Manager and it seems that the connectivity is lost at this exact moment so it is happenning during the snapshot creation.

[foggy]

...and without Veeam B&R in the picture.

[collinp]

These errors are occurring in the SQL logs during the Veeam backup. We didn't see these issues when the VM's lived in Hyper-v but we do with VMWare. Are there some easy workarounds? I can think of purchasing faster SSD source disk, adjusting app and sql timeouts or offloading the snapshot process to the storage.

Error: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The semaphore timeout period has expired.).

[foggy]

Hi Collin, do you see those during snapshot creation or snapshot commit? Are you able to reproduce it without Veeam B&R in presence - just by creating/committing the snapshot manually?

[collinp]

I don't think it would be necessary to manually commit a vmware snapshot because I believe the source of the problem is with VMWare. I wanted to find rather how other users in this forum are able to resolve this vmware issue associated with VMWare snapshots taking the Virtual machine offline for extended periods.

The disk consolidation happened at 12:36:49am and the job finished at 12:37:21AM. The error happened right around 12:37am. So it seems to me either the application or SQL is not able to withstand a 40 second pause in I/O during the consolidation process.

[foggy]

Measures you've mentioned are among the correct ones - I'm merging your post into an existing thread that also contains other hints allowing to eliminate similar issues.

[collinp]

I think I may be running into this issue: https://kb.vmware.com/s/article/2010953

I am going to switch from hot add to NBD mode tonight and see how it goes. I am currently seeing 3 minutes between vmware snapshot removal and vmware disk consolidation. The app loses its connection to the database if that interval goes over 1 minute.

[collinp]

This fixed it for me. I was using NFS 3. https://kb.vmware.com/s/article/2010953