"Snapshot deadlock detected" nearly 3 days into backup

[jturnbull]

Hi,

Using the Veeam Agent for Linux (free), we have a backup that has been processing for nearly 3 days (2.5TB into a 3TB backup) which then failed with the error "Snapshot deadlock detected".

This is after many failed backups with the error "Snapshot overflow" (all several hours or days into the backup) which we seem to have overcome by altering the snapshop size.

The snapshot overflow backups deleted anything backed-up, so any restart started all over again. The latest one with "Snapshot deadlock detected" does not appear to have deleted the backup, so I'm hoping it is continuining where it left off.

Has anyone come across this and do you know if the backup will indeed continue from where it failed, or are we going to need to wait another 3 days to see if it fails?

P.S. Apologies if this breaks forum rules, I just wanted advice from anyone who has seen this before, rather than raise a support case which may be unnecessary in this instance.

Rgds,

Jim

[Mike Resseler]

Hi Jim,

Three days? I would not consider that normal. I have heard about a few cases around "snapshot deadlock detected". I believe there was an issue with AV, another issue I saw was with DRBD, and a case with a RAID 10 where it was hardware related.

So apologies accepted for breaking the forum rules , but now is the time to create a support call since it can be many things and our engineers will need to investigate the logs.


As always, please post the case ID here and the result after investigation of our engineers

Cheers
Mike

[jturnbull]

Okay will do - just a footnote before I log the case...
...I have a second server with the same config which also took a few days to backup (about 3.3TB of data), but there were no issues with this one and subsequent backups don't take more than 10-15 minutes on that server because we're only backing up changes.

[Mike Resseler]

Jim,

Still worries me a bit to be honest. When you say the same config, I assume you mean the same hardware? Or the same software. Maybe there is something that causes this to go slow. (BTW: Did you check on the succeeded server where the bottleneck is?)

[jturnbull]

Same hardware (well actually same VM config).

They are both VMs running under Hyper-V with the Agent for Linux installed in the VM.

Backup is to a SMB share.

A possibility for the bottleneck is maybe that most of the data is already compressed (e.g. jpg, mp3 etc.). The network connection is 1Gbps between the Hyper-V host and the Backup Server. Backup server is physical. However, like you I'm surprised it takes this long. We have some other VMs on an ESXi host which are smaller but don't seem to have the same kind of bottleneck (backing up to the same backup server).

Anyway, case has been raised (Case #03230734).

Rgds,

Jim

[jturnbull]

Quick investigation into the bottleneck - looks like the person who configured the SLES11 guest has not configured the Ethernet driver correctly, so we're running much slower than we should on this virtual machine.

I'll reconfigure the guest machine (out of hours) and feedback when I have re-tested.

Rgds,

Jim

[PTide]

Hi,

<...>most of the data is already compressed (e.g. jpg, mp3 etc.).

Since you've mentioned the type of data, I wonder is it a volume-level backup, or a file-level?

Thanks!