Backup from SAN snap results in a stalled job

[dcolpitts]

I've got a strange one here... Just installed a new Alletra 6030 array (32Gbps FC) and a new StoreOnce 3660 (20Gbps LACP bond, and the VBR server is 2022 with a 40Gbps LACP NIC team). Both are running current firmware / OS. Veeam is 12.1.0.2131. And on 5 VMs (out of 30 some), the job starts to process them, and then just stalls. And it doesn't always stall at the same point. Looking at the task log for each of the 5 VMs, I keep seeing this repeated at the end of the logs:

[16.12.2023 12:12:10.455] <56> [001A] Info (3) VSSControl: KeepSnapshot started, ttl 1200 sec
[16.12.2023 12:12:10.455] <56> [001A] Info (3) VSSControl: KeepSnapshot completed, ttl 1200 sec

Looking at the log for the job, I see this repeated at the end of the log 3 or 4 times and then nothing:

[16.12.2023 11:53:03.892] <85> [001A] Info (3) Removing [CSoapConnectionKeyInfo]
[16.12.2023 11:53:03.892] <85> [001A] Info (3) HostNameOrIpAddress: 'vcenter01.fqdn'
[16.12.2023 11:53:03.892] <85> [001A] Info (3) Port: '443'
[16.12.2023 11:53:03.892] <85> [001A] Info (3) UserName: 'Veeam@vsphere.local'
[16.12.2023 11:53:03.892] <85> [001A] Info (3) UseProxy: 'False'
[16.12.2023 11:53:03.892] <85> [001A] Info (3) ProxyHostNameOrIpAddress: ''
[16.12.2023 11:53:03.892] <85> [001A] Info (3) ProxyPort: '0'
[16.12.2023 11:53:03.892] <85> [001A] Info (3) ManagedThreadId: '##'
[16.12.2023 11:53:03.892] <85> [001A] Info (3) from cache. Reason: Expired
[16.12.2023 11:53:03.892] <85> [001A] Info (3) [Soap] Logout from "https://vcenter01.fqdn:443/sdk"

These 5 VMs range in size from 68GB to 1.92TB in size... And it is consistently these 5... This last job it managed to read 538.5GB before it hung... Other jobs are more and some are less. The repo is a Catalyst store with Immutability enabled (set to 21 days, while the job is set to 14 days) and fixed block chunking is enabled.

And backing up these same VMs in a different job to a pair of RDX drives, works perfectly fine...

I haven't opened a ticket yet (it will be Monday before I can do that - trying to pack up to get a flight out of here...)

That said - has anyone seen anything like this?

dcc

[foggy]

Hi Dean, support case is definitely required for an investigation. Please open one when you have a chance. Thanks!

[shineknoxTN]

Having the same stall.

post508901.html#p508901

[shineknoxTN]

Are you using the setting "Limit Processed VM count per storage snapshot"? If so check out this post post508901.html#p508901