Veeam 12 Backup from replicated Nimble SAN rep.

[shineknoxTN]

We are running Veeam 12 Backup and replication server (Windows 2019) on a physical server, DL360 Gen10. We have iSCIS set up with two nics active/active, ipv4 only. We have a single network connection for the server, which is configured in dualstack (ipv4/ipv6). The backup repository we are backing up to is ReFS 64k.

We are replicating a san snap from an HP Nimble HF40 to an HP Alletra 5000. We are backing up from the replicated san snap on the HP Alletra 5000 to a backup repository on an HP Nimble HF20C. Compute is not being touched. The testing went good. Then we started adding some production servers and things went south. The jobs will now hang sporcadically on virtual machines. If we try the job it will hang again. In order for us to run the job successfully, we have to reboot the physical backup server. We have looked through the task logs on the virtual machine(s) that hang and it looks like it gets in a VSS loop. Here is what is repeatd over and over again in the task logs, "VSSControl: KeepSnapshot completed, ttl 1200 sec" and every 30 minutes it will say, "[AP] Waiting for completion. Id: [0x49ae], Time: 00:30:00.0192329", then continue back with "VSSControl: KeepSnapshot completed, ttl 1200 sec", until we kill the job. If we look in the agent logs for the virtual machine, it says "There was no finalized server sessions during the last 60 minutes." Today, we disabled application aware processing and retried the job, but it also failed. The VSS loop obviously was not there, but still failed.

a reboot will fix the issue.

Case #07081597

[PetrM]

Hello and Welcome to Veeam R&D Forums!

It does not look like a known issue, please continue working with our support team to sort it out.

Thanks!

[shineknoxTN]

Well looks like someone else is having the same problem.
post506513.html?hilit=alletra%20san%20snap#p506513

[shineknoxTN]

It looks like the issue has been located. There is a bug with Veeam and Nimble, with using the setting "Limit Processed VM count per storage snapshot". Veeam and Nimble have issues when there are numerous snaps within a single job. We had our jobs set at 10 VMs. In the log you can see where it does a group of 10 successfully and then fails when it starts a new group. It was very sporadic in which group it would fail in. We unchecked the setting to not use it and our backups have been consistently successful.

NOTE:
When we first started rolling this out it was running great. Our issues started to pop up once our vm count increased in the jobs. This caused the jobs to have more group of 10 snaps per job. Veeam support should have caught this. Hopefully Veeam and Nimble will provide a fix.