Creating VSS Snapshot time for each VM takes much of time

[mateusz.sobocinski]

Let me say Hello to everyone.

The Support case which I had opened for my issue is: 01882484

So I will firstly describe my infrastructure and then my problem.

I have two Hyper-V servers which I will label for the clarify as HV1 and HV2. Additionally I have two Veeam Backup Servers. Veeam1 and Veeam2.
The Backup infrastructure is really easy. My Veeam1 which is installed on HV1 takes backups for VMs located on HV2 and then do replicas of this VMs from HV2 to HV1. For Veeam2 the thing is the same. Veeam2 is installed on HV2, takes backups of VMs located on HV1 and then do replicas from HV1 to HV2. Everything works fine. We don't have any problems.

But from the certain time we have problem with Veeam1. On such Backup Server the time for creating VSS Snapshot for each VM takes much of time. It is about 4 minutes and 41 sec only for creating the snapshot... This time is very undesirable because now my backup job last about 35 minutes not as previous about 15-20 minutes... The closing snapshot time is longer than previously and it last about 2 minutes. For the comparison I of course have my Veeam2 and there the Creating VSS snapshot time is only 41 seconds for each VM.

I tried to configure new backup job with only one VM and the result was the same. Unfortunately such issue can be observed only when Veeam1 takes backups of VMs from HV2 Server. When I Configure on Veeam1 Backup Job for VM from HV1 Server, the VSS snapshot time is OK. So I definitely can assume that the problem is related to the HV2 Server... I was restarted the VSS and HV VMM services but with no result. Finally I rebooted the whole HV2 Server at the Saturday evening and then the VSS Snapshot time was 41 seconds. But now (after couple of days) I can see that VSS snapshot time is growing. firstly it was 2 minutes, now it is 3 minutes...

Please see attached Screenshots:
Veeam1 with problem:


Veeam2 without problem:

[foggy]

This is better to be investigated with the help support, however, could you please check whether shared snapshot ('Allow processing of multiple VMs with a single volume snapshot' check box) is enabled in both jobs, to check whether this is related?

[mateusz.sobocinski]

Hello Foggy,

Thank You for Your reply. Firstly I checked both Veeams and Jobs for "Allow procesing of multiple VMs with a single volume snapshot." Yes it is checked. Both on the Veeam where backup job working fine and on the Veeam where the job have VSS problem.

Additionally I talked with support, but I get the following response:

"The snapshot creation itself looks transparent:
1. Veeam sends API call to host, requesting snapshot.
2. In your case (onhost snapshot) HV host creates shadow copy of the volume locally on the host and starts transfer immediately.
Technically Veeam is just an overseer in this setup. We cannot control snapshot creation procedure directly."

So generally I was informed from the tech support that Veeam only triggers the VSS snapshot by HV host side. Okey I know it, but the thing is still not clear for me, why after rebooting the HV host times for VSS snapshot are ok, but after a few days it starts to grow up and now such VSS snapshot takes 4 minutes for each VM...

[foggy]

Neither clear to me, probably involving MS support is required. What if you disable the ability to include several VMs into a single snapshot? Just wonder whether the type of the snapshot matters.

[mateusz.sobocinski]

I imagine that I should contact Microsoft, but I wonder that Microsoft can tell me that it is problem related between Windows and Veeam, and there I won't get any support, but I will try...

[mateusz.sobocinski]

What do You mean that type of snapshot can be a root cause of issue? Do You mean the unchecked the allow procesing multiple vms per single snapshot will switch to the different type of vss snapshot?

Of course I will test it.

[foggy]

The assumption was that in one case shared snapshot is created and in another - separate snapshot for each VM, causing some timing difference.

[mateusz.sobocinski]

I unchecked this checkbox and the result is the same VSS snapshot time approx 4 minuts 41 sec, and Veeam process only one VM per snapshot. So I stopped the backup and check the checkbox again to process more VMS atg the same time. I think I should contact Microsoft support. Maybe they will wind the cause of VSS snapshot problem on the Windows side...

From the other side. If I can ask You for the other little issue which I have.

Is it normal behavior that while starting the backup the process of building VM list takes about 8 minutes for 21 VMs in the Job? Unfortunately I can't compare it with any other backup because in my second Veeam the time for building VM list is generally the same for 29 VMs - it is about 8 minutes. But I remember when I configured such backup job and I added the couple of VMs so then the Building VM List last about 30 seconds. After Putting the 20 VMs to the single Backup Job such Building VM List time now is about 8 minutes. Additionally the CPU usage on the Hyper-V host rises while building VM list to the 50% while normaly it is at the 10% of usage. Please see attached photo:

[foggy]

Better ask support to look at what it is doing at that time in the logs. Here's a similar issue, btw.

[mkaec]

It might help to do some testing with Windows Server Backup. If you can replicate the symptoms there, then Microsoft support will not be able to claim it is a Veeam issue. If you can't replicate there, then you can go back to Veeam support and say "maybe it is a Veeam issue after all".