Very slow template backup

[HendersonD]

ESXi 6.7 update 3 and Veeam 9.5 update 4b. We have two templates

• WinServer 2012 Std R2
• WinServer 2016

These templates backup very slowly at about 700 KB/s. All my other VMs backup very fast at somewhere between 250 and 700 MB/s. Has anyone else experienced this? Any idea what causes this?

[HendersonD]

I should add that our source SAN that holds the templates has a 10gig network connection
There is a 10gig network connection between our data center and the disaster recovery site
The destination SAN also has a 10gig network connection
I believe that template backup is strictly in network mode
We are seeing the 60GB disk on each template (that probably only has about 20GB worth of data) take about an hour to backup

[mkretzer]

We have the same issue but we do all other backup with NBD as well. Some templates are just backed up extremely slowly, much slower than the other NBD backups!

[Gostev]

Templates can only be backed up via NBD indeed. VDDK does not support template backups, so we have to use a workaround to pull them.

[HendersonD]

I will have to investigate further. Templates were taking a few minutes to back up, now they are taking well over an hour

[Gostev]

Check your ESXi management interface and switches first and foremost.

[mkretzer]

Gostev,

How does this workaround work? Does it use the same transfer methods/links as a normal backup would?

[Gostev]

No, since normal backup uses VDDK, which as I explained above does not support templates backup at all... we use another API instead, which was not designed for backup. But effectively, it is somewhat close to how VDDK works when backing up over NBD (data also goes through the ESXi management network, so it is quite slow on 1Gb Ethernet).

[HendersonD]

I figured out the problem. Every so often we take our templates, turn them into VMs, turn them on and update them. Once turned on we login to the VM, assign an IP address to the nic, and do Microsoft updates. Once we have the VM updated we shut it down and turn it back into a template. We did this process a short while ago and forgot to do one thing. Before shutting it down we left the nic configured with an IP address where we normally take off the IP address. You wouldn't think this would affect the ability of Veeam to backup the template but it did. Yesterday later in the day we took the IP addresses off the nics for these two templates and last night backup went well with each template taking between 4 and 8 minutes to backup.

[DGrinev]

That's interesting!
I've shared this finding with the responsible QA team. Thanks!

[DGrinev]

The QA team cannot confirm said behavior.
Can you back up two templates by a single job? One should have an IP the other shouldn't. Thanks!

[HendersonD]

I am doubting my own conclusion now. My template does not have an IP address and I am seeing varying behavior. Here are the last 3 backup jobs and what my Windows Server 2019 template did

1. Did not backup on first try with "Error: VDDK operation timed out", the second try of this job backed up the template in 3:28
2. Template backed up in 4:54
3. Template backed up in 3:28
4. Template took 30:30 to backup

Would I be better off to take my template and convert it to a VM and just leave it off? Each night it would backup as a VM and not a template. When I wanted to spin up a new server I can convert it back to a template, spin up a new VM from the template, and then convert the template back to a VM

[DGrinev]

You can do that, but I would ask the support team for assistance to find out the root cause of the issue. Thanks!

[skrause]

Do you need to backup your templates at times other than right before/after you run your windows update process?

You could just manually run the backup job while it is still a VM as part of your process.

[icetrain]

Has this issue ever been fixed?

[Gostev]

v10 can perform template backup in any transport mode except those involving direct SAN access, so:
1. Direct NFS
2. Virtual appliance (Hot Add)
3. Network (NBD)