Host-based backup of VMware vSphere VMs.
Post Reply
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Very slow template backup

Post by HendersonD »

ESXi 6.7 update 3 and Veeam 9.5 update 4b. We have two templates
  • WinServer 2012 Std R2
  • WinServer 2016
These templates backup very slowly at about 700 KB/s. All my other VMs backup very fast at somewhere between 250 and 700 MB/s. Has anyone else experienced this? Any idea what causes this?
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Very slow template backup

Post by HendersonD »

I should add that our source SAN that holds the templates has a 10gig network connection
There is a 10gig network connection between our data center and the disaster recovery site
The destination SAN also has a 10gig network connection
I believe that template backup is strictly in network mode
We are seeing the 60GB disk on each template (that probably only has about 20GB worth of data) take about an hour to backup
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Very slow template backup

Post by mkretzer »

We have the same issue but we do all other backup with NBD as well. Some templates are just backed up extremely slowly, much slower than the other NBD backups!
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow template backup

Post by Gostev » 1 person likes this post

Templates can only be backed up via NBD indeed. VDDK does not support template backups, so we have to use a workaround to pull them.
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Very slow template backup

Post by HendersonD »

I will have to investigate further. Templates were taking a few minutes to back up, now they are taking well over an hour
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow template backup

Post by Gostev »

Check your ESXi management interface and switches first and foremost.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Very slow template backup

Post by mkretzer »

Gostev,

How does this workaround work? Does it use the same transfer methods/links as a normal backup would?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow template backup

Post by Gostev »

No, since normal backup uses VDDK, which as I explained above does not support templates backup at all... we use another API instead, which was not designed for backup. But effectively, it is somewhat close to how VDDK works when backing up over NBD (data also goes through the ESXi management network, so it is quite slow on 1Gb Ethernet).
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Very slow template backup

Post by HendersonD » 2 people like this post

I figured out the problem. Every so often we take our templates, turn them into VMs, turn them on and update them. Once turned on we login to the VM, assign an IP address to the nic, and do Microsoft updates. Once we have the VM updated we shut it down and turn it back into a template. We did this process a short while ago and forgot to do one thing. Before shutting it down we left the nic configured with an IP address where we normally take off the IP address. You wouldn't think this would affect the ability of Veeam to backup the template but it did. Yesterday later in the day we took the IP addresses off the nics for these two templates and last night backup went well with each template taking between 4 and 8 minutes to backup.
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Very slow template backup

Post by DGrinev »

That's interesting!
I've shared this finding with the responsible QA team. Thanks!
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Very slow template backup

Post by DGrinev »

The QA team cannot confirm said behavior.
Can you back up two templates by a single job? One should have an IP the other shouldn't. Thanks!
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: Very slow template backup

Post by HendersonD »

I am doubting my own conclusion now. My template does not have an IP address and I am seeing varying behavior. Here are the last 3 backup jobs and what my Windows Server 2019 template did
  1. Did not backup on first try with "Error: VDDK operation timed out", the second try of this job backed up the template in 3:28
  2. Template backed up in 4:54
  3. Template backed up in 3:28
  4. Template took 30:30 to backup
Would I be better off to take my template and convert it to a VM and just leave it off? Each night it would backup as a VM and not a template. When I wanted to spin up a new server I can convert it back to a template, spin up a new VM from the template, and then convert the template back to a VM
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Very slow template backup

Post by DGrinev »

You can do that, but I would ask the support team for assistance to find out the root cause of the issue. Thanks!
skrause
Veteran
Posts: 487
Liked: 105 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Very slow template backup

Post by skrause »

Do you need to backup your templates at times other than right before/after you run your windows update process?

You could just manually run the backup job while it is still a VM as part of your process.
Steve Krause
Veeam Certified Architect
icetrain
Influencer
Posts: 15
Liked: 2 times
Joined: Sep 05, 2017 9:02 am
Full Name: ECE
Contact:

Re: Very slow template backup

Post by icetrain »

Has this issue ever been fixed?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow template backup

Post by Gostev »

v10 can perform template backup in any transport mode except those involving direct SAN access, so:
1. Direct NFS
2. Virtual appliance (Hot Add)
3. Network (NBD)
Post Reply

Who is online

Users browsing this forum: No registered users and 66 guests