Discussions specific to the VMware vSphere hypervisor
Post Reply
HendersonD
Expert
Posts: 125
Liked: 7 times
Joined: Jul 23, 2011 12:35 am

Very slow template backup

Post by HendersonD » Dec 04, 2019 1:20 am

ESXi 6.7 update 3 and Veeam 9.5 update 4b. We have two templates
  • WinServer 2012 Std R2
  • WinServer 2016
These templates backup very slowly at about 700 KB/s. All my other VMs backup very fast at somewhere between 250 and 700 MB/s. Has anyone else experienced this? Any idea what causes this?

HendersonD
Expert
Posts: 125
Liked: 7 times
Joined: Jul 23, 2011 12:35 am

Re: Very slow template backup

Post by HendersonD » Dec 04, 2019 1:32 am

I should add that our source SAN that holds the templates has a 10gig network connection
There is a 10gig network connection between our data center and the disaster recovery site
The destination SAN also has a 10gig network connection
I believe that template backup is strictly in network mode
We are seeing the 60GB disk on each template (that probably only has about 20GB worth of data) take about an hour to backup

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Very slow template backup

Post by mkretzer » Dec 04, 2019 7:47 am

We have the same issue but we do all other backup with NBD as well. Some templates are just backed up extremely slowly, much slower than the other NBD backups!

Gostev
SVP, Product Management
Posts: 25141
Liked: 3692 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow template backup

Post by Gostev » Dec 04, 2019 2:37 pm 1 person likes this post

Templates can only be backed up via NBD indeed. VDDK does not support template backups, so we have to use a workaround to pull them.

HendersonD
Expert
Posts: 125
Liked: 7 times
Joined: Jul 23, 2011 12:35 am

Re: Very slow template backup

Post by HendersonD » Dec 04, 2019 4:29 pm

I will have to investigate further. Templates were taking a few minutes to back up, now they are taking well over an hour

Gostev
SVP, Product Management
Posts: 25141
Liked: 3692 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow template backup

Post by Gostev » Dec 04, 2019 5:00 pm

Check your ESXi management interface and switches first and foremost.

mkretzer
Expert
Posts: 574
Liked: 129 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Very slow template backup

Post by mkretzer » Dec 05, 2019 11:41 am

Gostev,

How does this workaround work? Does it use the same transfer methods/links as a normal backup would?

Gostev
SVP, Product Management
Posts: 25141
Liked: 3692 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow template backup

Post by Gostev » Dec 05, 2019 2:09 pm

No, since normal backup uses VDDK, which as I explained above does not support templates backup at all... we use another API instead, which was not designed for backup. But effectively, it is somewhat close to how VDDK works when backing up over NBD (data also goes through the ESXi management network, so it is quite slow on 1Gb Ethernet).

HendersonD
Expert
Posts: 125
Liked: 7 times
Joined: Jul 23, 2011 12:35 am

Re: Very slow template backup

Post by HendersonD » Dec 06, 2019 1:08 pm 3 people like this post

I figured out the problem. Every so often we take our templates, turn them into VMs, turn them on and update them. Once turned on we login to the VM, assign an IP address to the nic, and do Microsoft updates. Once we have the VM updated we shut it down and turn it back into a template. We did this process a short while ago and forgot to do one thing. Before shutting it down we left the nic configured with an IP address where we normally take off the IP address. You wouldn't think this would affect the ability of Veeam to backup the template but it did. Yesterday later in the day we took the IP addresses off the nics for these two templates and last night backup went well with each template taking between 4 and 8 minutes to backup.

DGrinev
Veeam Software
Posts: 1818
Liked: 230 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Very slow template backup

Post by DGrinev » Dec 06, 2019 2:26 pm

That's interesting!
I've shared this finding with the responsible QA team. Thanks!

DGrinev
Veeam Software
Posts: 1818
Liked: 230 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Very slow template backup

Post by DGrinev » Dec 09, 2019 10:29 am

The QA team cannot confirm said behavior.
Can you back up two templates by a single job? One should have an IP the other shouldn't. Thanks!

HendersonD
Expert
Posts: 125
Liked: 7 times
Joined: Jul 23, 2011 12:35 am

Re: Very slow template backup

Post by HendersonD » Dec 10, 2019 6:25 pm

I am doubting my own conclusion now. My template does not have an IP address and I am seeing varying behavior. Here are the last 3 backup jobs and what my Windows Server 2019 template did
  1. Did not backup on first try with "Error: VDDK operation timed out", the second try of this job backed up the template in 3:28
  2. Template backed up in 4:54
  3. Template backed up in 3:28
  4. Template took 30:30 to backup
Would I be better off to take my template and convert it to a VM and just leave it off? Each night it would backup as a VM and not a template. When I wanted to spin up a new server I can convert it back to a template, spin up a new VM from the template, and then convert the template back to a VM

DGrinev
Veeam Software
Posts: 1818
Liked: 230 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Very slow template backup

Post by DGrinev » Dec 11, 2019 11:24 am

You can do that, but I would ask the support team for assistance to find out the root cause of the issue. Thanks!

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 22 guests