Discussions specific to the VMware vSphere hypervisor
Post Reply
brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Slow Config File Backups in v6

Post by brupnick » Dec 14, 2011 2:15 pm

Good morning-

After upgrading one of my v5 boxes to v6, I noticed that it takes approximately 10 minutes to back up my .vmx, .vmxf, and .nvram files. Since these files are a combined 13K, file size obviously isn't the problem. Is there a way for me to tell why this is taking so long? I have 31 VMs in this job, so even if I could get them down to a minute each, I'd shave 3 hours off of my backup time.

Thank you,
Brian

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow Config File Backups in v6

Post by Vitaliy S. » Dec 14, 2011 3:14 pm

Brian, the best way to find it out is to look through the corresponding backup job log files. You can contact our support team for assistance with that. Btw, do you see this performance degradation on all hosts which VMs you're backing up?

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Slow Config File Backups in v6

Post by brupnick » Dec 14, 2011 5:02 pm

Thanks Vitaliy; I'll do that. And yes, I see this in every VM in the job, which are spread over my 4 hosts. Some are worse than others, but in the example of last night's job, none are better than 3:03.

tsightler
VP, Product Management
Posts: 5476
Liked: 2294 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow Config File Backups in v6

Post by tsightler » Dec 14, 2011 5:03 pm

I saw this issue once at a client site and I think it turned out to be something to do with DNS.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Slow Config File Backups in v6

Post by brupnick » Dec 14, 2011 5:08 pm

Thanks Tom. I'm sure you would have included it if you could, but do you remember anything more specific about this potential cause?

tsightler
VP, Product Management
Posts: 5476
Liked: 2294 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow Config File Backups in v6

Post by tsightler » Dec 14, 2011 5:37 pm

Honestly I just don't remember the exact details, the client corrected the issue themselves but they were convinced it was something to do with DNS, I think they even put entries on the hosts file for the vCenter, Veeam server, and ESX hosts. Support should be able to dig through the logs and determine where the pauses are coming from.

Gostev
SVP, Product Management
Posts: 25149
Liked: 3700 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow Config File Backups in v6

Post by Gostev » Dec 14, 2011 8:41 pm

Logs would show the reason for these pauses right away, but I can add couple more guesses:
- SQL issues could be another if Veeam backup DB is hosted on the remote SQL
- vCenter server is overloaded

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Slow Config File Backups in v6

Post by brupnick » Dec 14, 2011 9:09 pm

I can find the spot in the log where it issues this command, then no entry until 3+ minutes later.

As for the SQL point, I'm using a local SQL install, so I wouldn't expect there to be any latency there. The overloading of vCenter is a possibility, but is there a way for me to check this?

Gostev
SVP, Product Management
Posts: 25149
Liked: 3700 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow Config File Backups in v6

Post by Gostev » Dec 14, 2011 9:44 pm

Please do not post log snippets on forums, and instead send full log package to support for investigation (as explained when you click New Topic). Regarding vCenter server load, CPU and memory usage are good indicators. Thanks.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Slow Config File Backups in v6

Post by brupnick » Dec 15, 2011 3:37 pm

Thanks Anton, and sorry about the log snippet.

Here are some snapshots of the performance of my VC server for the past day. None of these graphs are overly troubling to me and don't seem to show that VC is under any kind of load.

Image
Image
Image

Gostev
SVP, Product Management
Posts: 25149
Liked: 3700 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow Config File Backups in v6

Post by Gostev » Dec 15, 2011 3:42 pm

Yep, your vCenter looks good to me as well. But as I've said, these were nothing more than blind guesses.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Slow Config File Backups in v6

Post by brupnick » Dec 15, 2011 3:51 pm

But usually those yield the best information. :)

tsightler
VP, Product Management
Posts: 5476
Liked: 2294 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow Config File Backups in v6

Post by tsightler » Dec 15, 2011 4:18 pm

What's with all the network traffic between 10:20PM and 1:20AM? Is that when your backups are running?

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Slow Config File Backups in v6

Post by brupnick » Dec 15, 2011 4:25 pm

Yes. My backups start at 7:00 PM and lately one of the jobs has been running for about 18 hours. Hence my desire to cut down the time spent backing up 13K worth of files.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Slow Config File Backups in v6

Post by brupnick » Dec 21, 2011 4:35 pm

Because this has become a huge limitation for me, I've been spending some time over the last few days to try to see why this is happening. My production job is responsible for backing up two individual VMs and a resource group. When I create a new job with the same exact VMs and run it, the config files are backed up in about 10 seconds (as opposed to the 10 minutes in my other job). The only thing that's different between these two jobs is that one is mapped to an existing backup file set (the one that takes 10 minutes) and one is not (the one that takes 10 seconds). I have found the entry in the log that occurs immediately before the 3+ minute wait and am waiting on support to tell me what this command is doing.

That being said, is there anyone who is experiencing a similar issue? In my particular case, I upgraded one of my v5 boxes to v6, then imported the backup files to a clean installation of v6 on a different server. I am spending literally twice as much time as necessary to run this job because of this problem.

tsightler
VP, Product Management
Posts: 5476
Liked: 2294 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow Config File Backups in v6

Post by tsightler » Dec 21, 2011 4:40 pm

I'm curious, is this incremental backups? How long is the chain?

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Slow Config File Backups in v6

Post by brupnick » Dec 21, 2011 4:46 pm

Yes, they are incrementals. I run my fulls on the first Friday/Saturday of the month, so I have a .vbk from 12/3. I do incrementals M-F, so I'm creating .vib #14 now. My .vbk is about 700 G, the Monday .vib is about 210 GB and all the other daily .vibs are about 120 GB. Considering that my other daily production job creates a .vbk that's about 1.5 TB (but the .vibs are about half the size), this doesn't seem to me like it should be causing my processing to be SO slow.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Slow Config File Backups in v6

Post by brupnick » Dec 22, 2011 3:07 pm

Just an update on this after exchanging some emails with support. As you may or may not know, v5 used to store metadata within the actual backup files. In v6, they use the .vbm file for this data. Because I imported a v5-->v6 backup chain into my clean v6 environment, it looks as though v6 is still going back into the old v5 backup files to retrieve the necessary metadata. This is happening for the .vmdks as well as the .vmx, .vmxf, and .nvram. This would explain why it takes 10 minutes to back up these three files, which total about 13K in size.

One of my questions to support was, Why didn't the first run in the upgraded v5-->v6 environment extract the necessary metadata from the v5 files and write it into the v6 .vbm? Not having to go back into the .vbk and .vib files for every run would cut my backup window literally in half.

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow Config File Backups in v6

Post by Vitaliy S. » Dec 22, 2011 4:00 pm

brupnick wrote:That being said, is there anyone who is experiencing a similar issue? In my particular case, I upgraded one of my v5 boxes to v6, then imported the backup files to a clean installation of v6 on a different server.
...
Because I imported a v5-->v6 backup chain into my clean v6 environment, it looks as though v6 is still going back into the old v5 backup files to retrieve the necessary metadata
I believe this is the reason why you had this issue, as mapping into backup files created by different Veeam Backup server is not supported.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Slow Config File Backups in v6

Post by brupnick » Dec 22, 2011 4:05 pm

Wasn't this the whole reason for creating and including the Map Backup option?

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow Config File Backups in v6

Post by Vitaliy S. » Dec 22, 2011 4:21 pm

Mapping for backup files comes handy when you want to reuse existing backup files and do not want to perform full backups from scratch. With v6 architecture there is simply no reason to keep multiple backup servers, since all VM data processing has been offloaded to backup proxy servers. Hope this makes sense.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Slow Config File Backups in v6

Post by brupnick » Dec 22, 2011 4:28 pm

This makes complete sense and is exactly what I was trying to do. I run fulls on the first Friday of the month. Since v6 came out shortly after I ran my fulls, I didn't want to have to wait until my January full to upgrade. So, as per the manual (and support), I upgraded my v5 server to v6 and ran the job so that I could import that file into v6.
Note that backups created with Veeam Backup & Replication v5 do not include .vbm files, and therefore, cannot be used for mapping “as is”. To generate a .vbm file for a v5 backup, you need to perform at least one job pass after upgrading to Veeam Backup & Replication v6.
Maybe I missed it, but I didn't find anything in the manual that said I couldn't (shouldn't) import my backups to a different VBR server.

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow Config File Backups in v6

Post by Vitaliy S. » Dec 22, 2011 4:36 pm

Thanks for the feedback, I will pass this information to our technical writers team, sorry for inconvenience.

brupnick
Expert
Posts: 196
Liked: 13 times
Joined: Feb 05, 2011 5:09 pm
Full Name: Brian Rupnick
Location: New York, USA
Contact:

Re: Slow Config File Backups in v6

Post by brupnick » Dec 22, 2011 4:50 pm

No problem. But I would like to add that I just checked the logs on my v5-->v6 upgraded server for this job, and it looks like this behavior was happening there as well. So I don't know if anyone else is having this issue or if it's just me.

tsightler
VP, Product Management
Posts: 5476
Liked: 2294 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow Config File Backups in v6

Post by tsightler » Dec 22, 2011 5:55 pm

brupnick wrote:Just an update on this after exchanging some emails with support. As you may or may not know, v5 used to store metadata within the actual backup files. In v6, they use the .vbm file for this data. Because I imported a v5-->v6 backup chain into my clean v6 environment, it looks as though v6 is still going back into the old v5 backup files to retrieve the necessary metadata. This is happening for the .vmdks as well as the .vmx, .vmxf, and .nvram. This would explain why it takes 10 minutes to back up these three files, which total about 13K in size.
This is exactly why I asked if you were using incrementals and how long the chain was. Reading metadata from incrementals with V5 could sometimes be a time consuming task.
brupnick wrote:One of my questions to support was, Why didn't the first run in the upgraded v5-->v6 environment extract the necessary metadata from the v5 files and write it into the v6 .vbm? Not having to go back into the .vbk and .vib files for every run would cut my backup window literally in half.
I suspect this was simply a design compromise. If you upgrade from V5 you simply get the same behavior as you had with V5 for the older backup files, but as those are slowly aged out the problem would slowly go away as you would eventually have only V6 backup files.

Post Reply

Who is online

Users browsing this forum: No registered users and 27 guests