-
- Expert
- Posts: 196
- Liked: 13 times
- Joined: Feb 05, 2011 5:09 pm
- Full Name: Brian Rupnick
- Location: New York, USA
- Contact:
Slow Config File Backups in v6
Good morning-
After upgrading one of my v5 boxes to v6, I noticed that it takes approximately 10 minutes to back up my .vmx, .vmxf, and .nvram files. Since these files are a combined 13K, file size obviously isn't the problem. Is there a way for me to tell why this is taking so long? I have 31 VMs in this job, so even if I could get them down to a minute each, I'd shave 3 hours off of my backup time.
Thank you,
Brian
After upgrading one of my v5 boxes to v6, I noticed that it takes approximately 10 minutes to back up my .vmx, .vmxf, and .nvram files. Since these files are a combined 13K, file size obviously isn't the problem. Is there a way for me to tell why this is taking so long? I have 31 VMs in this job, so even if I could get them down to a minute each, I'd shave 3 hours off of my backup time.
Thank you,
Brian
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Slow Config File Backups in v6
Brian, the best way to find it out is to look through the corresponding backup job log files. You can contact our support team for assistance with that. Btw, do you see this performance degradation on all hosts which VMs you're backing up?
-
- Expert
- Posts: 196
- Liked: 13 times
- Joined: Feb 05, 2011 5:09 pm
- Full Name: Brian Rupnick
- Location: New York, USA
- Contact:
Re: Slow Config File Backups in v6
Thanks Vitaliy; I'll do that. And yes, I see this in every VM in the job, which are spread over my 4 hosts. Some are worse than others, but in the example of last night's job, none are better than 3:03.
-
- VP, Product Management
- Posts: 6035
- Liked: 2860 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Slow Config File Backups in v6
I saw this issue once at a client site and I think it turned out to be something to do with DNS.
-
- Expert
- Posts: 196
- Liked: 13 times
- Joined: Feb 05, 2011 5:09 pm
- Full Name: Brian Rupnick
- Location: New York, USA
- Contact:
Re: Slow Config File Backups in v6
Thanks Tom. I'm sure you would have included it if you could, but do you remember anything more specific about this potential cause?
-
- VP, Product Management
- Posts: 6035
- Liked: 2860 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Slow Config File Backups in v6
Honestly I just don't remember the exact details, the client corrected the issue themselves but they were convinced it was something to do with DNS, I think they even put entries on the hosts file for the vCenter, Veeam server, and ESX hosts. Support should be able to dig through the logs and determine where the pauses are coming from.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Slow Config File Backups in v6
Logs would show the reason for these pauses right away, but I can add couple more guesses:
- SQL issues could be another if Veeam backup DB is hosted on the remote SQL
- vCenter server is overloaded
- SQL issues could be another if Veeam backup DB is hosted on the remote SQL
- vCenter server is overloaded
-
- Expert
- Posts: 196
- Liked: 13 times
- Joined: Feb 05, 2011 5:09 pm
- Full Name: Brian Rupnick
- Location: New York, USA
- Contact:
Re: Slow Config File Backups in v6
I can find the spot in the log where it issues this command, then no entry until 3+ minutes later.
As for the SQL point, I'm using a local SQL install, so I wouldn't expect there to be any latency there. The overloading of vCenter is a possibility, but is there a way for me to check this?
As for the SQL point, I'm using a local SQL install, so I wouldn't expect there to be any latency there. The overloading of vCenter is a possibility, but is there a way for me to check this?
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Slow Config File Backups in v6
Please do not post log snippets on forums, and instead send full log package to support for investigation (as explained when you click New Topic). Regarding vCenter server load, CPU and memory usage are good indicators. Thanks.
-
- Expert
- Posts: 196
- Liked: 13 times
- Joined: Feb 05, 2011 5:09 pm
- Full Name: Brian Rupnick
- Location: New York, USA
- Contact:
Re: Slow Config File Backups in v6
Thanks Anton, and sorry about the log snippet.
Here are some snapshots of the performance of my VC server for the past day. None of these graphs are overly troubling to me and don't seem to show that VC is under any kind of load.
Here are some snapshots of the performance of my VC server for the past day. None of these graphs are overly troubling to me and don't seem to show that VC is under any kind of load.
-
- Chief Product Officer
- Posts: 31814
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Slow Config File Backups in v6
Yep, your vCenter looks good to me as well. But as I've said, these were nothing more than blind guesses.
-
- Expert
- Posts: 196
- Liked: 13 times
- Joined: Feb 05, 2011 5:09 pm
- Full Name: Brian Rupnick
- Location: New York, USA
- Contact:
Re: Slow Config File Backups in v6
But usually those yield the best information.
-
- VP, Product Management
- Posts: 6035
- Liked: 2860 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Slow Config File Backups in v6
What's with all the network traffic between 10:20PM and 1:20AM? Is that when your backups are running?
-
- Expert
- Posts: 196
- Liked: 13 times
- Joined: Feb 05, 2011 5:09 pm
- Full Name: Brian Rupnick
- Location: New York, USA
- Contact:
Re: Slow Config File Backups in v6
Yes. My backups start at 7:00 PM and lately one of the jobs has been running for about 18 hours. Hence my desire to cut down the time spent backing up 13K worth of files.
-
- Expert
- Posts: 196
- Liked: 13 times
- Joined: Feb 05, 2011 5:09 pm
- Full Name: Brian Rupnick
- Location: New York, USA
- Contact:
Re: Slow Config File Backups in v6
Because this has become a huge limitation for me, I've been spending some time over the last few days to try to see why this is happening. My production job is responsible for backing up two individual VMs and a resource group. When I create a new job with the same exact VMs and run it, the config files are backed up in about 10 seconds (as opposed to the 10 minutes in my other job). The only thing that's different between these two jobs is that one is mapped to an existing backup file set (the one that takes 10 minutes) and one is not (the one that takes 10 seconds). I have found the entry in the log that occurs immediately before the 3+ minute wait and am waiting on support to tell me what this command is doing.
That being said, is there anyone who is experiencing a similar issue? In my particular case, I upgraded one of my v5 boxes to v6, then imported the backup files to a clean installation of v6 on a different server. I am spending literally twice as much time as necessary to run this job because of this problem.
That being said, is there anyone who is experiencing a similar issue? In my particular case, I upgraded one of my v5 boxes to v6, then imported the backup files to a clean installation of v6 on a different server. I am spending literally twice as much time as necessary to run this job because of this problem.
-
- VP, Product Management
- Posts: 6035
- Liked: 2860 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Slow Config File Backups in v6
I'm curious, is this incremental backups? How long is the chain?
-
- Expert
- Posts: 196
- Liked: 13 times
- Joined: Feb 05, 2011 5:09 pm
- Full Name: Brian Rupnick
- Location: New York, USA
- Contact:
Re: Slow Config File Backups in v6
Yes, they are incrementals. I run my fulls on the first Friday/Saturday of the month, so I have a .vbk from 12/3. I do incrementals M-F, so I'm creating .vib #14 now. My .vbk is about 700 G, the Monday .vib is about 210 GB and all the other daily .vibs are about 120 GB. Considering that my other daily production job creates a .vbk that's about 1.5 TB (but the .vibs are about half the size), this doesn't seem to me like it should be causing my processing to be SO slow.
-
- Expert
- Posts: 196
- Liked: 13 times
- Joined: Feb 05, 2011 5:09 pm
- Full Name: Brian Rupnick
- Location: New York, USA
- Contact:
Re: Slow Config File Backups in v6
Just an update on this after exchanging some emails with support. As you may or may not know, v5 used to store metadata within the actual backup files. In v6, they use the .vbm file for this data. Because I imported a v5-->v6 backup chain into my clean v6 environment, it looks as though v6 is still going back into the old v5 backup files to retrieve the necessary metadata. This is happening for the .vmdks as well as the .vmx, .vmxf, and .nvram. This would explain why it takes 10 minutes to back up these three files, which total about 13K in size.
One of my questions to support was, Why didn't the first run in the upgraded v5-->v6 environment extract the necessary metadata from the v5 files and write it into the v6 .vbm? Not having to go back into the .vbk and .vib files for every run would cut my backup window literally in half.
One of my questions to support was, Why didn't the first run in the upgraded v5-->v6 environment extract the necessary metadata from the v5 files and write it into the v6 .vbm? Not having to go back into the .vbk and .vib files for every run would cut my backup window literally in half.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Slow Config File Backups in v6
I believe this is the reason why you had this issue, as mapping into backup files created by different Veeam Backup server is not supported.brupnick wrote:That being said, is there anyone who is experiencing a similar issue? In my particular case, I upgraded one of my v5 boxes to v6, then imported the backup files to a clean installation of v6 on a different server.
...
Because I imported a v5-->v6 backup chain into my clean v6 environment, it looks as though v6 is still going back into the old v5 backup files to retrieve the necessary metadata
-
- Expert
- Posts: 196
- Liked: 13 times
- Joined: Feb 05, 2011 5:09 pm
- Full Name: Brian Rupnick
- Location: New York, USA
- Contact:
Re: Slow Config File Backups in v6
Wasn't this the whole reason for creating and including the Map Backup option?
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Slow Config File Backups in v6
Mapping for backup files comes handy when you want to reuse existing backup files and do not want to perform full backups from scratch. With v6 architecture there is simply no reason to keep multiple backup servers, since all VM data processing has been offloaded to backup proxy servers. Hope this makes sense.
-
- Expert
- Posts: 196
- Liked: 13 times
- Joined: Feb 05, 2011 5:09 pm
- Full Name: Brian Rupnick
- Location: New York, USA
- Contact:
Re: Slow Config File Backups in v6
This makes complete sense and is exactly what I was trying to do. I run fulls on the first Friday of the month. Since v6 came out shortly after I ran my fulls, I didn't want to have to wait until my January full to upgrade. So, as per the manual (and support), I upgraded my v5 server to v6 and ran the job so that I could import that file into v6.
Maybe I missed it, but I didn't find anything in the manual that said I couldn't (shouldn't) import my backups to a different VBR server.Note that backups created with Veeam Backup & Replication v5 do not include .vbm files, and therefore, cannot be used for mapping “as is”. To generate a .vbm file for a v5 backup, you need to perform at least one job pass after upgrading to Veeam Backup & Replication v6.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Slow Config File Backups in v6
Thanks for the feedback, I will pass this information to our technical writers team, sorry for inconvenience.
-
- Expert
- Posts: 196
- Liked: 13 times
- Joined: Feb 05, 2011 5:09 pm
- Full Name: Brian Rupnick
- Location: New York, USA
- Contact:
Re: Slow Config File Backups in v6
No problem. But I would like to add that I just checked the logs on my v5-->v6 upgraded server for this job, and it looks like this behavior was happening there as well. So I don't know if anyone else is having this issue or if it's just me.
-
- VP, Product Management
- Posts: 6035
- Liked: 2860 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Slow Config File Backups in v6
This is exactly why I asked if you were using incrementals and how long the chain was. Reading metadata from incrementals with V5 could sometimes be a time consuming task.brupnick wrote:Just an update on this after exchanging some emails with support. As you may or may not know, v5 used to store metadata within the actual backup files. In v6, they use the .vbm file for this data. Because I imported a v5-->v6 backup chain into my clean v6 environment, it looks as though v6 is still going back into the old v5 backup files to retrieve the necessary metadata. This is happening for the .vmdks as well as the .vmx, .vmxf, and .nvram. This would explain why it takes 10 minutes to back up these three files, which total about 13K in size.
I suspect this was simply a design compromise. If you upgrade from V5 you simply get the same behavior as you had with V5 for the older backup files, but as those are slowly aged out the problem would slowly go away as you would eventually have only V6 backup files.brupnick wrote:One of my questions to support was, Why didn't the first run in the upgraded v5-->v6 environment extract the necessary metadata from the v5 files and write it into the v6 .vbm? Not having to go back into the .vbk and .vib files for every run would cut my backup window literally in half.
Who is online
Users browsing this forum: Google [Bot] and 18 guests