Slow Config File Backups in v6

[brupnick]

Good morning-

After upgrading one of my v5 boxes to v6, I noticed that it takes approximately 10 minutes to back up my .vmx, .vmxf, and .nvram files. Since these files are a combined 13K, file size obviously isn't the problem. Is there a way for me to tell why this is taking so long? I have 31 VMs in this job, so even if I could get them down to a minute each, I'd shave 3 hours off of my backup time.

Thank you,
Brian

[Vitaliy S.]

Brian, the best way to find it out is to look through the corresponding backup job log files. You can contact our support team for assistance with that. Btw, do you see this performance degradation on all hosts which VMs you're backing up?

[brupnick]

Thanks Vitaliy; I'll do that. And yes, I see this in every VM in the job, which are spread over my 4 hosts. Some are worse than others, but in the example of last night's job, none are better than 3:03.

[tsightler]

I saw this issue once at a client site and I think it turned out to be something to do with DNS.

[brupnick]

Thanks Tom. I'm sure you would have included it if you could, but do you remember anything more specific about this potential cause?

[tsightler]

Honestly I just don't remember the exact details, the client corrected the issue themselves but they were convinced it was something to do with DNS, I think they even put entries on the hosts file for the vCenter, Veeam server, and ESX hosts. Support should be able to dig through the logs and determine where the pauses are coming from.

[Gostev]

Logs would show the reason for these pauses right away, but I can add couple more guesses:
- SQL issues could be another if Veeam backup DB is hosted on the remote SQL
- vCenter server is overloaded

[brupnick]

I can find the spot in the log where it issues this command, then no entry until 3+ minutes later.

As for the SQL point, I'm using a local SQL install, so I wouldn't expect there to be any latency there. The overloading of vCenter is a possibility, but is there a way for me to check this?

[Gostev]

Please do not post log snippets on forums, and instead send full log package to support for investigation (as explained when you click New Topic). Regarding vCenter server load, CPU and memory usage are good indicators. Thanks.

[brupnick]

Thanks Anton, and sorry about the log snippet.

Here are some snapshots of the performance of my VC server for the past day. None of these graphs are overly troubling to me and don't seem to show that VC is under any kind of load.

[Gostev]

Yep, your vCenter looks good to me as well. But as I've said, these were nothing more than blind guesses.

[brupnick]

But usually those yield the best information.

[tsightler]

What's with all the network traffic between 10:20PM and 1:20AM? Is that when your backups are running?

[brupnick]

Yes. My backups start at 7:00 PM and lately one of the jobs has been running for about 18 hours. Hence my desire to cut down the time spent backing up 13K worth of files.

[brupnick]

Because this has become a huge limitation for me, I've been spending some time over the last few days to try to see why this is happening. My production job is responsible for backing up two individual VMs and a resource group. When I create a new job with the same exact VMs and run it, the config files are backed up in about 10 seconds (as opposed to the 10 minutes in my other job). The only thing that's different between these two jobs is that one is mapped to an existing backup file set (the one that takes 10 minutes) and one is not (the one that takes 10 seconds). I have found the entry in the log that occurs immediately before the 3+ minute wait and am waiting on support to tell me what this command is doing.

That being said, is there anyone who is experiencing a similar issue? In my particular case, I upgraded one of my v5 boxes to v6, then imported the backup files to a clean installation of v6 on a different server. I am spending literally twice as much time as necessary to run this job because of this problem.

[tsightler]

I'm curious, is this incremental backups? How long is the chain?

[brupnick]

Yes, they are incrementals. I run my fulls on the first Friday/Saturday of the month, so I have a .vbk from 12/3. I do incrementals M-F, so I'm creating .vib #14 now. My .vbk is about 700 G, the Monday .vib is about 210 GB and all the other daily .vibs are about 120 GB. Considering that my other daily production job creates a .vbk that's about 1.5 TB (but the .vibs are about half the size), this doesn't seem to me like it should be causing my processing to be SO slow.

[brupnick]

Just an update on this after exchanging some emails with support. As you may or may not know, v5 used to store metadata within the actual backup files. In v6, they use the .vbm file for this data. Because I imported a v5-->v6 backup chain into my clean v6 environment, it looks as though v6 is still going back into the old v5 backup files to retrieve the necessary metadata. This is happening for the .vmdks as well as the .vmx, .vmxf, and .nvram. This would explain why it takes 10 minutes to back up these three files, which total about 13K in size.

One of my questions to support was, Why didn't the first run in the upgraded v5-->v6 environment extract the necessary metadata from the v5 files and write it into the v6 .vbm? Not having to go back into the .vbk and .vib files for every run would cut my backup window literally in half.

[Vitaliy S.]

That being said, is there anyone who is experiencing a similar issue? In my particular case, I upgraded one of my v5 boxes to v6, then imported the backup files to a clean installation of v6 on a different server.
...
Because I imported a v5-->v6 backup chain into my clean v6 environment, it looks as though v6 is still going back into the old v5 backup files to retrieve the necessary metadata

I believe this is the reason why you had this issue, as mapping into backup files created by different Veeam Backup server is not supported.

[brupnick]

Wasn't this the whole reason for creating and including the Map Backup option?

[Vitaliy S.]

Mapping for backup files comes handy when you want to reuse existing backup files and do not want to perform full backups from scratch. With v6 architecture there is simply no reason to keep multiple backup servers, since all VM data processing has been offloaded to backup proxy servers. Hope this makes sense.

[brupnick]

This makes complete sense and is exactly what I was trying to do. I run fulls on the first Friday of the month. Since v6 came out shortly after I ran my fulls, I didn't want to have to wait until my January full to upgrade. So, as per the manual (and support), I upgraded my v5 server to v6 and ran the job so that I could import that file into v6.

Note that backups created with Veeam Backup & Replication v5 do not include .vbm files, and therefore, cannot be used for mapping “as is”. To generate a .vbm file for a v5 backup, you need to perform at least one job pass after upgrading to Veeam Backup & Replication v6.

Maybe I missed it, but I didn't find anything in the manual that said I couldn't (shouldn't) import my backups to a different VBR server.

[Vitaliy S.]

Thanks for the feedback, I will pass this information to our technical writers team, sorry for inconvenience.

[brupnick]

No problem. But I would like to add that I just checked the logs on my v5-->v6 upgraded server for this job, and it looks like this behavior was happening there as well. So I don't know if anyone else is having this issue or if it's just me.

[tsightler]

Just an update on this after exchanging some emails with support. As you may or may not know, v5 used to store metadata within the actual backup files. In v6, they use the .vbm file for this data. Because I imported a v5-->v6 backup chain into my clean v6 environment, it looks as though v6 is still going back into the old v5 backup files to retrieve the necessary metadata. This is happening for the .vmdks as well as the .vmx, .vmxf, and .nvram. This would explain why it takes 10 minutes to back up these three files, which total about 13K in size.

This is exactly why I asked if you were using incrementals and how long the chain was. Reading metadata from incrementals with V5 could sometimes be a time consuming task.

One of my questions to support was, Why didn't the first run in the upgraded v5-->v6 environment extract the necessary metadata from the v5 files and write it into the v6 .vbm? Not having to go back into the .vbk and .vib files for every run would cut my backup window literally in half.

I suspect this was simply a design compromise. If you upgrade from V5 you simply get the same behavior as you had with V5 for the older backup files, but as those are slowly aged out the problem would slowly go away as you would eventually have only V6 backup files.