Comprehensive data protection for all workloads
Post Reply
Deckers
Novice
Posts: 4
Liked: 2 times
Joined: Oct 31, 2012 12:00 am
Full Name: Kyle
Contact:

Job Overhead

Post by Deckers »

I'm hoping to get an idea of how much time other B&R users are seeing with job overhead. It seems like it takes quite a bit of time per VM for the setup and teardown operations before it even reads data. With 100+ VM's in the job this can add 1 to 2 hours of additional run time.

As a test I created a job with two 5GB VM's in it and ran a full backup, then immediately ran the job a second time to get a feel for the incremental time.

Building VM list: 38 seconds
VM-A
Stops on the 'Using source proxy [nbd]' task for 97 seconds
Creating VM Snapshot: 6 seconds
Saving VMX: 49 seconds
Saving VMXF: 41 seconds
Saving NVRAM: 50 seconds
Hard disk: 115 seconds (95 seconds elapse before the disk changes from 0KB read)
Finalizing: 45 sec

VM-B
Stops on the 'Using source proxy [nbd]' task for 90 seconds
Creating VM Snapshot: 9 seconds
Saving VMX: 37 seconds
Saving VMXF: 40 seconds
Saving NVRAM: 28 seconds
Hard disk: 115 seconds (95 seconds elapse before the disk changes from 0KB read)
Finalizing: 45 sec

I don't want to focus so much on the network vs hotadd modes as I've tried both and network was faster. I also used to use LAN free mode when I could and still saw this type of processing overhead. I can't understand why it takes 140 or 105 seconds to copy 11KB of VMX, VMFX, NVRAM data. So bottom line it just seems sluggish overall but I don't have anything to compare it to.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Job Overhead

Post by Gostev »

You may want to be indicating B&R versions when posting results, as presumably 6.5 had some improvements in this.
Deckers wrote:I can't understand why it takes 140 or 105 seconds to copy 11KB of VMX, VMFX, NVRAM data.
Actual copy completes very fast, it is chatting with VMware and establishing the download connection that takes forever in certain larger environments. For example, in my lab all of that takes just a few seconds to complete.
dellock6
VeeaMVP
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Job Overhead

Post by dellock6 »

I can confirm this behaviour, in our environment with almost 1000 VMs managed by a single vCenter, it takes long time to chat with it, then the proper backup operation is fast, and even without exact data, I can say it has increased its speed from 6.1 to 6.5.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
rkovhaev
Veeam Software
Posts: 39
Liked: 21 times
Joined: May 17, 2010 6:49 pm
Full Name: Rustam
Location: hockey night in canada
Contact:

Re: Job Overhead

Post by rkovhaev » 1 person likes this post

You can experience the behavior when you stage your backups to CIFS/SMBFS, it takes 50 seconds to open the storage, and couple of seconds to put file. CIFS/SMBFS is NAS, it is not block aware, so it needs to cache(download) some parts of the file first. Depends on VBK size and amount of allocated metadata banks.
dellock6
VeeaMVP
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Job Overhead

Post by dellock6 »

Hi Rustam,
I'm experiencing the same behaviour in a block storage connected to the Veeam repository via iscsi, and formatted as ntfs local disk. But as Anton explained, in my case maybe the problem is about the size of the backup more than the use of NAS storage.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Deckers
Novice
Posts: 4
Liked: 2 times
Joined: Oct 31, 2012 12:00 am
Full Name: Kyle
Contact:

Re: Job Overhead

Post by Deckers » 2 people like this post

Gostev wrote:You may want to be indicating B&R versions when posting results, as presumably 6.5 had some improvements in this.
Actual copy completes very fast, it is chatting with VMware and establishing the download connection that takes forever in certain larger environments. For example, in my lab all of that takes just a few seconds to complete.
Good point Gostev, these numbers were with 6.1. I've run it again today after the 6.5 update and it's improved considerably. More in line with were I think it should be performing...

VM-A
Stops on the 'Using source proxy [nbd]' didn't catch the time, but it was quicker (was 97 sec)
Creating VM Snapshot: 4 sec (was 6)
Saving VMX: 2 sec (was 49)
Saving VMXF: 3 sec (was 41)
Saving NVRAM: 3 sec (was 50)
Hard disk: 8 sec (was 115)
Finalizing: 1 sec (was 45)

Overall the job time for these two VM's went from ~13 min to ~4 min. I have not seen a major change in the duration of my normal jobs (110 vm & 44 vm) however.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Job Overhead

Post by tsightler »

With that many VMs in a single job my guess is that the backup files themselves are quite large. It's important to remember that you will always get the best performance from Veeam by scaling to many, smaller jobs (at the cost of some dedupe).

What would be interesting to see from your current setup is the following:

1. Total size of VM data being backed up
2. Total size of VBK files
3. Backup mode (forward or reverse incremental)
4. Backup target (disk/spindles/NAS/dedupe)
5. Memory usage of VeeamAgent process on both proxy and repository during backups
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Job Overhead

Post by Gostev »

Kyle, that is great to hear ;) nice reduction!
Post Reply

Who is online

Users browsing this forum: rjv1971 and 162 guests