Job Overhead

[Deckers]

I'm hoping to get an idea of how much time other B&R users are seeing with job overhead. It seems like it takes quite a bit of time per VM for the setup and teardown operations before it even reads data. With 100+ VM's in the job this can add 1 to 2 hours of additional run time.

As a test I created a job with two 5GB VM's in it and ran a full backup, then immediately ran the job a second time to get a feel for the incremental time.

Building VM list: 38 seconds
VM-A
Stops on the 'Using source proxy [nbd]' task for 97 seconds
Creating VM Snapshot: 6 seconds
Saving VMX: 49 seconds
Saving VMXF: 41 seconds
Saving NVRAM: 50 seconds
Hard disk: 115 seconds (95 seconds elapse before the disk changes from 0KB read)
Finalizing: 45 sec

VM-B
Stops on the 'Using source proxy [nbd]' task for 90 seconds
Creating VM Snapshot: 9 seconds
Saving VMX: 37 seconds
Saving VMXF: 40 seconds
Saving NVRAM: 28 seconds
Hard disk: 115 seconds (95 seconds elapse before the disk changes from 0KB read)
Finalizing: 45 sec

I don't want to focus so much on the network vs hotadd modes as I've tried both and network was faster. I also used to use LAN free mode when I could and still saw this type of processing overhead. I can't understand why it takes 140 or 105 seconds to copy 11KB of VMX, VMFX, NVRAM data. So bottom line it just seems sluggish overall but I don't have anything to compare it to.

[Gostev]

You may want to be indicating B&R versions when posting results, as presumably 6.5 had some improvements in this.

I can't understand why it takes 140 or 105 seconds to copy 11KB of VMX, VMFX, NVRAM data.

Actual copy completes very fast, it is chatting with VMware and establishing the download connection that takes forever in certain larger environments. For example, in my lab all of that takes just a few seconds to complete.

[dellock6]

I can confirm this behaviour, in our environment with almost 1000 VMs managed by a single vCenter, it takes long time to chat with it, then the proper backup operation is fast, and even without exact data, I can say it has increased its speed from 6.1 to 6.5.

Luca.

[rkovhaev]

You can experience the behavior when you stage your backups to CIFS/SMBFS, it takes 50 seconds to open the storage, and couple of seconds to put file. CIFS/SMBFS is NAS, it is not block aware, so it needs to cache(download) some parts of the file first. Depends on VBK size and amount of allocated metadata banks.

[dellock6]

Hi Rustam,
I'm experiencing the same behaviour in a block storage connected to the Veeam repository via iscsi, and formatted as ntfs local disk. But as Anton explained, in my case maybe the problem is about the size of the backup more than the use of NAS storage.

Luca.

[Deckers]

You may want to be indicating B&R versions when posting results, as presumably 6.5 had some improvements in this.
Actual copy completes very fast, it is chatting with VMware and establishing the download connection that takes forever in certain larger environments. For example, in my lab all of that takes just a few seconds to complete.

Good point Gostev, these numbers were with 6.1. I've run it again today after the 6.5 update and it's improved considerably. More in line with were I think it should be performing...

VM-A
Stops on the 'Using source proxy [nbd]' didn't catch the time, but it was quicker (was 97 sec)
Creating VM Snapshot: 4 sec (was 6)
Saving VMX: 2 sec (was 49)
Saving VMXF: 3 sec (was 41)
Saving NVRAM: 3 sec (was 50)
Hard disk: 8 sec (was 115)
Finalizing: 1 sec (was 45)

Overall the job time for these two VM's went from ~13 min to ~4 min. I have not seen a major change in the duration of my normal jobs (110 vm & 44 vm) however.

[tsightler]

With that many VMs in a single job my guess is that the backup files themselves are quite large. It's important to remember that you will always get the best performance from Veeam by scaling to many, smaller jobs (at the cost of some dedupe).

What would be interesting to see from your current setup is the following:

1. Total size of VM data being backed up
2. Total size of VBK files
3. Backup mode (forward or reverse incremental)
4. Backup target (disk/spindles/NAS/dedupe)
5. Memory usage of VeeamAgent process on both proxy and repository during backups

[Gostev]

Kyle, that is great to hear nice reduction!