Backup Job Duration

[Stephan23]

I have some trouble with the duration of one specific backup job and have no clue what the cause could be. Is anyone able able to point me in the right direction?

The job contains about 280 VMs with a total size of 17 TB and takes about 12 to 17 hours to finish.

Taken from one example:
Bottleneck: Source 49%, Proxy 6%, Network 40%, Target 34%
Processing rate 9 MB/s
Read 209 GB
Transfer 60 GB

The proxy is hardware based with direct (san) access to the VMware datastores.
The Veeam repository is connected to the same hardware via FC.

Because the job takes so much time, no other job is running simultaneously after a couple of hours. At this time the proxy, the repository and the vCenter is basically idling and very little load on the production storage.

The status of VMs that are processed is either 0% or 99% most of the time (30-50 min per VM). The time that is needed for the actual data transfer is relatively small. The throughput is not great, but that is no concern.




"Only" 8 minutes for data processing, but 40 minutes in total from start to finish for this VM as an example.

I haven't done much so far for troubleshooting.
I tried to lower the maximum number of concurrent task, which made it worse.
My next step is to set up a proxy as a VM and use hotadd, in the hope of isolating the cause.

I updated to v10 last week, no change. The job ran fine (3-4 hours) weeks ago, but no changes were made that would explain that behavior (for me).

Any ideas?

[PetrM]

Hi Stephan,

Based on the last screenshot, it seems that a VM was waiting for available proxy or repository slots more than 10 hours.
The data processing can be started only once infrastructure resources (usually proxy or repository slots) are assigned to a VM in a job.

You said that you tested the behavior with a decreased number of concurrent tasks but I would suggest to increase concurrent tasks for proxies and a repository and may be to add more proxies but don't forget about system requirements (pages: 8-10 for concurrent tasks of proxy and repository).

Feel free to open a support request if the suggestion above does not help and please provide us with support case number.

Thanks!

[Stephan23]

Hi Petr,

thanks for the reply. The amount of concurrent tasks is limited to the the number of CPU cores on the proxy (12 cores), which should be plenty. Raising the number further or adding more proxies would be brute force approach in my opinion, that should not be necessary. Other jobs on the same proxy run fine.

And like I said, the proxy is idling while the job is running.

[Stephan23]

Based on the last screenshot, it seems that a VM was waiting for available proxy or repository slots more than 10 hours.

I only now see what you are referring to regarding the 10 hours.
Of course the job was waiting for available slots that long. That's not the issue. The issue is that it takes 30-50 minutes per VM to process.

[Stephan23]

I created a new job for testing with a single VM. The initial Active Full took 10 minutes, the subsequent incremental 1 minute.
That leads me to believe that something is wrong with the backup chain. Is that possible? I don't see any IO load or access to backup files, when no data processing is happening.

[PetrM]

Hi Stephan,

By the way, you may take a look at this table which reflects a generic overview for I/O impact of different backup chain modes.
May be it would make sense to test forward incremental mode which does not require synthetic operations and to compare job duration (check "create active full backups periodically").

But I recommend to ask our support team to check logs and to define which operation (except waiting for slots) takes most of time because we can not get this information from the screenshots.

Thanks!

[gerling-sit]

Hi, Stephan,

What is the relationship between the data read (from the VM) and the data actually written (from the VM) in the backup job?
Does the backup job actually only read the changed data from the VM or does it read almost all data from the VM?

Does the Change Block Tracking (CBT) work for you ?

[Stephan23]

Yes, CBT works.
From one example:
Processed: 16,4 TB
Read: 369,2 GB
Transferred: 97,8 GB


It seams it has something to do with the backup chain. I created a new job with a per-VM repository (same back end) and moved some VMs over. The first Active Full was already a lot faster than the incremental before.

I also created a ticket (04123811), but so far not very helpful in pointing out, what the cause is, just some generic recommendations about load on proxy/repository/etc.

[foggy]

If you'd like to get to the bottom of this, I recommend pushing them to review the logs closely to find the exact operation that took too long for the old job.