Incrementals of some VMs take as long as full backups

[pshute]

Can anyone please tell me why some of my servers take just as long to do an Incremental backup as an Active Full, despite the TransferredSize being much smaller.

Some example durations for one of the servers concerned (you can tell which are incrementals by the smaller Transferred Size):
DurationReadSize in GBTransferredSize in GB
4:29:01853.8485.8
4:54:11853.8490.8
4:57:16833.321.9
4:55:18813.422.3
5:04:04817.828.3
4:28:31761.931.4
4:46:14836.333.4
4:54:49836.935.7
4:23:10763.643.3
4:41:17817.947.4
4:41:31813.8484.1

For that particular server, the history entries show that one particular operation takes most of the time:
Full 11/01/2017 9:52:27 AM :: Hard disk 7 (600.0 GB) 600.0 GB read at 45 MB/s [CBT] - duration 3:51:06
Incr 13/01/2017 12:22:29 AM :: Hard disk 7 (600.0 GB) 600.0 GB read at 45 MB/s [CBT] - duration 3:43:56

Does that mean the problem is to do with how long it's taking to read that disk? I have other servers nearly as big that are much quicker doing incrementals than fulls.

[Mike Resseler]

Peter,

Can you tell us what the bottleneck is? It seems indeed that it is a bit slow. How is the disk read of your other servers?

[Shestakov]

Do other VMs use same datastore and proxy servers?
Thanks!

[foggy]

The fact that it reads the entire disk during incremental run tells that CBT isn't working...

[pshute]

Yes, thank you, I see this line in the history for that machine:
"16/01/2017 11:35:06 PM :: CBT data is invalid, failing over to legacy incremental backup. No action is required, next job run should start using CBT again. If CBT data remains invalid, follow KB1113 to perform CBT reset. Usual cause is power loss. "

I'll try resetting CBT.

I would never have spotted that myself, as it's a bit buried in the job report, and it would take a bit of work to check if other machines are affected. Is there an easier way of finding them?

And shouldn't CBT failure be mentioned in the backup job report we get emailed?

[foggy]

You can monitor job duration using Backup Infrastructure Custom Data Veeam ONE report to be aware of such issues.

[Vitaliy S.]

Or if you want to be notified about this behavior in real-time, you can use backup job duration alarm in Veeam ONE, that will notify you when your job has exceeded its usual duration period.

[pshute]

The fact that it reads the entire disk during incremental run tells that CBT isn't working...

Now that I've enabled deleted block exclusion, it's not reading the whole disk. How can I now tell if CBT is working for this disk?

I'd like a way of auditing CBT for all the disks on all our VMs. Monitoring job duration won't help if they've always been failing, as I have no benchmark, nor will it tell me which disk has the problem. Looking for the warning in the job history will only tell me that at least one disk has a CBT problem in that job. It won't tell me how many or which ones.

Would the information be available to a Powershell script, or can I only extract the same information that's in the job histories?

[foggy]

Now that I've enabled deleted block exclusion, it's not reading the whole disk. How can I now tell if CBT is working for this disk?

With CBT in action, Veeam B&R will read just the changes occurred inside the VM, which are typically 3-5% of the total size for regular servers.