Discussions specific to the VMware vSphere hypervisor
Post Reply
pshute
Expert
Posts: 189
Liked: 9 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Incrementals of some VMs take as long as full backups

Post by pshute » Jan 13, 2017 4:16 am

Can anyone please tell me why some of my servers take just as long to do an Incremental backup as an Active Full, despite the TransferredSize being much smaller.

Some example durations for one of the servers concerned (you can tell which are incrementals by the smaller Transferred Size):
DurationReadSize in GBTransferredSize in GB
4:29:01853.8485.8
4:54:11853.8490.8
4:57:16833.321.9
4:55:18813.422.3
5:04:04817.828.3
4:28:31761.931.4
4:46:14836.333.4
4:54:49836.935.7
4:23:10763.643.3
4:41:17817.947.4
4:41:31813.8484.1

For that particular server, the history entries show that one particular operation takes most of the time:
Full 11/01/2017 9:52:27 AM :: Hard disk 7 (600.0 GB) 600.0 GB read at 45 MB/s [CBT] - duration 3:51:06
Incr 13/01/2017 12:22:29 AM :: Hard disk 7 (600.0 GB) 600.0 GB read at 45 MB/s [CBT] - duration 3:43:56

Does that mean the problem is to do with how long it's taking to read that disk? I have other servers nearly as big that are much quicker doing incrementals than fulls.

Mike Resseler
Veeam Software
Posts: 4974
Liked: 523 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Incrementals of some VMs take as long as full backups

Post by Mike Resseler » Jan 13, 2017 5:41 am

Peter,

Can you tell us what the bottleneck is? It seems indeed that it is a bit slow. How is the disk read of your other servers?

Shestakov
Veeam Software
Posts: 6028
Liked: 522 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Incrementals of some VMs take as long as full backups

Post by Shestakov » Jan 13, 2017 9:16 am

Do other VMs use same datastore and proxy servers?
Thanks!

foggy
Veeam Software
Posts: 16932
Liked: 1377 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incrementals of some VMs take as long as full backups

Post by foggy » Jan 13, 2017 10:02 pm

The fact that it reads the entire disk during incremental run tells that CBT isn't working...

pshute
Expert
Posts: 189
Liked: 9 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Re: Incrementals of some VMs take as long as full backups

Post by pshute » Jan 17, 2017 4:10 am

Yes, thank you, I see this line in the history for that machine:
"16/01/2017 11:35:06 PM :: CBT data is invalid, failing over to legacy incremental backup. No action is required, next job run should start using CBT again. If CBT data remains invalid, follow KB1113 to perform CBT reset. Usual cause is power loss. "

I'll try resetting CBT.

I would never have spotted that myself, as it's a bit buried in the job report, and it would take a bit of work to check if other machines are affected. Is there an easier way of finding them?

And shouldn't CBT failure be mentioned in the backup job report we get emailed?

foggy
Veeam Software
Posts: 16932
Liked: 1377 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incrementals of some VMs take as long as full backups

Post by foggy » Jan 18, 2017 12:43 pm 2 people like this post

You can monitor job duration using Backup Infrastructure Custom Data Veeam ONE report to be aware of such issues.

Vitaliy S.
Veeam Software
Posts: 21612
Liked: 1290 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incrementals of some VMs take as long as full backups

Post by Vitaliy S. » Jan 18, 2017 1:01 pm

Or if you want to be notified about this behavior in real-time, you can use backup job duration alarm in Veeam ONE, that will notify you when your job has exceeded its usual duration period.

pshute
Expert
Posts: 189
Liked: 9 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Re: Incrementals of some VMs take as long as full backups

Post by pshute » Jan 23, 2017 10:32 pm

foggy wrote:The fact that it reads the entire disk during incremental run tells that CBT isn't working...
Now that I've enabled deleted block exclusion, it's not reading the whole disk. How can I now tell if CBT is working for this disk?

I'd like a way of auditing CBT for all the disks on all our VMs. Monitoring job duration won't help if they've always been failing, as I have no benchmark, nor will it tell me which disk has the problem. Looking for the warning in the job history will only tell me that at least one disk has a CBT problem in that job. It won't tell me how many or which ones.

Would the information be available to a Powershell script, or can I only extract the same information that's in the job histories?

foggy
Veeam Software
Posts: 16932
Liked: 1377 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incrementals of some VMs take as long as full backups

Post by foggy » Jan 24, 2017 10:58 am

pshute wrote:Now that I've enabled deleted block exclusion, it's not reading the whole disk. How can I now tell if CBT is working for this disk?
With CBT in action, Veeam B&R will read just the changes occurred inside the VM, which are typically 3-5% of the total size for regular servers.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 31 guests