Incrementals of some VMs take as long as full backups

VMware specific discussions

Incrementals of some VMs take as long as full backups

Veeam Logoby pshute » Fri Jan 13, 2017 4:16 am

Can anyone please tell me why some of my servers take just as long to do an Incremental backup as an Active Full, despite the TransferredSize being much smaller.

Some example durations for one of the servers concerned (you can tell which are incrementals by the smaller Transferred Size):
DurationReadSize in GBTransferredSize in GB
4:29:01853.8485.8
4:54:11853.8490.8
4:57:16833.321.9
4:55:18813.422.3
5:04:04817.828.3
4:28:31761.931.4
4:46:14836.333.4
4:54:49836.935.7
4:23:10763.643.3
4:41:17817.947.4
4:41:31813.8484.1

For that particular server, the history entries show that one particular operation takes most of the time:
Full 11/01/2017 9:52:27 AM :: Hard disk 7 (600.0 GB) 600.0 GB read at 45 MB/s [CBT] - duration 3:51:06
Incr 13/01/2017 12:22:29 AM :: Hard disk 7 (600.0 GB) 600.0 GB read at 45 MB/s [CBT] - duration 3:43:56

Does that mean the problem is to do with how long it's taking to read that disk? I have other servers nearly as big that are much quicker doing incrementals than fulls.
pshute
Expert
 
Posts: 118
Liked: 8 times
Joined: Mon Nov 23, 2015 10:56 pm
Full Name: Peter Shute

Re: Incrementals of some VMs take as long as full backups

Veeam Logoby Mike Resseler » Fri Jan 13, 2017 5:41 am

Peter,

Can you tell us what the bottleneck is? It seems indeed that it is a bit slow. How is the disk read of your other servers?
Mike Resseler
Veeam Software
 
Posts: 3381
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Incrementals of some VMs take as long as full backups

Veeam Logoby Shestakov » Fri Jan 13, 2017 9:16 am

Do other VMs use same datastore and proxy servers?
Thanks!
Shestakov
Veeam Software
 
Posts: 5130
Liked: 430 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Incrementals of some VMs take as long as full backups

Veeam Logoby foggy » Fri Jan 13, 2017 10:02 pm

The fact that it reads the entire disk during incremental run tells that CBT isn't working...
foggy
Veeam Software
 
Posts: 15282
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Incrementals of some VMs take as long as full backups

Veeam Logoby pshute » Tue Jan 17, 2017 4:10 am

Yes, thank you, I see this line in the history for that machine:
"16/01/2017 11:35:06 PM :: CBT data is invalid, failing over to legacy incremental backup. No action is required, next job run should start using CBT again. If CBT data remains invalid, follow KB1113 to perform CBT reset. Usual cause is power loss. "

I'll try resetting CBT.

I would never have spotted that myself, as it's a bit buried in the job report, and it would take a bit of work to check if other machines are affected. Is there an easier way of finding them?

And shouldn't CBT failure be mentioned in the backup job report we get emailed?
pshute
Expert
 
Posts: 118
Liked: 8 times
Joined: Mon Nov 23, 2015 10:56 pm
Full Name: Peter Shute

Re: Incrementals of some VMs take as long as full backups

Veeam Logoby foggy » Wed Jan 18, 2017 12:43 pm 2 people like this post

You can monitor job duration using Backup Infrastructure Custom Data Veeam ONE report to be aware of such issues.
foggy
Veeam Software
 
Posts: 15282
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Incrementals of some VMs take as long as full backups

Veeam Logoby Vitaliy S. » Wed Jan 18, 2017 1:01 pm

Or if you want to be notified about this behavior in real-time, you can use backup job duration alarm in Veeam ONE, that will notify you when your job has exceeded its usual duration period.
Vitaliy S.
Veeam Software
 
Posts: 19954
Liked: 1144 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Incrementals of some VMs take as long as full backups

Veeam Logoby pshute » Mon Jan 23, 2017 10:32 pm

foggy wrote:The fact that it reads the entire disk during incremental run tells that CBT isn't working...

Now that I've enabled deleted block exclusion, it's not reading the whole disk. How can I now tell if CBT is working for this disk?

I'd like a way of auditing CBT for all the disks on all our VMs. Monitoring job duration won't help if they've always been failing, as I have no benchmark, nor will it tell me which disk has the problem. Looking for the warning in the job history will only tell me that at least one disk has a CBT problem in that job. It won't tell me how many or which ones.

Would the information be available to a Powershell script, or can I only extract the same information that's in the job histories?
pshute
Expert
 
Posts: 118
Liked: 8 times
Joined: Mon Nov 23, 2015 10:56 pm
Full Name: Peter Shute

Re: Incrementals of some VMs take as long as full backups

Veeam Logoby foggy » Tue Jan 24, 2017 10:58 am

pshute wrote:Now that I've enabled deleted block exclusion, it's not reading the whole disk. How can I now tell if CBT is working for this disk?

With CBT in action, Veeam B&R will read just the changes occurred inside the VM, which are typically 3-5% of the total size for regular servers.
foggy
Veeam Software
 
Posts: 15282
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 1 guest