Discussions specific to the VMware vSphere hypervisor
Post Reply
lrhazi
Enthusiast
Posts: 40
Liked: never
Joined: Jan 03, 2017 8:15 pm
Full Name: Mohamed Lrhazi
Contact:

How to detect that CBT was reset?

Post by lrhazi » Oct 28, 2018 12:41 am

Is there something I could look for in backup jobs that would indicate that CBT has been reset and that Veeam is now going to read all the blocks of the VM's VMDKs ?

Support referred me to this line:

Code: Select all

CBT not working:
[24.10.2018 07:41:58] <  9532> dsk|   Loading changed extents from 'C:\Windows\TEMP\{5aa67d14-58ca-403a-9eac-5c2c53cf4e23}\2d72b270-173c-42cc-bce9-fbded3d8da99_ctk.xml'.
[24.10.2018 07:41:58] <  9532> dsk|     Loaded '1' extents ('274877906944' bytes).
[24.10.2018 07:41:58] <  9532> dsk|   Loading changed extents from 'C:\Windows\TEMP\{5aa67d14-58ca-403a-9eac-5c2c53cf4e23}\2d72b270-173c-42cc-bce9-fbded3d8da99_ctk.xml'. ok.

CBT working:
[16.10.2018 05:49:07] <  8012> dsk|   Loading changed extents from 'C:\Windows\TEMP\{d75482df-7758-454f-83f3-b30efc983950}\21efb7a6-d78c-4818-9040-31b596bc6ee9_ctk.xml'.
[16.10.2018 05:49:07] <  8012> dsk|     Loaded '81' extents ('15335424' bytes).
[16.10.2018 05:49:07] <  8012> dsk|   Loading changed extents from 'C:\Windows\TEMP\{d75482df-7758-454f-83f3-b30efc983950}\21efb7a6-d78c-4818-9040-31b596bc6ee9_ctk.xml'. ok.

But searching the logs I found instances where Loaded 1 extent when CBT appears to have worked and only small fraction of the disks were read.

Gostev
SVP, Product Management
Posts: 25070
Liked: 3663 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How to detect that CBT was reset?

Post by Gostev » Oct 28, 2018 8:25 pm

There's the event that is logged when CBT data is invalid, but I don't believe there's an event about CBT reset. But since the reset can only be performed manually anyway, you'd already know that it happened, right?

lrhazi
Enthusiast
Posts: 40
Liked: never
Joined: Jan 03, 2017 8:15 pm
Full Name: Mohamed Lrhazi
Contact:

Re: How to detect that CBT was reset?

Post by lrhazi » Oct 29, 2018 1:49 am

Well, I don't know actually... What happened is that a job of ours, which backs up 4 VMs, and which is set like all our jobs, to do forever incremental, suddenly took a full backup!
Without calling it a ful backup... it was marked incremental, but looking at the amount of data read in the process, it was about 95% of the data processed, so it did read all the blocks! This is what I am calling "CBT got reset", maybe not the right word?

lrhazi
Enthusiast
Posts: 40
Liked: never
Joined: Jan 03, 2017 8:15 pm
Full Name: Mohamed Lrhazi
Contact:

Re: How to detect that CBT was reset?

Post by lrhazi » Oct 29, 2018 1:51 am

In case it is useful, we use VSphere and NetApp storage with sanpshots enabled.
Support said VSphere must have reset CBT... and am trying to figure out how to detect such event in the future.. other than tracking how much data gets read for each VM...

lrhazi
Enthusiast
Posts: 40
Liked: never
Joined: Jan 03, 2017 8:15 pm
Full Name: Mohamed Lrhazi
Contact:

Re: How to detect that CBT was reset?

Post by lrhazi » Oct 29, 2018 1:53 am

And I checked our logs and the words "CBT data is invalid", as mentioned in that KB, did not occur!

Vitaliy S.
Product Manager
Posts: 23174
Liked: 1600 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: How to detect that CBT was reset?

Post by Vitaliy S. » Oct 29, 2018 12:01 pm

lrhazi wrote:Without calling it a ful backup... it was marked incremental, but looking at the amount of data read in the process, it was about 95% of the data processed, so it did read all the blocks! This is what I am calling "CBT got reset", maybe not the right word?
The reason why it is called incremental is that it read the entire image to detect the changes and then re-used your existing backup chain to complete the operation. As Anton said there is no vSphere event for that (at least google didn't return me any results), but you can detect these events by the size/duration of the incremental job pass.

On a side note, you may want to try this predefined Veeam ONE alarm (Max allowed job duration) to get notifications on jobs that ran unexpectedly long time and then review what had caused this. It will not notify you on CBT reset operations, but at least automate the red button alert for you ;)

foggy
Veeam Software
Posts: 18430
Liked: 1587 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How to detect that CBT was reset?

Post by foggy » Oct 29, 2018 12:21 pm

If CBT cannot be used, there should be an indication in the job session log (no [CBT] tag next to the the reading disk record at the very least). Job debug logs should also contain detailed information (something like "Cannot use CBT", for example). The records you've referenced look like CBT is ok.

lrhazi
Enthusiast
Posts: 40
Liked: never
Joined: Jan 03, 2017 8:15 pm
Full Name: Mohamed Lrhazi
Contact:

Re: How to detect that CBT was reset?

Post by lrhazi » Oct 29, 2018 4:41 pm

Thank you guys.

Vitaliy,
It is incremental job, but uses CBT, to avoid having to read the entire image :)

Hi Foggy,
CBT was used indeed.. it's not that it could not be used... It's that it seems to have reported back that all blocks have changed!

So the job, using CBT, reads say, 10%, or less, of the blocks everyday... then one day it suddenly reads 100%, the next day goes back to the normal 10%ish....
This is the event am trying to figure out how to detect in the logs... I understand I can detect it by tracking the time the job takes to complete and also the amount of data it reads.. but I was wondering if Veeam itself logs this, as a warning for example... This is an incremental backup, we already have the full.. and we usually get small number of blocks changed, and today we are told that all the blocks have changed!

am I making sense, or am I still missing something about how Veeam/CBT works?

foggy
Veeam Software
Posts: 18430
Liked: 1587 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How to detect that CBT was reset?

Post by foggy » Oct 29, 2018 4:48 pm

So the job, using CBT, reads say, 10%, or less, of the blocks everyday... then one day it suddenly reads 100%, the next day goes back to the normal 10%ish....
This is the event am trying to figure out how to detect in the logs...
This might be caused either by CBT reset (external to Veeam B&R) or by the huge amount of changes inside guest OS (due to AV scan/defrag or something), in both cases you will not find indication of this in the logs, since Veeam B&R just copies everything it is told to copy.

veremin
Product Manager
Posts: 17047
Liked: 1470 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: How to detect that CBT was reset?

Post by veremin » Oct 29, 2018 4:52 pm

You can try writing a PS script that will:

- Check amount of data read
- Compare it to the amount of data read during the previous job run
- Notify you, if fluctuation is higher than usual

Thanks!

lrhazi
Enthusiast
Posts: 40
Liked: never
Joined: Jan 03, 2017 8:15 pm
Full Name: Mohamed Lrhazi
Contact:

Re: How to detect that CBT was reset?

Post by lrhazi » Oct 29, 2018 10:21 pm

Thanks guys.

Suppoet claims that the log I included in my initial post does indicate the event... extent = 1. Supposedly that means the number of extents that did not change is 1, or sometimes 0, and in both cases it would mean something is wrong with CBT.

But from inspecting my logs, am suspicious it is not always true.... I was hoping for an authoritative answer from Veeam people about the matter, but thank you all anyways.

foggy
Veeam Software
Posts: 18430
Liked: 1587 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How to detect that CBT was reset?

Post by foggy » Oct 30, 2018 12:49 pm

It's the opposite, the number of extents that contain changes is 1 - CBT returns the amount of changes inside guest OS (~256GB in this case). This could be either incremental changes or all blocks have changed (if it is close to the disk size) or CBT reset was performed.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 18 guests