How to detect that CBT was reset?

[lrhazi]

Is there something I could look for in backup jobs that would indicate that CBT has been reset and that Veeam is now going to read all the blocks of the VM's VMDKs ?

Support referred me to this line:

Code: Select all

CBT not working:
[24.10.2018 07:41:58] <  9532> dsk|   Loading changed extents from 'C:\Windows\TEMP\{5aa67d14-58ca-403a-9eac-5c2c53cf4e23}\2d72b270-173c-42cc-bce9-fbded3d8da99_ctk.xml'.
[24.10.2018 07:41:58] <  9532> dsk|     Loaded '1' extents ('274877906944' bytes).
[24.10.2018 07:41:58] <  9532> dsk|   Loading changed extents from 'C:\Windows\TEMP\{5aa67d14-58ca-403a-9eac-5c2c53cf4e23}\2d72b270-173c-42cc-bce9-fbded3d8da99_ctk.xml'. ok.

CBT working:
[16.10.2018 05:49:07] <  8012> dsk|   Loading changed extents from 'C:\Windows\TEMP\{d75482df-7758-454f-83f3-b30efc983950}\21efb7a6-d78c-4818-9040-31b596bc6ee9_ctk.xml'.
[16.10.2018 05:49:07] <  8012> dsk|     Loaded '81' extents ('15335424' bytes).
[16.10.2018 05:49:07] <  8012> dsk|   Loading changed extents from 'C:\Windows\TEMP\{d75482df-7758-454f-83f3-b30efc983950}\21efb7a6-d78c-4818-9040-31b596bc6ee9_ctk.xml'. ok.

But searching the logs I found instances where Loaded 1 extent when CBT appears to have worked and only small fraction of the disks were read.

[Gostev]

There's the event that is logged when CBT data is invalid, but I don't believe there's an event about CBT reset. But since the reset can only be performed manually anyway, you'd already know that it happened, right?

[lrhazi]

Well, I don't know actually... What happened is that a job of ours, which backs up 4 VMs, and which is set like all our jobs, to do forever incremental, suddenly took a full backup!
Without calling it a ful backup... it was marked incremental, but looking at the amount of data read in the process, it was about 95% of the data processed, so it did read all the blocks! This is what I am calling "CBT got reset", maybe not the right word?

[lrhazi]

In case it is useful, we use VSphere and NetApp storage with sanpshots enabled.
Support said VSphere must have reset CBT... and am trying to figure out how to detect such event in the future.. other than tracking how much data gets read for each VM...

[lrhazi]

And I checked our logs and the words "CBT data is invalid", as mentioned in that KB, did not occur!

[Vitaliy S.]

Without calling it a ful backup... it was marked incremental, but looking at the amount of data read in the process, it was about 95% of the data processed, so it did read all the blocks! This is what I am calling "CBT got reset", maybe not the right word?

The reason why it is called incremental is that it read the entire image to detect the changes and then re-used your existing backup chain to complete the operation. As Anton said there is no vSphere event for that (at least google didn't return me any results), but you can detect these events by the size/duration of the incremental job pass.

On a side note, you may want to try this predefined Veeam ONE alarm (Max allowed job duration) to get notifications on jobs that ran unexpectedly long time and then review what had caused this. It will not notify you on CBT reset operations, but at least automate the red button alert for you

[foggy]

If CBT cannot be used, there should be an indication in the job session log (no [CBT] tag next to the the reading disk record at the very least). Job debug logs should also contain detailed information (something like "Cannot use CBT", for example). The records you've referenced look like CBT is ok.

[lrhazi]

Thank you guys.

Vitaliy,
It is incremental job, but uses CBT, to avoid having to read the entire image

Hi Foggy,
CBT was used indeed.. it's not that it could not be used... It's that it seems to have reported back that all blocks have changed!

So the job, using CBT, reads say, 10%, or less, of the blocks everyday... then one day it suddenly reads 100%, the next day goes back to the normal 10%ish....
This is the event am trying to figure out how to detect in the logs... I understand I can detect it by tracking the time the job takes to complete and also the amount of data it reads.. but I was wondering if Veeam itself logs this, as a warning for example... This is an incremental backup, we already have the full.. and we usually get small number of blocks changed, and today we are told that all the blocks have changed!

am I making sense, or am I still missing something about how Veeam/CBT works?

[foggy]

So the job, using CBT, reads say, 10%, or less, of the blocks everyday... then one day it suddenly reads 100%, the next day goes back to the normal 10%ish....
This is the event am trying to figure out how to detect in the logs...

This might be caused either by CBT reset (external to Veeam B&R) or by the huge amount of changes inside guest OS (due to AV scan/defrag or something), in both cases you will not find indication of this in the logs, since Veeam B&R just copies everything it is told to copy.

[veremin]

You can try writing a PS script that will:

- Check amount of data read
- Compare it to the amount of data read during the previous job run
- Notify you, if fluctuation is higher than usual

Thanks!

[lrhazi]

Thanks guys.

Suppoet claims that the log I included in my initial post does indicate the event... extent = 1. Supposedly that means the number of extents that did not change is 1, or sometimes 0, and in both cases it would mean something is wrong with CBT.

But from inspecting my logs, am suspicious it is not always true.... I was hoping for an authoritative answer from Veeam people about the matter, but thank you all anyways.

[foggy]

It's the opposite, the number of extents that contain changes is 1 - CBT returns the amount of changes inside guest OS (~256GB in this case). This could be either incremental changes or all blocks have changed (if it is close to the disk size) or CBT reset was performed.