Job Warning regarding CBT

[erossman]

Hi Guys,

Some of my jobs are finished with the result "warning". The reason for this is one or more vm have a CBT issue.

"Changed block tracking cannot be enabled one or more snapshots present"

I know this is a limitation of CBT. But how can I suppress this warning. I don't wanna see a warning message in the job state.
Please help me.


Regards,
Eric

[foggy]

Hi Eric, I'm not sure this warning can be suppressed. Any chance you could remove the snapshots to be able to perform the first job run without them, to allow for CBT to be used further?

[Vitaliy S.]

As a workaround, you can regroups VMs running on the snapshot and place them to a separate job and disable CBT option for them.

[Stephan23]

Has there been an update on that matter?
I have one job with 300 VMs that generates 80 warnings every day, because CBT can't be enabled.
I would like to suppress only this specific warning for only this job but leave the option to enable CBT activated.

[Egor Yakovlev]

No updates. Split those VMs in a separate job or let us run a single backup without snapshots - you can reuse snapshots afterwards, for whatever reason you use them in the first place.
/Thanks!

[bhnelson]

I would like to +1 this request.

Actually, what I think would be best: The first time this occurs (per VM) post it as a warning. Each subsequent time, post as information only. And of course, it should reset after CBT is later disabled again (as per the new 'reset CBT on full backup' setting).

Thanks

[soncscy]

To those wanting it: out of curiosity...why?

This is a conversation I have with clients frequently, and usually it's days of explaining "look, you need to drop the snapshots; this is why your VMs are having trash performance".

VMs need to not be on snapshots. If my backup app is not warning me about snaps, what good is it? Snapshots are tremendously misunderstood, and VMware/Storage Vendors don't help with this conversation when it comes up because of sloppy marketing.

What is the benefit of not having such a warning? In fact, I'd love it if Veeam could include a toggle to flag/warn when it detects a VM is running on snaps, as it must know with the snapshot hunter, and it's far more active about such things than VMware who just passes it off to monitoring tools.

[Stephan23]

out of curiosity...why?

Part of what I'm backing up is a View cluster that is manged by another department. From my understanding (I'm not super familiar with View) snapshots are essential for the pools and cannot be removed without significant effort.

Edit: That's the reason why I want it to be an option per job. I agree with you, that there shouldn't be snapshots for an extended time for normal server VMs and like to be reminded with a warning every single time.

[Egor Yakovlev]

It will be worth checking with that department if backup of VMware Horizon(former View) VMs even needs a backup at all - in most of the cases those machines are non-persistent and created\deleted on the fly. Main idea behind VDI is to create disposable machines from a golden corporate desktop image and it is sufficient to protect just that image instead of all clones deployed from it.
If said Horizon machines do hold persistent data, you can always switch to Veeam Agent with in-guest backup of essential data - that will use in-guest CBT instead of relying on VMware CBT, and will use in-guest VSS snapshot instead of relying on VMware Snapshots.
/Cheers!

[bhnelson]

If my backup app is not warning me about snaps, what good is it?

I disagree. It's not my backup software's job or business to keep tabs on 'potential performance issues' in my virtual infrastructure. Rather, that's the job of my monitoring/alerting software.

Given, in this case the inability to re-enable CBT is a potential backup performance issue, so a warning is reasonable. But do I really need to hear about it every time a backup runs? Yes. I know. There's snapshots. They need to be there for now. At least give me some option(s) to suppress the warning so I can stop hearing about it.

As a corollary, take Veeam's treatment of independent disks. It will warn you ONE TIME that an independent disk won't be backed up, then never speak of it again. THIS seems bonkers to me, as it is a 'backup incomplete' situation with only one easily overlooked and forgotten notification. CBT/snaps don't affect the backup actually working. At worst they may affect performance.

[soncscy]

See something, say something

Hypervisor backup solutions must know about snapshots because of their use of CBT. Since VMware hasn't deigned to give proper warnings about snapshots/orphaned snapshots (despite their frequent articles about how you must not keep VMware snaps running), I'll take any solution that's willing to give a free heads-up about snapshots, orphaned or otherwise. Often, the backup solution finds these snapshots, or at least reports them more prominently, sooner than most monitoring softwares.

Similarly, I absolutely want to know when the backup window is going to be impacted. Even for my larger business clients we monitor, backup windows are tight, and anything that potentially affects the window I want to know about (or someone from my team).

Unaddressed snapshots are a plague, and anything that gets attention to snapshots is welcomed in my book.

[wishr]

Hi Harvey,

You may want to take a look at the B&R's Snapshot Hunter technology and ONE's Orphaned VM snapshots report.

Thanks

[Stephan23]

It will be worth checking with that department if backup of VMware Horizon(former View) VMs even needs a backup at all - in most of the cases those machines are non-persistent and created\deleted on the fly. Main idea behind VDI is to create disposable machines from a golden corporate desktop image and it is sufficient to protect just that image instead of all clones deployed from it.
If said Horizon machines do hold persistent data, you can always switch to Veeam Agent with in-guest backup of essential data - that will use in-guest CBT instead of relying on VMware CBT, and will use in-guest VSS snapshot instead of relying on VMware Snapshots.
/Cheers!

The non persistent clones are not not beieng backed up, there are 40 or so golden images. Also a lot of test / dev machines, that requires snapshots.
It's not like a regular VDI environment.

[Stephan23]

I updated to V11 (11.0.0.837) and no longer get those warnings. Has this been changed? Is this by design? I didn't notice any mentioning of that in the release notes.

[wishr]

Hi Stephan,

Yes, there were some improvements to the processing logic in v11 made as a part of bugfix activities.

Thanks

[Stephan23]

Wait. I just noticed it's not the warning that isn't shown any more, it's actually enabling cbt on machines with snapshots. That's awesome. 6-7 hours pre V11 are now 2 hours for that job I was talking about above.

[wishr]

Long story short, there was a bug on the VMware side causing potential data corruption which was forcing us to keep disabling CBT and raising that warning. Since now we have more details on what vSphere versions are affected, we've removed a few restrictions previously saving users from the worst of two evils. Now the warning should only appear for vSphere 6.5 and only in certain rare circumstances.

For future readers: if you are still getting the warning when on VBR v11 and think it should not be there - please raise a support case so we could collect more information. Please, don't forget to share your case ID here. Thanks.

[filippoAdb]

I recently moved from Veaam B&R 9.5 to current version 11.0.
A backup job has always reported a warning because a vm has a snapshot, so CBT cannot be used.
Here is the relevant part of the activity log:

Code: Select all

VM size: 12.0 GB 
Getting VM info from vSphere 
Changed block tracking cannot be enabled: one or more snapshots present. 
Creating VM snapshot 
Saving [Datastore_MSA1050_01] indicatori SPI python/indicatori SPI python.vmx 
Saving [Datastore_MSA1050_01] indicatori SPI python/indicatori SPI python.vmxf 
Saving [Datastore_MSA1050_01] indicatori SPI python/indicatori SPI python.nvram 
Using backup proxy VMware Backup Proxy for disk Hard disk 1 [hotadd] 
Hard disk 1 (12.0 GB) 12.0 GB read at 312 MB/s

The same job now doesn't give any warning:

Code: Select all

VM size: 12 GB 
Getting VM info from vSphere 
Creating VM snapshot 
Saving [Datastore_MSA1050_01] indicatori SPI python/indicatori SPI python.vmx 
Saving [Datastore_MSA1050_01] indicatori SPI python/indicatori SPI python.vmxf 
Saving [Datastore_MSA1050_01] indicatori SPI python/indicatori SPI python.nvram 
Using backup proxy VMware Backup Proxy for disk Hard disk 1 [hotadd] 
Hard disk 1 (12 GB) 259 MB read at 259 MB/s [CBT]

Besides from the line

Hard disk 1 (12 GB) 259 MB read at 259 MB/s [CBT]

it seems CBT is used.

I read many people complained about that warning and asked to remove it, so initially I thought B&R changed behavior, but couldn't find any confirmation. Anyway the vm still has a snapshot, so according to my knowledge CBT cannot be used.
Am I missing something ? Should I open a support case ?

Thanks
Filippo

[wishr]

Hi FilippoAdb,

Welcome to Veeam R&D Forums!

I'm merging your post with an existing thread - please take a look.

Thanks