CBT issue? - entire disk gets processed every few days

[frodooftheshire]

Hi everyone,

I'm having an intermittent CBT issue with only one client, out of many that are using Veeam/ESXi, and I'm wondering it's related to 8.0u2 as they're currently only client I have running this exact version of ESXi. I'm running Veeam B & R 12.1.0.2131 and backing up a single VM located on an ESXi host running 8.0 u2 (not patched with the most recent fix yet). On the VM there are only two drives (OS & Data). The "Data" drive is storing a QuickBooks company file and a few other files, but nothing crazy that's amounting to a lot of data changes. The total data on this drive is around 25 GB. Randomly, and sometimes several times a day, hourly incremental backups are reading/backing up the whole drive - in the log it shows hard disk 2 - 500 GB 24.5 GB read vs 360 MB or 800 MB for example. So, in a day, out of 8 incremental hourly backups, most of them will be between 100 -700 MB and then it shoots up to 24 GB. Sometimes I'll have 3 or 4 back-to-back incremental be 24 GB in size. Is there any chance that this CBT bug could be causing my backups to blow up in size randomly? What's odd is that it will sometimes go multiple days without any issues at all. I opened a ticket with Veeam support and they told me to inquire with VMware but then in my search I discovered this bug and wondered if the two could be connected. Curious what you guys thought. Thanks!

[Sturniolo]

Hi @frodooftheshire

Can you please provide the support ticket number you have for this case so we can look into it?

Thanks

[frodooftheshire]

Absolutely. Case # was 07135147. I thought the issue had resolved itself because I went about a 7 - 10 days without any large backups after deleting some older unneeded files/shares and then bam - they came back. Also, I attempted to open a support case with VMware but was unsuccessful because apparently while we're under contract for software updates, we don't have any support purchased for this customer. I attempted to buy support directly from VMware but support said we had to buy from a reseller, which because of the Broadcom buyout, I'm dreading because I'm guessing it will be much more expensive then if it was purchased up front. Of course if it comes to that we'll do what we need to do. Either way this issue is just so odd. Again, I can't tell you how little utilization this server gets. The only thing that's unique to this client's setup vs all the other ESXi hosts I manage is that a.) they're on version 8 (all other clients are on 7) and that they have a Dell Boss card in leu of dual SD cards for ESXi itself. Data is still stored on raid 1 SSDs connected to Perc.

[Origin 2000]

.. Sometimes I'll have 3 or 4 back-to-back incremental be 24 GB in size.

So the vDisk is a 500GB one but only 25 of data on? Anyway.... how likely is that there is some kind of script/task that touch or optimize all of the 25GB for some kind of reason? I have seen some kind of Databases than try an optimize ever week, try and internal backup or when it comes to an enabled Windows Dedup the garbage and optime task will touch every bit on a disk.

Regards,
Joerg

[frodooftheshire]

We sized the new server's "data" drive off the previous server and when it came to transferring files we discovered about 200 GB didn't need to be moved over and then we discovered another 50 GB or so was no longer necessary after the transfer. And yes, no databases stored on this drive. This company runs an application called CCC One which is a collision repair software. It downloads application files that get distributed to other computers in the office, but this only happens about once every two weeks. There are no scripts/tasks running and Windows deduplication is not enabled/configured.

[Andreas Neufert]

Based on the support case you enabled data deduplication from Microsoft on this volume. If this is the case then the described changes on disk level are expected as Microsoft reorganizes data frequently.
There are many similar documented cases with this.

[Origin 2000]

If Windows Dedup is anabled than the windows schedudeld Task will execute certain powershelll scripts for maintaining the dedup like running garbage collection, optimizations and more. There might be an additional thing that per default these scripts runs in the same period as your Veeam backup which means that a VM Snapshot can increase in size rapidly. Think about a 5 TB file server where a Active full take some time and in the same period a dedup maintanance script run. It can fill a VMFS Datastore in minutes

If you only have 25GB after your cleanup you should run the needed powershell cmdlet to dehydrate your data and than disable the dedup.

Regards,
Joerg

[Mildur]

Hello James

I split your discussion from the vSphere 8.0u2 topic.
Please continue your discussion here.

Best,
Fabian

[DonZoomik]

I have run Windows guest dedup VM backups for a long time (since the feature came out about a decade ago). Depending on your change rate, increased incremental size is expected.
The main point is to disable Full garbage collection that runs about once a month that may cause very large CBT delta, by design. I have discussed the issue many times in the forum, for example here: post366241.html#p366241

[frodooftheshire]

Based on the support case you enabled data deduplication from Microsoft on this volume. If this is the case then the described changes on disk level are expected as Microsoft reorganizes data frequently.
There are many similar documented cases with this.

That's not true. I said in my ticket that deduplication is *not* an enabled feature on this VM.

[Andreas Neufert]

I asked the Team to check the logs if there is a CBT reset that causes to read all data (CBT giving us a * for reading everything) or if CBT really tells us that all this has changed. This can take some time.

[Andreas Neufert]

Hi James, as your old case is closed, the logs where purged. Can you please open a new case and upload logs again. This will give us a chance to look at the CBT information we have received from VMware as described above. Please post new support case number here then. Thanks.

[frodooftheshire]

Hi everyone,

My apologies for the delay. I've created a new case with number #07199710. I uploaded the past logs as well as the most recent logs. The random large incremental backups are still occurring, but worse, I'm now getting a FcWriteFileEX error. I've read online that can be related to errors for cached credentials, but I tested the credentials for application aware processing, and they test just fine.

[Andreas Neufert]

To summarize and follow up a bit here:
Based on logs support stated that something is blocking access to the files and their first guess was Antivirus not having Veeam backup files excluded.
Feedback later from customer was that there was a hardware/driver issue and that it worked after some updates.
For now backups worked.