CBT OK - but Full Scan

[Berniebgf]

Hi All,

We have a couple of VM's (2) that are performing "full scan's" during backup and replication, there are no Errors with respect to CBT when the job runs but a "Full Scan" is still performed.

When this originally occurred we shutdown the VM's, cleared out a single old snapshot from each VM (1 month old) and did a "CBT reset" as per the Veeam TB, this fixed the issue for a few days but now it is back.
The source ESXi host is a 4.1 early patch level (260XXX), so my feeling is this could be part of the problem.

I will recheck for "stuck" snapshots as see if there are any listed (again) then proceed with the patching of the host.

Can anyone offer advice before I patch the 4.1 host up to the latest level.

thanks

Bernie.

[daphnis]

I'm actually seeing almost the same thing (if not the same) whereby a VM (running Exchange 2010) on which I have reset CBT and run a full backup appears to not ever clear the CBT log. So I'm seeing that I might have 5GB of "Read" data on day 1 after resetting CBT, then on day 2 I'll see 8 GB, then 11 GB etc. etc. Even after Veeam runs it's job, the amount of recorded Read data is always higher than the previous day's backup--it has never decreased. There are no errors or warnings in the job relating to CBT, no snapshots before Veeam runs against this VM, and no other strange settings (that I can find) on this VM. I've opened a case with Veeam support and they tell me it needs to go to VMware, which I'll probably do tomorrow.

[Berniebgf]

Hi Daphnis,

What Version and Build of VMware are you seeing this on?

Bernie

[daphnis]

Veeam B&R 7.0 (same behavior with GA and R2) and vSphere 5.1 U1.

[foggy]

Do you see the [CBT] tag next the hard disk processing rate in the job log? What are the processed/read/transferred counts for those VMs?

[daphnis]

Yes, as I keep telling support, Veeam shows all is good for the job containing this VM, and CBT is always used (or so it says).

Before resetting CBT on this VM, I was seeing Size (no column named "Processed") / Read / Transferred as: 560 GB / 470.3 GB / 15.1 GB. After resetting CBT and once the first job had run, here are some log entries for entries thereafter. Each job runs daily so these figures take into consideration about 24 hours of changes to a single VM:

560 / 34.9 / 21.7
560 / 45.8 / 12
560 / 59 / 21.1 (this one was two days removed from the previous)
560 / 66.6 / 14.9
560 / 70.5 / 13.5
560 / 75 / 13.6
560 / 78.6 / 14.9
560 / 80.4 / 8.7

As you can see, the "Read" amount is always greater than the previous value and therefore the processing time. But the "Transferred" is variable--sometimes more, sometimes less.

[foggy]

Assuming this VM has Exchange installed inside, this is pretty much expected. Exchange Server VMs always generate large increments because it is highly transactional application and perform many small changes across the entire disk.

[daphnis]

Yes, I understand that the increments are generally large because of the nature of the transactions and their impact at the block level, but I have Veeam backing up over a dozen other Exchange 2010 servers and have never observed this before. It is normal for Exchange to generate large incremental changes, but what isn't normal is for the Read rate to not be reset after incremental backups. The fact that those numbers are always greater than the previous tells me there's something not right with the CBT engine.

[foggy]

Don't take the Read counter value as accumulated amount of data. This is "newly" changed data between the job cycles that was returned by VMware CBT and has to be processed.

[daphnis]

Yeah I get that, but you're saying that somehow after each job runs the Read counter is always higher than the previous job...for almost two weeks, and this is legit? I don't buy it...

[foggy]

Yes, it should not be always higher. And according to your stats the gap decreases each day, so probably it will come to an acceptable level.

[daphnis]

Not sure you've seen my post, but the "Read" column is the middle column, and dates are from old to new starting at the top. The Read value is always increasing, not decreasing despite the Transferred amount being flexible.

[foggy]

I was talking about the decreasing "gap" in the amount of data read between job cycles. It was 11GB between 1st and 2d runs and it is less than 2GB between the last two job runs. Sorry for not being clear enough.

[daphnis]

But that's still not correct. When an incremental runs against a VM with persistent disks, the CBT log should be reset so the Read amount should be reset to 0. Then the changed blocks start getting counted.

[foggy]

I assume that CBT might not work properly in this case, it's just not 100% clear from the stats you've posted (especially keeping Exchange in mind). Have you observed similar increase of the Read counter value before resetting CBT (reaching the 470.3 GB mark)?

[daphnis]

I don't have accurate figures from before that time, but around the 470.3 GB mark for "Read" value, it sat right around that number every time the job ran. This obviously meant that Veeam had to run a manual check against every block, which even though the transferred amount would be 8 GB or less sometimes, it would still take 3+ hours to process that VM.

[emachabert]

I guess this exchange vm is configured to continuously do the database maintenance (defrag) so that all your DB get scanned everyday.
Try switching to weekly maintenance, eg: through saturday to sunday, this way only week end backups will be heavy, and you won't notice any performance drawback by defragmenting once a week.