Discussions specific to the VMware vSphere hypervisor
Post Reply
Berniebgf
Service Provider
Posts: 94
Liked: 9 times
Joined: Sep 01, 2010 11:36 pm
Full Name: Bernard Tyers
Contact:

CBT OK - but Full Scan

Post by Berniebgf »

Hi All,

We have a couple of VM's (2) that are performing "full scan's" during backup and replication, there are no Errors with respect to CBT when the job runs but a "Full Scan" is still performed.

When this originally occurred we shutdown the VM's, cleared out a single old snapshot from each VM (1 month old) and did a "CBT reset" as per the Veeam TB, this fixed the issue for a few days but now it is back.
The source ESXi host is a 4.1 early patch level (260XXX), so my feeling is this could be part of the problem.

I will recheck for "stuck" snapshots as see if there are any listed (again) then proceed with the patching of the host.

Can anyone offer advice before I patch the 4.1 host up to the latest level.

thanks

Bernie.

daphnis
Enthusiast
Posts: 47
Liked: 3 times
Joined: Jun 13, 2012 7:01 pm
Contact:

Re: CBT OK - but Full Scan

Post by daphnis »

I'm actually seeing almost the same thing (if not the same) whereby a VM (running Exchange 2010) on which I have reset CBT and run a full backup appears to not ever clear the CBT log. So I'm seeing that I might have 5GB of "Read" data on day 1 after resetting CBT, then on day 2 I'll see 8 GB, then 11 GB etc. etc. Even after Veeam runs it's job, the amount of recorded Read data is always higher than the previous day's backup--it has never decreased. There are no errors or warnings in the job relating to CBT, no snapshots before Veeam runs against this VM, and no other strange settings (that I can find) on this VM. I've opened a case with Veeam support and they tell me it needs to go to VMware, which I'll probably do tomorrow.

Berniebgf
Service Provider
Posts: 94
Liked: 9 times
Joined: Sep 01, 2010 11:36 pm
Full Name: Bernard Tyers
Contact:

Re: CBT OK - but Full Scan

Post by Berniebgf »

Hi Daphnis,

What Version and Build of VMware are you seeing this on?

Bernie

daphnis
Enthusiast
Posts: 47
Liked: 3 times
Joined: Jun 13, 2012 7:01 pm
Contact:

Re: CBT OK - but Full Scan

Post by daphnis »

Veeam B&R 7.0 (same behavior with GA and R2) and vSphere 5.1 U1.

foggy
Veeam Software
Posts: 19285
Liked: 1743 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT OK - but Full Scan

Post by foggy »

Do you see the [CBT] tag next the hard disk processing rate in the job log? What are the processed/read/transferred counts for those VMs?

daphnis
Enthusiast
Posts: 47
Liked: 3 times
Joined: Jun 13, 2012 7:01 pm
Contact:

Re: CBT OK - but Full Scan

Post by daphnis »

Yes, as I keep telling support, Veeam shows all is good for the job containing this VM, and CBT is always used (or so it says).

Before resetting CBT on this VM, I was seeing Size (no column named "Processed") / Read / Transferred as: 560 GB / 470.3 GB / 15.1 GB. After resetting CBT and once the first job had run, here are some log entries for entries thereafter. Each job runs daily so these figures take into consideration about 24 hours of changes to a single VM:

560 / 34.9 / 21.7
560 / 45.8 / 12
560 / 59 / 21.1 (this one was two days removed from the previous)
560 / 66.6 / 14.9
560 / 70.5 / 13.5
560 / 75 / 13.6
560 / 78.6 / 14.9
560 / 80.4 / 8.7

As you can see, the "Read" amount is always greater than the previous value and therefore the processing time. But the "Transferred" is variable--sometimes more, sometimes less.

foggy
Veeam Software
Posts: 19285
Liked: 1743 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT OK - but Full Scan

Post by foggy »

Assuming this VM has Exchange installed inside, this is pretty much expected. Exchange Server VMs always generate large increments because it is highly transactional application and perform many small changes across the entire disk.

daphnis
Enthusiast
Posts: 47
Liked: 3 times
Joined: Jun 13, 2012 7:01 pm
Contact:

Re: CBT OK - but Full Scan

Post by daphnis »

Yes, I understand that the increments are generally large because of the nature of the transactions and their impact at the block level, but I have Veeam backing up over a dozen other Exchange 2010 servers and have never observed this before. It is normal for Exchange to generate large incremental changes, but what isn't normal is for the Read rate to not be reset after incremental backups. The fact that those numbers are always greater than the previous tells me there's something not right with the CBT engine.

foggy
Veeam Software
Posts: 19285
Liked: 1743 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT OK - but Full Scan

Post by foggy »

Don't take the Read counter value as accumulated amount of data. This is "newly" changed data between the job cycles that was returned by VMware CBT and has to be processed.

daphnis
Enthusiast
Posts: 47
Liked: 3 times
Joined: Jun 13, 2012 7:01 pm
Contact:

Re: CBT OK - but Full Scan

Post by daphnis »

Yeah I get that, but you're saying that somehow after each job runs the Read counter is always higher than the previous job...for almost two weeks, and this is legit? I don't buy it...

foggy
Veeam Software
Posts: 19285
Liked: 1743 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT OK - but Full Scan

Post by foggy »

Yes, it should not be always higher. And according to your stats the gap decreases each day, so probably it will come to an acceptable level.

daphnis
Enthusiast
Posts: 47
Liked: 3 times
Joined: Jun 13, 2012 7:01 pm
Contact:

Re: CBT OK - but Full Scan

Post by daphnis »

Not sure you've seen my post, but the "Read" column is the middle column, and dates are from old to new starting at the top. The Read value is always increasing, not decreasing despite the Transferred amount being flexible.

foggy
Veeam Software
Posts: 19285
Liked: 1743 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT OK - but Full Scan

Post by foggy »

I was talking about the decreasing "gap" in the amount of data read between job cycles. It was 11GB between 1st and 2d runs and it is less than 2GB between the last two job runs. Sorry for not being clear enough.

daphnis
Enthusiast
Posts: 47
Liked: 3 times
Joined: Jun 13, 2012 7:01 pm
Contact:

Re: CBT OK - but Full Scan

Post by daphnis »

But that's still not correct. When an incremental runs against a VM with persistent disks, the CBT log should be reset so the Read amount should be reset to 0. Then the changed blocks start getting counted.

foggy
Veeam Software
Posts: 19285
Liked: 1743 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT OK - but Full Scan

Post by foggy »

I assume that CBT might not work properly in this case, it's just not 100% clear from the stats you've posted (especially keeping Exchange in mind). Have you observed similar increase of the Read counter value before resetting CBT (reaching the 470.3 GB mark)?

daphnis
Enthusiast
Posts: 47
Liked: 3 times
Joined: Jun 13, 2012 7:01 pm
Contact:

Re: CBT OK - but Full Scan

Post by daphnis »

I don't have accurate figures from before that time, but around the 470.3 GB mark for "Read" value, it sat right around that number every time the job ran. This obviously meant that Veeam had to run a manual check against every block, which even though the transferred amount would be 8 GB or less sometimes, it would still take 3+ hours to process that VM.

emachabert
Veeam Vanguard
Posts: 373
Liked: 165 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: CBT OK - but Full Scan

Post by emachabert »

I guess this exchange vm is configured to continuously do the database maintenance (defrag) so that all your DB get scanned everyday.
Try switching to weekly maintenance, eg: through saturday to sunday, this way only week end backups will be heavy, and you won't notice any performance drawback by defragmenting once a week.
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

Post Reply

Who is online

Users browsing this forum: No registered users and 22 guests