cktEnabled gets set to false after backup

[dkiernan]

I have one VM (the Veeam server itself) that I can *only* get Changed Block Tracking enabled by manually setting it and running a backup.

Then when the next scheduled Veeam backup runs, the CBT is disabled.

The server has one disk ( C: ), 75gb, 5gb free

Shut down VM
enable cktEnabled, generally and for each scsi disk (scsi0:0)
power on vm
run backup - runs with CBT, completes in 45min - 14gb read

next backup - 3hrs, 70gb read
Check cktEnabled - set to false, generally and on scsi0:0

How can I keep CBT enabled for this VM?

Let me know if you need more info!

[kjc3303]

Is the Veeam server a HotAdd Proxy? CBT is disabled on a Proxies that use hotadd proxy mode. This is the case even if the proxy is not being used for this particular job.

To enable CBT either remove the proxy service from the VM or use a different transport mode.

[veremin]

May I ask you if you’re backing up VB&R server, using default proxy server which is deployed on it?

If so, there is no wonder why CBT can’t be enabled, since whenever you’re trying to backup VB&R server via default proxy, CBT is automatically disabled and it’s VB&R proprietary mechanism that is being used.

This ad-hoc mechanism works like this:

1. VB&R scans through the full VM image and calculates a checksum for every data block.
2. After that, it checks what blocks have been changed on your VM in comparison with the VM image stored in the backup file.

In the way I see it, that’s why you see such a high read value.

Hope this helps.
Thanks.

[kjc3303]

I have found that even if the default proxy isn't used when backing up the VBR server but has the proxy service installed on it (Hotadd mode) then CBT will always be disabled so this will need to be addressed too.

Explanation courtesy of Tom Sightler

"Yes, as a workaround for the hotadd CBT "stall" all backup proxies now have CBT disabled automatically. I don't know if there is a way to change this behavior if you were not having CBT "stall" issue during hotadd, but it would be nice, as disabling CBT for proxies works against customers who were using existing VMs as their proxy servers. It's much less of an issue for customers using dedicated proxies as those VMs are typically not worth backing up anyway and even if you want to back them up they are small."

[veremin]

Furthermore, what Kevin said also seems reasonable.

If your VB&R server is being used as Hot-Add proxy for some other job, it inevitably results in the lack of CBT.

From my perspective, this process might have looked like this:

1) Firstly, you manually enabled CBT on your VB&R virtual machine.
2) Then you performed backup of this VM, using different proxy server.
3) Backup process was finished and after a while other backup job took place, however, this time using VB&R server as a proxy. Specifically in this moment, when VB&R was selected as proxy, CBT was disabled on it.

So, CBT had been already disabled, when you tried to backup up VB&R server second time and proprietary VB&R mechanism was used. It should explain the “read” difference you saw.

Hope this helps.
Thanks.

[dkiernan]

Wow - great information! Thanks everyone.

My follow-up question is whether I can get the VB&R server backed up *without* having it read all 70GB?

Is this a known "penalty" when backing up the VB&R server itself?

There is only a "default proxy" for the VB&R server itself, if that's what you mean. Let me know if you need more info on that.

[veremin]

My follow-up question is whether I can get the VB&R server backed up *without* having it read all 70GB?

Yes, you can. However, be sure that you back it up, using different proxy server, not the default one, and that VB&R server isn’t used as Hot-Add proxy for any other job.

Is this a known "penalty" when backing up the VB&R server itself?

As described in my first post, CBT is automatically disabled, whenever you’re trying to backup VB&R server, using default proxy that is deployed on it, and it is VB&R proprietary mechanism that is being used in this case.

There is only a "default proxy" for the VB&R server itself, if that's what you mean.

Actually, the situation is a lit bit different. The role of default proxy is played by VB&R server, nevertheless, you can easily deploy/use whatever number of additional proxies you want.

Hope this helps.
Thanks.

[kjc3303]

To avoid the 70gb read you will have to uninstall the proxy from the VBR server too, as Vladimir says you will need to deploy a proxy on a different server.
You will never be able to use CBT on a server that has the proxy service installed on it even if the proxy is not being used for this particular job.

Not sure what happens if you disable the proxy rather than remove it, I shall test and post results on here if one of the Veeam guys do not confirm before I have chance.

Kev

[foggy]

You will never be able to use CBT on a server that has the proxy service installed on it even if the proxy is not being used for this particular job.

Actually, this is not 100% true. If the proxy is not used as hotadd, you can enable CBT manually and it will not be disabled anymore (until the moment this proxy is selected as hotadd proxy in one of the jobs). So you can explicitly set the transport mode for the default proxy server to Network and Veeam B&R will backup itself using CBT (however, all other VMs will be backed up using Network mode also).

[kjc3303]

Correct, should have made that more clear, I did cover that in my original post

"To enable CBT either remove the proxy service from the VM or use a different transport mode."

How about disabling the proxy would that have the same effect? I know it's not needed as if the proxy is not used you would just set to Network mode mode although if you wanted to play about it's easier and more noticeable if you can just right click the proxy and disable.

Kev

[foggy]

Of course, you can just disable the default proxy, enable CBT on it manually and use another proxy to back it up with CBT (since the proxy is disabled, it will not be selected as hotadd proxy anymore, hence CBT will not be disabled).

[kjc3303]

Cheers foggy

[rsargeant]

Noticed VMware CBT seems disabled when backing up or replicating VM running Veeam B&R, because [CBT] doesn't appear after virtual disk?

This is happening to a number of Veeam 6.x installs we support, so I guess there's a reason and this is normal behaviour?

Any way of forcing VMware CBT, as jobs run really slow in comparsion to those with VMware CBT support??

Thanks guys.

R.

[veremin]

Hi, Richard. Most likely VB&R is being used in your case as a Hot-ADD proxy. It, in its turn, results in lack of CBT. In order to bypass this, you can explicitly specify default proxy to use Network mode, and also manually enable CBT on it.

Kindly check the answers provided above for more information regarding this issue.

Thanks.

[foggy]

Another reason for CBT being disabled for the VM is at least one existing snapshot.

[rsargeant]

Hi guys,

In most cases I tend to disable Hot-Add on Default and extra Proxies, and go for Direct SAN or Network, so this isn't the reason. Also the VM definitely has no
other snapshots except the one taken for the running job.

This behavior is happening with all our Veeam 6.x installs. Note: all Veeam installs are running as VMs (both on Windows 7 and 2008 R2).

Any ideas?

Cheers,

R.

[veremin]

Can you manually enable CBT on this machine and see whether it helps or not? Thanks.

[rsargeant]

CBT is already enabled in Configuration Parameters, already checked this.

I'm not getting the 'Soap CBT' errors which you might get with a 'broken job', maybe caused by power failure or hung Veeam server etc where you need to manually reset the ctk parameters.

I just get no [CBT] appearing after each virtual disk on these VMs???

[nefes]

It is worth opening support case then, since the reason of CBT not working can be found from reviewing job logs.

[foggy]

Might be a silly question, but are these VMs turned on during backup/replication? Have you tried to reset the CBT manually on them? During what job runs do you observe this behavior: full or incremental or both?

[rsargeant]

This is happening on all of our customers running v6.x, so if it's a bug,I'm amazed we are the only one to report this problem???

To recap:

Veeam Backup & Replication is running on a VM running Windows 2008 R2 or Windows 7;
All other VMs within jobs use [CBT] during jobs EXCEPT the VM running Veeam Backup & Replication;
Proxies are set to Network Mode;
CTK parameters are enabled;
VM running Veeam Backup & Replication has no Snapshots (other than when jobs starts);

All very strange...

R.

[veremin]

As mentioned above, kindly open a ticket with our support team and let them investigate it directly. Thanks.

[CharadeUR]

All my guest vmware 5.5 servers back up with no problem showing that it is using [hotadd] on the proxy line. However the guest vm that Veeam 7 is running on shows it is using [nbd] on the proxy line and takes twice as long a all the other guest put together. I don't see how to force it to use [hotadd] for all servers and I'm not sure that it should. Maybe there is a reason the guest vm running Veeam does not use [hotadd]?

[DavidReimers]

Sorry to dig up an old thread, but is there an advantage to putting a proxy on each ESXi host? I am trying to maximise performance, yet I'm still getting notable less performance when running Veeam as a VM, even with a dedicated separate proxy VM, compared to where I run Veeam on a physical server and use a proxy VM.

[foggy]

David, there's not much sense (in terms of performance) in whether Veeam B&R itself is virtual or physical. Are you sure the designated proxy is being effectively utilized for data processing? What transport mode does it use? What are the bottleneck stats for your jobs?

Basically, to be able to use hotadd to retrieve VM data, the proxy VM needs to have all the datastores where protected VMs' disks reside to be connected to it .

[DavidReimers]

It uses hotadd Foggy. I found an increase in performance when I deployed a dedicated proxy VM separate from the Veeam VM. Overall, the Veeam job reports don't say the bottleneck is the proxy, but each VM varies. It is some cases it is the proxy, in some the source, in others the target.

[foggy]

So are you effectively saying that with virtual Veeam B&R console and hotadd proxy you have lower performance than with physical Veeam console utilizing the same hotadd proxy? Could you then please describe your setup in a bit more detail? And what is the primary bottleneck for the entire job, still?

[DavidReimers]

It's difficult to say as I'm comparing two different sites

- One site uses a virtual VBR console + separate virtual proxy + separate physical server as the backup repository.
- The other site uses a physical VBR console, with local disk (which is also the backup repository) + separate virtual proxy.

I realise I'm not comparing apples with apples here.

FWIW, adding a second dedicated proxy (i.e. for a total of two proxies) and using just these two (not the proxy on the VBR console) has improved performance further - the job that was 4 hours with Veeam 6.5 shrunk to around 3 hours and then to 2.5 hours with the 2 dedicated proxies.

[foggy]

Adding second proxy is indeed should increase performance as the load is spread between them. With parallel processing introduced in v7, you would get even more for sure.