Discussions specific to the VMware vSphere hypervisor
Post Reply
dkiernan
Novice
Posts: 6
Liked: never
Joined: Apr 09, 2012 4:27 pm
Contact:

cktEnabled gets set to false after backup

Post by dkiernan » Feb 19, 2013 12:30 am

I have one VM (the Veeam server itself) that I can *only* get Changed Block Tracking enabled by manually setting it and running a backup.

Then when the next scheduled Veeam backup runs, the CBT is disabled.

The server has one disk ( C: ), 75gb, 5gb free

Shut down VM
enable cktEnabled, generally and for each scsi disk (scsi0:0)
power on vm
run backup - runs with CBT, completes in 45min - 14gb read

next backup - 3hrs, 70gb read
Check cktEnabled - set to false, generally and on scsi0:0

How can I keep CBT enabled for this VM?

Let me know if you need more info!

kjc3303
Expert
Posts: 167
Liked: 24 times
Joined: Dec 02, 2010 12:25 pm
Full Name: Kevin Clarke
Location: Cheshire
Contact:

Re: cktEnabled gets set to false after backup

Post by kjc3303 » Feb 19, 2013 9:18 am

Is the Veeam server a HotAdd Proxy? CBT is disabled on a Proxies that use hotadd proxy mode. This is the case even if the proxy is not being used for this particular job.

To enable CBT either remove the proxy service from the VM or use a different transport mode.

veremin
Product Manager
Posts: 16691
Liked: 1393 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: cktEnabled gets set to false after backup

Post by veremin » Feb 19, 2013 9:37 am

May I ask you if you’re backing up VB&R server, using default proxy server which is deployed on it?

If so, there is no wonder why CBT can’t be enabled, since whenever you’re trying to backup VB&R server via default proxy, CBT is automatically disabled and it’s VB&R proprietary mechanism that is being used.

This ad-hoc mechanism works like this:

1. VB&R scans through the full VM image and calculates a checksum for every data block.
2. After that, it checks what blocks have been changed on your VM in comparison with the VM image stored in the backup file.

In the way I see it, that’s why you see such a high read value.

Hope this helps.
Thanks.

kjc3303
Expert
Posts: 167
Liked: 24 times
Joined: Dec 02, 2010 12:25 pm
Full Name: Kevin Clarke
Location: Cheshire
Contact:

Re: cktEnabled gets set to false after backup

Post by kjc3303 » Feb 19, 2013 9:50 am

I have found that even if the default proxy isn't used when backing up the VBR server but has the proxy service installed on it (Hotadd mode) then CBT will always be disabled so this will need to be addressed too.

Explanation courtesy of Tom Sightler

"Yes, as a workaround for the hotadd CBT "stall" all backup proxies now have CBT disabled automatically. I don't know if there is a way to change this behavior if you were not having CBT "stall" issue during hotadd, but it would be nice, as disabling CBT for proxies works against customers who were using existing VMs as their proxy servers. It's much less of an issue for customers using dedicated proxies as those VMs are typically not worth backing up anyway and even if you want to back them up they are small."

veremin
Product Manager
Posts: 16691
Liked: 1393 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: cktEnabled gets set to false after backup

Post by veremin » Feb 19, 2013 11:14 am

Furthermore, what Kevin said also seems reasonable.

If your VB&R server is being used as Hot-Add proxy for some other job, it inevitably results in the lack of CBT.

From my perspective, this process might have looked like this:

1) Firstly, you manually enabled CBT on your VB&R virtual machine.
2) Then you performed backup of this VM, using different proxy server.
3) Backup process was finished and after a while other backup job took place, however, this time using VB&R server as a proxy. Specifically in this moment, when VB&R was selected as proxy, CBT was disabled on it.

So, CBT had been already disabled, when you tried to backup up VB&R server second time and proprietary VB&R mechanism was used. It should explain the “read” difference you saw.

Hope this helps.
Thanks.

dkiernan
Novice
Posts: 6
Liked: never
Joined: Apr 09, 2012 4:27 pm
Contact:

Re: cktEnabled gets set to false after backup

Post by dkiernan » Feb 21, 2013 4:52 am

Wow - great information! Thanks everyone.

My follow-up question is whether I can get the VB&R server backed up *without* having it read all 70GB?

Is this a known "penalty" when backing up the VB&R server itself?

There is only a "default proxy" for the VB&R server itself, if that's what you mean. Let me know if you need more info on that.

veremin
Product Manager
Posts: 16691
Liked: 1393 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: cktEnabled gets set to false after backup

Post by veremin » Feb 21, 2013 8:21 am

My follow-up question is whether I can get the VB&R server backed up *without* having it read all 70GB?
Yes, you can. However, be sure that you back it up, using different proxy server, not the default one, and that VB&R server isn’t used as Hot-Add proxy for any other job.
Is this a known "penalty" when backing up the VB&R server itself?
As described in my first post, CBT is automatically disabled, whenever you’re trying to backup VB&R server, using default proxy that is deployed on it, and it is VB&R proprietary mechanism that is being used in this case.
There is only a "default proxy" for the VB&R server itself, if that's what you mean.
Actually, the situation is a lit bit different. The role of default proxy is played by VB&R server, nevertheless, you can easily deploy/use whatever number of additional proxies you want.

Hope this helps.
Thanks.

kjc3303
Expert
Posts: 167
Liked: 24 times
Joined: Dec 02, 2010 12:25 pm
Full Name: Kevin Clarke
Location: Cheshire
Contact:

Re: cktEnabled gets set to false after backup

Post by kjc3303 » Feb 21, 2013 10:01 am

To avoid the 70gb read you will have to uninstall the proxy from the VBR server too, as Vladimir says you will need to deploy a proxy on a different server.
You will never be able to use CBT on a server that has the proxy service installed on it even if the proxy is not being used for this particular job.

Not sure what happens if you disable the proxy rather than remove it, I shall test and post results on here if one of the Veeam guys do not confirm before I have chance.

Kev

foggy
Veeam Software
Posts: 18029
Liked: 1533 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: cktEnabled gets set to false after backup

Post by foggy » Feb 21, 2013 10:22 am

kjc3303 wrote:You will never be able to use CBT on a server that has the proxy service installed on it even if the proxy is not being used for this particular job.
Actually, this is not 100% true. If the proxy is not used as hotadd, you can enable CBT manually and it will not be disabled anymore (until the moment this proxy is selected as hotadd proxy in one of the jobs). So you can explicitly set the transport mode for the default proxy server to Network and Veeam B&R will backup itself using CBT (however, all other VMs will be backed up using Network mode also).

kjc3303
Expert
Posts: 167
Liked: 24 times
Joined: Dec 02, 2010 12:25 pm
Full Name: Kevin Clarke
Location: Cheshire
Contact:

Re: cktEnabled gets set to false after backup

Post by kjc3303 » Feb 21, 2013 10:36 am

Correct, should have made that more clear, I did cover that in my original post :)

"To enable CBT either remove the proxy service from the VM or use a different transport mode."

How about disabling the proxy would that have the same effect? I know it's not needed as if the proxy is not used you would just set to Network mode mode although if you wanted to play about it's easier and more noticeable if you can just right click the proxy and disable.

Kev

foggy
Veeam Software
Posts: 18029
Liked: 1533 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: cktEnabled gets set to false after backup

Post by foggy » Feb 21, 2013 10:51 am

Of course, you can just disable the default proxy, enable CBT on it manually and use another proxy to back it up with CBT (since the proxy is disabled, it will not be selected as hotadd proxy anymore, hence CBT will not be disabled).

kjc3303
Expert
Posts: 167
Liked: 24 times
Joined: Dec 02, 2010 12:25 pm
Full Name: Kevin Clarke
Location: Cheshire
Contact:

Re: cktEnabled gets set to false after backup

Post by kjc3303 » Feb 21, 2013 10:54 am

Cheers foggy

rsargeant
Influencer
Posts: 11
Liked: never
Joined: Feb 16, 2012 2:03 pm
Full Name: Richard Sargeant
Contact:

[MERGED] : No VMware [CBT] backing up/replicating Veeam B&R

Post by rsargeant » Jul 07, 2013 11:20 pm

Noticed VMware CBT seems disabled when backing up or replicating VM running Veeam B&R, because [CBT] doesn't appear after virtual disk?

This is happening to a number of Veeam 6.x installs we support, so I guess there's a reason and this is normal behaviour?

Any way of forcing VMware CBT, as jobs run really slow in comparsion to those with VMware CBT support??

Thanks guys.

R.

veremin
Product Manager
Posts: 16691
Liked: 1393 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: cktEnabled gets set to false after backup

Post by veremin » Jul 08, 2013 8:13 am

Hi, Richard. Most likely VB&R is being used in your case as a Hot-ADD proxy. It, in its turn, results in lack of CBT. In order to bypass this, you can explicitly specify default proxy to use Network mode, and also manually enable CBT on it.

Kindly check the answers provided above for more information regarding this issue.

Thanks.

foggy
Veeam Software
Posts: 18029
Liked: 1533 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: cktEnabled gets set to false after backup

Post by foggy » Jul 08, 2013 10:58 am

Another reason for CBT being disabled for the VM is at least one existing snapshot.

rsargeant
Influencer
Posts: 11
Liked: never
Joined: Feb 16, 2012 2:03 pm
Full Name: Richard Sargeant
Contact:

Re: cktEnabled gets set to false after backup

Post by rsargeant » Jul 08, 2013 11:53 am

Hi guys,

In most cases I tend to disable Hot-Add on Default and extra Proxies, and go for Direct SAN or Network, so this isn't the reason. Also the VM definitely has no
other snapshots except the one taken for the running job.

This behavior is happening with all our Veeam 6.x installs. Note: all Veeam installs are running as VMs (both on Windows 7 and 2008 R2).

Any ideas?

Cheers,

R.

veremin
Product Manager
Posts: 16691
Liked: 1393 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: cktEnabled gets set to false after backup

Post by veremin » Jul 08, 2013 11:55 am

Can you manually enable CBT on this machine and see whether it helps or not? Thanks.

rsargeant
Influencer
Posts: 11
Liked: never
Joined: Feb 16, 2012 2:03 pm
Full Name: Richard Sargeant
Contact:

Re: cktEnabled gets set to false after backup

Post by rsargeant » Jul 08, 2013 1:52 pm

CBT is already enabled in Configuration Parameters, already checked this.

I'm not getting the 'Soap CBT' errors which you might get with a 'broken job', maybe caused by power failure or hung Veeam server etc where you need to manually reset the ctk parameters.

I just get no [CBT] appearing after each virtual disk on these VMs???

nefes
Veeam Software
Posts: 623
Liked: 155 times
Joined: Dec 10, 2012 8:44 am
Full Name: Nikita Efes
Contact:

Re: cktEnabled gets set to false after backup

Post by nefes » Jul 08, 2013 2:02 pm

It is worth opening support case then, since the reason of CBT not working can be found from reviewing job logs.

foggy
Veeam Software
Posts: 18029
Liked: 1533 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: cktEnabled gets set to false after backup

Post by foggy » Jul 08, 2013 2:06 pm

Might be a silly question, but are these VMs turned on during backup/replication? Have you tried to reset the CBT manually on them? During what job runs do you observe this behavior: full or incremental or both?

rsargeant
Influencer
Posts: 11
Liked: never
Joined: Feb 16, 2012 2:03 pm
Full Name: Richard Sargeant
Contact:

Re: cktEnabled gets set to false after backup

Post by rsargeant » Jul 09, 2013 7:04 am

This is happening on all of our customers running v6.x, so if it's a bug,I'm amazed we are the only one to report this problem???

To recap:

Veeam Backup & Replication is running on a VM running Windows 2008 R2 or Windows 7;
All other VMs within jobs use [CBT] during jobs EXCEPT the VM running Veeam Backup & Replication;
Proxies are set to Network Mode;
CTK parameters are enabled;
VM running Veeam Backup & Replication has no Snapshots (other than when jobs starts);

All very strange...

R.

veremin
Product Manager
Posts: 16691
Liked: 1393 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: cktEnabled gets set to false after backup

Post by veremin » Jul 09, 2013 9:32 am

As mentioned above, kindly open a ticket with our support team and let them investigate it directly. Thanks.

CharadeUR
Influencer
Posts: 21
Liked: never
Joined: Mar 05, 2012 2:29 pm
Full Name: George Streeter
Contact:

[MERGED] slow backup of veeam vm

Post by CharadeUR » Mar 14, 2014 2:05 pm

All my guest vmware 5.5 servers back up with no problem showing that it is using [hotadd] on the proxy line. However the guest vm that Veeam 7 is running on shows it is using [nbd] on the proxy line and takes twice as long a all the other guest put together. I don't see how to force it to use [hotadd] for all servers and I'm not sure that it should. Maybe there is a reason the guest vm running Veeam does not use [hotadd]?

DavidReimers
Enthusiast
Posts: 49
Liked: 2 times
Joined: Sep 20, 2010 4:39 am
Full Name: David Reimers
Contact:

Re: cktEnabled gets set to false after backup

Post by DavidReimers » Sep 03, 2014 3:56 am

Sorry to dig up an old thread, but is there an advantage to putting a proxy on each ESXi host? I am trying to maximise performance, yet I'm still getting notable less performance when running Veeam as a VM, even with a dedicated separate proxy VM, compared to where I run Veeam on a physical server and use a proxy VM.

foggy
Veeam Software
Posts: 18029
Liked: 1533 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: cktEnabled gets set to false after backup

Post by foggy » Sep 03, 2014 6:45 am

David, there's not much sense (in terms of performance) in whether Veeam B&R itself is virtual or physical. Are you sure the designated proxy is being effectively utilized for data processing? What transport mode does it use? What are the bottleneck stats for your jobs?

Basically, to be able to use hotadd to retrieve VM data, the proxy VM needs to have all the datastores where protected VMs' disks reside to be connected to it .

DavidReimers
Enthusiast
Posts: 49
Liked: 2 times
Joined: Sep 20, 2010 4:39 am
Full Name: David Reimers
Contact:

Re: cktEnabled gets set to false after backup

Post by DavidReimers » Sep 03, 2014 11:59 pm

It uses hotadd Foggy. I found an increase in performance when I deployed a dedicated proxy VM separate from the Veeam VM. Overall, the Veeam job reports don't say the bottleneck is the proxy, but each VM varies. It is some cases it is the proxy, in some the source, in others the target.

foggy
Veeam Software
Posts: 18029
Liked: 1533 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: cktEnabled gets set to false after backup

Post by foggy » Sep 04, 2014 6:16 am

So are you effectively saying that with virtual Veeam B&R console and hotadd proxy you have lower performance than with physical Veeam console utilizing the same hotadd proxy? Could you then please describe your setup in a bit more detail? And what is the primary bottleneck for the entire job, still?

DavidReimers
Enthusiast
Posts: 49
Liked: 2 times
Joined: Sep 20, 2010 4:39 am
Full Name: David Reimers
Contact:

Re: cktEnabled gets set to false after backup

Post by DavidReimers » Sep 05, 2014 6:00 am

It's difficult to say as I'm comparing two different sites

- One site uses a virtual VBR console + separate virtual proxy + separate physical server as the backup repository.
- The other site uses a physical VBR console, with local disk (which is also the backup repository) + separate virtual proxy.

I realise I'm not comparing apples with apples here.

FWIW, adding a second dedicated proxy (i.e. for a total of two proxies) and using just these two (not the proxy on the VBR console) has improved performance further - the job that was 4 hours with Veeam 6.5 shrunk to around 3 hours and then to 2.5 hours with the 2 dedicated proxies.

foggy
Veeam Software
Posts: 18029
Liked: 1533 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: cktEnabled gets set to false after backup

Post by foggy » Sep 05, 2014 6:36 am

Adding second proxy is indeed should increase performance as the load is spread between them. With parallel processing introduced in v7, you would get even more for sure.

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 14 guests