Slow backup of VM that has Veeam Backup Proxy installed

[KOPFteam]

Hi forum,

I have my Veeam B&R (7.0.0.771) running on my vCenter (physical Win 2008R2). At the moment this vCenter controls 1 ESXi 5.1.0 with 3 VMs.

The VMs "Server1" and "Server3" are quite small, the VM "Server2" is using about 900GB of it's 1 TB thin provisioned HDD. There are no snapshots on any of the VMs. All 3 VMs are either "VM version" 7 or 8. I resized the hdd of "Server2" after the initial installation.

I created a "Backup job" to backup these 3 VMs to a NAS over the network. CBT is enabled in that job.

I have two Backup Proxies: The default backup proxy running on the vCenter and an additional backup proxy running in the VM "Server2".

The backup tasks finished without any errors, but it needs over 5 hours each night.

"Server1" and "Server3" use the backup proxy in "Server2" [hotadd], they finish quite fast and they both show [CBT] in the history.
"Server2" uses the default backup proxy on the vCenter [nbd], needs a lot of time and it doesn't show [CBT] in the history.

What can I do that all 3 Vms use CBT?

Thanks for your help,
Florian

[Gostev]

Couple of reasons:

1. CBT is not leveraged, because CBT is disabled automatically on any VM that acts as backup proxy due to some VMware limitations (enabling may cause short server outage during backup).

2. Backup proxy cannot backup itself in hot add mode (VMware and Microsoft interop issue), and automatically fails over to network.

Moving backup proxy from Server 2 to the smallest VM you have will be the best solution in your case.

Thanks!

[KOPFteam]

Hi Gostev,

thanks for your reply on a Sunday evening! I just created a backup proxy on Server1 and removed the backup proxy from Server2 (Why is removal instant? Shouldn't this uninstall software from the VM?).
I'll check what happens during the scheduled backup in the night and I'll let you know.

Thanks,
Florian

[Gostev]

Uninstallation is not instant, but rather very quick because the backup proxy package is so small.

First backup of Server 2 will still be slow, as this will be the run when CBT gets first enabled. However, this run should already use hot add. And the following runs should start leveraging CBT, so Server 2 processing times should improve dramatically.

[KOPFteam]

Hi Gostev,

it happend exactly like you promised: CBT was activated on Server2 in the first night (because it started to use the proxy in Server1), but it still read the whole drive. Tonight it only read 48 GB of the 1 TB and it finished after 49 minutes - down from over 5 hours!

Perfect solution - Thanks
Florian

[Gostev]

Perfect Florian - and thanks for coming back to confirm this helped.

[keithlammers]

We do nightly incremental backups of all of our VMs. One of the VMs has the Veeam Backup Proxy installed on it. The nightly backup job is specifically set to not use that proxy, so there shouldn't be any sort of conflict there. When that VM gets backed up, it takes over an hour. If I uninstall the Veeam Backup Proxy from that VM, the next nightly incremental will take 10 minutes or less.

Is backing up a VM that has the Veeam Backup Proxy installed in it not supported? Just wondering why this issue would be occurring if that proxy isn't even being used at the time of the backup.

Thanks!

[keithlammers]

1. CBT is not leveraged, because CBT is disabled automatically on any VM that acts as backup proxy due to some VMware limitations (enabling may cause short server outage during backup).

2. Backup proxy cannot backup itself in hot add mode (VMware and Microsoft interop issue), and automatically fails over to network.

This answers my question, thanks for merging it into the appropriate thread! I had tried searching but some how didn't turn up this topic, so I apologize for the duplicate submission

[Gostev]

No problems, thanks for searching! I renamed the original topic after the merge, which is why you did not find it. And I actually used your topic name, because it is better from search relevance perspective. So future users having the same question should be able to find this topic easily.

[patrickds]

Couple of reasons:

1. CBT is not leveraged, because CBT is disabled automatically on any VM that acts as backup proxy due to some VMware limitations (enabling may cause short server outage during backup).

2. Backup proxy cannot backup itself in hot add mode (VMware and Microsoft interop issue), and automatically fails over to network.

Is this the official and final explanation in version 7?

Before, it used to be "CBT is disabled on any proxy configured for Hot-add", if I am not mistaken.
2 has always been the case afaik, so no surprise there.

I'm just checking, because my boss asked me to create a document with best practice recommendations for Veeam B&R installations, and this is something that could result in the need for extra Windows licenses when deploying proxies, to make sure the backup performance is as high as possible.

[Vitaliy S.]

Hi Patrick, yes, this is still the case as I haven't heard about any changes in vStorage API engine (VDDK) regarding that.

[Gostev]

this is something that could result in the need for extra Windows licenses when deploying proxies, to make sure the backup performance is as high as possible

Just remember that THE ONLY VMs that will be backed up without CBT are those few backup proxy VMs. If you use smaller VMs as your proxies (10-20 GB), the extended time it takes to backup them up will be negligible comparing to your overall backup window. I am assuming you have a big environment, if you are thinking about deploying additional proxies.

[dellock6]

In a big environment one could also think IF the proxies need to be saved. Before moving to physical proxies, we used to keep a win2008R2 template, and if something went wrong with a proxy, it was quicker to redploy it than to restore it from backups. I take care of the central Veeam server and repositories, proxies are expendable...

Luca.

[patrickds]

I am assuming you have a big environment, if you are thinking about deploying additional proxies.

quite the contrary.
For smaller customers, often with only 1 host, our sales people are always looking for ways to make cheaper offers and win more business.
Lately I've been running into installations, where they have really skimped on numbers of licenses, offering only the bare minimum, leaving me with no choice but to put Veeam on a production VM.
Of course this causes backup jobs to take much longer than with a dedicated VM, and the use of CBT.

Now that I know CBT will always be disabled on proxies, and not only when it is set to Hot-Add, I can tell them (the sales guys, and the potential customers), that there is a choice to be made:
longer backups, or extra license cost.

For some customers, an extra 1000 $/€ can make the difference.

[foggy]

Now that I know CBT will always be disabled on proxies, and not only when it is set to Hot-Add,

Actually, CBT is disabled for hotadd proxies only and even can be enabled manually on them if the proxy is not used as hotadd one anymore.

[patrickds]

Actually, CBT is disabled for hotadd proxies only and even can be enabled manually on them if the proxy is not used as hotadd one anymore.

That's exactly what I wanted to know.
Is this still the case for version 7, or has it changed to what Gostev wrote above?

To resume:
- is CBT always disabled on proxies, or only when used for hot-add?
- is the same true for the server running Veeam? (I would assume yes, since the proxy service is the one doing the actual work, and I wouldn't expect having the management service running on a VM to make a difference)

[foggy]

Is this still the case for version 7, or has it changed to what Gostev wrote above?

This is still the case for v7 and I do not see any contradictions with what Anton has stated in his post.

- is CBT always disabled on proxies, or only when used for hot-add?

CBT is always disabled for hotadd proxies (and only for them), however, can be manually enabled later if hotadd is not used.

- is the same true for the server running Veeam? (I would assume yes, since the proxy service is the one doing the actual work, and I wouldn't expect having the management service running on a VM to make a difference)

Yes, the same is true for the default proxy server as well.

[patrickds]

This is still the case for v7 and I do not see any contradictions with what Anton has stated in his post.

1. CBT is not leveraged, because CBT is disabled automatically on any VM that acts as backup proxy due to some VMware limitations (enabling may cause short server outage during backup).

CBT is disabled "on any VM that acts as backup proxy" or "for hotadd proxies (and only for them)" are 2 different statements, which make a huge difference in backup time.
But you have cleared that up for me now.

Thanks.

[patrickds]

Thanks to the info provided here, and a little script found there (among other places):
http://www.lazywinadmin.com/2013/01/ena ... bt-on.html
CBT now also works on the proxy, without having to power off the VM, and incremental backup time has been reduced from 10 hours to 30 minutes.

I just have to remember to set the proxy to network mode whenever I need to install Veeam on a non-dedicated VM, before any backup is taken, otherwise CBT gets disabled.

Thanks again for the clarification.

[keithlammers]

Thanks to the info provided here, and a little script found there (among other places):
http://www.lazywinadmin.com/2013/01/ena ... bt-on.html
CBT now also works on the proxy, without having to power off the VM, and incremental backup time has been reduced from 10 hours to 30 minutes.

That's interesting, but as Gostev wrote, having CBT enabled on a proxy VM can cause a short outage:

CBT is not leveraged, because CBT is disabled automatically on any VM that acts as backup proxy due to some VMware limitations (enabling may cause short server outage during backup).

Gostev, does the "short server outage" that you mentioned only affect the proxy VM? Just wondering, as the VM I use as the proxy is not critical and it would be ok for it to be down for a short period of time when we run our backups. Just wondering if it's safe to do patrickds' workaround. (i.e. is the short downtime of the proxy VM the only side effect of having CBT enabled on it?)

[foggy]

If CBT is enabled on hotadd proxy, then on attaching (hotadding) disks of the backed up VM to the proxy VM, CBT for those disks should be calculated. During this operation the proxy VM itself could experience a short stun and could become unavailable to Veeam B&R console. For this reason, we intentionally disable CBT for hotadd proxies in current version.

So, answering your questions, yes, only proxy VM is affected and enabling CBT manually is safe (since CBT will be anyway disabled to avoid this outage, once the proxy is used as hotadd one).

[patrickds]

I hate to have to come back on this, but once again I'm facing the issue.

On a new installation, version 7 + latest patches.
Backup and replication of Veeam server itself does not use CBT.

CBT is enabled on the VM, and CTK files exist in the datastore.
But still no CBT when backing up the (proxy) server.

[foggy]

On a new installation, version 7 + latest patches.
Backup and replication of Veeam server itself does not use CBT.

Why should it? Nothing has changed since your previous posts: CBT is automatically disabled for hotadd proxies for the reasons described above.

[patrickds]

Forgot to mention proxy was set to Network.

But it is fixed already.
Apparently CBT was not enabled properly on the proxy VM.
the scsi0:0.ctkEnabled = TRUE option existed, and so did the ctk files in the VMs directory.
But the ctkEnabled = TRUE was missing from the vmx file, which is probably why Veeam wouldn't use CBT.
I tried to add it to the Advanced Configuration Parameters in the Vsphere CLient, but it wouldn't take.
After adding it, and closing the window with the OK button, the option would not be present when reopening the configuration.
No error message either.
Got it to work by adding the option directly into the vmx file.