Discussions specific to the VMware vSphere hypervisor
Post Reply
KOPFteam
Enthusiast
Posts: 52
Liked: 7 times
Joined: Sep 30, 2011 6:59 pm
Full Name: Florian Pürner
Contact:

Slow backup of VM that has Veeam Backup Proxy installed

Post by KOPFteam »

Hi forum,

I have my Veeam B&R (7.0.0.771) running on my vCenter (physical Win 2008R2). At the moment this vCenter controls 1 ESXi 5.1.0 with 3 VMs.

The VMs "Server1" and "Server3" are quite small, the VM "Server2" is using about 900GB of it's 1 TB thin provisioned HDD. There are no snapshots on any of the VMs. All 3 VMs are either "VM version" 7 or 8. I resized the hdd of "Server2" after the initial installation.

I created a "Backup job" to backup these 3 VMs to a NAS over the network. CBT is enabled in that job.

I have two Backup Proxies: The default backup proxy running on the vCenter and an additional backup proxy running in the VM "Server2".

The backup tasks finished without any errors, but it needs over 5 hours each night.

"Server1" and "Server3" use the backup proxy in "Server2" [hotadd], they finish quite fast and they both show [CBT] in the history.
"Server2" uses the default backup proxy on the vCenter [nbd], needs a lot of time and it doesn't show [CBT] in the history.

What can I do that all 3 Vms use CBT?

Thanks for your help,
Florian

Gostev
SVP, Product Management
Posts: 29321
Liked: 5456 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: No CBT on 1 of 3 VMs - Veeam B&R shows no error

Post by Gostev » 1 person likes this post

Couple of reasons:

1. CBT is not leveraged, because CBT is disabled automatically on any VM that acts as backup proxy due to some VMware limitations (enabling may cause short server outage during backup).

2. Backup proxy cannot backup itself in hot add mode (VMware and Microsoft interop issue), and automatically fails over to network.

Moving backup proxy from Server 2 to the smallest VM you have will be the best solution in your case.

Thanks!

KOPFteam
Enthusiast
Posts: 52
Liked: 7 times
Joined: Sep 30, 2011 6:59 pm
Full Name: Florian Pürner
Contact:

Re: No CBT on 1 of 3 VMs - Veeam B&R shows no error

Post by KOPFteam »

Hi Gostev,

thanks for your reply on a Sunday evening! I just created a backup proxy on Server1 and removed the backup proxy from Server2 (Why is removal instant? Shouldn't this uninstall software from the VM?).
I'll check what happens during the scheduled backup in the night and I'll let you know.

Thanks,
Florian
Loving (and fighting with) Veeam for over a decade!

Gostev
SVP, Product Management
Posts: 29321
Liked: 5456 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: No CBT on 1 of 3 VMs - Veeam B&R shows no error

Post by Gostev »

Uninstallation is not instant, but rather very quick :D because the backup proxy package is so small.

First backup of Server 2 will still be slow, as this will be the run when CBT gets first enabled. However, this run should already use hot add. And the following runs should start leveraging CBT, so Server 2 processing times should improve dramatically.

KOPFteam
Enthusiast
Posts: 52
Liked: 7 times
Joined: Sep 30, 2011 6:59 pm
Full Name: Florian Pürner
Contact:

Re: No CBT on 1 of 3 VMs - Veeam B&R shows no error

Post by KOPFteam »

Hi Gostev,

it happend exactly like you promised: CBT was activated on Server2 in the first night (because it started to use the proxy in Server1), but it still read the whole drive. Tonight it only read 48 GB of the 1 TB and it finished after 49 minutes - down from over 5 hours!

Perfect solution - Thanks :D
Florian
Loving (and fighting with) Veeam for over a decade!

Gostev
SVP, Product Management
Posts: 29321
Liked: 5456 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: No CBT on 1 of 3 VMs - Veeam B&R shows no error

Post by Gostev »

Perfect Florian - and thanks for coming back to confirm this helped.

keithlammers
Influencer
Posts: 17
Liked: never
Joined: Dec 17, 2013 7:19 pm
Full Name: Keith Lammers
Location: Canada
Contact:

[MERGED] Slow backup of VM that has Veeam Backup Proxy insta

Post by keithlammers »

We do nightly incremental backups of all of our VMs. One of the VMs has the Veeam Backup Proxy installed on it. The nightly backup job is specifically set to not use that proxy, so there shouldn't be any sort of conflict there. When that VM gets backed up, it takes over an hour. If I uninstall the Veeam Backup Proxy from that VM, the next nightly incremental will take 10 minutes or less.

Is backing up a VM that has the Veeam Backup Proxy installed in it not supported? Just wondering why this issue would be occurring if that proxy isn't even being used at the time of the backup.

Thanks!

keithlammers
Influencer
Posts: 17
Liked: never
Joined: Dec 17, 2013 7:19 pm
Full Name: Keith Lammers
Location: Canada
Contact:

Re: [MERGED] Slow backup of VM that has Veeam Backup Proxy i

Post by keithlammers »

Gostev wrote:1. CBT is not leveraged, because CBT is disabled automatically on any VM that acts as backup proxy due to some VMware limitations (enabling may cause short server outage during backup).

2. Backup proxy cannot backup itself in hot add mode (VMware and Microsoft interop issue), and automatically fails over to network.
This answers my question, thanks for merging it into the appropriate thread! I had tried searching but some how didn't turn up this topic, so I apologize for the duplicate submission :)

Gostev
SVP, Product Management
Posts: 29321
Liked: 5456 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow backup of VM that has Veeam Backup Proxy installed

Post by Gostev »

No problems, thanks for searching! I renamed the original topic after the merge, which is why you did not find it. And I actually used your topic name, because it is better from search relevance perspective. So future users having the same question should be able to find this topic easily.

patrickds
Enthusiast
Posts: 26
Liked: 5 times
Joined: Feb 24, 2010 11:58 am
Full Name: Patrick De Smedt
Contact:

Re: No CBT on 1 of 3 VMs - Veeam B&R shows no error

Post by patrickds »

Gostev wrote:Couple of reasons:

1. CBT is not leveraged, because CBT is disabled automatically on any VM that acts as backup proxy due to some VMware limitations (enabling may cause short server outage during backup).

2. Backup proxy cannot backup itself in hot add mode (VMware and Microsoft interop issue), and automatically fails over to network.
Is this the official and final explanation in version 7?

Before, it used to be "CBT is disabled on any proxy configured for Hot-add", if I am not mistaken.
2 has always been the case afaik, so no surprise there.

I'm just checking, because my boss asked me to create a document with best practice recommendations for Veeam B&R installations, and this is something that could result in the need for extra Windows licenses when deploying proxies, to make sure the backup performance is as high as possible.

Vitaliy S.
Product Manager
Posts: 25349
Liked: 2235 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Slow backup of VM that has Veeam Backup Proxy installed

Post by Vitaliy S. »

Hi Patrick, yes, this is still the case as I haven't heard about any changes in vStorage API engine (VDDK) regarding that.

Gostev
SVP, Product Management
Posts: 29321
Liked: 5456 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: No CBT on 1 of 3 VMs - Veeam B&R shows no error

Post by Gostev »

patrickds wrote:this is something that could result in the need for extra Windows licenses when deploying proxies, to make sure the backup performance is as high as possible
Just remember that THE ONLY VMs that will be backed up without CBT are those few backup proxy VMs. If you use smaller VMs as your proxies (10-20 GB), the extended time it takes to backup them up will be negligible comparing to your overall backup window. I am assuming you have a big environment, if you are thinking about deploying additional proxies.

dellock6
Veeam Software
Posts: 6034
Liked: 1837 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow backup of VM that has Veeam Backup Proxy installed

Post by dellock6 » 1 person likes this post

In a big environment one could also think IF the proxies need to be saved. Before moving to physical proxies, we used to keep a win2008R2 template, and if something went wrong with a proxy, it was quicker to redploy it than to restore it from backups. I take care of the central Veeam server and repositories, proxies are expendable...

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2021
Veeam VMCE #1

patrickds
Enthusiast
Posts: 26
Liked: 5 times
Joined: Feb 24, 2010 11:58 am
Full Name: Patrick De Smedt
Contact:

Re: No CBT on 1 of 3 VMs - Veeam B&R shows no error

Post by patrickds »

Gostev wrote: I am assuming you have a big environment, if you are thinking about deploying additional proxies.
quite the contrary.
For smaller customers, often with only 1 host, our sales people are always looking for ways to make cheaper offers and win more business.
Lately I've been running into installations, where they have really skimped on numbers of licenses, offering only the bare minimum, leaving me with no choice but to put Veeam on a production VM.
Of course this causes backup jobs to take much longer than with a dedicated VM, and the use of CBT.

Now that I know CBT will always be disabled on proxies, and not only when it is set to Hot-Add, I can tell them (the sales guys, and the potential customers), that there is a choice to be made:
longer backups, or extra license cost.

For some customers, an extra 1000 $/€ can make the difference.

foggy
Veeam Software
Posts: 20523
Liked: 1970 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: No CBT on 1 of 3 VMs - Veeam B&R shows no error

Post by foggy »

patrickds wrote:Now that I know CBT will always be disabled on proxies, and not only when it is set to Hot-Add,
Actually, CBT is disabled for hotadd proxies only and even can be enabled manually on them if the proxy is not used as hotadd one anymore.

patrickds
Enthusiast
Posts: 26
Liked: 5 times
Joined: Feb 24, 2010 11:58 am
Full Name: Patrick De Smedt
Contact:

Re: No CBT on 1 of 3 VMs - Veeam B&R shows no error

Post by patrickds »

foggy wrote:Actually, CBT is disabled for hotadd proxies only and even can be enabled manually on them if the proxy is not used as hotadd one anymore.
That's exactly what I wanted to know.
Is this still the case for version 7, or has it changed to what Gostev wrote above?

To resume:
- is CBT always disabled on proxies, or only when used for hot-add?
- is the same true for the server running Veeam? (I would assume yes, since the proxy service is the one doing the actual work, and I wouldn't expect having the management service running on a VM to make a difference)

foggy
Veeam Software
Posts: 20523
Liked: 1970 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: No CBT on 1 of 3 VMs - Veeam B&R shows no error

Post by foggy »

patrickds wrote:Is this still the case for version 7, or has it changed to what Gostev wrote above?
This is still the case for v7 and I do not see any contradictions with what Anton has stated in his post.
patrickds wrote:- is CBT always disabled on proxies, or only when used for hot-add?
CBT is always disabled for hotadd proxies (and only for them), however, can be manually enabled later if hotadd is not used.
patrickds wrote:- is the same true for the server running Veeam? (I would assume yes, since the proxy service is the one doing the actual work, and I wouldn't expect having the management service running on a VM to make a difference)
Yes, the same is true for the default proxy server as well.

patrickds
Enthusiast
Posts: 26
Liked: 5 times
Joined: Feb 24, 2010 11:58 am
Full Name: Patrick De Smedt
Contact:

Re: No CBT on 1 of 3 VMs - Veeam B&R shows no error

Post by patrickds »

foggy wrote:This is still the case for v7 and I do not see any contradictions with what Anton has stated in his post.
Gostev wrote: 1. CBT is not leveraged, because CBT is disabled automatically on any VM that acts as backup proxy due to some VMware limitations (enabling may cause short server outage during backup).
CBT is disabled "on any VM that acts as backup proxy" or "for hotadd proxies (and only for them)" are 2 different statements, which make a huge difference in backup time.
But you have cleared that up for me now.

Thanks.

patrickds
Enthusiast
Posts: 26
Liked: 5 times
Joined: Feb 24, 2010 11:58 am
Full Name: Patrick De Smedt
Contact:

Re: Slow backup of VM that has Veeam Backup Proxy installed

Post by patrickds »

Thanks to the info provided here, and a little script found there (among other places):
http://www.lazywinadmin.com/2013/01/ena ... bt-on.html
CBT now also works on the proxy, without having to power off the VM, and incremental backup time has been reduced from 10 hours to 30 minutes.

I just have to remember to set the proxy to network mode whenever I need to install Veeam on a non-dedicated VM, before any backup is taken, otherwise CBT gets disabled.

Thanks again for the clarification.

keithlammers
Influencer
Posts: 17
Liked: never
Joined: Dec 17, 2013 7:19 pm
Full Name: Keith Lammers
Location: Canada
Contact:

Re: Slow backup of VM that has Veeam Backup Proxy installed

Post by keithlammers »

patrickds wrote:Thanks to the info provided here, and a little script found there (among other places):
http://www.lazywinadmin.com/2013/01/ena ... bt-on.html
CBT now also works on the proxy, without having to power off the VM, and incremental backup time has been reduced from 10 hours to 30 minutes.
That's interesting, but as Gostev wrote, having CBT enabled on a proxy VM can cause a short outage:
Gostev wrote:CBT is not leveraged, because CBT is disabled automatically on any VM that acts as backup proxy due to some VMware limitations (enabling may cause short server outage during backup).
Gostev, does the "short server outage" that you mentioned only affect the proxy VM? Just wondering, as the VM I use as the proxy is not critical and it would be ok for it to be down for a short period of time when we run our backups. Just wondering if it's safe to do patrickds' workaround. (i.e. is the short downtime of the proxy VM the only side effect of having CBT enabled on it?)

foggy
Veeam Software
Posts: 20523
Liked: 1970 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow backup of VM that has Veeam Backup Proxy installed

Post by foggy »

If CBT is enabled on hotadd proxy, then on attaching (hotadding) disks of the backed up VM to the proxy VM, CBT for those disks should be calculated. During this operation the proxy VM itself could experience a short stun and could become unavailable to Veeam B&R console. For this reason, we intentionally disable CBT for hotadd proxies in current version.

So, answering your questions, yes, only proxy VM is affected and enabling CBT manually is safe (since CBT will be anyway disabled to avoid this outage, once the proxy is used as hotadd one).

patrickds
Enthusiast
Posts: 26
Liked: 5 times
Joined: Feb 24, 2010 11:58 am
Full Name: Patrick De Smedt
Contact:

Re: Slow backup of VM that has Veeam Backup Proxy installed

Post by patrickds »

I hate to have to come back on this, but once again I'm facing the issue.

On a new installation, version 7 + latest patches.
Backup and replication of Veeam server itself does not use CBT.

CBT is enabled on the VM, and CTK files exist in the datastore.
But still no CBT when backing up the (proxy) server.

foggy
Veeam Software
Posts: 20523
Liked: 1970 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow backup of VM that has Veeam Backup Proxy installed

Post by foggy »

patrickds wrote:On a new installation, version 7 + latest patches.
Backup and replication of Veeam server itself does not use CBT.
Why should it? Nothing has changed since your previous posts: CBT is automatically disabled for hotadd proxies for the reasons described above.

patrickds
Enthusiast
Posts: 26
Liked: 5 times
Joined: Feb 24, 2010 11:58 am
Full Name: Patrick De Smedt
Contact:

Re: Slow backup of VM that has Veeam Backup Proxy installed

Post by patrickds »

Forgot to mention proxy was set to Network.

But it is fixed already.
Apparently CBT was not enabled properly on the proxy VM.
the scsi0:0.ctkEnabled = TRUE option existed, and so did the ctk files in the VMs directory.
But the ctkEnabled = TRUE was missing from the vmx file, which is probably why Veeam wouldn't use CBT.
I tried to add it to the Advanced Configuration Parameters in the Vsphere CLient, but it wouldn't take.
After adding it, and closing the window with the OK button, the option would not be present when reopening the configuration.
No error message either.
Got it to work by adding the option directly into the vmx file.

Post Reply

Who is online

Users browsing this forum: No registered users and 16 guests