Discussions specific to the VMware vSphere hypervisor
humbertoz
Novice
Posts: 5
Liked: never
Joined: Jul 16, 2018 10:45 pm
Full Name: Robert Zarate
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by humbertoz » Jul 17, 2018 2:05 pm

spiritie wrote:We've also seen huge increase in backup time, and are not able to reach our backup window anymore.

Busy adding another proxy to see if it helps.

Running VMware 6.5, and all proxies are VM's using hotadd.

We though don't have many VM's with many disks, but have some large jobs.

EDIT:
Just did an inventory of our VM's in on of our vCenters. Out of around 400-500 VM's,
only 9 VM's has over 3 disks (Ranges between 4-7 disks)

Around 70-75% of the VM's has 1 disk, and 20-25% has 2 disks, rest is above.
Spiritie,

Can you look at your jobs and see where the slowdown is to maybe shed some commonalities with our setup? When we look at our jobs, we can easily see the issue. All VMs process correctly (with low incremental backup times 5-10 minutes) until they hit those two "bad" VMs. Then the backup time for those VMs and any other VMs processing at the same time blows up to 2-4 hours. After those two VMs are processed, all remaining VMs are processed correctly again (5-10 minutes). You can literally see it in the backup summary report without even digging deeper. The "duration" column will be good, good, good, bad, bad, bad, then good , good again as soon as the "bad" VMs are done processing (good VMs are also affected while the "bad" VMs are processing concurrently, but everything goes back to normal as soon as the bad VMs are done). This customer only has two VMs with high disk counts out of 60 VMs, but those two are causing a real headache. Hopefully, when you find which VMs are causing an issue, you can share specs on them.

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Jul 17, 2018 2:08 pm

@Gert for those on 6.5, the quick and confirmed fix is to roll backup VDDK back to the version used in Update 3 - feel free to contact our support for assistance with this.

spiritie
Service Provider
Posts: 66
Liked: 7 times
Joined: Mar 01, 2016 10:16 am
Full Name: Gert van Niekerk
Location: Denmark
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by spiritie » Jul 18, 2018 7:17 am

Gostev wrote:@Gert for those on 6.5, the quick and confirmed fix is to roll backup VDDK back to the version used in Update 3 - feel free to contact our support for assistance with this.
Hi Gostev, we are upgrading vCenter 6.7 tomorrow (No ESXi), is this bug with vCenter or ESXi?

ottl05
Influencer
Posts: 11
Liked: 1 time
Joined: Oct 16, 2014 11:29 am
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by ottl05 » Jul 18, 2018 7:20 am

we have the same issue.
vm on 6.5, hotadd and a increase on backup time

whats the best way to solve this?

mcz
Expert
Posts: 253
Liked: 48 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by mcz » Jul 18, 2018 7:24 am

ottl05 wrote:we have the same issue.
vm on 6.5, hotadd and a increase on backup time

whats the best way to solve this?
I think rolling back VDDK as Anton Gostev wrote:
@Gert for those on 6.5, the quick and confirmed fix is to roll backup VDDK back to the version used in Update 3 - feel free to contact our support for assistance with this.

spiritie
Service Provider
Posts: 66
Liked: 7 times
Joined: Mar 01, 2016 10:16 am
Full Name: Gert van Niekerk
Location: Denmark
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by spiritie » Jul 18, 2018 7:37 am

humbertoz wrote: Spiritie,

Can you look at your jobs and see where the slowdown is to maybe shed some commonalities with our setup? When we look at our jobs, we can easily see the issue. All VMs process correctly (with low incremental backup times 5-10 minutes) until they hit those two "bad" VMs. Then the backup time for those VMs and any other VMs processing at the same time blows up to 2-4 hours. After those two VMs are processed, all remaining VMs are processed correctly again (5-10 minutes). You can literally see it in the backup summary report without even digging deeper. The "duration" column will be good, good, good, bad, bad, bad, then good , good again as soon as the "bad" VMs are done processing (good VMs are also affected while the "bad" VMs are processing concurrently, but everything goes back to normal as soon as the bad VMs are done). This customer only has two VMs with high disk counts out of 60 VMs, but those two are causing a real headache. Hopefully, when you find which VMs are causing an issue, you can share specs on them.
I've looked through them, but cannot find any redline on "bad VM's". I have 1 job where we have 27 VM's in it
and the only 2 VM'sthat was slow (a bit over 1 hour each) was tiny VM's that only had 40 GB and 1 disk on them.

But this seems to randomize, because I have some a lot of VM's that worked fine yesterday but "failed" this night.

All the VM's that is slow goes like this in the log, all of them stuck on hot adding the disk, and it fails over to NBD mode and processes them fairly quickly:

17-07-2018 21:35:24 :: Using backup proxy VMware Backup Proxy for disk Hard disk 1 [hotadd]
17-07-2018 22:47:36 :: Unable to hot add source disk, failing over to network mode...
17-07-2018 22:47:38 :: Hard disk 1 (40,0 GB) 1,1 GB read at 22 MB/s [CBT]
17-07-2018 22:48:47 :: Removing VM snapshot
17-07-2018 22:49:08 :: Finalizing

Gostev has reported that there is a fix for VMware vSphere 6.5

mcz
Expert
Posts: 253
Liked: 48 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by mcz » Jul 18, 2018 7:42 am

I had such issues in the past when proxie's bios.uuid wasn't unique withing the same vcenter. This would be the case if e.g. someone would replicate proxies to another host but still within the same vcenter. Has anybody checked if this isn't the case?

humbertoz
Novice
Posts: 5
Liked: never
Joined: Jul 16, 2018 10:45 pm
Full Name: Robert Zarate
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by humbertoz » Jul 18, 2018 8:19 am

humbertoz wrote: Spiritie,

Can you look at your jobs and see where the slowdown is to maybe shed some commonalities with our setup? When we look at our jobs, we can easily see the issue. All VMs process correctly (with low incremental backup times 5-10 minutes) until they hit those two "bad" VMs. Then the backup time for those VMs and any other VMs processing at the same time blows up to 2-4 hours. After those two VMs are processed, all remaining VMs are processed correctly again (5-10 minutes). You can literally see it in the backup summary report without even digging deeper. The "duration" column will be good, good, good, bad, bad, bad, then good , good again as soon as the "bad" VMs are done processing (good VMs are also affected while the "bad" VMs are processing concurrently, but everything goes back to normal as soon as the bad VMs are done). This customer only has two VMs with high disk counts out of 60 VMs, but those two are causing a real headache. Hopefully, when you find which VMs are causing an issue, you can share specs on them.

I've looked through them, but cannot find any redline on "bad VM's". I have 1 job where we have 27 VM's in it
and the only 2 VM'sthat was slow (a bit over 1 hour each) was tiny VM's that only had 40 GB and 1 disk on them.

But this seems to randomize, because I have some a lot of VM's that worked fine yesterday but "failed" this night.

All the VM's that is slow goes like this in the log, all of them stuck on hot adding the disk, and it fails over to NBD mode and processes them fairly quickly:

17-07-2018 21:35:24 :: Using backup proxy VMware Backup Proxy for disk Hard disk 1 [hotadd]
17-07-2018 22:47:36 :: Unable to hot add source disk, failing over to network mode...
17-07-2018 22:47:38 :: Hard disk 1 (40,0 GB) 1,1 GB read at 22 MB/s [CBT]
17-07-2018 22:48:47 :: Removing VM snapshot
17-07-2018 22:49:08 :: Finalizing

Gostev has reported that there is a fix for VMware vSphere 6.5
Spiritie,

Thanks for the info. Unfortunately, it looks like your issues is a little different than our customer's. Your hotadd takes a long time to process, like theirs, but in the end is failing and switching over to NBD (network) mode. Our customer's two "bad" VMs take a long time for the hotadd process just like you are also seeing, but the hotadd never fails. It finally completes and then the VM begins the drive read and backup process until it pauses again. Their "bad" VMs are never random. Yours seem to be random a little. I do not see any things in common with your issue and our issue except that hotadd is having problems. Hopefully, Gostev is able to track some issues down and find a common source to all our ailments.

ottl05
Influencer
Posts: 11
Liked: 1 time
Joined: Oct 16, 2014 11:29 am
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by ottl05 » Jul 20, 2018 3:22 pm 1 person likes this post

I have an open ticket (#03106424 ) about rollback vddk version, but nothing is happening :-(

ottl05
Influencer
Posts: 11
Liked: 1 time
Joined: Oct 16, 2014 11:29 am
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by ottl05 » Jul 23, 2018 6:20 am

ottl05 wrote:I have an open ticket (#03106424 ) about rollback vddk version, but nothing is happening :-(
I changed the vddk-files and now the backup times are back to "normal".

thanks.

DominikM
Influencer
Posts: 10
Liked: 1 time
Joined: Nov 19, 2014 12:44 pm
Full Name: Dominik Meier
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by DominikM » Jul 23, 2018 7:25 am

I'm having the same Problem since I've upgraded to 9.5 U3a. Since we're still on vSphere 6.5 I've requested a downgrade of the VDDK version.

omegagx
Enthusiast
Posts: 49
Liked: 2 times
Joined: May 09, 2017 6:33 pm
Full Name: Michael Gorn
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by omegagx » Jul 26, 2018 9:34 pm

So this issue doesn't occur on ESX 6.0 U2 ?

AlexWhit
Enthusiast
Posts: 40
Liked: never
Joined: Jul 21, 2011 10:10 am
Full Name: Alex Whittaker
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by AlexWhit » Jul 30, 2018 7:33 am

HI have seen this and dot told by Veeam support that it is normal

Andreas Neufert
Veeam Software
Posts: 3392
Liked: 596 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Andreas Neufert » Jul 30, 2018 8:33 am

omegagx wrote:So this issue doesn't occur on ESX 6.0 U2 ?
Hi,

the issue is not within the hypervisor it is within the vmware software development kit (VDDK) that backup vendors need to add to their products.
For vsphere 6.7 compatibillity reasons we upgraded to VDDK 6.7 which include the issue.

For those customers that do not have upgraded to vsphere 6.7 you can stay on our Update 3 (older vddk kit) or contact support to downgrade the vddk kit with Update 3a. Let‘s hope that VMware fixes the VDDK kit soon.

ChuckS42
Expert
Posts: 104
Liked: 13 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by ChuckS42 » Jul 30, 2018 9:39 pm 1 person likes this post

Has VMware acknowledged the issue and started working on a fix?

Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests