Discussions specific to the VMware vSphere hypervisor
Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Jul 31, 2018 2:12 pm

VMware Support is still "researching" the issue, which is the state that we all know can potentially take a very long time :D

From Veeam perspective, the issue is fully understood thanks to the verbose VDDK debug logs obtained from the affected customers. There's clearly the new method added to hot add logic in the latest version of VDDK 6.5 (which Update 3a brings with it). The new method makes a call to the Virtual Disk Service (VDS) of Microsoft Windows in a seemingly unoptimal manner: it queries each VM disk sequentially one by one, but also re-queries EVERY previously queried disk each time as it moves to query the next one. So the more disks the particular VM has, the exponentially longer the whole process takes. We shared all these findings with VMware Support.

The only unclear part is why these VDS queries, even if unoptimal, take very little time in most environments - but in some environments (28 to be precise) they take by an order of magnitude longer. For example, we're unable to reproduce the issue in our own test environments no matter what we do, including throwing huge load on every component.

omegagx
Enthusiast
Posts: 57
Liked: 3 times
Joined: May 09, 2017 6:33 pm
Full Name: Michael Gorn
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by omegagx » Jul 31, 2018 2:19 pm

We are still on 9.5.0.138. If we want to make sure to avoid this issue, should we hold off installing Update 3a and wait for Update 4 or is there a hotfix already available for this issue?

Thank you.

mcz
Expert
Posts: 273
Liked: 50 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by mcz » Jul 31, 2018 2:21 pm

Gostev wrote:There's clearly the new method added to hot add logic in the latest version of VDDK 6.5 (which Update 3a brings with it). The new method makes a call to the Virtual Disk Service (VDS) of Microsoft Windows in a seemingly unoptimal manner: it queries each VM disk sequentially one by one, but also re-queries EVERY previously queried disk each time as it moves to query the next one. So the more disks the particular VM has, the exponentially longer the whole process takes.
Wow sounds like you guys reverse engineered the vddk... :wink:

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Jul 31, 2018 2:25 pm 2 people like this post

@omegagx before we're able to produce the hot fix (or include one in Update 4), we need to get the new versions of VDDK 6.5 which contain a fix for this bug from VMware.

For now, there are two options of avoiding this issue:

1. Hold off installing Update 3a.

2. Install Update 3a and if you ARE affected by the issue (which is about 0.5% chance based on the current Update 3a download statistics and the number of support cases opened on this issue), then roll back VDDK version on your backup proxies to the previous version (used in Update 3) with the help of our support folks.

@mcz we're just trying to assist VMware in every possible way we can to accelerate the resolution for our mutual customers! But it certainly feels like to date, we've spent way more effort and man/hours troubleshooting this VDDK regression than they did.

mcz
Expert
Posts: 273
Liked: 50 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by mcz » Jul 31, 2018 2:51 pm 6 people like this post

Gostev wrote:@mcz we're just trying to assist VMware in every possible way we can to accelerate the resolution for our mutual customers! But it certainly feels like to date, we've spent way more effort and man/hours troubleshooting this VDDK regression than they did.
Well I think this is the reason why Veeam delivers such great products! I mean every other vendor would argue that the issue isn't within the managed source code and would refer to the vendor of the SDK. Veeam is pushing the limits by investigating every pice of software they are delivering to customers and you really can feel that. So big thank you to every Veeam team member for the big efforts!

syscons
Novice
Posts: 9
Liked: never
Joined: Oct 24, 2017 6:40 pm
Full Name: Jochen Seeger
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by syscons » Aug 07, 2018 7:49 am

Case # 03108110 seems to have the same Problem on one VM with multiple Disks, funny thing is that it started after an SQL Update on that machine (not the Veeam DB)

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 08, 2018 2:45 pm

VMware has just confirmed the issue, however they are not planning to provide the updated VDDK 6.5 build with the fix. Which means that most likely, we will have to roll back to using the previous version of VDDK 6.5 which does not yet have this bug. So basically, this is what our support does for the affected customers manually today.

I will update on the final decision in a few days.

Thanks!

NightBird
Service Provider
Posts: 175
Liked: 32 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by NightBird » Aug 09, 2018 5:08 pm

Same problem with vsphere 6.7 with the last VDDK 6.7 EP1 ? ==> ok issue resolved on 6.7 EP1...

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 10, 2018 12:55 pm

Correct, we will be using 6.7 EP1 in the next update for sure.

ngreen
Novice
Posts: 7
Liked: 1 time
Joined: Jun 15, 2017 2:48 pm
Full Name: Neil Green
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by ngreen » Aug 13, 2018 8:32 am

As I understand it, update 3a already uses VDDK 6.7 to backup vSphere 6.7 - is this the EP1 version? i.e. is this issue already fixed for vSphere 6.7 or do we need to wait for update 4 to fix this too?

We're still on 6.5 update 1 but getting twitchy about missing security patches. Have you got an approximate timescale for GA release of update 4 so we can move to 6.5u2 without downgrading VDDK 6.5? If this is going to be a while, is upgrading to 6.7 a safe option (from a backup perspective :))?

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 13, 2018 12:42 pm

Update 3a includes plain VDDK 6.7, not the Express Patch 1 build.

Our current plan is as follows:
1. Release a hot fix for Update 3a that brings older VDDK 6.5 back for vSphere 6.5 processing, and VDDK 6.7 EP1 for vSphere 6.7 processing.
2. In the Update 4, use VDDK 6.7 EP1 for both vSphere 6.5 and vSphere 6.7 (unless some our testing discovers some blockers).

Upgrading to vSphere 6.7 is a safe option from a backup perspective due to the vSphere API reliability issue that is specific to vSphere 6.5 U2.

ngreen
Novice
Posts: 7
Liked: 1 time
Joined: Jun 15, 2017 2:48 pm
Full Name: Neil Green
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by ngreen » Aug 14, 2018 8:15 am

Thanks for the update. Yes I'd been following that other thread but think I got them mixed up as being part of the same issue!

We've not experienced the hot add issue on 6.5U1 so I think we'll wait for the update 3a hotfix then move to 6.7 if there's no sign of a fix for the 6.5U2 API issue.

omegagx
Enthusiast
Posts: 57
Liked: 3 times
Joined: May 09, 2017 6:33 pm
Full Name: Michael Gorn
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by omegagx » Aug 14, 2018 3:49 pm

Hi
We are currently using Veeam 9.5.0.1038 to backup ESX 6.0 Update 2. Does that mean we would not be potentially affected by this issue if we upgrade to Veeam 9.5 Update 3a? We do have a few VMs with a very large number of large disks.
Thanks a lot.

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 14, 2018 10:32 pm

That is correct, Michael.

agoldenlife
Influencer
Posts: 17
Liked: 1 time
Joined: Mar 03, 2014 5:45 pm
Full Name: James Golden
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by agoldenlife » Aug 16, 2018 7:15 pm

Gostev wrote:VMware has just confirmed the issue, however they are not planning to provide the updated VDDK 6.5 build with the fix. Which means that most likely, we will have to roll back to using the previous version of VDDK 6.5 which does not yet have this bug. So basically, this is what our support does for the affected customers manually today.
This doesn't seem to be in my case. We updated, are having the issue, but I was told to enable NDB failover. Soo....

Full disclosure while troubleshooting why VCSA was not using hotadd I was instructed to turn off "failover to network mode". Then for the next several days we have had a lot of backup/hotadd failures. I had not looked at the logs since updating. So when I looked I found that since the update our hotadd were taking 1to2 hours for some, others a few minutes.

I just turned the "failover" option back on, but I am left wondering why I should need to.

Case # 03148615

Could anyone shed some light on what is the real fix. This is going to make our already bad backup situation worse. I was hoping we would not be in the .5%.. although is that number greater?

syscons
Novice
Posts: 9
Liked: never
Joined: Oct 24, 2017 6:40 pm
Full Name: Jochen Seeger
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by syscons » Aug 16, 2018 7:21 pm

syscons wrote:Case # 03108110 seems to have the same Problem on one VM with multiple Disks, funny thing is that it started after an SQL Update on that machine (not the Veeam DB)
Support downgraded the vddk Version and the Backup Time went down from 3 Hours back to around 9 Minutes!

bovie2k
Novice
Posts: 9
Liked: 1 time
Joined: Jan 30, 2017 2:14 am
Full Name: Derrick Bovenkamp
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by bovie2k » Aug 16, 2018 9:50 pm

Support gave me the VDDK 6.7 version (I think EP1) and it only works with 6.7 hosts. Problem is I have 5 6.7 hosts and two hosts that are stuck at 6.5 per Dell. Support says the fix only works with 6.7 hosts and I can't have any 6.0 or 6.5 hosts for it to work.

If I use the new VDDK support gave me I get the following 3 errors,
8/16/2018 2:03:54 PM :: Unable to hot add source disk, failing over to network mode...
8/16/2018 2:03:59 PM :: Hard disk 1 (0.0 B) 0.0 B read at 0 KB/s [CBT][fast clone]
8/16/2018 2:04:25 PM :: Error: The specified transports are not available. Available transports: [nbd]. Specified transports: [nbdssl]. VMDK path: [PATH HERE].
Failed to open disk for read.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.

with the Update 3a VDDK I get slowness sometime over an hour to mount a disk. So for now I've fallen back on nbdssl which is fast in my environment but not as fast as hot add. What should I do? Ask support for the older VDDK?

Thanks,
Derrick

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 17, 2018 11:58 am

Yes.

bovie2k
Novice
Posts: 9
Liked: 1 time
Joined: Jan 30, 2017 2:14 am
Full Name: Derrick Bovenkamp
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by bovie2k » Aug 17, 2018 3:52 pm

Support said there wasn't an older VDDK since 6.7 wasn't supported before Update 3a. Just the VDDK 6.7EP1 was available but that doesn't work with 6.0 and 6.5 hosts.

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 20, 2018 3:50 pm

Gostev wrote:Our current plan is as follows:
1. Release a hot fix for Update 3a that brings older VDDK 6.5 back for vSphere 6.5 processing, and VDDK 6.7 EP1 for vSphere 6.7 processing.
The hotfix is now available through our technical support. Thanks!

bovie2k
Novice
Posts: 9
Liked: 1 time
Joined: Jan 30, 2017 2:14 am
Full Name: Derrick Bovenkamp
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by bovie2k » Aug 20, 2018 4:31 pm

Gostev,

I called support this morning and they didn't know anything about the hotfix. They were very nice but said I'd have to wait for Update 4, Revert to vSphere 6.5 or keep in Network Mode. I have vSphere 6.7, 7 x 6.7 Hosts and 2 x 6.5 Hosts. Network mode is working fine its just slower than Hot Add. What do you suggest I do?

Thanks,
Bovie2k

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 21, 2018 12:33 pm

Come back to them again today :D due to the time difference, it may take up to a day for all support tiers and locations to become aware of the new hot fix.

postict
Novice
Posts: 4
Liked: never
Joined: Jun 22, 2012 12:48 pm
Full Name: Jan Post
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by postict » Aug 22, 2018 8:08 am

We encounter the same issue here upgrading to U3a, Hotadd is taking up to more than 1 hr for the actual backup cna start.
Opened a case (03155603) at support yesterday (August 21) and the response was a link to a VMWare KB and as solution to upgrade to vSphere 6.7 or changing the transport mode. Both are currently not an option.

Unfortunately the possibility of an available Hotfix was not mentioned. Is there a reference nr to the available hotfix I can refer to?
I'll ask them today.

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 22, 2018 3:15 pm

Yes, just ask them again today. They're also planning to create the official KB article in the coming days.

bovie2k
Novice
Posts: 9
Liked: 1 time
Joined: Jan 30, 2017 2:14 am
Full Name: Derrick Bovenkamp
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by bovie2k » Aug 22, 2018 3:17 pm

So I was told the Hotfix only works if I revert everything back to vCenter 6.5. I currently have vCenter 6.7 and all but two of my hosts running 6.7. Those two remaining hosts run 6.5 because that the highest they support. Can someone confirm the hotfix doesn't work with 6.7? and that update4 will fix this issue? For now I have reverted to nbdssl transport mode.

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 22, 2018 3:36 pm

This is an incorrect statement by your support engineer, as the hotfix does not have any vCenter Server version limitations.

postict
Novice
Posts: 4
Liked: never
Joined: Jun 22, 2012 12:48 pm
Full Name: Jan Post
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by postict » Aug 22, 2018 5:40 pm

Did asked support today and they will on our request transfer the case to the "VMware integration team".
Must say I've to push them for answers, referring to your "digest" from begin august ad this threat that resulted that the case will be transferred. If I had not "pushed" to only option would have been upgrade ( on the fly) form ESXi 6.0 to 6.7. That is obviously not a good practice in a production environment.
I got the (temporarily) Fix instructions, but that seems to be targeted to ESXi 6.5 and we are running ESXi6.0 with vCenter 6.5.

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 23, 2018 11:24 am

You can just refer your support engineer to xwiki bug ID 138831 to get the hotfix.

bovie2k
Novice
Posts: 9
Liked: 1 time
Joined: Jan 30, 2017 2:14 am
Full Name: Derrick Bovenkamp
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by bovie2k » Aug 23, 2018 10:48 pm 1 person likes this post

I got the hotfix and its working perfectly. Thanks for the quick work on this issue Gostev.

postict
Novice
Posts: 4
Liked: never
Joined: Jun 22, 2012 12:48 pm
Full Name: Jan Post
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by postict » Aug 24, 2018 5:47 am

Thanks for the ID.
Got the fix and replaced on all our proxies the VDDK_6_5 directory (renamed the old directory) and can confirm that the issue with the hotadd extensive delay has been solved by this fix.

Post Reply

Who is online

Users browsing this forum: No registered users and 12 guests