Discussions specific to the VMware vSphere hypervisor
Gostev
SVP, Product Management
Posts: 24020
Liked: 3255 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Jul 31, 2018 2:12 pm

VMware Support is still "researching" the issue, which is the state that we all know can potentially take a very long time :D

From Veeam perspective, the issue is fully understood thanks to the verbose VDDK debug logs obtained from the affected customers. There's clearly the new method added to hot add logic in the latest version of VDDK 6.5 (which Update 3a brings with it). The new method makes a call to the Virtual Disk Service (VDS) of Microsoft Windows in a seemingly unoptimal manner: it queries each VM disk sequentially one by one, but also re-queries EVERY previously queried disk each time as it moves to query the next one. So the more disks the particular VM has, the exponentially longer the whole process takes. We shared all these findings with VMware Support.

The only unclear part is why these VDS queries, even if unoptimal, take very little time in most environments - but in some environments (28 to be precise) they take by an order of magnitude longer. For example, we're unable to reproduce the issue in our own test environments no matter what we do, including throwing huge load on every component.

omegagx
Enthusiast
Posts: 49
Liked: 2 times
Joined: May 09, 2017 6:33 pm
Full Name: Michael Gorn
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by omegagx » Jul 31, 2018 2:19 pm

We are still on 9.5.0.138. If we want to make sure to avoid this issue, should we hold off installing Update 3a and wait for Update 4 or is there a hotfix already available for this issue?

Thank you.

mcz
Expert
Posts: 253
Liked: 48 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by mcz » Jul 31, 2018 2:21 pm

Gostev wrote:There's clearly the new method added to hot add logic in the latest version of VDDK 6.5 (which Update 3a brings with it). The new method makes a call to the Virtual Disk Service (VDS) of Microsoft Windows in a seemingly unoptimal manner: it queries each VM disk sequentially one by one, but also re-queries EVERY previously queried disk each time as it moves to query the next one. So the more disks the particular VM has, the exponentially longer the whole process takes.
Wow sounds like you guys reverse engineered the vddk... :wink:

Gostev
SVP, Product Management
Posts: 24020
Liked: 3255 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Jul 31, 2018 2:25 pm 2 people like this post

@omegagx before we're able to produce the hot fix (or include one in Update 4), we need to get the new versions of VDDK 6.5 which contain a fix for this bug from VMware.

For now, there are two options of avoiding this issue:

1. Hold off installing Update 3a.

2. Install Update 3a and if you ARE affected by the issue (which is about 0.5% chance based on the current Update 3a download statistics and the number of support cases opened on this issue), then roll back VDDK version on your backup proxies to the previous version (used in Update 3) with the help of our support folks.

@mcz we're just trying to assist VMware in every possible way we can to accelerate the resolution for our mutual customers! But it certainly feels like to date, we've spent way more effort and man/hours troubleshooting this VDDK regression than they did.

mcz
Expert
Posts: 253
Liked: 48 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by mcz » Jul 31, 2018 2:51 pm 6 people like this post

Gostev wrote:@mcz we're just trying to assist VMware in every possible way we can to accelerate the resolution for our mutual customers! But it certainly feels like to date, we've spent way more effort and man/hours troubleshooting this VDDK regression than they did.
Well I think this is the reason why Veeam delivers such great products! I mean every other vendor would argue that the issue isn't within the managed source code and would refer to the vendor of the SDK. Veeam is pushing the limits by investigating every pice of software they are delivering to customers and you really can feel that. So big thank you to every Veeam team member for the big efforts!

syscons
Novice
Posts: 9
Liked: never
Joined: Oct 24, 2017 6:40 pm
Full Name: Jochen Seeger
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by syscons » Aug 07, 2018 7:49 am

Case # 03108110 seems to have the same Problem on one VM with multiple Disks, funny thing is that it started after an SQL Update on that machine (not the Veeam DB)

Gostev
SVP, Product Management
Posts: 24020
Liked: 3255 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 08, 2018 2:45 pm

VMware has just confirmed the issue, however they are not planning to provide the updated VDDK 6.5 build with the fix. Which means that most likely, we will have to roll back to using the previous version of VDDK 6.5 which does not yet have this bug. So basically, this is what our support does for the affected customers manually today.

I will update on the final decision in a few days.

Thanks!

NightBird
Service Provider
Posts: 172
Liked: 32 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by NightBird » Aug 09, 2018 5:08 pm

Same problem with vsphere 6.7 with the last VDDK 6.7 EP1 ? ==> ok issue resolved on 6.7 EP1...

Gostev
SVP, Product Management
Posts: 24020
Liked: 3255 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 10, 2018 12:55 pm

Correct, we will be using 6.7 EP1 in the next update for sure.

ngreen
Novice
Posts: 7
Liked: 1 time
Joined: Jun 15, 2017 2:48 pm
Full Name: Neil Green
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by ngreen » Aug 13, 2018 8:32 am

As I understand it, update 3a already uses VDDK 6.7 to backup vSphere 6.7 - is this the EP1 version? i.e. is this issue already fixed for vSphere 6.7 or do we need to wait for update 4 to fix this too?

We're still on 6.5 update 1 but getting twitchy about missing security patches. Have you got an approximate timescale for GA release of update 4 so we can move to 6.5u2 without downgrading VDDK 6.5? If this is going to be a while, is upgrading to 6.7 a safe option (from a backup perspective :))?

Gostev
SVP, Product Management
Posts: 24020
Liked: 3255 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 13, 2018 12:42 pm

Update 3a includes plain VDDK 6.7, not the Express Patch 1 build.

Our current plan is as follows:
1. Release a hot fix for Update 3a that brings older VDDK 6.5 back for vSphere 6.5 processing, and VDDK 6.7 EP1 for vSphere 6.7 processing.
2. In the Update 4, use VDDK 6.7 EP1 for both vSphere 6.5 and vSphere 6.7 (unless some our testing discovers some blockers).

Upgrading to vSphere 6.7 is a safe option from a backup perspective due to the vSphere API reliability issue that is specific to vSphere 6.5 U2.

ngreen
Novice
Posts: 7
Liked: 1 time
Joined: Jun 15, 2017 2:48 pm
Full Name: Neil Green
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by ngreen » Aug 14, 2018 8:15 am

Thanks for the update. Yes I'd been following that other thread but think I got them mixed up as being part of the same issue!

We've not experienced the hot add issue on 6.5U1 so I think we'll wait for the update 3a hotfix then move to 6.7 if there's no sign of a fix for the 6.5U2 API issue.

omegagx
Enthusiast
Posts: 49
Liked: 2 times
Joined: May 09, 2017 6:33 pm
Full Name: Michael Gorn
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by omegagx » Aug 14, 2018 3:49 pm

Hi
We are currently using Veeam 9.5.0.1038 to backup ESX 6.0 Update 2. Does that mean we would not be potentially affected by this issue if we upgrade to Veeam 9.5 Update 3a? We do have a few VMs with a very large number of large disks.
Thanks a lot.

Gostev
SVP, Product Management
Posts: 24020
Liked: 3255 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by Gostev » Aug 14, 2018 10:32 pm

That is correct, Michael.

agoldenlife
Influencer
Posts: 17
Liked: 1 time
Joined: Mar 03, 2014 5:45 pm
Full Name: James Golden
Contact:

Re: Increase in Hot Add backup time after installing Update

Post by agoldenlife » Aug 16, 2018 7:15 pm

Gostev wrote:VMware has just confirmed the issue, however they are not planning to provide the updated VDDK 6.5 build with the fix. Which means that most likely, we will have to roll back to using the previous version of VDDK 6.5 which does not yet have this bug. So basically, this is what our support does for the affected customers manually today.
This doesn't seem to be in my case. We updated, are having the issue, but I was told to enable NDB failover. Soo....

Full disclosure while troubleshooting why VCSA was not using hotadd I was instructed to turn off "failover to network mode". Then for the next several days we have had a lot of backup/hotadd failures. I had not looked at the logs since updating. So when I looked I found that since the update our hotadd were taking 1to2 hours for some, others a few minutes.

I just turned the "failover" option back on, but I am left wondering why I should need to.

Case # 03148615

Could anyone shed some light on what is the real fix. This is going to make our already bad backup situation worse. I was hoping we would not be in the .5%.. although is that number greater?

Post Reply

Who is online

Users browsing this forum: fondazionetls and 43 guests