Increase in Hot Add backup time after installing Update 3a

[Gostev]

VMware Support is still "researching" the issue, which is the state that we all know can potentially take a very long time

From Veeam perspective, the issue is fully understood thanks to the verbose VDDK debug logs obtained from the affected customers. There's clearly the new method added to hot add logic in the latest version of VDDK 6.5 (which Update 3a brings with it). The new method makes a call to the Virtual Disk Service (VDS) of Microsoft Windows in a seemingly unoptimal manner: it queries each VM disk sequentially one by one, but also re-queries EVERY previously queried disk each time as it moves to query the next one. So the more disks the particular VM has, the exponentially longer the whole process takes. We shared all these findings with VMware Support.

The only unclear part is why these VDS queries, even if unoptimal, take very little time in most environments - but in some environments (28 to be precise) they take by an order of magnitude longer. For example, we're unable to reproduce the issue in our own test environments no matter what we do, including throwing huge load on every component.

[omegagx]

We are still on 9.5.0.138. If we want to make sure to avoid this issue, should we hold off installing Update 3a and wait for Update 4 or is there a hotfix already available for this issue?

Thank you.

[mcz]

There's clearly the new method added to hot add logic in the latest version of VDDK 6.5 (which Update 3a brings with it). The new method makes a call to the Virtual Disk Service (VDS) of Microsoft Windows in a seemingly unoptimal manner: it queries each VM disk sequentially one by one, but also re-queries EVERY previously queried disk each time as it moves to query the next one. So the more disks the particular VM has, the exponentially longer the whole process takes.

Wow sounds like you guys reverse engineered the vddk...

[Gostev]

@omegagx before we're able to produce the hot fix (or include one in Update 4), we need to get the new versions of VDDK 6.5 which contain a fix for this bug from VMware.

For now, there are two options of avoiding this issue:

1. Hold off installing Update 3a.

2. Install Update 3a and if you ARE affected by the issue (which is about 0.5% chance based on the current Update 3a download statistics and the number of support cases opened on this issue), then roll back VDDK version on your backup proxies to the previous version (used in Update 3) with the help of our support folks.

@mcz we're just trying to assist VMware in every possible way we can to accelerate the resolution for our mutual customers! But it certainly feels like to date, we've spent way more effort and man/hours troubleshooting this VDDK regression than they did.

[mcz]

@mcz we're just trying to assist VMware in every possible way we can to accelerate the resolution for our mutual customers! But it certainly feels like to date, we've spent way more effort and man/hours troubleshooting this VDDK regression than they did.

Well I think this is the reason why Veeam delivers such great products! I mean every other vendor would argue that the issue isn't within the managed source code and would refer to the vendor of the SDK. Veeam is pushing the limits by investigating every pice of software they are delivering to customers and you really can feel that. So big thank you to every Veeam team member for the big efforts!

[syscons]

Case # 03108110 seems to have the same Problem on one VM with multiple Disks, funny thing is that it started after an SQL Update on that machine (not the Veeam DB)

[Gostev]

VMware has just confirmed the issue, however they are not planning to provide the updated VDDK 6.5 build with the fix. Which means that most likely, we will have to roll back to using the previous version of VDDK 6.5 which does not yet have this bug. So basically, this is what our support does for the affected customers manually today.

I will update on the final decision in a few days.

Thanks!

[NightBird]

Same problem with vsphere 6.7 with the last VDDK 6.7 EP1 ? ==> ok issue resolved on 6.7 EP1...

[Gostev]

Correct, we will be using 6.7 EP1 in the next update for sure.

[ngreen]

As I understand it, update 3a already uses VDDK 6.7 to backup vSphere 6.7 - is this the EP1 version? i.e. is this issue already fixed for vSphere 6.7 or do we need to wait for update 4 to fix this too?

We're still on 6.5 update 1 but getting twitchy about missing security patches. Have you got an approximate timescale for GA release of update 4 so we can move to 6.5u2 without downgrading VDDK 6.5? If this is going to be a while, is upgrading to 6.7 a safe option (from a backup perspective )?

[Gostev]

Update 3a includes plain VDDK 6.7, not the Express Patch 1 build.

Our current plan is as follows:
1. Release a hot fix for Update 3a that brings older VDDK 6.5 back for vSphere 6.5 processing, and VDDK 6.7 EP1 for vSphere 6.7 processing.
2. In the Update 4, use VDDK 6.7 EP1 for both vSphere 6.5 and vSphere 6.7 (unless some our testing discovers some blockers).

Upgrading to vSphere 6.7 is a safe option from a backup perspective due to the vSphere API reliability issue that is specific to vSphere 6.5 U2.

[ngreen]

Thanks for the update. Yes I'd been following that other thread but think I got them mixed up as being part of the same issue!

We've not experienced the hot add issue on 6.5U1 so I think we'll wait for the update 3a hotfix then move to 6.7 if there's no sign of a fix for the 6.5U2 API issue.

[omegagx]

Hi
We are currently using Veeam 9.5.0.1038 to backup ESX 6.0 Update 2. Does that mean we would not be potentially affected by this issue if we upgrade to Veeam 9.5 Update 3a? We do have a few VMs with a very large number of large disks.
Thanks a lot.

[Gostev]

That is correct, Michael.

[agoldenlife]

VMware has just confirmed the issue, however they are not planning to provide the updated VDDK 6.5 build with the fix. Which means that most likely, we will have to roll back to using the previous version of VDDK 6.5 which does not yet have this bug. So basically, this is what our support does for the affected customers manually today.

This doesn't seem to be in my case. We updated, are having the issue, but I was told to enable NDB failover. Soo....

Full disclosure while troubleshooting why VCSA was not using hotadd I was instructed to turn off "failover to network mode". Then for the next several days we have had a lot of backup/hotadd failures. I had not looked at the logs since updating. So when I looked I found that since the update our hotadd were taking 1to2 hours for some, others a few minutes.

I just turned the "failover" option back on, but I am left wondering why I should need to.

Case # 03148615

Could anyone shed some light on what is the real fix. This is going to make our already bad backup situation worse. I was hoping we would not be in the .5%.. although is that number greater?

[syscons]

Case # 03108110 seems to have the same Problem on one VM with multiple Disks, funny thing is that it started after an SQL Update on that machine (not the Veeam DB)

Support downgraded the vddk Version and the Backup Time went down from 3 Hours back to around 9 Minutes!

[bovie2k]

Support gave me the VDDK 6.7 version (I think EP1) and it only works with 6.7 hosts. Problem is I have 5 6.7 hosts and two hosts that are stuck at 6.5 per Dell. Support says the fix only works with 6.7 hosts and I can't have any 6.0 or 6.5 hosts for it to work.

If I use the new VDDK support gave me I get the following 3 errors,
8/16/2018 2:03:54 PM :: Unable to hot add source disk, failing over to network mode...
8/16/2018 2:03:59 PM :: Hard disk 1 (0.0 B) 0.0 B read at 0 KB/s [CBT][fast clone]
8/16/2018 2:04:25 PM :: Error: The specified transports are not available. Available transports: [nbd]. Specified transports: [nbdssl]. VMDK path: [PATH HERE].
Failed to open disk for read.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.

with the Update 3a VDDK I get slowness sometime over an hour to mount a disk. So for now I've fallen back on nbdssl which is fast in my environment but not as fast as hot add. What should I do? Ask support for the older VDDK?

Thanks,
Derrick

[Gostev]

Yes.

[bovie2k]

Support said there wasn't an older VDDK since 6.7 wasn't supported before Update 3a. Just the VDDK 6.7EP1 was available but that doesn't work with 6.0 and 6.5 hosts.

[Gostev]

Our current plan is as follows:
1. Release a hot fix for Update 3a that brings older VDDK 6.5 back for vSphere 6.5 processing, and VDDK 6.7 EP1 for vSphere 6.7 processing.

The hotfix is now available through our technical support. Thanks!

[bovie2k]

Gostev,

I called support this morning and they didn't know anything about the hotfix. They were very nice but said I'd have to wait for Update 4, Revert to vSphere 6.5 or keep in Network Mode. I have vSphere 6.7, 7 x 6.7 Hosts and 2 x 6.5 Hosts. Network mode is working fine its just slower than Hot Add. What do you suggest I do?

Thanks,
Bovie2k

[Gostev]

Come back to them again today due to the time difference, it may take up to a day for all support tiers and locations to become aware of the new hot fix.

[postict]

We encounter the same issue here upgrading to U3a, Hotadd is taking up to more than 1 hr for the actual backup cna start.
Opened a case (03155603) at support yesterday (August 21) and the response was a link to a VMWare KB and as solution to upgrade to vSphere 6.7 or changing the transport mode. Both are currently not an option.

Unfortunately the possibility of an available Hotfix was not mentioned. Is there a reference nr to the available hotfix I can refer to?
I'll ask them today.

[Gostev]

Yes, just ask them again today. They're also planning to create the official KB article in the coming days.

[bovie2k]

So I was told the Hotfix only works if I revert everything back to vCenter 6.5. I currently have vCenter 6.7 and all but two of my hosts running 6.7. Those two remaining hosts run 6.5 because that the highest they support. Can someone confirm the hotfix doesn't work with 6.7? and that update4 will fix this issue? For now I have reverted to nbdssl transport mode.

[Gostev]

This is an incorrect statement by your support engineer, as the hotfix does not have any vCenter Server version limitations.

[postict]

Did asked support today and they will on our request transfer the case to the "VMware integration team".
Must say I've to push them for answers, referring to your "digest" from begin august ad this threat that resulted that the case will be transferred. If I had not "pushed" to only option would have been upgrade ( on the fly) form ESXi 6.0 to 6.7. That is obviously not a good practice in a production environment.
I got the (temporarily) Fix instructions, but that seems to be targeted to ESXi 6.5 and we are running ESXi6.0 with vCenter 6.5.

[Gostev]

You can just refer your support engineer to xwiki bug ID 138831 to get the hotfix.

[bovie2k]

I got the hotfix and its working perfectly. Thanks for the quick work on this issue Gostev.

[postict]

Thanks for the ID.
Got the fix and replaced on all our proxies the VDDK_6_5 directory (renamed the old directory) and can confirm that the issue with the hotadd extensive delay has been solved by this fix.