Repeated VDDK error 1 and 2

[ericcsinger]

Hi,

I've moved on to CommVault for our VM backup's, but we too are still seeing what I think is a similar issue. We saw the issue in Veeam on ESXi v5.5 and it followed with CV on ESXi v6. I opened a case with VMware and this is the KB they returned.

https://kb.vmware.com/selfservice/micro ... Id=1022543

Ultimately I think its a memory leak on their side, but that's tough to prove considering they couldn't tell me how to monitor the memory consumed, or even see the number of connections supposedly opened. As you can see the KB has existed since 2014, so it doesn't look like its anything that VMware is too concerned about. I realize the KB is just telling you what will happen when you hit the limit and ultimately doesn't tell you how to resolve the issue (other than restarting the management services), but perhaps if enough folks open a case and link to this KB, we can get some traction in VMware. It does however tell you where to look in the logs to see if you're in fact hitting the limit, so that might be of use too. Also, I realize the KB doesn't list ESXi or vCenter 6, but the VMware support person told me its still applicable. Finally, they of course blamed it on the backup vendor for leaving orphaned connections, hard to believe both CV *and* Veeam are doing this, but maybe its possible.

Hope that helps.

BTW, I don't recall if Veeam does this, but it would be helpful to throttle NFC connections per host and per vCenter in total, otherwise you'll hit this in the best of conditions. The limits for each are outlined here, and each disk counts as a unique connection. http://pubs.vmware.com/vsphere-50/index ... _4_0_2_3_1

Host Platform
When Connecting
Limits You To About
vSphere 4
to an ESX host
9 connections directly, 27 connections through vCenter Server
vSphere 4
to an ESXi host
11 connections directly, 23 connections through vCenter Server
vSphere 5
to an ESXi host
Limited by a transfer buffer for all NFC connections, enforced by the host; the sum of all NFC connection buffers to an ESXi host cannot exceed 32MB.
52 connections through vCenter Server, including the above per-host limit.

[picaroon]

After months of collecting logs and discussing with both Veeam and VMware SDK team, this is the last message I've received from VMware after enabling VDDK loglevel 6 and creating a live dump of the vxpa process:
Engineering analyzed the live dump and below is their findings:

From esxcfg-resgrp_-l.txt
Memory limit for vpxa : 311296 = 304 MB
Also from the memstats output, we can see the active/consumed memory of vpxa process is approx 295 MB.

From the captured vpxa core dump, here is the ah64 output count allocated Counting allocated chunks
8060522 chunks take 0x1032131c (271,717,148) bytes.
summarize allocated
Summarizing allocated chunks
Unsigned chunks have 5575559 instances taking 0xa358280(171278976) bytes.
Signature 179973bc (Vim::ElementDescription) has 1817283 instances taking 0x377d5b0(58185136) bytes.
Signature 179a169c (Vim::PerformanceManager::CounterInfo) has 605414 instances taking 0x2500f10(38801168) bytes.
Signature 15a8c904 (Vmomi::DataPropertyImpl) has 7498 instances taking 0x75388(480136) bytes.

PerformanceManager::CounterInfo instance count is very high.

This is a known issue and it is fixed in 55 P08(vSphere 5.5 U3e). Engineering suggested to upgrade the ESXi to P08 to avoid the abnormal memory growth of vpxa.

I haven't been able to update the environment, since I changed jobs. But according to this thread, the issue still exists with current versions...

[pidthepiper]

Ive been following this thread closely for a while, I lost all hope for it for ages. I made sure everyone was aware that if a bunch of VMs failed during backup and they were all on one host just do a services.sh restart and try again. Its happened about once a month randomly for us, I deployed a few more proxies and made sure they only used hotadd wherever possible.

I was unaware of this patch so I am going to add it to my list of roll outs, as it fixes another random issue for us :
Attempts to power on a virtual machine after upgrading the hardware version might fail. This is an issue seen on ESXi 5.5 Update 3a or later. An error message similar to the following is displayed:
cannot open the disk vmdk or one of the snapshot disks it depends on. Could not open/create change tracking file.

I am on 5.5u3b currently

[picaroon]

It has been a while, but it seems VMware fixed this by releasing patches on 2017-08-04: https://kb.vmware.com/selfservice/micro ... Id=2144799

After installing those patches (or later versions) we haven't seen the issues anymore.

[mef]

Really old topic, but I ran into this recently. A disconnect/reconnect of the host inside of vCenter allowed the backups to run successfully in my situation