Repeated VDDK error 1 and 2

[pidthepiper]

VMware came back and told me that there was an issue because of a vpxa.cfg file on the ESXi hosts, but they don't have an idea of the cause. They did say that 'too many delta disks or cloned disks depending on other disks which adversely affects the vpxa agent' , which is interesting because there was a single snapshot on each of these machines which is the one Veeam was trying to create.

They wanted me to edit the vpxa file on each of the hosts and change 128 to 256 and restart vpxa then enable sfcdb temporarily for a core dump.

I'm not sure I want to proceed with this change given that their reason for doing it doesn't seem to be correct

I also have been told to change the VddkPreReadBufferSize on my case with the wording that
"it doesn't appear that the root cause of the issue has been found - however, it does appear to be related to ESXi hosts that are added via vCenter: the error itself seems to be produced due to an issue with the datastore or connection between the host and the datastore"

I'm somewhat surprised at that statement, since we had no issue on v7 or v8 whatsoever for nearly two years with no changes to our environment.

I'll be adjusting the VddkPrereadBufferSize value today but if Picaroons response is correct, I doubt we will see much difference

Thats interesting, But he errors have started appearing out of no where

[ndolson]

We are getting the VDDK error 2 message now. Running vSphere 6.0 U2 and Veeam B&R v9 U1. I can't say for sure since it's been a while but it seems like it started happening after the installation of U2, which is the last time those hosts were patched and rebooted.

Case ID 01850397

[pidthepiper]

We are all still waiting for an update from veeam really

[ndolson]

I was basically told that because restarting hostd and vpxa temporarily resolves the issue, it indicates a problem on the VMware side and I need to pursue the issue with them.

[mwirz]

i have noticed that this error only appears with my virtual machines running on nfs storage systems (netapp, nfs), virtual machines running on iscsi storage are fine all the time. what about your storage systems? nfs as well?

[ndolson]

Yep, the cluster in which its happening has its storage presented to it via NFS. Another small cluster with SAS connectivity is not experiencing it. Same ESXi version under same vCenter.

[mwirz]

i'll take the time to open a case as well. maybe i can bring some more logs to identify the root cause...

[rpop]

Hi all,

Same problem for me, but only on job replication.

31 VMs failed on 85 in the job

Code: Select all

16.07.2016 06:12:06 :: Error: VDDK error: 2 (Memory allocation failed. Out of memory.). Value: 0x0000000000000002
Failed to write to a virtual disk
Failed to download disk.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.

After reboot (or restart mgmt service) concerned hosts (target hosts), it's ok, but for how many time ?

Code: Select all

vCenter : 6.0.0.20000 (Appliance postgressql)
ESXi :      6.0.0 , 3825889 + (patch last bug VMXNET3 !!!)
Veeam :  9u1 , 1x B&R server as a VM , 2x physical proxy/repo(landing zone) servers, one for test/DR site , the other for production site), Network 10Gbps everywhere, even between sites.

Best regards

[Vitaliy S.]

rpop, please contact our support team directly when posting about a technical issue as it is advised by our forum rules. Thanks!

[WRStone]

We've been getting error 2 on random servers. We can typically fix it by restarting management interfaces, but then the problem will pop up and we'll have to do it again elsewhere.

At least for us, Veeam has washed their hands of the issue with the following:

I am glad that the restart was able to help get past the error. If this VMware error continues to persist then I would highly advise leveraging VMware support in on the issue to further investigate the root cause. For the VDDK 2 error, specifically, it would point to a memory allocation issue with the host of the VM being processed:

https://www.vmware.com/support/develope ... rrors.html

I'm going to open a support case with VMware and tell them that Veeam sent me. As this has been an ongoing issue for over six months, I'll post the root cause (assuming that VMware and Veeam can agree on one).

[duukie]

Same issue here
Veeam Support Case # 01862154

I was able to conference in both Veeam and VMware Support. They basically blame each other.

I've tried the following without success:
* vMotion to another host and datastore
* Rebooted host
* services.sh restart
* Cloned VM
* Created new job and sent to different repository.

VMware is reviewing Veems logs to compare date stamps.

Will update this thread....

[WRStone]

I was able to conference in both Veeam and VMware Support. They basically blame each other.

Thank you muchly. I've not yet gotten them together, but I assume that blaming each other is what would happen. I've seen it repeatedly.

We have a Veeam engineer in the office some of this week. You can rest assured that we'll be asking about this and will update the thread.

[WRStone]

The Veeam engineer was in today. While he didn't recognize the issue, he recognized that it's now a bouncing game between VMware and Veeam.

One thing he could reliably do is escalate my support ticket. He and the guy he was with were very clear that escalating is something they do a lot and are good at.

I'm to provide him with this forum thread and my support ticket so as to escalate. I'll post here as I learn more.

[WRStone]

Unfortunately, escalating our ticket didn't really help. Here's what we got back:

At the moment we have several cases open with the same issue, so here's the update:

As far as I can tell, we have a case open with VMware SDK support, and they are working on there end to identify/resolve the issue. Beyond that I don't see any update.

I understand that the issue is ongoing, and we seem to have possibly narrowed it down to being an issue with the vpxa service on the host. What I'd suggest trying, if you are using the temporary 'fix' of rebooting the ESXi host, is the following:

1) Instead of rebooting the host next time you see the VDDK error, try just restarting the vpxa service. If backup still fails, you might try restarting the vpxa service on the host as well as the vpxd service on the VC

2) Another test (if this hasn't already been attempted) would be to add the ESXi host to Veeam as a standalone - see https://www.veeam.com/kb1777 - then try backing up via the standalone.

Breaking it down, what that means is:

No progress has been made since the vxpa issue was reported in this forum over four weeks ago.

I have started to make a stink that I won't post. I'll continue to update.

[pidthepiper]

Yep basically the same thing SDK case open, VMware are like its a SDK issue for the SDK team and Veeam, have a case open with them, but nothing is happening.

We dont use NFS for any VM storage, we use FC.

I have mitigated the errors by simply using HotAdd specifically, but it means using nbd doesnt work well and thats a feature we are paying for that we cant use, which doesn't sit well with me

[ndolson]

I restarted the vpxa and hostd services on July 17th and haven't had this issue return yet, for what it's worth. It took a little over 8 weeks from the last time the hosts were patched and rebooted til this issue manifested itself the first time.

[drose0]

We are having the same issues. Each day this week we've had a few fail. Each day they were on the same host, but that host has changed daily.

Support case #01878241.

[EBoucq]

Hi,

When I had this issue one day, I got a few jobs I couldn't stop the proper way (stayed in "stopping" status). When I restarted my Veeam server, the ESXi hosting the VMs with the VDDK error 1 suddenly got disconnected from the vCenter.
It's obviously not a coincidence but I'd like to understand what could technically cause a host to stop responding and disconnect from a vcenter because a veeam server is being restarted.
Any idea ?
Thanks.
Eric

[peterjm]

Were running Veeam B&R 9 U2 on VMware 5.5 with this issue. We started with Veeam B&R U1 when we were first experiencing it. We migrate the guests off the host and then reboot the host and migrate them back each time. Next time it happens I will try restarting the management agents instead.

I am Case # 01900486

[vmhosts_ge]

Veeam Support Case 01904571

Same issue, vMotioning the affected VM's allows them to be backed up. Have tested vMotioning back after a successful backup without a host reboot or restarting the management services and it backs up successfully still.

It has happened to 2 different hosts in 2 separate Virtual Centers.

[joechay]

Case #01902976

I first having some issues where the VMs start to fail but eventually completed after few retries on the backup jobs. It happen after the version is updated to Veeam Backup Replicator v9.01715.

Eventually all my VMs jobs start failing with the following VDDK error 1 and 2.

I have try the following (not in sequence)

- Verify connection from vCenter to Veeam backup server
- Restart vCenter
- Rebooting ESXi host

And then I received SOAP error (network connectivity issue)
- Add and remove vNIC adaptor on my Veeam VM Server
- Set proxy selection from automatic to fix to my Veeam server.

My backup is running fine now, but looks like for many of you the issues will return back?

[pidthepiper]

ok So I have had this happen again to one of my other hosts, still no resolution at all.

Its like Veeam have stood still with it and so have VMware

[virtual0ne]

Last night 2 VMs failed initially with this error, but after the 3rd retry, one of the VMs finally worked, while the second one still failed. There were 6 VMs on this particular host being backed up, so 4 of them worked with no issue. My guess is it will go through tonight without me doing anything to the host, as this has been the case in the past. Having said this, a few months ago an entire host's backed up VMs kept failing and I had to restart the management interface to fix it.

Veeam - 8.0.0.2084
VMware - 6.0 build 3620759

[AdamR]

We are having the same issues.
We rebooted one of the hosts and the problem disappeared for two days, currently we are waiting if the problem does not appear again.

Veeam - 9.0.0.1715
VMware - 5.5.0

[krogerss]

So is there no fix in sight for this??? This issue has been out there for so long, just thought I would check in. It's getting to be a real nuisance. Thanks.

[Ice-Dog]

There is no fix for this yet. Every time this happens I need to restart the ESXi Management Interface of the effected host.

[spf62]

I have found that restarting the management agents work every time. But this tends to be a pain in the a$$ after a while.
We are running Veeam 9u2, ESXi 5.5 latest build

Hopefully VMWARE and VEEAM can get this worked out

[tsightler]

Please make sure you have the following VMware patches installed if you are experiencing this issue. These patches were released in August 2016 and address one of the more common issues that cause this failure. I notice a lot of people above are using builds that are prior to these patches:

https://kb.vmware.com/selfservice/micro ... Id=2144799

Note that if restarting the management agents resolve the error you can be almost certain that the issue is not caused by Veeam and thus having tickets open with VMware is also highly recommended if you want to see the issue resolved. In many of these cases you can correlate the times of the failures to errors in the logs on the VMware hosts.

A VDDK error 1 is basically a generic error that Veeam requested a disk block from the VDDK API (also a VMware component) and it was unable to be retrieved. If your backups are running in network/NBD mode, you can be almost sure that you'll see a corresponding error on the host, one of the more common is an out of memory error. Restarting the agents simply frees memory and allows new memory to be allocated. It's certainly possible that the issues could be caused by other problems, corrupted VMDK for example, but obviously restarting management agents wouldn't fix that issue, so if restarting the management agents is the fix, then you need to figure out what's going wrong with the management agents.

Of course I'm not suggesting you can't open a ticket with Veeam, but if you do, be prepared to push the issue to completion. I looked at a number of the cases in this thread and most closed with "No response from customers", many of them did not even upload logs. These types of issues are difficult to resolve because, in many cases, once the issue is resolved, the person opening the ticket loses motivation to pursue the root cause. This is especially true if the root cause is outside of Veeam so requires opening a ticket and working with VMware as well, however, that's likely the only way to find the root cause.

There are also many other possible causes for the error, but the agent issue is probably the most common I have seen.

[krogerss]

I opened a case (# 01867282) regarding this a couple months ago and was told that Veeam had a case open with VMware for this issue. I was just told that it was a known issue and there was nothing that could be done except trying a couple of workarounds such as rebooting the host of the affected machines. This has happened two times in the past week. So do I need to open another case or strictly seek VMware support? I can't keep babysitting multiple backup failures. My case was closed on 9-16-16 and I wasn't told of these patches you mentioned. Thanks.

[tsightler]

First of all I would verify you have installed the patches I mentioned above as these correct at least one of the more common causes of the issues.

So I'll put on my customer hat and tell you what I'd do if I were still a customer. I would most likely open a case with both Veeam and VMware, and tell each that I was doing so. I'd explain to VMware that I know rebooting the host/restarting the agents will resolve the issue but I'd like to understand what state the agents are in that they require a restart, and why they are getting into this state. This will hopefully help them not just try to send me back to Veeam support, but recognize that there really is a problem with the agent. Regardless, I wouldn't expect Veeam support to just tell me that it's a VMware problem and close the ticket, and I certainly wouldn't accept that answer. I'd want a root cause analysis to determine what specifically was causing the error in my environment, even if it happens to be caused by VMware. Perhaps this will require a joing call with both Veeam and VMware support, but this should certainly be possible.

My guess is the logs of the VMware agent have all of the information that are really needed to determine the cause of this error, but I could be wrong. I'm currently working with a couple of customers that are experiencing this issue on a semi-regular basis, so hopefully if we continue to investigate we'll be able to get to the bottom if it.