Discussions specific to the VMware vSphere hypervisor
Post Reply
ndolson
Influencer
Posts: 13
Liked: 2 times
Joined: Jan 08, 2015 3:56 pm
Full Name: Neal
Contact:

Re: Repeated VDDK error 1 and 2

Post by ndolson » Aug 10, 2016 1:39 pm

I restarted the vpxa and hostd services on July 17th and haven't had this issue return yet, for what it's worth. It took a little over 8 weeks from the last time the hosts were patched and rebooted til this issue manifested itself the first time.

drose0
Service Provider
Posts: 1
Liked: never
Joined: Nov 21, 2011 5:14 pm
Full Name: DJ Rose
Contact:

Re: Repeated VDDK error 1 and 2

Post by drose0 » Aug 12, 2016 5:00 pm

We are having the same issues. Each day this week we've had a few fail. Each day they were on the same host, but that host has changed daily.

Support case #01878241.

EBoucq
Expert
Posts: 217
Liked: 15 times
Joined: Jan 21, 2011 10:10 am
Full Name: Eric Boucq
Contact:

Re: Repeated VDDK error 1 and 2

Post by EBoucq » Sep 06, 2016 10:06 am

Hi,

When I had this issue one day, I got a few jobs I couldn't stop the proper way (stayed in "stopping" status). When I restarted my Veeam server, the ESXi hosting the VMs with the VDDK error 1 suddenly got disconnected from the vCenter.
It's obviously not a coincidence but I'd like to understand what could technically cause a host to stop responding and disconnect from a vcenter because a veeam server is being restarted.
Any idea ?
Thanks.
Eric

peterjm
Lurker
Posts: 1
Liked: never
Joined: Sep 08, 2016 10:36 pm
Full Name: Peter McGrath
Contact:

Re: Repeated VDDK error 1 and 2

Post by peterjm » Sep 08, 2016 10:53 pm

Were running Veeam B&R 9 U2 on VMware 5.5 with this issue. We started with Veeam B&R U1 when we were first experiencing it. We migrate the guests off the host and then reboot the host and migrate them back each time. Next time it happens I will try restarting the management agents instead.

I am Case # 01900486

vmhosts_ge
Service Provider
Posts: 4
Liked: 3 times
Joined: Jul 04, 2016 9:30 am
Full Name: Gary Eastwood
Contact:

Re: Repeated VDDK error 1 and 2

Post by vmhosts_ge » Sep 14, 2016 4:27 pm

Veeam Support Case 01904571

Same issue, vMotioning the affected VM's allows them to be backed up. Have tested vMotioning back after a successful backup without a host reboot or restarting the management services and it backs up successfully still.

It has happened to 2 different hosts in 2 separate Virtual Centers.

joechay
Novice
Posts: 3
Liked: 1 time
Joined: Sep 15, 2016 1:09 am
Full Name: Joe Chay
Contact:

Re: Repeated VDDK error 1 and 2

Post by joechay » Sep 15, 2016 3:12 am

Case #01902976

I first having some issues where the VMs start to fail but eventually completed after few retries on the backup jobs. It happen after the version is updated to Veeam Backup Replicator v9.01715.

Eventually all my VMs jobs start failing with the following VDDK error 1 and 2.

I have try the following (not in sequence)

- Verify connection from vCenter to Veeam backup server
- Restart vCenter
- Rebooting ESXi host

And then I received SOAP error (network connectivity issue)
- Add and remove vNIC adaptor on my Veeam VM Server
- Set proxy selection from automatic to fix to my Veeam server.

My backup is running fine now, but looks like for many of you the issues will return back?

pidthepiper
Enthusiast
Posts: 80
Liked: 7 times
Joined: Aug 11, 2015 9:10 am
Full Name: Bilal AHmed
Contact:

Re: Repeated VDDK error 1 and 2

Post by pidthepiper » Sep 19, 2016 2:10 pm

ok So I have had this happen again to one of my other hosts, still no resolution at all.

Its like Veeam have stood still with it and so have VMware

virtual0ne
Lurker
Posts: 1
Liked: never
Joined: Sep 19, 2016 4:50 pm
Full Name: Troy Egan
Contact:

Re: Repeated VDDK error 1 and 2

Post by virtual0ne » Sep 19, 2016 5:23 pm

Last night 2 VMs failed initially with this error, but after the 3rd retry, one of the VMs finally worked, while the second one still failed. There were 6 VMs on this particular host being backed up, so 4 of them worked with no issue. My guess is it will go through tonight without me doing anything to the host, as this has been the case in the past. Having said this, a few months ago an entire host's backed up VMs kept failing and I had to restart the management interface to fix it.

Veeam - 8.0.0.2084
VMware - 6.0 build 3620759

AdamR
Novice
Posts: 4
Liked: never
Joined: Nov 04, 2013 11:42 am
Full Name: Adam
Contact:

Re: Repeated VDDK error 1 and 2

Post by AdamR » Sep 26, 2016 10:57 am

We are having the same issues.
We rebooted one of the hosts and the problem disappeared for two days, currently we are waiting if the problem does not appear again.

Veeam - 9.0.0.1715
VMware - 5.5.0

krogerss
Enthusiast
Posts: 66
Liked: 1 time
Joined: Aug 18, 2014 6:06 pm
Contact:

Re: Repeated VDDK error 1 and 2

Post by krogerss » Oct 03, 2016 12:32 pm

So is there no fix in sight for this??? This issue has been out there for so long, just thought I would check in. It's getting to be a real nuisance. Thanks.

Ice-Dog
Influencer
Posts: 11
Liked: 2 times
Joined: Feb 01, 2016 10:39 am
Full Name: Gunnar Olafsson
Contact:

Re: Repeated VDDK error 1 and 2

Post by Ice-Dog » Oct 04, 2016 8:41 am

There is no fix for this yet. Every time this happens I need to restart the ESXi Management Interface of the effected host.

spf62
Enthusiast
Posts: 65
Liked: never
Joined: Sep 17, 2010 12:33 pm
Full Name: Steve Fritsch
Location: Connecticut
Contact:

Re: Repeated VDDK error 1 and 2

Post by spf62 » Oct 06, 2016 12:48 pm

I have found that restarting the management agents work every time. But this tends to be a pain in the a$$ after a while.
We are running Veeam 9u2, ESXi 5.5 latest build

Hopefully VMWARE and VEEAM can get this worked out

tsightler
Veeam Software
Posts: 5199
Liked: 2082 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Repeated VDDK error 1 and 2

Post by tsightler » Oct 07, 2016 7:20 pm 2 people like this post

Please make sure you have the following VMware patches installed if you are experiencing this issue. These patches were released in August 2016 and address one of the more common issues that cause this failure. I notice a lot of people above are using builds that are prior to these patches:

https://kb.vmware.com/selfservice/micro ... Id=2144799

Note that if restarting the management agents resolve the error you can be almost certain that the issue is not caused by Veeam and thus having tickets open with VMware is also highly recommended if you want to see the issue resolved. In many of these cases you can correlate the times of the failures to errors in the logs on the VMware hosts.

A VDDK error 1 is basically a generic error that Veeam requested a disk block from the VDDK API (also a VMware component) and it was unable to be retrieved. If your backups are running in network/NBD mode, you can be almost sure that you'll see a corresponding error on the host, one of the more common is an out of memory error. Restarting the agents simply frees memory and allows new memory to be allocated. It's certainly possible that the issues could be caused by other problems, corrupted VMDK for example, but obviously restarting management agents wouldn't fix that issue, so if restarting the management agents is the fix, then you need to figure out what's going wrong with the management agents.

Of course I'm not suggesting you can't open a ticket with Veeam, but if you do, be prepared to push the issue to completion. I looked at a number of the cases in this thread and most closed with "No response from customers", many of them did not even upload logs. These types of issues are difficult to resolve because, in many cases, once the issue is resolved, the person opening the ticket loses motivation to pursue the root cause. This is especially true if the root cause is outside of Veeam so requires opening a ticket and working with VMware as well, however, that's likely the only way to find the root cause.

There are also many other possible causes for the error, but the agent issue is probably the most common I have seen.

krogerss
Enthusiast
Posts: 66
Liked: 1 time
Joined: Aug 18, 2014 6:06 pm
Contact:

Re: Repeated VDDK error 1 and 2

Post by krogerss » Oct 09, 2016 2:43 pm

I opened a case (# 01867282) regarding this a couple months ago and was told that Veeam had a case open with VMware for this issue. I was just told that it was a known issue and there was nothing that could be done except trying a couple of workarounds such as rebooting the host of the affected machines. This has happened two times in the past week. So do I need to open another case or strictly seek VMware support? I can't keep babysitting multiple backup failures. My case was closed on 9-16-16 and I wasn't told of these patches you mentioned. Thanks.

tsightler
Veeam Software
Posts: 5199
Liked: 2082 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Repeated VDDK error 1 and 2

Post by tsightler » Oct 10, 2016 1:46 pm 1 person likes this post

First of all I would verify you have installed the patches I mentioned above as these correct at least one of the more common causes of the issues.

So I'll put on my customer hat and tell you what I'd do if I were still a customer. I would most likely open a case with both Veeam and VMware, and tell each that I was doing so. I'd explain to VMware that I know rebooting the host/restarting the agents will resolve the issue but I'd like to understand what state the agents are in that they require a restart, and why they are getting into this state. This will hopefully help them not just try to send me back to Veeam support, but recognize that there really is a problem with the agent. Regardless, I wouldn't expect Veeam support to just tell me that it's a VMware problem and close the ticket, and I certainly wouldn't accept that answer. I'd want a root cause analysis to determine what specifically was causing the error in my environment, even if it happens to be caused by VMware. Perhaps this will require a joing call with both Veeam and VMware support, but this should certainly be possible.

My guess is the logs of the VMware agent have all of the information that are really needed to determine the cause of this error, but I could be wrong. I'm currently working with a couple of customers that are experiencing this issue on a semi-regular basis, so hopefully if we continue to investigate we'll be able to get to the bottom if it.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 31 guests