Repeated VDDK error 1 and 2

VMware specific discussions

Re: Repeated VDDK error 1 and 2

Veeam Logoby ndolson » Wed Aug 10, 2016 1:39 pm

I restarted the vpxa and hostd services on July 17th and haven't had this issue return yet, for what it's worth. It took a little over 8 weeks from the last time the hosts were patched and rebooted til this issue manifested itself the first time.
ndolson
Influencer
 
Posts: 13
Liked: 2 times
Joined: Thu Jan 08, 2015 3:56 pm
Full Name: Neal

Re: Repeated VDDK error 1 and 2

Veeam Logoby drose0 » Fri Aug 12, 2016 5:00 pm

We are having the same issues. Each day this week we've had a few fail. Each day they were on the same host, but that host has changed daily.

Support case #01878241.
drose0
Service Provider
 
Posts: 1
Liked: never
Joined: Mon Nov 21, 2011 5:14 pm
Full Name: DJ Rose

Re: Repeated VDDK error 1 and 2

Veeam Logoby EBoucq » Tue Sep 06, 2016 10:06 am

Hi,

When I had this issue one day, I got a few jobs I couldn't stop the proper way (stayed in "stopping" status). When I restarted my Veeam server, the ESXi hosting the VMs with the VDDK error 1 suddenly got disconnected from the vCenter.
It's obviously not a coincidence but I'd like to understand what could technically cause a host to stop responding and disconnect from a vcenter because a veeam server is being restarted.
Any idea ?
Thanks.
Eric
EBoucq
Expert
 
Posts: 215
Liked: 14 times
Joined: Fri Jan 21, 2011 10:10 am
Full Name: Eric Boucq

Re: Repeated VDDK error 1 and 2

Veeam Logoby peterjm » Thu Sep 08, 2016 10:53 pm

Were running Veeam B&R 9 U2 on VMware 5.5 with this issue. We started with Veeam B&R U1 when we were first experiencing it. We migrate the guests off the host and then reboot the host and migrate them back each time. Next time it happens I will try restarting the management agents instead.

I am Case # 01900486
peterjm
Lurker
 
Posts: 1
Liked: never
Joined: Thu Sep 08, 2016 10:36 pm
Full Name: Peter McGrath

Re: Repeated VDDK error 1 and 2

Veeam Logoby vmhosts_ge » Wed Sep 14, 2016 4:27 pm

Veeam Support Case 01904571

Same issue, vMotioning the affected VM's allows them to be backed up. Have tested vMotioning back after a successful backup without a host reboot or restarting the management services and it backs up successfully still.

It has happened to 2 different hosts in 2 separate Virtual Centers.
vmhosts_ge
Service Provider
 
Posts: 4
Liked: 1 time
Joined: Mon Jul 04, 2016 9:30 am
Full Name: Gary Eastwood

Re: Repeated VDDK error 1 and 2

Veeam Logoby joechay » Thu Sep 15, 2016 3:12 am

Case #01902976

I first having some issues where the VMs start to fail but eventually completed after few retries on the backup jobs. It happen after the version is updated to Veeam Backup Replicator v9.01715.

Eventually all my VMs jobs start failing with the following VDDK error 1 and 2.

I have try the following (not in sequence)

- Verify connection from vCenter to Veeam backup server
- Restart vCenter
- Rebooting ESXi host

And then I received SOAP error (network connectivity issue)
- Add and remove vNIC adaptor on my Veeam VM Server
- Set proxy selection from automatic to fix to my Veeam server.

My backup is running fine now, but looks like for many of you the issues will return back?
joechay
Novice
 
Posts: 3
Liked: 1 time
Joined: Thu Sep 15, 2016 1:09 am
Full Name: Joe Chay

Re: Repeated VDDK error 1 and 2

Veeam Logoby pidthepiper » Mon Sep 19, 2016 2:10 pm

ok So I have had this happen again to one of my other hosts, still no resolution at all.

Its like Veeam have stood still with it and so have VMware
pidthepiper
Enthusiast
 
Posts: 69
Liked: 6 times
Joined: Tue Aug 11, 2015 9:10 am
Full Name: Bilal AHmed

Re: Repeated VDDK error 1 and 2

Veeam Logoby virtual0ne » Mon Sep 19, 2016 5:23 pm

Last night 2 VMs failed initially with this error, but after the 3rd retry, one of the VMs finally worked, while the second one still failed. There were 6 VMs on this particular host being backed up, so 4 of them worked with no issue. My guess is it will go through tonight without me doing anything to the host, as this has been the case in the past. Having said this, a few months ago an entire host's backed up VMs kept failing and I had to restart the management interface to fix it.

Veeam - 8.0.0.2084
VMware - 6.0 build 3620759
virtual0ne
Lurker
 
Posts: 1
Liked: never
Joined: Mon Sep 19, 2016 4:50 pm
Full Name: Troy Egan

Re: Repeated VDDK error 1 and 2

Veeam Logoby AdamR » Mon Sep 26, 2016 10:57 am

We are having the same issues.
We rebooted one of the hosts and the problem disappeared for two days, currently we are waiting if the problem does not appear again.

Veeam - 9.0.0.1715
VMware - 5.5.0
AdamR
Novice
 
Posts: 4
Liked: never
Joined: Mon Nov 04, 2013 11:42 am
Full Name: Adam

Re: Repeated VDDK error 1 and 2

Veeam Logoby krogerss » Mon Oct 03, 2016 12:32 pm

So is there no fix in sight for this??? This issue has been out there for so long, just thought I would check in. It's getting to be a real nuisance. Thanks.
krogerss
Enthusiast
 
Posts: 66
Liked: 1 time
Joined: Mon Aug 18, 2014 6:06 pm

Re: Repeated VDDK error 1 and 2

Veeam Logoby Ice-Dog » Tue Oct 04, 2016 8:41 am

There is no fix for this yet. Every time this happens I need to restart the ESXi Management Interface of the effected host.
Ice-Dog
Influencer
 
Posts: 11
Liked: 2 times
Joined: Mon Feb 01, 2016 10:39 am
Full Name: Gunnar Olafsson

Re: Repeated VDDK error 1 and 2

Veeam Logoby spf62 » Thu Oct 06, 2016 12:48 pm

I have found that restarting the management agents work every time. But this tends to be a pain in the a$$ after a while.
We are running Veeam 9u2, ESXi 5.5 latest build

Hopefully VMWARE and VEEAM can get this worked out
spf62
Enthusiast
 
Posts: 65
Liked: never
Joined: Fri Sep 17, 2010 12:33 pm
Location: Connecticut
Full Name: Steve Fritsch

Re: Repeated VDDK error 1 and 2

Veeam Logoby tsightler » Fri Oct 07, 2016 7:20 pm 2 people like this post

Please make sure you have the following VMware patches installed if you are experiencing this issue. These patches were released in August 2016 and address one of the more common issues that cause this failure. I notice a lot of people above are using builds that are prior to these patches:

https://kb.vmware.com/selfservice/micro ... Id=2144799

Note that if restarting the management agents resolve the error you can be almost certain that the issue is not caused by Veeam and thus having tickets open with VMware is also highly recommended if you want to see the issue resolved. In many of these cases you can correlate the times of the failures to errors in the logs on the VMware hosts.

A VDDK error 1 is basically a generic error that Veeam requested a disk block from the VDDK API (also a VMware component) and it was unable to be retrieved. If your backups are running in network/NBD mode, you can be almost sure that you'll see a corresponding error on the host, one of the more common is an out of memory error. Restarting the agents simply frees memory and allows new memory to be allocated. It's certainly possible that the issues could be caused by other problems, corrupted VMDK for example, but obviously restarting management agents wouldn't fix that issue, so if restarting the management agents is the fix, then you need to figure out what's going wrong with the management agents.

Of course I'm not suggesting you can't open a ticket with Veeam, but if you do, be prepared to push the issue to completion. I looked at a number of the cases in this thread and most closed with "No response from customers", many of them did not even upload logs. These types of issues are difficult to resolve because, in many cases, once the issue is resolved, the person opening the ticket loses motivation to pursue the root cause. This is especially true if the root cause is outside of Veeam so requires opening a ticket and working with VMware as well, however, that's likely the only way to find the root cause.

There are also many other possible causes for the error, but the agent issue is probably the most common I have seen.
tsightler
Veeam Software
 
Posts: 4872
Liked: 1821 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Repeated VDDK error 1 and 2

Veeam Logoby krogerss » Sun Oct 09, 2016 2:43 pm

I opened a case (# 01867282) regarding this a couple months ago and was told that Veeam had a case open with VMware for this issue. I was just told that it was a known issue and there was nothing that could be done except trying a couple of workarounds such as rebooting the host of the affected machines. This has happened two times in the past week. So do I need to open another case or strictly seek VMware support? I can't keep babysitting multiple backup failures. My case was closed on 9-16-16 and I wasn't told of these patches you mentioned. Thanks.
krogerss
Enthusiast
 
Posts: 66
Liked: 1 time
Joined: Mon Aug 18, 2014 6:06 pm

Re: Repeated VDDK error 1 and 2

Veeam Logoby tsightler » Mon Oct 10, 2016 1:46 pm 1 person likes this post

First of all I would verify you have installed the patches I mentioned above as these correct at least one of the more common causes of the issues.

So I'll put on my customer hat and tell you what I'd do if I were still a customer. I would most likely open a case with both Veeam and VMware, and tell each that I was doing so. I'd explain to VMware that I know rebooting the host/restarting the agents will resolve the issue but I'd like to understand what state the agents are in that they require a restart, and why they are getting into this state. This will hopefully help them not just try to send me back to Veeam support, but recognize that there really is a problem with the agent. Regardless, I wouldn't expect Veeam support to just tell me that it's a VMware problem and close the ticket, and I certainly wouldn't accept that answer. I'd want a root cause analysis to determine what specifically was causing the error in my environment, even if it happens to be caused by VMware. Perhaps this will require a joing call with both Veeam and VMware support, but this should certainly be possible.

My guess is the logs of the VMware agent have all of the information that are really needed to determine the cause of this error, but I could be wrong. I'm currently working with a couple of customers that are experiencing this issue on a semi-regular basis, so hopefully if we continue to investigate we'll be able to get to the bottom if it.
tsightler
Veeam Software
 
Posts: 4872
Liked: 1821 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: Bing [Bot], Yahoo [Bot] and 1 guest