Repeated VDDK error 1 and 2

VMware specific discussions

Re: Repeated VDDK error 1 and 2

Veeam Logoby pidthepiper » Fri Jul 08, 2016 11:24 am

mgratla wrote:VMware came back and told me that there was an issue because of a vpxa.cfg file on the ESXi hosts, but they don't have an idea of the cause. They did say that 'too many delta disks or cloned disks depending on other disks which adversely affects the vpxa agent' , which is interesting because there was a single snapshot on each of these machines which is the one Veeam was trying to create.

They wanted me to edit the vpxa file on each of the hosts and change 128 to 256 and restart vpxa then enable sfcdb temporarily for a core dump.

I'm not sure I want to proceed with this change given that their reason for doing it doesn't seem to be correct

I also have been told to change the VddkPreReadBufferSize on my case with the wording that
"it doesn't appear that the root cause of the issue has been found - however, it does appear to be related to ESXi hosts that are added via vCenter: the error itself seems to be produced due to an issue with the datastore or connection between the host and the datastore"

I'm somewhat surprised at that statement, since we had no issue on v7 or v8 whatsoever for nearly two years with no changes to our environment.

I'll be adjusting the VddkPrereadBufferSize value today but if Picaroons response is correct, I doubt we will see much difference


Thats interesting, But he errors have started appearing out of no where
pidthepiper
Enthusiast
 
Posts: 65
Liked: 5 times
Joined: Tue Aug 11, 2015 9:10 am
Full Name: Bilal AHmed

Re: Repeated VDDK error 1 and 2

Veeam Logoby ndolson » Wed Jul 13, 2016 2:18 pm

We are getting the VDDK error 2 message now. Running vSphere 6.0 U2 and Veeam B&R v9 U1. I can't say for sure since it's been a while but it seems like it started happening after the installation of U2, which is the last time those hosts were patched and rebooted.

Case ID 01850397
ndolson
Influencer
 
Posts: 13
Liked: 2 times
Joined: Thu Jan 08, 2015 3:56 pm
Full Name: Neal

Re: Repeated VDDK error 1 and 2

Veeam Logoby pidthepiper » Thu Jul 14, 2016 4:55 pm

We are all still waiting for an update from veeam really
pidthepiper
Enthusiast
 
Posts: 65
Liked: 5 times
Joined: Tue Aug 11, 2015 9:10 am
Full Name: Bilal AHmed

Re: Repeated VDDK error 1 and 2

Veeam Logoby ndolson » Mon Jul 18, 2016 1:36 pm

I was basically told that because restarting hostd and vpxa temporarily resolves the issue, it indicates a problem on the VMware side and I need to pursue the issue with them. :|
ndolson
Influencer
 
Posts: 13
Liked: 2 times
Joined: Thu Jan 08, 2015 3:56 pm
Full Name: Neal

Re: Repeated VDDK error 1 and 2

Veeam Logoby mwirz » Mon Jul 18, 2016 2:53 pm

i have noticed that this error only appears with my virtual machines running on nfs storage systems (netapp, nfs), virtual machines running on iscsi storage are fine all the time. what about your storage systems? nfs as well?
mwirz
Service Provider
 
Posts: 12
Liked: never
Joined: Tue Nov 25, 2014 9:55 pm
Full Name: Mike Wirz

Re: Repeated VDDK error 1 and 2

Veeam Logoby ndolson » Mon Jul 18, 2016 2:56 pm

Yep, the cluster in which its happening has its storage presented to it via NFS. Another small cluster with SAS connectivity is not experiencing it. Same ESXi version under same vCenter.
ndolson
Influencer
 
Posts: 13
Liked: 2 times
Joined: Thu Jan 08, 2015 3:56 pm
Full Name: Neal

Re: Repeated VDDK error 1 and 2

Veeam Logoby mwirz » Mon Jul 18, 2016 3:27 pm

i'll take the time to open a case as well. maybe i can bring some more logs to identify the root cause...
mwirz
Service Provider
 
Posts: 12
Liked: never
Joined: Tue Nov 25, 2014 9:55 pm
Full Name: Mike Wirz

Re: Repeated VDDK error 1 and 2

Veeam Logoby rpop » Tue Jul 19, 2016 8:36 am

Hi all,

Same problem for me, but only on job replication.

31 VMs failed on 85 in the job

Code: Select all
16.07.2016 06:12:06 :: Error: VDDK error: 2 (Memory allocation failed. Out of memory.). Value: 0x0000000000000002
Failed to write to a virtual disk
Failed to download disk.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.


After reboot (or restart mgmt service) concerned hosts (target hosts), it's ok, but for how many time ?
Code: Select all
vCenter : 6.0.0.20000 (Appliance postgressql)
ESXi :      6.0.0 , 3825889 + (patch last bug VMXNET3 !!!)
Veeam :  9u1 , 1x B&R server as a VM , 2x physical proxy/repo(landing zone) servers, one for test/DR site , the other for production site), Network 10Gbps everywhere, even between sites.

Best regards
Best regards
rpop
Influencer
 
Posts: 17
Liked: never
Joined: Fri Feb 11, 2011 2:39 pm
Full Name: rpveeam

Re: Repeated VDDK error 1 and 2

Veeam Logoby Vitaliy S. » Tue Jul 19, 2016 10:24 am

rpop, please contact our support team directly when posting about a technical issue as it is advised by our forum rules. Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Repeated VDDK error 1 and 2

Veeam Logoby WRStone » Mon Jul 25, 2016 4:43 pm

We've been getting error 2 on random servers. We can typically fix it by restarting management interfaces, but then the problem will pop up and we'll have to do it again elsewhere.

At least for us, Veeam has washed their hands of the issue with the following:

I am glad that the restart was able to help get past the error. If this VMware error continues to persist then I would highly advise leveraging VMware support in on the issue to further investigate the root cause. For the VDDK 2 error, specifically, it would point to a memory allocation issue with the host of the VM being processed:

https://www.vmware.com/support/develope ... rrors.html


I'm going to open a support case with VMware and tell them that Veeam sent me. As this has been an ongoing issue for over six months, I'll post the root cause (assuming that VMware and Veeam can agree on one).
WRStone
Novice
 
Posts: 4
Liked: never
Joined: Fri Jul 22, 2016 4:24 pm
Full Name: William Stone III

Re: Repeated VDDK error 1 and 2

Veeam Logoby duukie » Mon Jul 25, 2016 9:24 pm 1 person likes this post

Same issue here
Veeam Support Case # 01862154

I was able to conference in both Veeam and VMware Support. They basically blame each other.

I've tried the following without success:
* vMotion to another host and datastore
* Rebooted host
* services.sh restart
* Cloned VM
* Created new job and sent to different repository.

VMware is reviewing Veems logs to compare date stamps.

Will update this thread....
duukie
Lurker
 
Posts: 1
Liked: 1 time
Joined: Mon Jul 25, 2016 9:09 pm
Full Name: Du Lam

Re: Repeated VDDK error 1 and 2

Veeam Logoby WRStone » Tue Jul 26, 2016 4:08 pm

duukie wrote:I was able to conference in both Veeam and VMware Support. They basically blame each other.


Thank you muchly. I've not yet gotten them together, but I assume that blaming each other is what would happen. I've seen it repeatedly.

We have a Veeam engineer in the office some of this week. You can rest assured that we'll be asking about this and will update the thread.
WRStone
Novice
 
Posts: 4
Liked: never
Joined: Fri Jul 22, 2016 4:24 pm
Full Name: William Stone III

Re: Repeated VDDK error 1 and 2

Veeam Logoby WRStone » Tue Jul 26, 2016 8:36 pm

The Veeam engineer was in today. While he didn't recognize the issue, he recognized that it's now a bouncing game between VMware and Veeam.

One thing he could reliably do is escalate my support ticket. He and the guy he was with were very clear that escalating is something they do a lot and are good at.

I'm to provide him with this forum thread and my support ticket so as to escalate. I'll post here as I learn more.
WRStone
Novice
 
Posts: 4
Liked: never
Joined: Fri Jul 22, 2016 4:24 pm
Full Name: William Stone III

Re: Repeated VDDK error 1 and 2

Veeam Logoby WRStone » Thu Jul 28, 2016 8:09 pm

Unfortunately, escalating our ticket didn't really help. Here's what we got back:

At the moment we have several cases open with the same issue, so here's the update:

As far as I can tell, we have a case open with VMware SDK support, and they are working on there end to identify/resolve the issue. Beyond that I don't see any update.

I understand that the issue is ongoing, and we seem to have possibly narrowed it down to being an issue with the vpxa service on the host. What I'd suggest trying, if you are using the temporary 'fix' of rebooting the ESXi host, is the following:

1) Instead of rebooting the host next time you see the VDDK error, try just restarting the vpxa service. If backup still fails, you might try restarting the vpxa service on the host as well as the vpxd service on the VC

2) Another test (if this hasn't already been attempted) would be to add the ESXi host to Veeam as a standalone - see https://www.veeam.com/kb1777 - then try backing up via the standalone.


Breaking it down, what that means is:

No progress has been made since the vxpa issue was reported in this forum over four weeks ago.

I have started to make a stink that I won't post. I'll continue to update.
WRStone
Novice
 
Posts: 4
Liked: never
Joined: Fri Jul 22, 2016 4:24 pm
Full Name: William Stone III

Re: Repeated VDDK error 1 and 2

Veeam Logoby pidthepiper » Mon Aug 01, 2016 2:56 pm

Yep basically the same thing SDK case open, VMware are like its a SDK issue for the SDK team and Veeam, have a case open with them, but nothing is happening.

We dont use NFS for any VM storage, we use FC.

I have mitigated the errors by simply using HotAdd specifically, but it means using nbd doesnt work well and thats a feature we are paying for that we cant use, which doesn't sit well with me
pidthepiper
Enthusiast
 
Posts: 65
Liked: 5 times
Joined: Tue Aug 11, 2015 9:10 am
Full Name: Bilal AHmed

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 13 guests