Repeated VDDK error 1 and 2

VMware specific discussions

Repeated VDDK error 1 and 2

Veeam Logoby Ice-Dog » Mon Feb 01, 2016 11:17 am 1 person likes this post

Hello,

Support ticket Case # 01676511

My environment consists of 230 VMs running on 14x ESXi 5.5 build 3116895 on Cisco UCS blades. I am running this across two sites with two 10Gb dark fiber links with no latency.

I have Veeam B&R v8 u3 (8.0.0.2084) + 2 Backup proxies.

Few VMs in various jobs seem to recently started failing with the same error.

Error: VDDK error: 1 (Unknown error). Value: 0x0000000000000001 Failed to read from a virtual disk Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: VDDK error: 1 (Unknown error). Value: 0x0000000000000001 Failed to read from a virtual disk Unable to retrieve next block transmission command. Number of already processed blocks: [7]. Failed to download disk.

One if these VMs is my Exchange server so I raised the case regarding that server.

1. Response was to storage migrate all the VM disks to a different date store. - done same error

2. I uploaded Support logs and Veeam found found this error,
"[NFC ERROR] NfcFssrvrProcessErrorMsg: received diskLib error 786441 from server: NFC_DISKLIB_ERROR
This error is generated by you ESXi host, please refer to following KB to try to sort out this issue: http://kb.vmware.com/selfservice/micros ... Id=1004424"


My ESX hosts are set to 256MB by default in ESXi 5.5 which is also the maximum. although, as per the kb it can grow to 640MB as required. That allows a maximum of 60 TB to be opened per host.

3. Asked to add the ESXi hosts as a standalone host to Veeam and try backing up the VM. - Backup successful using this workaround to skip vCenter

4. Veeam suggest this is an unusual CBT issue and ask me to try restting the CBT info on the VM. - I do that but the backup via vCener is still unsuccessful but the error changes to

"Processing Exchange01 Error: VDDK error: 2 (Memory allocation failed. Out of memory.). Value: 0x0000000000000002 Failed to read from a virtual disk Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: VDDK error: 2 (Memory allocation failed. Out of memory.). Value: 0x0000000000000002 Failed to read from a virtual disk Unable to retrieve next block transmission command. Number of already processed blocks: [0]. Failed to download disk. "

Last night I start playing around with other VMs that have had the same issue. After resetting the CBT on them they change the error message to the one above. But I discoveed that all the VMs that are failing are in site B. If I vmotion them to hosts in site A and retry the job Veeam can open up the disks and complete the job. I have DRS site affinity rules so VMs are not running split site (e.g. their disks in site A but they running on host memory in site B) Even if DRS migrates the VM back to site B the Veeam job carries on, they only need to be in site a as Veeam starts processing the VM and accessing the disks

Any toughs would be appreciated

Thanks
Ice-Dog
Influencer
 
Posts: 11
Liked: 2 times
Joined: Mon Feb 01, 2016 10:39 am
Full Name: Gunnar Olafsson

Re: Repeted VDDK error 1 and 2

Veeam Logoby isaako » Tue Feb 02, 2016 8:38 am

I've the same error during backup window.
I'm backing up about 550 VMs, all my failed backups are from the same ESXi hosts.
I'm going to investigate this issue.
isaako
Influencer
 
Posts: 19
Liked: never
Joined: Wed Sep 15, 2010 11:31 am
Full Name: Isaac González

Re: Repeted VDDK error 1 and 2

Veeam Logoby isaako » Wed Feb 03, 2016 9:01 am

Hi,

Yesterday my backups failed in a concrete ESxi host. Today some of them has failed in another ESXi host (but all the jobs in the same).
I'm trying to play whith ESXi buffers to solve the issue:

esxcfg-advcfg -g /BufferCache/MaxCapacity
esxcfg-advcfg -g /BufferCache/FlushInterval
esxcfg-advcfg -s 32768 /BufferCache/MaxCapacity
esxcfg-advcfg -s 20000 /BufferCache/FlushInterval
isaako
Influencer
 
Posts: 19
Liked: never
Joined: Wed Sep 15, 2010 11:31 am
Full Name: Isaac González

Re: Repeted VDDK error 1 and 2

Veeam Logoby Ice-Dog » Wed Feb 03, 2016 9:18 pm

Hi all,

I narrowed it down to the same host. Should have noticed that earlier but hey ...

I restarted that host today and moved some of the VMs that had been failing back onto it and I am the running backups at the moment.

The Exchange backup of the passive DAG works fine since yesterday via vCenter after I moved the VM of to a different host in the same site.

Would still like to know what caused.... I cannot see any errors in the hostd, vmkwarning or vm logs.
Ice-Dog
Influencer
 
Posts: 11
Liked: 2 times
Joined: Mon Feb 01, 2016 10:39 am
Full Name: Gunnar Olafsson

Re: Repeted VDDK error 1 and 2

Veeam Logoby isaako » Thu Feb 04, 2016 8:30 am

Hi,

I've just changed the memory buffers in the above post and just worked.
I've opened a case with vmware, it seems a NBD transport limitation or bug.
I fear that if the number of VM to backups increase we will have the problem again.
Currently I'm using Veeam network mode in our proxies and I'm going to plan a configuration to SAN mode.

Isaac
isaako
Influencer
 
Posts: 19
Liked: never
Joined: Wed Sep 15, 2010 11:31 am
Full Name: Isaac González

[MERGED] VDDK Error

Veeam Logoby Stokkolm » Thu Feb 11, 2016 11:35 pm 1 person likes this post

Today I encountered an error that I have never seen before from Veeam. I found the following KB article, which seems to correspond: https://www.veeam.com/kb1901 That KB was only created 16 days ago, so I'm curious if this is something that was recently discovered and if it's specifically related to vSphere 6.0 U1a or not. The error occurred when Veeam attempted to read the first VMDK, it would begin to process it and then fail:

Code: Select all
2/11/2016 12:18:41 AM :: Processing anc-sccm-dist01 Error: VDDK error: 1 (Unknown error). Value: 0x0000000000000001
Failed to read from a virtual disk
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: VDDK error: 1 (Unknown error). Value: 0x0000000000000001
Failed to read from a virtual disk
Unable to retrieve next block transmission command. Number of already processed blocks: [219].
Failed to download disk.

The KB article says to attempt a svMotion or clone the VM and try again. Neither of those suggestions worked, however doing a normal vMotion to another host did (we were able to determine that all of the VM's, about 20 of them, that had this error were all on one host). Based on that, we placed the host into maintenance mode and rebooted it. Once it came back up Veeam Backups were working again. I still don't have a clue as to the root cause, there were no major changes to any of our hosts recently and based on the severity of the case we started with VMware they aren't going to respond until tomorrow. The VM's involved were on varying datastores and across multiple arrays and there were other VM's on these datastores that backed up normally. Also, the VM's didn't show any signs of abnormal activity other than the failed backups. Anyway, I wanted to make this forum post mainly because I think the KB is lacking a little bit and hopefully if someone else runs into this error they can find this post for additional information based off of my experience today.
Stokkolm
Influencer
 
Posts: 11
Liked: 5 times
Joined: Thu Feb 11, 2016 11:11 pm
Full Name: John Zetterman

Re: Repeted VDDK error 1 and 2

Veeam Logoby foggy » Fri Feb 12, 2016 9:40 am

Thanks, John, much appreciated. I've passed your feedback to the responsible team for possible KB update. You can also see that you're not alone with this problem.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Repeted VDDK error 1 and 2

Veeam Logoby picaroon » Mon Feb 15, 2016 10:44 pm

Past two months I also experiencing this issue. Multiple cases with VMware and Veeam had been created, but today still no root cause and solution. Only known workaround is migrating the VMs and reboot the affected host. At this moment I've got another two VMs which can't be backupped due to this issue.
picaroon
Influencer
 
Posts: 13
Liked: 5 times
Joined: Fri Feb 12, 2016 1:44 pm
Full Name: Jeltsen Haagsma

Re: Repeted VDDK error 1 and 2

Veeam Logoby Ice-Dog » Wed Feb 24, 2016 9:59 am

Hi all,

Since first reporting this issue I have had two other hosts with the same issue. Using vmotion to move the failed VMs to a different host and retry the failed Backup job allows the job to complete succssfully. After rebooting the hosts Veeam is able to backup running VMs of it.

Last week a Senior Team lead from Veeam support contacted me to enqiure if I had done any further investigations on the issue and suggested that if this would happen again to raise a new ticket with Veeam and they would leverage VMware support on the case.

Last night VMs on the fourth host are failing in the Backup jobs. Same issue as before. Now I need to plan another host reboot.

New ticket #01708122
Support logs uploaded with the case

Regards,
Gunnar
Ice-Dog
Influencer
 
Posts: 11
Liked: 2 times
Joined: Mon Feb 01, 2016 10:39 am
Full Name: Gunnar Olafsson

Re: Repeted VDDK error 1 and 2

Veeam Logoby rlove » Tue Mar 01, 2016 8:37 pm

Ice-Dog,

Did you ever get a fix for this from Veeam or VMware?
Is there a VMware ticket you can reference?
rlove
Lurker
 
Posts: 2
Liked: never
Joined: Tue Mar 01, 2016 8:34 pm

Re: Repeted VDDK error 1 and 2

Veeam Logoby picaroon » Wed Mar 02, 2016 10:48 am 1 person likes this post

Last Saturday the same issue happened again. I changed the buffercache according to one of the posts in this thread, but no luck. I've ended up vMotioning the machine to another host in the cluster and Veeam is able to backup the VM.
We use 12 hosts in a Cisco UCS blade, running ESXi 5.5.0 build 3343343 and vCenter 5.5.0 build 30000241.

A while ago we created case #01303819 with Veeam support, but they advised us to create a case with VMware support since the backup is working correctly when vMotioning the VM.

Till now still no answer from VMware...
picaroon
Influencer
 
Posts: 13
Liked: 5 times
Joined: Fri Feb 12, 2016 1:44 pm
Full Name: Jeltsen Haagsma

Re: Repeted VDDK error 1 and 2

Veeam Logoby rlove » Wed Mar 02, 2016 4:19 pm

I feel your pain of being put in the middle of two vendors pointing fingers.
I tried a vMotion and and a SvMotion on the troubled VM's and it did not help.
I found this on Reddit - https://www.reddit.com/r/Veeam/comments ... s_failing/

It states that you have to actually reboot the node the VM was on. I rebooted the original node the VM used to be on and did a retry and it worked.
Not sure what the node was holding open/locked but it needed a complete reboot to get things moving again.
I've updated my Veeam case (01713005) and the VMware case with that info.
rlove
Lurker
 
Posts: 2
Liked: never
Joined: Tue Mar 01, 2016 8:34 pm

Re: Repeted VDDK error 1 and 2

Veeam Logoby picaroon » Thu Mar 03, 2016 8:45 am

Rebooting the host is just a temporarily workaround, I'm dealing with this issue every week. Sometimes there are nights I need to reboot 3 hosts because of this.
Anyone experienced the same issue when using hotadd instead of the NBD method? In our environment only NBD seems to be affected, but I only have had the chance to test it short.

So at this moment I'm considering hotadd method, but I'm a bit shivery to use it...
picaroon
Influencer
 
Posts: 13
Liked: 5 times
Joined: Fri Feb 12, 2016 1:44 pm
Full Name: Jeltsen Haagsma

Re: Repeted VDDK error 1 and 2

Veeam Logoby isaako » Wed Mar 16, 2016 9:38 am

Hi,
I've opened a case with VMWare and they don't know how to solve the issue.
Do you have some news about this issue?

After applying the options in my previous note the issue happens some time, only 1 VM per night and some days.

Can you try to apply this setting in the failing host an try again?

esxcfg-advcfg -g /BufferCache/MaxCapacity
esxcfg-advcfg -g /BufferCache/FlushInterval
esxcfg-advcfg -s 32768 /BufferCache/MaxCapacity
esxcfg-advcfg -s 20000 /BufferCache/FlushInterval


Isaac
isaako
Influencer
 
Posts: 19
Liked: never
Joined: Wed Sep 15, 2010 11:31 am
Full Name: Isaac González

Re: Repeted VDDK error 1 and 2

Veeam Logoby LeoKurz » Wed Mar 30, 2016 9:24 am

Any news on this? I'm having the same issue. Running von vSphere 6 and B&R 9. Could be coincidence, but occured after I updated vCenter Appliance to 6.0U2. ESX are still on plain 6.0. Had none of this before...

__Leo
LeoKurz
Veeam ProPartner
 
Posts: 22
Liked: 6 times
Joined: Wed Mar 16, 2011 8:36 am
Full Name: Leonhard Kurz

Next

Return to VMware vSphere



Who is online

Users browsing this forum: UT2015 and 29 guests