Discussions specific to the VMware vSphere hypervisor
Post Reply
Ice-Dog
Influencer
Posts: 11
Liked: 2 times
Joined: Feb 01, 2016 10:39 am
Full Name: Gunnar Olafsson
Contact:

Repeated VDDK error 1 and 2

Post by Ice-Dog » Feb 01, 2016 11:17 am 1 person likes this post

Hello,

Support ticket Case # 01676511

My environment consists of 230 VMs running on 14x ESXi 5.5 build 3116895 on Cisco UCS blades. I am running this across two sites with two 10Gb dark fiber links with no latency.

I have Veeam B&R v8 u3 (8.0.0.2084) + 2 Backup proxies.

Few VMs in various jobs seem to recently started failing with the same error.

Error: VDDK error: 1 (Unknown error). Value: 0x0000000000000001 Failed to read from a virtual disk Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: VDDK error: 1 (Unknown error). Value: 0x0000000000000001 Failed to read from a virtual disk Unable to retrieve next block transmission command. Number of already processed blocks: [7]. Failed to download disk.

One if these VMs is my Exchange server so I raised the case regarding that server.

1. Response was to storage migrate all the VM disks to a different date store. - done same error

2. I uploaded Support logs and Veeam found found this error,
"[NFC ERROR] NfcFssrvrProcessErrorMsg: received diskLib error 786441 from server: NFC_DISKLIB_ERROR
This error is generated by you ESXi host, please refer to following KB to try to sort out this issue: http://kb.vmware.com/selfservice/micros ... Id=1004424"


My ESX hosts are set to 256MB by default in ESXi 5.5 which is also the maximum. although, as per the kb it can grow to 640MB as required. That allows a maximum of 60 TB to be opened per host.

3. Asked to add the ESXi hosts as a standalone host to Veeam and try backing up the VM. - Backup successful using this workaround to skip vCenter

4. Veeam suggest this is an unusual CBT issue and ask me to try restting the CBT info on the VM. - I do that but the backup via vCener is still unsuccessful but the error changes to

"Processing Exchange01 Error: VDDK error: 2 (Memory allocation failed. Out of memory.). Value: 0x0000000000000002 Failed to read from a virtual disk Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: VDDK error: 2 (Memory allocation failed. Out of memory.). Value: 0x0000000000000002 Failed to read from a virtual disk Unable to retrieve next block transmission command. Number of already processed blocks: [0]. Failed to download disk. "

Last night I start playing around with other VMs that have had the same issue. After resetting the CBT on them they change the error message to the one above. But I discoveed that all the VMs that are failing are in site B. If I vmotion them to hosts in site A and retry the job Veeam can open up the disks and complete the job. I have DRS site affinity rules so VMs are not running split site (e.g. their disks in site A but they running on host memory in site B) Even if DRS migrates the VM back to site B the Veeam job carries on, they only need to be in site a as Veeam starts processing the VM and accessing the disks

Any toughs would be appreciated

Thanks

isaako
Service Provider
Posts: 25
Liked: never
Joined: Sep 15, 2010 11:31 am
Full Name: Isaac González
Contact:

Re: Repeted VDDK error 1 and 2

Post by isaako » Feb 02, 2016 8:38 am

I've the same error during backup window.
I'm backing up about 550 VMs, all my failed backups are from the same ESXi hosts.
I'm going to investigate this issue.

isaako
Service Provider
Posts: 25
Liked: never
Joined: Sep 15, 2010 11:31 am
Full Name: Isaac González
Contact:

Re: Repeted VDDK error 1 and 2

Post by isaako » Feb 03, 2016 9:01 am

Hi,

Yesterday my backups failed in a concrete ESxi host. Today some of them has failed in another ESXi host (but all the jobs in the same).
I'm trying to play whith ESXi buffers to solve the issue:

esxcfg-advcfg -g /BufferCache/MaxCapacity
esxcfg-advcfg -g /BufferCache/FlushInterval
esxcfg-advcfg -s 32768 /BufferCache/MaxCapacity
esxcfg-advcfg -s 20000 /BufferCache/FlushInterval

Ice-Dog
Influencer
Posts: 11
Liked: 2 times
Joined: Feb 01, 2016 10:39 am
Full Name: Gunnar Olafsson
Contact:

Re: Repeted VDDK error 1 and 2

Post by Ice-Dog » Feb 03, 2016 9:18 pm

Hi all,

I narrowed it down to the same host. Should have noticed that earlier but hey ...

I restarted that host today and moved some of the VMs that had been failing back onto it and I am the running backups at the moment.

The Exchange backup of the passive DAG works fine since yesterday via vCenter after I moved the VM of to a different host in the same site.

Would still like to know what caused.... I cannot see any errors in the hostd, vmkwarning or vm logs.

isaako
Service Provider
Posts: 25
Liked: never
Joined: Sep 15, 2010 11:31 am
Full Name: Isaac González
Contact:

Re: Repeted VDDK error 1 and 2

Post by isaako » Feb 04, 2016 8:30 am

Hi,

I've just changed the memory buffers in the above post and just worked.
I've opened a case with vmware, it seems a NBD transport limitation or bug.
I fear that if the number of VM to backups increase we will have the problem again.
Currently I'm using Veeam network mode in our proxies and I'm going to plan a configuration to SAN mode.

Isaac

Stokkolm
Influencer
Posts: 15
Liked: 5 times
Joined: Feb 11, 2016 11:11 pm
Full Name: John Zetterman
Contact:

[MERGED] VDDK Error

Post by Stokkolm » Feb 11, 2016 11:35 pm 1 person likes this post

Today I encountered an error that I have never seen before from Veeam. I found the following KB article, which seems to correspond: https://www.veeam.com/kb1901 That KB was only created 16 days ago, so I'm curious if this is something that was recently discovered and if it's specifically related to vSphere 6.0 U1a or not. The error occurred when Veeam attempted to read the first VMDK, it would begin to process it and then fail:

Code: Select all

2/11/2016 12:18:41 AM :: Processing anc-sccm-dist01 Error: VDDK error: 1 (Unknown error). Value: 0x0000000000000001
Failed to read from a virtual disk
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: VDDK error: 1 (Unknown error). Value: 0x0000000000000001
Failed to read from a virtual disk
Unable to retrieve next block transmission command. Number of already processed blocks: [219].
Failed to download disk.
The KB article says to attempt a svMotion or clone the VM and try again. Neither of those suggestions worked, however doing a normal vMotion to another host did (we were able to determine that all of the VM's, about 20 of them, that had this error were all on one host). Based on that, we placed the host into maintenance mode and rebooted it. Once it came back up Veeam Backups were working again. I still don't have a clue as to the root cause, there were no major changes to any of our hosts recently and based on the severity of the case we started with VMware they aren't going to respond until tomorrow. The VM's involved were on varying datastores and across multiple arrays and there were other VM's on these datastores that backed up normally. Also, the VM's didn't show any signs of abnormal activity other than the failed backups. Anyway, I wanted to make this forum post mainly because I think the KB is lacking a little bit and hopefully if someone else runs into this error they can find this post for additional information based off of my experience today.

foggy
Veeam Software
Posts: 16932
Liked: 1377 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Repeted VDDK error 1 and 2

Post by foggy » Feb 12, 2016 9:40 am

Thanks, John, much appreciated. I've passed your feedback to the responsible team for possible KB update. You can also see that you're not alone with this problem.

picaroon
Influencer
Posts: 13
Liked: 5 times
Joined: Feb 12, 2016 1:44 pm
Full Name: Jeltsen Haagsma
Contact:

Re: Repeted VDDK error 1 and 2

Post by picaroon » Feb 15, 2016 10:44 pm

Past two months I also experiencing this issue. Multiple cases with VMware and Veeam had been created, but today still no root cause and solution. Only known workaround is migrating the VMs and reboot the affected host. At this moment I've got another two VMs which can't be backupped due to this issue.

Ice-Dog
Influencer
Posts: 11
Liked: 2 times
Joined: Feb 01, 2016 10:39 am
Full Name: Gunnar Olafsson
Contact:

Re: Repeted VDDK error 1 and 2

Post by Ice-Dog » Feb 24, 2016 9:59 am

Hi all,

Since first reporting this issue I have had two other hosts with the same issue. Using vmotion to move the failed VMs to a different host and retry the failed Backup job allows the job to complete succssfully. After rebooting the hosts Veeam is able to backup running VMs of it.

Last week a Senior Team lead from Veeam support contacted me to enqiure if I had done any further investigations on the issue and suggested that if this would happen again to raise a new ticket with Veeam and they would leverage VMware support on the case.

Last night VMs on the fourth host are failing in the Backup jobs. Same issue as before. Now I need to plan another host reboot.

New ticket #01708122
Support logs uploaded with the case

Regards,
Gunnar

rlove
Service Provider
Posts: 2
Liked: never
Joined: Mar 01, 2016 8:34 pm
Contact:

Re: Repeted VDDK error 1 and 2

Post by rlove » Mar 01, 2016 8:37 pm

Ice-Dog,

Did you ever get a fix for this from Veeam or VMware?
Is there a VMware ticket you can reference?

picaroon
Influencer
Posts: 13
Liked: 5 times
Joined: Feb 12, 2016 1:44 pm
Full Name: Jeltsen Haagsma
Contact:

Re: Repeted VDDK error 1 and 2

Post by picaroon » Mar 02, 2016 10:48 am 1 person likes this post

Last Saturday the same issue happened again. I changed the buffercache according to one of the posts in this thread, but no luck. I've ended up vMotioning the machine to another host in the cluster and Veeam is able to backup the VM.
We use 12 hosts in a Cisco UCS blade, running ESXi 5.5.0 build 3343343 and vCenter 5.5.0 build 30000241.

A while ago we created case #01303819 with Veeam support, but they advised us to create a case with VMware support since the backup is working correctly when vMotioning the VM.

Till now still no answer from VMware...

rlove
Service Provider
Posts: 2
Liked: never
Joined: Mar 01, 2016 8:34 pm
Contact:

Re: Repeted VDDK error 1 and 2

Post by rlove » Mar 02, 2016 4:19 pm

I feel your pain of being put in the middle of two vendors pointing fingers.
I tried a vMotion and and a SvMotion on the troubled VM's and it did not help.
I found this on Reddit - https://www.reddit.com/r/Veeam/comments ... s_failing/

It states that you have to actually reboot the node the VM was on. I rebooted the original node the VM used to be on and did a retry and it worked.
Not sure what the node was holding open/locked but it needed a complete reboot to get things moving again.
I've updated my Veeam case (01713005) and the VMware case with that info.

picaroon
Influencer
Posts: 13
Liked: 5 times
Joined: Feb 12, 2016 1:44 pm
Full Name: Jeltsen Haagsma
Contact:

Re: Repeted VDDK error 1 and 2

Post by picaroon » Mar 03, 2016 8:45 am

Rebooting the host is just a temporarily workaround, I'm dealing with this issue every week. Sometimes there are nights I need to reboot 3 hosts because of this.
Anyone experienced the same issue when using hotadd instead of the NBD method? In our environment only NBD seems to be affected, but I only have had the chance to test it short.

So at this moment I'm considering hotadd method, but I'm a bit shivery to use it...

isaako
Service Provider
Posts: 25
Liked: never
Joined: Sep 15, 2010 11:31 am
Full Name: Isaac González
Contact:

Re: Repeted VDDK error 1 and 2

Post by isaako » Mar 16, 2016 9:38 am

Hi,
I've opened a case with VMWare and they don't know how to solve the issue.
Do you have some news about this issue?

After applying the options in my previous note the issue happens some time, only 1 VM per night and some days.

Can you try to apply this setting in the failing host an try again?

esxcfg-advcfg -g /BufferCache/MaxCapacity
esxcfg-advcfg -g /BufferCache/FlushInterval
esxcfg-advcfg -s 32768 /BufferCache/MaxCapacity
esxcfg-advcfg -s 20000 /BufferCache/FlushInterval


Isaac

LeoKurz
Veeam ProPartner
Posts: 25
Liked: 6 times
Joined: Mar 16, 2011 8:36 am
Full Name: Leonhard Kurz
Contact:

Re: Repeted VDDK error 1 and 2

Post by LeoKurz » Mar 30, 2016 9:24 am

Any news on this? I'm having the same issue. Running von vSphere 6 and B&R 9. Could be coincidence, but occured after I updated vCenter Appliance to 6.0U2. ESX are still on plain 6.0. Had none of this before...

__Leo

Post Reply

Who is online

Users browsing this forum: No registered users and 31 guests