Repeated VDDK error 1 and 2

[Ice-Dog]

Hello,

Support ticket Case # 01676511

My environment consists of 230 VMs running on 14x ESXi 5.5 build 3116895 on Cisco UCS blades. I am running this across two sites with two 10Gb dark fiber links with no latency.

I have Veeam B&R v8 u3 (8.0.0.2084) + 2 Backup proxies.

Few VMs in various jobs seem to recently started failing with the same error.

Error: VDDK error: 1 (Unknown error). Value: 0x0000000000000001 Failed to read from a virtual disk Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: VDDK error: 1 (Unknown error). Value: 0x0000000000000001 Failed to read from a virtual disk Unable to retrieve next block transmission command. Number of already processed blocks: [7]. Failed to download disk.

One if these VMs is my Exchange server so I raised the case regarding that server.

1. Response was to storage migrate all the VM disks to a different date store. - done same error

2. I uploaded Support logs and Veeam found found this error,
"[NFC ERROR] NfcFssrvrProcessErrorMsg: received diskLib error 786441 from server: NFC_DISKLIB_ERROR
This error is generated by you ESXi host, please refer to following KB to try to sort out this issue: http://kb.vmware.com/selfservice/micros ... Id=1004424"

My ESX hosts are set to 256MB by default in ESXi 5.5 which is also the maximum. although, as per the kb it can grow to 640MB as required. That allows a maximum of 60 TB to be opened per host.

3. Asked to add the ESXi hosts as a standalone host to Veeam and try backing up the VM. - Backup successful using this workaround to skip vCenter

4. Veeam suggest this is an unusual CBT issue and ask me to try restting the CBT info on the VM. - I do that but the backup via vCener is still unsuccessful but the error changes to

"Processing Exchange01 Error: VDDK error: 2 (Memory allocation failed. Out of memory.). Value: 0x0000000000000002 Failed to read from a virtual disk Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: VDDK error: 2 (Memory allocation failed. Out of memory.). Value: 0x0000000000000002 Failed to read from a virtual disk Unable to retrieve next block transmission command. Number of already processed blocks: [0]. Failed to download disk. "

Last night I start playing around with other VMs that have had the same issue. After resetting the CBT on them they change the error message to the one above. But I discoveed that all the VMs that are failing are in site B. If I vmotion them to hosts in site A and retry the job Veeam can open up the disks and complete the job. I have DRS site affinity rules so VMs are not running split site (e.g. their disks in site A but they running on host memory in site B) Even if DRS migrates the VM back to site B the Veeam job carries on, they only need to be in site a as Veeam starts processing the VM and accessing the disks

Any toughs would be appreciated

Thanks

[isaako]

I've the same error during backup window.
I'm backing up about 550 VMs, all my failed backups are from the same ESXi hosts.
I'm going to investigate this issue.

[isaako]

Hi,

Yesterday my backups failed in a concrete ESxi host. Today some of them has failed in another ESXi host (but all the jobs in the same).
I'm trying to play whith ESXi buffers to solve the issue:

esxcfg-advcfg -g /BufferCache/MaxCapacity
esxcfg-advcfg -g /BufferCache/FlushInterval
esxcfg-advcfg -s 32768 /BufferCache/MaxCapacity
esxcfg-advcfg -s 20000 /BufferCache/FlushInterval

[Ice-Dog]

Hi all,

I narrowed it down to the same host. Should have noticed that earlier but hey ...

I restarted that host today and moved some of the VMs that had been failing back onto it and I am the running backups at the moment.

The Exchange backup of the passive DAG works fine since yesterday via vCenter after I moved the VM of to a different host in the same site.

Would still like to know what caused.... I cannot see any errors in the hostd, vmkwarning or vm logs.

[isaako]

Hi,

I've just changed the memory buffers in the above post and just worked.
I've opened a case with vmware, it seems a NBD transport limitation or bug.
I fear that if the number of VM to backups increase we will have the problem again.
Currently I'm using Veeam network mode in our proxies and I'm going to plan a configuration to SAN mode.

Isaac

[Stokkolm]

Today I encountered an error that I have never seen before from Veeam. I found the following KB article, which seems to correspond: https://www.veeam.com/kb1901 That KB was only created 16 days ago, so I'm curious if this is something that was recently discovered and if it's specifically related to vSphere 6.0 U1a or not. The error occurred when Veeam attempted to read the first VMDK, it would begin to process it and then fail:

Code: Select all

2/11/2016 12:18:41 AM :: Processing anc-sccm-dist01 Error: VDDK error: 1 (Unknown error). Value: 0x0000000000000001
Failed to read from a virtual disk
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: VDDK error: 1 (Unknown error). Value: 0x0000000000000001
Failed to read from a virtual disk
Unable to retrieve next block transmission command. Number of already processed blocks: [219].
Failed to download disk.

The KB article says to attempt a svMotion or clone the VM and try again. Neither of those suggestions worked, however doing a normal vMotion to another host did (we were able to determine that all of the VM's, about 20 of them, that had this error were all on one host). Based on that, we placed the host into maintenance mode and rebooted it. Once it came back up Veeam Backups were working again. I still don't have a clue as to the root cause, there were no major changes to any of our hosts recently and based on the severity of the case we started with VMware they aren't going to respond until tomorrow. The VM's involved were on varying datastores and across multiple arrays and there were other VM's on these datastores that backed up normally. Also, the VM's didn't show any signs of abnormal activity other than the failed backups. Anyway, I wanted to make this forum post mainly because I think the KB is lacking a little bit and hopefully if someone else runs into this error they can find this post for additional information based off of my experience today.

[foggy]

Thanks, John, much appreciated. I've passed your feedback to the responsible team for possible KB update. You can also see that you're not alone with this problem.

[picaroon]

Past two months I also experiencing this issue. Multiple cases with VMware and Veeam had been created, but today still no root cause and solution. Only known workaround is migrating the VMs and reboot the affected host. At this moment I've got another two VMs which can't be backupped due to this issue.

[Ice-Dog]

Hi all,

Since first reporting this issue I have had two other hosts with the same issue. Using vmotion to move the failed VMs to a different host and retry the failed Backup job allows the job to complete succssfully. After rebooting the hosts Veeam is able to backup running VMs of it.

Last week a Senior Team lead from Veeam support contacted me to enqiure if I had done any further investigations on the issue and suggested that if this would happen again to raise a new ticket with Veeam and they would leverage VMware support on the case.

Last night VMs on the fourth host are failing in the Backup jobs. Same issue as before. Now I need to plan another host reboot.

New ticket #01708122
Support logs uploaded with the case

Regards,
Gunnar

[rlove]

Ice-Dog,

Did you ever get a fix for this from Veeam or VMware?
Is there a VMware ticket you can reference?

[picaroon]

Last Saturday the same issue happened again. I changed the buffercache according to one of the posts in this thread, but no luck. I've ended up vMotioning the machine to another host in the cluster and Veeam is able to backup the VM.
We use 12 hosts in a Cisco UCS blade, running ESXi 5.5.0 build 3343343 and vCenter 5.5.0 build 30000241.

A while ago we created case #01303819 with Veeam support, but they advised us to create a case with VMware support since the backup is working correctly when vMotioning the VM.

Till now still no answer from VMware...

[rlove]

I feel your pain of being put in the middle of two vendors pointing fingers.
I tried a vMotion and and a SvMotion on the troubled VM's and it did not help.
I found this on Reddit - https://www.reddit.com/r/Veeam/comments ... s_failing/

It states that you have to actually reboot the node the VM was on. I rebooted the original node the VM used to be on and did a retry and it worked.
Not sure what the node was holding open/locked but it needed a complete reboot to get things moving again.
I've updated my Veeam case (01713005) and the VMware case with that info.

[picaroon]

Rebooting the host is just a temporarily workaround, I'm dealing with this issue every week. Sometimes there are nights I need to reboot 3 hosts because of this.
Anyone experienced the same issue when using hotadd instead of the NBD method? In our environment only NBD seems to be affected, but I only have had the chance to test it short.

So at this moment I'm considering hotadd method, but I'm a bit shivery to use it...

[isaako]

Hi,
I've opened a case with VMWare and they don't know how to solve the issue.
Do you have some news about this issue?

After applying the options in my previous note the issue happens some time, only 1 VM per night and some days.

Can you try to apply this setting in the failing host an try again?

esxcfg-advcfg -g /BufferCache/MaxCapacity
esxcfg-advcfg -g /BufferCache/FlushInterval
esxcfg-advcfg -s 32768 /BufferCache/MaxCapacity
esxcfg-advcfg -s 20000 /BufferCache/FlushInterval


Isaac

[LeoKurz]

Any news on this? I'm having the same issue. Running von vSphere 6 and B&R 9. Could be coincidence, but occured after I updated vCenter Appliance to 6.0U2. ESX are still on plain 6.0. Had none of this before...

__Leo

[ChuckS42]

I had this happen on two VMs after the upgrade to v9 Update 1. For me a storage vMotion seems to have fixed the problem. Odd timing, but at least I didn't need one of the more drastic solutions (this time). Case # 01744042, FWIW.

[SERaines]

Add me to the list with this same issue.

I came in to work with a bunch of failures over the weekend. All VMs on the same host. Will be migrating them off and rebooting the host today. Would like to know the reason and how to prevent this in the future.

[picaroon]

At this moment there's a case with VMware (#16894064702) and Veeam (#01728838). Still no root cause, VMware would like to create a separate case with their SDK support team.

[ChuckS42]

Just to add that I had two more VMs do the same thing, but they worked on a retry. Very odd. Veeam support directed me to VMware support. We'll see what they find.

[eric.law]

Hi all

I have this problem in ESxi 5.5 / VC 6 U2/ veeam 8 2084.
I had created VM/Veeam support case to find out what happen in the same Email loop...
but still have not find out the root cause....

In last testing, Veeam support using vixDiskLib to connect VC/Host. it have success in proxy server. since does not related to VMware side.

[Vitaliy S.]

Eric, please let us know your support case ID for future reference. Thanks!

[eric.law]

Veeam case # 01717540.

[andyg]

Veeam Support - Case # 01766224

[ChuckS42]

Mine is case # 01744042, which is currently on ice. I was deflected to VMware instead.

[rtorrecillas]

Hi all,

Add me to the list with this same issue.
Vcenter Server Appliance 6.0 u1, hosts with esxi 6.0 u1. veeam backup v9 u1.
we have to move the virtual machines to another host and reboot de host.

[Vitaliy S.]

Raul, can you please also post your case ID as all others did?

[rtorrecillas]

I have no open case

[Vitaliy S.]

In order to troubleshoot any technical issue, opening a support case is required. Thanks!

[lux209]

Hi All,

I had the exact same problem yesterday for the first time (using Veeam 9, freshly updated from . I did a host vMotion of the failing VM to other host and it solved the problem. What is interesting is that not all VMs running on the problematic ESXi failed the backup, only few of them.. Anyway I did fully reboot the host and I will see tomorrow if the problem is still there.

Anyway thanks for the posts in this thread, it helps me debug the problem quickly !

[zapnet]

Hi,

I had also the exact same problem yesterday for the first time.