Host-based backup of VMware vSphere VMs.
Ice-Dog
Influencer
Posts: 11
Liked: 2 times
Joined: Feb 01, 2016 10:39 am
Full Name: Gunnar Olafsson
Contact:

Repeated VDDK error 1 and 2

Post by Ice-Dog » 1 person likes this post

Hello,

Support ticket Case # 01676511

My environment consists of 230 VMs running on 14x ESXi 5.5 build 3116895 on Cisco UCS blades. I am running this across two sites with two 10Gb dark fiber links with no latency.

I have Veeam B&R v8 u3 (8.0.0.2084) + 2 Backup proxies.

Few VMs in various jobs seem to recently started failing with the same error.

Error: VDDK error: 1 (Unknown error). Value: 0x0000000000000001 Failed to read from a virtual disk Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: VDDK error: 1 (Unknown error). Value: 0x0000000000000001 Failed to read from a virtual disk Unable to retrieve next block transmission command. Number of already processed blocks: [7]. Failed to download disk.

One if these VMs is my Exchange server so I raised the case regarding that server.

1. Response was to storage migrate all the VM disks to a different date store. - done same error

2. I uploaded Support logs and Veeam found found this error,
"[NFC ERROR] NfcFssrvrProcessErrorMsg: received diskLib error 786441 from server: NFC_DISKLIB_ERROR
This error is generated by you ESXi host, please refer to following KB to try to sort out this issue: http://kb.vmware.com/selfservice/micros ... Id=1004424"


My ESX hosts are set to 256MB by default in ESXi 5.5 which is also the maximum. although, as per the kb it can grow to 640MB as required. That allows a maximum of 60 TB to be opened per host.

3. Asked to add the ESXi hosts as a standalone host to Veeam and try backing up the VM. - Backup successful using this workaround to skip vCenter

4. Veeam suggest this is an unusual CBT issue and ask me to try restting the CBT info on the VM. - I do that but the backup via vCener is still unsuccessful but the error changes to

"Processing Exchange01 Error: VDDK error: 2 (Memory allocation failed. Out of memory.). Value: 0x0000000000000002 Failed to read from a virtual disk Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: VDDK error: 2 (Memory allocation failed. Out of memory.). Value: 0x0000000000000002 Failed to read from a virtual disk Unable to retrieve next block transmission command. Number of already processed blocks: [0]. Failed to download disk. "

Last night I start playing around with other VMs that have had the same issue. After resetting the CBT on them they change the error message to the one above. But I discoveed that all the VMs that are failing are in site B. If I vmotion them to hosts in site A and retry the job Veeam can open up the disks and complete the job. I have DRS site affinity rules so VMs are not running split site (e.g. their disks in site A but they running on host memory in site B) Even if DRS migrates the VM back to site B the Veeam job carries on, they only need to be in site a as Veeam starts processing the VM and accessing the disks

Any toughs would be appreciated

Thanks
isaako
Service Provider
Posts: 26
Liked: never
Joined: Sep 15, 2010 11:31 am
Full Name: Isaac González
Contact:

Re: Repeted VDDK error 1 and 2

Post by isaako »

I've the same error during backup window.
I'm backing up about 550 VMs, all my failed backups are from the same ESXi hosts.
I'm going to investigate this issue.
isaako
Service Provider
Posts: 26
Liked: never
Joined: Sep 15, 2010 11:31 am
Full Name: Isaac González
Contact:

Re: Repeted VDDK error 1 and 2

Post by isaako »

Hi,

Yesterday my backups failed in a concrete ESxi host. Today some of them has failed in another ESXi host (but all the jobs in the same).
I'm trying to play whith ESXi buffers to solve the issue:

esxcfg-advcfg -g /BufferCache/MaxCapacity
esxcfg-advcfg -g /BufferCache/FlushInterval
esxcfg-advcfg -s 32768 /BufferCache/MaxCapacity
esxcfg-advcfg -s 20000 /BufferCache/FlushInterval
Ice-Dog
Influencer
Posts: 11
Liked: 2 times
Joined: Feb 01, 2016 10:39 am
Full Name: Gunnar Olafsson
Contact:

Re: Repeted VDDK error 1 and 2

Post by Ice-Dog »

Hi all,

I narrowed it down to the same host. Should have noticed that earlier but hey ...

I restarted that host today and moved some of the VMs that had been failing back onto it and I am the running backups at the moment.

The Exchange backup of the passive DAG works fine since yesterday via vCenter after I moved the VM of to a different host in the same site.

Would still like to know what caused.... I cannot see any errors in the hostd, vmkwarning or vm logs.
isaako
Service Provider
Posts: 26
Liked: never
Joined: Sep 15, 2010 11:31 am
Full Name: Isaac González
Contact:

Re: Repeted VDDK error 1 and 2

Post by isaako »

Hi,

I've just changed the memory buffers in the above post and just worked.
I've opened a case with vmware, it seems a NBD transport limitation or bug.
I fear that if the number of VM to backups increase we will have the problem again.
Currently I'm using Veeam network mode in our proxies and I'm going to plan a configuration to SAN mode.

Isaac
Stokkolm
Influencer
Posts: 16
Liked: 5 times
Joined: Feb 11, 2016 11:11 pm
Full Name: John Zetterman
Contact:

[MERGED] VDDK Error

Post by Stokkolm » 1 person likes this post

Today I encountered an error that I have never seen before from Veeam. I found the following KB article, which seems to correspond: https://www.veeam.com/kb1901 That KB was only created 16 days ago, so I'm curious if this is something that was recently discovered and if it's specifically related to vSphere 6.0 U1a or not. The error occurred when Veeam attempted to read the first VMDK, it would begin to process it and then fail:

Code: Select all

2/11/2016 12:18:41 AM :: Processing anc-sccm-dist01 Error: VDDK error: 1 (Unknown error). Value: 0x0000000000000001
Failed to read from a virtual disk
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: VDDK error: 1 (Unknown error). Value: 0x0000000000000001
Failed to read from a virtual disk
Unable to retrieve next block transmission command. Number of already processed blocks: [219].
Failed to download disk.
The KB article says to attempt a svMotion or clone the VM and try again. Neither of those suggestions worked, however doing a normal vMotion to another host did (we were able to determine that all of the VM's, about 20 of them, that had this error were all on one host). Based on that, we placed the host into maintenance mode and rebooted it. Once it came back up Veeam Backups were working again. I still don't have a clue as to the root cause, there were no major changes to any of our hosts recently and based on the severity of the case we started with VMware they aren't going to respond until tomorrow. The VM's involved were on varying datastores and across multiple arrays and there were other VM's on these datastores that backed up normally. Also, the VM's didn't show any signs of abnormal activity other than the failed backups. Anyway, I wanted to make this forum post mainly because I think the KB is lacking a little bit and hopefully if someone else runs into this error they can find this post for additional information based off of my experience today.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Repeted VDDK error 1 and 2

Post by foggy »

Thanks, John, much appreciated. I've passed your feedback to the responsible team for possible KB update. You can also see that you're not alone with this problem.
picaroon
Influencer
Posts: 13
Liked: 5 times
Joined: Feb 12, 2016 1:44 pm
Full Name: Jeltsen Haagsma

Re: Repeted VDDK error 1 and 2

Post by picaroon »

Past two months I also experiencing this issue. Multiple cases with VMware and Veeam had been created, but today still no root cause and solution. Only known workaround is migrating the VMs and reboot the affected host. At this moment I've got another two VMs which can't be backupped due to this issue.
Ice-Dog
Influencer
Posts: 11
Liked: 2 times
Joined: Feb 01, 2016 10:39 am
Full Name: Gunnar Olafsson
Contact:

Re: Repeted VDDK error 1 and 2

Post by Ice-Dog »

Hi all,

Since first reporting this issue I have had two other hosts with the same issue. Using vmotion to move the failed VMs to a different host and retry the failed Backup job allows the job to complete succssfully. After rebooting the hosts Veeam is able to backup running VMs of it.

Last week a Senior Team lead from Veeam support contacted me to enqiure if I had done any further investigations on the issue and suggested that if this would happen again to raise a new ticket with Veeam and they would leverage VMware support on the case.

Last night VMs on the fourth host are failing in the Backup jobs. Same issue as before. Now I need to plan another host reboot.

New ticket #01708122
Support logs uploaded with the case

Regards,
Gunnar
rlove
Service Provider
Posts: 2
Liked: never
Joined: Mar 01, 2016 8:34 pm
Contact:

Re: Repeted VDDK error 1 and 2

Post by rlove »

Ice-Dog,

Did you ever get a fix for this from Veeam or VMware?
Is there a VMware ticket you can reference?
picaroon
Influencer
Posts: 13
Liked: 5 times
Joined: Feb 12, 2016 1:44 pm
Full Name: Jeltsen Haagsma

Re: Repeted VDDK error 1 and 2

Post by picaroon » 1 person likes this post

Last Saturday the same issue happened again. I changed the buffercache according to one of the posts in this thread, but no luck. I've ended up vMotioning the machine to another host in the cluster and Veeam is able to backup the VM.
We use 12 hosts in a Cisco UCS blade, running ESXi 5.5.0 build 3343343 and vCenter 5.5.0 build 30000241.

A while ago we created case #01303819 with Veeam support, but they advised us to create a case with VMware support since the backup is working correctly when vMotioning the VM.

Till now still no answer from VMware...
rlove
Service Provider
Posts: 2
Liked: never
Joined: Mar 01, 2016 8:34 pm
Contact:

Re: Repeted VDDK error 1 and 2

Post by rlove »

I feel your pain of being put in the middle of two vendors pointing fingers.
I tried a vMotion and and a SvMotion on the troubled VM's and it did not help.
I found this on Reddit - https://www.reddit.com/r/Veeam/comments ... s_failing/

It states that you have to actually reboot the node the VM was on. I rebooted the original node the VM used to be on and did a retry and it worked.
Not sure what the node was holding open/locked but it needed a complete reboot to get things moving again.
I've updated my Veeam case (01713005) and the VMware case with that info.
picaroon
Influencer
Posts: 13
Liked: 5 times
Joined: Feb 12, 2016 1:44 pm
Full Name: Jeltsen Haagsma

Re: Repeted VDDK error 1 and 2

Post by picaroon »

Rebooting the host is just a temporarily workaround, I'm dealing with this issue every week. Sometimes there are nights I need to reboot 3 hosts because of this.
Anyone experienced the same issue when using hotadd instead of the NBD method? In our environment only NBD seems to be affected, but I only have had the chance to test it short.

So at this moment I'm considering hotadd method, but I'm a bit shivery to use it...
isaako
Service Provider
Posts: 26
Liked: never
Joined: Sep 15, 2010 11:31 am
Full Name: Isaac González
Contact:

Re: Repeted VDDK error 1 and 2

Post by isaako »

Hi,
I've opened a case with VMWare and they don't know how to solve the issue.
Do you have some news about this issue?

After applying the options in my previous note the issue happens some time, only 1 VM per night and some days.

Can you try to apply this setting in the failing host an try again?

esxcfg-advcfg -g /BufferCache/MaxCapacity
esxcfg-advcfg -g /BufferCache/FlushInterval
esxcfg-advcfg -s 32768 /BufferCache/MaxCapacity
esxcfg-advcfg -s 20000 /BufferCache/FlushInterval


Isaac
LeoKurz
Veeam ProPartner
Posts: 28
Liked: 7 times
Joined: Mar 16, 2011 8:36 am
Full Name: Leonhard Kurz
Contact:

Re: Repeted VDDK error 1 and 2

Post by LeoKurz »

Any news on this? I'm having the same issue. Running von vSphere 6 and B&R 9. Could be coincidence, but occured after I updated vCenter Appliance to 6.0U2. ESX are still on plain 6.0. Had none of this before...

__Leo
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: Repeted VDDK error 1 and 2

Post by ChuckS42 » 1 person likes this post

I had this happen on two VMs after the upgrade to v9 Update 1. For me a storage vMotion seems to have fixed the problem. Odd timing, but at least I didn't need one of the more drastic solutions (this time). Case # 01744042, FWIW.
Veeaming since 2013
SERaines
Lurker
Posts: 1
Liked: never
Joined: Aug 05, 2014 12:01 pm
Full Name: Scott Raines

Re: Repeted VDDK error 1 and 2

Post by SERaines »

Add me to the list with this same issue.

I came in to work with a bunch of failures over the weekend. All VMs on the same host. Will be migrating them off and rebooting the host today. Would like to know the reason and how to prevent this in the future.
picaroon
Influencer
Posts: 13
Liked: 5 times
Joined: Feb 12, 2016 1:44 pm
Full Name: Jeltsen Haagsma

Re: Repeted VDDK error 1 and 2

Post by picaroon »

At this moment there's a case with VMware (#16894064702) and Veeam (#01728838). Still no root cause, VMware would like to create a separate case with their SDK support team.
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: Repeted VDDK error 1 and 2

Post by ChuckS42 »

Just to add that I had two more VMs do the same thing, but they worked on a retry. Very odd. Veeam support directed me to VMware support. We'll see what they find.
Veeaming since 2013
eric.law
Lurker
Posts: 2
Liked: never
Joined: Apr 18, 2016 3:32 am
Contact:

Re: Repeted VDDK error 1 and 2

Post by eric.law »

Hi all

I have this problem in ESxi 5.5 / VC 6 U2/ veeam 8 2084.
I had created VM/Veeam support case to find out what happen in the same Email loop...
but still have not find out the root cause....

In last testing, Veeam support using vixDiskLib to connect VC/Host. it have success in proxy server. since does not related to VMware side.
Vitaliy S.
VP, Product Management
Posts: 27025
Liked: 2709 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Repeted VDDK error 1 and 2

Post by Vitaliy S. »

Eric, please let us know your support case ID for future reference. Thanks!
eric.law
Lurker
Posts: 2
Liked: never
Joined: Apr 18, 2016 3:32 am
Contact:

Re: Repeted VDDK error 1 and 2

Post by eric.law »

Veeam case # 01717540.
andyg
Enthusiast
Posts: 58
Liked: 5 times
Joined: Apr 23, 2014 9:51 am
Full Name: Andy Goldschmidt
Contact:

Re: Repeted VDDK error 1 and 2

Post by andyg »

Veeam Support - Case # 01766224
-= VMCE v9 certified =-
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: Repeted VDDK error 1 and 2

Post by ChuckS42 »

Mine is case # 01744042, which is currently on ice. I was deflected to VMware instead.
Veeaming since 2013
rtorrecillas
Lurker
Posts: 2
Liked: never
Joined: Nov 17, 2015 9:27 am
Full Name: Raul Torrecillas
Contact:

Re: Repeted VDDK error 1 and 2

Post by rtorrecillas »

Hi all,

Add me to the list with this same issue.
Vcenter Server Appliance 6.0 u1, hosts with esxi 6.0 u1. veeam backup v9 u1.
we have to move the virtual machines to another host and reboot de host.
Vitaliy S.
VP, Product Management
Posts: 27025
Liked: 2709 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Repeted VDDK error 1 and 2

Post by Vitaliy S. »

Raul, can you please also post your case ID as all others did?
rtorrecillas
Lurker
Posts: 2
Liked: never
Joined: Nov 17, 2015 9:27 am
Full Name: Raul Torrecillas
Contact:

Re: Repeted VDDK error 1 and 2

Post by rtorrecillas »

I have no open case
Vitaliy S.
VP, Product Management
Posts: 27025
Liked: 2709 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Repeted VDDK error 1 and 2

Post by Vitaliy S. »

In order to troubleshoot any technical issue, opening a support case is required. Thanks!
lux209
Novice
Posts: 8
Liked: never
Joined: Feb 16, 2012 10:07 am
Full Name: Lucas Plancherel
Contact:

Re: Repeated VDDK error 1 and 2

Post by lux209 »

Hi All,

I had the exact same problem yesterday for the first time (using Veeam 9, freshly updated from 8). I did a host vMotion of the failing VM to other host and it solved the problem. What is interesting is that not all VMs running on the problematic ESXi failed the backup, only few of them.. Anyway I did fully reboot the host and I will see tomorrow if the problem is still there.

Anyway thanks for the posts in this thread, it helps me debug the problem quickly !
zapnet
Lurker
Posts: 2
Liked: never
Joined: Mar 27, 2015 2:22 pm
Full Name: Oliver Servay
Contact:

Re: Repeated VDDK error 1 and 2

Post by zapnet »

Hi,

I had also the exact same problem yesterday for the first time.
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 24 guests