Host-based backup of VMware vSphere VMs.
Post Reply
jaychunlai
Lurker
Posts: 1
Liked: never
Joined: Feb 01, 2019 11:01 am
Full Name: Chun Lai
Contact:

VDDK error 14009

Post by jaychunlai »

Hey guys,

I was hoping some of you guys could help me out, backup Veeam had been running good for some time, then I bumped in this error message:

Code: Select all

Error: VDDK error: 14009 (The server refused connection). Value: 0x00000000000036b9
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.  
A support case has been openend as well (03393772)
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: VDDK error 14009

Post by HannesK »

Hello,

please give support some time to work on your case, as you opened the case with evaluation support on Friday (evaluation tickets are not processed during weekend).

Best regards,
Hannes
L0g333
Novice
Posts: 7
Liked: never
Joined: May 10, 2017 9:36 am
Contact:

[MERGED] Backup Up VEEAM VM fails with Error 14009 - Failed to upload disk

Post by L0g333 »

Hi there :)

for some days i have a problem with backing up our VEEAM VM, that used to work like a charm for more then a year now. After ~19.5GB of transferred Data, the Backup of the VEEAM VM fails with the following error (All other VMs are backed up without any problem):

Code: Select all

06.02.2019 12:26:13 :: Error: VDDK async operation error: 14009. Value: 0x00000000000036b9
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: VDDK async operation error: 14009. Value: 0x00000000000036b9
Unable to retrieve next block transmission command. Number of already processed blocks: [62740].
Failed to download disk.
Some background information to the infrastructure:

VEEAM Backup Server (Windows Server 2016, VBR 9.5.4.2615) and Backup Target (QNAP NAS) are running in LAN A.
ESXi 6.7 Host (Build 11675023) and vCSA 6.7 (Build 11726888) are running in LAN B.

-> VMware and VEEAM are up-to-date.

LANs are seperated by Sophos Firewall. Any-Any FW Rule doesn't change the result.

VEEAM Backup Server is running as VM on the ESXi Host, and fails backup up its own VM. There's no seperate Backup Proxy in LAN B. VEEAM is connected to vCenter.

After hours of googling and reading, rebooting and updating, i am running out of ideas. Maybe someone here has an idea, what the cause of this could be.


I found this on that topic: https://vinfrastructure.it/2018/10/veea ... ror-14009/ But I am not sure if that can be the answer. We didn't touch the network and don't encounter other network problems.

Thanks a lot guys!
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: VDDK error 14009

Post by veremin » 1 person likes this post

Hello, what's the support ticket ID for this issue (as requested when you click New Topic) please? Thanks!
L0g333
Novice
Posts: 7
Liked: never
Joined: May 10, 2017 9:36 am
Contact:

Re: VDDK error 14009

Post by L0g333 »

Thank you for that hint @v.Eremin. I thought, I'd open a forum post instead of opening a ticket, since the problem isn't that severe. I'll catch up on this later, when i'm back in the office.

I actually wasn't clear about the fact i'd need a support ticket for technical issues. I must have overseen that little notice, when creating my forum post. Maybe it would be helpful to make the forum rules (veeam-backup-replication-f2/rules-of-po ... -t755.html) a sticky post.

Sorry for the inconvenience.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VDDK error 14009

Post by Andreas Neufert »

Just to give some additional background. This forum is not a support forum and only some of the support members are part of the community because of personal interest. Main part of the forum is to interact with Veeam product management, R&D and systems engineers/architects.
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: VDDK error 14009

Post by m.novelli »

I have a customer that is experiencing this issue after update to B&R 9.4 Update 4

Case ID 03398738

After Veeam update he get a "DataTransfer.SyncDisk" error on a VM. Support asked to try to clone / Storage vMotion the VM

I'll let you know

Marco
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: VDDK error 14009

Post by veremin »

Kindly, keep working with our support team directly. Also, don't post log snippets here (it's against the forum rules). Thanks!
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: VDDK error 14009

Post by m.novelli » 2 people like this post

Hello, the issue with my customer was solved after a Storage vMotion of the affected VM

Cheers
ppcat
Lurker
Posts: 2
Liked: never
Joined: Sep 20, 2018 3:21 pm
Full Name: wendy zhou
Contact:

Re: VDDK error 14009

Post by ppcat »

We are having exactly same issue.
support case 03404281

on the target vm :
we did Filesystem checks -- come back clean
Disk checks in the OS --come back clean
Disk checks at the VMFS level-- come back clean
All snapshots have been removed to eliminate any potential issue with that
Nothing configuration wise has changed on that machine
Then :
re-run backup still failed with same error .
re-create a new job for the target vm , returns same error
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VDDK error 14009

Post by Gostev »

Thanks everyone for including their support case IDs. The issue is not wide spread, so the more material we have we work on - the better!
Qasim
Lurker
Posts: 2
Liked: never
Joined: Jun 26, 2018 3:36 pm
Full Name: Qasim Umer
Contact:

Re: VDDK error 14009

Post by Qasim »

Hi
we are getting the same error for one of the machine. Do the Veeam has a solution for this error?
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: VDDK error 14009

Post by HannesK »

Hello,
and welcome to the forums.

I looked in all cases above and there were different reasons ans solutions in each case (networking, storage...).

Please contact support for further assistance

Thanks,
Hannes
mwilliam
Lurker
Posts: 1
Liked: never
Joined: Feb 14, 2019 4:48 pm
Full Name: Mattias William
Contact:

Re: VDDK error 14009

Post by mwilliam »

I am getting the same issue.

Support Case# 03665196

Started happening about a week after upgrading VMware to 6.7. Just the one backup failing, it is an application aware backup of a SQL server that only has 3 disks.
vRhinow
Influencer
Posts: 10
Liked: never
Joined: Nov 12, 2019 11:57 am
Full Name: Felix Heitmann
Contact:

Re: VDDK error 14009

Post by vRhinow »

Hi,

is there any update on this?

Best regards
Felix
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: VDDK error 14009

Post by wishr »

Hi vRhinow,

I suggest you open an independent case as each of the previous cases had a different nature. A few customers reported that cloning/migrating VMs to different storage has solved the issue for them.

Thanks
tha-lan-law
Lurker
Posts: 1
Liked: 2 times
Joined: Mar 16, 2016 12:35 pm
Full Name: YAN
Contact:

Re: VDDK error 14009

Post by tha-lan-law » 2 people like this post

Hi!

same error...

Code: Select all

Processing VM Error: VDDK error: 14009 (The server refused connection). Value: 0x00000000000036b9
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: VDDK error: 14009 (The server refused connection). Value: 0x00000000000036b9
Unable to retrieve next block transmission command. Number of already processed blocks: [4].
Failed to download disk 'VM-flat.vmdk'.
Veeam Backup & Replication Server at Datacenter, connected with VPN to multiple Sites with local ESX Servers.
ANY-ANY Rule on the Firewall -> Above Error.

:!: We had to disable the IPS Module on the (Site-)Firewall (Barracuda). :arrow: Jobs run successfully!

cu
bsieg
Influencer
Posts: 14
Liked: never
Joined: Jun 18, 2019 12:17 pm
Contact:

[MERGED] Backup Fails - VDDK async operations error: 14099

Post by bsieg »

Error: VDDK async operation error: 14009. Value: 0x00000000000036b9 Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

I am pulling my hair out with this one. We have one ESXi host running 6.7. There are a total of 5 VMs with two different domains. The backup device is a Buffalo TerraStation which is in the same server room as the host but it's connected by network. The Buffalo is in a workgroup and not in any domain.

The one domain which I have been taking up since last year works perfectly fine. The second domain I just setup does not work at all. It starts backing up and it does write files to the Buffalo so permissions are not an issue.


The Buffalo was in the same vlan as the domain that worked so I moved the Buffalo to the vlan that didn't work. That did not resolve the issue. Any help would be appreciated.


Thank you
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: VDDK error 14009

Post by Dima P. »

bsieg,

I've merged your post to the existing thread. Please make sure that nothing is blocking the connection to the ESXi host i.e. you don't have any firewall rules that might affect the connection. If that does not help, kindly, raise a support ticket and share the case ID. Thanks!
bsieg
Influencer
Posts: 14
Liked: never
Joined: Jun 18, 2019 12:17 pm
Contact:

Re: VDDK error 14009

Post by bsieg »

I did see this post when researching the error but unfortunately it was of no help.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VDDK error 14009

Post by Andreas Neufert »

Please use the latest Veeam version or contact our support and ask them for help.
bsieg
Influencer
Posts: 14
Liked: never
Joined: Jun 18, 2019 12:17 pm
Contact:

Re: VDDK error 14009

Post by bsieg »

It's the community edition so I cannot receive technical support.
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: VDDK error 14009

Post by PetrM »

Hi Bsieg,

You can get support even with Community Edition, it will be provided on a best-effort basis depending on available support resources.

Do you run backups in Network mode? I see the VDDK error code in the error message pasted above.

I'd suggest to try HotAdd to avoid VDDK read. The data is retrieved by virtual proxy from the directly attached disk instead of travelling through the network link from ESXi to proxy.
By the way, HotAdd supports Advanced Data Fetcher which is based on asynchronous read, this technology usually gives a significant grow of read speed.

Thanks!
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: VDDK error 14009

Post by wishr »

If technical support won't help you may just upgrade to the the latest version.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VDDK error 14009

Post by Andreas Neufert »

I don´t have the logs, but I belive this was addressed in one of the later versions of Veeam. So please download latest v10 and update your environment.
bsieg
Influencer
Posts: 14
Liked: never
Joined: Jun 18, 2019 12:17 pm
Contact:

Re: VDDK error 14009

Post by bsieg »

This issue was the Dell SonicWall. When the job failed and got the VDDK 14009 error, the log in the SonicWall generated the following event.....SonicWall Gateway anti-virus inform smb out of order read/write

To resolve this, I went into the internal settings of the Sonicwall to disable the Gateway AV SMB read/write option as per the article below.

https://www.sonicwall.com/support/knowl ... 070049309/


This finally resolved the 14009 error but then I got a new error message....an unexpected network error occurred. failed to flush buffers.
This was also a SonicWall issue. According to the SonicWall logs, it was picking up false positive virus signatures. I added my two servers to the exclusion list for both the Anti-Spyware and Anti-Virus security settings. I can now finally backup my server!
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VDDK error 14009

Post by Andreas Neufert »

It is not really best practices to use a firewall between the backup proxy and the VMware environments. You limit the dataflow by the firewall throughput limitation usually.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 75 guests