Comprehensive data protection for all workloads
Post Reply
StefanSpecht
Influencer
Posts: 16
Liked: never
Joined: Aug 17, 2010 12:21 pm
Full Name: Stefan Specht
Contact:

Failed to open VDDK disk / Parallel processing

Post by StefanSpecht »

Hi,

we just updated to Veeam B&R v7 R2. After we enabled "Parallel Processing" feature we encounter a lot of failed VM backups with the following error message:

"Error: Client error: Failed to open VDDK disk [[DSCLU01_UR_T1_02] ds1874/ds1874.vmdk] ( is read-only mode - [true] )
Failed to open VMDK.
Logon attempt with parameters [VC/ESX: [vcenter.fqdn.com];Port: 443;Login: [****];VMX Spec: [moref=vm-481];Snapshot mor: [snapshot-67431];Transports: [hotadd];Read Only: [true]] failed because of the following errors: "

We have parallel processing enabled as well as "HotAdd" with failover to network disabled.
We have 4 proxy servers with 6vCPU's each, backing up about 250 VM's to a CIFS share. All jobs are scheduled to start at the same time (but we did also had this error when scheduling one job after the other...)
When we disable parallel processing, or we allow the failover to network (from HotAdd), the messages are gone.

Support told me that I have to add an additional proxy server, but was not able to tell me why so far.
Any ideas, why this is happening here? I would like to avoid installing an additional server without understanding, why I have to.
Support Case: 00510068
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Failed to open VDDK disk / Parallel processing

Post by Vitaliy S. »

Hello Stefan,

I am also not sure why additional proxy server should address this issue, so I would double-check it with the support team. Anyway, what happens if you have parallel processing and force to network processing mode options enabled at the same time? Do you see the same errors?

BTW, do all VMs fail in this case? Additionally, can you please tell me if there is any common thing between the VMs that fail? Same host? Storage?

Thank you!
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Failed to open VDDK disk / Parallel processing

Post by tsightler »

Can you tell me more about your source storage. Does it happen to be NFS?
StefanSpecht
Influencer
Posts: 16
Liked: never
Joined: Aug 17, 2010 12:21 pm
Full Name: Stefan Specht
Contact:

Re: Failed to open VDDK disk / Parallel processing

Post by StefanSpecht »

Hi,

@Vitaly: The failed VM's don't seems to have anything special in common. They are located on diffrent datastores, used diffrent proxy servers, different hosts....
Also, this error seems to be very random. So every day, some other VMs are failing.
The issue only happens if I use Virtual-Appliance mode without failover to network enabled. Every day, about 10 VM's are affected.

@tsightler: Source storage are two NetApp FAS2370, connected by NFS. We present 10 datastores with 4TB each to our ESXi hosts.

Today, there also appeared new error messages for two VM's -> "vmx Datastore not found". Don't know, if these are related to the other message...
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Failed to open VDDK disk / Parallel processing

Post by tsightler »

Part of the reason I asked about whether your storage was NFS is that I suspected what you are seeing is being caused by known locking issues when using Hotadd with NFS datastores (which is why you don't see the issue with network mode). The most common issue when using hotadd with NFS is the VM stun caused by NFS locking when a proxy on one host is used to backup a VM on another host (something that works perfectly fine with block storage). However, with parallel processing there can be many more hotadd operations and these locks can also lead to random failures of the hotadd process itself, which can lead to even more trouble (stuck and/or orphaned snapshots for example).

The general recommendation is to just use network mode for these environments, especially if 10GbE is available on the management network. Network mode is typically fast enough if the bandwidth is available and has significantly less issues, and it's really the only reliable option to use with parallel processing and NFS for most setups. However, for customers that perhaps only have 1GbE for the management network, network mode might not provide enough performance so the recently released patch 3 for V7 introduced a new option that can be enabled via a registry key to prefer using a proxy that is on the same host as the VM. This tweak can allow you to use hotadd successfully with NFS. You can read more details in this excellent blog post writting by our very own Timothy Dewin.

Of course the one big disadvantage to this approach is that you must have at least one proxy on every host, but for now it's either that, or just use network mode if it provides acceptable performance.
StefanSpecht
Influencer
Posts: 16
Liked: never
Joined: Aug 17, 2010 12:21 pm
Full Name: Stefan Specht
Contact:

Re: Failed to open VDDK disk / Parallel processing

Post by StefanSpecht »

Thank you so much for this information! After I switched over to network mode, also the other "weired", random failures gone away.
As we have 10GbE and I don't want to install 10 Proxy Servers for 10 ESXi hosts, I will leave it on network mode now. Backup speed is still the same, as "source" is our bottleneck.

btw: Support still tells me that I need more proxy servers, as only one job can be assigned to a virtual appliance when parallel processing is used. All additional jobs would have to run in network mode. So I would either need more proxies or set some of the backup jobs to network mode.
This still makes not too much sense for me...But as tslighter gave this excelelent hint, I can let the SR be closed now.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Failed to open VDDK disk / Parallel processing

Post by foggy »

StefanSpecht wrote:btw: Support still tells me that I need more proxy servers, as only one job can be assigned to a virtual appliance when parallel processing is used. All additional jobs would have to run in network mode.
This is not completely true. They probably mean the VDDK limitation not allowing to hotadd several disks of the same VM to target proxies simultaneously.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 156 guests