Multiple jobs connecting to same Datastore

[flfb]

Recently several server backups within a few of our jobs have been failing with the following error:

Backing up file "[DataStore] Server/Server-flat.vmdk" Unable to establish direct connection to the shared storage (SAN). Please ensure that: - HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly. - SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server. - Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation). BackupDisk failed Client error: Failed to open VDDK disk [[DataStore] Server/Server.vmdk] ( is read-only mode - [true] ) Failed to open VMDK. Logon attempt with parameters [VC/ESX: [ConsoleServer.company.com];Port: 443;Login: [vmadmin];VMX Spec: [moref=vm-11976];Snapshot mor: [snapshot-16076];Transports: [san]] failed because of the following errors: Server error: End of file

We have 5 jobs with mutliple servers in each job. Each job is configured to use the VMware vStorage API in SAN mode Only. Our backup server is connected to the the datastores in read-only mode (connectivity to the datastores is definitely not the issue). Each job spans multiple datastores. We usualy see this error, for example, let's say job 1 is backing up server1 connected to datastore-1. Job 2 kicks off and it tries to backup server2 also connected to datastore-1 while job 1 is still backing up server1. Server2 in job 2 fails. However, on the retry of job 2 server2 is backed up successfully.

So with that long winded explanation my question is can two seperate jobs in Veeam access the same datastore at the same time? Or is this a function of the vStorage API? Is there a work around?

Any help is much appreciated!

[Gostev]

Hello Jeremy, generally we do not advise on running parallel jobs hitting the same LUN, because it causes a number of problems with VMFS like SCSI reservation timeouts (actually VMware has technical white paper about this, I will try to look it up for you).

Usually, this does not cause actual errors (just some operations like snapshot management start to take ridiculous time), but I guess depending on specific storage/HBA make and model this can lead to issues like that.

To help our customers better align with VMware recommendations around parallel datastore access, we are planning to give ability to create datastore-based jobs in v5 (in other words, stuff jobs with datastore containers). This has been common request from our customers, because it would allow to increase reliability of backups and simplify job management; so I am trying really hard to make sure this makes into v5 release.

Anyhow, if you'd like us to investigate this further to ensure the issue is not caused by something else (unreliable network, overloaded vCenter), then we will need full logs to be submitted to support@veeam.com.

Thank you.

[flfb]

Mr. Gostev,

Thank your for the quick reply. The white paper would be helpful if you have time to find it. The enhancement of datastore-based jobs would be very helpful. Matter of fact to get around this issue I was planning on doing that manually. I opened a case with support and I am submitting the logs as requested.

Once again your help is very much appreciated!


Best regards,

Jeremy

[Gostev]

Jeremy, I have finally tracked the white paper down for you
http://www.vmware.com/files/pdf/scalabl ... rmance.pdf

See page 6 for SCSI reservation issues described, and some actual numbers showing the performance effects of SCSI reservations.

Also, just as an update, datastore-based jobs have made it into v5.

Hope this helps!