random backup failures

[fgw]

have about 100 vm's distributed over 6 esxi servers. all servers are connected to an hp eva4400 san hosting the vms. backup jopbs are configured to run in "SAN only" mode.

when i run my backupjobs i usually have about 8 jobs running concurrently. there is one big job running on one backup server and all other jobs on a second server.

sometimes vm's which run fine last time, fail to backup with errors like this:

Backing up file "[GB1_VMW_OFFICE_04] VEX450 - ADH Development/VEX450 - ADH Development-flat.vmdk"
BackupDisk failed
Client error: The specified transports are not available. Available transports: [nbd]. Specified transports: [san]. VMDK path: [[GB1_VMW_OFFICE_04] VEX450 - ADH Development/VEX450 - ADH Development.vmdk].

Server error: End of file


or


Backing up file "[SG1_VMW_OFFICE_05] VEX339/VEX339-flat.vmdk"
Unable to establish direct connection to the shared storage (SAN).
Please ensure that:
- HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly.
- SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server.
- Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation).

BackupDisk failed
Client error: Failed to open VDDK disk [[SG1_VMW_OFFICE_05] VEX339/VEX339.vmdk] ( is read-only mode - [true] )
Failed to open VMDK.
Logon attempt with parameters [VC/ESX: [vex296];Port: 443;Login: [LocalAdmin];VMX Spec: [moref=vm-583];Snapshot mor: [snapshot-2295];Transports: [san]] failed because of the following errors:

Server error: End of file


next time when i run the backups the failing vm's get backed up without problems, but another vm migt run into an issue.

access to the san should not be the problem as the backup of the failing vm worked the last time i ran the job and also the next time ...

any ideas?

[Gostev]

Franz, try reducing the number of concurrent jobs to 3-4 and see if this helps. We do not generally recommend runnning multiple jobs in parallel specifically for backup reliability reasons. There are all sort of possible issues to expect with multiple concurrent jobs, for instance iSCSI reservation conflicts.

[fgw]

Franz, try reducing the number of concurrent jobs to 3-4 and see if this helps. We do not generally recommend runnning multiple jobs in parallel specifically for backup reliability reasons. There are all sort of possible issues to expect with multiple concurrent jobs, for instance iSCSI reservation conflicts.

anton, when you say to run just 3-4 jobs concurrently,

- do you mean 3-4 concurrent jobs per backup server. as i have 2 veeam backup servers this would result in 6-8 jobs in total

or

- 3-4 concurrent jobs in total, in the whole san?

[Gostev]

I meant per Backup server, assuming that your issue is primarily caused due to Backup server's HBA not being able to handle the load. I did not realize you have multiple backup server, but this should not cause any problems unless the concurrent jobs hit the same LUNs.