iSCSI link failing

[davidkeis]

Hi everyone.

I am writing to you, hoping you will be able to help me with the following problem (or at least point me in the right direction!).

The problem I am experiencing is, that the iSCSI link will fail on the Veeam backup server when I initiate a backup job.
Sometimes the backup job will run for a while before it fails, other times it seems to fail almost immediately after I start a backup job.
All backup jobs are configured to use direct SAN access.

When the iSCSI fails, the only way I can get the iSCSI link up again, is by restarting the server.

In the iSCSI Initiator I have 3 links defined (one for each volume in the SAN).
Only 2 links are used by Veeam; Volume 1 and Volume 2. Strangely it is only the connection to Volume 1 that seems to fail. The other links don’t seem to be affected, and backups from Volume 2 will work just fine with direct SAN access.

My setup is as follows:

Veeam backup 5.0 residing on a physical server with iSCSI connection through 2 stacked Dell PowerConnect 6224 switches, to a Dell EqualLogic PS4000 SAN.
Jumbo frames are enabled, and flow control set up in the switches.

Backup is taken from 2 vSphere hosts running VMware ESXi 4.1.0. Both hosts have redundant Gb iSCSI links to the switches, and redundant Gb Ethernet connections.
Backups are stored on a Synology RS810+. Throughput while backing up is roughly 70 MB/sec until direct SAN access fails on the Veeam backup server. After that it falls to roughly 35-40 MB/sec.

I hope that I have included enough information, and that you will be able to help me figure this one out!


EDIT: Images attached

[Vitaliy S.]

Hello David,

The first thing I would do is to update firmware/drivers for all the controllers. In addition, could you please tell if you have any errors in the Windows Event log, that might be somehow related to iSCSI initiator issue?

Thanks!

[davidkeis]

Thanks for the reply. I will check the firmware / driver versions, and update if applicable.

I have attached a screenshot from the system event log - which is the only place any information about this error seems to be recorded. There is no unusual information in the Veeam event log.

[Vitaliy S.]

Yes, that event doesn't tell much...By the way, how many backup jobs accessing the same SAN LUN do you run concurrently?

[davidkeis]

Yes, unfortunatly, the eventlog does not help much!

I currently have 14 backup jobs, scheduled to run once a day.
The jobs are executed by a PowerShell script that is triggered by the windows task scheduler.

Below is the script (and by the way, thanks to this forum for help creating the script).
As you can see the jobs are executed sequentially to prevent overloading the backup server / snapshot space used by the VM's

Code: Select all

############################################### 
# PowerShell script til backup af VM
#
# Forfatter:  	DKE  2011-02-14
#				Første version
#
############################################### 

# indlæs Veeam kommando-plugin
Add-PSSnapin VeeamPSSnapIn


# backup job: dkvam-ad2
Get-VBRJob | where {$_.Name –eq "backup dkvam-ad2"} | Start-VBRJob

# backup job: dkvam-cimio1
Get-VBRJob | where {$_.Name –eq "backup dkvam-cimio1"} | Start-VBRJob

# backup job: dkvam-ctxman
Get-VBRJob | where {$_.Name –eq "backup dkvam-ctxman"} | Start-VBRJob

# backup job: dkvam-etikser
Get-VBRJob | where {$_.Name –eq "backup dkvam-etikser"} | Start-VBRJob

# backup job: dkvam-fileserv
Get-VBRJob | where {$_.Name –eq "backup dkvam-fileserv"} | Start-VBRJob

# backup job: dkvam-ftassetc
Get-VBRJob | where {$_.Name –eq "backup dkvam-ftassetc"} | Start-VBRJob

# backup job: dkvam-man3
Get-VBRJob | where {$_.Name –eq "backup dkvam-man3"} | Start-VBRJob

# backup job: dkvam-messerv
Get-VBRJob | where {$_.Name –eq "backup dkvam-messerv"} | Start-VBRJob

# backup job: dkvam-test1
Get-VBRJob | where {$_.Name –eq "backup dkvam-test1"} | Start-VBRJob

# backup job: dkvam-vcenter
Get-VBRJob | where {$_.Name –eq "backup dkvam-vcenter"} | Start-VBRJob

# backup job: dkvam-web01
Get-VBRJob | where {$_.Name –eq "backup dkvam-web01"} | Start-VBRJob

# backup job: dkvam-xa1
Get-VBRJob | where {$_.Name –eq "backup dkvam-xa1"} | Start-VBRJob

# backup job: dkvam-xaman
Get-VBRJob | where {$_.Name –eq "backup dkvam-xaman"} | Start-VBRJob

# backup job: mesaep_V01
Get-VBRJob | where {$_.Name –eq "backup mesaep_V01"} | Start-VBRJob

[Vitaliy S.]

Everything seems fine from the backup organization process, so please double-check the firmware/drivers, should help. Let us know how it goes for you.

[davidkeis]

Hi again.

I have found the cause.
On the Veeam backup server, the network cards handling the iSCSI connections are from Broadcom.
Apparently the Broadcom driver does not handle the hardware offload very well, resulting in these errors.

By disabling the hardware offload for
- "IPv4 Large Send Offload"
- "TCP Connection Offload (IPv4)"

and also disabling any network pairing (BASP virtual adapter) in the iSCSI fabric, the backups have been running smoothly (apart from other glitches in the system, caused by other issues).

-So i'm still straightening out the kinks in the backup system. But at least this rather severe problem is now history.

[Vitaliy S.]

David, thank you for this update. I've never heard about Broadcom drivers issues...

[Gostev]

Nice troubleshooting effort... impressive. Thanks for taking time to post the update!