Comprehensive data protection for all workloads
Post Reply
davidkeis
Influencer
Posts: 10
Liked: never
Joined: Feb 14, 2011 8:09 am
Full Name: David Keis
Contact:

iSCSI link failing

Post by davidkeis »

Hi everyone.

I am writing to you, hoping you will be able to help me with the following problem (or at least point me in the right direction!).

The problem I am experiencing is, that the iSCSI link will fail on the Veeam backup server when I initiate a backup job.
Sometimes the backup job will run for a while before it fails, other times it seems to fail almost immediately after I start a backup job.
All backup jobs are configured to use direct SAN access.

When the iSCSI fails, the only way I can get the iSCSI link up again, is by restarting the server.

In the iSCSI Initiator I have 3 links defined (one for each volume in the SAN).
Only 2 links are used by Veeam; Volume 1 and Volume 2. Strangely it is only the connection to Volume 1 that seems to fail. The other links don’t seem to be affected, and backups from Volume 2 will work just fine with direct SAN access.

My setup is as follows:

Veeam backup 5.0 residing on a physical server with iSCSI connection through 2 stacked Dell PowerConnect 6224 switches, to a Dell EqualLogic PS4000 SAN.
Jumbo frames are enabled, and flow control set up in the switches.

Backup is taken from 2 vSphere hosts running VMware ESXi 4.1.0. Both hosts have redundant Gb iSCSI links to the switches, and redundant Gb Ethernet connections.
Backups are stored on a Synology RS810+. Throughput while backing up is roughly 70 MB/sec until direct SAN access fails on the Veeam backup server. After that it falls to roughly 35-40 MB/sec.

I hope that I have included enough information, and that you will be able to help me figure this one out! :)


EDIT: Images attached

Image Image
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: iSCSI link failing

Post by Vitaliy S. »

Hello David,

The first thing I would do is to update firmware/drivers for all the controllers. In addition, could you please tell if you have any errors in the Windows Event log, that might be somehow related to iSCSI initiator issue?

Thanks!
davidkeis
Influencer
Posts: 10
Liked: never
Joined: Feb 14, 2011 8:09 am
Full Name: David Keis
Contact:

Re: iSCSI link failing

Post by davidkeis »

Thanks for the reply. I will check the firmware / driver versions, and update if applicable.

I have attached a screenshot from the system event log - which is the only place any information about this error seems to be recorded. There is no unusual information in the Veeam event log.

Image
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: iSCSI link failing

Post by Vitaliy S. »

Yes, that event doesn't tell much...By the way, how many backup jobs accessing the same SAN LUN do you run concurrently?
davidkeis
Influencer
Posts: 10
Liked: never
Joined: Feb 14, 2011 8:09 am
Full Name: David Keis
Contact:

Re: iSCSI link failing

Post by davidkeis »

Yes, unfortunatly, the eventlog does not help much!

I currently have 14 backup jobs, scheduled to run once a day.
The jobs are executed by a PowerShell script that is triggered by the windows task scheduler.

Below is the script (and by the way, thanks to this forum for help creating the script).
As you can see the jobs are executed sequentially to prevent overloading the backup server / snapshot space used by the VM's

Code: Select all

############################################### 
# PowerShell script til backup af VM
#
# Forfatter:  	DKE  2011-02-14
#				Første version
#
############################################### 

# indlæs Veeam kommando-plugin
Add-PSSnapin VeeamPSSnapIn


# backup job: dkvam-ad2
Get-VBRJob | where {$_.Name –eq "backup dkvam-ad2"} | Start-VBRJob

# backup job: dkvam-cimio1
Get-VBRJob | where {$_.Name –eq "backup dkvam-cimio1"} | Start-VBRJob

# backup job: dkvam-ctxman
Get-VBRJob | where {$_.Name –eq "backup dkvam-ctxman"} | Start-VBRJob

# backup job: dkvam-etikser
Get-VBRJob | where {$_.Name –eq "backup dkvam-etikser"} | Start-VBRJob

# backup job: dkvam-fileserv
Get-VBRJob | where {$_.Name –eq "backup dkvam-fileserv"} | Start-VBRJob

# backup job: dkvam-ftassetc
Get-VBRJob | where {$_.Name –eq "backup dkvam-ftassetc"} | Start-VBRJob

# backup job: dkvam-man3
Get-VBRJob | where {$_.Name –eq "backup dkvam-man3"} | Start-VBRJob

# backup job: dkvam-messerv
Get-VBRJob | where {$_.Name –eq "backup dkvam-messerv"} | Start-VBRJob

# backup job: dkvam-test1
Get-VBRJob | where {$_.Name –eq "backup dkvam-test1"} | Start-VBRJob

# backup job: dkvam-vcenter
Get-VBRJob | where {$_.Name –eq "backup dkvam-vcenter"} | Start-VBRJob

# backup job: dkvam-web01
Get-VBRJob | where {$_.Name –eq "backup dkvam-web01"} | Start-VBRJob

# backup job: dkvam-xa1
Get-VBRJob | where {$_.Name –eq "backup dkvam-xa1"} | Start-VBRJob

# backup job: dkvam-xaman
Get-VBRJob | where {$_.Name –eq "backup dkvam-xaman"} | Start-VBRJob

# backup job: mesaep_V01
Get-VBRJob | where {$_.Name –eq "backup mesaep_V01"} | Start-VBRJob
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: iSCSI link failing

Post by Vitaliy S. »

Everything seems fine from the backup organization process, so please double-check the firmware/drivers, should help. Let us know how it goes for you.
davidkeis
Influencer
Posts: 10
Liked: never
Joined: Feb 14, 2011 8:09 am
Full Name: David Keis
Contact:

Re: iSCSI link failing

Post by davidkeis »

Hi again.

I have found the cause.
On the Veeam backup server, the network cards handling the iSCSI connections are from Broadcom.
Apparently the Broadcom driver does not handle the hardware offload very well, resulting in these errors.

By disabling the hardware offload for
- "IPv4 Large Send Offload"
- "TCP Connection Offload (IPv4)"

and also disabling any network pairing (BASP virtual adapter) in the iSCSI fabric, the backups have been running smoothly (apart from other glitches in the system, caused by other issues).

-So i'm still straightening out the kinks in the backup system. But at least this rather severe problem is now history.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: iSCSI link failing

Post by Vitaliy S. »

David, thank you for this update. I've never heard about Broadcom drivers issues...
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: iSCSI link failing

Post by Gostev »

Nice troubleshooting effort... impressive. Thanks for taking time to post the update!
Post Reply

Who is online

Users browsing this forum: brodyk and 262 guests