-
- Influencer
- Posts: 10
- Liked: never
- Joined: Feb 14, 2011 8:09 am
- Full Name: David Keis
- Contact:
iSCSI link failing
Hi everyone.
I am writing to you, hoping you will be able to help me with the following problem (or at least point me in the right direction!).
The problem I am experiencing is, that the iSCSI link will fail on the Veeam backup server when I initiate a backup job.
Sometimes the backup job will run for a while before it fails, other times it seems to fail almost immediately after I start a backup job.
All backup jobs are configured to use direct SAN access.
When the iSCSI fails, the only way I can get the iSCSI link up again, is by restarting the server.
In the iSCSI Initiator I have 3 links defined (one for each volume in the SAN).
Only 2 links are used by Veeam; Volume 1 and Volume 2. Strangely it is only the connection to Volume 1 that seems to fail. The other links don’t seem to be affected, and backups from Volume 2 will work just fine with direct SAN access.
My setup is as follows:
Veeam backup 5.0 residing on a physical server with iSCSI connection through 2 stacked Dell PowerConnect 6224 switches, to a Dell EqualLogic PS4000 SAN.
Jumbo frames are enabled, and flow control set up in the switches.
Backup is taken from 2 vSphere hosts running VMware ESXi 4.1.0. Both hosts have redundant Gb iSCSI links to the switches, and redundant Gb Ethernet connections.
Backups are stored on a Synology RS810+. Throughput while backing up is roughly 70 MB/sec until direct SAN access fails on the Veeam backup server. After that it falls to roughly 35-40 MB/sec.
I hope that I have included enough information, and that you will be able to help me figure this one out!
EDIT: Images attached
I am writing to you, hoping you will be able to help me with the following problem (or at least point me in the right direction!).
The problem I am experiencing is, that the iSCSI link will fail on the Veeam backup server when I initiate a backup job.
Sometimes the backup job will run for a while before it fails, other times it seems to fail almost immediately after I start a backup job.
All backup jobs are configured to use direct SAN access.
When the iSCSI fails, the only way I can get the iSCSI link up again, is by restarting the server.
In the iSCSI Initiator I have 3 links defined (one for each volume in the SAN).
Only 2 links are used by Veeam; Volume 1 and Volume 2. Strangely it is only the connection to Volume 1 that seems to fail. The other links don’t seem to be affected, and backups from Volume 2 will work just fine with direct SAN access.
My setup is as follows:
Veeam backup 5.0 residing on a physical server with iSCSI connection through 2 stacked Dell PowerConnect 6224 switches, to a Dell EqualLogic PS4000 SAN.
Jumbo frames are enabled, and flow control set up in the switches.
Backup is taken from 2 vSphere hosts running VMware ESXi 4.1.0. Both hosts have redundant Gb iSCSI links to the switches, and redundant Gb Ethernet connections.
Backups are stored on a Synology RS810+. Throughput while backing up is roughly 70 MB/sec until direct SAN access fails on the Veeam backup server. After that it falls to roughly 35-40 MB/sec.
I hope that I have included enough information, and that you will be able to help me figure this one out!
EDIT: Images attached
-
- VP, Product Management
- Posts: 27345
- Liked: 2785 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: iSCSI link failing
Hello David,
The first thing I would do is to update firmware/drivers for all the controllers. In addition, could you please tell if you have any errors in the Windows Event log, that might be somehow related to iSCSI initiator issue?
Thanks!
The first thing I would do is to update firmware/drivers for all the controllers. In addition, could you please tell if you have any errors in the Windows Event log, that might be somehow related to iSCSI initiator issue?
Thanks!
-
- Influencer
- Posts: 10
- Liked: never
- Joined: Feb 14, 2011 8:09 am
- Full Name: David Keis
- Contact:
-
- VP, Product Management
- Posts: 27345
- Liked: 2785 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: iSCSI link failing
Yes, that event doesn't tell much...By the way, how many backup jobs accessing the same SAN LUN do you run concurrently?
-
- Influencer
- Posts: 10
- Liked: never
- Joined: Feb 14, 2011 8:09 am
- Full Name: David Keis
- Contact:
Re: iSCSI link failing
Yes, unfortunatly, the eventlog does not help much!
I currently have 14 backup jobs, scheduled to run once a day.
The jobs are executed by a PowerShell script that is triggered by the windows task scheduler.
Below is the script (and by the way, thanks to this forum for help creating the script).
As you can see the jobs are executed sequentially to prevent overloading the backup server / snapshot space used by the VM's
I currently have 14 backup jobs, scheduled to run once a day.
The jobs are executed by a PowerShell script that is triggered by the windows task scheduler.
Below is the script (and by the way, thanks to this forum for help creating the script).
As you can see the jobs are executed sequentially to prevent overloading the backup server / snapshot space used by the VM's
Code: Select all
###############################################
# PowerShell script til backup af VM
#
# Forfatter: DKE 2011-02-14
# Første version
#
###############################################
# indlæs Veeam kommando-plugin
Add-PSSnapin VeeamPSSnapIn
# backup job: dkvam-ad2
Get-VBRJob | where {$_.Name –eq "backup dkvam-ad2"} | Start-VBRJob
# backup job: dkvam-cimio1
Get-VBRJob | where {$_.Name –eq "backup dkvam-cimio1"} | Start-VBRJob
# backup job: dkvam-ctxman
Get-VBRJob | where {$_.Name –eq "backup dkvam-ctxman"} | Start-VBRJob
# backup job: dkvam-etikser
Get-VBRJob | where {$_.Name –eq "backup dkvam-etikser"} | Start-VBRJob
# backup job: dkvam-fileserv
Get-VBRJob | where {$_.Name –eq "backup dkvam-fileserv"} | Start-VBRJob
# backup job: dkvam-ftassetc
Get-VBRJob | where {$_.Name –eq "backup dkvam-ftassetc"} | Start-VBRJob
# backup job: dkvam-man3
Get-VBRJob | where {$_.Name –eq "backup dkvam-man3"} | Start-VBRJob
# backup job: dkvam-messerv
Get-VBRJob | where {$_.Name –eq "backup dkvam-messerv"} | Start-VBRJob
# backup job: dkvam-test1
Get-VBRJob | where {$_.Name –eq "backup dkvam-test1"} | Start-VBRJob
# backup job: dkvam-vcenter
Get-VBRJob | where {$_.Name –eq "backup dkvam-vcenter"} | Start-VBRJob
# backup job: dkvam-web01
Get-VBRJob | where {$_.Name –eq "backup dkvam-web01"} | Start-VBRJob
# backup job: dkvam-xa1
Get-VBRJob | where {$_.Name –eq "backup dkvam-xa1"} | Start-VBRJob
# backup job: dkvam-xaman
Get-VBRJob | where {$_.Name –eq "backup dkvam-xaman"} | Start-VBRJob
# backup job: mesaep_V01
Get-VBRJob | where {$_.Name –eq "backup mesaep_V01"} | Start-VBRJob
-
- VP, Product Management
- Posts: 27345
- Liked: 2785 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: iSCSI link failing
Everything seems fine from the backup organization process, so please double-check the firmware/drivers, should help. Let us know how it goes for you.
-
- Influencer
- Posts: 10
- Liked: never
- Joined: Feb 14, 2011 8:09 am
- Full Name: David Keis
- Contact:
Re: iSCSI link failing
Hi again.
I have found the cause.
On the Veeam backup server, the network cards handling the iSCSI connections are from Broadcom.
Apparently the Broadcom driver does not handle the hardware offload very well, resulting in these errors.
By disabling the hardware offload for
- "IPv4 Large Send Offload"
- "TCP Connection Offload (IPv4)"
and also disabling any network pairing (BASP virtual adapter) in the iSCSI fabric, the backups have been running smoothly (apart from other glitches in the system, caused by other issues).
-So i'm still straightening out the kinks in the backup system. But at least this rather severe problem is now history.
I have found the cause.
On the Veeam backup server, the network cards handling the iSCSI connections are from Broadcom.
Apparently the Broadcom driver does not handle the hardware offload very well, resulting in these errors.
By disabling the hardware offload for
- "IPv4 Large Send Offload"
- "TCP Connection Offload (IPv4)"
and also disabling any network pairing (BASP virtual adapter) in the iSCSI fabric, the backups have been running smoothly (apart from other glitches in the system, caused by other issues).
-So i'm still straightening out the kinks in the backup system. But at least this rather severe problem is now history.
-
- VP, Product Management
- Posts: 27345
- Liked: 2785 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: iSCSI link failing
David, thank you for this update. I've never heard about Broadcom drivers issues...
-
- Chief Product Officer
- Posts: 31749
- Liked: 7253 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: iSCSI link failing
Nice troubleshooting effort... impressive. Thanks for taking time to post the update!
Who is online
Users browsing this forum: AdsBot [Google], Bing [Bot] and 121 guests