Backups fail with when using jumbo frames

[JaySt]

i'm having an issue with direct-san backups when using jumbo frames. I cannot really explain, maybe someone has some insights where to look further.
Veeam B&R 9.5 update 4
Nimble storage arrays
vSphere 6.7 update 1
10Gbit iSCSI network
DL380Gen10 Running WIndows server 2016 backup repository, connected to the 10Gbit iSCSI network.
Cisco Nexus 3548 switches, configured for Jumbo frames (confirmed and double checked that).

All hosts are running production workload just fine on the iSCSI network. All components are configured to use jumbo frames. The last one to be configured with jumbo frames was the physical repository server. I ran direct san backups before changing the MTU of the iSCSI adapters and everything went fine, extremely well actually. When i changed the MTU within windows to jumbo frames, the backups started to fail.

Code: Select all

Processing VM_redacted Error: Failed to open VDDK disk [[NIM-redacted] VM_redacted/VM_redacted_2.vmdk] ( is read-only mode - [true] )
Logon attempt with parameters [VC/ESX: [esx.redacted.local];Port: 443;Login: [veeam@vsphere.local];VMX Spec: [moref=vm-70];Snapshot mor: [snapshot-39835];Transports: [san];Read Only: [true]] failed because of the following errors:
Failed to open disk for read.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.

when i change back to MTU of 1500 on the iSCSI adapters, things go just fine again and really fast. I'm trying to understand what's going on here. Looking at the Veeam logs, i can see VDDK having trouble opening disks and reading them when jumbo frames are enabled. Windows eventviewer shows iscsi errors within the system log.

Code: Select all

Target sent an iSCSI PDU with an invalid opcode. Dump data contains the entire iSCSI header.

No such errors in the eventviewer when i change back to MTU 1500.

All ping test go just fine when i perform a ping with jumbo frame packetsizes (8000 for example). That proves a proper network connection i think. Switch shows all ports used with MTU 9216 (global setting).

Anyone got a suggestion for a next step? I'm not sure i need to bother Veeam support just yet, as i'm not sure it's veeam related.

For now, i'm just disabling jumbo frames on the repo, no worries there. I just can't explain it technically

[JaySt]

just found this one, pretty similar situation (also Nimble etc.):
https://www.experts-exchange.com/questi ... 7-153.html

Edit: good news. i can confirm the fix mentioned on page above!
In summary and maybe for future reference: The 10Gbit NIC feature "Recv Segment Coalescing (IPv4)" needs to set to disabled on the NICs used for iSCSI.

Looking at the amount of research-effort that was done for this by the person who reported the fix, i'm very happy i found this.

[Andreas Neufert]

Thanks for sharing.