Virtual Win2003 local raid5 SYMMPI error

[aseesing]

Hello everyone,


Everytime Veeam 3.0.1 with NFS patch starts a job, the VM with Veeam installed gets practically locked up (the VM's console or RDP session is extemely slow in respons) and creates entrys in the windows systemlog. I've searched for an explaination but besides several not so usefull workarounds I was unable to fix this problem. When Veeam runs a backup with best compression enabled, the chance this VM will crash rises with about 30% every hour resulting with a windows crash after at least 3 hours after the job started. This happens only with Veeam Backup. So far other high I/O demanding applications do not lock up the console. Veeam is running using this configuration:

Hardware backupserver:
Dell PowerEdgde 2900
CPU Xeon E5440 quadcore 2,83GHz
4GB RAM
PERC6/i 256MB Raid controller SAS disks:
1+2 = RAID-1 2x63GB = 63GB
3-8 = RAID-5 64KB WriteBack 6x1TB ~= 5TB = vDIsk1 2TB + vDisk2 2TB + vDisk3 558GB
2xBCM5708 GB NIC

Software:
ESXi 3.5 Update4 Dell Customized version (official release)\
One vSwitch, multiple portgroups
iSCSI storage with 4 paths holding the cluster V'MDK files
RAID1 holding 3i plus system/boot 20GB VMDK virtual Win2003 for Veeam
vDisk1 holding VMFS 2TB 8MB blocks with 1 persistant VCBSRV1.VMDK of 2TB
vDisk2 holding VMFS 2TB 8MB blocks with 1 persistant VCBSRV2.VMDK of 2TB
vDisk1 holding VMFS 558GB 4MB blocks with 1 persistant VCBSRV3.VMDK of 556GB

In Windows 2003 R2 SP2:
With diskmanager the 3 VMDK files combined as one logical spanned (JBOD alike) volume of 4,56RTB NTFS with 4KB clustersize (also tried 64KB) mounted as drive Z:
MS iSCSI initiator with no mutlipathing enabled but only one path configured.
The iSCSI LUN shows up as a healthy unknown volume.
VCB installed
Services for NFS installed (part of backup storage is used for ISO file sharing for ESX hosts)
Veeam Backup 3.0.1 with NFS patch installed

Job details:
Every configuration with Best Compression will eventualy result in a Windows 2003 crash with several mysterious error events writting in the windows systemlog:

(Took me about 80 minutes to get these without rebooting the VM The first 2 or 3 events are recorded for thousands of times within minutes)

Thanks

Code: Select all

Event Type:	Error
Event Source:	symmpi
Event Category:	None
Event ID:	9
Date:		5/28/2009
Time:		8:12:24 PM
User:		N/A
Computer:	VCBSRV
Description:
The device, \Device\Scsi\symmpi1, did not respond within the timeout period.
Data:
0000: 00 00 10 00 01 00 66 00   ......f.
0008: 00 00 00 00 09 00 04 c0   .......À
0010: 01 01 00 50 00 00 00 00   ...P....
0018: 00 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........
0028: 00 00 00 00 00 00 00 00   ........
0030: 00 00 00 00 07 00 00 00   ........


Event Type:	Error
Event Source:	symmpi
Event Category:	None
Event ID:	15
Date:		5/28/2009
Time:		8:12:20 PM
User:		N/A
Computer:	VCBSRV
Description:
The device, \Device\Scsi\symmpi1, is not ready for access yet.
Data:
0000: 0f 2a 18 00 01 00 6e 00   .*....n.
0008: 00 00 00 00 0f 00 04 c0   .......À
0010: 03 01 00 00 a3 00 00 c0   ....£..À
0018: 9d 8b 00 00 00 00 00 00   ‹......
0020: 00 00 00 00 00 00 00 00   ........
0028: 00 00 00 00 02 00 00 00   ........
0030: 00 00 00 00 00 00 00 00   ........
0038: 07 00 00 00 05 00 00 00   ........

Event Type:	Error
Event Source:	Disk
Event Category:	None
Event ID:	11
Date:		5/28/2009
Time:		8:12:20 PM
User:		N/A
Computer:	VCBSRV
Description:
The driver detected a controller error on \Device\Harddisk2.
Data:
0000: 04 08 68 00 01 00 ba 00   ..h...º.
0008: 00 00 00 00 0b 00 04 c0   .......À
0010: 03 01 00 00 00 00 00 00   ........
0018: 00 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........
0028: 8c 3a 18 00 00 00 00 00   Œ:......
0030: ff ff ff ff 07 00 00 00   ÿÿÿÿ....
0038: 40 00 00 05 00 00 02 00   @.......
0040: ff 20 0a 12 8c 03 20 40   ÿ ..Œ. @
0048: 00 00 01 00 78 00 00 00   ....x...
0050: 00 00 00 00 08 de 15 86   .....Þ.†
0058: 00 00 00 00 d8 43 11 86   ....ØC.†
0060: 10 fa 72 86 ef ea 89 28   .úr†ïê‰(
0068: 2a 00 28 89 ea ef 00 00   *.(䐕..
0070: 80 00 00 00 00 00 00 00   €.......
0078: 00 00 00 00 00 00 00 00   ........
0080: 00 00 00 00 00 00 00 00   ........
0088: 00 00 00 00 00 00 00 00   ........

Event Type:	Information
Event Source:	dmio
Event Category:	None
Event ID:	30
Date:		5/28/2009
Time:		8:12:20 PM
User:		N/A
Computer:	VCBSRV
Description:
dmio: Harddisk2 write error at block 680127215: status 0xc0000185
Data:
0000: 00 00 00 00 04 00 4a 00   ......J.
0008: 00 00 00 00 1e 00 05 40   .......@
0010: 00 00 00 00 00 00 00 00   ........
0018: 01 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........

Event Type:	Warning
Event Source:	dmio
Event Category:	None
Event ID:	37
Date:		5/28/2009
Time:		8:12:20 PM
User:		N/A
Computer:	VCBSRV
Description:
dmio: Disk Harddisk2 block 680127343 (mountpoint Z:): Uncorrectable write error
Data:
0000: 00 00 00 00 04 00 4a 00   ......J.
0008: 00 00 00 00 25 00 05 80   ....%..€
0010: 00 00 00 00 00 00 00 00   ........
0018: 01 00 00 00 00 00 00 00   ........
0020: 00 00 00 00 00 00 00 00   ........

[ssimakov]

Alex, what other applications with high I/O did you test and how? All these errors obviously indicate issues with accessing your iSCSI storage.

[aseesing]

Hello Mr. Simakov,

I tried several Disk performace tools and used 7zip to compress a 1GB random testfile using different compression setting and running copies at the same time. Copying from my ISCSI device to my local Raid-5 backup store goes with blazing speed. Well, when I manually set priority of VeeamAgent.exe on low, all my problems disapear. Any thought how to make this exe allways start with low prio? This would fix my problems cause when running with low prio I can use Best compression which is much better then NTFS compression which gives better compression then Optimal, at least for my VMs.

Thank you

[Gostev]

Guys, I suspect this may have to deal with MS iSCSI initiator not working properly when CPU load is high. We have never seen similar issues reported by customers using hardware iSCSI HBA, and I know many people like to use Best compression...

[aseesing]

Should I forward this problem to support?

[aseesing]

Hello Super Duper Guys @ Veeam!


Your 3.1 version on 64 bit Windows Storage Server with MS iSCSI rocks the planet! I installed the workgroup edition with 2GB RAM on the same ESXi and it its controlable, stable and no errors in the systemlog. Best compression inside ESXI with 2008.Chapeaux! Discard the whole problem with Windows 2003 and MS iSCSI.

Love it!

Sincerly

[Gostev]

Alex, I am glad everything works now. You sound very excited
So chances are high that I was right and there's indeed an issue with earlier version of MS software iSCSI initiator.

[aseesing]

I suppose it is possible though it is a bit strange. The MS iSCSI initiatior version was the last version published by Microsoft at 12/8/2008, 2.08 if I'm right. Could it be real that this version cannot handle the high stress? The weird part was the disk write errors only occured on the local storage, not in anyway the iSCSI. However, by replacing the OS with Windows Storage Server 2008 64 bit, and not using Veeam Backup 3.0.1 but 3.1 does make a difference.