Veeam 9.5U2 on ESXi6.5 blows up Windows I/O

VMware specific discussions

Veeam 9.5U2 on ESXi6.5 blows up Windows I/O

Veeam Logoby micheld » Thu Sep 14, 2017 9:22 pm

We build a 16 drives RAID-10 ESXi server to host a Windows 2016 VM that runs Veeam 9.5.0.1038 and drops the data to the V: drive which is a 6.5TB ReFS at 64K.

This setup will die during the backup job or just after the end where it will run the tape job.
The VM is still alive but the Windows I/O is down to 0. All attempts to work with the VM will hang forever until a reset.

We tried to limit the bandwidth to the V: drive, tried 1Gbps ports instead of the 10Gbps ports to no avail.

We see around 160MB/s of writes during the backups and 285MB/s of reads when the tape job starts and sends the data to the proxy on 10Gbps to an LTO-7 drive.

Since it is the Windows I/O stack that is blowing up and no ESXi errors are present, I looked at the V: drive and decided to change the removal policy of the VMware Virtual disk SCSI Disk Device from the default "Better Performance" to "Quick Removal". By doing this the Windows Caching is deactivated and for the first time the backup jobs on both sites worked to the end and the Windows servers where still available the next morning.

There must be an issue with ReFS caches and Veeam's way of handling its large backup files. There is now performance impact for us since the source is the bottleneck.

Not sure if any one else saw these issues.
micheld
Lurker
 
Posts: 1
Liked: never
Joined: Thu Sep 14, 2017 8:59 pm
Full Name: Michel Dionne

Re: Veeam 9.5U2 on ESXi6.5 blows up Windows I/O

Veeam Logoby DaveWatkins » Sun Sep 17, 2017 4:27 am

This is more than likely just the normal ReFS issues with file deletion we have all been seeing. There are multiple registry entries available from MS which mitigate this in conjunction with the recent OS patches.

A good starting point is here assuming you have the latest patch applied. Go back through that thread if you want to know the details

veeam-backup-replication-f2/refs-4k-horror-story-t40629-585.html#p254231
DaveWatkins
Expert
 
Posts: 274
Liked: 68 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: Veeam 9.5U2 on ESXi6.5 blows up Windows I/O

Veeam Logoby the_mentor » Thu Sep 21, 2017 9:10 pm

We had a ton of issues with ReFS despite using 64k and having the latest patches etc.
In the end we ended up reformatting back to NTFS since its a more proven file system without all the ReFS issues.
Maybe we'll try again in a year or so when ReFS becomes more mature.
-DeMentor
the_mentor
Enthusiast
 
Posts: 42
Liked: 5 times
Joined: Thu Jul 26, 2012 11:10 pm
Full Name: DeMentor


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 1 guest