I just thought I'd post some of our findings regarding our slow restores:
We're running ESX 5.5 and Veeam 7R2. Our storage platforms are DELL EQL (4 arrays: 15k RAID10, 10k RAID10 and 2x7.2k RAID6) and a Nimble CS460G. Now we were testing restores from tape on a fairly smallish server here (1.3TB used of 4TB). We were restoring to a new datastore on the EQL SATA storage pool made up of two RAID6 arrays. We were getting about 15MB/s, which is pretty horrendous and it took 29hours to restore. Now that pool is fairly heavily used, but should be giving better performance than that. So we built a smaller VM of around 150GB and tested that - the results were pretty much the same.
We figured it might the storage pool and so we moved it to our 15k RAID10 storage pool. That got to the dizzy heights of 42MB/s. So, we're starting to panic now as it would take us over a month to restore from a full failure - not good. We decided to restore to the Nimble and got the dizzy heights of 80MB/s, not brilliant, but more acceptable. So, we started doing some deeper tests and noticed that our write rates to some VM's were also fairly poor, although it was hard to try and pin it down.
Anyway, I found this thread and started working through it - Veeam restores were consistently poor so we could use that as a measuring stick. We tried HotAdd and other mechanisms and found it to make no difference and were fairly convinced it was more global an issue than Veeam.
So, we got to the post by Yizhar about disabling VAAI functionality on the host. We run vSphere 5.5 Standard edition, so we don't have VAAI support even though our Nimble and EQL support it to differing degrees.
BOOM - our restore to the SATA pool went up to 90MB/s+, the 15K pool around 120MB/s and the Nimble clocked a restore of nearly 200MB/s which is amazing considering it's a large Sequential Write - the workload the Nimble hates chewing on the most.
So, we're still testing this on a spare host, but it appears that disabling VAAI has made a considerable difference to all our systems including restoration operations from Veeam. It looks like ESX has VAAI enabled by default, although I'm not sure what affect it has if you're not licensed to use it. Surely not licensed means disabled?
Anyway, it's made a big difference to us and we'll continue testing. I just wanted to say thanks for the post Yizhar!