Synthetic full slow, reverse incr. slow?

[Oletho]

I opened a support call, but would like some input from this forum also.

SR:
Synthetic full runs very, very slow.

I have two jobs and during this weekend from friday 5PM to sunday 7PM (50 hours) one job is at 45% and another at 16%.

I run virtual appliance mode with 6 vCPU and 8GB RAM. CPU is at 1-5% and disk activity under 5-10MB/s most of the time. Seldom peeks go to 40-50 MB/s

Daily incrementals takes less than 2 and 3 hours. I changed to forward incremental because reverse daily did not complete within the 14 hour backup window.

Any ideas?

Ole Thomsen

[Gostev]

Hi Ole, you must include support ID when posting about technical issues, as explained when you click New Topic. Based on the above data, backup server is doing nothing most of the time. Assuming your backup files are fine, the only possible cause for that, obviously, is slow target storage performance (or connection to it), which makes backup server wait most of the time for target storage to complete required operation. Where are your backups located?

[Oletho]

At that moment I had not received the ID, it is 5128420.

Storage is Qnap NAS, where NFS normally runs at least ~80MB read and ~50MB write.

Target bottleneck is what I also suspected, but disk queue length is only 1-5ms looking at disk activity monitor from within the appliance.

Just came to think of the performance graphs of the Veeam VM. They show a different picture for the NAS datastore- 40ms write and 25 ms read average latency. Average rate is around 4MB/s for both read and write.

NAS CPU and memory usage is low.

Thanks,
Ole Thomsen

[Gostev]

What is the exact QNAP model? How do you have QNAP NAS added to Veeam? Are you writing backups to a CIFS share?

[Oletho]

It is TS-809U-RP running RAID5 which seemed a good idea at the time. Now I wish it was RAID10

It is mounted in the vSphere cluster as an NFS share, with several VMDKs striped in Veeam Windows 2008R2 guest.

On the Qnap I see that read and write ratio is almost equal most of the time. Perhaps this is a challenge for the NAS, as my previous use almost always was either write or read with much better performance.

Ole Thomsen

[Yuki]

We use QNAP at client sites as well and are typically seeing around 25-50mb/s sustained write speed for Veeam. Even though Veeam server is not pegging at 100% cpu or memory we still don't hit the same rate when we do an SMB file copy (70-105Mb/s)

[Oletho]

Yuki, these are excellent numbers compared to mine. How do you present the Qnap NAS to Veeam?

I mount the Qnap as an NFS share to the ESX host and present VMDKs from that to the Veeam virtual appliance.

[Gostev]

Ole, try attaching SAN LUN to Veeam Backup server using iSCSI Software Initator, and see if it helps to increase performance due to writing directly to storage. I believe NFS performance is not very good when it has to go through ESX storage I/O stack. At least I have never seen good numbers reported. Thanks.

[Oletho]

Thanks Gostev. You might be right about direct attached LUN is better when using low-end storage like Qnap.

But for high grade storage that is not always the case. I have compared performance of host mounted NFS vs. 2008R2 vm software iSCSI on Oracle Sun Open Storage and Netapp. Host NFS was fastest, and that is why this is always my preferred setup.

I have ordered a spare Qnap for some testing.

[Gostev]

Oh, Windows 2008 and iSCSI is a separate story, you may need to tweak some settings to achieve good performance there. Check out sticky FAQ for more info... the difference in performance can be as large as 2 times with adjusted Windows settings comparing to out of box performance.