Comprehensive data protection for all workloads
Post Reply
Oletho
Enthusiast
Posts: 67
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Synthetic full slow, reverse incr. slow?

Post by Oletho »

I opened a support call, but would like some input from this forum also.

SR:
Synthetic full runs very, very slow.

I have two jobs and during this weekend from friday 5PM to sunday 7PM (50 hours) one job is at 45% and another at 16%.

I run virtual appliance mode with 6 vCPU and 8GB RAM. CPU is at 1-5% and disk activity under 5-10MB/s most of the time. Seldom peeks go to 40-50 MB/s

Daily incrementals takes less than 2 and 3 hours. I changed to forward incremental because reverse daily did not complete within the 14 hour backup window.


Any ideas?

Ole Thomsen
Gostev
Chief Product Officer
Posts: 31812
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Synthetic full slow, reverse incr. slow?

Post by Gostev »

Hi Ole, you must include support ID when posting about technical issues, as explained when you click New Topic. Based on the above data, backup server is doing nothing most of the time. Assuming your backup files are fine, the only possible cause for that, obviously, is slow target storage performance (or connection to it), which makes backup server wait most of the time for target storage to complete required operation. Where are your backups located?
Oletho
Enthusiast
Posts: 67
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: Synthetic full slow, reverse incr. slow?

Post by Oletho »

At that moment I had not received the ID, it is 5128420.

Storage is Qnap NAS, where NFS normally runs at least ~80MB read and ~50MB write.

Target bottleneck is what I also suspected, but disk queue length is only 1-5ms looking at disk activity monitor from within the appliance.

Just came to think of the performance graphs of the Veeam VM. They show a different picture for the NAS datastore- 40ms write and 25 ms read average latency. Average rate is around 4MB/s for both read and write.

NAS CPU and memory usage is low.

Thanks,
Ole Thomsen
Gostev
Chief Product Officer
Posts: 31812
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Synthetic full slow, reverse incr. slow?

Post by Gostev »

What is the exact QNAP model? How do you have QNAP NAS added to Veeam? Are you writing backups to a CIFS share?
Oletho
Enthusiast
Posts: 67
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: Synthetic full slow, reverse incr. slow?

Post by Oletho »

It is TS-809U-RP running RAID5 which seemed a good idea at the time. Now I wish it was RAID10 :-)

It is mounted in the vSphere cluster as an NFS share, with several VMDKs striped in Veeam Windows 2008R2 guest.

On the Qnap I see that read and write ratio is almost equal most of the time. Perhaps this is a challenge for the NAS, as my previous use almost always was either write or read with much better performance.

Ole Thomsen
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Synthetic full slow, reverse incr. slow?

Post by Yuki »

We use QNAP at client sites as well and are typically seeing around 25-50mb/s sustained write speed for Veeam. Even though Veeam server is not pegging at 100% cpu or memory we still don't hit the same rate when we do an SMB file copy (70-105Mb/s)
Oletho
Enthusiast
Posts: 67
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: Synthetic full slow, reverse incr. slow?

Post by Oletho »

Yuki, these are excellent numbers compared to mine. How do you present the Qnap NAS to Veeam?

I mount the Qnap as an NFS share to the ESX host and present VMDKs from that to the Veeam virtual appliance.
Gostev
Chief Product Officer
Posts: 31812
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Synthetic full slow, reverse incr. slow?

Post by Gostev »

Ole, try attaching SAN LUN to Veeam Backup server using iSCSI Software Initator, and see if it helps to increase performance due to writing directly to storage. I believe NFS performance is not very good when it has to go through ESX storage I/O stack. At least I have never seen good numbers reported. Thanks.
Oletho
Enthusiast
Posts: 67
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: Synthetic full slow, reverse incr. slow?

Post by Oletho »

Thanks Gostev. You might be right about direct attached LUN is better when using low-end storage like Qnap.

But for high grade storage that is not always the case. I have compared performance of host mounted NFS vs. 2008R2 vm software iSCSI on Oracle Sun Open Storage and Netapp. Host NFS was fastest, and that is why this is always my preferred setup.

I have ordered a spare Qnap for some testing.
Gostev
Chief Product Officer
Posts: 31812
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Synthetic full slow, reverse incr. slow?

Post by Gostev »

Oh, Windows 2008 and iSCSI is a separate story, you may need to tweak some settings to achieve good performance there. Check out sticky FAQ for more info... the difference in performance can be as large as 2 times with adjusted Windows settings comparing to out of box performance.
Post Reply

Who is online

Users browsing this forum: eblack, Google [Bot] and 133 guests