Slow backup performance (source-san, target-san)

Availability for the Always-On Enterprise

Slow backup performance (source-san, target-san)

Veeam Logoby dignitamsick » Mon Apr 03, 2017 2:14 pm

Hi,

I have a 3PAR San which most of my VMs are housed

We make use of 3pars own snapshots for regular hourly snapshots which works great, we set the expiry on these to a couple of days

However i'm trying to just do 1 backup a day for all my clients (1 job per client) with 1 backup copy job for retention and i'm getting poor MB/s rates

Setup

3PAR SAN(Source)
IBM Storwize v7000(Target)
Server 2016 -REFS luns/disks with MPIO (2x8GB Fibre)
10GB Teeamed to 20GB network (irrelevent for this test)

If i do a backup to 1 of our synology shelves i get like 700MB/s (but then i either have to use active fulls or allow for long merges)
if i do a file copy to the IBM disks i get similar speeds so i know the disks can handle it

My backup jobs are getting no more than 100MB/s at best..

I've tried changing the block size as this resulted in better performance when doing disk tests but no joy on the job

Ticket is logged with Veeam just wondering if anyones had similar issues
dignitamsick
Novice
 
Posts: 5
Liked: never
Joined: Mon Apr 03, 2017 12:30 pm
Full Name: Michael Bradshaw

Re: Slow backup performance (source-san, target-san)

Veeam Logoby foggy » Mon Apr 03, 2017 3:46 pm

Hi Michael, what are the bottleneck stats for the affected job(s)?
foggy
Veeam Software
 
Posts: 14904
Liked: 1096 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Slow backup performance (source-san, target-san)

Veeam Logoby dignitamsick » Tue Apr 04, 2017 6:44 am

Target 93%

its a HP DL360 Gen8 with Fibre storage, MPIO enabled

The disks are capable of much higher throughput so doens't make much sense
dignitamsick
Novice
 
Posts: 5
Liked: never
Joined: Mon Apr 03, 2017 12:30 pm
Full Name: Michael Bradshaw

Re: Slow backup performance (source-san, target-san)

Veeam Logoby Shestakov » Tue Apr 04, 2017 8:52 am

Michael,
What backup method do you use?
What is the processing rate?
There is no jobs without bottleneck, if backup job requires lots of I/O operations on the repository, bottle = target is expected.
Thanks!
Shestakov
Veeam Software
 
Posts: 4951
Liked: 407 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Slow backup performance (source-san, target-san)

Veeam Logoby adruet » Tue Apr 04, 2017 9:56 am

Have you tried a backup on a NTFS volume on the same target and compare the results with the same job on the ReFS volume.

It is just to be sure you are not hitting ReFS related issues that you can checkout in this thread: post236549.html#p236549
adruet
Influencer
 
Posts: 22
Liked: 6 times
Joined: Wed Oct 31, 2012 2:28 pm
Full Name: Alex

Re: Slow backup performance (source-san, target-san)

Veeam Logoby dignitamsick » Tue Apr 04, 2017 3:32 pm

Backup method is Direct Storage

I'll try setting up a new lun/volume and present it as NTFS to see whether its any different

Tried 4k & 64K block sized luns

Really odd!
dignitamsick
Novice
 
Posts: 5
Liked: never
Joined: Mon Apr 03, 2017 12:30 pm
Full Name: Michael Bradshaw

Re: Slow backup performance (source-san, target-san)

Veeam Logoby foggy » Tue Apr 04, 2017 3:45 pm

dignitamsick wrote:Target 93%

Is it a full or incremental job run? Forward/reverse incremental?
foggy
Veeam Software
 
Posts: 14904
Liked: 1096 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Slow backup performance (source-san, target-san)

Veeam Logoby adruet » Wed Apr 05, 2017 8:13 am

dignitamsick wrote:Backup method is Direct Storage

I'll try setting up a new lun/volume and present it as NTFS to see whether its any different

Tried 4k & 64K block sized luns

Really odd!



So my guess is: you will be amazed with NTFS : https://forums.veeam.com/veeam-backup-replication-f2/refs-4k-horror-story-t40629-240.html#p236702
adruet
Influencer
 
Posts: 22
Liked: 6 times
Joined: Wed Oct 31, 2012 2:28 pm
Full Name: Alex

Re: Slow backup performance (source-san, target-san)

Veeam Logoby dignitamsick » Wed Apr 05, 2017 8:57 am

The problem i have with that is

I've seen REFS running at 400+ MB/S with 4 1gb ISCSI connections to a Synology so something is a miss

Think i'll just have to carry on tweaking / attempting to fix it because i really want to make use of the REFS benefits
dignitamsick
Novice
 
Posts: 5
Liked: never
Joined: Mon Apr 03, 2017 12:30 pm
Full Name: Michael Bradshaw

Re: Slow backup performance (source-san, target-san)

Veeam Logoby adruet » Wed Apr 05, 2017 1:42 pm

On my setup, I have been able to hit 1.8 GB/s on ReFS. That was with write only operations (first fulls).
My problems started when I had 5 to 10 jobs doing merges (synthetics) at the same time as backups running.
This implies a 50/50 ratio between reads and writes. And though ReFS and fast clone looked great on the paper, it is then that my ReFS repository has began to cripple. With backup window not being held during a night.
This is why I am in the process of migrating back to NTFS, and I would be very interested that you share your results comparing the same type of jobs on ReFS and NTFS.
But if you have very little concurrent jobs and tasks on the ReFS repository, it should work fine.
adruet
Influencer
 
Posts: 22
Liked: 6 times
Joined: Wed Oct 31, 2012 2:28 pm
Full Name: Alex

Re: Slow backup performance (source-san, target-san)

Veeam Logoby dignitamsick » Wed Apr 05, 2017 3:55 pm

What settings were oyu using to hit those numbers, and how was the repo setup, as CIFS or locally to the Windows server?

I have multiple physical proxies i could use as targets and could create multiple luns
dignitamsick
Novice
 
Posts: 5
Liked: never
Joined: Mon Apr 03, 2017 12:30 pm
Full Name: Michael Bradshaw

Re: Slow backup performance (source-san, target-san)

Veeam Logoby adruet » Mon Apr 10, 2017 9:02 am

All was ran on the very same hardware (repository configured as CiFS to use fast cloning feature), we have 4 servers like this:
- HP DL380 Gen9 with dual CPU Intel E5-2660 v4 2Ghz, 64 GB of RAM, raid 1 SSD for the OS
- Dual 10 Gbit network cards (HP 560FLR) supporting the offloading of SMB v3 (RDMA capabale)
- 2 x DAS HP D3700 with 25 x 1.8TB 12G 10K SAS disks configured as Raid 6 with HP p441 controller

To hit those numbers I used the diskspd utility to do some benchmarking before validating the final repository setup.
If you have never used it, Veeam has done a great job explaining the different parameters you can use to simulate the different type of Veeam backup jobs: https://www.veeam.com/kb2014

With write only on my setup could go as high as 1.8GB/s. But with transforms, which is a 50% reads / 50% writes, that could go as low as 250MB/s.

My issues with ReFS started when I had several jobs (6 to 10) running at the same time on the same repository. Backups would not finish during the night.

I have reformated everything to NTFS, I can now run 20 jobs in parallel without any performance issue, and backup file merge is faster with NTFS than with fastcloning when several jobs run at the same time.

So according to my experience, ReFS works good if you have no job concurrency, if it is the case, performances start to drop dramatically.

Have you tested with NTFS to see if it's better ?
adruet
Influencer
 
Posts: 22
Liked: 6 times
Joined: Wed Oct 31, 2012 2:28 pm
Full Name: Alex


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: benks, Bing [Bot], Google [Bot] and 30 guests