Slow backup performance (source-san, target-san)

[dignitamsick]

Hi,

I have a 3PAR San which most of my VMs are housed

We make use of 3pars own snapshots for regular hourly snapshots which works great, we set the expiry on these to a couple of days

However i'm trying to just do 1 backup a day for all my clients (1 job per client) with 1 backup copy job for retention and i'm getting poor MB/s rates

Setup

3PAR SAN(Source)
IBM Storwize v7000(Target)
Server 2016 -REFS luns/disks with MPIO (2x8GB Fibre)
10GB Teeamed to 20GB network (irrelevent for this test)

If i do a backup to 1 of our synology shelves i get like 700MB/s (but then i either have to use active fulls or allow for long merges)
if i do a file copy to the IBM disks i get similar speeds so i know the disks can handle it

My backup jobs are getting no more than 100MB/s at best..

I've tried changing the block size as this resulted in better performance when doing disk tests but no joy on the job

Ticket is logged with Veeam just wondering if anyones had similar issues

[foggy]

Hi Michael, what are the bottleneck stats for the affected job(s)?

[dignitamsick]

Target 93%

its a HP DL360 Gen8 with Fibre storage, MPIO enabled

The disks are capable of much higher throughput so doens't make much sense

[Shestakov]

Michael,
What backup method do you use?
What is the processing rate?
There is no jobs without bottleneck, if backup job requires lots of I/O operations on the repository, bottle = target is expected.
Thanks!

[adruet]

Have you tried a backup on a NTFS volume on the same target and compare the results with the same job on the ReFS volume.

It is just to be sure you are not hitting ReFS related issues that you can checkout in this thread: post236549.html#p236549

[dignitamsick]

Backup method is Direct Storage

I'll try setting up a new lun/volume and present it as NTFS to see whether its any different

Tried 4k & 64K block sized luns

Really odd!

[foggy]

Target 93%

Is it a full or incremental job run? Forward/reverse incremental?

[adruet]

Backup method is Direct Storage

I'll try setting up a new lun/volume and present it as NTFS to see whether its any different

Tried 4k & 64K block sized luns

Really odd!

So my guess is: you will be amazed with NTFS : veeam-backup-replication-f2/refs-4k-hor ... ml#p236702

[dignitamsick]

The problem i have with that is

I've seen REFS running at 400+ MB/S with 4 1gb ISCSI connections to a Synology so something is a miss

Think i'll just have to carry on tweaking / attempting to fix it because i really want to make use of the REFS benefits

[adruet]

On my setup, I have been able to hit 1.8 GB/s on ReFS. That was with write only operations (first fulls).
My problems started when I had 5 to 10 jobs doing merges (synthetics) at the same time as backups running.
This implies a 50/50 ratio between reads and writes. And though ReFS and fast clone looked great on the paper, it is then that my ReFS repository has began to cripple. With backup window not being held during a night.
This is why I am in the process of migrating back to NTFS, and I would be very interested that you share your results comparing the same type of jobs on ReFS and NTFS.
But if you have very little concurrent jobs and tasks on the ReFS repository, it should work fine.

[dignitamsick]

What settings were oyu using to hit those numbers, and how was the repo setup, as CIFS or locally to the Windows server?

I have multiple physical proxies i could use as targets and could create multiple luns

[adruet]

All was ran on the very same hardware (repository configured as CiFS to use fast cloning feature), we have 4 servers like this:
- HP DL380 Gen9 with dual CPU Intel E5-2660 v4 2Ghz, 64 GB of RAM, raid 1 SSD for the OS
- Dual 10 Gbit network cards (HP 560FLR) supporting the offloading of SMB v3 (RDMA capabale)
- 2 x DAS HP D3700 with 25 x 1.8TB 12G 10K SAS disks configured as Raid 6 with HP p441 controller

To hit those numbers I used the diskspd utility to do some benchmarking before validating the final repository setup.
If you have never used it, Veeam has done a great job explaining the different parameters you can use to simulate the different type of Veeam backup jobs: https://www.veeam.com/kb2014

With write only on my setup could go as high as 1.8GB/s. But with transforms, which is a 50% reads / 50% writes, that could go as low as 250MB/s.

My issues with ReFS started when I had several jobs (6 to 10) running at the same time on the same repository. Backups would not finish during the night.

I have reformated everything to NTFS, I can now run 20 jobs in parallel without any performance issue, and backup file merge is faster with NTFS than with fastcloning when several jobs run at the same time.

So according to my experience, ReFS works good if you have no job concurrency, if it is the case, performances start to drop dramatically.

Have you tested with NTFS to see if it's better ?