Processing rates on reverse incrementals

[PNWMtnBiker]

What are other folks are getting for their processing rates on reverse incremental jobs?

I'm currently halfway through migrating my backups into Veeam with 98 vm's so far and another 100 to go.

The initial fulls were between 150 MB/s to 500 MB/s depending on the SAN activity at the time, plenty fast enough in my opinion.

On the incremental's I get an average processing rate of 7 MB/s. They are all reverse incremental and I'm about a month in since the initial full. The primary bottleneck on almost all the jobs now says the target.

Target is a rack server with 12 disk array in RAID 6. I did make separate folders and a repository for each to get multiple streams going.

Just trying to get a feel for what speeds I should be expecting and any tuning suggestions.

[foggy]

Jon, transformation performed during each incremental job run with reverse incremental backup mode is a very I/O intensive operation, so it is expected that target is the bottleneck. Changing RAID level to the one with less I/O impact (RAID5 or RAID10, which has the least impact) should provide better performance.

[PNWMtnBiker]

I understand it's I/O intensive, it was already crossing my mind to redo the target with RAID5 instead. I'm working on setting up some performance capturing for the target today so I know if I'm really over doing it on the storage or not.

Mostly I'm checking if it's more common to see higher numbers or not and whether I should retune the backups.

[foggy]

Slower speed of the incremental run comparing to the full run is expected, however I would say that 7 MB/s is still too slow. How many tasks do you run concurrently on this repository?

[PNWMtnBiker]

Running 32 tasks concurrently.

[dellock6]

You need to check if your storage can handle 32 streams at the same time, otherwise you are killing it.
If the storage has any way to measure the throughput, I would start with few cuncurrent jobs by limiting them in the repository configuration, and from here increase little by little the limit; after some tests you will find the "best spot" for it.
Or if you can run I/O tests against it, check for the "I/O queue" parameters, since one job only writes one block at a time, its I/O queue is 1. A tests with different queue values would give you a better picture of the best value for that specific storage.

[PNWMtnBiker]

Thanks Luca, I appreciate the advice. I've cut the streams way back, from 32 to 4 and will watch the next run which is tomorrow evening.
The storage is a windows 2012 server with direct attached. I'll run some I/O benchmarks on it tomorrow morning.

Congrats on being #1 Veeam certified too

[dellock6]

Thanks Honestly, in my position it would have been a total shame to not pass the exam among the first ones!

Let us know what you find with your tests, probably you can go above 4 streams and the sweeet spot is something in between.

Luca.

[PNWMtnBiker]

Here's how it went when I dropped the maximum current tasks for the repository down for last night. I set each repository at 2, very low.

This is one target with separate folders and a repository created for each folder.

...and what happened?
There was no change at all in processing speed, it stayed right between 6 and 7 MB/s.
The backups were just taking longer since less of them were going simultaneously. There were 8 vm's going simultaneous on four repositories, so two each.

I was also watching resource monitor on the target while the backup was running. It was doing ~45 MB/s Disk I/O with 3.0 of Disk Queue Length.

So, while the backups were running I changed the maximum current tasks to 8 on the four repositories and watched the effect on the target.

About 20 VM's kicked in and started backing up
Disk I/O floated between 70 to 100 MB/s.
Disk Queue Length went to 4
Processing speed stayed right at 6 and 7 MB/s.

It crossed my mind that we have SIOC enabled on the datastores and its at the default 30ms for congestion threshold. Could this be having a throttling effect? I think I'm going to turn it off for tonight and see what the result is.

An additional detail, if I start an backup job that is incremental but exclude all VM's but one, I'll get 35 to 50 MB/s processing rate and that seems decent. When I do a full initial backup job with all VM's in the job (12 vm's), I've seen it go as high as 500 MB/s but usually around 350 MB/s.

[PNWMtnBiker]

I just discovered the MTU setting on my storage server's 10Gbe NIC was wrong. I'm correcting this and we will see if that had anything to do with it. We do know our NetApp responds strangely if the MTU isn't exactly right.

[dellock6]

Uhm, yes SIOC (and any QoS system on the production storage) can impact backup speed, since we can introduce an additional latency on the storage, that in return is seen by sioc and it "reacts" to this situation. And also MTU can have an impact if there is no consistent configuration across the complete stream, end to end, because of retransmissions happening. I too heard in the past NetApp having problems with mis-configured 10G networks.
Good guesses, let us know.

[PNWMtnBiker]

I'm still working on resolving the MTU on the 10Gbe connection today. It is confirmed we were getting errors in the Nexus switch with this connection so there is certainly an issue.

I'll keep you guys updated ;0)

[PNWMtnBiker]

Update:

I found that my storage server is operating perfectly fine with it's 10Gbe NIC and is set correctly with jumbo frames and is transmitting them properly (wire shark) but found it is receiving very tiny packets back from the NetApp.

Ultimately it was decided to accelerate updating the firmware in our UCS environment so we can connect this server directly to the fiber interconnects as intended from the beginning.

So, I disabled the 10Gbe NIC and connected the 1Gb NIC on the regular switch network and it's running fine now. I spread apart the backup jobs a whole lot just to even out the load a little more.

I just ran a full backup and was getting 550MB/s processing rate. I'm hoping to see the incremental go around 35MB/s to 50MB/s.

[dellock6]

Hi Jon,
nice numbers for the full. 10:1 performance reduction maybe is too much from full (that is a forward even when running at the beginning of a reversed), but it all really depends on the underlying storage. Let's see how it goes.

Luca.

[PNWMtnBiker]

New update:

Backups are running well and across all the jobs I'm getting an average of 20MB/s and individually some are reaching 35MB/s. My target is no longer the bottleneck on any of my jobs, it's all on the source now.

Still on the 1Gb connection but everything is running good

[dellock6]

Nice!
Thanks Jon for coming back each time and update the thread! And we're happy you found a performance level you are happy with.

Luca.