Trying something new - no MB or concurrent limit on repo

[Encode7]

Going to see if I can rely on Backup IO control alone, looking forward to seeing the results!

Initial results look promising.

[Encode7]

Full backup jobs are now showing Target or Network as the bottleneck, as opposed the source.

Although there was more overall throughout, individual VMs took longer to backup as there was more parallel processing occurring (which will keep snapshots open longer unless you have array integration).

Now to see what happens when upgrade the target next week.

[Gostev]

Yes, quite expected results so far, as you are flooding your backup storage with too much I/O threads, so each particular one runs slower than before. Thanks for sharing, and please keep us posted on your findings!

[Encode7]

Time to do one job (incremental, 75gb, 7 VMs each with about 3 vDisks) is higher as opposed less compared to previous week & similar data volumes, unrestricted has 9 backup threads going so switching it down to 6 concurrent on the repository. Bottleneck varies between source, network and target (vs precious when it was always the source).

Backup IO control did not kick in, so latency ok, just not as efficient for writes.

I suspect the sweet spot for the current environment will be somewhere between 3 (original concurrent value on repository) and 6.

Would be interesting to run a load testing app against the target to see what the optimal concurrent job count was

[tsightler]

Would be interesting to run a load testing app against the target to see what the optimal concurrent job count was

Then you should read and follow this excellent whitepaper from our very own Luca Dell'Oca. The FIO profiles in that paper do a pretty good job emulating our workload and estimating number of threads and jobs.

[Encode7]

Amazing thanks Tom, shall post back results

[dellock6]

Thanks Tom for the mention

[Encode7]

FIO benchmark on the repository results time

Number of concurrent tasks
1: bw=61,634KB/s, iops=240, 80th percentile latency: 5ms
2: bw=91,916KB/s, iops=359, 80th percentile latency: 6ms
3: bw=95,711KB/s, iops=373, 80th percentile latency: 9ms
4: bw=97,833KB/s, iops=382, 80th percentile latency: 11ms
5: bw=98,375KB/s, iops=384, 80th percentile latency: 13ms
6: bw=98,819KB/s, iops=386, 80th percentile latency: 16ms
7: bw=99,187KB/s, iops=387, 80th percentile latency: 19ms

Interesting results (for this repository, others will be different), anything more than 5 concurrent tasks gives negligible throughput increase but increases latency. Explains also why having it unlimited (which would have resulted in 9 concurrent tasks due to the proxies having that capacity) wasn't the best setting for this environment.

(This was an Iomega PX6-300d connected via 1gb NIC, which is being switched out for something better).

Thanks Luca & Tom!

[Gostev]

This is due reaching to repository IOPS capacity.

[dellock6]

Ehm, I think it's more the queue depth, but anyway the intended goal of that paper and the tests is exactly this, so thanks so much Ian for the validation of the methodology.