All flash array, how can bottleneck be Source?

Availability for the Always-On Enterprise

Re: All flash array, how can bottleneck be Source?

Veeam Logoby Delo123 » Mon Sep 05, 2016 12:11 pm

But your Source is still the bottleneck right? 400/500 Megs /s could be 1GB/s or 2 or 10 so it still is a bottleneck...
Delo123
Expert
 
Posts: 348
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: All flash array, how can bottleneck be Source?

Veeam Logoby stevenrodenburg1 » Mon Sep 05, 2016 12:18 pm 2 people like this post

Sure. According to the way Veeam calculates it yes. But the VSAN is not really pushed that hard during backups. We have two proxies, processing 4 VMDK's in parallel ( 8 in all ).

My point is: there will always be "the slowest component dragging things down".
I'm perfectly satisfied with our system and in my mind, ignore that "negative information" I get fed every day.

This topic is as old as the day when Veeam introduced the "bottleneck display feature". People have been "worrying over nothing" ever since (well, not always, folks with performance issues have valid things to worry about).

This feature, though meant with the best intentions, has a "dark side" too ;-)
stevenrodenburg1
Expert
 
Posts: 115
Liked: 18 times
Joined: Tue May 31, 2011 9:11 am
Location: Switzerland
Full Name: Steven Rodenburg

Re: All flash array, how can bottleneck be Source?

Veeam Logoby dellock6 » Mon Sep 05, 2016 6:51 pm 1 person likes this post

I feel that at some point it would just be enough to replace the term "bottleneck" with something less negative, and magically all these "issues" would disappear :)
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: All flash array, how can bottleneck be Source?

Veeam Logoby chrisdearden » Mon Sep 05, 2016 6:55 pm

I like "rate limiting/determining step" from my Chemistry days, but its probably harder to explain.
chrisdearden
Expert
 
Posts: 1529
Liked: 225 times
Joined: Wed Jul 21, 2010 9:47 am
Full Name: Chris Dearden

Re: All flash array, how can bottleneck be Source?

Veeam Logoby dellock6 » Mon Sep 05, 2016 6:58 pm 1 person likes this post

Let's just flip the numbers in the percentage and call it "best performing components" :D
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: All flash array, how can bottleneck be Source?

Veeam Logoby stevenrodenburg1 » Mon Sep 05, 2016 7:01 pm

Good suggestions. Keep em coming :mrgreen:
stevenrodenburg1
Expert
 
Posts: 115
Liked: 18 times
Joined: Tue May 31, 2011 9:11 am
Location: Switzerland
Full Name: Steven Rodenburg

Re: All flash array, how can bottleneck be Source?

Veeam Logoby tsightler » Mon Sep 05, 2016 8:18 pm

I thought about suggesting we just call it "Utilization", the step with the highest utilization percentage would still technically be the bottleneck, but it wouldn't have such a negative connotation, similar to the way network, CPU, and memory are reported in percent of utilization. However, the problem with that is, if a component is performing poorly, and we're spending a lot of time waiting on it, it would still show high utitlizaton, which wouldn't really be the right term in that case.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: All flash array, how can bottleneck be Source?

Veeam Logoby daniel.farrelly » Tue Sep 06, 2016 3:40 pm

Our source "bottleneck" is NVMe flash. We routinely hit ~2TB/s during nightly incrementals.
daniel.farrelly
Novice
 
Posts: 9
Liked: 4 times
Joined: Mon Feb 29, 2016 5:16 pm
Full Name: Daniel Farrelly

Re: All flash array, how can bottleneck be Source?

Veeam Logoby Delo123 » Tue Sep 06, 2016 6:09 pm 1 person likes this post

2TB/s sounds quite much, what array are you using to archieve that? We have nvme source but " only" get up to 6GB/s...
Delo123
Expert
 
Posts: 348
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: All flash array, how can bottleneck be Source?

Veeam Logoby HendersonD » Tue Sep 06, 2016 7:58 pm

I agree that the term bottleneck to an IT professional means it is time to hunt for a solution. I am the original poster and am coming to the realization that Source in my case is not really a bottleneck even though Veeam reports it this way. My new Nimble all flash array is barely breaking a sweat during backups. It is capable of churning out a lot more IOPS and throughput than Veeam is asking it to.
HendersonD
Enthusiast
 
Posts: 57
Liked: 3 times
Joined: Sat Jul 23, 2011 12:35 am

Re: All flash array, how can bottleneck be Source?

Veeam Logoby daniel.farrelly » Tue Sep 06, 2016 9:31 pm

daniel.farrelly wrote:Our source "bottleneck" is NVMe flash. We routinely hit ~2TB/s during nightly incrementals.

Typo. Meant 2 gigs a sec.
daniel.farrelly
Novice
 
Posts: 9
Liked: 4 times
Joined: Mon Feb 29, 2016 5:16 pm
Full Name: Daniel Farrelly

Re: All flash array, how can bottleneck be Source?

Veeam Logoby nmdange » Tue Sep 06, 2016 10:45 pm

I think the problem with the stats is that it doesn't take into account time spent doing things other than IO. It would be nice if things like time it takes to build the VM list, take a snapshot, mount a snapshot on a proxy, do a merge, etc. were included to get a better idea of how much the source storage system is really the bottleneck.
nmdange
Expert
 
Posts: 193
Liked: 57 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: All flash array, how can bottleneck be Source?

Veeam Logoby Gostev » Tue Sep 06, 2016 11:47 pm

You can already see in the job log how much time all of the above-mentioned operations take comparing to actual data movement. Bottleneck analysis is for data movement only, exactly for the reason to be able to show you "how much the source storage system is really the bottleneck" in actual data processing, excluding preparations - length of which makes no difference anyway considering that usually, multiple VMs are being processed in parallel and data movement through backup infrastructure is continuous.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: All flash array, how can bottleneck be Source?

Veeam Logoby nmdange » Wed Sep 07, 2016 12:33 am

For certain stats like time spent taking a snapshot or mounting it on a proxy, you can only see it by looking at each VM separately. Thus if some VMs take a lot longer than normal to take a snapshot, there's no obvious way to see if the backup is taking longer because of this without checking every single VM one at a time.

What I've noticed is when there is very little change data per VM, the time spent doing things like taking snapshots actually becomes significant enough that the job as a whole takes longer. If I look at the throughput graph of incrementals, I often see a lot of gaps where no data movement is happening. On the rare occasion I've had to do full backups (e.g. after a CBT reset), the throughput graph shows a constant and very high data rate (500MB/s-1GB/s).

If I watch a backup as it runs, I can see when VMs are waiting or in the middle of taking a snapshot as opposed to actually backing up data. However it is harder to figure out after the fact because each VM just lists how much time it spent and it's not easy to match that up to gaps in the throughput graph. Watching live, I can see VMs spend time in "Resource not ready" state. For me it is always "Resource not ready: snapshot" because with Hyper-V you can only get 4 VMs at a time per CSV. This could also be the backup proxy or the repository depending on the max number of tasks configured there. Having some way to correlate the gaps in the throughput graph to what is causing the wait is what would be helpful.
nmdange
Expert
 
Posts: 193
Liked: 57 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: All flash array, how can bottleneck be Source?

Veeam Logoby HendersonD » Wed Sep 07, 2016 12:45 am

So things brings us full circle, how can Source be the bottleneck with an all flash array? In my case my backup is about 40 minutes but 27 minutes is spent indexing a large file server. If this indexing operation is not counted in the bottleneck analysis how is Source the bottleneck? Are the percentages relative to each other? In other words, more is happening on the data read side than the network, proxy, and target side therefore the bottleneck analysis says that Source is the bottleneck?
HendersonD
Enthusiast
 
Posts: 57
Liked: 3 times
Joined: Sat Jul 23, 2011 12:35 am

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot] and 17 guests