Host-based backup of VMware vSphere VMs.
Post Reply
Vejete
Technology Partner
Posts: 65
Liked: 2 times
Joined: Apr 06, 2017 6:23 pm
Full Name: Jim Turner
Location: Edmond, OK
Contact:

Not getting desired number of parallel backup streams

Post by Vejete »

Hi everyone. I'm pulling out what little hair I have left. Here's the situation:

VMware environment. VBR 9.5 Ent+ backup server is a physical machine with 8 cores and 16 GB of RAM. It's also the proxy and repository gateway. Primary storage is 3PAR. Target storage is StoreOnce Catalyst. Transport is SAN on both sides. Parallel processing enabled. Storage snapshots enabled. Proxy set to read 8 snaps at a time. Repository set to write 8 streams at a time. We have a backup with two dozen VMs in it. When we run that backup, only four VMs show as active at any one time with the rest pending. Of those four, only three show any percentage of completion. The fourth sits at 0%. So really only three VMs being actively backed up when I would expect eight in parallel.

We've verified in the StoreOnce Catalyst FC settings that we have 64 devices per connection, so there's no shortage of available devices for the proxy. We're not getting any errors. The issue is that we simply aren't seeing the performance we should because we're not getting the desired number of parallel streams. I'm hoping I just took stupid pills this morning, and I'm missing something else that needs to be checked/changed. If we've told Veeam that it's okay to read 8 source streams at a time, and it's okay to write 8 target streams to the repository, and we have way more than 8 VMs in the backup, then why are we not seeing 8 parallel streams active in a running backup containing two dozen VMs?

Thanks for your help. If help isn't possible, please send alcohol - preferably Maker's Mark. ;-)
=====
Jim Turner, VMCE 2020
Master Technologist & BURA Evangelist
Hewlett Packard Enterprise
bdufour
Expert
Posts: 206
Liked: 41 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: Not getting desired number of parallel backup streams

Post by bdufour »

what are the bottleneck stats looking like for the 3 vms (that process)?
Vejete
Technology Partner
Posts: 65
Liked: 2 times
Joined: Apr 06, 2017 6:23 pm
Full Name: Jim Turner
Location: Edmond, OK
Contact:

Re: Not getting desired number of parallel backup streams

Post by Vejete »

Source was the "winner" by about 20 percentage points, IIRC.
=====
Jim Turner, VMCE 2020
Master Technologist & BURA Evangelist
Hewlett Packard Enterprise
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Not getting desired number of parallel backup streams

Post by Gostev »

Hi Jim,

Please remember to include the Veeam support case ID, as requested when you click New Topic. It really is mandatory when reporting any technical issues on these forums - no exceptions.

As for the problem, it should be clear from the debug log review - trying to guess is a big waste of time. Could be anything from something simple (each of those VMs have more than one disks) to more complex (scheduler is running into some "hidden" limits like max API connections count). Bottom line is, support will be able to say right away by just looking at the logs.

Thanks!
Vejete
Technology Partner
Posts: 65
Liked: 2 times
Joined: Apr 06, 2017 6:23 pm
Full Name: Jim Turner
Location: Edmond, OK
Contact:

Re: Not getting desired number of parallel backup streams

Post by Vejete »

Hello Gostev,

The cu said that Veeam support actually got them nowhere and the support engineer suggested they call HPE about the StoreOnce. I checked out the StoreOnce, and there is no reason it cannot ingest far more than it is in parallel. The Veeam support case ID is 03259156. Where would I find the debug log you reference, and does additional debug need to be enabled prior to the backup running?

Thanks,
Jim
=====
Jim Turner, VMCE 2020
Master Technologist & BURA Evangelist
Hewlett Packard Enterprise
bdufour
Expert
Posts: 206
Liked: 41 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: Not getting desired number of parallel backup streams

Post by bdufour »

Please post the bottleneck stats. Source would indicate vm storage as the problem here. Posting the full stats would paint a better picture.
JasonL
Veeam Software
Posts: 114
Liked: 9 times
Joined: Jan 17, 2011 4:04 pm
Full Name: Jason Leiva
Contact:

Re: Not getting desired number of parallel backup streams

Post by JasonL »

Hi Jim,

A couple of things come to mind - 1) how many virtual disks are on each VM?; 2) You said that storage integration is enabled, but can you confirm that Backup from Storage Snapshots is actually being used and it's not using SAN mode instead? I believe there is still a smaller limit on the number of concurrent VM snaps per datastore when in SAN mode vs BfSS; 3) While likely not the root issue here, our best practices for StoreOnce lists to disable repository load control with StoreOnce since we do that under the covers already based on the number of maximum connections available for each device. Also, as mentioned above, any support case #'s or stats windows would also be helpful to assist further.

Thanks,
Jason
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Not getting desired number of parallel backup streams

Post by foggy » 1 person likes this post

Vejete wrote: Jan 11, 2019 10:28 pm VBR 9.5 Ent+ backup server is a physical machine with 8 cores and 16 GB of RAM. It's also the proxy and repository gateway.
Hi Jim, seems this setup is just not capable of running more than 3 concurrent tasks, you're short both on CPU and RAM. Each task (VM disk) requires a separate core both on proxy and repository servers (so double them in your case, since they are running on the same server) plus 2 more cores for the proxy itself. You can also calculate RAM requirements summing up those for all Veeam B&R components running on the server (backup server, proxy, repository).
Vejete
Technology Partner
Posts: 65
Liked: 2 times
Joined: Apr 06, 2017 6:23 pm
Full Name: Jim Turner
Location: Edmond, OK
Contact:

Re: Not getting desired number of parallel backup streams

Post by Vejete »

Hi Foggy,

That makes sense. I'd like to give the cu concrete guidelines on aggregate sizing for their "one box does everything" implementation. The backup server and proxy requirements for CPU and RAM are easy to reference, but I've yet to find anything that spells out CPU and RAM requirements for a gateway writing to StoreOnce Catalyst. With other ISV backup apps, the Catalyst Client API requires 1 GHz of CPU per concurrent stream going to a store, but it may be one core per stream for VBR. Either way, RAM is still not addressed for the StoreOnce gateway.

Hoping your or some other kind soul can cite a reference to calculate core count and RAM required for a gateway writing low-bandwidth data to StoreOnce Catalyst.

Thanks,
Jim
=====
Jim Turner, VMCE 2020
Master Technologist & BURA Evangelist
Hewlett Packard Enterprise
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Not getting desired number of parallel backup streams

Post by foggy » 1 person likes this post

Hi Jim, aggregate sizing makes sense, I believe we should add it to documentation. Currently for the gateway requirements you can use those of repository server, since gateway replaces it for integrated targets.
Post Reply

Who is online

Users browsing this forum: Amazon [Bot], Lewpy, nvdwansem, tomo yasu and 78 guests