Discussions specific to the VMware vSphere hypervisor
Post Reply
Vejete
Technology Partner
Posts: 27
Liked: 1 time
Joined: Apr 06, 2017 6:23 pm
Full Name: Jim Turner
Location: Edmond, OK
Contact:

Not getting desired number of parallel backup streams

Post by Vejete » Jan 11, 2019 10:28 pm

Hi everyone. I'm pulling out what little hair I have left. Here's the situation:

VMware environment. VBR 9.5 Ent+ backup server is a physical machine with 8 cores and 16 GB of RAM. It's also the proxy and repository gateway. Primary storage is 3PAR. Target storage is StoreOnce Catalyst. Transport is SAN on both sides. Parallel processing enabled. Storage snapshots enabled. Proxy set to read 8 snaps at a time. Repository set to write 8 streams at a time. We have a backup with two dozen VMs in it. When we run that backup, only four VMs show as active at any one time with the rest pending. Of those four, only three show any percentage of completion. The fourth sits at 0%. So really only three VMs being actively backed up when I would expect eight in parallel.

We've verified in the StoreOnce Catalyst FC settings that we have 64 devices per connection, so there's no shortage of available devices for the proxy. We're not getting any errors. The issue is that we simply aren't seeing the performance we should because we're not getting the desired number of parallel streams. I'm hoping I just took stupid pills this morning, and I'm missing something else that needs to be checked/changed. If we've told Veeam that it's okay to read 8 source streams at a time, and it's okay to write 8 target streams to the repository, and we have way more than 8 VMs in the backup, then why are we not seeing 8 parallel streams active in a running backup containing two dozen VMs?

Thanks for your help. If help isn't possible, please send alcohol - preferably Maker's Mark. ;-)
=====
Jim Turner, VMCE
Master Technologist & BURA Evangelist
Hewlett Packard Enterprise

bdufour
Expert
Posts: 123
Liked: 19 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: Not getting desired number of parallel backup streams

Post by bdufour » Jan 11, 2019 10:41 pm

what are the bottleneck stats looking like for the 3 vms (that process)?

Vejete
Technology Partner
Posts: 27
Liked: 1 time
Joined: Apr 06, 2017 6:23 pm
Full Name: Jim Turner
Location: Edmond, OK
Contact:

Re: Not getting desired number of parallel backup streams

Post by Vejete » Jan 12, 2019 4:37 am

Source was the "winner" by about 20 percentage points, IIRC.
=====
Jim Turner, VMCE
Master Technologist & BURA Evangelist
Hewlett Packard Enterprise

Gostev
Veeam Software
Posts: 23333
Liked: 3033 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Not getting desired number of parallel backup streams

Post by Gostev » Jan 12, 2019 1:18 pm

Hi Jim,

Please remember to include the Veeam support case ID, as requested when you click New Topic. It really is mandatory when reporting any technical issues on these forums - no exceptions.

As for the problem, it should be clear from the debug log review - trying to guess is a big waste of time. Could be anything from something simple (each of those VMs have more than one disks) to more complex (scheduler is running into some "hidden" limits like max API connections count). Bottom line is, support will be able to say right away by just looking at the logs.

Thanks!

Vejete
Technology Partner
Posts: 27
Liked: 1 time
Joined: Apr 06, 2017 6:23 pm
Full Name: Jim Turner
Location: Edmond, OK
Contact:

Re: Not getting desired number of parallel backup streams

Post by Vejete » Jan 13, 2019 5:55 am

Hello Gostev,

The cu said that Veeam support actually got them nowhere and the support engineer suggested they call HPE about the StoreOnce. I checked out the StoreOnce, and there is no reason it cannot ingest far more than it is in parallel. The Veeam support case ID is 03259156. Where would I find the debug log you reference, and does additional debug need to be enabled prior to the backup running?

Thanks,
Jim
=====
Jim Turner, VMCE
Master Technologist & BURA Evangelist
Hewlett Packard Enterprise

bdufour
Expert
Posts: 123
Liked: 19 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: Not getting desired number of parallel backup streams

Post by bdufour » Jan 13, 2019 3:46 pm

Please post the bottleneck stats. Source would indicate vm storage as the problem here. Posting the full stats would paint a better picture.

JasonL
Veeam Software
Posts: 104
Liked: 9 times
Joined: Jan 17, 2011 4:04 pm
Full Name: Jason Leiva
Contact:

Re: Not getting desired number of parallel backup streams

Post by JasonL » Jan 14, 2019 7:27 pm

Hi Jim,

A couple of things come to mind - 1) how many virtual disks are on each VM?; 2) You said that storage integration is enabled, but can you confirm that Backup from Storage Snapshots is actually being used and it's not using SAN mode instead? I believe there is still a smaller limit on the number of concurrent VM snaps per datastore when in SAN mode vs BfSS; 3) While likely not the root issue here, our best practices for StoreOnce lists to disable repository load control with StoreOnce since we do that under the covers already based on the number of maximum connections available for each device. Also, as mentioned above, any support case #'s or stats windows would also be helpful to assist further.

Thanks,
Jason

foggy
Veeam Software
Posts: 17222
Liked: 1419 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Not getting desired number of parallel backup streams

Post by foggy » Jan 16, 2019 3:58 pm 1 person likes this post

Vejete wrote:
Jan 11, 2019 10:28 pm
VBR 9.5 Ent+ backup server is a physical machine with 8 cores and 16 GB of RAM. It's also the proxy and repository gateway.
Hi Jim, seems this setup is just not capable of running more than 3 concurrent tasks, you're short both on CPU and RAM. Each task (VM disk) requires a separate core both on proxy and repository servers (so double them in your case, since they are running on the same server) plus 2 more cores for the proxy itself. You can also calculate RAM requirements summing up those for all Veeam B&R components running on the server (backup server, proxy, repository).

Post Reply

Who is online

Users browsing this forum: Exabot [Bot] and 7 guests