Not getting desired number of parallel backup streams

[Vejete]

Hi everyone. I'm pulling out what little hair I have left. Here's the situation:

VMware environment. VBR 9.5 Ent+ backup server is a physical machine with 8 cores and 16 GB of RAM. It's also the proxy and repository gateway. Primary storage is 3PAR. Target storage is StoreOnce Catalyst. Transport is SAN on both sides. Parallel processing enabled. Storage snapshots enabled. Proxy set to read 8 snaps at a time. Repository set to write 8 streams at a time. We have a backup with two dozen VMs in it. When we run that backup, only four VMs show as active at any one time with the rest pending. Of those four, only three show any percentage of completion. The fourth sits at 0%. So really only three VMs being actively backed up when I would expect eight in parallel.

We've verified in the StoreOnce Catalyst FC settings that we have 64 devices per connection, so there's no shortage of available devices for the proxy. We're not getting any errors. The issue is that we simply aren't seeing the performance we should because we're not getting the desired number of parallel streams. I'm hoping I just took stupid pills this morning, and I'm missing something else that needs to be checked/changed. If we've told Veeam that it's okay to read 8 source streams at a time, and it's okay to write 8 target streams to the repository, and we have way more than 8 VMs in the backup, then why are we not seeing 8 parallel streams active in a running backup containing two dozen VMs?

Thanks for your help. If help isn't possible, please send alcohol - preferably Maker's Mark.

[bdufour]

what are the bottleneck stats looking like for the 3 vms (that process)?

[Vejete]

Source was the "winner" by about 20 percentage points, IIRC.

[Gostev]

Hi Jim,

Please remember to include the Veeam support case ID, as requested when you click New Topic. It really is mandatory when reporting any technical issues on these forums - no exceptions.

As for the problem, it should be clear from the debug log review - trying to guess is a big waste of time. Could be anything from something simple (each of those VMs have more than one disks) to more complex (scheduler is running into some "hidden" limits like max API connections count). Bottom line is, support will be able to say right away by just looking at the logs.

Thanks!

[Vejete]

Hello Gostev,

The cu said that Veeam support actually got them nowhere and the support engineer suggested they call HPE about the StoreOnce. I checked out the StoreOnce, and there is no reason it cannot ingest far more than it is in parallel. The Veeam support case ID is 03259156. Where would I find the debug log you reference, and does additional debug need to be enabled prior to the backup running?

Thanks,
Jim

[bdufour]

Please post the bottleneck stats. Source would indicate vm storage as the problem here. Posting the full stats would paint a better picture.

[JasonL]

Hi Jim,

A couple of things come to mind - 1) how many virtual disks are on each VM?; 2) You said that storage integration is enabled, but can you confirm that Backup from Storage Snapshots is actually being used and it's not using SAN mode instead? I believe there is still a smaller limit on the number of concurrent VM snaps per datastore when in SAN mode vs BfSS; 3) While likely not the root issue here, our best practices for StoreOnce lists to disable repository load control with StoreOnce since we do that under the covers already based on the number of maximum connections available for each device. Also, as mentioned above, any support case #'s or stats windows would also be helpful to assist further.

Thanks,
Jason

[foggy]

VBR 9.5 Ent+ backup server is a physical machine with 8 cores and 16 GB of RAM. It's also the proxy and repository gateway.

Hi Jim, seems this setup is just not capable of running more than 3 concurrent tasks, you're short both on CPU and RAM. Each task (VM disk) requires a separate core both on proxy and repository servers (so double them in your case, since they are running on the same server) plus 2 more cores for the proxy itself. You can also calculate RAM requirements summing up those for all Veeam B&R components running on the server (backup server, proxy, repository).

[Vejete]

Hi Foggy,

That makes sense. I'd like to give the cu concrete guidelines on aggregate sizing for their "one box does everything" implementation. The backup server and proxy requirements for CPU and RAM are easy to reference, but I've yet to find anything that spells out CPU and RAM requirements for a gateway writing to StoreOnce Catalyst. With other ISV backup apps, the Catalyst Client API requires 1 GHz of CPU per concurrent stream going to a store, but it may be one core per stream for VBR. Either way, RAM is still not addressed for the StoreOnce gateway.

Hoping your or some other kind soul can cite a reference to calculate core count and RAM required for a gateway writing low-bandwidth data to StoreOnce Catalyst.

Thanks,
Jim

[foggy]

Hi Jim, aggregate sizing makes sense, I believe we should add it to documentation. Currently for the gateway requirements you can use those of repository server, since gateway replaces it for integrated targets.