Isilon backup to SMB over multiple streams

[Jonathan]

We have an Isilon NAS cluster, which is to be backed up to a large Supermicro storage server. The official documentation refers to using multiple extents to scale up backup throughput (https://www.veeam.com/sys256). Given my hardware setup (below), how would I configure this? Create two different shares on the same target volume and add each as a separate SOBR extent? Will this make the backup job automatically use multiple source nodes? While we're at it, are there any other tweaks that would give me more throughput?

This freshly built setup is already working quite nicely, but at the current rate a full backup would take a little over 2 weeks. This was to be expected and it's quite acceptable. After the first full backup has finished, daily incrementals are expected to be at most 10TB.

Source: 8-node Isilon A200 cluster, 360/900TB used, 10Gb connectivity, mostly video content ranging from 500MB to 150GB per file.
Target: Supermicro storage server, 90x18TB SAS drives (capacity tier), 4x3.68TB NVME (performance tier), 2x7TB SSD (operating system), 2 x Xeon 6258R 28-core, 512GB RAM, 4x10Gbit ethernet (LACP dynamic load balancing). Disk are configured with Storage Spaces: NVME drives as 6TB mirrored performance tier, SAS drives as 1,270TB parity capacity tier.
Backup server: Dell R740, 2 x Xeon 5218 16-core, 256GB RAM, 2 x 10Gbit ethernet (LACP dynamic load balancing)

Performance monitor shows the NVME tier getting hit first, until the 6TB is full, then starts flowing over to the capacity tier. Performance for the first 6TB is around 350MB/s, then drops to about 230MB/s. The Isilon cluster shows only one node being used for this backup job. I know the target server can sustain much more than 350MB/s. While the backup job is running, I can start an SMB file transfer to the same share at 400MB/s without the backup job slowing down. So we know network isn't a bottleneck, nor target storage speed, and the cpu load on both the backup and target server are barely above idle.

Sidenote: I'm using Windows Server 2016 for the target server instead of 2019, because 2019 Storage Spaces gave me constant BSODs once the vdisk was hit with a decent amount of IO. Configuring mirrored ssd + parity hdd tiers was a pain in the behind as well, both using GUI and Powershell. This took a fair bit of tinkering in 2016 as well, but in the end it worked out fine. Honestly, you would expect a little more maturity by now.

[Gostev]

Assuming you backup over SMB protocol, the only way to scale is by adding additional file proxies. Due to SMB stack peculiarities, additional streams from the same SMB client don't help and you need to add physically separate SMB clients. You should see pretty linear performance improvement with additional file proxies.

When you eventually face a bottleneck on the target repository, then creation of additional SOBR extents in a form of general purpose servers is the way to go. We will use both storage AND compute resources of these additional nodes for what is effectively a distributed NOSQL database behind our NAS backup engine.

[Jonathan]

Thank you for answering my question, Anton. So if I get this straight, adding multiple extents to an SOBR will give us no performance benefits as long as we use just one backup proxy? Could I install the proxy role on the target server itself as well, so both the backup server and the target pull data from the Isilon cluster?

[Gostev]

That is correct. And yes, you can make any existing machine a file proxy, even if it already carries other Veeam roles.

[Jonathan]

Allright, so we can create multiple proxies. But where exactly is the 1:1 relationship between an SOBR extent and an Isilon cluster node configured?

The official documentation simply says:

Veeam Scale Out Backup Repository can distribute the load across the Isilon cluster via multiple Backup Repository Extents each defined to use a different cluster node DNS A record and path

Where do you define an extent to use a specific cluster node? I can define an affinity between an extent and a proxy, but I can't find an option to specify a source node.

[Dima P.]

Hello Jonathan,

But where exactly is the 1:1 relationship between an SOBR extent and an Isilon cluster node configured?

Scale-Out repository or it's extents do not access source data - all the backup reads or restore writes are performed by the proxies. Can you please share the link to the guide? Cant find it anywhere.

Update: Found it, this recommendation applies to the Isilon storage added as repository (in your case Isilon is the source).

[Jonathan]

Ah, I get it. That's a bit of a bummer. So I guess there's no way to make Veeam read from multiple Isilon nodes at once?

[Gostev]

I think you're confusing Isilon as source with Isilon as target.

[Jonathan]

I think you're confusing Isilon as source with Isilon as target.

Yes, that penny dropped after Dima’s reply. Although why anyone would use such an expensive system as a backup target is lost on me

[Jonathan]

Can I follow this up with another related question?

If I were to cut this server's storage up into smaller volumes:

SOBR extent 1: 12TB performance (NVME)
SOBR extent 2: 350TB capacity
SOBR extent 3: 350TB capacity
SOBR extent 4: 350TB capacity

Is it then possible for a single NAS backup job to first hit the performance tier and have the backup data offloaded from the performance tier to the capacity tier in the background? A single full backup will likely be around 360TB initially, but daily incrementals would perhaps fit within the performance tier.

The reason I'm asking is we've hit some problems configuring 90 spinning disks and 4 NVME drives as a single contiguous file system on one server. Storage Spaces 2019 turns out to be buggy as hell, hw RAID controllers usually don't scale upwards of 32 drives per array (and perhaps you wouldn't want them to either) and ZFS would present a steep learning curve and yet another operating sytem/distro to manage in our datacenter. So right now I'm looking for the most robust yet economical way to use these drives in Veeam B&R, preferably using Windows.

[Dima P.]

Hi Jonathan,

Capacity tier is not integrated with nas backups, so nas jobs can only land backup files to a performance tier but with scale-out repository the nas backup data is balanced across all available extents based on available free space.

If you need to send the retired file versions from the primary repository (could be SOBR) to another storage you can use archive repository in nas backup job settings.

[Jonathan]

with scale-out repository the nas backup data is balanced across all available extents based on available free space

Is this process parallel or serial? As in: backup data chunk A goes to extent 1, then chunk B to extent 2, then chunk C to extent 3 (one after the other, serial) or chunks A, B and C are written to extents 1,2 and 3 at the same time (parallel)? The latter would also presumably provide better performance.

[Dima P.]

Jonathan,

Parallel if you have multiple proxies. Thanks!

[Dream_On]

Is there any way to control, how many proxies to use? My Isilon SMB share backup job is using 3 proxies, but speed is only 50MB/s.

[Dima P.]

Hi Vladimir,

Sure. You can to tune the file share throttling via File Share Processing Settings > Backup I/O control. Likely your proxies are currently throttled to produce minimal impact on the source, that's why the performance is low.