Backup of NAS, file shares, file servers and object storage.
Post Reply
Jonathan
Enthusiast
Posts: 81
Liked: 5 times
Joined: Oct 15, 2009 8:52 am
Contact:

Isilon backup to SMB over multiple streams

Post by Jonathan »

We have an Isilon NAS cluster, which is to be backed up to a large Supermicro storage server. The official documentation refers to using multiple extents to scale up backup throughput (https://www.veeam.com/sys256). Given my hardware setup (below), how would I configure this? Create two different shares on the same target volume and add each as a separate SOBR extent? Will this make the backup job automatically use multiple source nodes? While we're at it, are there any other tweaks that would give me more throughput?

This freshly built setup is already working quite nicely, but at the current rate a full backup would take a little over 2 weeks. This was to be expected and it's quite acceptable. After the first full backup has finished, daily incrementals are expected to be at most 10TB.

Source: 8-node Isilon A200 cluster, 360/900TB used, 10Gb connectivity, mostly video content ranging from 500MB to 150GB per file.
Target: Supermicro storage server, 90x18TB SAS drives (capacity tier), 4x3.68TB NVME (performance tier), 2x7TB SSD (operating system), 2 x Xeon 6258R 28-core, 512GB RAM, 4x10Gbit ethernet (LACP dynamic load balancing). Disk are configured with Storage Spaces: NVME drives as 6TB mirrored performance tier, SAS drives as 1,270TB parity capacity tier.
Backup server: Dell R740, 2 x Xeon 5218 16-core, 256GB RAM, 2 x 10Gbit ethernet (LACP dynamic load balancing)

Performance monitor shows the NVME tier getting hit first, until the 6TB is full, then starts flowing over to the capacity tier. Performance for the first 6TB is around 350MB/s, then drops to about 230MB/s. The Isilon cluster shows only one node being used for this backup job. I know the target server can sustain much more than 350MB/s. While the backup job is running, I can start an SMB file transfer to the same share at 400MB/s without the backup job slowing down. So we know network isn't a bottleneck, nor target storage speed, and the cpu load on both the backup and target server are barely above idle.

Sidenote: I'm using Windows Server 2016 for the target server instead of 2019, because 2019 Storage Spaces gave me constant BSODs once the vdisk was hit with a decent amount of IO. Configuring mirrored ssd + parity hdd tiers was a pain in the behind as well, both using GUI and Powershell. This took a fair bit of tinkering in 2016 as well, but in the end it worked out fine. Honestly, you would expect a little more maturity by now.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Gostev »

Assuming you backup over SMB protocol, the only way to scale is by adding additional file proxies. Due to SMB stack peculiarities, additional streams from the same SMB client don't help and you need to add physically separate SMB clients. You should see pretty linear performance improvement with additional file proxies.

When you eventually face a bottleneck on the target repository, then creation of additional SOBR extents in a form of general purpose servers is the way to go. We will use both storage AND compute resources of these additional nodes for what is effectively a distributed NOSQL database behind our NAS backup engine.
Jonathan
Enthusiast
Posts: 81
Liked: 5 times
Joined: Oct 15, 2009 8:52 am
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Jonathan »

Thank you for answering my question, Anton. So if I get this straight, adding multiple extents to an SOBR will give us no performance benefits as long as we use just one backup proxy? Could I install the proxy role on the target server itself as well, so both the backup server and the target pull data from the Isilon cluster?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Gostev »

That is correct. And yes, you can make any existing machine a file proxy, even if it already carries other Veeam roles.
Jonathan
Enthusiast
Posts: 81
Liked: 5 times
Joined: Oct 15, 2009 8:52 am
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Jonathan »

Allright, so we can create multiple proxies. But where exactly is the 1:1 relationship between an SOBR extent and an Isilon cluster node configured?

The official documentation simply says:
Veeam Scale Out Backup Repository can distribute the load across the Isilon cluster via multiple Backup Repository Extents each defined to use a different cluster node DNS A record and path
Where do you define an extent to use a specific cluster node? I can define an affinity between an extent and a proxy, but I can't find an option to specify a source node.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Dima P. »

Hello Jonathan,
But where exactly is the 1:1 relationship between an SOBR extent and an Isilon cluster node configured?
Scale-Out repository or it's extents do not access source data - all the backup reads or restore writes are performed by the proxies. Can you please share the link to the guide? Cant find it anywhere.

Update: Found it, this recommendation applies to the Isilon storage added as repository (in your case Isilon is the source).
Jonathan
Enthusiast
Posts: 81
Liked: 5 times
Joined: Oct 15, 2009 8:52 am
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Jonathan »

Ah, I get it. That's a bit of a bummer. So I guess there's no way to make Veeam read from multiple Isilon nodes at once?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Gostev »

I think you're confusing Isilon as source with Isilon as target.
Jonathan
Enthusiast
Posts: 81
Liked: 5 times
Joined: Oct 15, 2009 8:52 am
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Jonathan » 1 person likes this post

I think you're confusing Isilon as source with Isilon as target.
Yes, that penny dropped after Dima’s reply. Although why anyone would use such an expensive system as a backup target is lost on me :)
Jonathan
Enthusiast
Posts: 81
Liked: 5 times
Joined: Oct 15, 2009 8:52 am
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Jonathan »

Can I follow this up with another related question?

If I were to cut this server's storage up into smaller volumes:

SOBR extent 1: 12TB performance (NVME)
SOBR extent 2: 350TB capacity
SOBR extent 3: 350TB capacity
SOBR extent 4: 350TB capacity

Is it then possible for a single NAS backup job to first hit the performance tier and have the backup data offloaded from the performance tier to the capacity tier in the background? A single full backup will likely be around 360TB initially, but daily incrementals would perhaps fit within the performance tier.

The reason I'm asking is we've hit some problems configuring 90 spinning disks and 4 NVME drives as a single contiguous file system on one server. Storage Spaces 2019 turns out to be buggy as hell, hw RAID controllers usually don't scale upwards of 32 drives per array (and perhaps you wouldn't want them to either) and ZFS would present a steep learning curve and yet another operating sytem/distro to manage in our datacenter. So right now I'm looking for the most robust yet economical way to use these drives in Veeam B&R, preferably using Windows.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Dima P. »

Hi Jonathan,

Capacity tier is not integrated with nas backups, so nas jobs can only land backup files to a performance tier but with scale-out repository the nas backup data is balanced across all available extents based on available free space.

If you need to send the retired file versions from the primary repository (could be SOBR) to another storage you can use archive repository in nas backup job settings.

Image
Jonathan
Enthusiast
Posts: 81
Liked: 5 times
Joined: Oct 15, 2009 8:52 am
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Jonathan »

with scale-out repository the nas backup data is balanced across all available extents based on available free space
Is this process parallel or serial? As in: backup data chunk A goes to extent 1, then chunk B to extent 2, then chunk C to extent 3 (one after the other, serial) or chunks A, B and C are written to extents 1,2 and 3 at the same time (parallel)? The latter would also presumably provide better performance.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Dima P. »

Jonathan,

Parallel if you have multiple proxies. Thanks!
Dream_On
Influencer
Posts: 21
Liked: 1 time
Joined: Jan 19, 2022 1:30 am
Full Name: Vladimir Popov
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Dream_On »

Is there any way to control, how many proxies to use? My Isilon SMB share backup job is using 3 proxies, but speed is only 50MB/s.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Isilon backup to SMB over multiple streams

Post by Dima P. » 1 person likes this post

Hi Vladimir,

Sure. You can to tune the file share throttling via File Share Processing Settings > Backup I/O control. Likely your proxies are currently throttled to produce minimal impact on the source, that's why the performance is low.
Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests