Veeam NAS backup best practices with backup from Dell PowerScale/Isilon Smart Connect

[Andreas Neufert]

Veeam NAS backup best practices with backup from Dell PowerScale/Isilon Smart Connect
Updated on June 6. 2024 to reflect product changes.

Note: This guide does NOT describe the usage of PowerScale/Isilon as a backup target!

Dell PowerScale (old name Isilon) is a OneFS (Software name) grid-based filer. It uses server-like hardware with local disks to build a filesystem across the nodes and deliver all capacity within a single namespace to the users.

The system was built to deliver unstructured data to a huge number of parallel users with mainly sequential read/write requests. A single connection will usually get from DNS a node IP of the grid cluster to work with, and the load is distributed for many connections throughout all nodes by DNS-based load balancing. The backup (and regular client) throughput of a single node is limited; therefore, the backup load must be distributed across nodes.

Veeam NAS backup without Dell Smart Connect:
The Veeam Backup server works with the share’s given IP address or DNS name, and all (data mover and scan) NAS Proxy Servers will work with this single IP address.
- Even when multiple Proxies are used, Veeam will likely use the same storage node for all actual backup streams.
- Usage of multiple nodes is only the case if multiple shares are added to Veeam with different DNS names so that Veeam gets a chance during DNS resolution to get a different node IP address from DNS (round robin) for each share.

So it is highly recommended to configure Dell Smart Connect.

Veeam NAS backup with Dell Smart Connect integration:

Dell Smart Connect design:
Dell Smart Connect will be integrated into customers’ DNS system, where the DNS server delegates a cluster DNS name to the PowerScale integrated DNS server. This Smart Connect DNS server alters the resolution of the cluster DNS name to different storage nodes for load distribution. By using DNS resolution the clients that access the data get different IP addresses from different nodes to work with. Depending on Dell licensing, there can be load balancing based on Round Robin and CPU/Network Throughput/Connection Count. IP addresses in the storage cluster can be distributed manually or automatically to storage nodes and a single node can have multiple IP addresses with and without failover between nodes.

Note: SmartConnect is usually set up to best serve the regular production workload. Altering Smart Connect for backup would require coordination with the business owner of the system, and changes are usually not possible. This means that the backup system needs to work around potentially the non ideal load distribution.

The following steps illustrate how Veeam works with Smart Connect:
1. The storage SmartConnect service IP (SSIP) will be added as Veeam Storage to Veeam Backup & Replication.
2. NAS Proxies are rolled out (distribution of load). The more Proxies the better as Veeam distributes rescan and backup load throughout the NAS Proxies. We recommend adding as many general-purpose backup proxies as many Dell Powerscale storage nodes you plan to leverage for back up.
3. NAS backup Jobs need to be created, and a share from the storage can be selected from the list.
4. When the NAS Backup Job starts, each NAS Proxy will reach out to DNS to get the IP address for the cluster SmartConnect DNS name individually. Veeam Proxies will get different node IPs or not depending on the Smart Connect configuration. It can happen that SmartConnect is not able or not configured to change the IP address in their load balancing fast enough for our backup workload and all Veeam Proxies that start backup nearly at same time will get the same node IPs to work with. Please find below how to address this situation.

Best Practices implementation:

1. Enable Smart Connect on the Storage and integrate into DNS.
2. Add the Storage with the Smart Connect management IP (SSIP) to the Veeam Backup Server (Smart Connect usage is detected, and Veeam checks the DNS names on each Proxy instead of centrally on the backup server).
3. Roll out NAS Proxies at a high count. We recommend adding as many general-purpose backup proxies as Dell Powerscale storage nodes you plan to leverage for backup. These nodes can be used to backup multiple PowerScale systems and clusters.
4. Now run the first backup tests and check if Smart Connect is fast enough with IP address load balancing and the right amount of storage nodes are used for backup. If the load is not distributed correctly, you need to distribute the load in the following way:
Option a: Update Veeam to 12.1.1.56 or 12.1.2.172 and open a support ticket to get the fix 716019 (for issue 704170). This fix will delay the start of the processing on each Proxy for some seconds to allow Smart Connect to give Veeam different IP addresses.
Option b: Update Veeam to 12.2 when available. It has the in Option a mentioned change by default implemented.
Option c: Distribute the load manually. For this, edit the hosts file of each general-purpose backup proxy and map the cluster FQDN name to a storage node IP address. For each proxy, use a different storage node IP.
5. (Optionally) Enable Dynamic IP allocation on the PowerScale for IP fallback processing in case a PowerScale node goes down).

How to access the Smart Connect settings on PowerScale:
Go to: Cluster Management – Network Configuration
At the subnets you can view/edit the SmartConnect Service IP SSIP
A Zone Name (DNS name) can be configured that later needs to be added to the regular DNS server as a forward zone.
As well a pool of IP addresses and the static/dynamic IP node allocation can be configured there.

[Andreas Neufert]

One clarification:
The more Proxies you use, likely the fewer resources (RAM, CPU) you need per Proxy. Please check on regular sizing guidelines and contact your Veeam SE/SA for sizing assistance.

[Andreas Neufert]

Second clarification:
As the best practice states, that we should create as many proxies as nodes are in the system, does this as well count for high node counts?

The answer depends a bit on how you implement the system, their east-west network, IO and throughput capabilities and how many shares you backup from that cluster at the same time. As well what load you want to allow for backup on the system (stealing it from your clients for the time of backup).

For each share that you want to backup at the same time, you can calculate the throughput that you want to put on the PowerScale system. Then see how many Veeam NAS proxies would lead to that throughput (check on network and PowerScale node limitations as well) and calculate from there how many proxies you should use. Then check again on how many shares you want to backup with that same speed and calculate from there the amount of RAM and CPU resources needed for it per Proxy (check again on if network or the nodes become a bottleneck).
See the best practices guide for NAS sizing for details:
https://bp.veeam.com/vbr/2_Design_Struc ... oxies.html


Sample:
You have 100 nodes and all networks are 100GbE (to just say that network will not be the bottleneck here in that theoretical example).
Let´s say your selected PowerScale node type can deliver under any circumstances 150MB/s random IO read throughput (ask documentation or Dell for what your storage nodes can provide realistically).
You want to backup 4 shares parallel each with 1GB/s which means you overall need 4GB/s.

4GB/s : 150MB/s = 28 Nodes

Then let´s calculate on what the Proxies would look like for 4 parallel share backups and how many resources would mean how many throughput they can handle.

Each Task slot on the Proxy process 1 share at a time.
In our configuration, we want to process 4 shares at a time, which means 4 task slots per Proxy.
Each Proxy Task Slot uses 2 Cores 4GB RMAN (see above link to the best practices documentation) which means we need 8 Cores 16GB RAM per task slot for all 4 tasks.
Now what is the theoretical speed the 4 task slots and 8 cores can provide:
Each Proxy Core can process 100MB/s (see best practices documentation link above).
Each Proxy could process 8x 100MB/s = 800MB/s
4 GB/s : 800GB/s = 6 Proxies = Read from 6 nodes based on the original post here.

When you compare the two number results from above, then the node throughput is the bottleneck in this sample. Which means, that you need 28 Proxies. Each Proxy with 8 Cores and 16GB RAM and a 4 task slot setting.

Second scenario with different node types:
Now let's check if the node type you use could provide 850MB/s on random IO read throughput.
4GB/s (that you want to have... see above) : 850MB/s = 5 nodes
Now let´s compare again with what we have calculated above a Proxies could handle. You can see that the bottleneck shifted from node to the Proxy. In this case you need at least 6 Proxies to deliver the needed speed.

Remember all this is under ideal lab conditions when only backup operations are running and no client read/write on the same nodes. You should always add a bit more Proxies. In this last sample I would for sure go to 10 Proxies. As well remember that the network needs to be really tested. You can not calculate with the on paper available network throughput. Always check with iperf or other tools and do parallel tests. I saw so many firewalls become the bottleneck if you put backup streams through them.

[Andreas Neufert]

I updated the original entry a bit to reflect latest product changes and feedback from customer situations.

[Andreas Neufert]

I corrected one of the build numbers above based on feedback that the patch was created for these specific versions.

[Pkfyre]

Hi Andreas, thanks for this. We are looking to start backing up our Isilon cluster straight to AWS S3 and this has been helpful.

I was curious if you know why the supported OneFS version only goes up to 9.5 in the v12.2 release, 9.7 is now LTS as of May and on the surface, there does not appear to be any stand-out changes in 9.6 or 9.7 that would require Veeam to update functionality.

[Andreas Neufert]

We ran into some issues with access to test labs unrelated to our processing capabilities which delayed the support plans for later versions.
The contractual side is now fixed and we continue. I will update here when we have updated the document with latest support for 9.9 (which should be the current one).

[Pkfyre]

Hey Andreas,

We were meeting with our Dell rep the other day and when we mentioned wanting to backup our Isilon straight to AWS s3 with Veeam they mentioned that would not be possible and we would have to take a local copy beforehand. I'm about to try and demo this out to confirm that limitation, but does that sound correct?

[Andreas Neufert]

See here: https://helpcenter.veeam.com/docs/backu ... ml?ver=120

You can directly backup to S3.
Maybe there is some confusion about the Cache Repository. The Cache Repository is not used for the backup data, but for some metadata optimizations (faster processing, less IO on the backup target).