Veeam NAS backup best practices with backup from Dell PowerScale/Isilon Smart Connect

[Andreas Neufert]

Veeam NAS backup best practices with backup from Dell PowerScale/Isilon Smart Connect

Note: This guide does NOT describe the usage of PowerScale/Isilon as a backup target!

Dell PowerScale (old name Isilon) is a OneFS (Software name) grid-based filer. It uses server-like hardware with local disks to build a filesystem across the nodes and deliver all capacity within a single namespace to the users.

The system was built to deliver unstructured data to a huge number of parallel users with mainly sequential read/write requests. A single connection will usually get from DNS a node IP of the grid cluster to work with, and the load is distributed for many connections throughout all nodes. The backup (and regular client) throughput of a single node is limited, and therefore a distribution of the backup load is needed across nodes.

Veeam NAS backup without Dell Smart Connect:
The Veeam Backup server works with the share’s given IP address or DNS name, and all (data mover and scan) NAS Proxy Servers will work with this (single) IP.
- Even when multiple Proxies are used, Veeam will likely use the same node for all actual backup streams.
- Usage of multiple nodes is only the case if multiple shares are added with different DNS names so that Veeam gets a chance during DNS resolution to get from DNS (round robin) a different node IP address for each share.

Veeam NAS backup with Dell Smart Connect integration:

Dell Smart Connect design:
Dell Smart Connect will be integrated into customers’ DNS system, where the DNS server delegates a cluster DNS name to the PowerScale integrated DNS server. This Smart Connect DNS server alters the resolution of the cluster DNS name to different storage nodes for load distribution. By using DNS resolution the clients that access the data get different IP addresses from different nodes to work with. Depending on Dell licensing, there can be load balancing based on Round Robin and CPU/Network Throughput/Connection Count. IP addresses in the storage cluster can be distributed manually or automatically to storage nodes and a single node can have multiple IP addresses with and without failover between nodes.

Note: SmartConnect is usually setup in a way that it serves the regular production workload best. Altering Smart Connect for backup needs coordination, and usually, changes are not possible. The Backup system needs to use potential workarounds for ideal load distribution then.

The following steps illustrate how Veeam works with Smart Connect:
1. The storage (management) IP will be added as Veeam Storage to Veeam Backup & Replication.
2. NAS Proxies are rolled out (distribution of load). The more Proxies the better as Veeam distributes rescan and backup load throughout the NAS Procies. It is best practices to rollout the same amount of (smaller VMs) NAS Proxies as the storage system has nodes.
3. NAS backup Jobs needs to be created and a share from the storage can be selected from the list.
4. When the NAS Backup Job starts, each NAS Proxy will reach out to DNS to get the IP address for the cluster DNS name individually. It depends on which Smart Connect setting and mode is used, if Veeam Proxies will get different node IPs or not. It can happen that SmartConnect is not able to change the IP address in their load balancing fast enough based on setting and all Veeam Proxies that start backup nearly at same time will get the same node Ips to work with. See below for best practice implementation.

Best Practices implementation:

1. Enable Smart Connect on the Storage and integrate into DNS.
2. Add the Storage with the Smart Connect management IP (SSIP) to the Veeam Backup Server (Smart Connect usage is detected, and Veeam checks the DNS names on each Proxy instead of centrally on the backup server).
3. Rollout NAS Proxies at a high count. We suggest to use one NAS Proxy per Dell PowerScale/Isilon storage node. These nodes can be used to backup multiple PowerScale systems and clusters.
4. Manually distribute the load, by editing the operating system hosts file of each Veeam NAS Proxy and resolve the cluster DNS name (and short name) to a different PowerScale node IP address.
5. (Optionally) Enable Dynamic IP allocation on the PowerScale for IP fallback processing in case a PowerScale node goes down).

How to access the Smart Connect settings on PowerScale:
Cluster Management – Network Configuration

At the subnets you can view/edit the SmartConnect Service IP SSIP
A Zone Name (DNS name) can be configured that later need to be added to the regular DNS server as forward zone.
As well a pool of IP addresses and the static/dynamic IP node allocation can be configured.

[Andreas Neufert]

One clarification:
The more Proxies you use, likely the fewer resources (RAM, CPU) you need per Proxy. Please check on regular sizing guidelines and contact your Veeam SE/SA for sizing assistance.

[Andreas Neufert]

Second clarification:
As the best practice states, that we should create as many proxies as nodes are in the system, does this as well count for high node counts?

The answer depends a bit on how you implement the system, their east-west network, IO and throughput capabilities and how many shares you backup from that cluster at the same time. As well what load you want to allow for backup on the system (stealing it from your clients for the time of backup).

For each share that you want to backup at the same time, you can calculate the throughput that you want to put on the PowerScale system. Then see how many Veeam NAS proxies would lead to that throughput (check on network and PowerScale node limitations as well) and calculate from there how many proxies you should use. Then check again on how many shares you want to backup with that same speed and calculate from there the amount of RAM and CPU resources needed for it per Proxy (check again on if network or the nodes become a bottleneck).
See the best practices guide for NAS sizing for details:
https://bp.veeam.com/vbr/2_Design_Struc ... oxies.html


Sample:
You have 100 nodes and all networks are 100GbE (to just say that network will not be the bottleneck here in that theoretical example).
Let´s say your selected PowerScale node type can deliver under any circumstances 150MB/s random IO read throughput (ask documentation or Dell for what your storage nodes can provide realistically).
You want to backup 4 shares parallel each with 1GB/s which means you overall need 4GB/s.

4GB/s : 150MB/s = 28 Nodes

Then let´s calculate on what the Proxies would look like for 4 parallel share backups and how many resources would mean how many throughput they can handle.

Each Task slot on the Proxy process 1 share at a time.
In our configuration, we want to process 4 shares at a time, which means 4 task slots per Proxy.
Each Proxy Task Slot uses 2 Cores 4GB RMAN (see above link to the best practices documentation) which means we need 8 Cores 16GB RAM per task slot for all 4 tasks.
Now what is the theoretical speed the 4 task slots and 8 cores can provide:
Each Proxy Core can process 100MB/s (see best practices documentation link above).
Each Proxy could process 8x 100MB/s = 800MB/s
4 GB/s : 800GB/s = 6 Proxies = Read from 6 nodes based on the original post here.

When you compare the two number results from above, then the node throughput is the bottleneck in this sample. Which means, that you need 28 Proxies. Each Proxy with 8 Cores and 16GB RAM and a 4 task slot setting.

Second scenario with different node types:
Now let's check if the node type you use could provide 850MB/s on random IO read throughput.
4GB/s (that you want to have... see above) : 850MB/s = 5 nodes
Now let´s compare again with what we have calculated above a Proxies could handle. You can see that the bottleneck shifted from node to the Proxy. In this case you need at least 6 Proxies to deliver the needed speed.

Remember all this is under ideal lab conditions when only backup operations are running and no client read/write on the same nodes. You should always add a bit more Proxies. In this last sample I would for sure go to 10 Proxies. As well remember that the network needs to be really tested. You can not calculate with the on paper available network throughput. Always check with iperf or other tools and do parallel tests. I saw so many firewalls become the bottleneck if you put backup streams through them.