Isilon as Veeam target

[Jeff M]

I just learned that my org. has purchased Isilon for our Veeam backup target. Any information on making this work would be appreciated

[Egor Yakovlev]

Hi Jeff,
check out Veeam Isilon support\configuration KB!
/Thanks!

[Jeff M]

Thank You. So, with dsmISI you have to use Linux repositories? How would you configure for SOBR better yet, how would we migrate our existing SOBR (Windows mapped SAN drives) to Isilon? Is that even possible?

[Egor Yakovlev]

Sure it is possible.
- you can use Path #3 from this KB and then create a Scale-Out Backup Repository with Isilon, or
- you can add Isilon as an additional extent to the existing Scale-Out Backup Repository and use Evacuate Backups option on SAN extents to move backups over to Isilon extent.

p.s. evacuating backups will put quite a stress on your storage systems immediately, so use it with caution. In under 2 weeks we will release VB&R v10 with a new "Seal Mode" for extents, that will allow to block new incoming backup data to to be put on existing extents and will rather use Isilon as a target for new backups. Read operations will still be available on sealed extent data. With time, when retention hits your backup chains, older backups from original extents will get removed. That will be much more "gentle" way to migrate, yet more time sensitive!

/Cheers!

[Jeff M]

Thank You, Egor

[Jeff M]

We have decided not to migrate, we are going to get new system in place leave old system up for restores till retention runs out. So I am past needing help with Migration to Isilon Storage. Now I just need help with Isilon. Is there any Veeam Engineer or someone out there with Isilon experience. I have a design idea that came to me in a dream (Weird Huh). This system is to backup our Virtual Environment only. Can someone tell me if this might work? OK, Outside of using dsmISI, would it be possible to Have 5 physical Backup Servers each with 10GB fiber connections. Either NSF or SMB file shares on Isilon (still to be determined). Then deploying 5 physical Windows repository servers also with 10 Gig Fiber. Creating 3 repositories each from Isilon shares. Then making these extents of a SOBR for each backup server. SO, three extents on each Physical Repository Server 1 SOBR to each Backup Server Would that not provide the multi IO streams Isilon likes? Can a Windows Repository server also be a Gateway server? Or would I still need separate Gateway servers for each Backup server. If so I plan to deploy several VM proxy to each backup server. Can VM proxy be configured as gateway between repository server and Isilon?

[Jeff M]

I have several architectural type questions. When backing up to SMB Share on Isilon, Can the backup server also run as Gateway? We have beefy Physical Backup Servers with 10 Gig fiber to both VMware vSan and Isilon target. Can I place repositories on virtual host with say like 3 SMB shares, each as SOBR extents then set backup server as Gateway or can gateway run on repository server? Would it be better to have physical repository servers or will virtual work as well. Would multiple repository servers each as a SOBR extent be better than 1 repository host with multiple SOBR extents?

[HannesK]

Hello,
I merged your question with the exiting thread.

To make it short: I don't recommend using Isilon as backup target at all.

1) only use with dsmISI
2) only use active full (to avoid merges) or 3
3) XFS block cloning works fine. - Update 2024-01: XFS works fine
4) even with dsmISI and active full, be prepared for issues.
5) make sure the Isilon is configured correctly. I remember that there is a "streaming" and another setting. The default setting seems to be better, but for whatever reason it was changed at one of the customers I know.
6) technically you can have the backup server as gateway. but you don't want that

In general: any scale-out storage architecture will "break" with merges. It's not Isilon specific. XFS with block cloning might be an option.

Best regards,
Hannes

[Andreas Neufert]

As Hannes said, you need to calculate the IOs that you create on the Isilon. I saw many installtions where they just have selected an nearline Isilon version without doing the math so that even Active Full + Incremental was not possible as the target couldn´t handle the load and went to heavy latency because of it.

1) Plan for the IOs (with the DSMisi team
2) use DSMisi to workaround the throughput limitation, otherwise single stream is limited by the controler throughput of a single node.
3) use DSMisi to workaound the performance issues of files bigger than 1TB (EDIT: Based on Isilon documentation there is no such performance impact anymore with latest Isilon software)
4) use DSMisi to workaround the file size limitation of 4TB of the Isilon (EDIT: 16TB in latest Isilon software version)
5) Plan with Active Full + Incremental ... spread the Active Full accross 7 days (min 7 backup Jobs).

[Jeff M]

I need any updated information on job and repository settings for backing up to Dell/EMC Isilon Storage I have read KB2351: HCL - Dell EMC Isilon - Veeam which has very minimal Repository and Job settings and is somewhat dated. Does anyone have experience with this? Any discovered best practices? Anything really. We are not migrating. Starting all new. We have Physical backup servers and combo Vm and Physical Proxy servers. Our Isilon is new A2000 Any help would be most appreciated.
Thanks,
Jeff

[HannesK]

Hi Jeff,
I merged your question again with your existing questions and existing answers (please don't re-open new questions again and again in different forums).

Nothing has changed except for the maximum file size on Isilon. There are only a hand full of customers using that system and with Andreas and me from Veeam side you have the collected knowledge of most of these customers.

Best regards,
Hannes

[Andreas Neufert]

The maximum filesize is now 16TB which allows you to backup depending on compression factor 22-32TB VMs in VM chain mode.
The rest of the limitations and best practices still apply.

[Jeff M]

Why do NAS like Isilon (scale-out storage architecture) break with merges? Is there any fix for this? And why are weekly fulls recommended? Isilon also seems to break with synthetics or they get hung and never complete. And for some reason when running Isilon repositories with SOBR and weekly fulls, Veeam doesn't control retention well. Our jobs are set to keep 30 restore points when I look at disk backups many have 54 or more restore points and 3 full backups. That is taking up a lot of extra storage space. Anyone know why this is happening?

[veremin]

Can you provide a screenshot of the backup chain that does not honor retention policy?

Since you're using forward incremental with periodic full backup mode you might not have required number of restore points from the last or second-to-last full backup, and this doesn't allow retention policy to take place.

Check this thread for more explanation.

Thanks!

[HannesK]

Why do NAS like Isilon (scale-out storage architecture) break with merges?

because they are built for a different kind of workload

Is there any fix for this?

active full, or XFS block cloning in combination with dsmISI

And why are weekly fulls recommended?

It just needs to be "active full" to avoid merges. "weekly" is not required, can be scheduled rarer.

[sstorholm]

Is there any updates regarding support for XFS Reflinking?

[Gostev]

Can you please clarify your question?

[HannesK]

Is there any updates regarding support for XFS Reflinking?

please see my answer above. If you use dsmISI then you can use XFS reflink / block cloning. By doing that, the Isilon works fine as far as I hear from the field.

[sstorholm]

Ok, I'm having some problems understanding this setup, is dsmISI some 3rd party add-on to a "normal" Dell EMC Isilon setup, or something else?

[HannesK]

let me google that for you http://www.general-storage.com/PRODUCTS ... veeam.html

yes, it's an additional product. Without dsmISI as load balancer I see no realistic use case for Isilon as backup storage.

[sstorholm]

Cheers mate, I did have a look at that link, but got a bit confused as Dell has a similar component called SmartConnect, which does similar things, so I started wondering if this was a case of Dell integrating 3rd party software in their solution.

We're looking at both Isilon and Exagrid as a potential target for Veeam, hence my questions.

[Jeff M]

I am using multiple gateway servers each as an Isilon SMB share host and each a separate SOBR extent pointing to Isilon to load balance. This works pretty well without any third party product. Merges are slow but they are working.
FYI

[Gujumax]

@Jeff M

"I am using multiple gateway servers each as an Isilon SMB share host and each a separate SOBR extent pointing to Isilon to load balance. This works pretty well without any third party product. Merges are slow but they are working."

Can you provide a little more detail about this setup? I just started at a new company and they use Isilon A300 8nodes as backup targets but the performance is terrible. My company won't go for purchasing dsmISI so looking for ways to improve the performance without the need of 3rd party products.

[HannesK]

Hello,
are you already using "active full"? Synthetic fulls must be terrible on SMB / NFS in general and especially on scale-out storage systems, yes.

I also updated my old post: XFS with dsmISI works fine and would be the recommended setup.

Best regards,
Hannes

[Andreas Neufert]

Hi Max, I would really consider dsmISI in your case as it is a real performance boost and will likely be night and day. Above in this chat there are other recommendation to better distribute the load accross the dsmISI.

If no dsmISI:
Create a share on the Isilon. Create one folder per node and name them after the node.
Create for each folder a repository by using the individual IP addresses of each node and folder. Then create a SOBR on top of it. (Potentially use multiple Repository Server and split the load accross them). Overall the idea is that each node is used for processing. DNS or Isilong based DNS load balancing is not working that well in our case. If you still want to use DNS load balancing, disable the windows DNS client (which is a DNS cache).
This way Veeam will leverage more parallelization.
Use active full + incremental processing as the synthetic fulls will likely kill the backend of the Isilon system. The isilon is usually fast enough that the VMware snapshot lifetime is not an issue for VMware. Maybe distribute the active fulls through your environment and 7 days, so that each days load is the same.
Potentially increase max file size setting.

[Jeff M]

I am posting as a solution so I hope this gets by. First we use Physical VBS servers. These are ex ESXi hosts so very beefy. Next I create VM proxy several depending on the load. usually 4 or 5. These will be used as Gateway servers for the system. I will name these accordingly to help keep things organized. IE: VeeamGW001, VeeamGW002 etc. I will then usually create the same number of standard VM proxy servers. Then on Isilon, I will create the Shares in a main folder which I name the same as the VBS server. Then create your repository shares, I name them according to the gateway server to host it. (GW001Repo1, GW002Repo2 etc.) Each of your gateway servers will host one of these shares in the VBS Console. Again the naming convention is just to help me stay organized. Then in the VBS console, add your separate repositories. When adding a NAS repo you have your SMB share option, which will give you a name option (VM gateway001, gateway002 etc.). and a path option where you point each repo to its respective path on Isilon, and you are given a gateway option, where you select your corresponding Gateway VM Proxy. Your repositories should look like this:

ZXGW001 Repo1 SMB Gateway \\IsilonDomainName\VBSnameShare\GW001_Repo1 1.1 PB 492.6 TB 8.1 TB Created by ____ 11/21/2021 03:55 AM.
ZXGW002 Repo2 SMB Gateway \\isilonDomainName\VBSnameShare\GW002_Repo2 1.1 PB 492.6 TB 190.9 GB Created by ____ 11/21/2021 03:55 AM.
ZXGW003 Repo3 SMB Gateway \\silonDomainName\VBSnameShare\GW003_Repo3 1.1 PB 492.6 TB 21.6 TB Created by ____ 11/21/2021 03:55 AM.

Then you just add each repository (Gateway Host Share) as a SOBR extent. This gives you a widened path for Veeam to write to and Isilon to load balance to the separate shares. This has been working this way for 4 years. The only issue is sometimes in cases of network blip or VBS reboot you can get repository corruption which will appear as an unfixable failed job, at which time I will recreate the job with new backup stream. I leave the old job and backups in place These are still restore-able till the new job retention is met. Also merges take a long time as we run forward incremental with no periotic fulls. Hope this makes sense.

Jeff

[Gujumax]

Hello,
are you already using "active full"? Synthetic fulls must be terrible on SMB / NFS in general and especially on scale-out storage systems, yes.

I also updated my old post: XFS with dsmISI works fine and would be the recommended setup.

Best regards,
Hannes

We're doing active full, and we got a quote for dsmISI, and it was crazy expensive. My company already shot that idea down.

I wish my company would've gone with another storage solution, like HP StoreOnce or something better, rather than Isilon. I think of Isilon as an archiving solution that you need to keep data for longer retention periods and not as a primary landing space for backup jobs.

[Gujumax]

If we went with a different NAS appliance (not looking to spend a lot of $$$) just to backup M365 data, like Synology Business, would I get better, worse, or the same type of performance as the Isilon?

[Andreas Neufert]

Isilon was built for the access of many clients that work with relatively small office files or large sequential written files (media stream). The backend and load balancing was optimized for this.
Each node can run 150MB/s- 450MB/s under full load of the cluster when you write to it. If you read and write massively to it, it is a fraction of it. The reason of this is that ONEfs uses 64KB block sizes and so our IO are actually multiple times the IO on the Isilon. That said, dsmISI does something clever by splitting our backup files and heavily parallelize the IO including node load balancing for each sub section that they are reading/writing. We had some environments where we could get 3GB/s (yes Byte per second) in a single Veeam Write stream through dsmISI with Isilon. As well dsmISI adds block clonging to it. In many cases the solution outperformed many primary storages when we compared Instant Restore performance vs. the actual storage IO in disk IO tests. I remember that the customer sent me the test results and I belived in a mistake with the data being interchanged between production and Isilon, but it was not a mistake.

Any technology has its sweetspot and if you leverage it well it shines. Same with HPE StoreOnce, it shines in data reduction when you keep long term retention on it.

I can not speak for the Synology Business, but when you want to go with such systems, we recommend to go always with iSCSI (you can use block cloning which reduces the IO massively and the system works more stable).