EMC Isilon - how best to use

[umad]

I have joined a new co and have a nice 16 node 2PB EMC Isilon setup purchased prior to my joining. It's currently just used as an SMB repo. Backups jobs are pretty basic and need some work (modest sized backups with rollback Syn Full conversions on Isilon are taking 2 weeks to complete). I'm reasonably familiar with Veeam from prior employers but used only locally-attached repos on windows servers.

I've adjusted a few things but I'm looking for insights from anyone who's worked with an Isilon re:
1) What are the benefits of a Scale out Backup Repo? It seems it's good for performance if I can reorg it and possibly if I may need to do a repo migration in the future but anything else? any pressing reason to look at SOBR w/Isilon?
2) I'd like to use the immutable storage option of Isilon SmartLock to protect against potential ransomware attack. Anyone ever configure this on Isilon or a similar platform? How to set up jobs etc.
3) Anything I should NOT do with Isilon? The merge operations in incrementals with syn-fulls w/rollback are definite no-no. e.g. a full backup takes 5hrs, incremetal < 1hr but a syn-full takes literally 13-14 days. Ouch.

Thanks

[soncscy]

Isilons can't handle any random IO, so you need to make sure you're avoiding this at all costs.

Veeam has a handy article on it here: https://www.veeam.com/kb2351

I've dealt with this at client sites before and I'm sorry you have to deal with Isilon, but you are going to be constantly fighting with the Isilon limitations. Surebackup probably is a no-go for you, as is Instant recovery.

SOBR probably won't help you here even with the performance mode as the only way that really works as I get it, is if you have a performant extent for the full backups to merge on.

I'd be wary of doing any non-application controlled smart locking since there's no way for the backup application to be aware of it for things like retention. I suppose it should probably work exactly as you want but veeam is going to throw errors I'd assume about not being able to delete files, and there's nothing you can really do to make veeam 'aware' of smart lock. But, if you can live with that then I don't really see a reason it "wouldn't" work.

[umad]

Thanks for your feedback soncscy.

What you said about SOBR makes sense.

The immutable stuff as you said is going to be tricky as Veeam doesn't know what is locked and what is not. Have you any experience configuring AWS S3 with object lock to other S3 tiers or Azure Immutable blob storage? IIRC I think a special gateway was required to take full advantage of that in AWS, not sure about Azure.

Happy New Year!

[Mildur]

There is no „Azure Immutable blob storage“.
Microsoft needs to develope this first.

For other S3 Storages, you can find a list here

object-storage-f52/unoffizial-compatibi ... 56956.html

[Andreas Neufert]

For Isilo the following need to be cosidered.

1. older Firmwares had a limitation for maximum file size of 4TB. I think it is 16TB now. That means your biggest VMs that you can backup shouldn´t be bigger than 16-30TB (depending on compression level).
2. Isilon was built for massive parallelized access to many files with a sequential work pattern per file. Veeam Synthetic Full or Merges perform massive random IOs which the system can not be handeled well. I suggest to use it with Active Full + Incremental processing. There the system performs well and fast.
3. Scale out... Isilon usually uses DNS round robin to spread the load accross multiple nodes (or external DNS load balancers). In both cases out Repository Gateway Server will get an IP address from one of the controllers, will cache it and use only this controller for the workload. Therefore it is best to use multiple Repository Gateway Server (1:1 with the number of Isilon nodes maybe) and use individual folders for each of them within the share (do not use a central folder or just the share for all of them). That way Veeam will not only use one of the nodes for IO.
4. Any locking mechanism that prevents Veeam from deletion of the backup files is not compatible. If this happens we marke the chain for repair and the chain will not accept new backups (you need to create Active Fulls to start a new chain and maintain the deletion of the old backups manually). Usually no one has space for this. Maybe the Isilon can create Snapshot and make these snapshot immutable which both would be transparent for Veeam.

There is an external software from General Storage dsmISI that workaround multiple things with the Isilon. It will span a virtual filesystem accross the Isilon in combination with a load balancer to leverage the full speed of all controllers (even with a single Veeam Server). It will also enable fast cloning so that you can work with Synthetic Full + Incremental or Forever Forward Incremental without generating random IO workloads on the Isilon. It allows as well to use multiple nodes for one read/write stream which makes a single VM backup/restore faster.

[umad]

There is no „Azure Immutable blob storage“.
Microsoft needs to develope this first.

For other S3 Storages, you can find a list here

object-storage-f52/unoffizial-compatibi ... 56956.html

https://docs.microsoft.com/en-us/azure/ ... le-storage

[Gostev]

He meant "no object-level immutability in Azure". Container-level immutability you linked does not help, as it can only be used as a tape replacement through VTL with all the associated drawbacks of massive storage and bandwidth consumption due to the periodic full backups requirement.

[umad]

For Isilo the following need to be cosidered.

1. ...
2. Isilon was built for massive parallelized access to many files with a sequential work pattern per file. Veeam Synthetic Full or Merges perform massive random IOs which the system can not be handeled well. I suggest to use it with Active Full + Incremental processing. There the system performs well and fast.

Thanks for the clarification re better to avoid merges.

3. Scale out... Isilon usually uses DNS round robin to spread the load accross multiple nodes (or external DNS load balancers). In both cases out Repository Gateway Server will get an IP address from one of the controllers, will cache it and use only this controller for the workload. Therefore it is best to use multiple Repository Gateway Server (1:1 with the number of Isilon nodes maybe) and use individual folders for each of them within the share (do not use a central folder or just the share for all of them). That way Veeam will not only use one of the nodes for IO.

This is good info. I'll look into this a bit more re how to use re inserting gateways to spread the load.

4. Any locking mechanism that prevents Veeam from deletion of the backup files is not compatible. If this happens we marke the chain for repair and the chain will not accept new backups (you need to create Active Fulls to start a new chain and maintain the deletion of the old backups manually). Usually no one has space for this. Maybe the Isilon can create Snapshot and make these snapshot immutable which both would be transparent for Veeam.

There is an external software from General Storage dsmISI that workaround multiple things with the Isilon. It will span a virtual filesystem accross the Isilon in combination with a load balancer to leverage the full speed of all controllers (even with a single Veeam Server). It will also enable fast cloning so that you can work with Synthetic Full + Incremental or Forever Forward Incremental without generating random IO workloads on the Isilon. It allows as well to use multiple nodes for one read/write stream which makes a single VM backup/restore faster.

The point about the locking Veeam doesn't know about makes sense. It may work if I put enough logic in place but Murpthy's Law has a way of working its way into the best plans and breaking things.

I've seen the General Storage dsmISI product talked about on the Veeam forums before and I checked it out. Looks like it may offer some nice solutions and I'll reach out to them for a quote too.

Thanks for all of your great feedback on this Andreas!

[umad]

There is no „Azure Immutable blob storage“.
Microsoft needs to develope this first.

For other S3 Storages, you can find a list here

object-storage-f52/unoffizial-compatibi ... 56956.html

Apologies- misunderstood the context of your reply and didn't click the link you sent.

Makes sense.

Thanks!