Compellent snapshot integration?

[PapaSmurf]

I'm about to chose between a new Netapp VS Compellent. I know the latter has been waiting for support for snapshot integration and this is high on my list too.

I have been told by Dell that this is about to arrive with their new SC SCOS ver 7.4, arriving at some point during "H1". They've named this release "Dangerous Man"

Does anyone at Veeam have the ability to confirm this?

[Andreas Neufert]

Hi Ben,
within Veeam we have a very strict policy regarding upcoming features and the communication arround it. As long as a feature did not reached beta stage, we will not comment about this. The reason for this is that we keep the things very fluent to be able to react on customer requests and market trends very fast. As well to release features whenever they are ready, while others need more polishing and wait for next release. In the end customer profit from this, as a lot of requested features can be added to the products. I think the last couple release notes speak for themself.

Regarding the storage integrations... The vendors create the plug-in with us together. That said the plug-ins will depend on development from the vendor and as well compatibillity of the storages. As we do not have this under our control and are not allowed to comment by contract, the only option for customers is to speak with their storage vendors.
Thanks Andreas

[PapaSmurf]

Thanks for the response Andreas!

I understand the reasons you cannot comment, and all I can take from this is that if compellent snapshot integration is on the horizon that it isn't yet in Beta.

I shall cross my fingers and assume this is forthcoming, and use that to make my decision.

[Andreas Neufert]

There are no betas for the storage integrations. If a vendor completes the plug-in and pass our QA completely, then it just a matter of updating the websites and documentation before the release. So vendors can give you the best idea and you create additional pressure to them to work/start on the integrations.

Oh and usually there is no depenancy on newer Veeam Backup & Replication releases... so we release when the plug-in is ready.

[alexs]

Sigh,, i have compellent, contemplating veeam...

[jmmarton]

Keep in mind that snapshot integration isn't a requirement for performing backup. There are multiple options, including the ability even perform Direct SAN backups. So Alex in your case if you have Compellent there's no reason not to consider Veeam even if snapshot integration isn't in the solution today.

Joe

[alexs]

Hi

yes I like the other stuff, I am moving from VDP. so I am trialing now, i like the idea of talking directly to AWS storage, thats going to make my life easier.

My problem though is it hasn't been done in such a long time its not going to happen at all. well I will not hold my breath

I'm currently using my compellent for snap backups, so lots of luns for application groups - works well as it doesn't stun my VM's thats the issue I can't do veeam backups during the week as it causing issues on boxes.

the compellent snap -> vbr backup would have been awesome

[Andreas Neufert]

As I said, you should speak with the vendor about this.

On a separate note, we have som many technologies implemented in our product that address the usual issues with VMware backup processing.
For example we reduce the snapshot lifetime as much as we can to have a streamline snapshot commit. For example we read disks in parallel and do not backup deleted file and pagefile blocks which have usually a high change rate on block level. As well you can protect your workload with out Latency control which basically reduce the backup speed if the source storage is overloaded and start to have higher latency because of it. Regarding vcenter, we have an infrasturcture cache to reduce the SOAP connection during backup to the vcenter dramatically. Then we have special transport modes that can work faster and workaround some VMware specific issues. For example our HotAdd processing use async. read write which lead into 2-3 faster processing (less data in the VM snapshot => less problems to commit the VM snapshot). As well for environments with NFS storage we address the usual VM stuns with HotAdd by reading from the NFS share direclty with our DirectNFS.

[Lang]

From the release notes of SCOS 7.4.2, which has been released on August 30th:

Veeam Snapshot Integration
Storage Center 7.4 contains support for snapshot integration with Veeam Backup and Replication 9.5.

The Veeam Backup and Replication Plug-in for Storage Center is available for download from the Veeam website.

The plug-in does not support the following Storage Center objects:

FluidFS volumes
Live Volumes
VVols

Unfortunately i can not find the plugin on the veeam-website yet...

Regards,
Marc

[Gostev]

It will be shipped this month. Thanks!

[cerberus]

That is interesting news, I'll be on the lookout for Compellent integration with Veeam.

What benefit is there to be had when integrating with the storage array? We already map the CML LUNs to the backup server (read-only) so that we can perform [san mode] backups of our VMware environment with CBT enabled.

[Gostev]

Here are some of the key features and benefits:
https://www.veeam.com/backup-from-stora ... shots.html
https://www.veeam.com/san-snapshots-explorer.html
https://helpcenter.veeam.com/docs/backu ... l?ver=95u4

[TrevorBell]

Good Timing, Just installed 2 5020 All flash arrays

Some Notes from My Dell Support Manager just for info..

"There are actually two new SCOS codes in the pipeline..

SCOS 7.3.20 – This is a minor release, an upgrade to the 7.3.11 you’re running at the moment and contains patches and minor updates.

SCOS 7.4.2 – This is a major code release which as you say has support for Veeam and also some impressive performance enhancements too!

7.3.20 is available now and has actually become the recommended code level for SC customers, 7.4.2 is still very new and may take a while before it’s adopted by the masses.

Our team recommend staying away from the major release codes until they are in the minor release stage, this gives engineering time to work with early adopters on bugs and such.

There’s also a new version of DSM (2019 R1) that you’ll need to be using when making the move to 7.4.2."

Thanks

Trev

[squebel]

We're getting ready to set up our first SC5020 running 7.4.x with the new plugin and test out the integration. Where is the best place to go for specific configuration requirements to work with Dell SC specifically around how to set up permissions, the backup proxy, zoning, etc? Thanks!

[evilaedmin]

@Squebel:

Dell has everything which you need here

And Veeam has everything which you need here.

[Frosty]

I'll be interested to hear how your SC5020 goes with v7.4.x firmware. I'm running an SC5020 on v7.2.x and it has been running without a reboot for about 18 months, so I would be interested in v7.4.x performance and stability observations that you might have.

[TrevorBell]

At this moment in time i have installed the plugin and upgraded the DSM. My last communication from Dell Proactive Support was -:

" Normally I’d advise waiting for the first minor release of any particular SCOS code to come out before upgrading.

We’ve not heard any news internally about a minor release yet but based on past experience I’d say we’ll see it around December / January.

Speaking with some of the guys this morning who’ve already moved customers onto 7.4, early impressions are very good with no bugs reported.

All that said, I’ll keep and eye on the adoption rate and once enough people are on it and all the feedback is good I’ll let you know.


Thanks

Trev

[kewdah]

So i thought I would post my experience with the new storage plugin.
Upgraded DSMs to 2019R1 and 5020's to SCOS 7.4.2
Upgraded Veeam to U4b and installed SC plugin.
Veeam storage sees Arrays just fine but no volumes after scan.
Go back and dig deeper into the documentation and find a 3 word note that Live Volumes are not supported.
Tested this by converting a test volume to replication and the re-scanned with Veeam which was then able to pick it up just fin...convert back to LV and *Poof* gone...Oof..oh well maybe next time.

[foggy]

Hi Tony, thanks for the feedback. Correct, Live Volumes are not supported in this first Compellent plug-in version. Dell EMC is aware of this limitation and chances are they will address in the future plug-in updates.

[Andreas Neufert]

It is important to give vendor like Dell EMC direct feedback that this is a highly demanded feature as they usually follow the requests from their sales teams.

[squebel]

And here is our experience so far using only iSCSI:
- Configuration was a breeze. Make sure your proxies have access to the storage, configure the array in Veeam and then let it do a rescan. The initiators pop up automatically in DSM. No masking/zoning/mapping to the volumes needed (very cool).
- Orchestration of all the snapshots in VMware and then on the array, the backup job, etc handled as you'd expect.
- Seems to work really well
- Very cool that you don't even need to run any backup jobs with Veeam. As long as it can see the snapshots created on the array, you can do vm restores right from those snaps.

A few things:
1. The snapshot taken by Veeam is left on the SC and will not delete until you take another snapshot. This is a design of the SC array. To get around that (if you want), you must set the advanced setting on each volume to "Allow snapshots to coalesce into active snapshot".
2. Snapshots on the SC arrays kick off what's called On Demand Data Progression (ODDP) which moves blocks into different RAID types (if you have it set this way which most do). I'm assuming it still does this even with the Veeam snapshot. This can cause quite a bit of load on the storage subsystem in addition to the load being done by Veeam
3. If you allow the advanced option in #1 (above), this will generate even more load as the system will have to move changes "locked" in the Veeam snapshot back up into the volumes Active snapshot

Some initial thoughts:
- On a brand new array with VERY little running, the storage subsystem was generating nearly 20K iops while the snapshots and backups were running for just 4 tiny vm's running on two different datastores. I'm not sure how this would scale across dozens of datastores and hundreds of vm's. My theory is that all of the ODDP and movement of data between RAID types and allowing data to coalesce is driving more i/o than would be typically done in some arrays.
- We will probably reserve this type of backup for large vm's that have very high change rates where we are concerned about the amount of time the vmware snapshot is left running while the backup job is being performed.

[Andreas Neufert]

Thanks for the feedback. Please allow me to give some background information.

1.) When you perform Snapshot orchestration (no backup out of storage) we just trigger the snapshot. This cause no I/O from our side. You stated that there are some offloading options or better delete options for snapshots available on the storage. These I/O are independent from Veeam load. If you use the same with native snapshot orchestration or any other 3rd party software, then those I/O would be the same.

2) We rescann any snapshot once to check if there are VMs on it. This will cause some minor IO load like you would mount a volume and read the file table. No extra IO planning needed.

3) The backup load. If you measure IOs then this is somehow misleading. We backup on incremental forever base. First backup run 100% read and then usually 3-5% on daily changes. This data we read as fast as we can which is the goal of any backup solution. This can cause some spikes in the IOs. But remember there is latency control in Veeam that can monitor the latency on the datastores and reduce backup load before your production would be affected.

[Andreas Neufert]

Regarding numbers... You said 20.000 I/Os We read at 1MB blocks. that means 18GB/s ... Veeam can handle this but my guess is more that this IO peak came from the backend processes you described and not Veeam.

Our Storage integrations are used in the most difficult to handle and huge environments and scale usually with the storage to ultimo.
Specifically you can backup with multi GB/s as needed with the parallelization.

[squebel]

@Andreas - I agree with you 100% that all of the additional load is from back-end process on the array and not necessarily generated by Veeam. These are supposedly low-priority on the SC array (as one would expect) but it should be considered for any SC user looking to do the integrated backups. The SC aka Compellent is just a weird beast in general on how it handles snapshots and data progression.

[Gostev]

Very cool that you don't even need to run any backup jobs with Veeam. As long as it can see the snapshots created on the array, you can do vm restores right from those snaps.

That's a very bad idea though

[TrevorBell]

Did the upgrade to 7.4 on a couple of SC5020`s yesterday...

As above i do like the idea of seeing snapshots made by the SC array and just mount them and use.. Its slightly longer to mount for say a file recovery as it mounts the VM and presents within Vcenter and then you can see the files, just a slightly different way from how a flat backup file recovery works.

The plug-in does add extra protection as what i have setup is 15 min snapshots, deleted after 12 hours to completment the backup stragegy i already have.


Note: The compellent took 25 mins per controller to upgrade ( this is with 13x SSD 4TB disk ) . You need to call Dell who will do a health check of the array via SSH then release the latest FW for install.