NetApp Snapcenter VS. Veeam

[dasfliege]

A customer of ours has two NetApp Arrays:
- Primary FAS2650 All-Flash
- Secondary FAS2650 SAS

At the moment, they backup their virtual vSphere infrastructure via the NetApp Snapcenter (VSC) and all the other stuff (SQL, Exchange, Tape, etc) via BackupExec. We offered Veeam to make things easier. The customer has unlimited licenses for all Snapcenter products. What i'm not sure about now, is if it would be a good idea to replace all the native NetApp mechanisms with Veeam, or if i should use Snapcenter to Backup the VMs, SQL, Exchange and just use Veeam to orchstrate the Snapmirrors/Vault and Tape Backup.

What would you guys recommend?

[foggy]

I would definitely recommend a full switch to Veeam B&R, otherwise, you will have limited restore and tape backup capabilities. Moreover, does Snapcenter create just snapshots or some sort of backup files in the repository (I was not able to quickly find this kind of information)? Just keep in mind that storage snapshots cannot be considered as true backups since they are lost along with the storage/get replicated to the secondary storage along with "bad" data.

[Adam.Bergh]

Hi,

The first thing to understand about NetApp's SnapCenter is that it only orchestrates ONTAP snapshots and that it does not create backups. Remember, snapshots are not backups! A snapshot is an instantly created image level point-in-time recovery point on the primary storage. Although it is a recovery point, it leaves data vulnerable to a primary storage outage. A backup is a process which creates a consistent state of the application data and operating system and then copies the data to a different storage system. It's full copy of data which is no longer depending on the source system (so-called "media break").

Veeam always recommends a combination of snapshots AND backups for a complete data protection strategy.

Veeam's integration with NetApp's ONTAP and HCI has near identical overlap with SnapCenter with regards to snapshot orchestration for VMware workloads. When implementing VBR, you should discontinue the use of SnapCenter and use VBR as a single interface for both snapshots and backup management.

In additon here are a few gaps that customers using SnapCenter only vs. Veeam Backup & Replication have:

• No ability to restore VM workloads to 3rd party storage, hypervisors, or cloud targets like AWS or Azure. (No Portability of Backups)

• Gaps in data item recoverability, such as Linux files, Active Directory items, SharePoint items etc.

• No ability to tier backup data to S3/Cloud Targets in the event of unexpected data growth.

• Data availability concerns due to only having data on one storage OS (ONTAP) Risks could be a common bug in ONTAP effecting all systems

• No ability to create Dev/Test labs from Storage Snapshots

• No Bare Metal Backup or Restore

Benefits of adding Veeam to the NetApp Solution:

• Single tool to manage all NetApp snapshots, SnapMirror, and SnapVault data protection schedules

• Ability to create a portable off ONTAP backup for extended capabilities like restore to cloud, alternate hypervisors, or 3rd party storage for unknown future needs.

• Close data recoverability gaps

• Leverage secondary copies of data to create “DataLabs” for Dev/Test, patch testing, security testing, etc.

• Add additional security to your data by using role based access controls between data platforms - ONTAP Admin cannot access Veeam data repositories for example

• Ability to tier long term storage to cloud/s3 targets if needed

• Much More!

Feel free to reach out to me if you have any additional questions! Thanks!
Adam Bergh
adam.bergh@veeam.com

[segfault]

A few benefits to NOT using Veeam for NetApp Snapshots:

• Independent system to fall back on in the event that the Veeam B&R server fails, is stopped, or compromised in some way.

• SnapLock provides a higher level of assurance that somebody did not go in and remove the snapshots.

• Add additional security to your data: Veeam Admin cannot access ONTAP config to modify snapshot/snapmirror policy

For me, it is not an either/or decision but a "both" type of scenario since I assume that one of the systems will be compromised. There is a bit more coordination required to prevent them from walking all over each other, but I found that the result was worth the effort.

Along the reasons outline above, I also have our replication Veeam instance separate from the Veeam backup instance. The Backup side of the house does not even have rights to the DR vCenter server. I guess I've seen too many ransomware horror stories to trust any one system, so I have 3 that work in parallel: Backup (with tape out), Replication, and Snapshot/SnapMirror.

[chjones]

I will preface this as I work for NetApp, but also spent about 6 years as a Veeam Customer using HPE 3PAR Storage. I know both Veeam and NetApp very well.

I agree and disagree with both comments above.

1. NetApp SnapCenter does create backups. It's not just "ONTAP Snapshots". SnapCenter registers a Plugin to vCenter where you can orchestrate VM or Datastore Backups. SnapCenter integrates with VMware Tools on the Guest VM and creates an OS-Consistent snapshot, then an ONTAP snapshot, and can then run a SnapMirror or SnapVault update to a secondary destination. This is a backup, it is not just a crash-consistent snapshot.

Two major benefits to this is maintaining NetApp Storage Efficiencies end-to-end, and also the backups are almost instant. ONTAP snapshots complete in milliseconds. There is no need to mount a datastore and read out the data and write to disk backup repository and lose the storage efficiencies.

2. The secondary snapshots (or even the primary ONTAP snapshots if you wish) can be tiered to an S3 Cloud Target directly via NetApp FabricPool capability. This is transparent to Veeam and maintains NetApp efficiencies such as deduplication, compaction and compression. ONTAP 9.7 also includes the ability to move your tiered data between cloud targets transparently to the end users and Veeam.

3. You can use NetApp FlexClone to create instant read-write clones of datastore backups and present them to VMware for test labs. This isn't automated as nicely end-to-end like Veeam does, so Veeam does still win here.

I am a supporter of both approaches. They both have their values and drawbacks.

Personally, I'd choose to let Veeam orchestrate everything, but I'd leverage Veeam's Snapshot Only backups and ONTAP efficiencies and replication and ONTAP tiering to cloud.

One area Veeam is a clear winner is the application-aware backups and the Veeam Explorers. These are awesome! But these work with Snapshot Only backups, you don't need to write data out to a Veeam Backup Repository to be able to leverage them, so you get the best of both worlds.

Also, Veeam can integrate with any other existing ONTAP snapshots that have been created outside of Veeam, and as long as the volume is a vmware datastore, you can use all the explorers and restore from them. Win win!

[dasfliege]

Thanks for all the feedback. As chjones said: NetApp Snapcenter of course is able to create application consistent backups. Else i wouldn't even consider using it. NetApp did achive that in the past by using the Snapdrive / Snapmanagers capabilities, which was quite a pain to managed, as there were a whole bunch of individual componenets which had to be maintained. I didn't used the Snapcenter approach myself yet, but i have been told that it is much easier to maintain. What i personally liked with Netapp native backups in the past, was that they are created in seconds. There is almost zero impact on the production system, while we had several problems with impacts using veeam. Especially with realtime-critical SQL-Servers during VMWare Snapshot creation or removal, even though we're using storage-integration and therefor the snapshots doesn't really exists for a long time. I of course know that this isn't a veeam but a VMWare problem.

However, as our customers currently has a quite volantile infrastructure, where some parts will get migrated to cloud-services etc, we will choose the most flexible approach, which definately is to use veeam to orchestrate everything.

[Adam.Bergh]

I just wanted to reply to @segfault on his post above, because I think there is some misconceptions in what was written.

Independent system to fall back on in the event that the Veeam B&R server fails, is stopped, or compromised in some way.

If VBR were unavailable for some reason, the snapshots that Veeam orchestrated do not go away. Simply restore from them using the ONTAP System Manager. Adding in SnapCenter does not improve your situation in this regard, it just adds a second software tool to maintain.

SnapLock provides a higher level of assurance that somebody did not go in and remove the snapshots.

Many of our customers use SnapLock in such a way. You don't need SnapCenter or another orchestration tool to take advantage of SnapLock technology. One of my goals for this year is to publish a guide on SnapLock with Veeam.

Add additional security to your data: Veeam Admin cannot access ONTAP config to modify snapshot/snapmirror policy

The Veeam operator does not and cannot modify any ONTAP specific settings. All snapshot/snapmirror scheduling is done though the Veeam backup job scheduling. However, it is a valid point that a Veeam operator could maliciously modify a snapshot schedule through the Veeam backup job policies. It's important to audit changes to these policies of course.

[Adam.Bergh]

This is a reply to @chjones and @dasfliege

NetApp Snapcenter of course is able to create application consistent backups

NetApp SnapCenter does create backups.

NetApp has spent years and years telling their customers that snapshots are backups. Simply not true. Unless you can restore that data to another storage platform, the cloud, another hypervisor, etc, it is not a backup. It is and should be part of your data protection strategy, but should not be a complete replacement for data protection, and you shouldn't consider yourself fully protected. Remember Veeam's 3-2-1 rule! The "2" is two different media types - meaning that one of the copies of your data should be off of ONTAP.

To be clear, Veeam loves snapshots and SnapMirror, and it's absolutely part of our best practices for NetApp customers to take advantage of. We've had SnapMirror/Vault management capabilities in our product for years and want to encourage customers to use that, but not as a complete replacement for a true backup.

The secondary snapshots (or even the primary ONTAP snapshots if you wish) can be tiered to an S3 Cloud Target directly via NetApp FabricPool capability.

Please please please do not consider FabricPool a data protection technology. It absolutely is NOT. The data in the S3 bucket cannot be recovered from in the event of the source ONTAP system outage. FabricPool is a cost saving tool for customers using All-Flash ONTAP arrays, nothing more. You still need backups of your data if you have implemented FabricPool.

You can use NetApp FlexClone to create instant read-write clones of datastore backups and present them to VMware for test labs. This isn't automated as nicely end-to-end like Veeam does, so Veeam does still win here.

Part of our "secret sauce" if you will is the virtual router and auto-configuration of this router for network segregation of the VMs in the virtual lab. This is actually the hardest part of creating a virtual lab with copies of production data.

I'll leave you with this: It's a diagram of what we consider to be the "holy grail" architecture with Veeam and ONTAP. Veeam orchestrates all application consistent snapshots/mirrors/vaults as well as moving data to the cloud.

[segfault]

I'd like to clear some misconceptions (to use @Adam.Bergh term for it) in Adam's response.

If VBR were unavailable for some reason, the snapshots that Veeam orchestrated do not go away.

With Veeam as the snapshot orchestration tool, older snapshots may still be there but new snapshots are not being created, and to me that is the crux of the issue. If Veeam goes offline for any reason (Like a service pack taking extra long to install), my backup SLA's are immediately at risk. With a split system, I have the ONTAP orchestrated backups to fall back on during that window. Are they as nice as a Veeam orchestrated one? No, but they are better than the alternative of having nothing.

I work in an environment that is at a very high risk for a sophisticated and targeted ransomware attack. I have to assume that when an attacker gets into our environment (not if but when), it will be worth the attackers time and effort to look around and do it "right" in an effort to make sure that the bitcoin payment is the only viable option left. That means finding the backup systems and deleting/disabling them. Data changes frequently, so going to last night's backups on those tapes are an option of last resort.

In the end, I'm of the belief that data protection should never be a single product. So I take the 3-2-1 rule and add another layer: At least two systems that are independent of each other (but the marketing message on that is not as clean as "3-2-1").

As a corollary to the Holy Grail architecture image you posted, I present to you my companion belief that should be factored in for some high risk environments:

Trust No One!



(If the image does not appear, go here: https://i.imgur.com/3dfDDMU.jpg )

--john

[chjones]

I agree with Adam's statement that ONTAP snapshots are not backups. They aren't, 100% agree. If you lose the primary array you lose your snapshots. This is not at backup.

We will always recommend you use snapmirror to replicate those snapshots to a secondary array at a different data centre. For disaster recovery we recommend SnapMirror relationships (whatever is at the primary array is what you have at the secondary array) and for backups we recommend SnapVault (different snapshot retention at the secondary array, for long-term archival retention).

Tip: One really cool feature we have is you can do both DR (snapmirror) and Backup (snapvault) with a single ONTAP replication relationship and provide both protection operations with a single movement of snapshot data. You don't have to perform DR and Backups as separate operations.

In the past 12-18 months we've increased the number of supported snapshots per volume (each VMware Datastore) to over 1,000, allowing for a full 7-years of data retention via incredibly bandwidth and storage efficient SnapVault replication, assuming 1 daily backup.

I understand Veeam's position on storage replication. If your data is corrupted at the source then you replicate that corruption. I've also been a member of this community for nearly 8 years and am well aware of the issues (mostly originating from VMware themselves) regarding efficient VM backups using CBT that Veeam has to constantly workaround. In my previous role working with Veeam, 3PAR and StoreOnce I lost count of the number of times I had to reset CBT and perform an Active-Full backup (not great when a full of all jobs took over 15 hours and you had to do this on a weeknight).

At NetApp we don't have to worry about CBT. Seriously, we don't use it. The way ONTAP works "under the hood" is every single ONTAP snapshot is a full backup of all VMs on that datastore. We don't do incremental backups with ONTAP snapshots; there are no backup chains to worry about. Every time you perform a restore, even Orchestrated via Veeam, a single ONTAP snapshot is presented directly to an ESXi host and the VM is restored, or powered on, directly from the storage array. There is no intermediate that is reconstructing the data for you. There is no rehydration or performance degradation due to random reads having to fetch deduplicated data from multiple files in a backup chain (gets even worse the longer the backup chain).

For your primary VM protection I would always recommend the solution Adam has in his image. Primary ONTAP snapshot followed by a SnapMirror/Vault to a secondary location. And yes, you can very easily configure Veeam to mount the snapshots at the secondary location and write them out to a Veeam SOBR which can then have a cloud tier attached. This works quite well and I have recently finished deploying this for a large customer in Australia. However, my first concern with writing out to Veeam Backup Repositories and then Veeam's Cloud Tiering is the loss of storage efficiencies. It's easiest to explain this with a couple of points:

• As we know, Veeam in-line deduplication occurs within each individual run of the backup job. A full backup today does not deduplicate against a full backup of the same job yesterday. You need to rely on storage (or Windows post-process) deduplication to achieve this. I recently saw a customer who was achieving ~10.5:1 with a HP StoreOnce backup target achieving 1.2:1 with the exact same backups when writing to a SOBR.

• When using Veeam SOBRs with multiple LUNs and using Windows ReFS and Block Cloning you will end up with (not immediately, but it will happen) full backups from the same job on different LUNs. When this happens, you lose the block cloning benefits. Also, if you have to redistribute your backup files across your LUNs for better balancing of capacity usage you lose any block cloning savings when you copy a VIB or VBK to another LUN.

• When tiering to an S3 Cloud Target, Veeam sends backup blocks once. If I run a full backup, then run another full backup, Veeam will only tier the common blocks between those full VBKs to the S3 Tier. This is fantastic. However, this is only within each single backup job. If I have two backup jobs writing to the same Veeam SOBR and are both tiered to the same S3 Tier, any common blocks between those backup jobs will be duplicated in the cloud tier. With aggregate level deduplication (referred to as Cross-Volume Dedupe) in ONTAP if those backup jobs are the same ONTAP disk aggregate (aggregates can grow up to 800TB in capacity) any common blocks are only stored once. This even includes when tiering data directly from ONTAP to the cloud. And ONTAP also deduplicates at a 4KB block-level, which is incredibly more powerful.

To address the concern about losing the cloud data if you use NetApp FabricPool and lose that ONTAP array. This is 100% true. Adam is correct. However, we negate this by recommending that you tier data to a cloud target using ONTAP to maintain our efficiencies such as cross-volume deduplication, but to also SnapMirror that data to another site. Yes, I know people immediately think "what?! you mean i need more arrays and more cloud storage?". This is true, but not the complete picture. Since all of our efficiencies are maintained you are only replicating change 4KB blocks between data centres/arrays, and because you are maintaining our efficiency for the blocks that are tiered to the cloud, overall we do see our customers actually consume less cloud storage when tiering data at two sites using FabricPool than using the Veeam method.

As I said before, there are pros and cons to each approach.

I still recommend using Veeam to orchestrate as much as it can. Veeam is a phenomenal product (it actually won me a Global IT Award at my previous employer by what I was able to achieve with it) and I highly recommend it. There are a number of things Veeam does better than ONTAP and SnapCenter, and there are things ONTAP does better than Veeam. Whichever suits your business is best for you. Me? I'll also go with ONTAP storage level operations where I can, and have Veeam orchestrate it.

Oh ... and did I mention that SnapCenter is also free? If you run VMware on ONTAP you can install and use SnapCenter (fully featured VMware integration for VM DR and Backups) today at zero cost. I forgot to mention that early, sorry.

[Adam.Bergh]

Great conversation @segfault and @chjones! Thanks for much for your thoughts and posting in the forums. Discussions like these always get people thinking of new ways to achieve greater data protection and efficiencies!

[Gostev]

I felt this was just too important to leave hanging:

1. NetApp SnapCenter does create backups. It's not just "ONTAP Snapshots". SnapCenter registers a Plugin to vCenter where you can orchestrate VM or Datastore Backups. SnapCenter integrates with VMware Tools on the Guest VM and creates an OS-Consistent snapshot, then an ONTAP snapshot, and can then run a SnapMirror or SnapVault update to a secondary destination. This is a backup, it is not just a crash-consistent snapshot.

I'm sorry, but this does not classify as backup in my book, because there's no media break that is required by the 3-2-1 rule. You're merely replicating storage snapshots within a single NetApp deployment, so effectively it's a single distributed media spread across multiple arrays from the same vendor. And such distributed storage system ought to share all firmware bugs and/or corruptions across all snapshots.

Here's the real story from Veeam POC on high-end NetApp deployment in Germany last year:
1. The prospect selected a group of VMs for POC, but Veeam would refuse to backup one of them (error reading the VMDK). Not a great start of POC for us
2. Luckily they were willing to troubleshoot, so as one of the steps we asked to Storage VMotion this VM to another datastore... same error!
3. The prospect then mounted and checked SnapMirror and SnapVault copies of the datastore on a secondary array... same error!

Needless to say, the deal was an easy win for us. They were blown away that if not this POC, they would have never known (until restore time) that they had this corruption in all of their snapshot copies. Only then, they really understood the need for real backups.

Now, I do agree there are some great use cases around storage-snapshot-only "backup" jobs! In fact, v10 brings a couple of major new features around this concept specifically. You really can't do anything like that without storage snapshots, not even close - so I'm really excited to put this in hands of our customers. Nevertheless, it is critically important to always have the real backup on a different media, and don't call "backup" anything that does not actually meet the 3-2-1 rule. Because if anyone wants to argue the latter, then... well, that would be a whole different discussion

[Andreas Neufert]

As Anton said, it is not about trusting a vendor's technology or not. It is all about detecting if the application/OS/platform that has written the data did not make a mistake.
A real backup goes to the application that has written the data and requests from there data for backup. It will read the data and know if it has issues with it or not. In your 3-2-1 backup rule, the "2" defines such a situation as a media break. It is not all about just the media break, but you have to have as well what I call a "media logic break". Media logic break is defined as Reading the data with the application again and writing it in a new format (on another storage with other firmware/storage type).
Anton above shared a customer situation, but there are more examples, and we all can tell stories about the human factor that deleted stuff by mistake that was replicated across storages.
Veeam has a reg key to ignore VMFS or Hyper-V consistency issues at backup, to allow somehow customers to backup whatever was written damaged to disk before they start to take repair or replacement actions. So we have a pretty good view of the market that these kinds of issues are real, and not just a marketing spook to justify our technology.

Btw. This media logic break does not apply only to the primary backup data for backup, but the backup application should read it's own data for copy processing with the native method and write it into new backup independent backup chains. This is again important to know that the data is readable, and the chain is not corrupt. Again not a matter of Veeam does not trust the storage replication, but you do not know if something broke the chain in the first place. An example here is that customers sometimes use CIFS storages without write through support which could damage the backup chains in some situations (precisely in that cases when your datacenter is somehow affected, and you need restores the most), and every storage replication technology will not detect this and just replicate the corruptly written data.

[stuart_little1874]

I just wanted to reply to @segfault on his post above, because I think there is some misconceptions in what was written.


If VBR were unavailable for some reason, the snapshots that Veeam orchestrated do not go away. Simply restore from them using the ONTAP System Manager. Adding in SnapCenter does not improve your situation in this regard, it just adds a second software tool to maintain.


Many of our customers use SnapLock in such a way. You don't need SnapCenter or another orchestration tool to take advantage of SnapLock technology. One of my goals for this year is to publish a guide on SnapLock with Veeam.


The Veeam operator does not and cannot modify any ONTAP specific settings. All snapshot/snapmirror scheduling is done though the Veeam backup job scheduling. However, it is a valid point that a Veeam operator could maliciously modify a snapshot schedule through the Veeam backup job policies. It's important to audit changes to these policies of course.

Hello, did you get round to creating the Veeam and Snaplock document? We are in the process of deploying Veeam (and Snaplock) and would be interested in reviewing that doc