API plug-in for Dell EMC (Compellent)

[nunciate]

I know this just came out but I was wondering if anyone has had time to test or use this new plug-in.

We have a Dell Compellent SC8000 setup both here and in our DR site. I perform backups via FC on physical proxies in both locations.
We only have volumes on the Compellent setup and presented to VMware 6.5 and the are configured as standard datastores within vCenter.
All VMs are stored on these datastores. We have a few physical servers with drives mapped to the Compellent but I am not worried about those.

I'd be very interested if anyone has any experience they could share. I'd like to know if my setup will work the way it is or if I'd have to make some changes.

Currently, our version level is at 7.2 so I'll definitely need to upgrade our SANs.

[Frosty]

I'm also interested in this topic. We've got a Dell EMC SC5020 (also with older v7.2 firmware), backups via 10GbE, running vCenter/vSphere v6.7 with all storage presented as standard datastores. I'm not really expecting that I would use this functionality to be honest, as our current setup is running really well and I would be reluctant to disturb it, but if there are big benefits I am all ears.

[Andreas Neufert]

The standard VMware backup works fine in nearly all environment but the integration is all about optimizing this and adding additional value.

1) For huge VMs with huge change rates the Backup from Storage Snapshot will help to reduce IO load on the VMware storage by not allowing to grow the VMware redolog snapshot and so less data to commit.
2) You can create crash-consistent and application aware storage snapshots in addition to backups more frequently to optimize RPO window.
3) usually your production storage is faster than your backup storage, so restores are faster out of the storage snapshots.
4) Labs run faster if you use them

[foggy]

I'd like to know if my setup will work the way it is or if I'd have to make some changes.

Hi Alan, your setup will work fine with the Dell EMC plug-in provided you configure the infrastructure appropriately and keep in mind system requirements and limitations.

[nunciate]

OK I will plan upgrades to my SAN as I have needed to do that for a while now anyway. I'll let you all know how things work out though it will probably take me a while to get my project approved and completed.

[nunciate]

I spoke to Compellent Support to determine if there were interim steps to get from my current SAN software version 7.2.11.4 up to the new 7.4. There are none so I could go straight to 7.4. However, 7.4 has only been out about a month and I would never put something that new into my production environment so I will have to revisit this idea sometime down the road one 7.4 has been proven stable.

[foggy]

Sounds fair. Look forward to hearing how it goes.

[evilaedmin]

@foggy

Is there any specialized support path or keywords we should use if we encounter problems with this integration? Since documentation is making it very clear that this is written by Dell EMC against a Veeam-provided API, should we always start with a Dell support case?

(Asking for tips because of prior experiences with points-of-integration such as StoreOnce Catalyst).

[Andreas Neufert]

All Universal Storage Plug-ins are fully Veeam QA tested and verified offerings. You always should start with a Veeam Support case and our team can interact and work with the vendor R&D department if there is any bug.

The only sitaution where you should contact the vendor is, if you have a question with the storage configuration itself. For example if you want to use FC with Veeam Proxy and you do not know how to create the Host entry for the Veeam Proxy within the storage.

[nunciate]

I am very excited to report that we have upgraded our DR SAN and I now have the Dell plugin installed for integration. Before I set up my jobs I watched the How-To video.
I had no idea I could simply run a job that would create storage snapshots and then restore data from those. I am super interested in that but do have some questions.

My biggest concern is, of course, storage space. Will creating daily snapshots dramatically increase storage space utilization?

Is anyone running these snapshot-only jobs multiple times per day? I'd love to do snapshots a couple of times during the day each day on some systems. For example, file servers with a lot of change between actual backups. My concern is that doing so could interrupt access or cause dramatic growth in the SAN space utilization. Will running a snapshot only job for VM's cause any access issues? I assume not since it is just a storage snapshot but need to be sure.

[Andreas Neufert]

Hello Alan,

Storage Snapshots do not affect the uptime of your workloads. If you do Crash consistent Snapshot processing then there is as well no VM snapshot.
If you go with Guest Processing for consistency, then you have the consistency processing and in many cases as well the VM snapshot (at least for some seconds). Usually customers perform once a days a Backup from Storage Snapshots and then create additional crash consistent snapshots.

Space: Yes, you need some space for the changes. It is basically the change rate on disk that you hold then. DellEMC can likely give you an idea how many space you need.

[nunciate]

Thanks for the info. I am planning to talk to Co-Pilot support to see what space requirements I will need.

My volumes are already configured to create a storage snapshot on each volume every night and keep for 24 hours so effectively I could use those for most full VM restore.

I'll continue my backup and replication jobs as normal so we always have the real backup to use if needed.

For the high priority VM I might consider creating 2 or 3 snapshots each day to get more up-to-date restore options though we have never really needed that.

For File servers which are pretty large and exist each on their own data store, I'd like to do the normal backup at night and then create a storage snapshot maybe 3-4 times during the day and those only need to last until the next backup runs. Just a means to restore files that get deleted between backup runs which we have had sometimes. Usually, we have to tell those folks the files are just gone.

[Andreas Neufert]

Our NetApp customers do every 10minutes snapshots for file servers and the additional overhead compared with an hourly snapshot is minimal in case of space for those systems. You need to check as well the best practices in case of how many snapshots you can create and keep for what time to not stress the storage controller. But the above concept any storage system should be able to handle without issues.

[nunciate]

In case anyone else is reading and cares here is my experience so far.
No issues to report. I added the plugin and added my SAN to Veeam.
I ran a test backup with no issues.
I also ran a test snapshot-only job on a somewhat large file server. It is about 4Tb.
That snapshot job starts at 6:00 AM and runs every hour until 8:00 PM with 14 restore points configured.
I only want to run it every hour during the day and keep snapshots for around 24 hours or until the next real backup runs so this appears to work.

SAN performance does not appear to be affected at this time. I do see some different looking I/O spikes but it doesn't appear to be causing any issues.
Since this is our DR SAN I do have some backup jobs running out there during the day so that is probably the I/O I am seeing.

As for space utilization, the server I am performing the hourly snapshots for is on a dedicated VMware Datastore/SAN Volume. I see 14 snapshots there.
The vast majority of the snapshots are between 10-20Gb in size but I did notice a couple yesterday that were 100-200Gb in size. I see 1 now that is 200Gb in size.
My SAN automatically creates a snapshot at 7:00 PM each night and keeps that one for 24 hours. The one from last night is around 450Gb right now. I think I will end the job before that time each night since we won't need to have 2 at the same time. So currently that volume has a 760Gb snapshot overhead. Most of that appears to be that nightly auto snapshot. I suspect that number will be reduced by tomorrow.

So far I am very happy with the performance and plan to implement more snapshot jobs on the other file servers next week I'll aslo implement this on my production SAN on Monday after I have updated the firmware on it this weekend.

[foggy]

Hi Alan, great feedback, appreciate it!

[nunciate]

Here is a question. Should my backup job still be creating VM snapshots in vCenter and also creating storage snapshots?
When I enabled this I did not create new backup jobs. I just left the existing ones.

I noticed that the VM backup job starts by creating a VM snapshot. Then it creates a Storage snapshot. Then immediately removes the VM snapshots and continues to do the backup.

Is that right? What would be the purpose of a VM snapshot if you are also doing the storage one?

On this job I wanted I have the Integration tab set to Enable backup from storage snapshots and the other 2 options are disabled to limited processed VM count and failover to standard.
I assume that is the same on all my jobs as I never changed the defaults.

[Andreas Neufert]

Yes, we create snapshots on the VMs when needed.

In case of Backup from Storage Snapshot, this is the only way to get the Change Block Trackinginformation for our Incremental Forever processing (it is bound to the snapshot IDs).

In case of Storage Snapshot only processing we only create VM Snapshots when it is needed for correct consistency processing. For example VSS snapshot processing within the VM has a 30sec or similar timeout. In that time all the VMs have to go into the consistent state that belongs to the same storage volume + the storage snapshots need to be created in that time as well. This is impossible to do. This is why we create the consistecy within the VM, preserve the consistency in a VM snapshot, do the same for all the VMs selected on the same storage volume, create the storage snapshot and then release the VM snapshots.

In case of crash consistent snapshot only processing, there is no VM snapshot.
As well when you have your VMs on individual storage volumes with no other VM within the same job has VM disks on the same volume, we do not use VM snapshots then with Snapshot only processing.