InstantVM recovery and storage snapshots = WIN

[ChrisGundry]

Hey all,

I don't often get to post something positive because we tend to use a lot of the advanced features of Veeam and sometimes they have their issues, because not as many people use them I feel they are not quite as polished as they could be... However, I have been using Instant VM Recovery more and more lot lately, because I have found ways to use it that maybe were not it's original intention, and it is making my life a lot easier so I thought I would share some love!

Basically if you have production storage that allows you to take storage snapshots, such as a StoreVirtual or Nimble array, and you have configured scheduled snapshots then you can use Veeam B&R to perform Instant VM Recovery of the VM, back to your environment and it takes about 2 minutes. Veeam clones the snapshot to a new volume, adds that volume to the VMware host you specified, re-scans, finds the VM, renames/reconfigures it as requested and adds it to VMware, in under 2 minutes!

We have many snapshot schedules as well as backups and replicas and storage replication, but being able to power up an old copy of a VM from a storage snapshot and have production levels of performance is awesome. Today when tracking down a tricky issue I powered up 5 different versions of the same VM some from the last few days, one from last week, one from the week before and one from last month. All powered up as expected, all within 2 minutes and all running at normal levels of performance. I could then check for changes between the VMs within the dates I restored to, I found a difference and rectified it. I don't know how I would have been able to do what I did today without using Instant Recovery.

Instant recovery from backups does of course work well too, but from storage snapshots on production storage it just flies through and gives you really quick restore options. When someone boxed a server a a little while back I was able to restore it back to a storage snapshot within just a few minutes of being advised there was a problem. We kept the broken one online and servicing some requests until the restored one was tested and then switched over with no real down time to speak of.

In a similar vein I also love using 'Restore guest files' from storage snapshots. If you spin up a few snapshots from different times you can then browse them all through vPowerNFS/mount server using the local mount points, or as I do, use PowerShell to pull the files out, run searches, compare items/folders etc. Once you start doing it regularly it is a very powerful and handy process.

Anyway, good work Veeam!

Anyone else use Veeam to do some things that it was perhaps not quite intended for?

[HannesK]

Hello,
we are happy to hear that you leverage storage snapshots and that it works fast & easy for you.

We even thought a step further: for test / dev environments you can use virtual lab based on storage snapshots

Best regards,
Hannes

[ChrisGundry]

Yep, we have done several pre-upgrade test upgrades using vLabs and storage snapshots, it works really well.

[cbrasga]

Something Veeam should tout that not everyone may know about when it comes to Veeam’s SAN integration. Is that it can not only browse and restore SAN snapshots taken by Veeam, but also the ones taken just by the SAN’s snapshot schedule.

We actually don’t even use Veeam’s SAN snapshot jobs, we just browse and restore from our native 3Par SAN snapshots. That way we don’t have to worry about possible snapshot conflicts or failures.

[SBarrett847]

I would love this feature in Veeam for Hyper-V.

[ChrisGundry]

Something Veeam should tout that not everyone may know about when it comes to Veeam’s SAN integration. Is that it can not only browse and restore SAN snapshots taken by Veeam, but also the ones taken just by the SAN’s snapshot schedule.

We actually don’t even use Veeam’s SAN snapshot jobs, we just browse and restore from our native 3Par SAN snapshots. That way we don’t have to worry about possible snapshot conflicts or failures.

Definitely, we mostly use native array schedules as well, with just a few Veeam controlled snapshot jobs for greater visibility. We find that Veeams scheduling is not flexible enough for us generally, so tend to use the native schedules most of the time.

[aman4God]

Does anyone have any feedback about using the storage integration for all VM backups? I am trying to find some information to decide if it is better to do all backups with storage integration turned on or not. Some of the concerns would be we have large volumes of 20-50TBs and backup job may not have all the VMs from the volume in one backup job. Not all of our arrays are compatible with the functions discussed above so we can't browse snapshots on our Pure array. When Veeam prof services helped with the install they really only suggested using storage integration with large servers to limit the amount of time a snap was held in VMWare. Any feedback on personal experiences with this would be helpful.

[cbrasga]

@aman4god Storage snapshots help with reducing VMware snapshot consolidation & stun times for VMs with lots of data changes.

We use storage snapshots for all of our VMs without issues, we're only seeing the benefits.

I would recommend reading up on SAN Integration and doing some testing, then set it up how you see fit best for your environment.

[foggy]

Some of the concerns would be we have large volumes of 20-50TBs and backup job may not have all the VMs from the volume in one backup job.

Could you please explain why is having several jobs a concern for you?

[aman4God]

foggy, maybe the word "concern" isn't accurate. I don't think there is an issue with having multiple jobs. The aspect in question is we have multiple jobs built on the VMs and the folder structure inside VMWare as opposed to where they reside on the datastores. I was just trying to understand if it is a concern to have multiple jobs all taking multiple storage snapshots if you have 50 VMs on one datastore/volume and 50 VMs on another one. It would snapshot both and then run the backups and then the next job would do the same. That in conjunction with Veeam prof services making it sound like it is just used on larger higher change rate VMs, not necessarily all VMs. I would honestly like to employ this for everything we can, but just wanted to hear some successes or concerns and understand the process better. Thank you cbrasga for the link, I will read up on the details. We have done some testing and it works well, but it is not necessarily much faster, but there is definitely a lightened load on the VMWare subsystem.

[tsightler]

The general best practice for large environments is to use Backup from Storage Snapshot (BfSS) for only VMs which will significant benefit from it, i.e. VMs with higher change rate.

That's a general statement and there's lots to unpack there if you want to look at all of the technical details as to why this is the recommendation, but the simple answer is that BfSS requires a lot of complex operations on the storage system and, based on the exact job configuration and layout you can hit a lot of storage based limits that cause backup delays or failures. For example, some arrays have a fixed amount of total snapshots, or a fixed amount of activated clones, or simply limits on parallel snapshot operations, or number of connections to the storage system itself.

If you have a few hundred VMs, and a few datastores, it's probably not a big issue, however, as you get larger, and have lots of VMs and lots of datastores, and you don't design your jobs by datastore, you can literally end up taking 100's of storage snapshots during a backup window. I've seen this fall apart on some storage systems with as few as 700 VMs.

Not only that, but for low change rate VMs there's virtually no benefit to BfSS as it's almost certainly going to take longer to perform the vm snap/storage snap/vm snap remove/clone storage volume/mount volume/rescan volume operations (all functions that have to complete before the transfer of data can even start) that it will to just VM snap, copy data via NBD/Direct SAN, remove VM snap).

Because there are so many storage arrays models supported, and so many different limits even among those models, it's difficult to draw a line and say this is how many VMs before you will start to see problems. I've seen systems that easily handle a few hundred snapshots, and others that fall over with just a few dozen, so, to attempt to make a generic recommendation that works for everyone, when we implement Veeam we will suggest minimizing it's use to the VMs that get the most benefit from the feature, especially during the initial implementation.

However, you are more than welcome to begin enabling it on a per-job basis and seeing how you particular storage system handles it. It should become clear pretty quickly if you have any issues or see any benefits from it. That being said, I've see a few hundred VBR installs in my time here, and my experience is the customers that have the least issues use the feature sparingly, for the systems that really benefit from it. I've probably walked into a couple of dozen customers that were having high backup failure rates and the only thing we did was turn this feature off for 90% of their jobs to solve the problem and decrease the backup window to boot.

[aman4God]

Thank you for that response! That is exactly what I was looking for, outside of just starting to use it extensively and then beginning to see or experience issues.

[cfizz34]

Jumping in here for anyone using Veeam and PURE and Data Domina to have a complete ecosystem for backups/restores using the snapshots off the array for backups and recovery points.