Integration with Storage Snapshots : Tintri

[JimmyO]

Working for a global company and implementing Tintri VMstore in 3 regions I really see a need for storage snapshots integration for Tintri.

Sice we also (along with many other Tintri customers) plan to (or use) Veeam to back up all our Tintri VMstores, any priority on this functionality would be appreciated.

[lorengordon]

+1 for Veeam integration with Tintri snapshots!

[vClintWyckoff]

+1 for Veeam integration with Tintri VMReplication and VMSnapshots!

[ggrice]

Thought this may be interesting for some folks. I am running Veeam V7 on ESX5.5 backed by a Tintri T540. I am very happy with the backup speeds we are achieving, I run six jobs per night on an incremental basis, and two during working hours. Here is a pic I took of one of the backups from last night. Roughly 600gb of raw vm size, 35gb of changed data to backup and completed in just over 5 minutes We do this five more times every night and every job is running in under 10 minutes.



I checked the Tintri performance at that time and it was practically idling. Of course a lot of this performance is also due to Veeam's compression on the fly, but I can honestly say if you are holding out on buying Veeam or Tintri until there is a software integration - reconsider if you even need to as they work as a really great combo already.

[depps]

Snapshot integration becomes important when you have busy servers, VMWare snapshots are slow so the ability to leverage fast array snapshots is killer funtionality.

What the max number of snapshots the Tintri can have per VM?

We're evaluating both, I like the Tintri for its NFS/VWmare capabilities but Nimble seems to have more advanced snapshot capability and is cheaper. We want to keep hundreds of snaps of our SQL up to 15 minute increments and Nimble allows for almost unlimited snaps.

[veremin]

We want to keep hundreds of snaps of our SQL up to 15 minute increments

What you're going to snapshot SQL server every 15 minute? Are you after SQL point in time recovery? Thanks.

[depps]

For historical information and protection against corruption

RPO and RTO is currently seconds with CA/ArcServe RHA

Snapshot schedule/retention will be like this

15 min snaps kept for 1 week
daily's kept for 1 month
weeklies for 6 months
monthlies for 12-24 months
yearlies indefinitely, prob dump to tape or cheap storage a after 1 yr

[robgirard]

+1 for Tintri integration

[jwdiii]

+1 for Tintri integration with snapshots an replication

[ggrice]

Snapshot integration becomes important when you have busy servers, VMWare snapshots are slow so the ability to leverage fast array snapshots is killer funtionality.

What the max number of snapshots the Tintri can have per VM?

We're evaluating both, I like the Tintri for its NFS/VWmare capabilities but Nimble seems to have more advanced snapshot capability and is cheaper. We want to keep hundreds of snaps of our SQL up to 15 minute increments and Nimble allows for almost unlimited snaps.

Hi Depps, Tintri uses the VMware VAAI plug in so VMware snapshots are almost instant, I can snapshot my 1TB exchange VM in under 5 seconds with no degredation to the VM. VAAI completely unloads the servers so like I said Tintri + Veeam it super fast already. My Exchange cluster is on an 8 hour backup rotation, I just checked the Veeam settings, it would be possible to reduce this to every 15 minutes, although that does sound less like a backup requirement and more like a DR style requirement which might be better served with VM replication through Veeam, SRM, or even Tintri replication which happens with zero overhead to your front end infrastructure.

[vClintWyckoff]

@Daniel- Tintri T650 VMstore supports 50,000 VM Snapshots maximum on OS 3.0. As Geoff mentions, offloading vSphere cloning, snapshots and replication tasks off the hypervisor down to the storage layer completely eliminates all of the vSphere snapshot overhead and issues resulting from it (VM Stun / Commit...etc). I'm sure you're aware that this is much more efficient than the native VMware Hypervisor based snapshots. Nimble snapshots and cloning activities are provided at the entire volume / LUN level where as Tintri DOES EVERYTHING at the individual virtual machine and even individual vDisk layer, including providing dedicated QoS for that said SQL Server. But I'm sure you already knew that too

[mdaltoriob]

We have 3 sites with Tintri units. 2x540s and 1x650 currently. We are using Veeam to go to secondary storage for longer rentention, but I bet we could do some amazing stuff if we had integration with SAN snapshots.

Has there been any response from Veeam people on this?

[Gostev]

Has there been any response from Veeam people on this?

Please see my answer here.

And while Tintri clearly lacks market share to be our priority target for integration (note that we look at the actual market statistics, not the amount of +1s in this thread) , I am still curious where exactly do you see the value of Veeam integrating with Tintri? Based on the feedback from actual users on the previous page, Tintri's ability to offload VM snapshots to storage already solves that very same issue our Backup from Storage Snapshot feature is designed to address (namely, impact on production environment from large VM snapshots commits).

Perhaps there could be some lack of understanding here what our integration with storage snapshots is about... I would be happy to clarify this, if needed. Likewise, there might be lack of knowledge of some Tintri drawbacks on my part, too.

But reading through the feedback on the previous page, it sounds like there is absolutely nothing preventing Tintri users to do regular Veeam backups every 15 minutes today, without any sort of special integration. And perhaps the same can be applied to any SSD-based storage, really... because let's admit it - infinite IOPS will chew through anything regardless, even without any "special sauce".

[lorengordon]

Apologies, I was unaware that VAAI for NAS would offload a VMware snapshot to the array. My understanding was that the snap primitive was used to clone a VM -- the primitive names are sometimes misleading and confusing. I'm trying to find more information on exactly how the snapshot offload works. VMware KB 1021976 doesn't have nearly enough details. Is the VM still stunned when creating the snapshot? When the snapshot is deleted, is there still the potential for a very long commit period? I'm fairly certain the commit details are not significantly different, based on what I've seen (even though I didn't know I was using the offload feature), and commits are where all our issues have been with VMware snapshots.

As for the feedback on the previous page, we all know that there is a lot more to designing a backup architecture than just the primary storage device. The backup target design and network architecture design are incredibly important, as well. Also, 600GB is only 5% of the T540 capacity. In a rather small environment I was managing previously, we were backing up something like 10TB from a T540, and the backup target was an older DataDomain (daily incrementals and weekly fulls). We had a few very active SQL servers and a couple other high IO systems, and we would regularly have significant issues with backup performance and snapshot commits (and yes, we did have the Tintri VAAI for NAS vib installed to all hosts). The VMstore wasn't working hard enough to be a concern, and thankfully it's per-VM QoS is good enough that no other systems were ever impacted. Throughput issues were typically a result of job configuration + DataDomain, but throughput was secondary. The timing of the application issues we'd encounter could near always be traced to the snapshot commit period. In a much larger environment I helped design, we had EMC Clariion's with hundreds of spindles, and we'd have the same issues with snapshot commits.

In the end, my experience is that if your VMs are sufficiently busy, you will have problems with VMware snapshots, regardless of the storage solution and how fast it is. So yes, I advocate for avoiding VMware snapshots, and offloading snapshots and commits to the storage array entirely.

The ideal workflow to me would be for Veeam to quiesce the VM, then send the Tintri VMstore the command to snapshot the VM, then release the VM, then copy the snapshot from the VMstore to the backup target, then delete the Tintri snapshot. Note the data path there -- from the VMstore to the backup target -- not from the VMstore to the vSphere host to the backup target. Also note in this workflow that there may be very little interaction with vCenter or the vSphere hosts.

I see another use case for Veeam integration with storage/Tintri snapshots, though. It would be wonderful to utilize storage snapshots as the first tier backup, with a backup copy job to copy the snapshots to another storage device. Backups would then be nearly instant, and the backup copy of the snapshot would be "out of band" in a sense so it would not impact performance of the VM in any way.

[mdaltoriob]

Please see my answer here.

And while Tintri clearly lacks market share to be our priority target for integration (note that we look at the actual market statistics, not the amount of +1s in this thread) , I am still curious where exactly do you see the value of Veeam integrating with Tintri? Based on the feedback from actual users on the previous page, Tintri's ability to offload VM snapshots to storage already solves that very same issue our Backup from Storage Snapshot feature is designed to address (namely, impact on production environment from large VM snapshots commits).

Perhaps there could be some lack of understanding here what our integration with storage snapshots is about... I would be happy to clarify this, if needed. Likewise, there might be lack of knowledge of some Tintri drawbacks on my part, too.

But reading through the feedback on the previous page, it sounds like there is absolutely nothing preventing Tintri users to do regular Veeam backups every 15 minutes today, without any sort of special integration. And perhaps the same can be applied to any SSD-based storage, really... because let's admit it - infinite IOPS will chew through anything regardless, even without any "special sauce".

I guess I am confused here. If I use Veeam to backup VMs that reside on the Tintri that have Tintri snapshots I still get long commits that can cause brief outages. This may be an issue with the secondary storage I am using, but if I were working off of a storage snapshot from the beginning wouldn't I not need to worry about commits on a running VM? When I do a Tintri snapshot that quiesces the VM it stay open for only the briefest possible time. If Veeam could backup from a Tintri snapshot that happens 15 minutes before the job starts then we would not impact production VMs at all from a VMware snapshot perspective. If I am wrong please help me understand.

[Gostev]

From what I've been told previously about Tintri, I was under impression that Tintri's ability to perform snapshot on VM level, and offload the actual snapshot processing to storage, which eliminates the VM snapshot commit issue. If you say this is not the case as per your own experience, then perhaps those people did not know what they were saying.

[acoman]

My vote goes to Tintri as I'm already using that smart storage and will help us in our VEEAM day to day replication!!

+1 for Veeam integration with Tintri VMReplication and VMSnapshots!

Cheers

[ggrice]

I guess I am confused here. If I use Veeam to backup VMs that reside on the Tintri that have Tintri snapshots I still get long commits that can cause brief outages. This may be an issue with the secondary storage I am using, but if I were working off of a storage snapshot from the beginning wouldn't I not need to worry about commits on a running VM? When I do a Tintri snapshot that quiesces the VM it stay open for only the briefest possible time. If Veeam could backup from a Tintri snapshot that happens 15 minutes before the job starts then we would not impact production VMs at all from a VMware snapshot perspective. If I am wrong please help me understand.

Are you using the Tintri VAAI plugin? it makes a massive difference as it essentially converts your VMware snapshot to a Storage level snapshot. It's an add in that you need to install through Update Manager to each host that is accessing your Tintri filer.

The longer you leave a snapshot open the longer it takes to commit as it has to roll up more changes in to the image, but in the space of what is required for my backups I still get very fast snaps and commits.

[lorengordon]

When the snapshot command is issued from the vSphere interface or the regular vSphere APIs, I'm fairly certain that it is a traditional VMware snapshot that is created. The snapshot is not offloaded to the array in any way. From what I can tell, the VAAI snapshot offload capability is a special command available only through the API, and it's leveraged by VMware View (VCAI) and by vCloud Director. This information comes from http://www.vmware.com/files/pdf/techpap ... ration.pdf. If anyone can find anything stating otherwise, I would love to know.

Until then, again, +1 for Veeam integration with Tintri snapshots.

[tsightler]

I think it's important to remember that with the current implementation of Veeam's Backup from Storage Snapshot feature, we still require taking a VM snapshot because it's the only way to get access to the CBT data. If the problem is that you're VM cannot deal with snapshot at all, especially during commit, then even with BfSS the problem may not go away. Certainly it can be a huge help because the snapshot is not held very long.

The simple reality is VMware should have addressed their poor approach to snapshot a LONG time ago, although based on the announcement at VMworld it looks like they are finally providing features that should address this issue in future versions. Sure, Veeam could simply not use CBT at all, take hardware snapshots and then scan the entire VMDKs each time, but that's not really very scalable.

[vClintWyckoff]

@Loren / @Daltorio - @Geoff's correct. Yes you can offload the Snapshot from the Hypervisor through the use of the VAAI NAS Primitives - the Tintri VAAI plugin needs to be present on your ESX hosts (You download this off of your Tintri Support Portal).

In fact, Tintri is one of the only few storage vendors to support the Horizon View Composer API's for fast provisioning of VDI linked clones. Nevertheless, it's quite simple to install the VAAI. This will allow you to take VM level snapshots through the vSphere Web Client.

[Gostev]

Note to self: I wish Clint was as active on the public forums during his many years at Veeam, as he is now

[lorengordon]

@Loren / @Daltorio - @Geoff's correct. Yes you can offload the Snapshot from the Hypervisor through the use of the VAAI NAS Primitives - the Tintri VAAI plugin needs to be present on your ESX hosts (You download this off of your Tintri Support Portal).

In fact, Tintri is one of the only few storage vendors to support the Horizon View Composer API's for fast provisioning of VDI linked clones. Nevertheless, it's quite simple to install the VAAI. This will allow you to take VM level snapshots through the vSphere Web Client.

Clint, there's definitely something missing somewhere. Can you provide a reference? I can't find any documentation that says the VCAI snapshot offload is exposed via the vSphere Web Client. Do you mean the Tintri plugin for the vSphere Web Client, http://www.tintri.com/sites/default/fil ... asheet.pdf? Also, even if the capability is exposed through the web client only, that would seem to imply there is a different mechanism for triggering an offloaded snapshot. If that is correct, can we confirm that Veeam is issuing the command that triggers that capability, rather than the standard snapshot?

Here's the only information I can find on enabling and using the VCAI native snapshots, http://kb.vmware.com/kb/2061611.

[ggrice]

When the snapshot command is issued from the vSphere interface or the regular vSphere APIs, I'm fairly certain that it is a traditional VMware snapshot that is created. The snapshot is not offloaded to the array in any way. From what I can tell, the VAAI snapshot offload capability is a special command available only through the API, and it's leveraged by VMware View (VCAI) and by vCloud Director. This information comes from http://www.vmware.com/files/pdf/techpap ... ration.pdf. If anyone can find anything stating otherwise, I would love to know.

Until then, again, +1 for Veeam integration with Tintri snapshots.

Hi Loren, the VMware article you cited is very relevant . On page 3, Introduction to VAAI it states:
"VMware vSphere® Storage APIs – Array Integration (VAAI), ... enable the ESXi host to offload certain storage operations to the array, which reduces resource overhead on the ESXi hosts and can significantly improve performance for storage-intensive operations such as storage cloning, zeroing, and so on....."

Tintri's documentation also show that the snapshot is offloaded to the array:


This all happens at the level of the VMware Host and it is not a special function that is only available to View, VCloud director, web client or whatever.

Full Vsphere bypass for creating/storing backups would still be neat/fun/interesting to see, IMO Tintri could be the perfect platform for this type of integration because everything is managed at the level of VM's and not whole volumes.

BTW, thanks Gostev for making me post of the week in the last Digest

[depps]

Am I also wrong in assuming that Veeam Storage Snapshots would also add Application Aware snapshots? That is a huge advantage.

Is anyone here backing up from/to a Nimble array? During our Nimble POC vmware snapshots continued to cause VMWare stun...

[depps]

I did just do some digging and it appears that essentials plus isn't licensed for VAAI which probably explains why I still saw vm stun.

[LiamGP]

We use 2x Tintri T650's and would love to have full Veeam integration with Tintri snapshots too please!

[vClintWyckoff]

Just wanted to close the loop so there's no confusion. Tintri snapshots taken directly through the VMstore UI don't require VAAI, since you're already making the requests through the storage infrastructure. VAAI allows the hypervisor the ability to offload those snapshot requests to the storage layer...hence the requirement of the VAAI Plugin @ the hypervisor level. Yes, THESE snapshot requests are being made through vCenter and WILL USE VAAI, regardless if you have the Tintri Web Plugin. The Tintri Web Client Plugin exposes the per VM-Statistics and end-to-end per VM Visualizations along with the ability to add Tintri VMstores very quickly to multiple ESX hosts.

Veeam uses the Data Protection API set provided by VMware, which would leverage traditional hypervisor snapshots as well as technologies like HotAdd.

Here's link a to the VAAI Plugin release notes from the Tintri site:

https://tintri-support-static-us.s3.ama ... ionId=null

[Gostev]

All, I am splitting this discussion into the dedicated topic, since last two pages is nothing but Tintri talk anyway.

BTW, thanks Gostev for making me post of the week in the last Digest

You are very welcome, although it's not me - the post of the week is selected automatically by the number of likes

[lorengordon]

Hi Loren, the VMware article you cited is very relevant . On page 3, Introduction to VAAI it states:
"VMware vSphere® Storage APIs – Array Integration (VAAI), ... enable the ESXi host to offload certain storage operations to the array, which reduces resource overhead on the ESXi hosts and can significantly improve performance for storage-intensive operations such as storage cloning, zeroing, and so on....."

Tintri's documentation also show that the snapshot is offloaded to the array:

Geoff (and Clint), "offload certain storage operations to the array," does not necessarily mean that snapshots are offloaded to the array. The information in the Tintri release notes also does not specifically mention the offloading of VMware snapshots. I'm starting to feel like this is an emperor has no clothes moment. No amount of wishful thinking is going to help here, and the documentation is not proving all that helpful either.

So, what can we test? I wish I still had access to my old lab...how about this... Hypothesis: if VMware does offload the snapshot to the VMstore, then each VMware snapshot should create a Tintri snapshot. Test: Can someone with a Tintri VMstore go into the vSphere client and take a VMware snapshot of a VM? Validation: Go to the VMstore interface and check for a Tintri snapshot. Take a couple screenshots. Make sure your hosts have the Tintri VAAI plugin installed first, of course.