Any Experience Backing up with VVOL's?

[gpomanti]

I was wondering if anyone has any experience with VVOL's and backups. We will be upgrading to vSphere 6.0 soon and will definitely be looking into VVOL's. Has anyone been backing up/replicating VM's with VVOL's? If so what have you seen and how does the backup performance compare? Any other general VVOL information would be greatly appreciated!

[Vitaliy S.]

I'm not sure there is much information about vVols yet, but this quick search might give you some hints on the future deployment.

[Gostev]

After getting to know VVols better, I personally am not impressed with v1 architecture and functionality limitations, and would not use them myself. This technology needs to mature - A LOT.

[gpomanti]

Thanks for the replies! Gostev, is there anything in particular that you are not impressed with, or is it just that the overall "version 1" is not quite ready yet?

[Gostev]

Sure, I can share a few particular things.

From the architecture perspective, I don't like that VVols are not transportable. Basically, all metadata is stored inside vCenter Server, in the VASA 2.0 provider database. This means that if you lose vCenter, or get some unrecoverable database corruption - you will also lose access to all of your data. Forever! Because you can't just plug your VVols storage into another host to get to your data, such as the case with VMFS. Most people probably don't even realize this limitation, they take it for granted that their storage will be readable by any vSphere host (because this is the case with VMFS, and basically any other file system).

And then, there are some random functionality limitations which will need to be removed to make VVols generally usable. For example, inability to perform direct SAN backup (at all), or hot add backup (for virtual disks over 2TB). So, you are stuck with NBD backup of those huge disks - which is super slow, so good luck with backing up those large VMs with multi-TB disks.

Again, it's a promising technology overall, but v1 is always v1.

[joergr]

One thing more to mention - if you lose your VASA provider (some SAN vendors put their VASA providers in virtual appliances), your VM residing on VVOLS do stay on and online and can communicate with their residing VVOLS. However - the moment you shutdown one of these VMs with the VP still absent the VM will go grey and you will not be able to turn it on again till the VP comes back. You too will not be able to do many things with this VM once it´s turned off till the VP comes back. So - if you happen to have your VP VM placed on a VVOL Datastore and at some time this VP VM is down you will have an intersting time i guess. With VVOLs it is extremely crucial to have very regular backups of a) your vcenter server and b) your vp. This has to be taken care of in the future. I see two ideas which will do that: a) VMware integrates very regular auto-backups of all vvol-metadata to every single esxi connected to it or b) the vvol interop design is expanded to the fact that the san vendors will take care of all vvol metadata inside the protected san storage. Ideal would be both.

[dellock6]

Indeed all the moving parts involved into VVOLs are somewhat scary to think about. I can't imagine all the possible combinations of data loss into the vcenter only, in the VP only, in both, corruptions in data, misalignments between the two, and so on. I hope there's at least a way for the two to re-align informations if there's a restore to be done...

In a way the design with the VP running in a VM (netapp for example) makes backup of their data easy, what about when the VP/PE is the storage processor itself? It's easier to design its availability as the dual controller itself offers this capability, compared to maybe having to leverage multi-processor fault tolerance into vSphere for the virtual appliance running VASA, but on the other end do storage vendors have designed a backup solutionfor the VASA informations?

Also the problem is that few sources are listing all these really needed informations. I hope it's not because VMware doesn't want to let people know too much these limits, as it would be really dangerous to let people use VVOLs at least without being aware of this.

[joergr]

There is one intersting post by netapp - take a look....

https://library.netapp.com/ecmdocs/ECMP ... 9C322.html

Best,
Joerg

[dellock6]

I've seen that one during my researches
In these regards it makes sense indeed to have the VASA provider as a VM, as long as you don't place it inside a VVOL...

[joergr]

I will ask a few folks at vmware dev during vmworld europe especially regarding this. There has to be something on the horizon to take care of this metadata to keep it 100% secure and at best on many places at the same time without depending on a vcenter or a vp appliance.

[dellock6]

I was going to leverage some contacts I have too, maybe we can meet at VMworld Europe after our searches and share the info we collected. As I like the concept of VVOLs, these "limits" are huge roadblocks to adopt them in a production environment.

Maybe (suggestion for Anton ) if we understand how to extract and restore these metadata from both location, we can improve Veeam to become a tool to also protect VVOL metadata together with the VM itself. Our support today is based on the VADP libraries but I'm not sure those libraries also allow for extracting those info, to me seems they just allow for the extraction of VM data from the VVOLs.

[joergr]

These are very good ideas from my point of view

And yeah lets meet in Barca.

[gpomanti]

Thanks for all the information! I was unaware of the issue with data loss if you lose vCenter. That sounds like kind of a big deal to me I guess we will hold off on VVOls for now.

[Cormac]

A few people have reached out and asked me to comment on this post.

Its probably worthwhile clarifying a few things first.

In the context of VVols, the VASA provider should ideally be stateless. In other words, if it goes away for some reason, this should have no impact on your VM I/O. Yes, it will impact certain operations such as creation of new VMs, snapshots, etc, but will not impact the current running VMs. This is because the VASA API calls are currently out of band from ESXi to the storage array via the VP. Now a VASA provider can be configured in a HA mode, so that one VP is the primary and the other(s) are backups. This is what we have done with VSAN which also uses a VP. So this would be a good question to ask your storage vendor .. can your VP be setup in a HA configuration? If not, then you would just need to deploy out a new one. Now, I prefaced the start of this response by saying "ideally", the VP would be stateless. There is no need to storage anything on the VP. However that is not to say that any of our storage array partners have put something into their VPs that would no longer make them stateless. Yet another question to ask your storage vendor.

The other query is in relation to vCenter server. The VVol information that is stored by vCenter server is related to the policies (SPBM) that the VMs are using. Again, if vCenter goes away, this has no impact on the running VMs. The policy information associated with the VM continues to be used by the VMs. The issue is that if you do not have a vCenter server backup and you deploy a new vCenter, then it will not have the policy information for your running VMs. So these policies will need to be recreated, which is possible, though it might be a little tedious/manual. Again, we have ways of doing this for VSAN, and I suspect it is the same for VVols (though I admit I have not tried this scenario). I suspect that this might be why it is a good idea to backup your vCenter (or the VC db).

I'm a bit confused as to why people feel that failures in either of these components can lead to data loss in VVols. I/O to VVols does not depend on vCenter or the VASA Provider. These are components needed for configuration and management, but should not lead to any sort of data loss issues if they are not present. The paradigm with the VVol architecture is that the storage is the source of truth, whilst the VP is your guide. We would hope that the storage vendors do due diligence on their respective implementations, but just like the VAAI implementations of the past, that will probably vary from vendor to vendor.

Hope this helps.

[dellock6]

Thanks Cormac for coming here!

I think the "worst" worry is surfacing here is that vCenter holds the "mapping" of the VVOLs belonging to a given VM, and as such if we loose vCenter, the VVOLs are just a bunch of small luns without any sense. You are now saying that these info are IN the VVOLs themselves? Say it differently, if you blow away completely vCenter and I do not have a backup, I just deploy a new vCenter, mount again the storage container + PE + VASA provider, and after a rescan I see again all the VMs with all their virtual volumes? This sounds comforting if it's so.

Indeed loosing SPBM is not ideal, but it's still better than loosing a VM at all.

Thanks,
Luca

[Cormac]

That is exactly how I understand it Luca. There is definitely metadata associated with each VVol, but one would expect this to be persisted at the storage layer by the storage array vendors. This would not/should not be persisted in vCenter in any way, and one would also hope that this would not be persisted in the VASA provider, but actually persisted down on the storage layer (again, a great question to ask the storage vendor).

So to answer your question, yes, it should only be a matter of deploying out the new vCenter, discovering the hosts/storage, and since the VMs are already bound to PEs which have not changed, you only need to worry about recovering the SPBM stuff.

HTH

Cormac

[joergr]

Thanks so much, Cormac, for joining the discussion.

I have a suggestion / idea:

How about I create a testlab in the next 1-2 weeks, consisting of a) one DELL-EQL array with FW 8.0.4 (fully VVOL capable up to three group members), a dell r730 acting as exi 6u1, create a virtual vcenter 6u1, create a virtual VP (dell eql VP). I will then create a VVOL store, add some VMs to it. Then i will destroy (yes DESTROY, totally delete so i myself can´t get it back) the VC VM and also the VP VM. Then i will try to recreate a fresh VP and a fresh VC VM and will try to somehow get access to the VVOL store i created before. That is gonna be an interesting task i think. What do you guys say? Would this be worth the work or was this already done somewhere in the vmware labs before? I am speaking of losing BOTH, the vc and the VP. But not the array of course. If yes, please, Cormac, let me know, then i don´t have to do it. But if you haven´t tried that scenario i would love to check that out.

Best,
Joerg

[dellock6]

I think is a nice idea indeed, and you can even use the VNX virtual appliance to do it even quicker. however, I still would prefer an official answer from VMware, and I'm sure Cormac can find it internally. I understood there are some degrees of freedom in the implmentation of the VVOLs design, but as much as VMware has pushed some strict rules on some design aspects, also the exact placement of VVOLs "pointers" (for a lack of a better term)are inside the VVOL itself. As Anton said at the beginning of the thread, VMFS has always been good cause it was self contained: every needed information (vmx and the linked files) are all in the filesystem, regardless what you connect to it.
Cormac is saying this should be the case also for VVOLs (I guess those info are in the config VVOL at this point, and with it you can trace the other VVOLS belonging to the same VM), but is this "it should" that still worries me. It means that a VVOL implementation could be potentially awesome or extremely dangerous (and I'm being polite...) depending on the storage vendor. I'd prefer VMware to force storage vendors to follow some design principles, and if not, the storage not being recognized and mounted by vCenter.

And by the way, since all the connection is done by vCenter, it means we cannot have a VVOL directly attached to a single ESXi I guess.

Anyway, this is becoming a really interesting conversation, the first in many months here in the forums.

[joergr]

Indeed, Luca, this discussion is one of the most intersting

So it would be very interesting to know how much freedom the storage vendors have designing their VVOL capable implementations. I could imagine if it´s possible to stuff every kind of metadata on a virtual VP and literally no metadata at all - only the blocks with no info whatsoever to the actual array, that could be a very bad one, because the data is lying there it´s just no one can read it anymore in a logical way. On the nother hand, if the design dictates to store the logical assignment at least per VVOL, maybe in the config VVOL so that the internal implementation of the array can "understand" that - that would be a good one. There may also be a scenario thinkable where indeed there is only raw data on the array and the array knows nothing about that but there is some kind of scan-tool which can re-discover the logical VVOL-contents. I also can imagine it would be easier to get hands on VVOL data in case of an emergency lying on file based SAN like NFS because at the end of the day it´s only mount points and directories. On the other hand, on block based array - i think some special tool would be needed maybe.

And for the future i am sure there has to be a way to connect an ESXi standalone to a VVOL capable storage array and somehow read the VVOL contents. That also would make data recovery sessions much more effective.

[Cormac]

Let me ask one of my contacts at DELL EQL to comment on this scenario. I'm pretty sure they would have tested it, or if they haven't they certainly should

[joergr]

Thanks a lot, Cormac, highly appreciated. Netapp would also be interesting - same approach.

[dellock6]

Thanks Cormac, so it's confirmed that at least as VMware designed vvols, the complete loss of vcenter is not an issue to restore them? Then we can talk about "non optimal" implementations by storage vendors (even if I guess then what the vmware certification is checking before giving the ok), but there should be a "this is how it work" from you. After all you designed the libraries...

Thanks again for the help Cormac

[deevie]

I had been testing vvols on vsphere beta 6.0 with a 3PAR. When GA came out, I did setup a brand new vcenter and was able to re-add the existing vm's (like I would do with a VMFS datastore). Off course, in my case all crusial data is on the 3Par itself (VP). The 3par knows which datablocks belongs to which vm.

[dellock6]

Thanks for your input.
But then the question is: what if you loose data into the VP? Is there any backup available in the 3PAR machine for the metadata stored into VP? Even if this honestly is not any different than loosing a raid conf from a raid controller and have disks that makes no sense without the controller knowing how to group them.

Luca

[deevie]

Shouldn't you trust your storage vendor that the don't loose data within their system? If you create a volume on a SAN storage system for a VMFS datastore, you also trust it won't loose datablocks that belong to that volume. Same for the vmware vvols.

[bmeadowcroft]

Thanks Cormac, so it's confirmed that at least as VMware designed vvols, the complete loss of vcenter is not an issue to restore them?

That is correct, confirmed (as VVol PM). You will lose the SPBM policies you have created but not the VMs or their VVols, if you spin up a new VC you will see the VMs when you restore the VVol datastore to the inventory. Recovering the policies should be possible by restoring from a backup of VC or by using the Import-SpbmStoragePolicy PowerCLI cmdlet to import a previously exported XML representation of the policy information that was generated by the Export-SpbmStoragePolicy cmdlet.

Then we can talk about "non optimal" implementations by storage vendors (even if I guess then what the vmware certification is checking before giving the ok)

I will defer to the storage vendors concerning their best practices around VP availability and resiliency. VMware also provides a VASA HA capability that vendors can implement using multiple redundant instances of their VP (which we can certify and is listed on the VVols compatibility guide under VPHA, see https://www.vmware.com/resources/compat ... ureCats=39). This may not be necessary if the vendor runs the VP on their array itself however so lack of a listing there does not imply that they don't have their own availability solution, as I said talk to your vendors!

[Peter Faber]

is there any news on this subject in v9 ?
i understand that in 8.0 u2 backups of vVols in SAN mode is not supported, is this still the case in v9 ?

Regards Peter

[foggy]

Yes, nothing has changed in this regard.

[DaveWatkins]

Foggy, could you expand on where the problem is? Is it just that VMWare hasn't released API's that let you talk to the VASA provider and PE to get access to the associated LUN's, or is it something more fundamental about the VVVOL architecture that doesn't allow you to access those LUN's.... or is it something else?

Thanks

[foggy]

VDDK just does not have support for that mode yet, only hotadd and nbd are supported for vVol.