VSphere 7 / Kubernetes Storage

[mkretzer]

Hello,

now that VSphere 7 focuses on k8s alot is there any hope for a deeper integration between Veeam and VMware/Kubernetes storage like first class disk?

Markus

[nielsengelen]

Hi Markus, it would be interesting to see/hear how you are using Kubernetes (or plan to).

[mkretzer]

Right now we are using Rancher-Provisioned Kubernetes on VSphere VMs with local persistent storage (in short LPS - for exaple for applications like elasticsearch) or NFS provisioner (for applications needing a specific backup state which we can provide by backing up the NFS VM via Veeam).
This is all not a perfect solution. LPS is only working as long as a node not fails and NFS is not really redundant.

The best solution would be to use the VMware provisioner with first class disk. In that solution an independent disk (first class disk) is attached to the nodes but can switch nodes as needed. As far as i understand VDDK already supports backing up first class disk and other solutions like kasten already support this.

Now Veeam just has to implement backing up these things on the disk level (independent from the VM). For example i would envision a backup Job backing up all vmdks on one specific Datastore Cluster by finding the attached VM, snapshotting these (if necesarry) and taking a backup.

[Woidla27]

I‘m also in evaluation of the VMware Kubernetes solutions and looking for a solid backup possibility of that first class disks.

[bkc]

I also have the same concerns. It seems like 2 years ago I tried to connect Veeam engineers with vmware engineers (who were then working on vsphere storage for docker, pre-csi times). I sure hope these 2 years haven't gone by with no work at all from Veeam on solving this need for k8s users.

Its no longer fair to say that k8s is only for stateless workloads. Vsphere CSI hosted pvs need to be backed up. What's the solution? As a long-time Veeam user I've come to expect Veeam to be ahead of the curve. Where's the beta?

I hope the 'upcoming announcement' from https://siliconangle.com/2019/09/04/vee ... y-vmworld/ wasn't "use Veeam agent on your master nodes"..

Sure, we could try https://velero.io/ to backup to S3, except we're going to be running our own S3 on Minio, on a pv in k8s, so that's not useful. I can't justify needing to duplicate all pv's just to get a backup with Veeam.

How about a pre-backup task that snapshots csi-backed pv (not sure that works), and attaches the snapshots to a helper vm that Veeam backs up?

Will need to keep careful track of which snapshot was associated with each pv.

[bkc]

follow-up, here's an enterprising person facing the same issue, but with synology https://www.reddit.com/r/vmware/comment ... lume_vmdk/

Looks like snaphots of vsphere csi pv doesn't work

--

I forgot to mention that our management wants to move from docker swarm to k8s sooner than later. They're bringing in 'consultants' who will certainly recommend pure-cloud rather than on-prem or hybrid. Having no way to back up pvs will no doubt bolster their case. That means bye bye to vsphere and veeam.

We're a small customer, I doubt Veeam will miss us, but I will miss Veeam.. always works great.

[mkretzer]

The funny thing is that as far as I understand VMware VDDK really supports first class disk and Veeam could use that functionality. I hope the problems are "how do you restore fcd in a user friendly manner" instead of "how can we switch to a per-container license model like our competitors and make more money".

Still even with cloud only I would not feel good when the backup is not under our control in a way that I can export the backup if necessary on prem or cross cloud. Never trust only one company even if it is a big one.

[tsightler]

Certainly the challenges are mostly technical. It's true that the VDDK can backup such volumes, but that's just step one. You basically just get a bunch of volumes with UUIDs and are left with trying to figure out how to restore/associate those volumes with existing infrastructure. You also have to figure out what type of metadata to store with these objects. The entire Veeam product base is built around storing systems, e.g. VMs, instances, servers, workstations. Even our selection criteria for jobs are based on selecting systems, not disks. Suddenly, with FCD backup, this concept is turned on it's head, and everything is about disks, but you still need to have some ability to associate these persistent volumes with something. Of course, there is some limited metadata available in the vSphere API, but the question is if this approach is really even that useful/functional.

The above might be OK in some simple cases when you have only a limited amount of persistent data per-application, but it certainly leaves a lot of work for the admin, especially in the restore cases, and it gets complex quickly in larger environments with complex applications that need specific ordering and consistency.
As we venture into the K8S world, it seems obvious that we need higher level visibility into what we are backing up and how the pieces of data fit together within the application, so that we can "do the right thing" for both backup and restore, and have enough data that backups are recoverable.

We're not ignoring the space, we believe that vSphere 7 and Tanzu will be a catalyst for increased K8S adoption, which is already quite strong, especially in the enterprise. Veeam continues to actively collaborate with VMware on the problems that needs to be solved in this space, but it will certainly take some time for these efforts to turn into a product within VBR.

Admittedly, licensing is also an issue that will have to be addressed.

[mkretzer]

@tsightler Even if in v1 it is only possible to restore the fcd in place or do a file-level recovery and restore to a new volume from there is much better than having *nothing at all*.
We are trying to survive with a combination of dynamic NFS provisioner and local persistent provisioner. It works well but it feels completely wrong (and possibly is)... But right now easy backup&restore is more important for us than total dynamic and automatic scaling of ressources.