vo365 backups in containers

[doubleddav]

Hi. For some time we've been running Veeam backups for Office 365 in Windows docker containers. We're moving away from this now, as there are some problems, and we have confirmed with Veeam support that this is not supported.

But, I'd like to post our experiences anyway, which Veeam suggested. I think it should be supported and the software should be designed with this in mind. Sorry this is long.

The very basic is as follows:

• Setup docker running on Windows Server

• Create a dockerfile. This is a text file that is basically step by step silent installs of Veeam and it's explorers - just the powershell components

• Build the docker image

• Create containers from this standard image

• Store your data in object storage (or other non local storage)

• Access and configure Veeam either from the GUI on another machine, or the api's, or powershell

Why? Here are some of the advantages, in theory:

• Doing this the Veeam way through Cloud Connect is horrible

• Everything boils down to a text file and an object store

• Your client's Veeam backup environments are identical

• Setup a new client by running one command

• Your containers are disposable and stateless

• Upgrades are done once when you build the image. Deploying an upgrade involves destroying and recreating a container - that's it

• Scalability and automation

• Run it in the cloud if you want (I tested it, it works)

Now some of the problems we faced:

• Not supported, you're on your own

• The containers aren't actually stateless, because the software hasn't been designed that way. You can work around this by mounting volumes, but you're still relying on Veeam's database, which really should be disposable

• We found weird issues with repository id's changing on a new container and strange network dropout every 10 seconds. This can be avoided by blowing away the database

• Recreating (upgrading) a container doesn't work as expected. The Veeam database is on one version but the software and proxy are suddenly on a newer version. Can't we just upgrade the database? No sadly it doesn't work

• Recreating a client's config from scratch is easy (and automate-able) but requires a full backup the first run. This puts load on your server, particularly if you upgrade all of your containers at once

I could go into more detail into what I tried to work around some of this, but I think this is long enough.

What I'd like to see in the future is vo365 being designed with containers in mind.
Has anyone else tried this? Would anyone else be interested in seeing this as a supported setup?

Thanks

[HannesK]

Hello,
and welcome to the forums.

Just to understand the "why": Do I understand it correct, that you are a service provider and you wanted to have multi-tenancy access with self-service without programming your own portal?

I believe that whatever you tried to do can be done without containers, we just need to understand the goal

When you talk about scalability: how many users are we talking about? 10k? 50k? 100k? 500k?

Best regards,
Hannes

[doubleddav]

Hi Hannes,

That's correct we're a service provider. I agree this can be done without containers, which is what we're moving to now.

I like containers because they're lightweight compared to a full server or a vm. I guess the goal is statelessness and automation. Containers are very suitable for this.

In terms of scalability I think it would go both ways. 5 users to 500k, running on a single HyperV box or multiple data centres - the basic setup and ideas are the same. For large setups the benefits should be obvious. At very large scales or in the cloud, you might create a container only when a backup runs or someone needs to restore, then destroy it again to save on resources.

For smaller setups, consider an example.

Suppose we have 1 HyperV server. We are running 10 Windows Server VMs, each with Veeam installed and running. I think many providers (not all, but generally speaking) at this scale would manually create new VMs, and manually log into each VM and update Veeam. Each VM requires a Windows license, you need to manage Windows Updates for each machine. If something happens to the Veeam database or one of the VMs, you log in and try and set everything up again, trying to remember what settings you used for each client. Maybe you try to solve some of this by having them all part of a domain and managing them that way. But now all of your client's machines are part of the same network. Even writing this sounds complicated.

Compare to single Windows Container server. No Windows licenses for each client. You setup the software once, deploy it for all clients. Windows updates are not necessary, a domain isn't needed, so nice separation of clients. Each container uses almost no disk space. Problems with container config are fixed once and deployed to everyone simply by recreating their container, which takes 1 second.

I don't want to go on and on, I hope that all make sense

Thanks

[HannesK]

Hi,
well, calling windows containers "lightweight" is a bold statement

I'm talking about your environment. If you have to back up 500.000 users, please talk to your local Veeam solutions architect. It's a massive difference backing up a hand full or half a million users.

A domain is not required anymore.

From my point of view, containers would increase the level of complexity especially for small customers. A stateless container with the app is only half of the product. There is still the data(base) that needs to be covered. Sure, if we would re-write the whole software, it would be possible. But that's unlikely to happen.

The main question is, why you have 10 VBO instances for such a small environment (I'm looking for the feature request behind the container request). One VBR server can scale up to 250k users https://bp.veeam.com/vbo/guide/design/maxconfig.html

Best regards,
Hannes

[doubleddav]

A domain is not required anymore.

Sorry I should have been more clear. In the context of my example, someone might have the idea to use a domain to manage all of the VM using policies etc

From my point of view, containers would increase the level of complexity

I have to disagree. One reason is we had actually set it up this way, not just in theory. Aside from the problems I mentioned, managing this was really really easy. And visualizing it was simple, there's a 1:1 relationship, 1 customer = 1 container. Everyone thinks differently, but for me this was much simpler than the setup in the guide for providers.

There is still the data(base) that needs to be covered

Exactly. Extending this to other Veeam products, and lots of other software in general. In my opinion a dependency on the database is a weakness and a single point of failure. We have definitely seen many examples of [problem with the database] = [backups stop working]. The database is (or should be?) just a cached reference to other stuff that exists elsewhere (eg. last backup size, username etc, name of last backup file etc). This info should be able to be easily recreated if the database is lost.

I'm looking for the feature request behind the container request

When I started looking at this, I didn't initially have containers in mind, it gradually grew that way as I came across different road blocks or things I just didn't like. I couldn't get vo365 working with Cloud Connect. We did have a meeting at the time with some guys at Veeam, who were definitely knowledgeable. It could be that back then things didn't work as well as it was a much earlier version, but in the end we concluded it wasn't worth doing it the way it says in the guide. So we looked to work around the limitations and containers solved a lot of the problems.

I also just didn't like having all clients and their jobs, credentials, storage etc in a single console. Having 1 client per container was clean and neat. Everyone was nicely separated from each other. A single client was a simple basic setup. Many clients were the same simple design repeated again and again. Anything to deal with scale was handled by other tools designed for scale - and not by Veeam.

Sure, if we would re-write the whole software

Sounds like a plan
Really I just wanted to see everyone's thoughts on something we tried and worked well for us (to a point), and potentially could work well for everyone. I don't necessarily expect the software to be redesigned.