veeam agent backup to immutable object storage repository

[mcz]

Hi everyone,

it's 2023 and as well all know, object storage is a very good repository. Cheap, flexible, huge capacity, reliable, almost bulletproof immutability, etc. So why not use it for agents? I have already done a test for a small PoC, but some questions are still unanswered to me:

• If a backup job runs that doesn't fit into the local cache (for instance the full backup), the object storage is immutable and you're not having the best bandwidth to it (like when using a public object storage of cloud providers), you might be in an endless loop where the backup job can't complete in time, say the user shuts down the device and the backup job is aborted - the same happened the next time the device is started. Because the repo is immutable, every block will stay there until the period has expired and will only then be deleted or even later. Correct or is there an advanced feature that would be aware of that fact?

• in addition to the last point, I assume there is no cache on the vbr-side that would temporarily store the backup so that the client can finish its job while the upload is still taking place?

As you might guess I'm having some notebooks in offices that don't support Gigabit uplinks to our S3 provider and so the backup would eventually take longer than users are connected with their devices. Now you might argue that restore isn't too fast then either, but as long as I can recover in one day, I'm totally fine (I can provide a spare device meanwhile). But if the backup job just never finishes and wastes a lot of money due to filling the bucket with blocks that are obsolete after the aborted job, it's just impossible to go that route...

What I could imagine is doing the initial backup to a local repo and move it then to the object storage, but not sure if that was ideal for the further backup flow...

Thanks in advance!

[mcz]

as an addition, there's the idea that the backup job doesn't abort when being shut down or when the repository isn't available, so that it could resume the day after or whenever. of course, there has to be a timeout, but in our case it wouldn't be that important to finish a backup job of an agent within an hour or so...

[Dima P.]

Hello mcz!

Because the repo is immutable, every block will stay there until the period has expired and will only then be deleted or even later. Correct or is there an advanced feature that would be aware of that fact?

The data block becomes immutable as soon as it's uploaded to object storage, so, generally, whenever the connection is interrupted and is not restored the block will sit in the object storage for the entire immutability period. I've said 'generally' because there is a resume functionality that can continue the backup where it was left. The requirements for resume to work - the backup should fail due to disconnect and retry itself to make it work.

in addition to the last point, I assume there is no cache on the vbr-side that would temporarily store the backup so that the client can finish its job while the upload is still taking place?

Correct, otherwise it would not be possible to upload data directly to object storage and make it truly immutable.

As you might guess I'm having some notebooks in offices that don't support Gigabit uplinks to our S3 provider and so the backup would eventually take longer than users are connected with their devices.

Possibly you can take a full backup while machine has a decent connection (say it's connected over LAN/WAN in the office) and only after that allow laptop to travel.

What I could imagine is doing the initial backup to a local repo and move it then to the object storage, but not sure if that was ideal for the further backup flow...

Could be an option too, sure!

as an addition, there's the idea that the backup job doesn't abort when being shut down

You can control via schedule how agent should behave if the backup schedule was missed due to shutdown but there is not option to retry when accidental power off was performed. Last one was already requested by few folks, so I'll add your vote to this feature request.

or when the repository isn't available, so that it could resume the day after or whenever.

When the agent is up and running and cannot connect we have 23 hours retry cycle for workstation license, so, yup, that should work.

Hope it helps!

[mcz]

Thanks Dmitry,

here are my answers:

Possibly you can take a full backup while machine has a decent connection (say it's connected over LAN/WAN in the office) and only after that allow laptop to travel.

this is very difficult to achive. I know that the base issue here is connectivity, but on the other hand you don't really control how much data would change since the last backup...

Could be an option too, sure!

So to be more precise: I'm having a workstation policy where agents would write to a local repository (ReFS). Now when I move that whole chain to an object storage and reconfigure the policy it just goes on as nothing ever would have happened. Correct?

You can control via schedule how agent should behave if the backup schedule was missed due to shutdown

Not really what I thought... Just like a vmware snapshot still remains when the vm is being shut down, I assume that the same happens or can be applied to a VSS snapshot. So if you shut down your windows notebook, veeam would probably be able to containue the same restore point/backup once it got bootet up again. So something like this:

=> veeam starts backup job and creates vss snapshot
=> user shuts down device, veeam service does stop, but keeps state and vss snapshot
=> device is started again after X hours
=> veeam service starts, remembers that it is in the middle of the backup process and proceeds with the backup job

Of course, sounds fancy, but probably possible??

Thanks!

[mcz]

ok, just tried to move agent backups to the immutable object storage and then I received the following errormessage:

This operation is not supported for backups on object storage repositories created by standalone agents and Managed by Agent backup jobs.

Too bad. This means that we cannot seed the backups. Any ideas or other options? Thanks.

[mcz]

another fact to note is that I only have the "managed by agent" option as soon as I choose the workstations, so there is no way to escape...

[mcz]

And there's another clarification needed on my side: When there's a running backup job to an object storage, client looses connection to the server and the "rest" of the backup would fit into its local cache, it will write it there, synchronize and finish it later even when the object storage was immutable. Correct?

Thanks!

[Dima P.]

another fact to note is that I only have the "managed by agent" option as soon as I choose the workstations, so there is no way to escape...

Unfortunately as we've discussed. We do our best to remove this limitation in the next versions.

When there's a running backup job to an object storage, client looses connection to the server and the "rest" of the backup would fit into its local cache, it will write it there, synchronize and finish it later even when the object storage was immutable. Correct?

Yes and no. At first agent will try to resume the connection with OS, if that's not possible and remote location is unreachable agent will switch to a backup cache, make a local backup and synchronize it later as soon as connection is established.

[mcz]

agent will switch to a backup cache, make a local backup and synchronize it later

does it create a new backup or continue on the "interrupted one"?

last question (hopefully): Assume a backup job gets interrupted and is re-started at a later point: Most of the time, a huge amount of data is unchanged. Now the restore point on the OS is incomplete but the blocks there don't have to be uploaded again if they are identical - is this deduplication considered even with incomplete restore points? Because I don't see the deduplication in terms of upload speed (say if you had 50 GB already uploaded, those 50 GB should be processed at a much higher speed than it did before)...

Thanks!

[Dima P.]

Hello Michael,

It depends. If the backup job got interrupted and then made a retry by itself, then the incomplete backup on the repository will be reused and continued. If the job was interrupted and failed (for instance machine was shut down or rebooted) - the backup will be created from scratch and the incomplete backup will be deleted from the repository.

last question (hopefully): Assume a backup job gets interrupted and is re-started at a later point: Most of the time, a huge amount of data is unchanged. Now the restore point on the OS is incomplete but the blocks there don't have to be uploaded again if they are identical - is this deduplication considered even with incomplete restore points?

Depends on the job behavior - if backup is resumed we can say that deduplication will take some effect. If not and restore point will be recreated from scratch the data will be deduplicated but excluding what's been stored in the incomplete restore point because it was not finalized properly and can be considered as corrupted.

[mcz]

Thanks Dmitry,

but excluding what's been stored in the incomplete restore point because it was not finalized properly and can be considered as corrupted

Hmm... The whole point is for sure "corrupted" because it wasn't finished, but the individual blocks are probably written in one transaction, so they could be re-used. Do you agree? If so, please consider it as a feature request.

It's just very benificial if you're having a small uplink to the OS, then A backups will finish faster, B they will finish (otherwise it would probably never get to the finish line) and C you can reduce costs as wasted blocks don't stay on the OS for the whole immutability period.

Thanks!

[Dima P.]

Hmm... The whole point is for sure "corrupted" because it wasn't finished, but the individual blocks are probably written in one transaction, so they could be re-used. Do you agree? If so, please consider it as a feature request.

Totally agree and we are planning to discuss the possibility of re-using the incomplete backup with RnD folks. Thank you for the feedback Michael!

[mcz]

Thanks Dmitry, as always: it's a pleasure