Restore test with S3 compatible object storage

[archvector]

Hi,

I'm currently using a VBR v10 server with a scalable repository composed of:
- one performance tiers on local disk on site A
- one capacity tiers composed of an S3 compatible object storage on site B

Our customer is requesting to validate the backups by simulating the whole site A is inaccessible. He wants us to restore the data from the S3 compatible object storage on site B

We can't restore VBR server on site B since it's still performing daily backups on site A. So i was planning to install a fresh temporary VBR server on site B to do that test.
However this documentation : https://helpcenter.veeam.com/docs/backu ... ml?ver=110
is stating that the object storage must not be plugged to a secondary VBR server and it can lead to data loss.

Is there a way to restore my data from the S3 repository on site B without making any update on my S3 repository to ensure we don't expire/delete anything on it? (a sort of S3 readonly mode)


Thanks,

Note: my S3 compatible object doesn't support immutability yet

[dariusz.tyka]

Hi archvector,

put temporary your performance extent within SOBR into maintenance mode and initiate a test restore. All data will be pulled from capacity tier extent so from S3 object storage. Later on when finished testing exit simply maintenance mode.

[archvector]

Hi Darius,

Thanks for this suggestion. If i put in maintenance mode the performance extent, it means i cannot perform backup during that test (which is not feasible in my case, i cannot change the VBR site A configuration)

Or you mean restoring the VBR VM on site B and put in maintenance mode the performance extent in site B ? I'm not sure of the consequence of this action

[PetrM]

Hello,

Do you run some jobs in continuous mode? Why don't execute a restore test out of backup window?

Thanks!

[veremin]

If i put in maintenance mode the performance extent, it means i cannot perform backup during that test

Correct, currently there is no way to bypass this requirement - Capacity Tier cannot be managed simultaneously by multiple backup servers. So, you need to find a time period without backup activity running, put Performance Tier into maintenance mode and execute restore process (like Petr's suggested). Thanks!

[archvector]

Hi,

@PetrM, no there is no replication jobs. Only huge file backups that run during the whole evening/night.
@veremin, Unfortunely the restore test from S3 takes is a long operation so i it means skipping at least backup day. However i will ask the customer.

Is the following process could lead to issues/data loss ?

1°) Stop all job schedules concerned by SOBR on VBR site A
2°) Put performance extent in maintenance mode on VBR site A
3°) Backup the VBR configuration from site A
4°) Restore the VBR configuration on site B
5°) Since the performance extent is in maintenance mode + the job schedules are not enabled, it shouldn't change anything on S3, right?
6°) Perform the restore on site B (i understand the data will be taken from performance tier automatically since performance tier is in maintenance mode)
7°) Remove maintenance mode + re-enable all job schedules on VBR site A to go back to normal

[veremin]

The process sounds OK, however, it is a bit overcomplicated in my opinion - after step 2 you can simply start restore process from VBR site A and confirm that it works fine for cloud restore points. Thanks!

[archvector]

The aim of the process is to validate the data is restorable if site A has crashed (thus there will be no more VBR on site A)

Thanks for your feedbacks, i will think about it

[veremin]

In this case you can skip configuration backup and restore steps, simply put source extent into maintenance mode on site A and import backups from object storage repository on site B. Thanks!

[Regnor]

Correct, currently there is no way to bypass this requirement - Capacity Tier cannot be managed simultaneously by multiple backup servers. So, you need to find a time period without backup activity running, put Performance Tier into maintenance mode and execute restore process (like Petr's suggested). Thanks!

Hi Vladimir,
what are the technical implications of having the object storage online on the primary and the DR VBR server at the same time?
The helpcenter isn't very detailed on this:

Otherwise, metadata on the object storage can get corrupted or out of sync, and you will not be able to restore data.

Where and how could the meta data corruption occur?One the primary server, the DR server or both?
Is the reason for this potential issue that the primary VBR servers adds/deletes restorepoints, of which the DR VBR isn't aware of?

A customers of ours plans to do a complete DR test from object storage.
Putting the object storage into maintenance mode would create an enormous backlog of offloads, as they have a daily change rate of multiple TBs and the restore itself will take some days at minimum.

[veremin]

During our tests we discovered that simultaneous access to backup and metadata objects could result in data being corrupted and out of sync - one backup server might commit a change that the other one will not know about. So we decided to block this behavior.

We are aware of this request (and the use cases behind it) and keeping an eye on it as a potential enhancement for future product versions.

Thanks!

[Regnor]

Thanks for your reply.
Perhaps you could implement a read-only flag for object storage, which masks all permissions except for list and get (in case of S3). I think after v12 with direct object storage, testing/restore cases will increase so it would definitely be a useful addition.

[veremin]

Yeah, you might be right and the demand for this functionality will increase moving forward. So we will be forced to deliver it sooner rather than later. Thanks!

[Regnor]

@Vladimir: If we limit the permissions via the IAM policy to prevent any changes, would it be supported to access the bucket in parallel? It was just asked on Reddit and I was wondering if we could just leave out
DeleteObject and PutObject.

[veremin]

We have never tested it, so one would not get official support, if something goes wrong. Thanks!

[Regnor]

Understood, thanks. Looking forward to any future updates/changes

[veremin]

Sure, we will update the thread, when (or if) we have more information to share. Thanks!

[claudiofolu]

Hello Veremin,
We are trying this same procedure and we cant get it to work.
Site A is a VBR with a SOBR with a performance tier of an inmutable linux server, and capacity tier is a AWS S3 bucket.
We followed steps 1 and 2, disable all jobs on site A, put in maintenance mode the performance tier in site A, then we add and import the object storage to site B VBR.

After that we lauch a restore for a VM, and it fails with the following errors:

Restore job failed Error: File does not exist. File: (XXXXX)
Restore job failed Error: Failed to open storage for read access. Storage (xxxx)
Restore job failed Error: Failed to restore file from local backup. VFS link:


Any clues?

[veremin]

Doesn't seem expected, so might be worth reaching our support team. Thanks!

[claudiofolu]

Thank you. Here is the ticket
Can the inmutability feature be cause? Or maybe because we are using two differents AWS keys?

[veremin]

The case number is missing.

Immutability should not affect an ability to import backups to a new backup server.

Not sure what is meant by two different AWS keys, though. Can you clarify?

Thanks!

[claudiofolu]

Sorry, here it is #06110298

What do you mind by "Immutability should affect an ability to import backups to a new backup server"

So there is a restriction regarding immutability? Its a DR escenario, were we lost out production backupp server, and we are attaching the capacity tier (s3) into a new backup server.

[veremin]

Sorry, there was a typo - "the immutability should not affect an ability to import backups".