Discussions related to using object storage as a backup target.
Post Reply
archvector
Enthusiast
Posts: 33
Liked: 3 times
Joined: Mar 09, 2021 7:53 am
Contact:

Restore test with S3 compatible object storage

Post by archvector »

Hi,

I'm currently using a VBR v10 server with a scalable repository composed of:
- one performance tiers on local disk on site A
- one capacity tiers composed of an S3 compatible object storage on site B

Our customer is requesting to validate the backups by simulating the whole site A is inaccessible. He wants us to restore the data from the S3 compatible object storage on site B

We can't restore VBR server on site B since it's still performing daily backups on site A. So i was planning to install a fresh temporary VBR server on site B to do that test.
However this documentation : https://helpcenter.veeam.com/docs/backu ... ml?ver=110
is stating that the object storage must not be plugged to a secondary VBR server and it can lead to data loss.

Is there a way to restore my data from the S3 repository on site B without making any update on my S3 repository to ensure we don't expire/delete anything on it? (a sort of S3 readonly mode)


Thanks,

Note: my S3 compatible object doesn't support immutability yet :cry:
dariusz.tyka
Enthusiast
Posts: 54
Liked: 4 times
Joined: Jan 21, 2019 1:38 pm
Full Name: Dariusz Tyka
Contact:

Re: Restore test with S3 compatible object storage

Post by dariusz.tyka »

Hi archvector,

put temporary your performance extent within SOBR into maintenance mode and initiate a test restore. All data will be pulled from capacity tier extent so from S3 object storage. Later on when finished testing exit simply maintenance mode.
archvector
Enthusiast
Posts: 33
Liked: 3 times
Joined: Mar 09, 2021 7:53 am
Contact:

Re: Restore test with S3 compatible object storage

Post by archvector »

Hi Darius,

Thanks for this suggestion. If i put in maintenance mode the performance extent, it means i cannot perform backup during that test (which is not feasible in my case, i cannot change the VBR site A configuration)

Or you mean restoring the VBR VM on site B and put in maintenance mode the performance extent in site B ? I'm not sure of the consequence of this action
PetrM
Veeam Software
Posts: 3229
Liked: 519 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Restore test with S3 compatible object storage

Post by PetrM »

Hello,

Do you run some jobs in continuous mode? Why don't execute a restore test out of backup window?

Thanks!
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore test with S3 compatible object storage

Post by veremin »

If i put in maintenance mode the performance extent, it means i cannot perform backup during that test
Correct, currently there is no way to bypass this requirement - Capacity Tier cannot be managed simultaneously by multiple backup servers. So, you need to find a time period without backup activity running, put Performance Tier into maintenance mode and execute restore process (like Petr's suggested). Thanks!
archvector
Enthusiast
Posts: 33
Liked: 3 times
Joined: Mar 09, 2021 7:53 am
Contact:

Re: Restore test with S3 compatible object storage

Post by archvector »

Hi,

@PetrM, no there is no replication jobs. Only huge file backups that run during the whole evening/night.
@veremin, Unfortunely the restore test from S3 takes is a long operation so i it means skipping at least backup day. However i will ask the customer.

Is the following process could lead to issues/data loss ?

1°) Stop all job schedules concerned by SOBR on VBR site A
2°) Put performance extent in maintenance mode on VBR site A
3°) Backup the VBR configuration from site A
4°) Restore the VBR configuration on site B
5°) Since the performance extent is in maintenance mode + the job schedules are not enabled, it shouldn't change anything on S3, right?
6°) Perform the restore on site B (i understand the data will be taken from performance tier automatically since performance tier is in maintenance mode)
7°) Remove maintenance mode + re-enable all job schedules on VBR site A to go back to normal
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore test with S3 compatible object storage

Post by veremin »

The process sounds OK, however, it is a bit overcomplicated in my opinion - after step 2 you can simply start restore process from VBR site A and confirm that it works fine for cloud restore points. Thanks!
archvector
Enthusiast
Posts: 33
Liked: 3 times
Joined: Mar 09, 2021 7:53 am
Contact:

Re: Restore test with S3 compatible object storage

Post by archvector »

The aim of the process is to validate the data is restorable if site A has crashed (thus there will be no more VBR on site A)

Thanks for your feedbacks, i will think about it
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore test with S3 compatible object storage

Post by veremin »

In this case you can skip configuration backup and restore steps, simply put source extent into maintenance mode on site A and import backups from object storage repository on site B. Thanks!
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Restore test with S3 compatible object storage

Post by Regnor »

veremin wrote: Apr 19, 2021 4:18 pm Correct, currently there is no way to bypass this requirement - Capacity Tier cannot be managed simultaneously by multiple backup servers. So, you need to find a time period without backup activity running, put Performance Tier into maintenance mode and execute restore process (like Petr's suggested). Thanks!
Hi Vladimir,
what are the technical implications of having the object storage online on the primary and the DR VBR server at the same time?
The helpcenter isn't very detailed on this:
Otherwise, metadata on the object storage can get corrupted or out of sync, and you will not be able to restore data.
Where and how could the meta data corruption occur?One the primary server, the DR server or both?
Is the reason for this potential issue that the primary VBR servers adds/deletes restorepoints, of which the DR VBR isn't aware of?

A customers of ours plans to do a complete DR test from object storage.
Putting the object storage into maintenance mode would create an enormous backlog of offloads, as they have a daily change rate of multiple TBs and the restore itself will take some days at minimum.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore test with S3 compatible object storage

Post by veremin »

During our tests we discovered that simultaneous access to backup and metadata objects could result in data being corrupted and out of sync - one backup server might commit a change that the other one will not know about. So we decided to block this behavior.

We are aware of this request (and the use cases behind it) and keeping an eye on it as a potential enhancement for future product versions.

Thanks!
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Restore test with S3 compatible object storage

Post by Regnor »

Thanks for your reply.
Perhaps you could implement a read-only flag for object storage, which masks all permissions except for list and get (in case of S3). I think after v12 with direct object storage, testing/restore cases will increase so it would definitely be a useful addition.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore test with S3 compatible object storage

Post by veremin »

Yeah, you might be right and the demand for this functionality will increase moving forward. So we will be forced to deliver it sooner rather than later. Thanks!
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Restore test with S3 compatible object storage

Post by Regnor »

@Vladimir: If we limit the permissions via the IAM policy to prevent any changes, would it be supported to access the bucket in parallel? It was just asked on Reddit and I was wondering if we could just leave out
DeleteObject and PutObject.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore test with S3 compatible object storage

Post by veremin »

We have never tested it, so one would not get official support, if something goes wrong. Thanks!
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Restore test with S3 compatible object storage

Post by Regnor »

Understood, thanks. Looking forward to any future updates/changes 😉
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore test with S3 compatible object storage

Post by veremin »

Sure, we will update the thread, when (or if) we have more information to share. Thanks!
claudiofolu
Enthusiast
Posts: 78
Liked: 4 times
Joined: Jan 12, 2012 3:45 am
Full Name: claudiofolu
Contact:

Re: Restore test with S3 compatible object storage

Post by claudiofolu »

Hello Veremin,
We are trying this same procedure and we cant get it to work.
Site A is a VBR with a SOBR with a performance tier of an inmutable linux server, and capacity tier is a AWS S3 bucket.
We followed steps 1 and 2, disable all jobs on site A, put in maintenance mode the performance tier in site A, then we add and import the object storage to site B VBR.

After that we lauch a restore for a VM, and it fails with the following errors:

Restore job failed Error: File does not exist. File: (XXXXX)
Restore job failed Error: Failed to open storage for read access. Storage (xxxx)
Restore job failed Error: Failed to restore file from local backup. VFS link:


Any clues?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore test with S3 compatible object storage

Post by veremin »

Doesn't seem expected, so might be worth reaching our support team. Thanks!
claudiofolu
Enthusiast
Posts: 78
Liked: 4 times
Joined: Jan 12, 2012 3:45 am
Full Name: claudiofolu
Contact:

Re: Restore test with S3 compatible object storage

Post by claudiofolu »

Thank you. Here is the ticket
Can the inmutability feature be cause? Or maybe because we are using two differents AWS keys?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore test with S3 compatible object storage

Post by veremin »

The case number is missing.

Immutability should not affect an ability to import backups to a new backup server.

Not sure what is meant by two different AWS keys, though. Can you clarify?

Thanks!
claudiofolu
Enthusiast
Posts: 78
Liked: 4 times
Joined: Jan 12, 2012 3:45 am
Full Name: claudiofolu
Contact:

Re: Restore test with S3 compatible object storage

Post by claudiofolu »

Sorry, here it is #06110298

What do you mind by "Immutability should affect an ability to import backups to a new backup server"

So there is a restriction regarding immutability? Its a DR escenario, were we lost out production backupp server, and we are attaching the capacity tier (s3) into a new backup server.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore test with S3 compatible object storage

Post by veremin » 1 person likes this post

Sorry, there was a typo - "the immutability should not affect an ability to import backups".
Post Reply

Who is online

Users browsing this forum: No registered users and 20 guests