GFS backup file migration S3 to Azure without downloading

[tjdoc]

Hi,
We need to copy long term GFS backups from an S3 repo to an Azure Blob storage account.
The S3 repo is a capacity tier managed by a VBR server in our DC.
The S3 repo needs to stay online (so we cant seal it). It's not possible to connect to the S3 bucket from another VBR server (unless someone can tell me otherwise).

The only way I know of to obtain the backup files we need (selected M or Y backups held in S3) is to "Export" them using the VBR to a local disk on the server, then copy them somewhere so they can then be copied up into the Azure Blob storage account, using something like VeeamZip.
However, we dont have the capacity or network speed available to run the export, which will download TB of data...it will take forever.

Are we able to copy directly from S3 to Azure?
We cant see a way to avoid the data being downloaded to the VBR server first, before being copied anywhere else.
We thought of creating an Azure VM and adding as a Gateway / Data Mover, but still cant see how the Export process wont pull the data down to the server first.

Ideally we would connect a 2nd standalone VBR from our secondary DC to the S3 bucket then do the Exports from there, but I don't think that's possible either.

Any thoughts / help on this appreciated...

Thanks

[veremin]

A quick question - are you referring to copying one or a few GFS points rather than the entire contents of the capacity tier? Thanks!

[tjdoc]

Hi, we want to be selective with the GFS backups and do not need the entire data set currently in S3. It's quite unlikely we would ever actually have to restore from them anyway, but need to have the data just in case.
Worst case we would consider copying the whole lot if we really had to.

Many thanks

[veremin]

In that case, you can use the export backup procedure. To do this, launch the backup export wizard, select the restore point located on the S3 capacity extent as the source, and choose the Azure Blob object storage repository as the target. In this scenario, the traffic will flow between the gateway server selected for the S3 capacity extent and the gateway server selected for the Azure Blob object storage repository. If gateway servers are not specified (direct connection mode), then the mount servers associated with the source and target repository will be responsible for transferring the traffic. And no, the servers responsible for transferring the traffic will not need free space for the exported point during the process itself.

Thanks!

[tjdoc]

Hi, thanks for this update.
On the VBR server, I can see that the connection mode to S3 is Direct.
How is a Gateway server added into S3? Is it just a VM running in AWS that is on the same network as the S3 bucket, which is then added to the VBR infrastructure as a managed server and given the role of a Gateway?
The Azure storage isn't yet connected, but would that be the same kind of setup?

Each VM (in S3 and Azure) would then need to be able to resolve each others public address to communicate with each other - if that's setup as well, would it mean the S3 gateway would send directly to the Azure gateway?

Thanks very much!

[veremin]

A gateway server is an additional component associated with a backup repository that hosts the Veeam Data Mover. You can assign the gateway server role to any Microsoft Windows or Linux machine that is added to the backup infrastructure as a managed server. In your case, you can indeed add an EC2 machine to the backup server console and assign it the gateway server role, and do the same operation with an Azure IaaS VM.

If you assign separate gateway servers for both the source and target repositories, the traffic will indeed flow between them during the backup export.

Thanks!

[tjdoc]

Hi,
This does sound like it will help and not send traffic via the on-premise VBR server, if I'm understanding this correctly?

Do you know if the Helper Appliance can be used for this activity, or is it only for health checks?

thank you!

[veremin]

Yes, you understand correctly. In this case, the backup server will not be involved in the data traffic transfer. And no, a helper appliance cannot be used for these purposes — you need to set up gateway servers. Thanks!

[thynes]

It is always best to have the gateway server as close to the object repository as possible for this purpose.
It being S3, as previously mentioned, you would want an EC2 instance in the same VPC; for Blob storage, an Azure a VM within the same tenant; and for S3-Compatible again a VM nearest the source.

[tjdoc]

ok will work on that, thank you for your help...really useful