Backup directly to AWS S3 via 3rd party gateways

[itbackups]

If anyone wants to backup directly to S3 you can in an unsupported way

You need a virtual environment (VMware / Hyper-V) where you can deploy an S3 SMB Gateway VM with a large cache (bigger than one full job)

Add it as a repository, then just point a job at it. I set the repository to use individual files for each VM

I found the best thing to do is a Backup Copy job with weekly GFS enabled and the "copy from source not increments" option enabled

Every week it does a full copy of the latest chain to the cache, then uploads it async, it goes slow but it seems to work great, been able to pull them back down to restore when needed without much trouble

The new option to use it for scale-out capacity is a nice option between my solution and VTL. since VTL is more archival.

I can see the reason it's scale out only is you don't want to send backup data directly to s3, the higher teirs probably act like the gateway's cache, giving the system a place to read from that is not a live VM snapshot.

I wanted my latest weekly chain in the cloud, recovering 1 month ago or later in a disaster isn't worth it for my setup

I will be testing the new option though just to see how performance is

[veremin]

You can achieve pretty much the same thing by adding capacity tier to SOBR and setting operational restore window to 7 days. This way, all sealed backups older than 7 days will be automatically transferred to object storage. Thanks!

[davide.asts]

If anyone wants to backup directly to S3 you can in an unsupported way

You need a virtual environment (VMware / Hyper-V) where you can deploy an S3 SMB Gateway VM with a large cache (bigger than one full job)

I have tried the same, but in my environment performances for the GFS Copy job were very low; how did you mount the AWS gateway? With SMB or NFS?

[itbackups]

You can achieve pretty much the same thing by adding capacity tier to SOBR and setting operational restore window to 7 days. This way, all sealed backups older than 7 days will be automatically transferred to object storage. Thanks!

Ok so I tried to set it up to achieve the same thing.

Existing setup was:
Basic Repositories on NAS1,NAS2. AWS Gateway cache on NAS3
Primary Backup Job to NAS1 runs daily
Primary Copy Job to NAS2 runs continuously
Secondary Copy Job to AWS Gateway with cache on NAS3, runs every 7 days, with weekly GFS, and copy full from source enabled (this prevents injecting increments which incurs S3 read costs)

New setup is:
Basic Repositories on NAS1,NAS2. Scale out Repository for NAS3 with Capacity Tier in S3, with Move backups older than 0 days set. (this essentially turns the performance tier into just a cache)
Primary Backup Job to NAS1 runs daily
Primary Copy Job to NAS2 runs continuously
Secondary Copy Job to Scale out Repository (NAS3 + S3), runs every 7 days, with weekly GFS and copy full from source enabled

So hopefully after the Secondary Copy completes, the system will start uploading to S3. I'll check back in a week after it's got something to backup. It's essentially the exact same packets to the same devices just different containers. NAS3 was a gateway cache, now it's essentially a cache for the scale out capacity tier.

So far the only real advantage I can see is that I don't have to supply a gateway with it's own allocated RAM. Not a huge difference but not bad either. And it's reasonably simpler to have it all on one place.

P.S. (not directly veeam related)

I use two S3 gateways, one is all NFS mounts for system level (no credentials, IP whitelist) robocopy file backups to buckets, the other is SMB mounts of the same buckets for AD user authenticated access. This means I won't actually be able to remove the SMB gateway, but that's just my use case.

[itbackups]

I have tried the same, but in my environment performances for the GFS Copy job were very low; how did you mount the AWS gateway? With SMB or NFS?

I think slow performance is unavoidable in this case. My job of ~1.5TB take 30-50 hours. I think there is some deduplication in the gateway as the jobs seem to run faster week after week.There's also the small issue that just because Veeam says the job is complete (all data is written to the gateway cache) it's still another few hours before the cache empties into the bucket.

I had done it with NFS before Amazon added SMB capability, but Veeam wouldn't talk to it directly, so I mounted it to a linux box, I could then point veeam at that an it worked, but was a workaround to a workaround so I was very glad when SMB was added.

[JMcG26]

If anyone wants to backup directly to S3 you can in an unsupported way

You need a virtual environment (VMware / Hyper-V) where you can deploy an S3 SMB Gateway VM with a large cache (bigger than one full job)

Add it as a repository, then just point a job at it. I set the repository to use individual files for each VM

I found the best thing to do is a Backup Copy job with weekly GFS enabled and the "copy from source not increments" option enabled

Every week it does a full copy of the latest chain to the cache, then uploads it async, it goes slow but it seems to work great, been able to pull them back down to restore when needed without much trouble

The new option to use it for scale-out capacity is a nice option between my solution and VTL. since VTL is more archival.

I can see the reason it's scale out only is you don't want to send backup data directly to s3, the higher teirs probably act like the gateway's cache, giving the system a place to read from that is not a live VM snapshot.

I wanted my latest weekly chain in the cloud, recovering 1 month ago or later in a disaster isn't worth it for my setup

I will be testing the new option though just to see how performance is

Excuse my ignorance, and for bumping an old thread but having issues adding the gateway as a repository. Are there any specific ways to do this? Keep getting the "Failed to get disk space error" and all the troubleshooting I have found for that doesn't work.

[JMcG26]

Nevermind, I'm an idiot.

[l.vinokur]

Secondary Copy Job to Scale out Repository (NAS3 + S3), runs every 7 days, with weekly GFS and copy full from source enabled

Question: how did it work for you? I would imagine that once the data is copied to the NAS3 repo, it would just sit there and gets uploaded to S3 only next week when NAS3 receives a new backup chain and the old one is closed.