Using object storage as a backup target
Post Reply
bhnelson
Enthusiast
Posts: 30
Liked: 9 times
Joined: Oct 01, 2018 5:32 pm
Full Name: Brian Nelson
Contact:

Huge file offload problem - incremental/partial upload to S3?

Post by bhnelson »

FYI Case 02387844

I've an SOBR with AWS:S3 as the capacity tier. And I have a backup job with a bunch of VMs but one in particular that is rather large (~8TB). I just recently moved this job to the SOBR and have been waiting for it to complete offload of the initial full backup set.

I have a offload window set (12h) so as not to offload while backups are running (logical, and best practice I think?). This window is not long enough for veeam to be able to upload the whole 8TB file in one go (single thread offload performance (encrypted) is poor, around 70MB/s). Every day, the offload appears to be restarting *at the beginning* of the large file. For the 12h window it is able to process about 3TB. Of course if that's the case it will never finish and is just blowing our money with pointless S3 charges. Being block-based, I had fully expected Veeam to support continuing incomplete offloads rather than starting over. However support has informed my that is not the case?! If so that's a enormous and extremely short-sighted limitation.

Please tell me support is wrong??

Gostev
SVP, Product Management
Posts: 28942
Liked: 5291 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Huge file offload problem - incremental/partial upload to S3?

Post by Gostev »

Support is wrong indeed :D or you simply misunderstood each other. You're correct in that offload is block-based. It is also forever-incremental, so blocks that have already made it to the object storage will never be uploaded again.
bhnelson wrote: Mar 25, 2021 10:45 pmI have a offload window set (12h) so as not to offload while backups are running (logical, and best practice I think?).
This is definitely NOT the best practice. There's actually no reason not to do offload while backups are running. Object storage offload is the lowest priority process, and will only take backup repository task slots when no other jobs need them.

bhnelson
Enthusiast
Posts: 30
Liked: 9 times
Joined: Oct 01, 2018 5:32 pm
Full Name: Brian Nelson
Contact:

Re: Huge file offload problem - incremental/partial upload to S3?

Post by bhnelson »

Whew! Glad that's the case. I can happily report the offload finally completed today and everything looks good now! I must say that the offload job status/statistics should REALLY reflect this incremental progress in some way though. As a new user to SOBR, seeing the job grind day after day with no apparent progress was really worrying!
Gostev wrote: Mar 25, 2021 10:52 pm This is definitely NOT the best practice. There's actually no reason not to do offload while backups are running. Object storage offload is the lowest priority process, and will only take backup repository task slots when no other jobs need them.
Interesting. There was a different post somewhere that I think I read only part of and must have misinterpreted. So you're saying that an active offload process will stop/pause/yield its repository slot if a backup job starts that wants it? That's good to know.

My thought was that adding additional IO (ie offload) to the mix during backup time is inherently going to slow things down overall. Depending on storage performance, bandwidth, parallel jobs, etc, that slow down may or may not be negligible. I have plenty of time and bandwidth to offload during the day and let only backups have full run overnight so they complete as fast as possible. I'd think that would be generally applicable but certainly could depend on the specific situation and admin's desires.

-B

Gostev
SVP, Product Management
Posts: 28942
Liked: 5291 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Huge file offload problem - incremental/partial upload to S3?

Post by Gostev » 1 person likes this post

Glad to hear this :)

The one scenario where starting offload right away is essential, is when customers have strict off-site backup RPO requirements to meet, and are using the Copy policy of the SOBR Capacity Tier to create those off-site copies. Basically, some businesses require that every backup that is created on-prem is copied off-site within a very short time, for example a couple of hours max.

If you don't face such a requirement, then there's no reason to start offload immediately, as of course you're right - there will be at least some impact to your backup window, even if negligible. The impact can be regulated with a max concurrent task limit setting on the repository, but of course you cannot cheat the physics here: the more active tasks you have accessing a backup storage at the same time, the less IOPS and bandwidth each particular task will have to work with.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests