Huge file offload problem - incremental/partial upload to S3?

[bhnelson]

FYI Case 02387844

I've an SOBR with AWS:S3 as the capacity tier. And I have a backup job with a bunch of VMs but one in particular that is rather large (~8TB). I just recently moved this job to the SOBR and have been waiting for it to complete offload of the initial full backup set.

I have a offload window set (12h) so as not to offload while backups are running (logical, and best practice I think?). This window is not long enough for veeam to be able to upload the whole 8TB file in one go (single thread offload performance (encrypted) is poor, around 70MB/s). Every day, the offload appears to be restarting *at the beginning* of the large file. For the 12h window it is able to process about 3TB. Of course if that's the case it will never finish and is just blowing our money with pointless S3 charges. Being block-based, I had fully expected Veeam to support continuing incomplete offloads rather than starting over. However support has informed my that is not the case?! If so that's a enormous and extremely short-sighted limitation.

Please tell me support is wrong??

[Gostev]

Support is wrong indeed or you simply misunderstood each other. You're correct in that offload is block-based. It is also forever-incremental, so blocks that have already made it to the object storage will never be uploaded again.

I have a offload window set (12h) so as not to offload while backups are running (logical, and best practice I think?).

This is definitely NOT the best practice. There's actually no reason not to do offload while backups are running. Object storage offload is the lowest priority process, and will only take backup repository task slots when no other jobs need them.

[bhnelson]

Whew! Glad that's the case. I can happily report the offload finally completed today and everything looks good now! I must say that the offload job status/statistics should REALLY reflect this incremental progress in some way though. As a new user to SOBR, seeing the job grind day after day with no apparent progress was really worrying!

This is definitely NOT the best practice. There's actually no reason not to do offload while backups are running. Object storage offload is the lowest priority process, and will only take backup repository task slots when no other jobs need them.

Interesting. There was a different post somewhere that I think I read only part of and must have misinterpreted. So you're saying that an active offload process will stop/pause/yield its repository slot if a backup job starts that wants it? That's good to know.

My thought was that adding additional IO (ie offload) to the mix during backup time is inherently going to slow things down overall. Depending on storage performance, bandwidth, parallel jobs, etc, that slow down may or may not be negligible. I have plenty of time and bandwidth to offload during the day and let only backups have full run overnight so they complete as fast as possible. I'd think that would be generally applicable but certainly could depend on the specific situation and admin's desires.

-B

[Gostev]

Glad to hear this

The one scenario where starting offload right away is essential, is when customers have strict off-site backup RPO requirements to meet, and are using the Copy policy of the SOBR Capacity Tier to create those off-site copies. Basically, some businesses require that every backup that is created on-prem is copied off-site within a very short time, for example a couple of hours max.

If you don't face such a requirement, then there's no reason to start offload immediately, as of course you're right - there will be at least some impact to your backup window, even if negligible. The impact can be regulated with a max concurrent task limit setting on the repository, but of course you cannot cheat the physics here: the more active tasks you have accessing a backup storage at the same time, the less IOPS and bandwidth each particular task will have to work with.

[fklein]

Hi there,

I stumbled into this thread while searching for "Storage has not been uploaded".
My Case #05528793

Please send the hotfix.

Thanks

[Mildur]

Hi Florian

Thanks for the case number. Please keep working with support and let them analyze the logs to confirm the real issue. If there is a hotfix, support will provide you with it. If there is another solution, support will help you with a step by step guide. We can‘t provide hotfixes over the forum.

Thanks
Fabian