No proper off-load resume for Object Storage

[CaliMSP]

Veeam was uploading a full backup to object storage (Wasabi S3 compatible) and was 70% with our largest VM backup, then it hit the job window limit (start of the day for the business) and the job was terminated. My assumption was that the job will be resumed, where it was left off and the upload would continue. Instead it looks like it's starting from the beginning (0% and counting up). I thought it would know the chunks it uploaded or at least would be able to read them quickly at the remote storage and resume?

I've reconfigured it to run non-stop with a single universal throttle rule and will let it run for a week or so (since ANY and Internet is only available in the default rule and I need to get the correct IP for our multi-homed Veeam server and Wasabi's server to make the sencond rule that is based on IPs)

[Gostev]

Your assumption is correct: the offload is forever-incremental and chunks which have already been uploaded to object storage will not be uploaded again. The progress for each offload session starts from 0% and is calculated off of the number of remaining chunks still to be offloaded.

[CaliMSP]

Ah. That's interesting. The confusion came from the fact that it still shows the entire backup file size in the off-load job (how big it is), and not the remaining amount of that job. That lead me to believe that it was restarting from the very beginning.

[CaliMSP]

Something doesn't add up.
The backup file it is trying to offload is 1.5TB. it shows that it's done with 2% (52GB)...This would put it at 2.5TB (the size of the entire backup job and not just this VM backup at 1.5TB) and certainly doesn't look like it is resuming from where it left (should have had about 300GB left as it uploaded about 1.2-1.3TB)

[Gostev]

We do not offload actual backup files (which are compressed) but rather the data they contain. So it's safe to assume that offload tracks progress based on the raw data size, which is normally at least double the backup file size. Just let it run, and once the offload finishes the debug logs can be used to confirm the actual amount of data moved - and that resumes worked correctly.

[CaliMSP]

Ok, they did complete earlier than the system was estimating (around 6%) and now all VMs are in Wasabi.

One question i have - I still have two "jobs" hanging at 38% (initial full backup) and 0% (off-load first full backup). Each one had an error (full was missing permissions to backup a linux VM and off-load ran out of time to upload). Anytime we have an active job, these two show up and can't be cleaned up. These two stuck jobs have been "stopped" via veeam console, but it doesn't remove them from the "running" view.

[Gostev]

Press F5 to force refresh the view and see if that helps.