offload. incomplete restore points

[PetrM]

Hi Michael,

Could you please share a support case ID with us so that we can keep an eye on it?

Thanks!

[mcz]

Hi Petr,

sure. #05060085 is the first case, I'll post the following tickets once we've opened them.

[mcz]

Hi Petr,

the case with the storages is #05098300.

Regarding the incomplete restore points on the object storage (which happens when BCJ gets interrupted by backup-job and offload-job offloads the incomplete restore points of bcj): The devs don't see it as bug, they won't change anything (case #05060085). For me it's not an expected situation because I've ticked the box 'copy backups to object storage as soon as they are created'. You could argue that they are copied once the (shell) has been created, but the full backup file containing the blocks as well is not copied once the file has been completely written.

So I'd say it doesn't comply with the 'rule'.

What do you say?

[veremin]

Since most case is in German, can we confirm what exactly has been identified by R&D team as expected behaviour. This?

Backup copy job runs, creates a restore point, but then gets interrupted by backup job, and SOBR tiering job offloads incomplete restore point to object storage. After backup job ends, backup copy finishes creating complete restore point and so does SOBR tiering job (or you are saying that the last part never happens).

Thanks!

[mcz]

Hi Vladimir, yes, this is correct - BUT: the second run of the SOBR offload doesn't happen immediately, it will take some time (at least the 4 hour-interval if you haven't configured it different).

[Gostev]

Are you talking about Copy or Move policy here?

[mcz]

copy

[Gostev]

That 4 hours interval definitely does not apply to Copy jobs. It is used strictly for Move policy, which has its offload task builder engine doing backups scan for offload candidates and creating offload tasks every N hours (4 by default). Copy policy does not use this task builder in principle, so it does not matter how often it is set to run. So the "wait" must be caused by some other reasons, for example waiting for task slots occupied by other tasks to become available.

[mcz]

Anton, if your (copy) offload job gets interrupted (or finished due to incomplete restore points), it will do the job on the 4-hour interval as well. I've proven it to the engineer - all documented in this case. The thing really is that the (copy) offload job can't finish the whole restore point when the BCJ gets interrupted in between.

As I said, all proven in the mentioned case. Also it happens every time and I can do demos as well.

[Gostev]

@veremin please check on this, as 4 hours wait has nothing in common with "immediate copy"

[veremin]

Already checked, the QA statement provided within the support ticket is correct:

Automatic copy mode offload is triggered as soon as a new restore point appears in configuration database. Already created, but incomplete restore points are not "new", so their completion does not result in execution of automatic copy offload.

As to regular 4 hour tiering process, it works for both move and copy policies. For move policy it looks for new restore points, for copy policy - for finally completed points.

To work around it you can either schedule backup and backup copy jobs so that they do not interfere or decrease regular tiering interval via registry key.

Thanks!

[mcz]

Thanks Vladimir, that's perfect!

To work around it you can either schedule backup and backup copy jobs so that they do not interfere or decrease regular tiering interval via registry key.

I wasn't informed about that option. What's the needed regkey?

Thanks!

[veremin]

This one:

Code: Select all

SOBRArchivingScanPeriod (set number of hours with it, 4 is the default)

Thanks!

[mcz]

Thanks Vladimir, I got that key (thought we were talking about another one to stop the interference). It's a nice workaround but it isn't enabled per default and I doubt if all veeam-users are aware of the situation. In the worst case you've got a data loss for more than 4 hours (until the data got offloaded to the immutable storage).

So still no need to change the logic?

[veremin]

Sure we will discuss how the current logic can be enhanced in future product versions, but for now the safest bet is to configure jobs the way they do not interfere with each other on regular basis.

[mcz]

okay understood. I was talking about the future as well
Just one last question regarding the interference: I think avoiding this is not really possible cause the BCJ triggers the (copy) offload and so it will always collide...

[veremin]

I was talking about backup and backup copy job collision that resulted in incomplete restore point As far as I could understand this is the root cause of the behaviour experienced.

[mcz]

yep, correct. but the copy job runs in immediate mode, so I can't choose when it would run...

[veremin]

You can configure backup window for backup copy job so that it does not run at the same time backup job does, but we understand the drawbacks here.

So thanks again for raising this, gives us food for thought.

[mcz]

ah, very good idea. thanks Vladimir!

[mcz]

Vladimir,

I'm sorry that this whole thread is a bit confusing and containts multiple topics... Just wannted to add another note regarding the certificate issue during offload:

I think that the offload-job fails when the TCP/SSL session gets interrupted (or reset/finished) during the handshake or the certificate exchange or probably before that - at least before the SSL session is established. Normally, when you end sessions during the offload, veeam just reconnects and continues. But I think that it doesn't do retries when there's something 'wrong with the certificate' and it looks like to be the same exception (or probably catch-block) when a ssl-session wasn't established and the tcp connection gets reset. Maybe I'm wrong but this is my observation.

Can you please confirm that?

[veremin]

Before offload process starts, backup server establishes a secure connection to object storage. If connection cannot be established (say, backup server cannot receive certificate), the operation will be retried. This does not apply to a situation where backup server receives an invalid certificate – in this case operation just fails.

If you experience different behaviour, kindly, collect fresh debug logs, create a separate support ticket and describe the issue there. Also, if you want to discuss this issue further here, please, create another forum thread with the problem description and support ticket number, cause current thread becomes a bit messy at this point.

Thanks!

[mcz]

Vladimir, I'm waiting for the promised patch since weeks now (case #05098300). Can you please check why that would take so long?

[veremin]

Sorry for a timeout, but I just got back from vacation and haven't been following the investigation for some time. I will check the current issue state and post an update, as soon as I got it.

Thanks!

[veremin]

I checked the case internally - the ticket was updated yesterday with the engineer setting proper backup window configuration and collecting fresh log bundle for experienced "Error: Storage has not been uploaded" issue. Also, the R&D team seems to have been involved already.

So the investigation process continues actively.

Thank you for understanding!

[mcz]

Thanks Vladimir, yes, meanwhile things have changed. I always thought RnD or QA would provide us a patch and I was told in November. Meanwhile, it still isn't clear what is causing the issue so further investigations are going on. As I said, I expected the patch which was obviously never in work.

At the moment I'll wait for the response from my engineer.

Thanks for your efforts!

[mcz]

Hi Vladimir,

this is the current status or better said the statement from my engineer:

I have received an update from my colleagues in RND.

It was confirmed that this behavior is indeed caused by the interrupted session, which leads to 2 storage versions for the same backup file to be created. While the checkpoint file in the object storage will contain the latest version, the index file will contain both.

During the creation of the stub file (.vib.meta) the information is read from the index and the earliest version is requested, however that is no longer present in the Checkpoint, as it should not be, and we received the "Storage has not been uploaded" error message.

Our developers confirmed this is an issue related to the indexes and v12 will not be affected by this behavior. A private fix for v11a cannot be pursued at the moment sadly, but we can work around this behavior by recreating the Archive Index folder. The other option to work with blackout periods for SOBR Offloading is not viable for us due to the multiple job runs per day.

I just don't see a solution by recreating the index folder, because I've done this (by rescan) and it was only a temporary fix... V12 will be release in summer (approx)?

[veremin]

The main cause was the interrupted offload session. The session starts offloading restore point, then gets interrupted and leaves incomplete restore point. Once the offload gets restarted, it completes the restore point. Due to current problem upon downloading such restore point backup servers gets the incomplete restore point instead of finished one and generates the error.

The problem has been identified and will be fixed in the next product version (check this to get a better view on its release dates) .

For now setting proper backup window should help you to avoid such situations (I'm not sure what the support engineer meant by saying it's impossible due to multiple jobs present).

Thanks!