object storage rescan took 5 days - what does it do?

[mcz]

Hello everybody,

we're having a SOBR which has a local extent (NAS CIFS share) and a object storage as capacity extent (wasabi). Now to be honest I mad a small mistake: Instead of using copy jobs I just created a reverse incremental backup job (using this SOBR as repository) and specified the offload policy. Now I've got 7 VM's in my backup chain and more or less 750 restore points per VM, 650 have been offloaded.

Now something went wrong some weeks ago and that's why veeam started to offload to a new folder on s3 which means that all "old" restore points on s3 became unavailable (case # 03620219). We don't know why but tried to fix it by using the following action plan:

- download already offloaded restore points (from the new folder)
- drop the new folder on s3
- db-update for pointing again to the right s3 folder
- run rescan

Now we're still working on the issue but the reason why I created this the thread is the rescan-performance. I've got the feeling that this process could be improved, here are my observations:

• when veeam downlaods the index to the temp directory (c:\windows\temp\) then it does this step by step. It doesn't download files in parallel and it waits for some other task to finish and downloads the next file - it would be faster to download in advance and parallel

• veeam synchronizes the indexes one by one and compares/synchronizes the index from s3 and the local one. I had some attempts when we run out of disk space (because of the huge index size which has been downloaded) after a successful run of several other indexes and when I restarted the rescan, it started from the beginning!! It didn't realize that it has already processed some indexes and therefore wasted a lot of time. It would be better to somehow integrate checksums to compare s3 contents with the local ones to speed up the process

• the whole rescan took 5 days and some indexes were processed 24 hours. I noticed that the first 100 index files are processed very quickly (in about one hour) but the more the processing went on, the slower it got. So once again: For e.g. 650 restore points the first 100 were processed in about 1 hour and the other 550 needed 23 hours. I noticed that veeam was reading somewhere around 300-600 Mbit/s from the CIFS share (local extent) and I really don't get why it reads that much of data

Maybe I don't understand this process very much but I really don't understand why so much transformation has to be done. I'd appreciate a rescan which would simply just download the index from s3 and probably writes the contents into a file but without that much reading/comparing. Please note that for some VM's we already had the (same) index on the local extent, for some not - in both ways it took many hours to complete the synchronization.

Can please somebody shed some light on this process? Why is it that slow? How could it be accelerated (on my side)? For how many restore points has object storage rescan been tested/built? As I said, my approach wasn't ideal but what if you have a desaster in the future and you'd like to access backups from s3 - if you then have to wait 5 days until the rescan has finished, it would be more than painful.

By the way, during rescan the vm had 1 GB of RAM available and the local temp directory points to a SSD. I've already asked the support team for clarification but they told me that QA is currently under heavy load by testing the new v10 which of course has a high priority so I hope that I get my answers here on the forum.

Thanks in advance!

[Gostev]

Hi, Michael. Checked with the dev leads and they would be happy to review rescan debug logs to understand "what did it do" and consider if any optimizations can be made to the rescan logic (as long as logs show there were no 3rd party or connectivity issues). Thanks!

[mcz]

ok, thanks Anton. I guess they should have the rescan logs - or was that a question to upload the logs once again?

[Gostev]

What was the last time you uploaded logs to the case - was it after 5 days rescan completed?

[mcz]

yes, it was yesterday and back than the rescan completed after 5 days.

[Gostev]

OK, then they should be good, thanks!

[mcz]

Hi Anton, is there any update at the moment? I'm currently doing another rescan and this time it's already running since 5 days and is still working on the 4th of 6 indexes...

[Gostev]

Hi, Michael. I am not monitoring this, so all updates will be provided directly in the support case. If you don't hear back for extended time, please ping them through the case. If your support engineer tries to close the case, please request to keep it open until you receive a satisfactory response. Thanks!

[mcz]

Very interesting, support engineer has written the following:

This is to inform you that I've started discussion on this ticket with Q&A. Based on what I see in the logs, there's no real bottleneck here. Instead, rescan seems to require performing a lot of small actions, where each of them takes no more than just 1 second. Therefore, it's basically a matter of how many actions it has to perform, but the amount is unclear for me.

....and:

According to R&D completion time depends on the number of existing backup files, plus, from what I understood, it gets even worse if they are encrypted. Nevertheless, they say there's definitely a room for improvement, so they marked the behavior as bug, and we're planning to fix it in v10.

So it looks like that our current situation maybe has not been tested in the past. I hope that the devs can make a decent improvement on this process - otherwise it would be very painful in case of an issue/desaster.

Any thoughts on that? Thanks!

[veremin]

The slow performance is partly caused by the bug in re-scan logic - if for some reason it fails, backup server starts re-creating indexes upon next try, instead of re-reading them. This process takes significant time.

The behaviour is marked as a bug and will be fixed in version 10.

Thanks!

[Gostev]

That makes sense. As per the adjacent thread, Michael was hit by those temporary Wasabi issues.

[mcz]

You've got a very good memory, Anton