FLR from object storage data

[backupquestions]

Just to see how it would perform, I attempted a file level restore and selected a date that is data that has been offloaded to Wasabi capacity tier. The mounting restore point session sat there for about 20 minutes and never did load. I ended up killing the session out.

I watched the network traffic on the nic and did not see anything significant happening. Shouldn't it be downloading blocks of data from Wasabi in order to mount the image? What could cause it to hang up like this? Using latest version 4b.

We have 1gb/sec internet connection up and down.

For comparison I can view the same VM on a more recent restore point that has not been offloaded and it comes up in usually 1 or 2 min. A little worried by the performance as it seems like it wasn't even really trying to request the data.

[Gostev]

Yes, there should be quite significant network activity during mount. Perhaps some temporary issue with network connectivity, or Wasabi itself?

With 1Gb connection, the restore performance from object storage should be comparable with one from local storage. In fact, if you search this forum, you will find another Wasabi user reporting slightly FASTER restore from object storage than from local storage

[backupquestions]

My case is 03749065.

I was told from the tech (after he consulted with a tier 2 tech and some others) that the delay in the mount time was normal or that they were not surprised. They mentioned how Azure throttles and they suspected Wasabi does too. They also told me that it is meant as cold storage and for only older stuff. I do mostly understand that and that a delay is not as big of a deal in that scenario, but I could have sworn Veeam is also marketing capacity tier as an option for a smaller client to have a bigger repo that they can't afford. Meaning that they would be using object storage not only for really old stuff, but maybe stuff only a few weeks old that they might draw from on a somewhat regular basis. If it takes an hour or more to even mount each time, that doesn't seem acceptable.

Wasabi told me that I should be able to pull the data as fast as I can go with my internet speed as their storage is hot storage and performs much faster than (up to 6x I think) Amazon S3 itself.

I didn't have a big issue with actual restore of a file once it mounted, though I only restored a tiny file. I'm having issue specifically with the long wait time for it to even "mount" the restore point.

It took over 45 minutes when I let it go again to get it to mount. This was attempted yesterday, today, and then again tonight. So it doesn't seem like a temporary issue.

After more testing I'm seeing still 35-45 min on VMs with way different sizes as well. Almost same time to mount on a 500GB VM as it is on a 4TB one.

Could you or someone ask them to look into our logs or help figure out what might be needed?

[veremin]

I cannot see any debug logs provided within the case, can you post them there, so that we can double check the issue internally? Thanks!

[backupquestions]

I've attached the last few days of logs to the case just now.

[Gostev]

Thanks for sharing the logs. And please disregard comments made by the support engineer for now

We will investigate the issue appropriately, and contact Wasabi directly if we determine the issue is on their side. At a first sight, this should be really easy to troubleshoot from the logs, considering that you noted seeing no network activity during mount, meaning the whole process basically "hangs" doing nothing for 35-45 min.

[backupquestions]

Thank you Gostev,

I have worked today with tier 2 engineer and he did some looking at the logs and talked about the issue with me. So far we don't have a clear answer other than demonstrating that the mount times are super fast locally on any backups of these VMs including oldest ones. It is only ones in object storage that take a long time to mount. He did a registry weak that he thought could possibly help with the symptom of slow mount, but not something that would fix the actual cause understandably. But, this tweak did not seem to have an affect.

He gathered some more logs from today and is going to get back to me later after discussing it.

I'm really wanting to learn if a long initial mount time is just going to be normal for any object storage or if the mount itself should be pretty fast. It doesn't look good when our previous backup solution could retrieve old files pretty quickly but this is taking around an hour to actually get a file. I know it isn't necessarily going to be Veeam's fault in the end if it ends up being a Wasabi issue but we'll see. I'm interested to keep following this up and see what we can find.

[DWZH]

We work with SWITCHengines (https://www.switch.ch/engines/) object store and can say, we don't have such long load time. Even on the Veeam Demo 2-3 Weeks ago with SWITCHengines, it was under 5 minutes, till the file-browser was ready for restores.
(on the demo, we deactivated the local extend, so everything was loaded from S3 and nothing from the local storage)

I just tested it again in our production:

Setup1
- SOBR with S3 (SWITCHengines) as capacity Tier
- Backup Copy Job to SOBR
- Offload after 3 days
- Backup encrypted
- Nothing in maintenance mode

it took me about 3 minutes, till I got the FLR Explorer. Restore Speed about 140MB/s (for 700MB File).

[backupquestions]

Thanks for the reply. Could you tell me how large your vm backup file was in this test? Most of mine that are in object storage are large like 1tb or more. But those same vms open very fast in vcc or on premise backups.

Tier 2 is working on this and testing. But it's a great sign that you are able to mount quickly from object storage. Might be a wasabi specific issue.

[dalbertson]

Have you opened a case with wasabi? I have informed wasabi of this post and will work with them to figure this out. From all my testing with restores (file, instant, and full) I have never had and issue with wasabi.

[backupquestions]

First off to answer your question, no, I had only pushed it from Veeam's angle so far.

It's weird though.... Today I've done more tests and it is working fast! All restore points are mounting in 1-3 minutes and restore speeds are fine.

I just got off the phone with Veeam support engineer tier 2 and we demonstrated this and took fresh logs. I was seeing network activity very early on with these mounts etc.

The thing is, it was a problem for over a week I believe initially so it wasn't like a temporary glitch. I was testing each day for a while and not seeing it improve. So the Veeam tech is going to test this in his lab and see if he can re produce any slow mounting.

Veeam support does have my earlier logs showing the very long mounts so that might still be useful for Wasabi to look at possibly, not sure. Just in terms of whether it says Wasabi was asked for data and was not responsive in the replies.

I am hoping the problem is just magically solved and will not return but we will see.

Thank you to Gostev and to the Veeam support engineers. It was great service and they were always responsive and worked to try to find a solution.

With it performing like this I am happy!