Discussions related to using object storage as a backup target.
Post Reply
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

performance while streaming data from object storage via instant vm recovery

Post by mcz »

Hi veeam-guys,

I have backed up a vm and moved it to the object storage for archiving purposes. Now I was able to do a IR of that vm and to boot/login but then my application reported strange error messages. It was very strange to me because the "original" vm on prem was fine, so why should the exact clone (only on a different storage and a different structure) ever be different?

One idea I had was the performance difference due to the location of the object storage (cloud) which would never be as fast as for a local backup. Please note that we have a 100 Mbit internetconnection. So I took a look at the resmon on the veeam-server and saw that only one veeamagent process would pull the data (so far so good, it could run on multiple threads) but the throughput wasn't that great. It was somewhere around 1,2 MB/s while our connection would provide us 10 times more, if needed.

So I did a restore of the whole vm and the errors were gone - the same application of the same vm is running fine. So I assume that this application is very sensitive to latency and that some timeouts are set very low.

Anyway, I still don't unterstand the whole picture. First of all, I'd say that an IR from an cloud object storage is expected to not be that fast - I think latency to the provider is the culprit - but veeam has introduced a cache in v10 and this cache has boosted IR a lot. So when the first read attempt fails, why would the second one also fail? The second read should have the datablocks in the cache, so I don't get that. The other question is around the throughput. If read requests are queued, I assume that veeam would use multiple threads (up to 64, correct?) to pull the data from the object storage. But 1,2 MB/s is a quite low number for me. Compared to the full vm restore when the whole 100 Mbit/s were used, IR isn't pulling that much data from the object storage. I wonder if there's a limit for the maximum allowed threads which would pull the data from the object storage. And if that number was quite low and read requests are higher than available threads, I would understand why it would be that low.

In any case, I'm curious about the outcome!

Thanks!
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: performance while streaming data from object storage via instant vm recovery

Post by Gostev »

Actually, IR from a cloud object storage can be faster than some local storage, if you have the dedicated 10Gb connection to it - like many of our U.S. customers do ;) so you are right, it totally depends on the bandwidth and latency.

Sounds like it takes longer than two read attempts to pull the block from object storage, which is why the block is still not yet in cache by the time when the second read attempt happens.

Veeam can use multiple threads whenever we control the workload, such as backup offload or download - where we can process multiple different blocks at once. For example, when we're downloading the entire VMDK from object storage for the entire VM restore, we can request up to 64 different blocks of VMDK at once from object storage, and "land" their content into the appropriate VMDK file areas on VMFS volume as they are received. So, there's a massive concurrency at all times.

But with Instant Recovery, we don't control the workload - we merely deliver disk blocks as they are requested by OS/applications. When some application reads its database file, it usually requests block by block as its code execution requires, instead of concurrently "downloading" all blocks of the entire database file into RAM first. And most applications do synchronous I/O, meaning they don't move to performing the next I/O operation until the previous one completes - at least within each given logic path (for example, reading or updating the given database among multiple databases running).
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: performance while streaming data from object storage via instant vm recovery

Post by mcz »

Thanks Anton! I guess I'd have to wait a little bit until we get a 10Gb connection... Your explanation makes absolutely sense but to understand the whole picture I've got some additional questions ;)

You said that veeam only uses multithreading if the workload is controlled by itself... Does that mean that the "on the fly object-storage-pulldown" is running single-threaded? If not, how much threads would veeam use then? Why not also use 64? Say you've got 10 read-requests in the queue, how are these processed?

BTW, why are the threads limited to 64? I mean I know that 2 to the power of 6 is 64, but why not use more threads for instance during offload?

You see I'm digging deep because I wannt to understand the product... Thanks!
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: performance while streaming data from object storage via instant vm recovery

Post by Gostev »

We always use up to 64 concurrent API threads when working with backups in object storage. No activity that we ourselves perform with backups in object storage is single-threaded, as this would result in abysmal performance. So, for any Veeam-controlled task, there are always up to 64 API requests in the queue, which are all processed concurrently.

The value is based on our testing. We saw little performance increases after this number even with most capable object storage, however on the other hand we saw some object storages device struggle if we throw too many concurrent API requests at it - so 64 was the perfect middle ground. No one ever had to increase it, but reductions are not uncommon to reduce the load on object storage.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: performance while streaming data from object storage via instant vm recovery

Post by mcz »

ok, makes sense, thanks a lot. And what about pull-requests caused by IR sessions? Still not single-threaded? I mean after vm has been bootet, not before.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: performance while streaming data from object storage via instant vm recovery

Post by Gostev »

We will process concurrent I/O requests issued by the instantly recovery VM OS concurrently, if this is what you're asking. For example, if application A requests data from file A at the same time as application B requests data from file B, then both requests will be served concurrently, and required backup blocks will be retrieved from object storage at the same time.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: performance while streaming data from object storage via instant vm recovery

Post by mcz » 1 person likes this post

Thanks Anton, this is what I wannted to hear ;)

BTW, I love object storage allthough it caused a lot of "pain" on my side... :)
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 11 guests