Intra-disk parallelism for full-VM restore (single-disk VMs limited to one stream)

[MrvnWlr]

TL;DR

On a full-VM restore, each virtual disk is restored by a single data-mover stream. A VM with one disk therefore gets exactly one stream, which tops out at ~80 MB/s in our environment — even though the same repository extent delivers ~2800 MB/s single-stream in a raw read test, the restore proxy CPU is ~0.7% busy, and the transport is Direct SAN over FC. No hardware resource is saturated (measured). Our working hypothesis for the remaining single-stream limit is a serialized / shallow-pipeline per-disk restore path inside the data mover — which is exactly what we would like R&D to confirm or correct. Request: allow intra-disk parallelism (or a deeper, configurable per-stream I/O pipeline) for restore, so single-disk VMs are no longer capped by one stream.


Environment

• Veeam Backup & Replication v13 (behavior also present on 12.3.2).
• SOBR: primary = 9 Linux Hardened Repository extents; copy = 8 extents. Backup files on XFS.
• Proxies: Windows, Transport Mode = Auto, high MaxConcurrentTasks.
• Primary storage: enterprise all-flash array, FC fabric. Restore write path negotiated as Direct SAN [san] (confirmed in the restore session log).
• Test VM: 2 virtual disks, ~1.8 TB each, filled with incompressible data (worst case).

Problem statement

A full-VM restore parallelizes at the task = virtual disk level. There is no parallelism within a single disk. Consequently:

• A 1-disk VM restore = 1 data-mover stream (observed).
• That single stream tops out at ~79-81 MB/s per disk in our environment, i.e. a 2-disk VM ~160 MB/s, a 1-disk VM ~80 MB/s (measured).
• A 3.6 TB / 2-disk restore took ~405 min (measured).
• The same per-stream figure (~80-98 MB/s) is reproducible across different proxies and hardware generations (measured) — which points to a software pipeline constant, not a hardware limit.

Our hypothesis (to be confirmed by R&D) for why one stream caps at ~80 MB/s while every external resource is idle: the per-disk restore path is effectively latency-bound, not bandwidth-bound — throughput ~ 1 / (per-block read + decompress + decrypt + transfer + write-ack latencies) — because the stages are not overlapped with enough in-flight depth (read-ahead / write queue depth per stream appears shallow). We cannot prove this from the outside: the v13 restore session log does not expose a per-component "Busy:" breakdown (see Questions).


Why there is no acceptable workaround today

For a single-disk VM there is currently no way to make the full restore faster: parallelism only helps VMs with multiple disks or bulk/mass DR, not a 1-disk VM. Instant VM Recovery is not a viable substitute for us either — a large share of our protected workloads are latency-sensitive database servers that cannot run acceptably while being served from the backup repository (nor during the subsequent Storage vMotion). So for a single-disk database server we are stuck with the ~80 MB/s serial restore and its long RTO. That is exactly why we are asking for intra-disk parallelism / a deeper restore pipeline.


Evidence — what we measured (read-only)

Code: Select all

Measurement                                              Result                              Implication
-------------------------------------------------------  ----------------------------------  ------------------------------------
Veeam restore, per disk                                  ~79-81 MB/s [san] over 2 proxies    Single-stream ceiling
Raw repo read, fio libaio direct bs=1M qd=8, 1 stream    ~2800 MB/s (1 stream ~ saturates)   Storage is NOT the limit (~35x)
Raw repo read, fio, 4 parallel streams                   ~2900 MB/s (device ceiling)         Plenty of aggregate headroom
Restore proxy CPU (sampled locally during restore)       total ~0.7%, busiest core ~12%      Data mover / decompress NOT CPU-bound
Repo host iostat -xm during restore                      ~500 MB/s @ <50% util, low r_await  Repo disks NOT the limit
FC / proxy NIC during restore                            idle / far below line rate          Fabric / network NOT the limit
Restore write transport                                  Direct SAN [san]                    Not an NBD artifact

Key contrast: raw single-stream repo read = ~2800 MB/s, Veeam single-stream restore = ~80 MB/s -> ~35x gap with every hardware resource idle.


What we explicitly ruled out

• Repository storage throughput — 2800 MB/s single-stream in fio, <50% util during restore.
• Proxy CPU / decompression — not even one core saturated (~12%).
• Transport mode — Direct SAN over FC (optimal), not NBD/HotAdd.
• FC / network — idle.
• Proxy contention — same per-stream rate on two independent proxies simultaneously.

Everything hardware-side has massive headroom. The only thing that scales throughput is running more disk streams in parallel — which does nothing for a single-disk VM.


Feature request (the ask)

Please make a single virtual disk's restore able to use more than one worker, e.g. one or more of:

1. Intra-disk parallelism — split a single disk's block map into N contiguous sub-ranges, each restored by its own reader/writer worker (conceptually the existing multi-task model, but within one disk). This is the change that would speed up 1-disk VMs.
2. Configurable per-stream I/O pipeline depth — deeper read-ahead on the source data mover and a deeper write queue on the target, so a single stream becomes bandwidth-bound instead of latency-bound.
3. Multiple decompress/decrypt threads per stream (if per-block CPU serialization contributes).

Ideally exposed as an advanced/registry setting first (opt-in), like other data-mover tunables.


Questions for R&D

1. Is the per-disk restore pipeline processed synchronously block-by-block, or is there read-ahead? What is the default in-flight queue depth per restore stream?
2. Are there any (even unofficial / support-only) settings that increase restore stream depth or workers per disk today?
3. Is intra-disk parallelism fundamentally compatible with the backup block/storage format and with Direct SAN / HotAdd write paths?
4. Can you confirm the internal limiter for a single stream, given all external resources are idle? (The v13 restore session log does not expose the "Busy: Source/Proxy/Network/Target" breakdown that backup jobs have, so we cannot localize it from logs.)

[david.domask]

Hi Marvin, welcome to the forums.

Thank you for the detailed post, but a lot of this is expected with DirectSAN restores, DirectSAN write is fairly limited by VDDK for writes and does better with multiple disk parallelism.

I'm trying to find public document on it from Broadcom but seems they've taken down all public facing VDDK documentation, but as I recall, it's single thread synchronous per disk for DirectSAN write (I want to say that a more recent VDDK library introduced read improvements, but not write)

The fio tests are likely not super useful here as even with DirectSAN you're still going through VDDK so there is going to be some overhead there that will not be reflected by a fio test.

80 MB/s though still seems a little low, did you restore the disk as lazy-zeroed or eager-zeroed thick disk?

Regardless, understood on the request, but there are some limitations from the VDDK side on DirectSAN restores that make this not so feasible. Hotadd is usually recommended for restores.