Sequential write feature idea/theory

[kjo@deif.com]

I had an idea popup into my head today and I had to write it down somewhere, maybe you could use it.

With hard drives, sequential write speed is fast and random writes are slow. So the more jobs are run in parallel, the slower the data will be written to disk, and if we just set 1 job to run then a lot of time will be spent on waiting for snapshots etc.

So my idea is. If Veeam had a feature where it would take all data from the jobs and write it all in sequence to one big file, it would be fast. (job1block|job2block|job1block|...)

Afterwards it could use block clone to split it into different backup files. This should also be quick since it's just meta data.

As backups are taken and deleted the storage will get fragmented, so it will be important to keep the free space collected and performant (defrag /x).

And lastly since the amount of parallel jobs doesn't matter anymore, an additional feature where Veeam would automatically start another job if there is no bottleneck.

This might not be useful for all storage/solutions, will require a lot of work and testing, and I might not have thought everything through, however if this is possible then it would give better performance for hard drives.

[mkretzer]

That is exactly what DataCore Storage virtualisation is able to do! The results are very good and you get near-SSD write speed on rotating disk!

It is really a good idea but it would mean that Veeam would have to change its storage format, so i do not think they will implement it.

[PetrM]

Hello,

Many thanks for the feedback!

if we just set 1 job to run then a lot of time will be spent on waiting for snapshots etc.

I'm not sure I understand what is the challenge? Basically, time on waiting for snapshot does not depend on jobs number, the request to create a snapshot is sent once backup infrastructure
resources (f.e. proxy, repository) are assigned to a VM which is being processed by job.

If you set just 1 concurrent task in the backup repository options, you will have 1 stream of sequential write and block cloning will not be needed.

And lastly since the amount of parallel jobs doesn't matter anymore, an additional feature where Veeam would automatically start another job if there is no bottleneck.

In fact, bottleneck always exists in every system, there will be always some processing stage which slows down data flow and it's up to you to decide which processing rate is acceptable for you.
For example, there are storage latency control and read and write data limit options to throtte I/O on both source and target sides.

Thanks!