Veeam Transform does not saturate iSCSI link

[bryanwieg]

Hello,

Writing regarding very slow Veeam transform. I have data that possible shows Veeam does not fully utilize the iSCSI Link.
TICKET ID 00507674

Veeam ...771, R2A
HyperV
Veeam Server, Windows Server 2012
iSCSI Device, Sinology DS412+
iSCSI MPIO in use, Dual 1Gbps on iSCSI Device

Transform takes hours. Please see screenshot.
Top shows Hard drive activity and network utilization with only transform going on.
Bottom shows during a transform, a copy of a large file from the iSCSI device.
CPU usages remains at less than 10% on all 6x core during transform.
RAM has 3 GB free during transform.

Notice that before the copy network and disk activity are 'low', and during the copy activity increases by 4x. All this is during the transform, so the file copy is using bandwidth the transform is not. A regular Veeam backup does appear to perform better, at about 100MBps or more. Although I have seen similar poor performance to the transform when resuming an incomplete vm backup in Veeam. Right now according to the performance monitor, the transform is going at the rate of around 20MBs.

[Gostev]

I have data that possible shows Veeam does not fully utilize the iSCSI Link.

This means your storage does not provide sufficient IOPS required to perform transformation at dual 1 Gbps speed. Which is quite expected, as Synology DS412+ has just 4 bays, so IOPS with regular hard drives will be extremely low. The rule of thumb for transformation performance is 2MB/s per spindle.

A regular Veeam backup does appear to perform better, at about 100MBps or more.

You should never compare sequential I/O with random I/O. Even slowest hard drive available on the market today will be able to sustain over 100MB/s sequential I/O, but with random I/O, IOPS becomes the most important factor. The hard drives you are using likely provide 75-100 IOPS, so even with 4 of them, horrible random I/O performance is to be expected. Compare these IOPS with over 100'000 IOPS that modern SSD drives provide, for example.

Generally speaking, your storage device is extremely low end, I am actually using this exact model as my home NAS... you might be able to significantly increase transform performance if you stuff it with SSD drives, but for backup storage purposes it might be better to instead by 12-16 bay NAS for the same money (considering the cost of high capacity SSDs).

[bryanwieg]

Hello,

Thank you for thorough answer. I did some testing with IOMeter, and found that -yes-, this raid has ok sequential throughput, and very low random.

For the record, the setup is:
Synology DS412+
4x Seagate NAS SATA Drives - ST2000VN000

I found that even a Seagate ES.3 SATA would have had about 50% to 75% more IOPS than the Consumer grade Seagate NAS SATA Drive.
http://www.storagereview.com/seagate_nas_hdd_review
http://www.storagereview.com/seagate_en ... es3_review

So I have learned my lesson.

Still, might I humbly suggest a possibility for Veeam. This is an issue for many people. Typing "Veeam transformation slow" into Google brings up results all over the internet, including these forums. Plus, spending so much $$$$$ on high performance NAS with SSD or 9-16 drives is out of the question for my many clients. I can use better drives from now on, yes, and better NAS enclosure, but please consider also the following:

In the case of the initial test I ran in the first post, it appears that combining Random Read and Sequential Write allows better use of the NAS possible potential. Would it be possible to have the engine read all the prior incremental, and write sequentially? Perhaps to a new VBK. Or perhaps implement a method similar to WAN Caching, having a higher performing volume were the data is taken from the past backups, deduped and compressed, and then written sequentially to the iSCSI backup device.

I admit that I do not not know the internal architecture of Veeam. But it appears on the surface that taking advantage of random read + sequential write would speed up Transforms on even the higher performing Storage Devices.

[Gostev]

Reading the whole prior incremental before writing it sequentially requires the RAM cache equal to the size of incremental backup, which is an unreasonable requirement. Besides, writing sequentially is "impossible" anyway, because this means the new data has to be written into the new area of VBK file vs. reusing random freed up blocks (as it works today) - which in turn means forever growing VBK, killing the main benefit of the reversed incremental backup (backup set size).

Reversed incremental is essentially a trade off where you are sacrificing backup performance for backup space... nothing comes for free, as always.

Pick any two out of these three: fast backup, small backup set, cheap storage - and I will tell you what backup mode you need to be using.

[bryanwieg]

Well, yes, I see that obviously caching entire incremental's or using existing VBK is out of the question for the stated reasons.

I was thinking in line with something similar to a "Backup Copy" inside an existing set.

For Example
As far as I know, a backup copy does something similar.
-------------------------------------\
Initial Full (backup1.vbk) -------|
Incremental 1 (backup2.vbk) ---|
Incremental 2 (backup3.vbk)----|----- > Backup Copy --> Single Full (BackupCopy.vbk)
Incremental 3 (backup4.vbk)----|
Incremental 4 (backup5.vbk)----|
Incremental 5 (backup6.vbk)----|
-------------------------------------/
Following file access, I can see a backup copy job read multiple VBks and write to a single new VBK. And (I admit a leap of faith here) it appears to me to be a sequential write. On top of that, a WAN Accelerator cache can be used to dedupe further compression of blocks from all over the backup set before writing into the new VBK. This makes the backup copy even more efficient in terms of write speed. I don't know if this cache approach could be used in a local sense. But Veeam has already implemented it in principle.

What if that same approach was taken to creating forward fulls. Instead of a classic transform, an intensive operation, take a localized backup copy within the backup set.

-------------------------------------\
Initial Full (backup1.vbk) -------|``````````````````````````````````````````````-------------------------------------\
Incremental 1 (backup2.vbk)----|``````````````````````````````````````````````Previous Incrementals Remain |
Incremental 2 (backup3.vbk)----|-----INTERNAL BACKUP COPY with DEDUPE CACHE --> Transformed Full (Backup7.vbk) | (NEW VBK)
Incremental 3 (backup4.vbk)----|``````````````````````````````````````````````Incremental 1 (Backup8.vbk) |-- Next Transform...
Incremental 4 (backup5.vbk)----|``````````````````````````````````````````````Incremental 2 (Backup9.vbk) |
Incremental 5 (backup6.vbk)----|``````````````````````````````````````````````Incremental 3 (Backup10.vbk) |
-------------------------------------/`````````````````````````````````````````````` --------------------------------------/
And of course, as with WAN caching, the cache here above would be on a high performance volume.

Just brain storming.. The only difference I can think of, a Transform from a Backup Copy, is that a transform only takes certain data forward. That said, at the moment, I am not aware of a reason why the above still could not work. The "Internal Backup Copy" just caches, dedupes, and brings forward to the new VBK only certain blocks. And then it cleans out the old full and incrementals of duplicate blocks.

[bryanwieg]

I think that it could either work that it leaves all existing Incremental files in creating the new VBK and bringing forward the data, or do as a real backup copy does, and put all restore points into one compressed VBK But I don't know, there could be a reason that is not done that way already.

[Gostev]

Backup Copy is completely identical to reversed incremental backup, except the full backup file is always the oldest file in chain (instead of the newest).

Creating a new VBK from the existing data will indeed do sequential writes, however:

1. This kills the space saving idea of reversed incremental backup, as you now have to have space for 2 VBK files: one from previous incremental chain, because it is still need for previous incremental, and newly created. Your 2nd schema says it all (except it wrongly drops the original VBK while keeping the dependent incrementals - when they are useless without said VBK).

Perhaps, you meant that "Internal Backup Copy" will go from "Incremental 1"? Even then, the time it takes to create the new VBK file will offset any possible benefits. Because we are now talking about reading and writing 100% of data, instead of performing random I/O on just 5% of data (a single incremental backup file). And, you still need to have disk space for holding 2 VBK files, because you cannot delete the original VBK file until the new VBK is created.

2. We do already have similar backup mode anyway (incremental with synthetic fulls). This creates the new full backup synthetically from the existing incremental chain, writing the new VBK file sequentially (of course, there is still added random I/O to read data from the existing incremental chain). So, if you have sufficient disk space to keep more than one full back file pn your NAS, then you should just go with this backup mode.

[bryanwieg]

Gustav, thank you for assisting me. I am really am listening to what you are saying. In my case, I am using "Full Synthetic" with "Transform previous full backup chains into rollbacks".

I do not mean to say that it should exactly be a backup copy, but be like a backup copy. Basically, what I am trying to say is, Is it possible for a transform to read from multiple VBK and write into one new one sequentially? (Similar to a backup copy) In fact, the best question I think I should ask is, why does a transform perform so many random writes?

I have been trying to imagine what a transform does.

Perhaps it reads data from the previous full and incrementals, and writes the common data to the newest VBK. After that is done, it alters the vib incrementals, and they become vrb.

Even if the vib to vrb conversion must be random, if it was possible to bring the bulk of the data blocks (data which was common and unchanged) to a new file sequentially, and then to the rest randomly, it would greatly speed up the transform. Basically, I am saying, is it possible to write the %95 of common data in the set forward into a new full vbk sequentially?

[Gostev]

Basically, I am saying, is it possible to write the %95 of common data in the set forward into a new full vbk sequentially?

Yes. The key here is *new* full backup file. If you uncheck the transform option, this is exactly what will be happening - a new full backup file will be created with sequential writes. However, transformation of the *existing* full backup file cannot be sequential for obvious reasons (as the process involves moving randomly places VBK blocks to VRB file, and reusing randomly placed free blocks to store the new data).

[bryanwieg]

Thank you Gostev,

I think what I get out of this is that it is not possible for Veeam to implement a method of transform that has random reads and sequential writes. I will just have to work around the limit that Veeam Transforms are going to be so intensely random read/write heavy, and keep that in mind when choosing backup hardware for Veeam backups of my clients.

You summed it up well earlier:

.. nothing comes for free, as always.
Pick any two out of these three: fast backup, small backup set, cheap storage - and I will tell you what backup mode you need to be using.

I think that given the option it will be cheaper to build NAS with good processing, plenty of TB, and Enterprise Drives for mediocre transforms, than try to make a backup device of SSD for pure IOP performance. And try to avoid forced into using transforms unless I need them.

An option I am going to look at is synthetic (or active) fulls with retention that deletes the oldest full. It does not give the the same level of retention, but may be the only option for poorer performing NAS devices.

[bryanwieg]

I just to clarify, I really do prefer the use of reversed incrementals for a number of reasons.. but I was not prepared for the cost in IOPS. I will plan better for that in future builds.

[Gostev]

An option I am going to look at is synthetic (or active) fulls with retention that deletes the oldest full. It does not give the the same level of retention, but may be the only option for poorer performing NAS devices.

Indeed, this is a good option for slow performing NAS devices, and it includes added reliability due to multiple full backups stored (single bit rot no longer render all following backups useless). Active fulls can be a good idea when using consumer grade hard drives with high URE.

By the way, whenever you have a choice in future deployments, always recommend a server with a bunch local disks instead of a NAS device. Any decommissioned or existing physical server will work. Servers are preferred, because they make much faster and more reliable backup targets than NAS. Not only you will get much faster transform speed due data mover agent running locally on the server (network is no longer involved), but this is also more reliable target overall. Most data corruption cases we see in support are reported for backups stored on a share residing on some low end NAS. Besides, you will often save money due to not having to buy NAS

[bryanwieg]

By the way, whenever you have a choice in future deployments, always recommend a server with a bunch local disks instead of a NAS device. Any decommissioned or existing physical server will work. Servers are preferred, because they make much faster and more reliable backup targets than NAS. Not only you will get much faster transform speed due data mover agent running locally on the server (network is no longer involved), but this is also more reliable target overall. Most data corruption cases we see in support are reported for backups stored on a share residing on some low end NAS. Besides, you will often save money due to not having to buy NAS

Thank you for this idea. I can already see ways to use this.....

[dellock6]

Also, you can then add a proper NAS as a secondary location, and move backups there with backup copy jobs for longer retention.
In this way, the first backup to local storage only needs to hold few retention points. Thus, the "pick two between fast, large, cheap" dilemma can be solved by eliminating the "large" part

Luca.

[bryanwieg]

Wow. Good practical point. Thank you.