Performance drop as backup file grows?

[Yuki]

Hey all,
Just wanted to check if anyone else is seeing the same issue. It seems that as our backup file grows, the performance on the reverse incremental runs gets lower and lower. I understand that this is a target storage issue and not Veeam's per se, but wanted to see if anyone else is observing similar results.

Our 30TB NAS (linux based on 3TB SATA drives and EXT4 RAID5 volume) is getting progressively slower as the reverse incremental file grows. Right now a full backup file is at 5.8TB. Anyone else noticed this on other systems?

[tsightler]

What block size? (Local/LAN/WAN)

[Yuki]

seems to happen at all block sizes, but right now it is configured for Local/Optimal.

[tsightler]

Yes, it will happen at all block sizes, but it will happen sooner with smaller block sizes. With a 5.8TB VBK I would have to assume that the total size of the VM data in the backup is probably twice that much (assuming at least 2x compression). Is that correct?

As a best practice it is recommended to put no more that 8TB of VM size data into a single backup job if you are looking to achieve maximum performance. In general you will see much better performance out of running multiple smaller jobs rather than one huge job.

Also, VBK files do become fragmented over time which can also lead to some performance degradation and is another reason why running a full backup every few months is sometimes recommended, especially for huge backup files.

[Yuki]

Well, it's a single 10TB VM, so we can't split it. We have very little benefit from the compression. This is a 2012 with dedupe enabled and total used space (After dedupe) at 5.7TB, which is also the size of our backup file. Windows dedupe does save us about 2.8TB of space though.

[tsightler]

With a VM like that you'd see the best performance by disabling compression and dedupe in the job since the data is already deduped. Not only would this reduce the amount of CPU being used, but it should also reduce the memory overhead and disk I/O since there will be no hash table for the dedupe to hold in memory or flush to disk. You would still see some degradation due to fragmentation but it should be better.

[ncyclon]

Hi, I have same issue. VM size 6.6Tb. After start of job I see 30-40 MB/s, but after transfer 500-600Gb speed down to 16 MB/s. Dedupe and optimal compression set.

Is it normal behavior?

[Gostev]

Alexandr - assuming you are running current product version - no, performance reduction within the given job run is not normal. However, I can say that we've seen this before with certain storage devices. In fact, this one problem we have probably spent more time researching more than any other problem we ever had with the product. In previous similar support cases, this came down to the issue with the storage itself. Our support now has a tool that can do basic continuous I/O to a large file (thus removing Veeam from the picture completely). Best course of actions would to open a support case, confirm if this is the same storage issue using this tool, and if so then escalate to your storage vendor. Thanks!

[Yuki]

Any chance to get this tool without a support case? We are on active support, but calling/e-mailing and explaining what it is we need and getting it will take longer than to write this post

[Vitaliy S.]

You can always send the link to this forum post in order not to explain everything from scratch

[Yuki]

I did. Had to open support case, now sending logs and will be doing webex session most likely in order for Veeam team to run the tool.

[Gostev]

The tool I was referring to is for different issue (data copy slowdown within the given job run), this was my response to Alexandr's query. In your case (job slows down every day more and more), as Tom explained, the issue is most likely due VBK fragmentation and large VBK size. There is no need to run the support tool in your case.

[Yuki]

Do you mean fragmentation of VBK on target file system or within VBK itself? If it is the target system, it is on EXT4 which is said not to need defrag AFAIK. If it is withint he VBK - then i would love to see a VBK defrag tool.

[tsightler]

Within the VBK file. Because of compression and dedupe, blocks are not stored within the VBK in sequence, but rather as new blocks are stored in a block in which they fit, once old blocks are rolled out. For VM disks with lots of changed blocks, the VBK can become fragmented quite quickly. This is why it is generally recommended to use forward incremental and synthetic fulls with this VMs with a high change rate.

I believe that you would see at least some improvement (perhaps significant) by disabling compression and dedupe on this job, although you would need to run a new full backup for the changes to take effect since the current full is already compressed and deduped.

[Yuki]

Oh yea, does it look anything like this?

[Gostev]

If it is within he VBK - then i would love to see a VBK defrag tool.

There is one, it is called Active Full and this is just one of many reasons to perform those periodically (also for VBK shrinking, and to avoid a very long incremental chain). At least once in a few months would be my recommendation.

[Yuki]

I would love to have ability to run pereodic active full backups, and while we can do that on-site, we can't push 8TB of data over an internet connection. Seeding it also is a problem as a single full on a file server is 5.7TB.

[Gostev]

Backup across WAN is a whole different story though. With that, I highly doubt the performance slowdown we are talking about in this topic will have any material impact... instead, the primary performance bottleneck is going to be the internet link speed, and by far...

[ncyclon]

We have different target devices like iscsi on synology, raid10 on promise storage and mixed storage of local and external disk storage. All this storages have same problem. Looks like 12-18-25-30-36-35-32-28....etc until 10 MB/s.

[deduplicat3d]

I have this same problem. It is especially bad because I'm using a dedupe device (Forward Incremental). I talked to someone in R&D and it turns out that during any operation that will write to the backup file (before each one of the vm config files and before each vdisk), Veeam reads through ALL of the metadata that is written into the backup files. This metadata is distributed throughout the entire file (vbk and incremental files NOT vbm file).

[Gostev]

Well, this is obviously not the same problem, as reading metadata once BEFORE processing vdisk cannot cause performance degradation DURING the disk processing. Plus, deduplicating storage is the whole different story overall. Please, let's not mix different issues into the same topic - there are as many as 3 different issues described here, and this makes it totally unmanageable conversation. I recommend we continue discussing OP's problem exclusively here. Thanks!

[deduplicat3d]

Sorry I wasn't trying to complicate things, I had already accepted my problem for what it is. I was just noting that since there are multiple vdisks it would then re-read the metadata each time on the successive vdisks and on each of the next vdisk there is much more to read.