Comprehensive data protection for all workloads
Post Reply
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Performance drop as backup file grows?

Post by Yuki » Mar 21, 2013 6:13 am

Hey all,
Just wanted to check if anyone else is seeing the same issue. It seems that as our backup file grows, the performance on the reverse incremental runs gets lower and lower. I understand that this is a target storage issue and not Veeam's per se, but wanted to see if anyone else is observing similar results.

Our 30TB NAS (linux based on 3TB SATA drives and EXT4 RAID5 volume) is getting progressively slower as the reverse incremental file grows. Right now a full backup file is at 5.8TB. Anyone else noticed this on other systems?

tsightler
VP, Product Management
Posts: 5383
Liked: 2217 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Performance drop as backup file grows?

Post by tsightler » Mar 21, 2013 11:53 am

What block size? (Local/LAN/WAN)

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Performance drop as backup file grows?

Post by Yuki » Mar 21, 2013 4:17 pm

seems to happen at all block sizes, but right now it is configured for Local/Optimal.

tsightler
VP, Product Management
Posts: 5383
Liked: 2217 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Performance drop as backup file grows?

Post by tsightler » Mar 21, 2013 8:53 pm

Yes, it will happen at all block sizes, but it will happen sooner with smaller block sizes. With a 5.8TB VBK I would have to assume that the total size of the VM data in the backup is probably twice that much (assuming at least 2x compression). Is that correct?

As a best practice it is recommended to put no more that 8TB of VM size data into a single backup job if you are looking to achieve maximum performance. In general you will see much better performance out of running multiple smaller jobs rather than one huge job.

Also, VBK files do become fragmented over time which can also lead to some performance degradation and is another reason why running a full backup every few months is sometimes recommended, especially for huge backup files.

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Performance drop as backup file grows?

Post by Yuki » Mar 21, 2013 10:07 pm

Well, it's a single 10TB VM, so we can't split it. We have very little benefit from the compression. This is a 2012 with dedupe enabled and total used space (After dedupe) at 5.7TB, which is also the size of our backup file. Windows dedupe does save us about 2.8TB of space though.

tsightler
VP, Product Management
Posts: 5383
Liked: 2217 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Performance drop as backup file grows?

Post by tsightler » Mar 22, 2013 12:19 am

With a VM like that you'd see the best performance by disabling compression and dedupe in the job since the data is already deduped. Not only would this reduce the amount of CPU being used, but it should also reduce the memory overhead and disk I/O since there will be no hash table for the dedupe to hold in memory or flush to disk. You would still see some degradation due to fragmentation but it should be better.

ncyclon
Novice
Posts: 3
Liked: never
Joined: Mar 22, 2013 7:57 am
Full Name: Alexandr

Re: Performance drop as backup file grows?

Post by ncyclon » Mar 22, 2013 8:05 am

Hi, I have same issue. VM size 6.6Tb. After start of job I see 30-40 MB/s, but after transfer 500-600Gb speed down to 16 MB/s. Dedupe and optimal compression set.

Is it normal behavior?

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Performance drop as backup file grows?

Post by Gostev » Mar 22, 2013 10:36 am

Alexandr - assuming you are running current product version - no, performance reduction within the given job run is not normal. However, I can say that we've seen this before with certain storage devices. In fact, this one problem we have probably spent more time researching more than any other problem we ever had with the product. In previous similar support cases, this came down to the issue with the storage itself. Our support now has a tool that can do basic continuous I/O to a large file (thus removing Veeam from the picture completely). Best course of actions would to open a support case, confirm if this is the same storage issue using this tool, and if so then escalate to your storage vendor. Thanks!

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Performance drop as backup file grows?

Post by Yuki » Mar 22, 2013 4:09 pm

Any chance to get this tool without a support case? We are on active support, but calling/e-mailing and explaining what it is we need and getting it will take longer than to write this post :)

Vitaliy S.
Product Manager
Posts: 22777
Liked: 1526 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Performance drop as backup file grows?

Post by Vitaliy S. » Mar 22, 2013 10:02 pm

You can always send the link to this forum post in order not to explain everything from scratch ;)

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Performance drop as backup file grows?

Post by Yuki » Mar 23, 2013 5:35 am

I did. Had to open support case, now sending logs and will be doing webex session most likely in order for Veeam team to run the tool.

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Performance drop as backup file grows?

Post by Gostev » Mar 23, 2013 1:30 pm

The tool I was referring to is for different issue (data copy slowdown within the given job run), this was my response to Alexandr's query. In your case (job slows down every day more and more), as Tom explained, the issue is most likely due VBK fragmentation and large VBK size. There is no need to run the support tool in your case.

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Performance drop as backup file grows?

Post by Yuki » Mar 23, 2013 3:36 pm

Do you mean fragmentation of VBK on target file system or within VBK itself? If it is the target system, it is on EXT4 which is said not to need defrag AFAIK. If it is withint he VBK - then i would love to see a VBK defrag tool.

tsightler
VP, Product Management
Posts: 5383
Liked: 2217 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Performance drop as backup file grows?

Post by tsightler » Mar 23, 2013 3:57 pm

Within the VBK file. Because of compression and dedupe, blocks are not stored within the VBK in sequence, but rather as new blocks are stored in a block in which they fit, once old blocks are rolled out. For VM disks with lots of changed blocks, the VBK can become fragmented quite quickly. This is why it is generally recommended to use forward incremental and synthetic fulls with this VMs with a high change rate.

I believe that you would see at least some improvement (perhaps significant) by disabling compression and dedupe on this job, although you would need to run a new full backup for the changes to take effect since the current full is already compressed and deduped.

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Performance drop as backup file grows?

Post by Yuki » Mar 23, 2013 8:58 pm

Oh yea, does it look anything like this?

Image

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Performance drop as backup file grows?

Post by Gostev » Mar 23, 2013 9:16 pm

Yuki wrote:If it is within he VBK - then i would love to see a VBK defrag tool.
There is one, it is called Active Full :wink: and this is just one of many reasons to perform those periodically (also for VBK shrinking, and to avoid a very long incremental chain). At least once in a few months would be my recommendation.

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Performance drop as backup file grows?

Post by Yuki » Mar 23, 2013 9:26 pm

I would love to have ability to run pereodic active full backups, and while we can do that on-site, we can't push 8TB of data over an internet connection. Seeding it also is a problem as a single full on a file server is 5.7TB.

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Performance drop as backup file grows?

Post by Gostev » Mar 23, 2013 11:34 pm

Backup across WAN is a whole different story though. With that, I highly doubt the performance slowdown we are talking about in this topic will have any material impact... instead, the primary performance bottleneck is going to be the internet link speed, and by far...

ncyclon
Novice
Posts: 3
Liked: never
Joined: Mar 22, 2013 7:57 am
Full Name: Alexandr

Re: Performance drop as backup file grows?

Post by ncyclon » Mar 24, 2013 5:14 pm

We have different target devices like iscsi on synology, raid10 on promise storage and mixed storage of local and external disk storage. All this storages have same problem. Looks like 12-18-25-30-36-35-32-28....etc until 10 MB/s.

deduplicat3d
Expert
Posts: 100
Liked: 11 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: Performance drop as backup file grows?

Post by deduplicat3d » Mar 24, 2013 8:59 pm

I have this same problem. It is especially bad because I'm using a dedupe device (Forward Incremental). I talked to someone in R&D and it turns out that during any operation that will write to the backup file (before each one of the vm config files and before each vdisk), Veeam reads through ALL of the metadata that is written into the backup files. This metadata is distributed throughout the entire file (vbk and incremental files NOT vbm file).

Gostev
SVP, Product Management
Posts: 24465
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Performance drop as backup file grows?

Post by Gostev » Mar 24, 2013 9:08 pm

Well, this is obviously not the same problem, as reading metadata once BEFORE processing vdisk cannot cause performance degradation DURING the disk processing. Plus, deduplicating storage is the whole different story overall. Please, let's not mix different issues into the same topic - there are as many as 3 different issues described here, and this makes it totally unmanageable conversation. I recommend we continue discussing OP's problem exclusively here. Thanks!

deduplicat3d
Expert
Posts: 100
Liked: 11 times
Joined: Nov 04, 2011 8:21 pm
Full Name: Corey
Contact:

Re: Performance drop as backup file grows?

Post by deduplicat3d » Mar 24, 2013 10:34 pm

Sorry I wasn't trying to complicate things, I had already accepted my problem for what it is. I was just noting that since there are multiple vdisks it would then re-read the metadata each time on the successive vdisks and on each of the next vdisk there is much more to read.

Post Reply

Who is online

Users browsing this forum: lyya, Majestic-12 [Bot] and 44 guests