Comprehensive data protection for all workloads
Post Reply
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Improving backup speed and time window / duration

Post by albertwt »

People,

I'm having a problem with the large VM backup window that is getting longer lately.

The VMs:
Exchange Server Mailbox servers ~ 8 TB+ each VM took 17 hrs to complete.
SQL Servers ~ 2 TB each VM took 7-8 hrs to complete.
File Servers NAS ~ 5-6 TB took 3-4 hrs to complete.

Backup Repository:
LUN 1 - RAID 50 [HP D2700 25x 450GB 6G SAS 10K SFF Dual Port HDD 11.2TB]
LUN 2 - RAID 6 ADG [HP D2700 25x 450GB 6G SAS 10K SFF Dual Port HDD 11.2TB]
LUN 3, 4, 5 and 6 - RAID 5 [HP D2600 12x 3TB 6G SAS 7.2K LFF Dual port MDL HDD 36TB]

From the VBR console, the bottleneck is the Target.

So can I convert them all from Reverse Incremental into Forward Forever incremental ?

Thank in advance.
--
/* Veeam software enthusiast user & supporter ! */
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Improving backup speed and time window / duration

Post by nielsengelen » 1 person likes this post

That would be one option. Reverse IO requires quite some IO and RAID5 and RAID6 may not be the best.

If you switch from the reverse incremental method to the forward incremental method, VBR will create a set of incremental backups next to the reverse incremental chain. The full backup in the reverse incremental chain will be used as a starting point for produced increments.

In regards to the time, are these timeframes about an incremental or full backup ?
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Improving backup speed and time window / duration

Post by albertwt »

Thanks Niels,

This is in regards to the incremental backup daily.
--
/* Veeam software enthusiast user & supporter ! */
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Improving backup speed and time window / duration

Post by Vitaliy S. » 1 person likes this post

As another possible enhancement you can consider switching to RAID 10 in any new similar deployment, should also give you a bit more IOPs performance.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Improving backup speed and time window / duration

Post by albertwt »

Hi Vitaliy,
I can't switch it to raid 10 because the disk space is running very low, hence I create raid 5 to get more disk capacity.

Does changing from reverse incremental to forward incremental can trigger full backup on the next backup schedule ?
--
/* Veeam software enthusiast user & supporter ! */
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Improving backup speed and time window / duration

Post by veremin » 1 person likes this post

Does changing from reverse incremental to forward incremental can trigger full backup on the next backup schedule ?
The latest restore point in that case would be already a full backup, so no new full backup cycle should take place. Thanks.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

[MERGED] Backing up Exchange Server taking 13+ hours to fini

Post by albertwt »

Hi All,

I've got large Exchange Server (Mailbox server) VMs that is backed up by Veeam daily, but the problem is the job taking long to finish around 13+ hours

VM size total is 9 TB+ according to VBR
there are 7 VMDK in the VM that is 1.89 TB in size which hold large Exchange mailbox .edb file per drive.

Job settings:
Image

Bottleneck is on the: Target - Processing rate is 15 - 20 MB per second.

The target backup physical repo server is in the same data center same rack with the Exchange Server vSphere environment.
Image

The RAID 5 is consists of 12 x 3 TB SAS HDD.


Any kind of tips and suggestion would be greatly appreciated.

Thanks.

I'm using Veeam Backup & Replication 9.0.0.1715
--
/* Veeam software enthusiast user & supporter ! */
ACC
Enthusiast
Posts: 25
Liked: 5 times
Joined: Apr 29, 2015 10:38 am
Full Name: ACC
Contact:

Re: Backing up Exchange Server taking 13+ hours to finish

Post by ACC » 1 person likes this post

Based on the information provided, the cause of the low performance is likely the high random IO workload. Reverse incremental is more disk space efficient, but it does come with a penalty that the workload will consist of primarily random IOs. This kind of workload on a raid 5 set (or 6 for that matter) is tough on the performance. Raid 5 is, to put it mildly, not great in handling random write IOs. To improve the performance, switch the job to forward incremental to get more sequential IO workload and/or use raid 10 which has a lesser penalty on dealing with random write IOs.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Backing up Exchange Server taking 13+ hours to finish

Post by albertwt »

Hi Acc,

Thanks for the suggestion.
I need to maximize the disk space or the capacity hence I selected RAID 5 in my DAE configuration.

If I change the backup job from Reverse Incremental to Forward Incremental forever, does it going to consume more disk space ?
--
/* Veeam software enthusiast user & supporter ! */
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Improving backup speed and time window / duration

Post by foggy » 2 people like this post

You need to switch to simple forward incremental, since forever forward will still require random I/O during transformation. Yes, you would need more space with forward incremental mode.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Improving backup speed and time window / duration

Post by albertwt »

Alex,
Is there any calculator or estimator of how much more disk space is needed for the Forever forward incremental ?
--
/* Veeam software enthusiast user & supporter ! */
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Improving backup speed and time window / duration

Post by foggy » 1 person likes this post

Forever forward incremental doesn't require more space, however, forward incremental does. You can perform space estimation here.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Improving backup speed and time window / duration

Post by albertwt »

Alex,

So what's the different on those two mode ?

Image

My understanding is:

Forever forward incremental mode: Incremental (Recommended) selected, Create Synthetic Full Backups periodically option unchecked and Transform previous backup chains into rollbackoption unchecked. Hence disk space usage remains the same.

Forward incremental mode: Incremental (Recommended) selected, Create Synthetic Full Backups periodically option checked and Transform previous backup chains into rollbackoption checked. Additional disk space usage needed.

Is that correct ?
--
/* Veeam software enthusiast user & supporter ! */
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Improving backup speed and time window / duration

Post by foggy » 2 people like this post

Please review the corresponding user guide section.
sanjaykrk
Influencer
Posts: 11
Liked: 3 times
Joined: Apr 09, 2016 12:12 am
Full Name: Sanjay kumar
Contact:

Re: Improving backup speed and time window / duration

Post by sanjaykrk » 1 person likes this post

Albertwt,
As per my understanding, Your description of two modes is correct.

In forever forward incremental mode, after first initial full backup, you will only be writing incrementals (forever).
But, once you exceed the number of restore points based on your retention policy, Veeam will merge this 'First Full backup' with 'next increment' to prepare updated Full backup to stay within the number of restore point allowed. This may involve random I/O and this merge operation can become frequent (may be somebody can confirm if my understanding is correct).
Advantage: disk space use is on the lower side (One full backup image +incrementals).

In Forward incremental, you start with a full backup, followed by incrementals and periodic Synthetic or Active Full backup.
Depdending on your retention policy, disk space utilization should stablize after a few cycles.
This should be the fastest in terms of backup window. But depending on your retention policy/restore points/frequency of full backup, your disk space usage will vary.

For Reverse, disadvantage: its slow. Advantage: you most recent backup image is always 'full'.
ACC
Enthusiast
Posts: 25
Liked: 5 times
Joined: Apr 29, 2015 10:38 am
Full Name: ACC
Contact:

Re: Improving backup speed and time window / duration

Post by ACC »

sanjaykrk wrote:For Reverse, disadvantage: its slow.
Reverse incremental is slow because raid 5 and spinning disks as said aren't friends with a continuous high random IO workload, however would replace it with raid 0/1/10 with ssds or even better nvme-ssds it would be fast because an enterprise ssd is very capable handling such workload. Lets say the disks Albertwt is using can do 100 IOs (im rounding up) and have perfect schaling the array handles 1200 IOs. One Intel DC P3700 2TB ssd is rated at 450k IOs read / 175k IOs write, even if Veeam is only able to sqeeze a quarter of the rated performance from the P3700 it outperforms the spinning disk array with ease. However most backup budgets doesnt allow to spend a couple grand on a highend enterprise ssd.

Reverse incremental:
Pro: last backup is the full backup, storage use is very stable.
Con: backup is in essence two operations (backup and merge) combined in one operation which tend to be high random IO workload.

Forever forward incremental:
Pro: storage use is stable, backup is usually sequential IO workload, merge happens when backup is done.
con: merge tend to be random IO workload.

Forward incremental:
Pro: backup is usually sequential IO workload, creating (synthetic) full backups tend to be sequential IO workload, no merge operations.
con: storage use fluctuates based on settings.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Improving backup speed and time window / duration

Post by foggy » 1 person likes this post

Sanjay, your understanding of the backup methods is correct.
Post Reply

Who is online

Users browsing this forum: Semrush [Bot] and 224 guests