Improving backup speed and time window / duration

[albertwt]

People,

I'm having a problem with the large VM backup window that is getting longer lately.

The VMs:
Exchange Server Mailbox servers ~ 8 TB+ each VM took 17 hrs to complete.
SQL Servers ~ 2 TB each VM took 7-8 hrs to complete.
File Servers NAS ~ 5-6 TB took 3-4 hrs to complete.

Backup Repository:
LUN 1 - RAID 50 [HP D2700 25x 450GB 6G SAS 10K SFF Dual Port HDD 11.2TB]
LUN 2 - RAID 6 ADG [HP D2700 25x 450GB 6G SAS 10K SFF Dual Port HDD 11.2TB]
LUN 3, 4, 5 and 6 - RAID 5 [HP D2600 12x 3TB 6G SAS 7.2K LFF Dual port MDL HDD 36TB]

From the VBR console, the bottleneck is the Target.

So can I convert them all from Reverse Incremental into Forward Forever incremental ?

Thank in advance.

[nielsengelen]

That would be one option. Reverse IO requires quite some IO and RAID5 and RAID6 may not be the best.

If you switch from the reverse incremental method to the forward incremental method, VBR will create a set of incremental backups next to the reverse incremental chain. The full backup in the reverse incremental chain will be used as a starting point for produced increments.

In regards to the time, are these timeframes about an incremental or full backup ?

[albertwt]

Thanks Niels,

This is in regards to the incremental backup daily.

[Vitaliy S.]

As another possible enhancement you can consider switching to RAID 10 in any new similar deployment, should also give you a bit more IOPs performance.

[albertwt]

Hi Vitaliy,
I can't switch it to raid 10 because the disk space is running very low, hence I create raid 5 to get more disk capacity.

Does changing from reverse incremental to forward incremental can trigger full backup on the next backup schedule ?

[veremin]

Does changing from reverse incremental to forward incremental can trigger full backup on the next backup schedule ?

The latest restore point in that case would be already a full backup, so no new full backup cycle should take place. Thanks.

[albertwt]

Hi All,

I've got large Exchange Server (Mailbox server) VMs that is backed up by Veeam daily, but the problem is the job taking long to finish around 13+ hours

VM size total is 9 TB+ according to VBR
there are 7 VMDK in the VM that is 1.89 TB in size which hold large Exchange mailbox .edb file per drive.

Job settings:


Bottleneck is on the: Target - Processing rate is 15 - 20 MB per second.

The target backup physical repo server is in the same data center same rack with the Exchange Server vSphere environment.


The RAID 5 is consists of 12 x 3 TB SAS HDD.


Any kind of tips and suggestion would be greatly appreciated.

Thanks.

I'm using Veeam Backup & Replication 9.0.0.1715

[ACC]

Based on the information provided, the cause of the low performance is likely the high random IO workload. Reverse incremental is more disk space efficient, but it does come with a penalty that the workload will consist of primarily random IOs. This kind of workload on a raid 5 set (or 6 for that matter) is tough on the performance. Raid 5 is, to put it mildly, not great in handling random write IOs. To improve the performance, switch the job to forward incremental to get more sequential IO workload and/or use raid 10 which has a lesser penalty on dealing with random write IOs.

[albertwt]

Hi Acc,

Thanks for the suggestion.
I need to maximize the disk space or the capacity hence I selected RAID 5 in my DAE configuration.

If I change the backup job from Reverse Incremental to Forward Incremental forever, does it going to consume more disk space ?

[foggy]

You need to switch to simple forward incremental, since forever forward will still require random I/O during transformation. Yes, you would need more space with forward incremental mode.

[albertwt]

Alex,
Is there any calculator or estimator of how much more disk space is needed for the Forever forward incremental ?

[foggy]

Forever forward incremental doesn't require more space, however, forward incremental does. You can perform space estimation here.

[albertwt]

Alex,

So what's the different on those two mode ?



My understanding is:

Forever forward incremental mode: Incremental (Recommended) selected, Create Synthetic Full Backups periodically option unchecked and Transform previous backup chains into rollbackoption unchecked. Hence disk space usage remains the same.

Forward incremental mode: Incremental (Recommended) selected, Create Synthetic Full Backups periodically option checked and Transform previous backup chains into rollbackoption checked. Additional disk space usage needed.

Is that correct ?

[foggy]

Please review the corresponding user guide section.

[sanjaykrk]

Albertwt,
As per my understanding, Your description of two modes is correct.

In forever forward incremental mode, after first initial full backup, you will only be writing incrementals (forever).
But, once you exceed the number of restore points based on your retention policy, Veeam will merge this 'First Full backup' with 'next increment' to prepare updated Full backup to stay within the number of restore point allowed. This may involve random I/O and this merge operation can become frequent (may be somebody can confirm if my understanding is correct).
Advantage: disk space use is on the lower side (One full backup image +incrementals).

In Forward incremental, you start with a full backup, followed by incrementals and periodic Synthetic or Active Full backup.
Depdending on your retention policy, disk space utilization should stablize after a few cycles.
This should be the fastest in terms of backup window. But depending on your retention policy/restore points/frequency of full backup, your disk space usage will vary.

For Reverse, disadvantage: its slow. Advantage: you most recent backup image is always 'full'.

[ACC]

For Reverse, disadvantage: its slow.

Reverse incremental is slow because raid 5 and spinning disks as said aren't friends with a continuous high random IO workload, however would replace it with raid 0/1/10 with ssds or even better nvme-ssds it would be fast because an enterprise ssd is very capable handling such workload. Lets say the disks Albertwt is using can do 100 IOs (im rounding up) and have perfect schaling the array handles 1200 IOs. One Intel DC P3700 2TB ssd is rated at 450k IOs read / 175k IOs write, even if Veeam is only able to sqeeze a quarter of the rated performance from the P3700 it outperforms the spinning disk array with ease. However most backup budgets doesnt allow to spend a couple grand on a highend enterprise ssd.

Reverse incremental:
Pro: last backup is the full backup, storage use is very stable.
Con: backup is in essence two operations (backup and merge) combined in one operation which tend to be high random IO workload.

Forever forward incremental:
Pro: storage use is stable, backup is usually sequential IO workload, merge happens when backup is done.
con: merge tend to be random IO workload.

Forward incremental:
Pro: backup is usually sequential IO workload, creating (synthetic) full backups tend to be sequential IO workload, no merge operations.
con: storage use fluctuates based on settings.

[foggy]

Sanjay, your understanding of the backup methods is correct.