Synthetic Full Backups Slow

[amatteo78]

B&R now is VM inside infrastructure vmware sphere.

[Vitaliy S.]

tomorrow I create new server windows as repository and other server as proxy.

Make sure you do this on service provider's location/site.

[amatteo78]

every my server are inside location of service provider.

[Vitaliy S.]

Then repository server is located next to backup server. In this case installing additional proxy/repository server would not help, as everything is located on the local network. Seems like the NAS device cannot handle random I/O load and active full is a way to go.

[amatteo78]

I think same.
Thanks very much to yours helps.

M.

[amatteo78]

Hello,

this night backup job spent only 50 minutes, I disable "synthetic full backup" and add 2 new VM. Now seems good.
This is print screen my config.




Thanks

M.

[tienlamnguyen]

Hello is this normal ? On of my synthetic full jobs is still running after 15 hours (11 VM). On the other hand, another synthetic full (4 VM - all DC) took only 41 minutes.

What could be causing it ? is it because I have too many VM on my backup jobs ? should I split my backup jobs in 2 separate Backup jobs ?

I have 1 physical server that host everything (backup+proxy)
My repository is SYNOLOGY NAS with direct iSCSI SAN connectivity.

thanks for your help

[Shestakov]

Hello,
What is the length of the backup chain, i.e. how many incrementals become the Full?
What version of VBR are you at?
Thanks!

[tienlamnguyen]

What is the length of the backup chain-how many incrementals become the Full?
7 restore points

What version of VBR are you at?
version 8 latest patch

From the post chain - Am i to understand that If my server has dedupe drives - It will take longer ?

I will try amatteo78 settings and see if it will improve. If not i will simply disable synthetic full.

thanks

[Shestakov]

Yes, it might take time, but since you have 2 backup jobs in the same infrastructure, it should not be the reason.

I would also try to run Active Full manually and then your synthetics should run faster.
Thanks!

[MarkusRadtke]

Hi,

We have the problem that the weekly scheduled synthetic full take a lot of time and so the datacopy-jobs failed because of the little remaining time-windows.

Our virtual environment:

- vSphere Standard with 5 hosts, all 192-256 GB RAM, all 2x Intel CPU with 6 or 10 core and HT, all 2x10 GBits connected and all connected to 8 GBits- F/C SAN (FTS Eternus DX 200, 48x 1,2 TB SAS 10k)

Backup environment:
- virtual Veeam Server, Windows Server 2008 R2 with 6vcpu and 10 GB RAM, (proxy disabled)
- 3 virtual Proxy-Servers, Windows Server 2008 R2 or 2012 R2 with 6vcpus and 6 GB RAM

Targets:
- Primary and secondary target are equal Windows server 2008 R2 based Servers with 10 GBit, 6-Core-Intel CPU 2,6 GHz (+HT), 32 GB RAM, 24x 3 TB NL SAS (7,2k), 22 Disks in Raid 6 Mode for the backups.

Backup Strategy:
- one primare Backup-Job for all VMs except the templates (205 VMs with ~11 TB allocated Storage)
- incremental with synth. fulls
- daily schedule with weekly synthetic full, monthly active full
- 28 Days retention (= 4 weeks)

- 2 Jobs for the templates (primary and secondary target) <-- no problems

- 4 backupCopy-Jobs (splitted because of performance)
- daily cycle
- 45 days retention

so far, so good....

The primary backup-job runs fine if no synthetic full is scheduled and take ~5 hours.

The secondary backups runs good after investigating and splitting the big job.

But when the synthetic full is running, the job takes nearly 20 hours and so the backup-copy-jobs failed and need 2 or 3 cycles to become normal again.

We've updated from v7 to v8 latest patch and it seems that the overall performance slow down...

Are we able to change something to increase the performance (except changing the hardware, this is not an option at the moment)? Is the new v8-backup-option useable to optimize this?

Thanks,

Markus

[Shestakov]

Hello Markus,
Thanks for the comprehensive description.
The way to get faster Synthetic Full is to have backup proxy residing on the same site with the repository, so traffic don`t need to go back and forth.
Another tips are to use active Fulls instead of Synthetic ones or to split 205 VMs into two/several "primary" backup jobs.
Please review the topic where your post has been merged for more info. Thanks.

[MarkusRadtke]

Hi Shestakov,

thanks for your reply. What do you mean with "proxies on the same site with repository"? The vsphere-environment, the proxies and the (primary) Backup-target are on the same network-site. The backup-Server is in another vlan but on the same network-switch/core-cluster than the veeam-proxy-VMs.

We really like the only one primary backup job because of the good dedupe-results and easier handling.

Active-Fulls mean "real" full backup every week, right? So the pruduction environment will be used longer time by backup processes?

I also will have a look at the other posting in this thread.

Thanks,

Markus

[MarkusRadtke]

Hi again,

will try with weekly active full backups instead synthetic full.. hope that helps

Thanks,

Markus

[Shestakov]

Markus,

Active-Fulls mean "real" full backup every week, right? So the pruduction environment will be used longer time by backup processes?

Yes, Active means Real. As was mentioned above, Real/Active full backup is a sequential write, even single spindle of slowest eco hard drive can handle that easily. Synthetic full, is lots of random IOPS. You need to chose where you want to put the load from full backup to - either you put it on your production environment by doing active (real) full backups, or you isolate it to backup storage (but this requires a storage with decent IOPS).

Here is also the outline of I/O operations for active / synthetic backup modes:
Active (real) full backup = 1x I/O on target (write for each block)
Synthetic full backup for incremental backup mode = 2x I/O on target (read + write for each block)
Synthetic full backup for incremental backup mode with transform = 4x I/O on target (read + write + read + write for each block)

Thanks.

[mwya]

Hey there,

Currently we’re using Veeam 8.0.0.2084.
Our backup job do always incremental backups and every Friday a synthetic full backup.
The virtual machines have a full size of 6,1TB.
An incremental backup takes about 3 hours. (Processed: 6,1TB; Read: 1,5TB; Transfered: 67,1GB; Processing Rate: 221MB/s)
An synthetic full backup takes about 50 hours. (Final .vbk file have a full size of 2,5TB)
The target storage is connected as a CIFS backup repository. It is a Synology NAS.
On the source storage we have a backup proxy to use hotadd functionality.
By the way only this backup job is performing on the target storage and no other job.

Are there any solutions to get more performance of such a backup job?

Thanks in advance

[Shestakov]

Hello,
Do you have repository on the same site with the proxy? What`s the statistics of synthetic full run?
You can also review the topic for more information. Thanks!

[geordi]

Hi

mwya is colleague and I am writing instead of him.
The repository is on the same physical LAN segment as the proxy.
I have read a bit in the User Guide. As far as I understood, the synth full is done from the data mover service which can not be run directly on a SMB share. So the whole creation is done from one of the Backup Proxy Servers and the whole data is transferred via LAN which is limited to 1GB physical at the moment. And as the backup repository is using the 1GB LAN port also for a Shared Folder Sync and the creation of the normal backup jobs this is our bottelneck. Am I right with my assumption?

So the perfect deal would be to re-design our environment, put the NAS for example as JBOD via SAS or 10G LAN to a physical windows machine which can hold the Data Mover and the LAN-Limitation would be gone. Correct?

Regards,
geordi

[foggy]

Correct, however also keep in mind that your Synology NAS might not be able provide enough random IOPS that synthetic full requires. So the actual bottleneck may be on its writer component side as it is not able to accept data any faster.