Synthetic Backup

[kratos31]

We are finding that doing a Synthetic full takes about twice the amount of time to do a full backup.

Is this normal?

[Gostev]

Yes, depending on the backup storage speed, and how it is connected to the backup server.

[kratos31]

its a NAS, with 2gb iSCSI connection

[Gostev]

Basically, synthetic full takes load off your production storage, network and VM itself, and makes backup storage pay for this

Active full backup means just sequential writes from backup target perspective, so each active full backup block causes in 1 write (1 I/O operation on backup storage). On the other hand, synthetic full causes random reads in addition to sequential writes. Each synthetic full backup block causes in 1 random read, and 1 write (so 2 I/O operations). In other words, that is 2x more I/O on target storage. And if you enable transform, 3x more I/O.

So it should be clear now why in cases with very fast source data retrieval and processing speed, synthetic full may take longer than active full. Where synthetic full backup performance shines, is when connection to backup target is slow (remote backups over WAN).

Although, regardless performance, the biggest benefit of synthetic fulls for any environment is, of course, reduction of the amount of time during which source VM runs off snapshot when full backup is created. Shorter snapshot TTL = > smaller snapshots => faster snapshot commit => less time when production VM's performance is affected.

[Daveyd]

Basically, synthetic full takes load off your production storage, network and VM itself, and makes backup storage pay for this

Although, regardless performance, the biggest benefit of synthetic fulls for any environment is, of course, reduction of the amount of time during which source VM runs off snapshot when full backup is created. Shorter snapshot TTL = > smaller snapshots => faster snapshot commit => less time when production VM's performance is affected.

Would this matter if backups are done during off peak hours with little change to each VM's vmdk?

I run multiple jobs to the Exagrid appliance and have found issues where running synthetic fulls take a considerable bit of time and run into other jobsbackup windows. The Exagrid appliance is being hit with a lot of IOs doing the synthetic full backup and then adding more IOs when another job(s) kick off causing things to slow down. A normal full usually does not take as long as a synthetic one and in our case does not run into a window of another backup job.

[Gostev]

There is no univeral answer if this would matter, you need to test and see. Some applications like big Exchange or SQL servers are really "chatty" even during off peak hours, and cause snapshots to grow fast. Some other applications, or small servers are less chatty. Some customer are observing noticeable effects of large snapshot commit on production VM performance and that is a real issue for them, other are seeing this as well - but don't care if this happens during off hours, and some customers are not seeing any issues with commit at all. Every environment is different (think backup targets alone), provided above are simply universal considerations that everyone should keep in mind when evaluating their options with synthetic fulls.

[Daveyd]

Thanks for the reply.

We are a 24x7 Hospital and are backing up a little over 100 VMs but do not have a lot of heavy hitting VMs during non peak hours or commit issues after they are backed up. Both synthetic fulls and regular fulls have worked well for us. I like the fact that the regular fulls get done a lot sooner so the other jobs can have access to all the Exagrid's IOs when they run, making those jobs faster as well.

If Synthetic fulls gave us other benefits like better de-deupe ratio or something along those lines, I think it would be more benefical for us.

[aeccles]

So it should be clear now why in cases with very fast source data retrieval and processing speed, synthetic full may take longer than active full. Where synthetic full backup performance shines, is when connection to backup target is slow (remote backups over WAN).

Do you guys have a calculation that we could use to determine the cut off? For example I would consider a 5mbps link slow, but I think it's faster than waiting for a synthetic with transform turned on. I imagine there is some type of formula we could apply to figure out when synthetics make sense (at least in regard to time- I understand there are other benefits)

I'm struggling with some incredibly long synthetic times (10 hours for 3 small IIS servers that only take up about 50gb combined and have very little changes each day)
Also, would you expect synthetic jobs to take longer if the storage is a CIFS share vs. local storage?

[Vitaliy S.]

Also, would you expect synthetic jobs to take longer if the storage is a CIFS share vs. local storage?

Local storage or any Windows-enabled repository is a more preferable target for your backups. Synthetic full will take considerable amount of time if you're backing up over the WAN link to the CIFS share without any "proxying" server.

[aeccles]

Ok, my CIFS share is actually local. Here are the full details on a job - can you tell me if this is "normal" ? It doesn't make any sense that this should take a week to run.

Storage is a CIFS share - 1gbps connection on the same switch
Job is single server (Exchange) and the total disk size is 650gb.
Here are the stats from the initial backup of the server (1 week ago)
Summary:
Duration:3:47:48
Processing Rate: 50MB/s
Bottleneck: Network

Data:
Processed: 650.00GB (100%)
Read: 630.1GB
Transferred: 451.2 GB (1.4)

Status:
Success: 1
Warnings: 0
Errors: 0

Keeping 7 restore points
Storage settings on the job are: Incremental, Enable Synthentic Fulls, Transform previous full backup chains into roll backs.
Compression level is optimal
Storage optimization for LAN target
CBT is on

Daily incrementals take about 1 hour to run and transfer about 30gb.

All of that is happening with expected time frames/speed, however when the synthetic full with transform runs it's taking FOREVER. For example, one is running right now - it's been running for 14 hours and I can tell by the size of the VBR on the CIFS share that it is les than 8% complete. Rough math has this job taking 175 hours to complete.

All of the other Veeam jobs are disabled so nothing is waiting for resources. Both the backup proxy and the CIFS share are dedicate for Veeam (nothing else is running)


Thanks for your help. (and yes, I have a case open - 00530692 - but unfortunately like most calls we've made lately, support has been slow and apathetic (just being honest- a year ago support was outstanding)

[Vitaliy S.]

What kind of storage device have you exposed via CIFS protocol?

[aeccles]

It's a Windows server, but a CIFS share. And the reason it's a CIFS share has to do with the network. I am doing backup copy jobs to a data center and while the backup proxy server has connectivity to my data center, the backup repository server does not. I initially thought this would not be a problem since the backup proxy would act as the WAN accelerator, however I found (through support) that the source backup repository actually makes a connection to the target backup repository prior to the WAN accelerator being invoked. Since those could not connect directly, support recommended that I change the backup repository to CIFS (instead of Windows) and setup the connection to it from the backup proxy (which has connectivity to the data center) That change did resolve the connectivity issue.

I want to point out though that the problem I described in the post above does not involve the data center storage or wan accelerators.... it's the local synthetic im having a problem with (unless I don't understand something that is happening)

[Vitaliy S.]

Is there any reason why you have selected synthetic full and not active full instead? Seems like your target storage cannot keep up with random I/O, so active full might give you a much better performance.

There is an existing topic that can give you more information on the observed behavior > Synthetic Full Backups Slow

[aeccles]

Target storage is pretty good- I would be surprised if it is that overworked? It's a Dell R520 with a Dell PERC H710 w/ 1gb of RAM on the card. 6 Disk RAID5- disks are 6.0 SAS 7200PRM. 32 CPU cores @ 2.5ghz and 16gb of RAM.

I read through the post you linked and it actually just makes me think there is in fact something wrong here because the first poster talked about a 700gb VM synthetic taking 12 hours and the second poster spoke of "double the time" for his synthetic now I realize there are a ton of factors and you can't really compare job times, but remember this synthetic job I am talking about is taking 7 days while a full backup takes less than 4 hours.

I'm using synthetic because I have Backup Copy job associated to this job and the Backup Copy job is going over the WAN. It was my understanding that synthetic is the recommended setting for this scenario?

[veremin]

I'm using synthetic because I have Backup Copy job associated to this job and the Backup Copy job is going over the WAN. It was my understanding that synthetic is the recommended setting for this scenario?

In fact, the backup copy job doesn't t copy backup file as a whole, but rather creates required restore points in "target" location, using its synthetic logic and VM data in source backup repositories. So, it doesn't matter what backup mode is being used by source backup job. The type of scheduled full backups doesn't affect backup copy job, as well.

For the purpose of testing can you switch temporarily to active full backup, instead of synthetic, and see whether it makes any difference?

Thanks.

[Vitaliy S.]

Yes, you're correct that there are many factors that can affect synthetic full performance, however the most important one is storage performance.