Comprehensive data protection for all workloads
cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Synthetic Full Backups Slow

Post by cffit »

I have a VM that is about 700GB in size. I realize this is large. The incremental backups for this machine go pretty fast, but the synthetic full takes about 8 hours to complete. I'm on v7 now as I saw there were others getting increased speed with v7 and synthetic full backups. Our SAN is an HP MSA P2000 and there is nothing else on here other than backup storage for VEEAM. Is this typical or is there something I can check see if I can get these synthetic fulls to go faster?

Vitaliy S.
Product Manager
Posts: 24341
Liked: 1888 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Synthetic Full Backups Slow

Post by Vitaliy S. »

Hi Christensen, what type of backup storage do you have? Is it a dedupe device by any chance?

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: Synthetic Full Backups Slow

Post by cffit »

No it is not. It is just a regular iSCSI storage SAN.

foggy
Veeam Software
Posts: 19540
Liked: 1781 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Synthetic Full Backups Slow

Post by foggy »

How it is added to Veeam B&R console (I mean what type of repository it is)?

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: Synthetic Full Backups Slow

Post by cffit »

It's an iSCSI volume that is presented as a drive letter to the local Windows 2008 R2 server. Last night it took 12 hours to create a synthetic backup of a VM that is 700GB or so and 544GB after backed up. The iSCSI SAN has nothing else connected to it other than the VEEAM server.

foggy
Veeam Software
Posts: 19540
Liked: 1781 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Synthetic Full Backups Slow

Post by foggy »

Generally, synthetic backup activity puts very high load on the target storage and time required to perform it depends mostly on the storage performance. Probably the time you see is expected for such a large full, however I would recommend to open a case with technical support so that they could review logs and verify what operations actually take most of the time during job operation.

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: Synthetic Full Backups Slow

Post by cffit »

Ok, I will open a case. This is an HP MSA G3 P2000 SAN with 10K SAS drives, 18 to be exact. And the only thing running to it was the synthetic full. Before v7 it was taking about half this long. Thanks

foggy
Veeam Software
Posts: 19540
Liked: 1781 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Synthetic Full Backups Slow

Post by foggy »

Keep us updated on the investigation results.

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: Synthetic Full Backups Slow

Post by cffit »

So support suggested I do a regular full backup since I've been doing synthetic fulls for a long time. They said after I do a "real" full backup, then I can go back to synthetics and they should run faster again. Has anyone ever heard of this?

veremin
Product Manager
Posts: 17882
Liked: 1673 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Synthetic Full Backups Slow

Post by veremin »

Yep, “forever synthetic” scenario might result in decreased speed performance, thus, it’s recommended to run active full backup on monthly basis.

Thanks.

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: Synthetic Full Backups Slow

Post by cffit »

Ok, that is great to know. Is this documented anywhere or how would I ever know this?

veremin
Product Manager
Posts: 17882
Liked: 1673 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Synthetic Full Backups Slow

Post by veremin »

From the forums, as you’ve just done. Also, this post might be helpful if you need additional clarification. Thanks.

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: Synthetic Full Backups Slow

Post by cffit »

I am thankful for the VEEAM forums and find them VERY helpful. I just wish some of the great information here would be noted in official documentation or an online "Best Practices" document that could be updated regularly.

And also, like the person in the other article you linked to, I didn not notice this until after the upgrade to v7, which is very odd.

veremin
Product Manager
Posts: 17882
Liked: 1673 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Synthetic Full Backups Slow

Post by veremin »

Actually, community forums thoroughly monitored and analyzed by our technical writers team. And significant part of useful tricks is eventually written either to user guide or to best practice guide. Though, in case of newly-introduced version it might take some time, until those best practices are formed.

Anyway, thanks for the feedback; much appreciated.

Thanks.

rrnworks
Lurker
Posts: 2
Liked: never
Joined: Jan 23, 2013 8:48 pm
Full Name: Chris Porosky
Contact:

Re: Synthetic Full Backups Slow

Post by rrnworks »

We have a new backup job and our very first synthetic full is taking a very long time (over twice as long as the normal full backup).

Gostev
SVP, Product Management
Posts: 26877
Liked: 4357 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Synthetic Full Backups Slow

Post by Gostev »

rrnworks wrote:We have a new backup job and our very first synthetic full is taking a very long time (over twice as long as the normal full backup).
This means your backup storage is too low on IOPS capacity. Normal full backup is sequential write, even single spindle of slowest eco hard drive can handle that easily. Synthetic full, on the other hand, is lots of random IOPS. You need to chose where you want to put the load from full backup to - either you put it on your production environment by doing active (real) full backups, or you isolate it to backup storage (but this requires a storage with decent IOPS).

BHB3805
Enthusiast
Posts: 28
Liked: 1 time
Joined: Nov 04, 2011 1:45 pm
Full Name: Dan Andryszak
Contact:

Re: Synthetic Full Backups Slow

Post by BHB3805 »

Is there target disk savings using a synthetic full vs. a "real" full backup?

foggy
Veeam Software
Posts: 19540
Liked: 1781 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Synthetic Full Backups Slow

Post by foggy »

No, synthetic full backup is completely identical to an active full, except the way it is created (how VM data is retrieved).

amatteo78
Influencer
Posts: 15
Liked: 1 time
Joined: Jun 07, 2014 9:32 am
Full Name: Matteo Abrile
Contact:

Re: Synthetic Full Backups Slow

Post by amatteo78 »

Hello,

from few days I start test Veeam B&R, I have about 15VM with 5TB disks. First backup spend about 2 hours, and consuming about 150GB on my remote storage to do it. This night Incremental spent very few time about 5 minutes each vm, but when do "synthetic full backup" spent more 4 hours. It's very slow instead first full backup.
Can I check somethings ? Thanks

M.

Vitaliy S.
Product Manager
Posts: 24341
Liked: 1888 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Synthetic Full Backups Slow

Post by Vitaliy S. »

Hi Matteo,

What do you see as bottleneck stats and what is your target storage?

Thanks!

amatteo78
Influencer
Posts: 15
Liked: 1 time
Joined: Jun 07, 2014 9:32 am
Full Name: Matteo Abrile
Contact:

Re: Synthetic Full Backups Slow

Post by amatteo78 »

Hello,

I have 2 NAS the first one where I have discs of my VMs, it's FreeNAS with iSCSI connect to host ESXi, it's no problem I see on stats and all fine. Then I have other NAS to put backup, this is shared NAS that I rent from my service provider where I have housing. They give me partition and I can use CIFS/NFS to share it. I don't know if it's has saturation because I can't see its stats.
But when I do first backup (full) it spend about 2 hours to do. Then when do incremental it spend few minutes every vm, but about 4 hours to do synthetic process.
Thanks

M.

Vitaliy S.
Product Manager
Posts: 24341
Liked: 1888 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Synthetic Full Backups Slow

Post by Vitaliy S. »

What is the connection link to this NAS box? If it is over the WAN, then do you have "proxying" server configured on the other side to keep traffic locally during synthetic full creation?

amatteo78
Influencer
Posts: 15
Liked: 1 time
Joined: Jun 07, 2014 9:32 am
Full Name: Matteo Abrile
Contact:

Re: Synthetic Full Backups Slow

Post by amatteo78 »

Yes, I connect it by WAN. what you mean with "take traffic locally" ??

Vitaliy S.
Product Manager
Posts: 24341
Liked: 1888 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Synthetic Full Backups Slow

Post by Vitaliy S. »

When you're using CIFS repository and use synthetic fulls, then traffic has to travel back and forth to create a VBK file (new full). If your WAN link is slow, then creating new VBK file can take some time.

Please see this link to know more about the option I'm referring to > http://helpcenter.veeam.com/backup/70/v ... b=proxying

amatteo78
Influencer
Posts: 15
Liked: 1 time
Joined: Jun 07, 2014 9:32 am
Full Name: Matteo Abrile
Contact:

Re: Synthetic Full Backups Slow

Post by amatteo78 »

Yes, I setup it. I use only 1 server to B&R server and proxy.... so I setup "this server".

Vitaliy S.
Product Manager
Posts: 24341
Liked: 1888 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Synthetic Full Backups Slow

Post by Vitaliy S. »

You should setup any Windows server as a "proxying" server on the site where your CIFS share is located, not on the source site. I don't think that you have Veeam B&R installed on the remote site, do you?

If you cannot do this, then I would suggest switching to active full backups instead. Thanks!

amatteo78
Influencer
Posts: 15
Liked: 1 time
Joined: Jun 07, 2014 9:32 am
Full Name: Matteo Abrile
Contact:

Re: Synthetic Full Backups Slow

Post by amatteo78 »

I have only 1 server with installed B&R and Proxy, and CIFS shred only in this proxy. So I think it's correct. but not solve problem.
Now I setup "active full backup" and check if it goes.

Vitaliy S.
Product Manager
Posts: 24341
Liked: 1888 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Synthetic Full Backups Slow

Post by Vitaliy S. »

Another way to address that is to mount this NAS box to any Windows server and then use it as a repository. Keep us updated on your testings.

amatteo78
Influencer
Posts: 15
Liked: 1 time
Joined: Jun 07, 2014 9:32 am
Full Name: Matteo Abrile
Contact:

Re: Synthetic Full Backups Slow

Post by amatteo78 »

sure, this night i test with "full", tomorrow I create new server windows as repository and other server as proxy. What cpu/ram need have this server, i mean proxy and repository ?
Thanks very much

M.

foggy
Veeam Software
Posts: 19540
Liked: 1781 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Synthetic Full Backups Slow

Post by foggy »

It is still not clear where this Veeam B&R server is located and what are the bottleneck stats for your backup jobs. Bottleneck stats are available by right-clicking the job and selecting the Statistic command on the shortcut menu.

System requirements for all Veeam B&R components are available in the release notes document.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 41 guests