Discussions specific to the VMware vSphere hypervisor
Post Reply
rubydragon
Novice
Posts: 6
Liked: never
Joined: Jan 15, 2020 1:16 am
Contact:

Super slow synthetic full backup

Post by rubydragon »

Hi,

New member here for a not so new question but I would like the input of others on how to diagnose a speed issue on synthetics.

Setup:
VMWare 5.5
3 VMs including one with 3.3TB
Proxy = VM with 4 cores, 6GB ram
Repository = Offsite NAS with 50/50 WAN speed and 12TB storage

Image

Any idea on the super slow backup job and the total lack of throughput?


Thanks

wishr
Veeam Software
Posts: 2539
Liked: 329 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Super slow synthetic full backup

Post by wishr »

Hi Rubydragon,

As I see the bottleneck is the network. Do you experience the same with incremental backups or only synth. full backups are affected? What is your network bandwidth?

Thanks

foggy
Veeam Software
Posts: 20326
Liked: 1919 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Super slow synthetic full backup

Post by foggy »

How NAS is added to Veeam B&R - what type of repository is used? Any chance it is CIFS without a gateway at the remote site?

rubydragon
Novice
Posts: 6
Liked: never
Joined: Jan 15, 2020 1:16 am
Contact:

Re: Super slow synthetic full backup

Post by rubydragon »

Hi,

As a late response, I do not have the same issue with incrementals. I did not try a full backup as this would send over 2TB of data and would take forever.

The NAS is added as a CIFS repository and without a proxy at the remote site.

After further review, I think the 50/50 offsite WAN connection is simply not enough for that much data to transfer. The proxy machine was at the maximum bandwidth of 50MB/sec the whole time. I find it weird that it was not showing in the throughput graph of VEEAM. Is it not reliable when watching live throughput?

wishr
Veeam Software
Posts: 2539
Liked: 329 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Super slow synthetic full backup

Post by wishr »

Hello Rubydragon,

As an alternative, you may point your backup job to a repository located at the main site, then deploy WAN accelerators at both locations and configure a Backup Copy job to copy the data from the main location to the offsite. WAN acceleration should help you to deal with low bandwidth. I'd recommend taking a loot at this BP article for more info on WAN acceleration.

Speaking about the graph within the backup job, please refer here for additional information.

Thanks

tdewin
Veeam Software
Posts: 1671
Liked: 588 times
Joined: Mar 02, 2012 1:40 pm
Full Name: Timothy Dewin
Contact:

Re: Super slow synthetic full backup

Post by tdewin »

For the remote connected CIFS, it is highly recommended to deploy a remote windows server as a gateway server. This way, the synthetic full, reading all the data/and writing it back to the same NAS but a different file, will be done at the remote site by the gateway server and not by the backup server/proxy at the source site

rubydragon
Novice
Posts: 6
Liked: never
Joined: Jan 15, 2020 1:16 am
Contact:

Re: Super slow synthetic full backup

Post by rubydragon »

After reading what a synthetic backup is, it is supposed to be taking alot of ressources from the backup destination. As far as network trafic over WAN, it should only be doing a regular incremental. From what I can see after 5 days of the running task, it is still using the WAN connection at maximum speed. A regular incremental backup is only taking like 2 hours. Why is is still transfering data after 5 days when most of the work should be done "localy" on the remote NAS?

veremin
Product Manager
Posts: 18692
Liked: 1869 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Super slow synthetic full backup

Post by veremin »

How remote NAS is connected to your backup server? Does it have gateway server (running at remote site) associated with it? Thanks!

rubydragon
Novice
Posts: 6
Liked: never
Joined: Jan 15, 2020 1:16 am
Contact:

Re: Super slow synthetic full backup

Post by rubydragon »

Hi,

Remote repository is connected as a CIFS storage using \\NAS_IP\Share

It does not have a remote gateway as it is stored in a co-location datacenter.

foggy
Veeam Software
Posts: 20326
Liked: 1919 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Super slow synthetic full backup

Post by foggy »

To make the work "locally" on the remote NAS, gateway server located close to it is required as the data mover cannot be installed directly on NAS. In your case, the data mover installed on the backup server itself is used that's why all the synthetic activity is being performed over the network. Thanks!

rbance67
Novice
Posts: 6
Liked: 3 times
Joined: Aug 16, 2016 10:55 pm
Full Name: Richard Bance
Contact:

Re: Super slow synthetic full backup

Post by rbance67 » 2 people like this post

I came across this recently. A site had a Veeam server connecting to a Synology NAS via 1Gb Ethernet. The synthetic full backup was taking 38hrs here. Having a deeper look at what was going on I noticed that the actual backup component was around 50 minutes, meaning there was 37hrs unaccounted for.
It became apparent that the creation of the synthetic full was taking 37hrs, due to fact the bottleneck was the exchange of data between the veeam server and the synology nas. Previously this site had a SAS tray connected via 6Gb, but when this failed a 1GB connected NAS was installed as a replacement. It was noted that the previous synthetic full happened in a lot less time...

I hope this helps...it is based on my observation of the process and having a chat to a local Veeam SE to confirm
I am actually in the process of fixing this, looking at a server with plenty of local disk (REFS format). The bus speeds for the storage sub-system should make this job finish a lot quicker....

Post Reply

Who is online

Users browsing this forum: No registered users and 26 guests