Synthetic Backup with QNAP NAS takes over 30 hours to complete

[AlvinTheLAW]

I have a customer who is utilising a QNAP NAS for their backup repository, the NAS repository was set up to be using SMB protocol rather than NFS.

We have setup 1 job to be backing up 1 VM which is 6.1TB in size and i can see in the progress it is only processing at around 18MB/s which is way too slow.

This job is utilising Forever incremental with Saturday as synthetic full day (14 days retention pol)

And from our calculation:
A: synthetic full creating duration: 36 hours 34 mins 20 secs
B: overall durantion: 36 hours 52 mins 8 secs

A/B = 99.19% (99.19% of the time was spent at creating Synthetic Full)

Is there any solution as to improve the speed of the synthetic full? Because over 60 hours for a weekly synthetic full is awful.

FYI，I already raised a ticket to support but he only suggest to change some Registry key settings of the VBR server. We changed it and there was no improvement
CASE ID: Case # 06164348

[Mildur]

Hello Alvin

Welcome to the forum.
With SMB/NFS repositories, our recommendation is to use Active Full backups instead of synthetic full backups.

A synthetic full backup session will read the amount of an entire full backup to the gateway server and write it back as a new full backup. How large is your full backup file?

Let's assume 5TB. It would be reading 5TB from the NFS share to the gateway server and then writing 5TB back to the NFS share. If there is a slow link between the gateway server and your QNAP NAS, you will experience slow synthetic full backups.
An Active full backup session would only write those 5TB once to the NFS Share. No read of 5TB involved.

This job is utilising Forever incremental with Saturday as synthetic full day (14 days retention pol)

With full backups each saturday, it won't be called "forever incremental". It's called "forward incremental".

Best,
Fabian

[AlvinTheLAW]

Hey Fabian,

We have'nt created nor utilise another gateway server for the NAS. So its just using the VBR VM server as a gateway by default.

Would you suggest that we try to make another gateway server for the NAS to use to see if it improves the performance? Or do we just use Active full backups only and untick the synthetic full in the job settings.

I have some questions for both solutions:

1. For creating a new gateway server, do we just deploy a new windows 2019 sever VM and give it access to the QNAP NAS, then just add it to VBR and assign it to the repository? Or is there way more steps and configuration for me to do to setup a gateway server.

2. If we abandon the synthetic full in the backup job and only use Active Full. We can disregard the gateway server config I assume.

[Mildur]

Hi Alvin

May I ask, what's the bandwidth between the backup server VM and the QNAP NAS?
And what's the reported bottleneck in the job session details?

Best,
Fabian

[AlvinTheLAW]

Hi Fabian,

Bandwidth i need to ask the customer to reconfirm for me.
As for what the bottleneck was it displays source.

[AlvinTheLAW]

Hey Fabian,

Also on another note about Synthetic Full, from the user guide: https://helpcenter.veeam.com/docs/backu ... ml?ver=120

The synthetic full backup has a number of advantages:

The synthetic full backup does not use network resources: it is created from backup files you already have on disk.
The synthetic full backup produces less load on the production environment: it is synthesized right on the backup repository.

So I don't quite understand why it would require gateway servers and any networking to perform the merge phase of the synthetic full.

I need to clarify, that the synthetic full is not long on the backing up of the incremental to the NAS phase. It's the merging of all the previous incremental .vib files to synthesise a Synthetic full back up this part is the one taking over 35 hours.

The customer has confirmed to me that they are running 1GbE bandwidth between the VBR VM and the NAS btw

[pirx]

With a NAS the synthetic backup clearly needs network resources. We had ~20 of those QNAP at smaller locations in use, as soon as there was more data and syn fulls got bigger the backup time was >24h. You could try to switch from SMB to iSCSI but this is a major change. We usw standard x64 server with Linux and XFS filesystem. Never had any issues again.

[AlvinTheLAW]

Well, i'm not too familiar with this. Since that in the official guide they said

The synthetic full backup has a number of advantages:

The synthetic full backup does not use network resources: it is created from backup files you already have on disk.
The synthetic full backup produces less load on the production environment: it is synthesized right on the backup repository.

So for me it wasn't "Clear" per se. Unless this statement from the official guide is doesnt apply to NAS deployments. Which it did not mention.

FYI, changing protocol isn't an option for the end-user.

But if it IS the case of it will need network resources EVEN DURING the .vib merge (Synthesis) stage, then i'll ask again：
1. For creating a new gateway server, do we just deploy a new windows 2019 sever VM and give it access to the QNAP NAS, then just add it to VBR and assign it to the repository? Or is there way more steps and configuration for me to do to setup a gateway server.

2. If we abandon the synthetic full in the backup job and only use Active Full. We can disregard the gateway server config I assume and this will improve on the backup speed?

[quangng]

Hi Alvin,

1. You can read this link > https://helpcenter.veeam.com/docs/backu ... er&ver=120
2. Yes, you can try this option to see whats happen.

[tom.schmidt]

My 2 cents:
SMB on QNAP is quite slow. The better option would be an iSCSI-LUN mapped directly to the backup server. Then use a local ReFS-repository.

[YouGotServered]

Just a note on what the user guide says regarding "not using network recourses". What it really means (and maybe could be updated to say) is that it doesn't use network resources transferring data from your production environment to your backup environment.

Picture if you had an "all in one" Veeam server that also housed your repository on a locally attached disk. Doing a synthetic full wouldn't touch networking at all because it would be manipulating backup files locally on disk.

In your environment, you are using an SMB share over the network for your repository. Here, there is no way to avoid using network resources between your backup server and the repository (but you still aren't using the network from your production environment to your backup server / environment). The problem with using plain SMB shares for your backup is that your backup server has to go over the network each time to read each piece of data, see what it is, then go back to tell the repository what to do with it. I believe the process is 3 I/O for every block written to a new backup file instead of x1 I/O, and now this is aaaaalllll happening over a slow 1GB link on what is likely not enterprise-grade hardware.

For best performance, you're going to want a Windows or Linux repository with locally attached disks or at least a repository that has some sort of native integration from Veeam. As a secondary option, you can do what others have suggested and attach your repository over iSCSI but you'd probably need to blow away your data and recreate it which isn't great. In my experience on lower end NAS devices, iSCSI has actually performed worse than SMB in testing. If you're going to use something like iSCSI to attach your disks, you'll need a solid iSCSI network and equipment to get good performance.

As a funny and polite side note - a QNAP NAS is far from recommended. Typically devices like that are more likely, ESPECIALLY over SMB, to suffer data corruption issues. When I was taking my Veeam Advanced certification course a few years ago, the running joke in class when we were presented with a big environment to build a backup for was "OK guys, so we'll go and get the BIGGEST QNAP we can find", and it always got laughs. I would truthfully not expect stellar performance, even if you enabled Active Full instead of synthetic fulls, although it should be better.

It might be wise to re-evaluate the backup strategy vs cost. For example, is it important to have weekly full backups? How critical are the company backups? Depending on those answers, it may be time to invest in new infrastructure. Even if you do switch to active full and get 2x the performance, you're going to have a full backup job running for potentially 15+ hours which is a big hit on production and VERY risky when you factor in that you will now have an open snapshot in production for your backup. A power outage with an open snapshot (especially when merging) is begging for production data corruption in my experience.

[ITP-Stan]

The Veeam user guides assumes you use local storage.
If you use NAS storage, you always have to factor in the network connection/bandwith to the NAS.

Merge/Synthetic operations on a NAS device with spinning disks are very slow with SMB & NFS protocol.
Even worse if it's slow disks (5400rpm) or a low amount of disks (2 disks in RAID1 for example).
It's because all the data is being read and written to the device at the same time on merge/synthetic.

When I use NAS devices as repositories, I use iSCSI and weekly Active Full for back-up.
For back-up copies to NAS, I use iSCSI and ReFS 64k file system to get fast merge/synthetic operations using fast block clone technology.

PS: But just so you know, using ReFS and a NAS via iSCSI is not supported by Microsoft. You shouldn't be surprised if the ReFS file system becomes corrupt when the NAS looses power or network suddenly.

[cbeugene]

I have a customer who is utilising a QNAP NAS for their backup repository, the NAS repository was set up to be using SMB protocol rather than NFS.

We have setup 1 job to be backing up 1 VM which is 6.1TB in size and i can see in the progress it is only processing at around 18MB/s which is way too slow.

This job is utilising Forever incremental with Saturday as synthetic full day (14 days retention pol)

And from our calculation:
A: synthetic full creating duration: 36 hours 34 mins 20 secs
B: overall durantion: 36 hours 52 mins 8 secs

A/B = 99.19% (99.19% of the time was spent at creating Synthetic Full)

Is there any solution as to improve the speed of the synthetic full? Because over 60 hours for a weekly synthetic full is awful.

FYI，I already raised a ticket to support but he only suggest to change some Registry key settings of the VBR server. We changed it and there was no improvement
CASE ID: Case # 06164348

Hello! Just in case: Can you please share the exact hard drives model from this problematic QNAP? If they are “SMR” drives then it is a drive problem.

Best redards,
Eugene

[AlvinTheLAW]

Just a note on what the user guide says regarding "not using network recourses". What it really means (and maybe could be updated to say) is that it doesn't use network resources transferring data from your production environment to your backup environment.

Picture if you had an "all in one" Veeam server that also housed your repository on a locally attached disk. Doing a synthetic full wouldn't touch networking at all because it would be manipulating backup files locally on disk.

In your environment, you are using an SMB share over the network for your repository. Here, there is no way to avoid using network resources between your backup server and the repository (but you still aren't using the network from your production environment to your backup server / environment). The problem with using plain SMB shares for your backup is that your backup server has to go over the network each time to read each piece of data, see what it is, then go back to tell the repository what to do with it. I believe the process is 3 I/O for every block written to a new backup file instead of x1 I/O, and now this is aaaaalllll happening over a slow 1GB link on what is likely not enterprise-grade hardware.

For best performance, you're going to want a Windows or Linux repository with locally attached disks or at least a repository that has some sort of native integration from Veeam. As a secondary option, you can do what others have suggested and attach your repository over iSCSI but you'd probably need to blow away your data and recreate it which isn't great. In my experience on lower end NAS devices, iSCSI has actually performed worse than SMB in testing. If you're going to use something like iSCSI to attach your disks, you'll need a solid iSCSI network and equipment to get good performance.

As a funny and polite side note - a QNAP NAS is far from recommended. Typically devices like that are more likely, ESPECIALLY over SMB, to suffer data corruption issues. When I was taking my Veeam Advanced certification course a few years ago, the running joke in class when we were presented with a big environment to build a backup for was "OK guys, so we'll go and get the BIGGEST QNAP we can find", and it always got laughs. I would truthfully not expect stellar performance, even if you enabled Active Full instead of synthetic fulls, although it should be better.

It might be wise to re-evaluate the backup strategy vs cost. For example, is it important to have weekly full backups? How critical are the company backups? Depending on those answers, it may be time to invest in new infrastructure. Even if you do switch to active full and get 2x the performance, you're going to have a full backup job running for potentially 15+ hours which is a big hit on production and VERY risky when you factor in that you will now have an open snapshot in production for your backup. A power outage with an open snapshot (especially when merging) is begging for production data corruption in my experience.

Hey Cory, Thanks for great explanation and insight. However, this infrastructure is that of the end-user and they wanted to use what they currently have to deploy a VBR solution. And Yes, it was deployed a month ago so theres already production VMs backed up to the repository so changing protocol for the NAS from SMB to ISCSI would not be a simple fix. FYI, I have a support case with this issue and me and the support engineer has also addressed to the end-user that if all the little optimisation and fixes we have tried doesn't improve, then it is ultimately a hardware limitation.

For context, their connection between the QNAP and VBR is indeed 1GbE.