Transforming Full backup (Backup Copy Job) is taking 3 days

[TWBrowning]

Can you please clarify why do you need this? Do you experience any issues with backup copy jobs?

This thread specifically concerns Backup Copy Job transformations taking too long. Further up Gostev said that 'you can always trade disk space for I/O performance by switching to a backup mode that does not use backup file transformation', but that is obviously not true of backup copy jobs.

If people choose to use Veeam Backup Copies they have to use transformation. There is no way to use Veeam to simply copy the files off site. There are Veeam file copies, but they have very limited scheduling options, and won't delete older copies of files.

We're having similar issues to the original poster. If it turns out to be an I/O problem then we're going to be left changing our backups at head office to forward incremental and using robocopy or something to mirror directories. Obviously I'd rather not do that.

Tom

[Vitaliy S.]

Further up Gostev said that 'you can always trade disk space for I/O performance by switching to a backup mode that does not use backup file transformation', but that is obviously not true of backup copy jobs. There is no way to use Veeam to simply copy the files off site. There are Veeam file copies, but they have very limited scheduling options, and won't delete older copies of files.

Correct, however Anton was referring to regular backup jobs with forward incremental backup mode and scheduled active fulls with their own retention, but thanks for the clarification.

[Reimold]

Hello,

we have the same problem with the backup copy job transformation taking up way too long. The last transformation did take over 80 hours. I understand that our repository (11 disks SATA 7.2k in a Raid5) is not ideal for that kind of IO pattern and that changing that to a raid10 could bring some improvement it would result in investing in disk space that I lose by changing the raid level.

I understand that the backup copy job as it is implemented right now depends on backup transformation but it would be great to have a copy job that behaves like a “reversed incremental” backup job. As the new feature “Backup copy job” and “WAN Accelerators” where announced I understand it that way, that it copies the data from a remote repository so that I have the same files in both repositories and not as it right now – reversed incremental files in the WAN repository and incremental files and transformation cycles in the central one.

Our goal is to have all VM´s from remote sites throughout the world backed up to a local repository and to our central repository at the headquarters. We have accomplished that in running a second backup job (reversed incremental) to the central repository. Since that second job is running much longer because of the WAN speed the VM´s are in snapshot state for quite some time and there is always the risk of a short connection outage in the WAN links it would be great if a job simply did not take the remote site VM as a source to backup to the central repository – instead it should take the files in the remote repository as a source and use the WAN accelerators to move the data over the WAN link. That would result in reduced VM snapshot time and less VMware operations like hotadd on the source ESX.

Since the transformation cycle consumes so much time it makes more sense for me to run the backup jobs as I did in the past since.

I hope you get my point and maybe there will be the opportunity to select a repository as a source for a reversed incremental job in a future release.

Dirk

[Gostev]

Actually, Backup Copy job transformation does 33% less I/O operations than Reversed Incremental transformation (2 versus 3 I/O operations per block). I have no idea why reversed incremental backup worked better for this repository...

[Reimold]

All I can say is that I have seen a copy job transforming for a very long time (1.3 TB .vbk file) and I have never seen such a long run within any other reversed incremental job. And since I am not the only one with that “long copy job transformation problem” as seen in the other posts I do not think that the problem is only located at my repository hardware.

We have upgraded to enterprise plus licenses just to find out that a copy job sometimes takes up to 80 hours transforming some files while a reversed incremental backup from the same VM runs between 2-8 hours.

I will do some more tests with different repository configurations in the next weeks, maybe I find out under what circumstances it takes that long to transform the backup files.

Dirk

[Gostev]

All I can say is that I have seen a copy job transforming for a very long time (1.3 TB .vbk file) and I have never seen such a long run within any other reversed incremental job.

Well, this may be because with reversed incremental backup job, the transform is done inline (as the job progresses), while Backup Copy job does it after the new incremental backup is created. So, you can never see separate "transform" operation there.

But, if you say that your reversed incremental backup job completes in 2-8 hours, then Backup Copy should certainly take noticeably less than that for the same VM in the scenario when backup storage is your primary bottleneck.

[Reimold]

I forgot to mention that the reversed incremental job only takes those 2-8 hours because it is running over a WAN connection with 10Mbit (showing network as the bottleneck). So the question is: why do I and the other people here see sometimes such long transformation times.

[Gostev]

It is quite normal for transformation to take long on slow storage, and especially low end NAS. What is not normal is when it takes more than reversed incremental backup over 10 Mbit WAN, this makes no sense and needs to be investigated closely. The sooner you include our support, the better - with the diagnostic tools they have, they may be able to find the issue more quickly.

[tsightler]

I'd like to ask specifically what transformation you are referring to as Backup Copy really has two different transform options based on whether you are using GFS or not.

If you are not using GFS then transformations happen at the end of the job when you've hit the retention limit. Typically these should never be longer than a reverse incremental as they actually perform less I/O.

On the other hand, if you have GFS enabled, then there will be a very long transformation when a new full is built, this is tremendously more I/O than what a normal reverse incremental would create since it requires moving enough blocks to create an entire full backup.

[Reimold]

If i remeber it correctly this is what happens: The Copy Job fails because the source WAN Accelerator was not reachable (connection problems on the WAN link) and in the next cycle it starts run a long transformation. I see in the repository folder of that job, that it creates a new .vbk file resulting writing 1.3 TB and that is what took the job more than 80 hours.

We are not using GFS and I have set the job to keep 7 restore points.

[foggy]

Dirk, please check the job advanced settings, do you have compact full scheduled for that time by any chance?

[Gostev]

Yep, new VBK creation in Backup Copy job with GFS disabled has nothing to deal with transform in the first place. Something else is going on here!

[b.vanhaastrecht]

I agree with Bastiaan_van_Utrecht (and other users who hijacked this post); It would be great if a Copy Job would have an option to choose how the fulls are generated. Like him we have a 1Gbit connection between sites, and relatively slow storage for the offsite copy (ours is dedupped). If the Copy Job had an option to (periodic) full copy, it would be much faster than a transform. (Or somekind of offsite staging option, where fast storage is used to generate the transforms on, and when finished copy's it to slow/dedupped storage.)

[mazur1000]

I use for backups ibm 3700 (12 discs 3TB raid6). I have several storages, backups are done alternately on each. Full backup takes 8TB and done 20 hours, the increment takes about 600-800 GB and is 4-5 hours, and transforming each increment takes 35-40 hours. Total stored 7 increments at each store. As backup server system is used 24 cores, 96 GB of memory. All storages connected Fc SAN. In the analysis of bottlenecks in the transformation of an analysis of the load on storages IOPS. Surprisingly it was at the level of 10-20% of capacity ibm 3700 for this configuration in random. A more confidence were launched several parallel backups/transformations on the same storage and copying the files from several other TB storage at speeds over 100 MB/s. As a result of the transformation backup virtually unchanged. There are suspicions that while transforming depends not only on the amount storage IOPS. I beg your comment.

[Gostev]

Transformation depends on storage IOPS capability, and that alone. Rule of thumb is that transformation performance is about 2MB/s per higher end [spinning]. So in your case, you should be getting about 20 MB/s or so transform performance, depending on hard drives used.

Sequential I/O performance from copying files does not matter at all. I get close to 100 MB/s when copying files to my old notebook with the single 5400 rpm hard drive

[Eppich]

I've had the same problem for a couple weeks now and I got no help from support (case 00609843) other than them telling me I need to use faster storage. Isn't the point of offsite backup storage to be high capacity instead of high performance? I'm using a netapp with 10 3TB 7.2k drives! I wasn't told by a veeam engineer when designing my backup infrastructure that I needed to buy 15k SAS drives for my backup copy repository.

[nunciate]

I still use a NetApp FAS-2240-4 as my back end storage in DR for my Veeam physical server. It has 24 3Tb drives. It is slow but it works. I have better results from 2Tb 7200rpm drives in a physical server but the NetApp does work. The best way I have found to use the NetApp is to carve up multiple volumes and LUNs and present them to the physical server over fiber channel. You could do iSCSI as well but it will be slower. Transforms and such take a very long time and you have to be very careful about how many you run at once.

My recommendation is to run direct attached storage like an HP server with 7200 rpm drives. If you did 10k or 15k I would imagine that work work way better but be much more expensive since you have limited drive capacity in those speeds.

[Vitaliy S.]

Hi Joshua,

What is the size of your backups you're trying to transform? How did you present your NetApp box to the Veeam server?

Thanks!

[mazur1000]

You say that to improve performance transforming required storage system with a lot of iops. How to calculate the required storage performance for the passage of transformation during the day, if a full backup takes 10TB, and incremental 10% of the full backup. At the moment I do not understand how this can be done, according to the monitoring system bottlenecks are neither disk subsystem or processor or memory. Testing the dependence of the transformation of the load on the disk subsystem was done by creating an additional burden both on iops and in MB / s on the disk subsystem. The transformation was almost the same with the additional load without it. Just monitoring can be seen that during the transformation of a lot of time there is generally no load on the disk subsystem or it is minimal 100-200 iops. What's going on at the moment I do not know how to define.

[Gostev]

I've had the same problem for a couple weeks now and I got no help from support (case 00609843) other than them telling me I need to use faster storage. Isn't the point of offsite backup storage to be high capacity instead of high performance? I'm using a netapp with 10 3TB 7.2k drives! I wasn't told by a veeam engineer when designing my backup infrastructure that I needed to buy 15k SAS drives for my backup copy repository.

I still use a NetApp FAS-2240-4 as my back end storage in DR for my Veeam physical server. It has 24 3Tb drives. It is slow but it works. I have better results from 2Tb 7200rpm drives in a physical server but the NetApp does work.

Hello, please try these recommendations from NetApp support from a similar support case on bad transform performance:

Code: Select all

options cifs.max_mpx 1124
options cifs.tcp_window_size 2096560
options cifs.neg_buf_size 65535
options raid.mirror_read_plex_pref alternate

[andreash]

Hi,
I'm afraid I need to add an "me too" reply here. Transforming a 2.6TB Backup Copy Job takes several days, but I can not see any obvious bottleneck.

Some figures:
VM Size: 2.6TB
Full Backup Size: ~1.6TB
Data Processed (Inc. Size): ~560GB (at 24MB/s, multiple Jobs were running at that time)
Bottleneck detected by VEEAM: Source (as expected, only 4 disks, and several Jobs at once. I am ok with the 24MB/s though)

Source Repository: Synology rs10613xs+ (Xeon E3, 3.3GHz, 8GB Ram), 4x4TB 7.2k in Raid 5, 500GB SSD r/w Cache
Target Repository: same Synology, 7x4TB 7.2k in Raid 5
Network: 3x1 GBit/s iSCSI with MPIO

I can't really find any bottleneck. The transform is dead slow, but all components seem to be idle:
Backup Server CPU: 25%
Backup Server Ram: 20%
Network: ~2-5MBit/s
Synology CPU: 1%
Synology RAM: 32% (because of the SSD Cache)
Synology Disks: 2% Utilisation, 20 IOPS/disk, ~200KB/s read/write per disk

Any idea how to find the bottleneck here?

Regards,
Andreas

[luckyinfil]

The bottleneck is always going to be your target storage when doing transforms. The source data is copied over to the target, AND THEN the transforms are done on the target. This is why Backup copy jobs won't work well with any dedupe appliances unless it has significant IOPS (which most don't since they're not designed for performance!)

[andreash]

@luckyinfil: I understand that this process is IOPS-heavy, but 2MByte/s still seem to slow to me for 7x4TB 7.2k in Raid 5, don't you think?
My job is now running since 38 hours and is at 86%. 24 hours ago it was at 82%.

[dellock6]

On some conditions and based on cache performances of a given storage don't be so sure it's an unexpected result.
On a 12 disks SATA array I'm able to run transforms of 512k blocks (the default size if you didn't changed it in Veeam) at 32 iops obtaining 17108 KBs. Given the fact you use half the I/O to read data and the other half to inject it, the real speed is around 8MBs. And again, I have 12 disks in RAID6, you have less.

I'm not saying this cannot be an issue, but don't be so sure a small array can sustain this kind of activity with ease.

[Gostev]

Default is 1024KB.

[dellock6]

Which on the target storage becomes 512kb thanks to compression
Sorry I didn't mention this, thanks Anton.

[tsightler]

@luckyinfil: I understand that this process is IOPS-heavy, but 2MByte/s still seem to slow to me for 7x4TB 7.2k in Raid 5, don't you think?
My job is now running since 38 hours and is at 86%. 24 hours ago it was at 82%.

Another huge factor is the underlying RAID stripe size. I believe that the Synology may default to 64K, which is pretty low for our workload. I've never found a better resource online for calculating RAID IOPS than http://www.wmarow.com/strcalc/. They don't have a 4TB drive listed, and you didn't say which specific one you had, but I selected the 3TB 7.2K SATA disk they had in the list which is probably similar with seek times and built a 7 drive RAID 5 with a 64K stripe size and used 512K I/O and and a 50/50 read/write mix this was the result:

average random IOPS: 23.63
bandwidth (MiB/s): 11.81

So if you're transforming 1.6TB of data that's 3.2TB total data to move around which is about 85 hours!! If the RAID stripe is 256K then you get results more like:

average random IOPS: 70.89
bandwidth (MiB/s): 35.44

That's an 3x increase in performance just from having an optimal RAID stripe size so you're down to 28 hours. Having the right stripe size is HUGE factor in transformation performance. Of course caching will help some as well, and it's not quite a 100% random I/O pattern as we try to batch some reads and writes, so it probably won't quite ever be as bad as this "worst case" calculation, but the point is, you really need to have an optimal setup for transforms to perform adequately.

[dellock6]

Tom is spot on as usual!
Just an addition, also check the RAID level, if you keep every parameter nd simply try to change the raid level for example from Raid5 to Raid10, you will see an additional jump in performances. Obviously, with a trade-off in available space.

[andreash]

@Tom, luca: Thanks for your valuable input. The http://www.wmarow.com/strcalc/ page is a great tool, too.

Unfortunately the Synology Units don't allow to change the Chunk/Stripe Size, and there is no option to set it when creating a raid array...
Maybe I can work around this on command line.
Looking at http://www.wmarow.com/strcalc/ the Performance should increase as I increase the stripe size. Does it make sense to have a 1024KB stripe size for the Backup Copy target?
I assume the reshape will take several weeks, so I'd rather get it right the first time

Raid 10 unfortunately is not an option for the Backup Copy target for capacity reasons.

Thanks for all your help!

Andreas

[dellock6]

No problem.
I was curious and did some research, the only information I've found is here:
http://forum.synology.com/enu/viewtopic.php?t=8926
but even if the answer came from a Synology employee, it's pretty old (2008). But 64k sounds pretty reasonable since it's a really common value. I'm not an expert on Synology, so your best choice is to engage their support. Especially to verify the reshape can be done online, if it's mdadm as I suspect (that by the way uses 64k as default... ) it can change the chunk size indeed, but again better involve their support.

I wouldn't go over 512k unless you choose to disable compression (that reduces default 1024k to around half size) or go for the large blocks (called 16TB+ in our interface that is 8MB per block).