Transforming Full backup (Backup Copy Job) is taking 3 days

[Bastiaan_van_Utrecht]

Dear all,

System size 1,9 Tb
Full backup size 1,8 Tb
Incremental backup size 25-50 Gb
Running Veeam version 7 with patch3

For this VM a backup copy job is running, the transforming to full backup is taking around 3 days to finish.
CPU load on backup copy system ( windows server 2008) is around 60%, Disk I/O on system, is quit low
between 0 / 10 MB/sec. System is able to serve much more disk I/O.

Does someone have an advise on this?

Regards,

Bastiaan

[Vitaliy S.]

Bastiaan,

Did you add this server as a Windows repository or a CIFS share to the backup console?

Thanks!

[Bastiaan_van_Utrecht]

Hi Vitaly,

Repository is defined as a windows server..

Regards,

Bastiaan

[Gostev]

System is able to serve much more disk I/O.

Well, thing is, to perform well with random I/O the system must be able to serve IOPS, not just generic I/O.
How many spindles there are behind the repository volume, what is the RAID level, and what are the hard drives used?

[Bastiaan_van_Utrecht]

Hi Gostev,

System is using 10 spindles with Near Line SAS drives 7,2K Rpm using Raid 6.

Another backup copy job is performing better:

VMs size in total 1,4 TB
VM backup size = 350 Gb
VM Incremental size = 100 GB
Transforming to full backup 4-5hours

Is the backup size making the difference?
This issue is breaking our backup design..

It also seems that patch3 is making it even slower, transforming is running now for 43hours and still on 20%

Regards,

Bastiaan

[foggy]

Bastiaan, what kind of link do you have to the target repository? Do you run backup copy job in direct or WAN accelerated mode? Have you considered to perform initial seed for this job?

[Bastiaan_van_Utrecht]

Hi Foggy,

Getting the initial file to the target repository is not the issue as the link is a 1Gb link. THe backup copy jobs is using the direct mode.
The main issue is the merge of the incremental backup into the full backup (transforming) to generate a new full backup file, this is
not using the link, I expect.

Regards,

Bastiaan

[veremin]

The main issue is the merge of the incremental backup into the full backup (transforming) to generate a new full backup file, this is not using the link, I expect.

Yes, only target storage component is involved in this kind of activity.

According to the backup copy job statistics, what is identified as the main bottleneck? Target? The said RAID mode (RAID 6) has certain I/O penalty. From my perspective, RAID 10 should provide better performance.

Thanks.

[Bastiaan_van_Utrecht]

Hi Eremin,

Hmm, the job statistics is showing "The Network" as the bottleneck..
Raid 10 will provide faster performance, but this is not something what I can test right now.
I am wondering what kind of system you need to get this working in a reasonal time.. 60 disks?

Instead of transforming the incremental chunk into the full backup, it will me much faster from me
to extract a complete new full backup from the source over the line.. ( will take around 12 hours I expect.)

Is this something I can manage, or other work arounds to prevent the transforming?

Regards,

Bastiaan

[foggy]

Network being the bottleneck means that the problem is the speed of connection between source and target data movers. Could you try to copy some files from source to target repository server to check the performance? Also, even though the bottleneck stats report the network as the weakest point, you could test any other device to act as a target repository (just to eliminate the target storage issues).

[Bastiaan_van_Utrecht]

Hi Foggy,

Copy from the source to the destinaton is going well, using the 1Gb link up to 800Mb/Sec..
At the moment I have no other sytem available to store multiple copies of the 1,8TB Veaam backup..


Regards,

Bastiaan

[Vitaliy S.]

VMs size in total 1,4 TB
VM backup size = 350 Gb
VM Incremental size = 100 GB
Transforming to full backup 4-5hours

Is the backup size making the difference?
This issue is breaking our backup design..

I might have missed it, but I don't see how many backup copy jobs you run at the same time which are targeted to the same repository? I would also suggest taking a look at your job log (you can ask our support team for assistance with this) to cofirm that target storage is the bottleneck.

[nunciate]

What is the traget storage type? SAN, NAS, direct attached disks? Also how is the storage connected to your target server? Fiber, SAS, iSCSI?

I pushed backup copy jobs to a CIFs share on a NetApp and it was terrribly slow. I also, tried synthetic full transferms for normal backup jobs to a NetApp which was FC attached. Also, very slow speeds transforming.

Lastly, what type of server are you backing up with this job? I have a Symantec Enterprise Vault server that we backup and it has a pretty good amount of change. The transforms take a very long time.

[Bastiaan_van_Utrecht]

Hi Vitaly,

The repository was configured to run 2 jobs at the same time, I also tried to reduce it to 1.. it does not show
any visible improvements.

I already created a case # 00517291

Regards,

Bastiaan

[Bastiaan_van_Utrecht]

Hi Nunciate,

It a swindows server system with direct attached storage. The server itself is a windows 2008
file server, with small incremental ( normally 50Gb or less a day)

Regards,

Bastiaan

[Vitaliy S.]

The repository was configured to run 2 jobs at the same time, I also tried to reduce it to 1.. it does not show
any visible improvements.

What about the the job that took 4-5 hours? Did it also take the same time when you have reduced the number of concurrent tasks? Anyway, thanks for the case, let's see what our technical team finds out after comparing two job logs.

[Bastiaan_van_Utrecht]

New update

I see 2 different messages

"Creating synthetic full restore point" -> in this case the job runs for more then 3 days

"Starting full backup transformation completed successfully" -> such a job will take about 3-4 hours

For me it's unclear when it's starting the "synthetic full backup" and when it choosing the "Full backup transformation".

I am afraid I have to go back to the good old tape backup, I cannot guaranteed uptodate backups when jobs takes
more then 1 day.

Regards,

Bastiaan

[Vitaliy S.]

According to your support case ID two jobs had different configuration in terms of full backup creation (Transform type: create new full vs.Transform type: patch existing full), that is why you had this time difference. Seems like your target storage is indeed a bottleneck (random I/O is the key here), as job that is configured with GFS runs quicker compared to the job with regular retention.

[Bastiaan_van_Utrecht]

I think it is the other way around..

GFS runs slower as this will create a comple new full backup, the regular retention just merge the incremental into the last full.
Are you aware of any tool to measure the random I/O, and what are the requirements fom Veeam?

[zuerom]

Hello Together,

we have the same problem here in our company...

The Backup Copy Job run since 50h and 34h-Transforming full Backup at (87% done)...

Windows Server 2012 r2 with FC SAN Storage (HP)

we need a solution here too!

[Vitaliy S.]

Bastiaan,

Here is an existing topic that can you give you an insight on how to measure IOPs > Reverse Incremental read/write operation details.

Thanks!

[Vitaliy S.]

Hi Roman,

Are you using GFS or simple retention policy? What is the size of your job? How many concurrent jobs you run at the same? Also some information on the hardware configuration of your target storage would help.

Thanks!

[mintyevertonian]

I also have this issue

I am running one backup copy job that appears to complete ok and the trasforamtion can take 10-12 hours

The target repository is a Netgear NAS device which i suspect is the bottleneck but its all I have and suits the business needs.

This is a simple retention model with 7 restore points being kept...I cannot understand why trasformation takes so long

[Vitaliy S.]

Fintan,

Is your target NAS device located over the WAN link? How did you add it to the backup server? Your target storage should have enough IOPs capacity to support high performance of the transform process.

Thanks!

[Gostev]

suits the business needs

Perhaps it suits your business needs only in terms of disk space capacity, but not in terms of IOPS capacity (otherwise, you would not have to post about transformation taking too long). But remember that you can always trade disk space for I/O performance by switching to a backup mode that does not use backup file transformation.

[mintyevertonian]

it is a local backup on the same subnet ......transforming as i type zzzzz

[Vitaliy S.]

Try using any other local storage for testing purposes to see if you have a different performance for transform operation, but basically I would agree with Anton - storage should provide more IOPS if you want to make this process quicker.

[TWBrowning]

But remember that you can always trade disk space for I/O performance by switching to a backup mode that does not use backup file transformation.

Is it possible to disable transformations for Backup Copy jobs?

[foggy]

No. Backup copy job is all about transformation.

[Vitaliy S.]

Is it possible to disable transformations for Backup Copy jobs?

Can you please clarify why do you need this? Do you experience any issues with backup copy jobs?