Comprehensive data protection for all workloads
Bastiaan_van_Utrecht
Influencer
Posts: 10
Liked: never
Joined: Jun 28, 2011 12:03 pm
Full Name: Bastiaan_van_Utrecht
Contact:

Transforming Full backup (Backup Copy Job) is taking 3 days

Post by Bastiaan_van_Utrecht »

Dear all,

System size 1,9 Tb
Full backup size 1,8 Tb
Incremental backup size 25-50 Gb
Running Veeam version 7 with patch3

For this VM a backup copy job is running, the transforming to full backup is taking around 3 days to finish.
CPU load on backup copy system ( windows server 2008) is around 60%, Disk I/O on system, is quit low
between 0 / 10 MB/sec. System is able to serve much more disk I/O.

Does someone have an advise on this?

Regards,

Bastiaan
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Vitaliy S. »

Bastiaan,

Did you add this server as a Windows repository or a CIFS share to the backup console?

Thanks!
Bastiaan_van_Utrecht
Influencer
Posts: 10
Liked: never
Joined: Jun 28, 2011 12:03 pm
Full Name: Bastiaan_van_Utrecht
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Bastiaan_van_Utrecht »

Hi Vitaly,

Repository is defined as a windows server..

Regards,

Bastiaan
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Gostev »

Bastiaan_van_Utrecht wrote:System is able to serve much more disk I/O.
Well, thing is, to perform well with random I/O the system must be able to serve IOPS, not just generic I/O.
How many spindles there are behind the repository volume, what is the RAID level, and what are the hard drives used?
Bastiaan_van_Utrecht
Influencer
Posts: 10
Liked: never
Joined: Jun 28, 2011 12:03 pm
Full Name: Bastiaan_van_Utrecht
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Bastiaan_van_Utrecht »

Hi Gostev,

System is using 10 spindles with Near Line SAS drives 7,2K Rpm using Raid 6.

Another backup copy job is performing better:

VMs size in total 1,4 TB
VM backup size = 350 Gb
VM Incremental size = 100 GB
Transforming to full backup 4-5hours

Is the backup size making the difference?
This issue is breaking our backup design..

It also seems that patch3 is making it even slower, transforming is running now for 43hours and still on 20%

Regards,

Bastiaan
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by foggy »

Bastiaan, what kind of link do you have to the target repository? Do you run backup copy job in direct or WAN accelerated mode? Have you considered to perform initial seed for this job?
Bastiaan_van_Utrecht
Influencer
Posts: 10
Liked: never
Joined: Jun 28, 2011 12:03 pm
Full Name: Bastiaan_van_Utrecht
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Bastiaan_van_Utrecht »

Hi Foggy,

Getting the initial file to the target repository is not the issue as the link is a 1Gb link. THe backup copy jobs is using the direct mode.
The main issue is the merge of the incremental backup into the full backup (transforming) to generate a new full backup file, this is
not using the link, I expect.

Regards,

Bastiaan
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by veremin »

The main issue is the merge of the incremental backup into the full backup (transforming) to generate a new full backup file, this is not using the link, I expect.
Yes, only target storage component is involved in this kind of activity.

According to the backup copy job statistics, what is identified as the main bottleneck? Target? The said RAID mode (RAID 6) has certain I/O penalty. From my perspective, RAID 10 should provide better performance.

Thanks.
Bastiaan_van_Utrecht
Influencer
Posts: 10
Liked: never
Joined: Jun 28, 2011 12:03 pm
Full Name: Bastiaan_van_Utrecht
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Bastiaan_van_Utrecht »

Hi Eremin,

Hmm, the job statistics is showing "The Network" as the bottleneck..
Raid 10 will provide faster performance, but this is not something what I can test right now.
I am wondering what kind of system you need to get this working in a reasonal time.. 60 disks?

Instead of transforming the incremental chunk into the full backup, it will me much faster from me
to extract a complete new full backup from the source over the line.. ( will take around 12 hours I expect.)

Is this something I can manage, or other work arounds to prevent the transforming?

Regards,

Bastiaan
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by foggy »

Network being the bottleneck means that the problem is the speed of connection between source and target data movers. Could you try to copy some files from source to target repository server to check the performance? Also, even though the bottleneck stats report the network as the weakest point, you could test any other device to act as a target repository (just to eliminate the target storage issues).
Bastiaan_van_Utrecht
Influencer
Posts: 10
Liked: never
Joined: Jun 28, 2011 12:03 pm
Full Name: Bastiaan_van_Utrecht
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Bastiaan_van_Utrecht »

Hi Foggy,

Copy from the source to the destinaton is going well, using the 1Gb link up to 800Mb/Sec..
At the moment I have no other sytem available to store multiple copies of the 1,8TB Veaam backup..


Regards,

Bastiaan
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Vitaliy S. »

Bastiaan_van_Utrecht wrote:VMs size in total 1,4 TB
VM backup size = 350 Gb
VM Incremental size = 100 GB
Transforming to full backup 4-5hours

Is the backup size making the difference?
This issue is breaking our backup design..
I might have missed it, but I don't see how many backup copy jobs you run at the same time which are targeted to the same repository? I would also suggest taking a look at your job log (you can ask our support team for assistance with this) to cofirm that target storage is the bottleneck.
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by nunciate »

What is the traget storage type? SAN, NAS, direct attached disks? Also how is the storage connected to your target server? Fiber, SAS, iSCSI?

I pushed backup copy jobs to a CIFs share on a NetApp and it was terrribly slow. I also, tried synthetic full transferms for normal backup jobs to a NetApp which was FC attached. Also, very slow speeds transforming.

Lastly, what type of server are you backing up with this job? I have a Symantec Enterprise Vault server that we backup and it has a pretty good amount of change. The transforms take a very long time.
Bastiaan_van_Utrecht
Influencer
Posts: 10
Liked: never
Joined: Jun 28, 2011 12:03 pm
Full Name: Bastiaan_van_Utrecht
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Bastiaan_van_Utrecht »

Hi Vitaly,

The repository was configured to run 2 jobs at the same time, I also tried to reduce it to 1.. it does not show
any visible improvements.

I already created a case # 00517291

Regards,

Bastiaan
Bastiaan_van_Utrecht
Influencer
Posts: 10
Liked: never
Joined: Jun 28, 2011 12:03 pm
Full Name: Bastiaan_van_Utrecht
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Bastiaan_van_Utrecht »

Hi Nunciate,

It a swindows server system with direct attached storage. The server itself is a windows 2008
file server, with small incremental ( normally 50Gb or less a day)

Regards,

Bastiaan
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Vitaliy S. »

Bastiaan_van_Utrecht wrote:The repository was configured to run 2 jobs at the same time, I also tried to reduce it to 1.. it does not show
any visible improvements.
What about the the job that took 4-5 hours? Did it also take the same time when you have reduced the number of concurrent tasks? Anyway, thanks for the case, let's see what our technical team finds out after comparing two job logs.
Bastiaan_van_Utrecht
Influencer
Posts: 10
Liked: never
Joined: Jun 28, 2011 12:03 pm
Full Name: Bastiaan_van_Utrecht
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Bastiaan_van_Utrecht »

New update

I see 2 different messages

"Creating synthetic full restore point" -> in this case the job runs for more then 3 days

"Starting full backup transformation completed successfully" -> such a job will take about 3-4 hours

For me it's unclear when it's starting the "synthetic full backup" and when it choosing the "Full backup transformation".

I am afraid :| I have to go back to the good old tape backup, I cannot guaranteed uptodate backups when jobs takes
more then 1 day.

Regards,

Bastiaan
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Vitaliy S. »

According to your support case ID two jobs had different configuration in terms of full backup creation (Transform type: create new full vs.Transform type: patch existing full), that is why you had this time difference. Seems like your target storage is indeed a bottleneck (random I/O is the key here), as job that is configured with GFS runs quicker compared to the job with regular retention.
Bastiaan_van_Utrecht
Influencer
Posts: 10
Liked: never
Joined: Jun 28, 2011 12:03 pm
Full Name: Bastiaan_van_Utrecht
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Bastiaan_van_Utrecht »

I think it is the other way around..

GFS runs slower as this will create a comple new full backup, the regular retention just merge the incremental into the last full.
Are you aware of any tool to measure the random I/O, and what are the requirements fom Veeam?
zuerom
Novice
Posts: 8
Liked: 2 times
Joined: Feb 28, 2014 11:14 am
Full Name: Roman Zürcher
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by zuerom »

Hello Together,

we have the same problem here in our company...

The Backup Copy Job run since 50h and 34h-Transforming full Backup at (87% done)...

Windows Server 2012 r2 with FC SAN Storage (HP)

we need a solution here too!
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Vitaliy S. »

Bastiaan,

Here is an existing topic that can you give you an insight on how to measure IOPs > Reverse Incremental read/write operation details.

Thanks!
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Vitaliy S. »

Hi Roman,

Are you using GFS or simple retention policy? What is the size of your job? How many concurrent jobs you run at the same? Also some information on the hardware configuration of your target storage would help.

Thanks!
mintyevertonian
Enthusiast
Posts: 27
Liked: never
Joined: Nov 12, 2013 9:55 am
Full Name: fintan quinn
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by mintyevertonian »

I also have this issue

I am running one backup copy job that appears to complete ok and the trasforamtion can take 10-12 hours

The target repository is a Netgear NAS device which i suspect is the bottleneck but its all I have and suits the business needs.

This is a simple retention model with 7 restore points being kept...I cannot understand why trasformation takes so long
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Vitaliy S. »

Fintan,

Is your target NAS device located over the WAN link? How did you add it to the backup server? Your target storage should have enough IOPs capacity to support high performance of the transform process.

Thanks!
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Gostev »

mintyevertonian wrote:suits the business needs
Perhaps it suits your business needs only in terms of disk space capacity, but not in terms of IOPS capacity (otherwise, you would not have to post about transformation taking too long). But remember that you can always trade disk space for I/O performance by switching to a backup mode that does not use backup file transformation.
mintyevertonian
Enthusiast
Posts: 27
Liked: never
Joined: Nov 12, 2013 9:55 am
Full Name: fintan quinn
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by mintyevertonian »

it is a local backup on the same subnet ......transforming as i type zzzzz
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Vitaliy S. »

Try using any other local storage for testing purposes to see if you have a different performance for transform operation, but basically I would agree with Anton - storage should provide more IOPS if you want to make this process quicker.
TWBrowning
Influencer
Posts: 13
Liked: 1 time
Joined: Feb 22, 2012 10:35 am
Full Name: Tom Browning
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by TWBrowning »

Gostev wrote:But remember that you can always trade disk space for I/O performance by switching to a backup mode that does not use backup file transformation.
Is it possible to disable transformations for Backup Copy jobs?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by foggy »

No. Backup copy job is all about transformation.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Transforming Full backup (Backup Copy Job) is taking 3 d

Post by Vitaliy S. »

TWBrowning wrote:Is it possible to disable transformations for Backup Copy jobs?
Can you please clarify why do you need this? Do you experience any issues with backup copy jobs?
Post Reply

Who is online

Users browsing this forum: No registered users and 155 guests