Comprehensive data protection for all workloads
Post Reply
lando_uk
Veteran
Posts: 385
Liked: 39 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Transformation of copy jobs, concerns

Post by lando_uk »

Picture the enviroment.
Primary veeam server that backs up an exchange server installation of 18 servers.
It runs 5 days a week, has 20 restore points so this server hold 1 month of backups. The full backup is about 2.5TB.
We run a Backup copy job that goes to secondary server for archive, so that runs once a week and has 26 restore points, so that we can store 6 months of weekends on a different server(repository)

When we hit the 26 (6 month) retention, on that day it has to transform, it has to read all 26 vib files that are roughly 400-600GB into the full backup (vbk).

We have yet to hit the 26, but I worry when we do then the transform will never work. It would have to be on an 20TB SSD array to achieve the kind of IO requirements that are needed to process this job.

We are having simular problems already with standard jobs that are smaller with less restore points than these exchange vibs. ( case 00533881 ) - Our offsite/archive repository 12 x 4TB RAID6 with Server 2012 Dedupe doesnt have the IO's and we are getting delayed write failures. The trouble is we can't schedule copy job transforms, they only happen when the retention policy is exceeded, and because all our copy jobs have the same amount of restore points, they are all tranforming at the same time and there's no enough hours in the day limit the jobs so that only 1 transforms at the same time.

Server 2012 dedupe was working great, until we hit those retention limits and it had to transform, so it adds even more IO due to the hydration of the old vibs.

I feel the copy job implimentation is flawed - we were better of with complicated robocopy scripts :(
veremin
Product Manager
Posts: 20406
Liked: 2299 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Transformation of copy jobs, concerns

Post by veremin »

When we hit the 26 (6 month) retention, on that day it has to transform, it has to read all 26 vib files that are roughly 400-600GB into the full backup (vbk).
The backup copy job doesn't have to read all existing restore points. Once it exceeds specified retention, the backup files starts moving on daily basis. So, the only affected files are the oldest .vbk and .vib files.
The trouble is we can't schedule copy job transforms, they only happen when the retention policy is exceeded, and because all our copy jobs have the same amount of restore points, they are all tranforming at the same time and there's no enough hours in the day limit the jobs so that only 1 transforms at the same time.
Since transformation is considered as a task, you can set 1 as a number of concurrent tasks the backup repository can handle, so that, only one backup copy job is transferring or transforming at the given moment.

Thanks.
lando_uk
Veteran
Posts: 385
Liked: 39 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Transformation of copy jobs, concerns

Post by lando_uk »

v.Eremin wrote: The backup copy job doesn't have to read all existing restore points. Once it exceed specified retention, the backup files starts moving on daily basis. So, the only affected files are the oldest .vbk and .vib files.
So only the oldest files get touched during a transform?
eg

Exchange 2010 Backup - Archive2013-12-20T120000.vbk
Exchange 2010 Backup - Archive2013-12-27T120000.vib

Exchange 2010 Backup - Archive2014-01-03T120000.vib
Exchange 2010 Backup - Archive2014-01-10T120000.vib
Exchange 2010 Backup - Archive2014-01-17T120000.vib
Exchange 2010 Backup - Archive2014-01-24T120000.vib
Exchange 2010 Backup - Archive2014-01-31T120000.vib
Exchange 2010 Backup - Archive2014-02-12T120000.vib
Exchange 2010 Backup - Archive2014-02-19T120000.vib
Exchange 2010 Backup - Archive2014-02-26T120000.vib
Exchange 2010 Backup - Archive2014-03-05T120000.vib
Exchange 2010 Backup - Archive2014-03-12T120000.vib
Exchange 2010 Backup - Archive2014-03-19T120000.vib
Exchange 2010 Backup - Archive2014-03-26T120000.vib
Exchange 2010 Backup - Archive.vbm

So having more restore points makes no difference whatsoever to the amount of IO required to transform?
veremin
Product Manager
Posts: 20406
Liked: 2299 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Transformation of copy jobs, concerns

Post by veremin »

Correct.
lando_uk
Veteran
Posts: 385
Liked: 39 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Transformation of copy jobs, concerns

Post by lando_uk »

v.Eremin wrote:Since transformation is considered as a task, you can set 1 as a number of concurrent tasks the backup repository can handle, so that, only one backup copy job is transferring or transforming at the given moment.
If I have 6 copy jobs that are sheduled for the same time, out of interest, how does the repository decide which job runs first, as unlike backup jobs, the copy jobs cant be run off the back of each other.
veremin
Product Manager
Posts: 20406
Liked: 2299 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Transformation of copy jobs, concerns

Post by veremin »

On a first come, first serve basis.
lando_uk
Veteran
Posts: 385
Liked: 39 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Transformation of copy jobs, concerns

Post by lando_uk »

I have set the remote proxy to only run a max of 1 concurrent tasks, but its still processing multiple jobs and having to work very hard (500MB/sec+, disk queue length is hitting 5+). I dont think the current task number has any effect on copy jobs...
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Transformation of copy jobs, concerns

Post by Vitaliy S. »

Remote proxy servers are not used in backup copy jobs, you should set max concurrent tasks on the repository server instead.
lando_uk
Veteran
Posts: 385
Liked: 39 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Transformation of copy jobs, concerns

Post by lando_uk »

OK thanks, my mistake.

I've set it to 2 now.
lando_uk
Veteran
Posts: 385
Liked: 39 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Transformation of copy jobs, concerns

Post by lando_uk »

A question, now that I've limited the repository tasks, how does that effect Restores that may be required when large copy jobs are running? Would a restore task get priority above copy job tasks?
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Transformation of copy jobs, concerns

Post by foggy »

Yes, restore tasks have priority over other tasks in the sense that they pick up a released slot first, even if there are other backup tasks in the queue that.
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot], Majestic-12 [Bot] and 287 guests