-
- Veteran
- Posts: 385
- Liked: 39 times
- Joined: Oct 17, 2013 10:02 am
- Full Name: Mark
- Location: UK
- Contact:
Transformation of copy jobs, concerns
Picture the enviroment.
Primary veeam server that backs up an exchange server installation of 18 servers.
It runs 5 days a week, has 20 restore points so this server hold 1 month of backups. The full backup is about 2.5TB.
We run a Backup copy job that goes to secondary server for archive, so that runs once a week and has 26 restore points, so that we can store 6 months of weekends on a different server(repository)
When we hit the 26 (6 month) retention, on that day it has to transform, it has to read all 26 vib files that are roughly 400-600GB into the full backup (vbk).
We have yet to hit the 26, but I worry when we do then the transform will never work. It would have to be on an 20TB SSD array to achieve the kind of IO requirements that are needed to process this job.
We are having simular problems already with standard jobs that are smaller with less restore points than these exchange vibs. ( case 00533881 ) - Our offsite/archive repository 12 x 4TB RAID6 with Server 2012 Dedupe doesnt have the IO's and we are getting delayed write failures. The trouble is we can't schedule copy job transforms, they only happen when the retention policy is exceeded, and because all our copy jobs have the same amount of restore points, they are all tranforming at the same time and there's no enough hours in the day limit the jobs so that only 1 transforms at the same time.
Server 2012 dedupe was working great, until we hit those retention limits and it had to transform, so it adds even more IO due to the hydration of the old vibs.
I feel the copy job implimentation is flawed - we were better of with complicated robocopy scripts
Primary veeam server that backs up an exchange server installation of 18 servers.
It runs 5 days a week, has 20 restore points so this server hold 1 month of backups. The full backup is about 2.5TB.
We run a Backup copy job that goes to secondary server for archive, so that runs once a week and has 26 restore points, so that we can store 6 months of weekends on a different server(repository)
When we hit the 26 (6 month) retention, on that day it has to transform, it has to read all 26 vib files that are roughly 400-600GB into the full backup (vbk).
We have yet to hit the 26, but I worry when we do then the transform will never work. It would have to be on an 20TB SSD array to achieve the kind of IO requirements that are needed to process this job.
We are having simular problems already with standard jobs that are smaller with less restore points than these exchange vibs. ( case 00533881 ) - Our offsite/archive repository 12 x 4TB RAID6 with Server 2012 Dedupe doesnt have the IO's and we are getting delayed write failures. The trouble is we can't schedule copy job transforms, they only happen when the retention policy is exceeded, and because all our copy jobs have the same amount of restore points, they are all tranforming at the same time and there's no enough hours in the day limit the jobs so that only 1 transforms at the same time.
Server 2012 dedupe was working great, until we hit those retention limits and it had to transform, so it adds even more IO due to the hydration of the old vibs.
I feel the copy job implimentation is flawed - we were better of with complicated robocopy scripts
-
- Product Manager
- Posts: 20406
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Transformation of copy jobs, concerns
The backup copy job doesn't have to read all existing restore points. Once it exceeds specified retention, the backup files starts moving on daily basis. So, the only affected files are the oldest .vbk and .vib files.When we hit the 26 (6 month) retention, on that day it has to transform, it has to read all 26 vib files that are roughly 400-600GB into the full backup (vbk).
Since transformation is considered as a task, you can set 1 as a number of concurrent tasks the backup repository can handle, so that, only one backup copy job is transferring or transforming at the given moment.The trouble is we can't schedule copy job transforms, they only happen when the retention policy is exceeded, and because all our copy jobs have the same amount of restore points, they are all tranforming at the same time and there's no enough hours in the day limit the jobs so that only 1 transforms at the same time.
Thanks.
-
- Veteran
- Posts: 385
- Liked: 39 times
- Joined: Oct 17, 2013 10:02 am
- Full Name: Mark
- Location: UK
- Contact:
Re: Transformation of copy jobs, concerns
So only the oldest files get touched during a transform?v.Eremin wrote: The backup copy job doesn't have to read all existing restore points. Once it exceed specified retention, the backup files starts moving on daily basis. So, the only affected files are the oldest .vbk and .vib files.
eg
Exchange 2010 Backup - Archive2013-12-20T120000.vbk
Exchange 2010 Backup - Archive2013-12-27T120000.vib
Exchange 2010 Backup - Archive2014-01-03T120000.vib
Exchange 2010 Backup - Archive2014-01-10T120000.vib
Exchange 2010 Backup - Archive2014-01-17T120000.vib
Exchange 2010 Backup - Archive2014-01-24T120000.vib
Exchange 2010 Backup - Archive2014-01-31T120000.vib
Exchange 2010 Backup - Archive2014-02-12T120000.vib
Exchange 2010 Backup - Archive2014-02-19T120000.vib
Exchange 2010 Backup - Archive2014-02-26T120000.vib
Exchange 2010 Backup - Archive2014-03-05T120000.vib
Exchange 2010 Backup - Archive2014-03-12T120000.vib
Exchange 2010 Backup - Archive2014-03-19T120000.vib
Exchange 2010 Backup - Archive2014-03-26T120000.vib
Exchange 2010 Backup - Archive.vbm
So having more restore points makes no difference whatsoever to the amount of IO required to transform?
-
- Product Manager
- Posts: 20406
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
-
- Veteran
- Posts: 385
- Liked: 39 times
- Joined: Oct 17, 2013 10:02 am
- Full Name: Mark
- Location: UK
- Contact:
Re: Transformation of copy jobs, concerns
If I have 6 copy jobs that are sheduled for the same time, out of interest, how does the repository decide which job runs first, as unlike backup jobs, the copy jobs cant be run off the back of each other.v.Eremin wrote:Since transformation is considered as a task, you can set 1 as a number of concurrent tasks the backup repository can handle, so that, only one backup copy job is transferring or transforming at the given moment.
-
- Product Manager
- Posts: 20406
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Transformation of copy jobs, concerns
On a first come, first serve basis.
-
- Veteran
- Posts: 385
- Liked: 39 times
- Joined: Oct 17, 2013 10:02 am
- Full Name: Mark
- Location: UK
- Contact:
Re: Transformation of copy jobs, concerns
I have set the remote proxy to only run a max of 1 concurrent tasks, but its still processing multiple jobs and having to work very hard (500MB/sec+, disk queue length is hitting 5+). I dont think the current task number has any effect on copy jobs...
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Transformation of copy jobs, concerns
Remote proxy servers are not used in backup copy jobs, you should set max concurrent tasks on the repository server instead.
-
- Veteran
- Posts: 385
- Liked: 39 times
- Joined: Oct 17, 2013 10:02 am
- Full Name: Mark
- Location: UK
- Contact:
Re: Transformation of copy jobs, concerns
OK thanks, my mistake.
I've set it to 2 now.
I've set it to 2 now.
-
- Veteran
- Posts: 385
- Liked: 39 times
- Joined: Oct 17, 2013 10:02 am
- Full Name: Mark
- Location: UK
- Contact:
Re: Transformation of copy jobs, concerns
A question, now that I've limited the repository tasks, how does that effect Restores that may be required when large copy jobs are running? Would a restore task get priority above copy job tasks?
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Transformation of copy jobs, concerns
Yes, restore tasks have priority over other tasks in the sense that they pick up a released slot first, even if there are other backup tasks in the queue that.
Who is online
Users browsing this forum: Bing [Bot] and 236 guests