Comprehensive data protection for all workloads
Post Reply
lxzndr
Novice
Posts: 8
Liked: 2 times
Joined: Jun 24, 2011 3:26 pm
Contact:

Archive Job option

Post by lxzndr » Jan 07, 2014 8:58 pm

Are there plans for have the option for an Archive job? I like the copy job feature as an alternative to manually copying the data to another location, but it would be nice to be able to archive the data separately without any incremental parts to it.

I would like do my daily backup jobs, copy job for offsite, and have an archive copy job that will give me weekly, monthly, etc copies on a deduplication device. there is no need for there to be a daily copy to that device, only the archive copy. I would also want the ability to be able to run an archive job on a the offsite copy so that we can offsite archive as well as the onsite archive, and have that archive job created from the copy job files, not built from the source backup files to avoid any WAN traffic on those. (particularly for those of us without enterprise license)

As others have said before, it makes sense to have archives go to dedupe device, but doesn't make sense to have a daily changing vbk file going there.

I have been running a script to copy the backup vbk files to archive device, but then I don't have any Veeam history on those copies.

Mike

foggy
Veeam Software
Posts: 18790
Liked: 1650 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Archive Job option

Post by foggy » Jan 07, 2014 9:41 pm

lxzndr wrote:I would like do my daily backup jobs, copy job for offsite, and have an archive copy job that will give me weekly, monthly, etc copies on a deduplication device. there is no need for there to be a daily copy to that device, only the archive copy.
To achieve that, you could set the "copy every" interval to one week and set minimum number of restore points to keep (2).
lxzndr wrote:I would also want the ability to be able to run an archive job on a the offsite copy so that we can offsite archive as well as the onsite archive, and have that archive job created from the copy job files, not built from the source backup files to avoid any WAN traffic on those. (particularly for those of us without enterprise license)
Currently, you can try to use a "dummy" backup job mapped to the files produced by the backup copy job and use this job as a source for the "archive" backup copy job.

lxzndr
Novice
Posts: 8
Liked: 2 times
Joined: Jun 24, 2011 3:26 pm
Contact:

Re: Archive Job option

Post by lxzndr » Jan 09, 2014 2:17 pm

foggy wrote: Currently, you can use a "dummy" backup job mapped to the files produced by the backup copy job and use this job as a source for the "archive" backup copy job.
I'm not sure I understand how to setup the "dummy" job for this.

foggy
Veeam Software
Posts: 18790
Liked: 1650 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Archive Job option

Post by foggy » Jan 09, 2014 2:29 pm

Create a new backup job pointing to the remote repository and map it to the files produced by the backup copy job (do not run this job). Then specify this backup job as a source for the backup copy job that will run in the remote site for archiving purposes.

lxzndr
Novice
Posts: 8
Liked: 2 times
Joined: Jun 24, 2011 3:26 pm
Contact:

[MERGED] copy job using other copy job as source

Post by lxzndr » Apr 02, 2014 1:30 pm

Can or will a copy job use the files of another copy job as its source?

I have a copy job that copies a set of servers to offsite repository daily and keeps them for 7 days.
I want to have another copy job that archives one of those servers once a week (keep 4 weeks, and 36 months of archives)

If I create second copy job for that server, will it pull from the offsite repository to build on the offsite archive repository? Or will it only use the local repository files to send across the wan?
the offsite and offsite archive are in the same location (actually two different drive sets on the same server)

Thank you,
Mike

foggy
Veeam Software
Posts: 18790
Liked: 1650 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Archive Job option

Post by foggy » Apr 02, 2014 1:39 pm

Mike, currently backup copy job cannot be used as a source for another backup copy job. A possible workaround could be to use a "dummy" backup job, map it to the backup copy job files, and then use this job as a source for the backup copy job.

lxzndr
Novice
Posts: 8
Liked: 2 times
Joined: Jun 24, 2011 3:26 pm
Contact:

Re: Archive Job option

Post by lxzndr » Apr 02, 2014 1:49 pm

When I go to create the dummy backup job, it says that the files are already in use by the copy job, do I want to map it to this job instead?
Will that mess up the copy job? Do I need to include all the servers in the original copy job in the dummy job? or just the ones I want to use in the archive job?
Due to other reasons long ago, we have two veeam servers, one in each of our locations. do I set up the dummy and archive on the Local site (to the originals) or on the Remote site (where the offsite copy is stored, and where the archive repository is) I'm sure it probably doesn't really matter, but is one a better location than the other? Having it local makes it easier to see all the backups for the server in one spot. having it remote may help with processing?

Mike

foggy
Veeam Software
Posts: 18790
Liked: 1650 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Archive Job option

Post by foggy » Apr 02, 2014 3:27 pm

You're right, it does not work this simple way. Moreover, I've just talked to R&D guys and it turns out that backup copy job cannot process backups created by another backup copy job (hardcoded limitation). We will discuss the ability to address this in the future, though.

Currently you have to run both backup copies over WAN or use some 3d party utility to copy files in the remote site to some other location.

lxzndr
Novice
Posts: 8
Liked: 2 times
Joined: Jun 24, 2011 3:26 pm
Contact:

Re: Archive Job option

Post by lxzndr » Apr 02, 2014 4:13 pm

It would be great if there was the option to store archival copies of the copy job in a different location than the copy. Or to be able to use the copy job as source for another copy job.
Or have an Archive job option, that doesn't do any incremental backups at all. just builds full backup copies from previous and changed blocks.

Also it can be confusing how the copy job archival processing works. Does it use the backup closest to the selected day for the archive? I notice it only tags them as archives when the file would normally be purged due to the retention limit.

Should be an option to specify the day to run the copy job, but not limit when it operate. How would I define the job if I wanted to copy only Friday (or closest to friday) backup? but I also don't want to limit the amount of time it gets to run (in case it takes more than allotted) and not get messed up if someone needs to edit the job (which starts a new cycle). Maybe allow the option to schedule the copy job the same as a regular backup job? have the continuous as an option, but not the only scheduling.

One goal would be to schedule it to start on Saturday 1am, and use the most recent backup to make the copy (be that Friday, or Tuesday, or even Saturday 12:30am as long as it is newer than the previous backup copy was) then not run again until next Saturday. Allow it run until completion (as long as it completes before the next start time) Keeping 4 weekly copies, and 36 Monthly copies. (or maybe 4W, 4M, 4Q, 3Y)

foggy
Veeam Software
Posts: 18790
Liked: 1650 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Archive Job option

Post by foggy » Apr 02, 2014 4:18 pm

lxzndr wrote:Also it can be confusing how the copy job archival processing works. Does it use the backup closest to the selected day for the archive? I notice it only tags them as archives when the file would normally be purged due to the retention limit.
Probably, the corresponding user guide section will make it clearer for you.

And thanks for the feedback!

lxzndr
Novice
Posts: 8
Liked: 2 times
Joined: Jun 24, 2011 3:26 pm
Contact:

Re: Archive Job option

Post by lxzndr » Apr 02, 2014 4:50 pm

That covers a daily job, but not what happens if it only runs once a week. If I say I want an archive copy 1st of every month, but my job only runs every 7 days, then what determines the Monthly archive? That is the part that I haven't seen covered in the guide.

What is the best way to accomplish the goal mentioned above?
Define the job to run every 7 days starting at 1am, with limit of when it can run from now until that start time, and then clear the restrictions after the first backup runs?

I will say that with the copy job, Veeam has improved for GFS archiving, but isn't quite there yet. 1 week of daily (or weekday) stored to fast storage as initial backup, and then weekly/monthly/etc copies to archival (deduplicating) storage using the copy (without incrementals in the archive or reading the previous backups there), plus same 1 week of daily copies offsite, and weekly/monthly/etc offsite archive is what we would love to get from it.

lando_uk
Expert
Posts: 315
Liked: 27 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Archive Job option

Post by lando_uk » Apr 02, 2014 10:28 pm

I would also like the option to create multiple full copies for archive rather than incrementals. I get paranoid that my full will become corrupt and I'll loose the whole chain.

I'd much rather have lots of fulls that are then deduped by the OS.

veremin
Product Manager
Posts: 17419
Liked: 1548 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Archive Job option

Post by veremin » Apr 03, 2014 8:09 am

That covers a daily job, but not what happens if it only runs once a week. If I say I want an archive copy 1st of every month, but my job only runs every 7 days, then what determines the Monthly archive? That is the part that I haven't seen covered in the guide.
With backup copy job running on weekly basis, you can simply propagate the logic described in the said section of User Guide; just replace days with weeks, so to say, Monday = 1st week, Tuesday = 2nd week. The idea is quite simple. The backup copy job has the retention settings. Once the specified retention policy is exceeded, the full backup starts moving ("transforming") forward during each run: be it daily run or the weekly one (as is in your case). When the full backup reaches the day chosen for GFS archive purpose, the new GFS point is created.
I would also like the option to create multiple full copies for archive rather than incrementals. I get paranoid that my full will become corrupt and I'll loose the whole chain.
The GFS restore points are stored as full backups, and backup copy job itself has built-in integrity check of the restore point with automated remediation of the detected corruption. If this isn't enough, you might switch to remote backup job and select to run full backups each time.

Thanks.

larry
Expert
Posts: 387
Liked: 92 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

[MERGED] daily data and retention data split

Post by larry » Jun 09, 2014 2:29 pm

Question on best way to do this task.
At our DR site we store a backupcopy from our main site, I push it there. This backup contains SQL data and I do not use the WAN accelerator for the job. Full backups once a week on main job. 150 meg wan link for backups only at night. Some other jobs do use the WAN accelerator but this job performed best without.
We want to move our retention data, weekly monthly and yearly to a separate storage location. While increasing our daily retention in the current location. For our DR site we are moving retention data to USB SATA 4tb disks. We will rotate the USB disks out as they fill.

I do not want to send the data over the WAN twice. I want the daily data and retention data split between servers and repositories at the DR site.
Should I change the main backup copy to a pull backup job and make a backup copy from that? I hate to hit production twice but could.
Make an ugly script to robo full backups? Then swap USD drive when full.

Any way to use a backupcopy as a source and use a local proxy to move the data?
Can the Veeam server at the DR site use the backup copies in the repositories the Veeam server at the main site made?
thanks
Is everyone’s data doubling every year? :shock:

foggy
Veeam Software
Posts: 18790
Liked: 1650 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Archive Job option

Post by foggy » Jun 09, 2014 2:44 pm

Larry, am I understanding right that you just want to store backup files created by regular retention and GFS retention of the backup copy job in different locations? Then I'm afraid your options are either using two backup copy jobs or utilizing some external "ugly script" or tool to copy files to different location.

larry
Expert
Posts: 387
Liked: 92 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

Re: Archive Job option

Post by larry » Jun 09, 2014 3:30 pm

ok thanks
My vote is for a Archive job to do this.
Options to include.
Don't check for existing files. This way removable media is automatic with archives. If I have my archive folder set to drive X folder X just do it. If folder not there, make it.
Store media name with Archive record, with media name during a restore if files are not there Veeam can display mount media name. Allow user to choses where to find files this way I can mount a two year old disk to a different drive letter while leaving current setup alone.
Will need a import Archive.

Maybe a flag for destruction of archive, I select to save for X years and after that time create a warning to mount old media so it can be deleted when job runs. I need to find all old backups and destroy after they are expired currently.

I like current retention options and can do all our necessity retentions.

foggy
Veeam Software
Posts: 18790
Liked: 1650 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Archive Job option

Post by foggy » Jun 10, 2014 1:57 pm

Thanks for the reasonable feedback; much appreciated.

DaFresh
Enthusiast
Posts: 64
Liked: 1 time
Joined: Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand
Contact:

Re: Archive Job option

Post by DaFresh » Jul 16, 2014 4:51 pm

Hello,

Sorry but this post is quiet confusing for me. As Mike, my archiving needs are simple, I need to produce a full backup (or a copy of a full backup...) of critical VMs every 6 months, with a retention policy like 1years. these VMs are backuped every day, so the best option would be the use of the Backup Copy mode, for example every first Sunday of January and July.

Could someone explain how schedule properly that simple use case with in the Backup Copy mode ?

Thanks,

Cédric

Gostev
SVP, Product Management
Posts: 25794
Liked: 3969 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Archive Job option

Post by Gostev » Jul 16, 2014 9:23 pm

Hi, Cédric

Why not just create a dedicated backup job with the required schedule and retention?

Seems like Backup Copy is unnecessary complication in this case. Also, it is always a good idea to create check point backups of critical VMs by obtaining the data directly from production storage (as this reduces the chance of introducing data corruption).

Thanks!
Anton

DaFresh
Enthusiast
Posts: 64
Liked: 1 time
Joined: Aug 30, 2011 9:31 pm
Full Name: Cedric Lemarchand
Contact:

Re: Archive Job option

Post by DaFresh » Jul 16, 2014 9:51 pm

Hi Anton,

Well, I was having that in my minds as a secondary option. I find more elegant to use the Backup Copy mode as it only require to *copy*, thus is more light for the underlaying hardware, faster, and safer than a full that could overlap the backup window.

I think it would have been maybe more flexible to allow, in addition to "Minute, Hour, Day", something like "Week, Month" ? If it was doable, that would have done the trick !

Cheers

veremin
Product Manager
Posts: 17419
Liked: 1548 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Archive Job option

Post by veremin » Jul 17, 2014 8:58 am

If you want to, you can still stick to backup copy job. As to weekly, monthly backup copy job schedule, you can achieve that by setting 7 day or 30 day copy interval, respectively. Thanks.

puters
Lurker
Posts: 2
Liked: never
Joined: Jun 01, 2016 1:34 am
Full Name: Ryan Grant
Contact:

Re: Archive Job option

Post by puters » Jun 01, 2016 2:27 am

I saw mention before of using an 'ugly script' or 3rd party tool to move the archive restore points out to deduped storage - what impact would this have on Veeam's retention policies?

At the moment I'm thinking of making a Powershell script to manage retention and move out archive restore points to a Windows Deduplication enabled volume. I have moved one of the yearly restore points and Veeam has replaced it with a ~30MB file of the same name. I'm just not sure if Veeam is going to run into errors if it tries to remove a backup that is no longer there.

Would the best option perhaps be to have the backup copy jobs set to keep just one of each restore point (yearly / monthly / weekly) and the script can copy them out for a longer retention elsewhere?

foggy
Veeam Software
Posts: 18790
Liked: 1650 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Archive Job option

Post by foggy » Jun 01, 2016 9:57 am

Retention shouldn't be confused by the fact that some of the GFS points are not physically there anymore. But why move? You can copy them for longer retention to a third location and let backup copy job retention to do its job, maintaining the required number of points in its target location.

puters
Lurker
Posts: 2
Liked: never
Joined: Jun 01, 2016 1:34 am
Full Name: Ryan Grant
Contact:

Re: Archive Job option

Post by puters » Jun 02, 2016 3:05 am

Thanks, I'll give it a go. The DDPEval tool predicted a 42% space saving on a given backup set, should save us a fair bit of space.

Post Reply

Who is online

Users browsing this forum: Brian2M, bytewiseits, Dima P., Google [Bot], rh0783 and 59 guests