Comprehensive data protection for all workloads
Post Reply
clemJ
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2016 5:49 am
Full Name: Clément Juillac
Contact:

Veeam backup and copy jobs reorganization to resolve out-of

Post by clemJ »

Hello,

I have two sites, one hosting the Veeam Management Server, the other hosting a Proxy. Both server have proxy and repository roles, and both are running one job to backup all production VMs of on-site ESXis.
Also, they are both running Copy jobs to move and retain all backups on local and distant site (GFS retention policy enabled).

Our goal is to kept a part of our production VMs for archive purpose , with maintaining protection for all of our production VMs.

I plan to re-organize our backup jobs and repositories, because the actual plan is consuming nearly all of our storage on repositories : all of our production VMs are kept with GFS retention policy.

So my plan is :

1. Create new repositories "Archives" (only new folders on same storage) on each sites, to split things for monitoring purpose.
2. Move all my backups and backups-copy files to those new repositories, then launch rescan on them.
3. Create new Backup jobs, with only part of our VMs which we want to archive, set the repository to our new "Archives" repos, then map moved backups to those jobs.
4. Create new Copy jobs for newly created backup jobs, with GFS retention enabled, and set 0 restore points to keep, retain backup locally and on the other site, set repository to our new "Archives" repos, then map moved backups-copy files to those jobs.
5. Remove production VMs to archive from originals Backup and Copy jobs (with repos rescan et backups re-maping).
6. Remove backups of production VMs (those not to be archived), from moved Copy jobs files that will retain data to be archived.

Does have I made mistakes, or forgot any step to clean original backups from data that we not want to archive ?

Regards,
Clément.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by Vitaliy S. »

Hi Clément,
clemJ wrote:Our goal is to kept a part of our production VMs for archive purpose , with maintaining protection for all of our production VMs.
If that's the goal, then why not to keep running all your existing backup jobs (to protect production VMs) and pick and choose just a few VMs in the backup copy wizard for which you would like to create archives for?

Thanks!
clemJ
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2016 5:49 am
Full Name: Clément Juillac
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by clemJ »

Hi Vitaly,

Thank you for the reply.

You're right, it should be the simplest way to go.
But for future exploitation needs, I need to simplify tasks for our operators :
  • One backup job where they put VMs to archive.
  • One backup job whre they put VMs to protect.
No more configuration to do from them. I will do underlying configuration of Copy-Jobs.

I also need to simplify monitoring of repositories, so it is why I want to split things in different folders and check space consumed by each folder.

Regards,
Clément.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by Vitaliy S. »

Well... yes, even with the proposed variant above your backup operators will be able to choose VMs to backup and then which ones to archive.
clemJ wrote:4. Create new Copy jobs for newly created backup jobs, with GFS retention enabled, and set 0 restore points to keep, retain backup locally and on the other site, set repository to our new "Archives" repos, then map moved backups-copy files to those jobs.
Not sure I follow this, as you cannot set retention policy to 0 for the backup copy jobs.
clemJ wrote:I also need to simplify monitoring of repositories, so it is why I want to split things in different folders and check space consumed by each folder.
For that please consider taking a look at Veeam ONE, which allows you to monitor and report on backup components and its activity.
clemJ
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2016 5:49 am
Full Name: Clément Juillac
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by clemJ »

Hi Vitally,

Thanks for the tips about Copy Job set to 0, I will set them to 1, not a big deal.
To be clear about my goal I will try to make a picture of what I do.

Actually, for each of our sites, I have this configuration (from local site point-of-view):

Code: Select all

            Local            Remote
VM   \
       -->   BKP     -->      CPY
VMa  /        |
             CPY
Where :
  • VM : local production VMs to protect
  • VMa : local production VMs to protect AND Archive
  • BKP : Backup Job
  • CPY : GFS Copy Job to retain backup copies for months
  • Local : Local Veeam Proxy and Repository
  • Remote : Remote Veeam Proxy and Repository
I have the same thing from the remote site, so I run globally 2 Backups Jobs and 4 Copy Jobs.

What I plan to do :

Code: Select all

            Local            Remote
VM     -->   BKP     -->      CPY
VMa    -->   BKPa    -->      CPYa
              |
             CPYa
Where :
  • CPY : Copy Job to protect VMs remotely
  • BKPa : Backup Job with only VM to archive
  • CPYa : GFS Copy Job for Archive purpose (and protect VMa remotely with retention of last week data)
I think I need this configuration because I need to have local and remote protection for every VMs. Moreover, the only way to not use multiple Backup Jobs is to set CPYa jobs to retain VMs from Remote CPY job. As far as I understand Veeam B&R, it's not possible to use Copy job data as a source for another Copy Job. Maybe I'm wrong ?
Vitaliy S. wrote:For that please consider taking a look at Veeam ONE, which allows you to monitor and report on backup components and its activity.
Thanks, but I have already a great monitoring tool that can reply to this need with few scripting ;)

Regards,
Clément
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by Vitaliy S. »

Hi Clement,
clemJ wrote:Thanks for the tips about Copy Job set to 0, I will set them to 1, not a big deal.
Actually, the minimal retention policy it 2 > Backup copy for GFS only
clemJ wrote:I think I need this configuration because I need to have local and remote protection for every VMs. Moreover, the only way to not use multiple Backup Jobs is to set CPYa jobs to retain VMs from Remote CPY job. As far as I understand Veeam B&R, it's not possible to use Copy job data as a source for another Copy Job. Maybe I'm wrong ?
Yes, it seems like your setup is optimal. While you cannot select backup copy job as a source for another backup copy job, but it should be possible to select a repository (target for your 1st backup copy job) as a source for your second backup copy job to transfer restore points for archival purposes.

Thanks!
clemJ
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2016 5:49 am
Full Name: Clément Juillac
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by clemJ »

Thanks to you Vitaly for your validation of this setup :)
I will try it next week.

Regards,
Clément.
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by DaveWatkins » 1 person likes this post

You mentioned your staff put VM's into the appropriate job, that requirement can be removed by using VMWare tags and just tagging the VM when it's created (or Tag a parent folder and have the VM's moved to that folder)
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by Vitaliy S. »

You're welcome. Please post back on your results. Thanks!
clemJ
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2016 5:49 am
Full Name: Clément Juillac
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by clemJ »

DaveWatkins wrote:You mentioned your staff put VM's into the appropriate job, that requirement can be removed by using VMWare tags and just tagging the VM when it's created (or Tag a parent folder and have the VM's moved to that folder)
Thank you a lot DaveWatkins, I will look in this direction, this can simplify our workflow a lot :)

@Vitaliy : I've started to move things :
  1. Repos created.
  2. Copy-Job's files moved to new Repo (Archive), and repo rescanned.
  3. Made a clone of an existing Copy-Job, to keep configuration (schedules, windows,...), and mapped it to the new Repo.
  4. Made a clone of the linked Backup Job, to keep configuration, and mapped it to the existing Backup files in old Repo. Set it with only VMs to-archive.
  5. Cleaned linked Backup Job to keep only VMs not-to-archive.
  6. VMs to-not-archive deleted from Copy-Job's files.
This last point is disappointing me because after the removal, I've checked the size of my files : it was the same as before VMs removal. I then check on this forum and Veeam KB about a kind of space-reclaim inside those files (Copy-Job's files) and find no good news.

It seems that I cannot reclaim this recently freed space. It's a pain, because I've done all of this to resolve a coming out-of-space problem on my repositories. And I want to keep my last 6 months of archived GFS backups for VMs to-archive...

Vitaliy, please tell me that there is an obscure command line tool at Veeam that can help me to reclaim this space, a kind of shrink tool that make copy of existing file without the zero-ed blocks ?

Regards,
Clément
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by foggy »

The space isn't reclaimed immediately, you need to compact the full backup for that.
clemJ
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2016 5:49 am
Full Name: Clément Juillac
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by clemJ »

Thanks for your quick reply foggy, but I'm talking about Copy Job's files, with GFS retention scheme, not classical Backup Job's files.

So I cannot use Compact parameters, they are grayed out because of the GFS retention scheme with a message that say :
Compact is not required when full backup retention is enabled
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by Vitaliy S. »

Clément, you need to run a compact operation or trigger new active full backup for a backup copy job before enabling GFS retention policy. In this case, all unused data will be to ported to a GFS restore point.
clemJ
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2016 5:49 am
Full Name: Clément Juillac
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by clemJ »

I've tried to disable GFS on this Copy-Job, then enable "Compact Full backup file" in Advanced Settings. After I had click on "OK" on Advanced Setting's window, Veeam is showing this message :
Delete existing full backups ?
I've clicked "No", then I've discarded my change on this job.

In my understanding, it is asking me if it could delete existing GFS full backups, right ?
So I understand that it will only try to compact "simple retention" full backup files for this job, my existing GFS full backups will not be compacted anyway ?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by Vitaliy S. »

Yes, you're correct in your understanding. If you want to use compact operation for the upcoming GFS restore point (full backup), you need to copy existing points and then manually delete them (according to the retention policy), run compact and switch back to GFS retention policy. However, please keep in mind that all new GFS restore points (created after job reconfiguration) will NOT have these "dirty blocks", only the upcoming (the very next) restore point will have them.
clemJ
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2016 5:49 am
Full Name: Clément Juillac
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by clemJ »

OK thank you Vitaliy, now it's clear to me.
I'm not very happy with this but I will move those files to offline disks...

Is there a way to not fall in this situation in the futur ? For example, is it possible to have per-VM full backup files in a Copy-Job with GFS enable ?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by Vitaliy S. »

clemJ wrote:Is there a way to not fall in this situation in the future ?
Not sure I fully understand that. If you re-organize the backup job, you cannot simply shrink existing files with existing data, you always need to run a new full pass to get rid of "dirty" blocks.

Here is a similar topic, please take a look > Remove clients from a backup but VBK file never shrinks.
clemJ
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2016 5:49 am
Full Name: Clément Juillac
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by clemJ »

Vitaliy S. wrote:Not sure I fully understand that.
My concern is about the actual need to keep full backup files created by GFS retention (terabytes of data) in which I cannot remove VMs nor tell Veeam B&R to reuse blocks of those VM marked as removed...
Vitaliy S. wrote: If you re-organize the backup job, you cannot simply shrink existing files with existing data, you always need to run a new full pass to get rid of "dirty" blocks.
Your link, and others that I've read on this forum and elsewhere on the web always talk about Backup Job's files and not about Copy Job with GFS retention files.

The new full pass you are talking about is doable only on Backup Jobs, not on existing Copy Jobs with GFS retention enabled.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by Vitaliy S. »

clemJ wrote:full backup files created by GFS retention (terabytes of data) in which I cannot remove VMs nor tell Veeam B&R to reuse blocks of those VM marked as removed...
Yes, it is not possible to remove VMs from existing backup files (already created). Try to follow the solution I've proposed above (copying files elsewhere), if you have enough storage on another media.
clemJ
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2016 5:49 am
Full Name: Clément Juillac
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by clemJ »

Ok, thank you Vitaliy.
I hope someday Veam will publish a little tool to reclaim this lost space from GSF's full backup files ;)

Regards,
Clément
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam backup and copy jobs reorganization to resolve out

Post by Vitaliy S. »

Ok, good luck with your project and thanks for the feedback!
Post Reply

Who is online

Users browsing this forum: orb and 202 guests