Veeam backup and copy jobs reorganization to resolve out-of

[clemJ]

Hello,

I have two sites, one hosting the Veeam Management Server, the other hosting a Proxy. Both server have proxy and repository roles, and both are running one job to backup all production VMs of on-site ESXis.
Also, they are both running Copy jobs to move and retain all backups on local and distant site (GFS retention policy enabled).

Our goal is to kept a part of our production VMs for archive purpose , with maintaining protection for all of our production VMs.

I plan to re-organize our backup jobs and repositories, because the actual plan is consuming nearly all of our storage on repositories : all of our production VMs are kept with GFS retention policy.

So my plan is :

1. Create new repositories "Archives" (only new folders on same storage) on each sites, to split things for monitoring purpose.
2. Move all my backups and backups-copy files to those new repositories, then launch rescan on them.
3. Create new Backup jobs, with only part of our VMs which we want to archive, set the repository to our new "Archives" repos, then map moved backups to those jobs.
4. Create new Copy jobs for newly created backup jobs, with GFS retention enabled, and set 0 restore points to keep, retain backup locally and on the other site, set repository to our new "Archives" repos, then map moved backups-copy files to those jobs.
5. Remove production VMs to archive from originals Backup and Copy jobs (with repos rescan et backups re-maping).
6. Remove backups of production VMs (those not to be archived), from moved Copy jobs files that will retain data to be archived.

Does have I made mistakes, or forgot any step to clean original backups from data that we not want to archive ?

Regards,
Clément.

[Vitaliy S.]

Hi Clément,

Our goal is to kept a part of our production VMs for archive purpose , with maintaining protection for all of our production VMs.

If that's the goal, then why not to keep running all your existing backup jobs (to protect production VMs) and pick and choose just a few VMs in the backup copy wizard for which you would like to create archives for?

Thanks!

[clemJ]

Hi Vitaly,

Thank you for the reply.

You're right, it should be the simplest way to go.
But for future exploitation needs, I need to simplify tasks for our operators :

• One backup job where they put VMs to archive.
• One backup job whre they put VMs to protect.

No more configuration to do from them. I will do underlying configuration of Copy-Jobs.

I also need to simplify monitoring of repositories, so it is why I want to split things in different folders and check space consumed by each folder.

Regards,
Clément.

[Vitaliy S.]

Well... yes, even with the proposed variant above your backup operators will be able to choose VMs to backup and then which ones to archive.

4. Create new Copy jobs for newly created backup jobs, with GFS retention enabled, and set 0 restore points to keep, retain backup locally and on the other site, set repository to our new "Archives" repos, then map moved backups-copy files to those jobs.

Not sure I follow this, as you cannot set retention policy to 0 for the backup copy jobs.

I also need to simplify monitoring of repositories, so it is why I want to split things in different folders and check space consumed by each folder.

For that please consider taking a look at Veeam ONE, which allows you to monitor and report on backup components and its activity.

[clemJ]

Hi Vitally,

Thanks for the tips about Copy Job set to 0, I will set them to 1, not a big deal.
To be clear about my goal I will try to make a picture of what I do.

Actually, for each of our sites, I have this configuration (from local site point-of-view):

Code: Select all

            Local            Remote
VM   \
       -->   BKP     -->      CPY
VMa  /        |
             CPY

Where :

• VM : local production VMs to protect
• VMa : local production VMs to protect AND Archive
• BKP : Backup Job
• CPY : GFS Copy Job to retain backup copies for months
• Local : Local Veeam Proxy and Repository
• Remote : Remote Veeam Proxy and Repository

I have the same thing from the remote site, so I run globally 2 Backups Jobs and 4 Copy Jobs.

What I plan to do :

Code: Select all

            Local            Remote
VM     -->   BKP     -->      CPY
VMa    -->   BKPa    -->      CPYa
              |
             CPYa

Where :

• CPY : Copy Job to protect VMs remotely
• BKPa : Backup Job with only VM to archive
• CPYa : GFS Copy Job for Archive purpose (and protect VMa remotely with retention of last week data)

I think I need this configuration because I need to have local and remote protection for every VMs. Moreover, the only way to not use multiple Backup Jobs is to set CPYa jobs to retain VMs from Remote CPY job. As far as I understand Veeam B&R, it's not possible to use Copy job data as a source for another Copy Job. Maybe I'm wrong ?

For that please consider taking a look at Veeam ONE, which allows you to monitor and report on backup components and its activity.

Thanks, but I have already a great monitoring tool that can reply to this need with few scripting

Regards,
Clément

[Vitaliy S.]

Hi Clement,

Thanks for the tips about Copy Job set to 0, I will set them to 1, not a big deal.

Actually, the minimal retention policy it 2 > Backup copy for GFS only

I think I need this configuration because I need to have local and remote protection for every VMs. Moreover, the only way to not use multiple Backup Jobs is to set CPYa jobs to retain VMs from Remote CPY job. As far as I understand Veeam B&R, it's not possible to use Copy job data as a source for another Copy Job. Maybe I'm wrong ?

Yes, it seems like your setup is optimal. While you cannot select backup copy job as a source for another backup copy job, but it should be possible to select a repository (target for your 1st backup copy job) as a source for your second backup copy job to transfer restore points for archival purposes.

Thanks!

[clemJ]

Thanks to you Vitaly for your validation of this setup
I will try it next week.

Regards,
Clément.

[DaveWatkins]

You mentioned your staff put VM's into the appropriate job, that requirement can be removed by using VMWare tags and just tagging the VM when it's created (or Tag a parent folder and have the VM's moved to that folder)

[Vitaliy S.]

You're welcome. Please post back on your results. Thanks!

[clemJ]

You mentioned your staff put VM's into the appropriate job, that requirement can be removed by using VMWare tags and just tagging the VM when it's created (or Tag a parent folder and have the VM's moved to that folder)

Thank you a lot DaveWatkins, I will look in this direction, this can simplify our workflow a lot

@Vitaliy : I've started to move things :

1. Repos created.
2. Copy-Job's files moved to new Repo (Archive), and repo rescanned.
3. Made a clone of an existing Copy-Job, to keep configuration (schedules, windows,...), and mapped it to the new Repo.
4. Made a clone of the linked Backup Job, to keep configuration, and mapped it to the existing Backup files in old Repo. Set it with only VMs to-archive.
5. Cleaned linked Backup Job to keep only VMs not-to-archive.
6. VMs to-not-archive deleted from Copy-Job's files.

This last point is disappointing me because after the removal, I've checked the size of my files : it was the same as before VMs removal. I then check on this forum and Veeam KB about a kind of space-reclaim inside those files (Copy-Job's files) and find no good news.

It seems that I cannot reclaim this recently freed space. It's a pain, because I've done all of this to resolve a coming out-of-space problem on my repositories. And I want to keep my last 6 months of archived GFS backups for VMs to-archive...

Vitaliy, please tell me that there is an obscure command line tool at Veeam that can help me to reclaim this space, a kind of shrink tool that make copy of existing file without the zero-ed blocks ?

Regards,
Clément

[foggy]

The space isn't reclaimed immediately, you need to compact the full backup for that.

[clemJ]

Thanks for your quick reply foggy, but I'm talking about Copy Job's files, with GFS retention scheme, not classical Backup Job's files.

So I cannot use Compact parameters, they are grayed out because of the GFS retention scheme with a message that say :

Compact is not required when full backup retention is enabled

[Vitaliy S.]

Clément, you need to run a compact operation or trigger new active full backup for a backup copy job before enabling GFS retention policy. In this case, all unused data will be to ported to a GFS restore point.

[clemJ]

I've tried to disable GFS on this Copy-Job, then enable "Compact Full backup file" in Advanced Settings. After I had click on "OK" on Advanced Setting's window, Veeam is showing this message :

Delete existing full backups ?

I've clicked "No", then I've discarded my change on this job.

In my understanding, it is asking me if it could delete existing GFS full backups, right ?
So I understand that it will only try to compact "simple retention" full backup files for this job, my existing GFS full backups will not be compacted anyway ?

[Vitaliy S.]

Yes, you're correct in your understanding. If you want to use compact operation for the upcoming GFS restore point (full backup), you need to copy existing points and then manually delete them (according to the retention policy), run compact and switch back to GFS retention policy. However, please keep in mind that all new GFS restore points (created after job reconfiguration) will NOT have these "dirty blocks", only the upcoming (the very next) restore point will have them.

[clemJ]

OK thank you Vitaliy, now it's clear to me.
I'm not very happy with this but I will move those files to offline disks...

Is there a way to not fall in this situation in the futur ? For example, is it possible to have per-VM full backup files in a Copy-Job with GFS enable ?

[Vitaliy S.]

Is there a way to not fall in this situation in the future ?

Not sure I fully understand that. If you re-organize the backup job, you cannot simply shrink existing files with existing data, you always need to run a new full pass to get rid of "dirty" blocks.

Here is a similar topic, please take a look > Remove clients from a backup but VBK file never shrinks.

[clemJ]

Not sure I fully understand that.

My concern is about the actual need to keep full backup files created by GFS retention (terabytes of data) in which I cannot remove VMs nor tell Veeam B&R to reuse blocks of those VM marked as removed...

If you re-organize the backup job, you cannot simply shrink existing files with existing data, you always need to run a new full pass to get rid of "dirty" blocks.

Your link, and others that I've read on this forum and elsewhere on the web always talk about Backup Job's files and not about Copy Job with GFS retention files.

The new full pass you are talking about is doable only on Backup Jobs, not on existing Copy Jobs with GFS retention enabled.

[Vitaliy S.]

full backup files created by GFS retention (terabytes of data) in which I cannot remove VMs nor tell Veeam B&R to reuse blocks of those VM marked as removed...

Yes, it is not possible to remove VMs from existing backup files (already created). Try to follow the solution I've proposed above (copying files elsewhere), if you have enough storage on another media.

[clemJ]

Ok, thank you Vitaliy.
I hope someday Veam will publish a little tool to reclaim this lost space from GSF's full backup files

Regards,
Clément

[Vitaliy S.]

Ok, good luck with your project and thanks for the feedback!