changing of backups design

[lravelo]

Hi everyone,

So for the longest time I've been creating single backup jobs per VM (currently have 200+ jobs) with varying restore points. I would like to change the way I'm doing these backups today to eliminate some of the overhead of trying to keep track of new VM's that need backups or VM's that have been decommissioned and no longer need backups...stuff of that nature. In addition, I need to include backups to tape in order to take the data offsite.

Here's what I'm thinking:

Create a single backup job for our production VMware datacenter cluster which is just over 33 TB in size, run it daily and keep, say, 10 days worth of restore points on disk. (This will definitely pose an issue in order to get everything re-seeded unless there's a way I can import my existing per-VM backups into this new backup job...anyone know if this is possible?). The point is to treat all data equally (which I know it isn't) and to make sure everything is "recoverable" to more or less the same time frame in the event of an outage.

From here I need the data to duplicate to tape and maintain a GFS rotation.

I don't think I'll have an issue using a Dell TL-4000 library with two LTO-6 drives in order to get everything backed up to tape. The idea would be to run my backups at night and then duplicate to tape during business hours. The tape library would be directly attached to the backup repo.

This is an entirely new concept for me at this point...am hoping someone could give me some advice/recommendations/criticism.

Thanks in advance!!

[Shestakov]

Hi Laz an welcome to the community!
Yes, if you choose VM container(host/tag/folder) as a source of the job, it will resolve the issue.
You are not able to import all backups to the new job, but once the new job has enough successful runs, you can delete the older backups.
Thanks!

[foggy]

Create a single backup job for our production VMware datacenter cluster which is just over 33 TB in size, run it daily and keep, say, 10 days worth of restore points on disk. (This will definitely pose an issue in order to get everything re-seeded unless there's a way I can import my existing per-VM backups into this new backup job...anyone know if this is possible?).

You can create a backup copy job that has the per-VM repository as a source and is pointing to a repository with per-VM setting disabled, this will result in a single VBK that can then be used to seed the new remote job.

[lravelo]

Hi Laz an welcome to the community!
Yes, if you choose VM container(host/tag/folder) as a source of the job, it will resolve the issue.
You are not able to import all backups to the new job, but once the new job has enough successful runs, you can delete the older backups.
Thanks!

Thanks!

This is reassuring. However, the part that confuses me is how does Veeam know from daily snapshots to create different retention once it starts going to tape?

[lravelo]

You can create a backup copy job that has the per-VM repository as a source and is pointing to a repository with per-VM setting disabled, this will result in a single VBK that can then be used to seed the new remote job.

This works for me in theory but the thought of having just one large .vbk worries me. Large files, in my experience, have greater potential for data corruption.

[foggy]

You can then do the same to switch back to per-vm chains.

[Shestakov]

This is reassuring. However, the part that confuses me is how does Veeam know from daily snapshots to create different retention once it starts going to tape?

You need to set different retention for each backup/backup copy job.
As for tapes, retention is set on media pools.

[lravelo]

So what you're saying is create something like this:

Backup Job:
Daily Backup - Run daily from Mon to Sat - Number of snapshots = 21
Weekly Backup - Run weekly on Sunday - Number of snapshots = 3
Monthly Backup - Run on last Sunday of month - Number of snapshots = 12
Yearly Backup - Run on last Sunday of the year - Number of snapshots =7

Then for tapes I'd just create a media pool for weekly, monthly, and yearly backups as well as a scratch pool.

I keep seeing "backup copy job" showing up in some of the replies here...how would that work if I only have one storage repo? Do I have to intentionally duplicate data for this to work?

[Shestakov]

Once you have 200+ jobs and want to combine them by the periodicity and restore points number you will have N primary jobs.
Then you can copy the backups on disk using backup copy jobs or to tape using backup to tape jobs. Both can work with or without GFS retention.

I keep seeing "backup copy job" showing up in some of the replies here...how would that work if I only have one storage repo? Do I have to intentionally duplicate data for this to work?

Technically you can create a folder and add it as another repository to VBR console.
Thanks!

[lravelo]

So in theory I could use my original idea of just creating one backup job for an entire VMware cluster with X number of restore points and just let the backup to tape job handle the retention regardless of how many restore points I have on disk?

[larry]

I do backup jobs by datastore. If a VM gets placed in a datastore it is backed up. You can use a vm tag if there is one VM you don’t want backed up. I have my datastores divided by RTO/RPO requirements, which fits with Veeam jobs by datastore. I have use this setup for years and am still happy with it.

[Shestakov]

So in theory I could use my original idea of just creating one backup job for an entire VMware cluster with X number of restore points and just let the backup to tape job handle the retention regardless of how many restore points I have on disk?

Yes, but this way all VMs will have equal amount of restore points on disk and various number of backups on tapes.

[lravelo]

Yes, but this way all VMs will have equal amount of restore points on disk and various number of backups on tapes.

Having an equal number of restore points is fine with me. It's actually the policy my employer uses globally. By "various number of backups on tape" do you mean that there will be an unequal number of restore points once it goes to tape?

[lravelo]

I do backup jobs by datastore. If a VM gets placed in a datastore it is backed up. You can use a vm tag if there is one VM you don’t want backed up. I have my datastores divided by RTO/RPO requirements, which fits with Veeam jobs by datastore. I have use this setup for years and am still happy with it.

You're using storage integration features for this though, right? I've never seen targeted backups of datastores without using that...I could be wrong.

[Shestakov]

I meant that you can vary number of backups on tapes if you create several backup-to-tape jobs.
You can create one job and have same amount of backups for all VMs.

[Shestakov]

You're using storage integration features for this though, right? I've never seen targeted backups of datastores without using that...I could be wrong.

Storage integration is not required for this, you just need to create jobs based on the datastores. Job A works with VMs from Datastore A, Job B works with VMs from Datastore B etc.

[lravelo]

My bad. I just realized that this could be done when adding objects. Forgive my brain fart.

[Shestakov]

No problem

[lravelo]

Ok so I've been able to successfully migrate to backups targeting the three datastores that I have and that's working great. The issue now is with the tape backups. I'm wondering if there's anything I can do to speed this up.

I have created three different GFS pools (one for each datastore) and have created one tape backups job per datastore each pointing to their respective GFS pool. The problem I'm having now is that two of the three jobs are idle saying that there are no GFS candidates available and the one job that is running has been running for about right hours now and it's only able 25% of the way through (granted, there's like 12.6 TB worth of data in there but both my repo are hooked up to a 10GbE network). Just wondering if there any work that can be done preemptively to minimize the amount of time this is taking? Right now I basically have another tape drive that's just sitting doing nothing. Would be nice to put it to work. That's kind of why I created independent backups jobs and pools since I know GFS tape jobs won't do parallel processing

[Shestakov]

Do you want to have 3 copies of same backup set on different media pools or you want to distribute backups among the pools?
Thanks!

[lravelo]

What I was really aiming for was just to finish the tape backups job as fast as possible. While reading the documentation, I noticed that GFS pools don't do parallel processing so I'll always have an idle tape drive which doesn't work for me so I created three GFS pools (one for each datastore that is backed up) in order to have at least two tape backups jobs running concurrently.

EDIT:

So I logged on this morning to check the status of the backups. The first one took 24 hours to back up a little over 4TB. The second one has been "idle" for almost 14 hours and has been actually running for over 7 hours and is only at 51%. The last job was stuck on idle for 24 hours and just started building the full backup map about 6.5 hours ago and still isn't done.

[lravelo]

OK so I think this is one of the culprits for the issue I'm having right now:

When Veeam Backup & Replication executes a GFS job, it performs the following operations:

At 00:00 of the day set in the GFS job schedule the GFS job starts and checks periodically if the source job created a backup:

- If the source job creates a full backup, the GFS job copies it.
- If the source job creates an incremental backup, the GFS job synthesizes a virtual full backup.
- If no backup appears in 24 hours, the GFS job looks for the most recent restore point available. If it is a full backup, the GFS job copies it. If it is an increment, the GFS job uses it to synthesize a virtual full backup.

Specifically the second item on that list. Seems like I either need to move to reverse incrementals or run a full backup prior to the the tape job. Any recommendations? The full backups size for each datastore is 3.3 TB, 5.26 TB, and 4.65 TB respectively.

[Shestakov]

What I was really aiming for was just to finish the tape backups job as fast as possible. While reading the documentation, I noticed that GFS pools don't do parallel processing so I'll always have an idle tape drive which doesn't work for me so I created three GFS pools (one for each datastore that is backed up) in order to have at least two tape backups jobs running concurrently.

Got it. That makes sense to split the objects between the tape jobs. Just make sure there are more or less equal amount of data written on each media pool.

Seems like I either need to move to reverse incrementals or run a full backup prior to the the tape job. Any recommendations?

Synthetic full is created if source job is on forever forward incremental mode. What backup mode do you use now?
Thanks!

[lravelo]

Right now I use forever forward incrementals. I have some wiggle room in the backups window though so I can change it if the tape backup duration will shorten

[Shestakov]

Good. So let's give it a try.

[lravelo]

Ok so after a couple of weeks of testing different methods, I've narrowed it down to having to run staggered active full backups prior to the GFS tape backup job. So far I can get all jobs to run within a 30 hour window during the weekends so that's doable for me. The reverse increments took too long as my storage repo couldn't handle the random IO as well as the sequential. So far it looks like this is the approach that I'll be taking.

[Shestakov]

Glad the goal os achieved. Thanks for updating!