Tape backup scenarios

[perjonsson1960]

Folks,

We are backing up our servers daily through backup jobs to a scale-out repository, and we are also doing backup copies daily of all the backups to another scale-out repository at a different physical location. Now we have installed a tape library as well, a HPE MSL6480 with room for 70 tapes, and we have two LTO-8 drives.

Since I am new to this, how should I go about doing backups to tape? Basically we want to copy the backup copy chains from disk to tape. Some of the backup copy jobs use GFS policies. If I want to copy the GFS chains to tape, do I need a GFS Media Pool in order to maintain a GFS retention policy on tape?

And what about copies that do not use GFS, that have for example only a standard seven day retention period? Can I put those in a Standard Media Pool?

Bottom line, do I need to divide the tape library into, for example, one GFS pool and one Standard pool in order to pull this off?

Sincerely,
Per Jonsson
Sweden

[Dima P.]

Hello Per,

To cut the long story short you have to options:

1. File to tape jobs. Can be used to copy files from file shares or full backup files.
2. Backup to tape jobs. Should be used to write backup files to tape (as it provides tight integration with source jobs)

If I want to copy the GFS chains to tape, do I need a GFS Media Pool in order to maintain a GFS retention policy on tape?

Retention in backup copy and retention in tape jobs are completely independent from each other. You can use simple media pool and backup all the backup copy jobs to tape with simple retention schema for your tape media within this media pool. Another option is to use GFS media pool and organize tape media retention based on GFS schema.

And what about copies that do not use GFS, that have for example only a standard seven day retention period? Can I put those in a Standard Media Pool?

Media pool is just a target (basically a box with assigned tape media and set retention), so you can use backup copy jobs with any media pool type.

Bottom line, do I need to divide the tape library into, for example, one GFS pool and one Standard pool in order to pull this off?

Only if you want to keep the data on tape media with different retention. For example, some customers are using simple media pool just to keep the secondary of their backups and GFS media pools for long term archiving. Cheers!

[perjonsson1960]

I have just created and started my first backup to tape job. We want to do a tape backup of all our servers once a month, for long term archiving, so I created a job that backs up the whole Scale-out repository where the backup copies on disk are stored, and chose to backup only full backup files. I estimate that this job will take around 72 hours.

Our backups to disk are run daily, seven days a week, and our backup copy jobs to disk run continuously 24 hours a day. The synthetic fulls are created during the weekends, and the health checks and defragmentation tasks are also done during the weekends. Should we avoid running the tape backup during the weekend, and instead maybe start the job on Monday so that it will be finished before the weekend?

I assume that the tape job locks the backup file that it is currently writing to tape, and that the backup copy job will have to wait until the tape job releases the file, before it can do something with it?

When the tape backup job is finished, and we want to take out the tapes that have been written, and put them in a safe, is it enough to do an "Export" of the tapes before we take them out of the library? I have also seen that there is a "Move to vault" option... But maybe the Export puts the tape in a vault?

Next month, when I start the tape backup again, should I use the same tape job, or is it better to create a totally new job, since the tapes that were used last are no longer present?

I know that these are rookie questions, but there it is...

[perjonsson1960]

The tape job has been running for 68 hours now, and I don't understand what is happening... It seems that when backup copy jobs have created new restore points, it seems that the tape job discovers them and copies them as well, as if the tape job is continuous... At first the tape job said that it was 283 objects to back up, and suddenly it changed to 390 objects... It seems that this tape job will run forever... What am I doing wrong? The log is filled with entries like "backup files have changed, retry is required"... And in one of the sources it says "Storage bla bla bla not found" for about 100 servers...

What should I have done? Do I have to disable all the backup copy jobs before the tape job starts? I cannot disable backup copies to disk for 72 hours or more... No way.

[perjonsson1960]

Now the job has been running for 74 hours. The job progress says "99%" and "268 of 390 objects", which doesn't quite add up in my head. I am THIS CLOSE from stopping the job now, erasing all the tapes and start over. But I will let it run until tomorrow morning (Monday) at 8 (ish) CEST.

Can somebody please tell me how to create a backup to tape job with our backup copy scale-out repository as the source, that will copy all of the latest full backup copies (no incrementals) that are accessible, and that will NOT be changed by any of the running backup copy jobs? Right now it seems impossible to me.

1. The backup jobs put the backups on our "Scale-out repository 1" once every 24 hours.
2. Our backup copy jobs, which run continuously 24 hours a day, copies the latest backup restore points whenever they appear, and put them on our "Scale-out repository 2".

Those things work like a charm.

3. The backup to tape job should take the latest full backup copies that it can access without any interventions, and put them on tape, in order to, once a month, create a long term archive on tape that we can put in a safe. How do we accomplish this? I have read the help pages about tape backups, and I have read some blog entries with "best practices", and my head is spinning, quite frankly...

[Dima P.]

Per,

Can somebody please tell me how to create a backup to tape job with our backup copy scale-out repository as the source, that will copy all of the latest full backup copies (no incrementals) that are accessible, and that will NOT be changed by any of the running backup copy jobs? Right now it seems impossible to me.

1. If that's a new backup to tape job, make sure that you've instucted backup job to process only latest backup chains during setup
2. Do not enable incremental backup on the incremental backup step of the wizard
3. Check the box Process latest full backup chain only under Advanced settings of your backup to tape job
4. Make sure the checkbox Prevent this job from being interrupted by source backup jobs remains unchecked

[perjonsson1960]

Thanks for your reply!

All these criteria have been met. That is exactly how the job is configured.
However, it seems that the tape job discoveres new restore points made by the backup copy jobs while the tape job is running, and copies those to tape, as well. This means that the tape job takes much longer, and uses much more tape than necessary. We have around 240 servers, most of them in VMware and about 15 physical, and I would estimate that half of them got one full backup on tape, and the other half two full backups. And for one source, even three were copied to tape...

PJ

[Dima P.]

Per,

tape job discoveres new restore points made by the backup copy jobs while the tape job is running

I wonder if these backup copy jobs produce new full backup or they are running in forever incremental backup mode (thus tape starts to create synthetic full from incremental restore point)? Thanks!

[perjonsson1960]

All of the backup copy jobs are set to keep seven restore points, which is the default, I think. Most of the jobs are set to keep one weekly backup, which means that there are two full backups present, the "current" one listed as "R", and one listed as "W", in the "Disk (Copy)" properties. And three jobs use more of a full GFS policy with four weekly backups, twelve monthly backups and two yearly backups.

The sources that got two or even three fulls on the tape are from both of those categories, and in some cases there are some servers, within the same backup copy source, that got one full on tape and others got two fulls on tape.

I am not saying that is is a bad thing to get many backups, but what we had in mind was to do a tape backup of the latest available full backup copy for each server once a month, to put in a safe for long term archiving. This tape job took 92 hours and used eight LTO-7 tapes (there are no LTO-8 tapes available for purchase), and if the job had backed up only the latest available full backup copy for each server, perhaps it would have taken half that time and half the number of tapes.

[Dima P.]

Understood, thank you! Can I please ask you to raise a support case and share the case ID with us? It might be the issue of restore point selection logic but hard to say without the application logs.

[perjonsson1960]

Case #03513366

[douglas.fernandes]

Hello Guys

I have the same problems and the support team not solved my case is Veeam Support - Case # 03451259

Let me explain my environment.

We have 35 backup jobs with 28 Vms each totalized 44TB full backup, a robo LTO 7 with 4 tapes.

Every month we need create a full tape job to save at external guard.

The first time that I tried to do it my jobs were running for more that one week and finished with several erros. This tape jobs was configurated to running with simple Media Pool

The second time was created a GFS pool and splite in 2 tape Jobs. Same problem after running for several days and error at the end.


The solution that support team gave me:

1 - Create single tape jobs to be used with big Vms - This procedure solved a small problem that was finished a tape job before iniciated other Backup Job.

2 - Was created 6 tape jobs with +/- 150 vms by job --> The problem still the same, because "When th Backup Job start -- Tape Job identified new points and Added this point to tape again

Final soluction that support team gave me:

Create a Tape Job to each Backup Job --> This solution not was implemented. But I think that be the only that will be fix this problem.

I would like to know why this will be the only maner to solve it?

Why need be created one by one?

How I will schedule of theses tapes jobs?

Each time that i create a new jobs needs createad a new tape job, new schedule time.

The administration of this can be very dificult to manage.

Regards

[perjonsson1960]

I am copying backup copies to tape. Some of the copies have full GFS policies with weekly, monthly and yearly full backups, and most of the copies have a standard seven day retention period and one extra weekly backup. All these weekly backups are synthesized during the weekend, and the previous weekly backups are deleted.

I was thinking that what if I avoid running the tape job over the weekend, and let it run during the week? Then the tape job will not be "disturbed" by new fulls being synthesized and old fulls being deleted in the meantime. The only thing that happens during the week is that new incrementals are created, and I am not copying incrementals to tape, anyway.

Maybe that would avoid all these "rescans" and "retries" in the tape job?

[Dima P.]

Douglas,

We are reviewing you case details with QA folks. You've mentioned that you have 4 tape drives, can you please clarify if you were using tape drive parallel processing or not? What was the disk repository? Thanks!

perjonsson1960,

Maybe that would avoid all these "rescans" and "retries" in the tape job?

To be honest, the best solution would be to run the tape job after the source disk job (another option is to use 'Prevent this job from being interrupted by source backup jobs, but that would affect disk jobs).

[perjonsson1960]

No, we have two tape drives, but paralell processing is indeed used. The disk repository is a scale-out called "Scale-Out Repository 2" with two extents.

[douglas.fernandes]

We are reviewing you case details with QA folks. You've mentioned that you have 4 tape drives, can you please clarify if you were using tape drive parallel processing or not? What was the disk repository?

Thanks Dima,

Yes I m using parallel processing and My repository is a Windows 2016 Server with several Luns formatted with REFS 64

One more information -- The repository and tape are at the same Windows Server connected by FC at the same zonning.

[perjonsson1960]

The solution to my monthly "snapshot" backup of all servers to tape is:

Media pool:
1. Create a GFS Media Pool.
2. Avtivate only the "Monthly" overwrite protection period under "GFS Media Set".
3. Use parallel processing to speed things up a bit. This may result in "number of used drives -1" more tapes being used.

Tape job:
1. I choose our backup copy scale-out repository as the source of the tape job.
2. The "Export media sets" option can be left default as it will ignore all but "monthly" in this case, but I checked only "Monthly media sets" for clarity.
3. In the advanced options i check "Process the most recent restore point instead of waiting" to speed things up.
4. Under "Schedule" only the "Monthly backup" is active because of the GFS media pool settings, and I chose to run on the first Monday every month.
5. I choose to check the "Prevent this job from being interrupted by source backup jobs", because in this case complete backups on tape are more important than daily backup copies to disk.

We'll see what happens on May 6.

Many thanks to Ivan Bochagov and the rest of the support team for coming up with this solution!

PJ