Discussions specific to tape backups
Post Reply
nunciate
Expert
Posts: 172
Liked: 23 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

GFS Tape Daily Full! Why?

Post by nunciate » Mar 13, 2019 8:10 pm

So I have created all new GFS tape jobs to replace my existing standard jobs and started running some of the GFS jobs today. I have the pools set for Daily, Weekly, Monthly and Yearly. I have the jobs set to run the weekly and monthly on the same day of the week only the monthly is only set to run the first Thursday of the month. Based on this I would assume on my next tape job run, being a first run, would send incremental files to the daily media set and a full backup over to the weekly media set.

Surprise! The job is outputting a Full Backup to a Daily media set. Why? Daily should be for incremental backups only. If it is going to do a Full Backup to tape it should do it under the Weekly, Monthly or Yearly media set.

Is anyone else experiencing this? Is this something I can expect every week when a new Full is created by the primary backup job? Is it going to output my full backup to a daily media set every time? If it is I might as well go back to using regular media pools and using a stupid powershell script to update my jobs each month so I can do monthly and yearly full backups.

I waited years for Veeam to add Daily rotation to the GFS media pools. Now I wonder if it was fully thought through.

nunciate
Expert
Posts: 172
Liked: 23 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate » Mar 13, 2019 8:23 pm

I manually right clicked the jobs and chose Active Full Weekly for these so that will force them to the correct retention policy.
Really hoping this doesn't try to send full backups to daily tapes again after the initial run. I'll keep you updated

lyapkost
Veeam Software
Posts: 153
Liked: 21 times
Joined: Nov 27, 2015 2:26 pm
Full Name: Konstantin
Location: Saint Petersburg
Contact:

Re: GFS Tape Daily Full! Why?

Post by lyapkost » Mar 14, 2019 12:02 pm

Hi Alan. Tape job generally writes incremental backups to the Daily media set. In your case there were no full backups in the other media sets since it was the first job's run, so the full was copied. UG:
If there is no full restore point that can be used for linking the daily increments, the tape job will create a virtual full backup and put it to the Daily media set.
If your source job is forward incremental I recommend to schedule synthetic full to Thursdays, the same as Weekly schedule.

nunciate
Expert
Posts: 172
Liked: 23 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate » Mar 14, 2019 12:24 pm

All of my backup jobs are set to Incremental, Synthetic Full & Transform previous backup chains. Jobs are staggered over 4 days with the exception of a few archive servers which go on Wednesdays. The rest are all on Thur, Fri, Sat & Sun nights for their full backups. The corresponding tape jobs are then set to the same day for the Full.

I think just manually running the active Full on the first run of the GFS tape job should do the trick.

nunciate
Expert
Posts: 172
Liked: 23 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate » Mar 19, 2019 3:17 pm

Unfortunately, manully running these as weekly full tape backups didn't really work. Some of those simply failed and then re-ran themselves full to daily media. Right now I am just waiting for all my full backups to get run off to tape and hope that after this first time they don't send full backups to the daily retention. If they do I'll have to remove all of these GFS jobs and re-create all my old tape jobs.

Dima P.
Product Manager
Posts: 9755
Liked: 771 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: GFS Tape Daily Full! Why?

Post by Dima P. » Mar 19, 2019 6:50 pm

Hello Alan,

Just to make sure we are on the same page: daily media set copies the content from the disk to tape. It cannot create incremental backup from existing full backup on disk, so it may copy full backup.

Since you have multiple backup jobs with different full backup schedule, I wonder if it's possible to split disk backups across multiple GFS media pools based on full backup schedule? For example:

Disk jobs scheduled to create full backup on Thursday can go to GFS media pool where Weekly full backup is scheduled on Thursday. As a result you will write full backup from disk only on Thrusday while incremental will be sorted in Daily media pool. Makes sense?

nunciate
Expert
Posts: 172
Liked: 23 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate » Mar 19, 2019 7:19 pm

The media pool settings themselves don't decide which day to do a full. That is all controlled within the tape job itself or the backup job. Here is how mine are setup.

Single GFS Media Pool configured for Daily keep for 7 days, Weekly keep for 4 weeks, Monthly keep for 36 months and Yearly keep for 100 years. As stated there are no settings here for what days to send full to tape just retention periods.

Backup to Disk Job:
Backs up VM to disk storage. Set for Incremental with Synthetic Full on a specific day and transform previous incremental the same day. For examlple, Full every Thursday. That works fine and the job is configured with a secondary target of Backup To Tape job. I have many of these jobs and they are split up to do synthetic full on 4 different days. For each of these jobs I have a matching tape job. So I have 50 backup jobs and 50 corresponding tape jobs for example.

Backup to Tape Job:
Set as secondary to above job and usually sit in idle waiting for files to be detected. Configured to send to the single GFS Media Pool above. Configured to do a weekly backup on the same day as it's corresponding backup job above. Also, the Monthly is set run on the First (same say as weekly) of the month, and yearly is set to a specific day of the year which corresponds to the same day of the week as the weekly and monthly. For example, we do yearly backup the first week in April every year so a backup job configured with a weekly and monthly on Thursdays currently has it's yearly setting at a specific day of Thursday April 4th.

I think the problem is that this is the first time we have ever used these jobs and for whatever reason the system thinks it needs to store a Full backup within the daily retention. I don't know why it would do that since what I did was manually kick off the full backup to the weekly retention on the day it was set to go. So on Thursday I waited for my backup jobs to complete. I enabled all my Thursday full tape jobs and manually did an active full to the weekly retention. That did work in some cases but I am still seeing jobs attempting new full backups to a daily retention. If this completes this week and then I see more full backups going to daily next week I will know this is essentially broken and will have to fail back.

Make Sense?

nunciate
Expert
Posts: 172
Liked: 23 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate » Mar 26, 2019 12:35 pm

Unfortunately, GFS daily rotation just isn't ready for prime time. I am removing the tape jobs and re-creating my old jobs with standard media pools. It is a big disappointment because I had been waiting for the GFS media pool to add the daily options for years. It simply doesn't work right. I have using up twice the tapes and much of the full backup data is being targeted to both weekly and daily media pools even though only incremental data should ever go to a daily retention.

The other issue is I never know what really completed by looking at the console. Every day I come in I have some jobs using tape drives and many of them have been sitting for hours or even days with no activity even though many of them are at 99% complete. The remaining 100+ jobs are all running sitting at 0% complete waiting on resource availability. I have to cancel every job and the go back an manually run them one a time just to know for sure they are completing.

Veeam, I think you need to go back to the drawing board. I am sorry, I don't have the time to spend days or weeks troubleshooting this with you. I have to get my tapes out and I have to know that the jobs are running and completing.

Dima P.
Product Manager
Posts: 9755
Liked: 771 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: GFS Tape Daily Full! Why?

Post by Dima P. » Mar 26, 2019 7:31 pm

Hello nunciate,
I think the problem is that this is the first time we have ever used these jobs and for whatever reason the system thinks it needs to store a Full backup within the daily retention.
Backup chain in daily media set needs a full backup on the first run and unless you are starting GFS job on Thursday (when full backup to elder media sets is configured) you will get additional full backup during fist backup to tape job run for daily, so it looks ok, but only for the first run to daily media set.
I enabled all my Thursday full tape jobs and manually did an active full to the weekly retention. That did work in some cases but I am still seeing jobs attempting new full backups to a daily retention. If this completes this week and then I see more full backups going to daily next week I will know this is essentially broken and will have to fail back.
Just to clarify, all your source disk jobs are creating full backup on Thursday?
The other issue is I never know what really completed by looking at the console. Every day I come in I have some jobs using tape drives and many of them have been sitting for hours or even days with no activity even though many of them are at 99% complete. The remaining 100+ jobs are all running sitting at 0% complete waiting on resource availability. I have to cancel every job and the go back an manually run them one a time just to know for sure they are completing.
May I ask you to open a support case regarding this issue?

BackupMonkey
Novice
Posts: 9
Liked: 2 times
Joined: May 23, 2018 12:14 pm
Contact:

Re: GFS Tape Daily Full! Why?

Post by BackupMonkey » Mar 31, 2019 10:31 am

nunciate wrote:
Mar 26, 2019 12:35 pm
Veeam, I think you need to go back to the drawing board. I am sorry, I don't have the time to spend days or weeks troubleshooting this with you. I have to get my tapes out and I have to know that the jobs are running and completing.
I'm in the same boat, moving away from GFS due to it's substandard design.
GFS on paper sounds amazing but the way Veeam have done it is more of a hack job sold as a professional one which is a shame since the disk based stuff is amazing.

Dima P.
I've stopped raising support tickets on my issues now
All the official support know how to do is tell you that that feature, option or fix is coming in the next release.

nunciate
Expert
Posts: 172
Liked: 23 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate » Apr 01, 2019 1:13 pm

It looks like the full backups to daily retention finally worked itself out. I had to check my jobs a couple of times this weekend and I did find today that I only had 4 daily tapes which are about what I would expect from a weekend run. So it does appear that my fulls are now running off to weekly media pools as they should. I'll be interested to see what happens when the full's that were written to daily media expire.

Anyway, I didn't roll back to standard media pools yet but may still need too because of the way the GFS tape jobs handle schedules.
Every day when I come in I see over 100 jobs running. Remember, I have over 50 jobs per datacenter and have 2 locations where I tape from.
This makes it super difficult to find out exactly what jobs finished and were successful and what jobs were not. They are all running, with a few in pending mode and others just waiting on tape.
The way I used to do this previously was to run 4 jobs (1 per tape drive) at a scheduled time and then I chained all other jobs one after the other from those. That way I would always know what had run, what was running and what was set to run next. That cannot be done with GFS media pool jobs as they sit and scan and wait and if they see something they just start and sit forever or they sit in idle.

Another issue I see is that some of these jobs start, get close to finishing and then some other jobs steal the tape drive and the job that was almost complete is stuck waiting. I checked on Sunday and found many jobs that had been stuck waiting for 3 days or more. Unacceptable behavior. Once a job has the drive it should maintain access to that drive until the job is 100% complete.

So for now what I have is a feature request. Given that the last time I made a request to add Daily backups to GFS it took like 3 years I am not hopeful this will ever be done but hey I'll request it anyway.
Give us normal scheduling options for GFS tape jobs. Just give me the ability to start my jobs at a specific time or to chain the jobs one after the other as you have when I use normal media pools.

nunciate
Expert
Posts: 172
Liked: 23 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate » Apr 01, 2019 1:33 pm

I'll be installing Update 4a this week. Maybe that will help with some of these jobs hung issues.

Dima P.
Product Manager
Posts: 9755
Liked: 771 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: GFS Tape Daily Full! Why?

Post by Dima P. » Apr 09, 2019 7:21 pm

That cannot be done with GFS media pool jobs as they sit and scan and wait and if they see something they just start and sit forever or they sit in idle.
You can define the GFS tape job start time in the job schedule to the time when the source jobs are nearly completed. Will that help in your case?
Another issue I see is that some of these jobs start, get close to finishing and then some other jobs steal the tape drive and the job that was almost complete is stuck waiting. I checked on Sunday and found many jobs that had been stuck waiting for 3 days or more. Unacceptable behavior. Once a job has the drive it should maintain access to that drive until the job is 100% complete.
Tape drives should be locked by the tape job till it's completion. Can you please clarify what are the tape jobs in the question (gfs or regular tape jobs)? Are you using parallel processing? Thank you!

nunciate
Expert
Posts: 172
Liked: 23 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate » Apr 12, 2019 8:24 pm

I am sorry to say that my jobs are sending full backups to daily media sets again today. I thought things stabilized as the week after I had the initial issue all full backups went to weekly media pools. Then last week as expected all full backups went to yearly retention as we do that once a year in April. Unfortunately, my full tape outs started today and I see a ton of daily tape usage and jobs are showing daily media set usage for full backups again.

I'll try to get a ticket together on Monday.

Dima P.
Product Manager
Posts: 9755
Liked: 771 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: GFS Tape Daily Full! Why?

Post by Dima P. » Apr 16, 2019 6:02 pm

Alan,

Sorry to hear that. Can you please submit a support case and share the case ID with us (we will need a potion of debug logs to identify the root cause)? Thank you in advance!

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests