Discussions related to exporting backups to tape and backing up directly to tape.
Post Reply
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

GFS Tape Daily Full! Why?

Post by nunciate »

So I have created all new GFS tape jobs to replace my existing standard jobs and started running some of the GFS jobs today. I have the pools set for Daily, Weekly, Monthly and Yearly. I have the jobs set to run the weekly and monthly on the same day of the week only the monthly is only set to run the first Thursday of the month. Based on this I would assume on my next tape job run, being a first run, would send incremental files to the daily media set and a full backup over to the weekly media set.

Surprise! The job is outputting a Full Backup to a Daily media set. Why? Daily should be for incremental backups only. If it is going to do a Full Backup to tape it should do it under the Weekly, Monthly or Yearly media set.

Is anyone else experiencing this? Is this something I can expect every week when a new Full is created by the primary backup job? Is it going to output my full backup to a daily media set every time? If it is I might as well go back to using regular media pools and using a stupid powershell script to update my jobs each month so I can do monthly and yearly full backups.

I waited years for Veeam to add Daily rotation to the GFS media pools. Now I wonder if it was fully thought through.
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate »

I manually right clicked the jobs and chose Active Full Weekly for these so that will force them to the correct retention policy.
Really hoping this doesn't try to send full backups to daily tapes again after the initial run. I'll keep you updated
lyapkost
Expert
Posts: 221
Liked: 48 times
Joined: Nov 27, 2015 2:26 pm
Full Name: Konstantin
Location: Saint Petersburg
Contact:

Re: GFS Tape Daily Full! Why?

Post by lyapkost »

Hi Alan. Tape job generally writes incremental backups to the Daily media set. In your case there were no full backups in the other media sets since it was the first job's run, so the full was copied. UG:
If there is no full restore point that can be used for linking the daily increments, the tape job will create a virtual full backup and put it to the Daily media set.
If your source job is forward incremental I recommend to schedule synthetic full to Thursdays, the same as Weekly schedule.
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate »

All of my backup jobs are set to Incremental, Synthetic Full & Transform previous backup chains. Jobs are staggered over 4 days with the exception of a few archive servers which go on Wednesdays. The rest are all on Thur, Fri, Sat & Sun nights for their full backups. The corresponding tape jobs are then set to the same day for the Full.

I think just manually running the active Full on the first run of the GFS tape job should do the trick.
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate »

Unfortunately, manully running these as weekly full tape backups didn't really work. Some of those simply failed and then re-ran themselves full to daily media. Right now I am just waiting for all my full backups to get run off to tape and hope that after this first time they don't send full backups to the daily retention. If they do I'll have to remove all of these GFS jobs and re-create all my old tape jobs.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: GFS Tape Daily Full! Why?

Post by Dima P. »

Hello Alan,

Just to make sure we are on the same page: daily media set copies the content from the disk to tape. It cannot create incremental backup from existing full backup on disk, so it may copy full backup.

Since you have multiple backup jobs with different full backup schedule, I wonder if it's possible to split disk backups across multiple GFS media pools based on full backup schedule? For example:

Disk jobs scheduled to create full backup on Thursday can go to GFS media pool where Weekly full backup is scheduled on Thursday. As a result you will write full backup from disk only on Thrusday while incremental will be sorted in Daily media pool. Makes sense?
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate »

The media pool settings themselves don't decide which day to do a full. That is all controlled within the tape job itself or the backup job. Here is how mine are setup.

Single GFS Media Pool configured for Daily keep for 7 days, Weekly keep for 4 weeks, Monthly keep for 36 months and Yearly keep for 100 years. As stated there are no settings here for what days to send full to tape just retention periods.

Backup to Disk Job:
Backs up VM to disk storage. Set for Incremental with Synthetic Full on a specific day and transform previous incremental the same day. For examlple, Full every Thursday. That works fine and the job is configured with a secondary target of Backup To Tape job. I have many of these jobs and they are split up to do synthetic full on 4 different days. For each of these jobs I have a matching tape job. So I have 50 backup jobs and 50 corresponding tape jobs for example.

Backup to Tape Job:
Set as secondary to above job and usually sit in idle waiting for files to be detected. Configured to send to the single GFS Media Pool above. Configured to do a weekly backup on the same day as it's corresponding backup job above. Also, the Monthly is set run on the First (same say as weekly) of the month, and yearly is set to a specific day of the year which corresponds to the same day of the week as the weekly and monthly. For example, we do yearly backup the first week in April every year so a backup job configured with a weekly and monthly on Thursdays currently has it's yearly setting at a specific day of Thursday April 4th.

I think the problem is that this is the first time we have ever used these jobs and for whatever reason the system thinks it needs to store a Full backup within the daily retention. I don't know why it would do that since what I did was manually kick off the full backup to the weekly retention on the day it was set to go. So on Thursday I waited for my backup jobs to complete. I enabled all my Thursday full tape jobs and manually did an active full to the weekly retention. That did work in some cases but I am still seeing jobs attempting new full backups to a daily retention. If this completes this week and then I see more full backups going to daily next week I will know this is essentially broken and will have to fail back.

Make Sense?
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate »

Unfortunately, GFS daily rotation just isn't ready for prime time. I am removing the tape jobs and re-creating my old jobs with standard media pools. It is a big disappointment because I had been waiting for the GFS media pool to add the daily options for years. It simply doesn't work right. I have using up twice the tapes and much of the full backup data is being targeted to both weekly and daily media pools even though only incremental data should ever go to a daily retention.

The other issue is I never know what really completed by looking at the console. Every day I come in I have some jobs using tape drives and many of them have been sitting for hours or even days with no activity even though many of them are at 99% complete. The remaining 100+ jobs are all running sitting at 0% complete waiting on resource availability. I have to cancel every job and the go back an manually run them one a time just to know for sure they are completing.

Veeam, I think you need to go back to the drawing board. I am sorry, I don't have the time to spend days or weeks troubleshooting this with you. I have to get my tapes out and I have to know that the jobs are running and completing.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: GFS Tape Daily Full! Why?

Post by Dima P. »

Hello nunciate,
I think the problem is that this is the first time we have ever used these jobs and for whatever reason the system thinks it needs to store a Full backup within the daily retention.
Backup chain in daily media set needs a full backup on the first run and unless you are starting GFS job on Thursday (when full backup to elder media sets is configured) you will get additional full backup during fist backup to tape job run for daily, so it looks ok, but only for the first run to daily media set.
I enabled all my Thursday full tape jobs and manually did an active full to the weekly retention. That did work in some cases but I am still seeing jobs attempting new full backups to a daily retention. If this completes this week and then I see more full backups going to daily next week I will know this is essentially broken and will have to fail back.
Just to clarify, all your source disk jobs are creating full backup on Thursday?
The other issue is I never know what really completed by looking at the console. Every day I come in I have some jobs using tape drives and many of them have been sitting for hours or even days with no activity even though many of them are at 99% complete. The remaining 100+ jobs are all running sitting at 0% complete waiting on resource availability. I have to cancel every job and the go back an manually run them one a time just to know for sure they are completing.
May I ask you to open a support case regarding this issue?
BackupMonkey
Influencer
Posts: 14
Liked: 4 times
Joined: May 23, 2018 12:14 pm
Contact:

Re: GFS Tape Daily Full! Why?

Post by BackupMonkey »

nunciate wrote: Mar 26, 2019 12:35 pm Veeam, I think you need to go back to the drawing board. I am sorry, I don't have the time to spend days or weeks troubleshooting this with you. I have to get my tapes out and I have to know that the jobs are running and completing.
I'm in the same boat, moving away from GFS due to it's substandard design.
GFS on paper sounds amazing but the way Veeam have done it is more of a hack job sold as a professional one which is a shame since the disk based stuff is amazing.

Dima P.
I've stopped raising support tickets on my issues now
All the official support know how to do is tell you that that feature, option or fix is coming in the next release.
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate »

It looks like the full backups to daily retention finally worked itself out. I had to check my jobs a couple of times this weekend and I did find today that I only had 4 daily tapes which are about what I would expect from a weekend run. So it does appear that my fulls are now running off to weekly media pools as they should. I'll be interested to see what happens when the full's that were written to daily media expire.

Anyway, I didn't roll back to standard media pools yet but may still need too because of the way the GFS tape jobs handle schedules.
Every day when I come in I see over 100 jobs running. Remember, I have over 50 jobs per datacenter and have 2 locations where I tape from.
This makes it super difficult to find out exactly what jobs finished and were successful and what jobs were not. They are all running, with a few in pending mode and others just waiting on tape.
The way I used to do this previously was to run 4 jobs (1 per tape drive) at a scheduled time and then I chained all other jobs one after the other from those. That way I would always know what had run, what was running and what was set to run next. That cannot be done with GFS media pool jobs as they sit and scan and wait and if they see something they just start and sit forever or they sit in idle.

Another issue I see is that some of these jobs start, get close to finishing and then some other jobs steal the tape drive and the job that was almost complete is stuck waiting. I checked on Sunday and found many jobs that had been stuck waiting for 3 days or more. Unacceptable behavior. Once a job has the drive it should maintain access to that drive until the job is 100% complete.

So for now what I have is a feature request. Given that the last time I made a request to add Daily backups to GFS it took like 3 years I am not hopeful this will ever be done but hey I'll request it anyway.
Give us normal scheduling options for GFS tape jobs. Just give me the ability to start my jobs at a specific time or to chain the jobs one after the other as you have when I use normal media pools.
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate »

I'll be installing Update 4a this week. Maybe that will help with some of these jobs hung issues.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: GFS Tape Daily Full! Why?

Post by Dima P. »

That cannot be done with GFS media pool jobs as they sit and scan and wait and if they see something they just start and sit forever or they sit in idle.
You can define the GFS tape job start time in the job schedule to the time when the source jobs are nearly completed. Will that help in your case?
Another issue I see is that some of these jobs start, get close to finishing and then some other jobs steal the tape drive and the job that was almost complete is stuck waiting. I checked on Sunday and found many jobs that had been stuck waiting for 3 days or more. Unacceptable behavior. Once a job has the drive it should maintain access to that drive until the job is 100% complete.
Tape drives should be locked by the tape job till it's completion. Can you please clarify what are the tape jobs in the question (gfs or regular tape jobs)? Are you using parallel processing? Thank you!
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate »

I am sorry to say that my jobs are sending full backups to daily media sets again today. I thought things stabilized as the week after I had the initial issue all full backups went to weekly media pools. Then last week as expected all full backups went to yearly retention as we do that once a year in April. Unfortunately, my full tape outs started today and I see a ton of daily tape usage and jobs are showing daily media set usage for full backups again.

I'll try to get a ticket together on Monday.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: GFS Tape Daily Full! Why?

Post by Dima P. »

Alan,

Sorry to hear that. Can you please submit a support case and share the case ID with us (we will need a potion of debug logs to identify the root cause)? Thank you in advance!
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate »

I have not had a chance to submit a ticket because I have been overwhelmed with other work but I have been holding my backups together with chewing gum and baling wire for too long. It will take me about 30 minutes to remove all these GFS tape jobs and re-create the original regular tape out jobs and that is what I am doing now. I really don't have the time to spend weeks on this with support so hopefully, someone else with this issue will speak up. For now, I am failing back. We have literally spent thousands of dollars bolstering our tapes with new ones to compensate for the big increase in the number used since converting. I can't buy more and honestly, I shouldn't have too. So, for now, I'll go back to my old way and maybe look at this again at some point later.

Just to be clear, I am still seeing a lot of tapes used up by daily retentions getting full backups once a month. That should not be happening with a properly designed environment but I can't make it stop.
EdNewman
Influencer
Posts: 14
Liked: never
Joined: Jan 15, 2018 4:24 pm
Full Name: Ed Newman
Contact:

Re: GFS Tape Daily Full! Why?

Post by EdNewman »

Did you ever get anywhere with this? We are looking to move from old style tape backups to GFS with dailies and our first test had the same issue as you are describing, full backups being written on the daily job, despite a full backup being run the night before.

Cheers,
Ed.
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate »

Nope, I never did fix this and I had converted many jobs to GFS. I ended up just recreating my old style jobs and that was a pain. The way I do it now is I have a scheduled task that runs a set of PowerShell commands to move all my tape jobs to a Monthly pool once a month and then the week after that another job moves them back to the weekly pools.

I will test again after B&R v10 releases.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: GFS Tape Daily Full! Why?

Post by Dima P. »

Hello Ed,
old style tape backups to GFS with dailies and our first test had the same issue as you are describing, full backups being written on the daily job, despite a full backup being run the night before.
Can you please clarify your configuration:

1. What is the source job backup mode (forever forward incremental, forward incremental with periodic full backups or reverse incremental) and how this job is scheduled? If periodic full backups are enabled please let us know the defined schedule.
2. What retention is set for GFS media sets?
3. How tape job in the question is scheduled?

Thank you in advance!
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate »

How unfortunate. I finally got around to testing v10 on my Dev system. Looks like GFS tapes are still outputting full backups to a daily retention.
This is confirmed by looking at the tape after the job is successful and I can see a VBK file stored there.

See from the log:

Code: Select all

5/5/2020 9:56:32 AM :: Building source backup files list started at 5/5/2020 9:56:32 AM  
5/5/2020 9:56:33 AM :: No GFS candidate for Daily media set was found for today, waiting  
5/5/2020 9:56:33 AM :: No backup files found  
5/5/2020 10:06:35 AM :: Synthetic full backup from <myserver>.vib (5/5/2020 12:00:00 AM) will be placed into the daily media set  
5/5/2020 10:06:36 AM :: Source backup files detected. VBK map: 1  
5/5/2020 10:06:36 AM :: Queued for processing at 5/5/2020 10:06:36 AM  
5/5/2020 10:06:36 AM :: Required backup infrastructure resources have been assigned  
5/5/2020 10:06:37 AM :: [<myserver>] Using tape library IBM 3573-TL D.70  
5/5/2020 10:06:37 AM :: Backup to media set daily started at 5/5/2020 9:56:32 AM  
5/5/2020 10:06:41 AM :: [<myserver>] Drive 1 (Server: <myserver>, Library: IBM 3573-TL D.70, Drive ID: Tape0) locked successfully  
5/5/2020 10:07:39 AM :: [<myserver>] Current tape is 070291L4  
5/5/2020 10:08:00 AM :: New tape backup session started, encryption: disabled  
5/5/2020 10:08:25 AM :: Creating full backup map for <myserver>.vib  
5/5/2020 10:08:43 AM :: [<myserver>] Processing <myserver>.vbk  
5/5/2020 10:27:30 AM :: 0 folders and 1 files have been backed up  
5/5/2020 10:27:30 AM :: Busy: Source 0% > Proxy 3% > Network 40% > Target 84%  
5/5/2020 10:27:30 AM :: Primary bottleneck: Target  
5/5/2020 10:27:30 AM :: Network traffic verification detected no corrupted blocks  
5/5/2020 10:27:30 AM :: Processing finished at 5/5/2020 10:27:30 AM
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: GFS Tape Daily Full! Why?

Post by Dima P. »

Hello Alan,

Daily media set requires full backup to link the following incrementals, so that's the first case when you can see full backup in daily media set. Another scenario is when you already have full backup on disk: tape job cannot create incremental restore point from full backup thus it will tape out what is stored on your repository.

Can you please clarify how the source job was configured (forever incremental job type or with periodic full backups) and how weekly full backup was scheduled in the gfs tape job? Thanks!
nunciate
Expert
Posts: 247
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: GFS Tape Daily Full! Why?

Post by nunciate »

My backup jobs are set as Incremental, Create Synthetic Full Weekly on Thursday and Transform previous backup chains into rollbacks.
I don't have the space even with deduplications to do anyting other than that. I can't have multipe full copies of backups for every VM. I simply don't ahve the space.
I have tried and it brought my storage down so that isn't an option for me.

Currently I only use GFS within my test system and it is a mirror of production.

GFS is set to keep Daily for 7 days, Weekly for 4 weeks, Monthly for 36 Months and Yearly forever (100 years). No Quarterly.
All pools are set to add tapes automatically and append incomplete tapes.

Tape job is configured to perform backups Daily every day, Weekly on Saturday, Monthly on First Saturday, and Yearly on April 10th every year.

The thing is that with every backup system I have ever worked with Daily pools are reserved for incremental backups while the others are meant for fulls. Or at miniumn there should be some configuration that allows for that. Something to let me say I only want incremental backups to be stored in Daily media sets and force all full backups into the Weekly, Monthly and Yearly pools. Otherwise I am just eating up tapes with extra backups.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: GFS Tape Daily Full! Why?

Post by Dima P. »

Thanks for the details Alan! Was the backup to daily media set performed before the backup to weekly media set? Cheers!
fga
Influencer
Posts: 22
Liked: 23 times
Joined: Feb 04, 2019 10:08 pm
Location: Germany
Contact:

Re: GFS Tape Daily Full! Why?

Post by fga »

This might be a somewhat older Topic but this problem still exists in the latest V10...
Our GFS Tape Job does the same.
Source jobs are configured for Sunday to Friday increments, Saturday Virtual full, Last Saturday in the Month: Active full.
Yet today surprise... The GFS Job creates a Full backup map for todays increment and fills up my daily tapes.
Yes last Saturdays Full Backups were created and are on tape. They were even online in the Changer.
Last week the Job did the same thing on wednesday. So i deleted the old GFS job in case something went wrong and recreated it. Yet today it does the same thing again.
Support Case: #04584834
Post Reply

Who is online

Users browsing this forum: No registered users and 24 guests