How to wipe File to Tape job tape without breaking Veeam?

[mcsmithSOP]

We have a physical file server in our head office that contains 1.6TB of data. Approximately 900GB is originates in head office and the other 700GB is replicated from our remote locations.

Due to bandwidth limitations, we cannot protect the entire server with EP and send a disk copy off-site. (EP copy jobs do not have the option to specify which volumes should be excluded from the copy job, nor can you set up separate Full/Incremental EP jobs.) Therefore, the volumes that contain head office data only are protected by EP and copied off-site and to tape as Backup to Tape job. The data that originates from the remote locations is backed up as File to Tape. (End result is we have two disk copies which are in separate locations and one tape copy.)

We had a case where the File to Tape job failed but a senior engineer determined it was a network error. Other than that everything has been working flawlessly. The first tapes (which are expired) have made their way back into the library so I decided to put them back in the Free pool. (Tapes are automatically put into either Full or Incremental pool on schedule.) The first two tapes (only containing Backup to Tape jobs) were fine but the third tape containing a File to Tape job began writing to the SQL database until the OS drive was full and the server crashed. Support Case # 01819311. Once the initial problem (not able to log in to console was solved) it was passed onto another tech.

Even though I told the Senior Tech how much data has being backed up and he never said anything, the current tech says I am backing up way more data than Veeam is supposed to be used for. (Would've been nice to see something in the documentation....)

His solution is that I add another 50GB to my C: drive so the database can expand without crashing the server. I'm not a fan of that solution for multiple reasons. I only need the last full and latest incrementals of the job. Is there a way I can just nuke the data without causing the database to expand?

[Dima P.]

Hi Jason,

If you are using file to tape job each file is added to the backup catalogue, which is stored in the SQL database. To remove the tape and residing files you can use remove the tape from the catalogue. That might temporary solve your issue but if you keep using the file to tape jobs the suggestion to increase the disk space sound like a perfect plan.

[mcsmithSOP]

Hi Dima,

What I am failing to understand is why the database grows when trying to delete/remove/wipe a tape. I understand that Veeam catalogues the files so the database will grow each time the job runs, but when the tape is no longer required, why does it expand the database rather than shrink it?

[Dima P.]

Jason,

Give me a short time to check with QA team and thanks for the case - I'll use is as a reference.

[mcsmithSOP]

Thanks. I responded to the ticket (automated follow up), but I haven't had a chance to implement the proposed solution. (Way too many fires to put out ... )
I also added that question there, but I don't want to have two people at Veeam trying to find the same answer for me. Thanks.

[skrause]

Did you try removing the tape from the catalog while it was offline before putting it back into the library?

[mcsmithSOP]

Did you try removing the tape from the catalog while it was offline before putting it back into the library?

Nope, I didn't. The first two tapes I marked Free were fine because they contained Backup to Tape jobs.

I just noticed the reply from the engineer that it is temporary transaction log growth. So it doesn't look like there's a way around it. Still confused why it was suggested I add 4 times the total DB size - it's not like I'm trying to delete all the tapes. None of my other heavy transaction DBs see that kind of growth in transaction logs. (But I'm not a DBA, so what do I know...)

[Dima P.]

Transaction logs growth indeed might be the root cause. Are you using a file to tape backup from the VBR server with VSS (or any other solution like VEB to backup your VBR server)?

[mcsmithSOP]

I added the file server to Managed Servers in VBR, and then set up the File to Tape job with VSS enabled.

Very long story how we got ourselves into this situation....

Last spring during the product demo, I mentioned that we had a couple physical servers that I needed to back up. The engineer mentioned that VEB was meant for desktops and laptops but it was supported all the way up to WS2012 R2 *wink wink*. I asked on the forum if VEB could handle 1.5+ TB of data and was assured it could, so great, we had our new backup solution.

Late spring/early summer I provisioned our new file server (the old one died) and created Users and Shares volumes (among others). Late summer we got our licenses for Veeam so I ordered a new server for our off-site backup copies and set up the on-site backup jobs. Two (aggravating) months later, the off-site server showed up so I provisioned it and pre-seeded it locally. At that point, now that we knew how much data we were backing up (at the outset we had to ball park the number since we didn't know what our actual compression/deduplication ratios would be), we realized that we would not be able to get all the data off-site within a 24 hour interval.

That's how we ended up in the situation (mentioned in the first post) where four volumes are backed up using VEB and three volumes are backed up using F2T. Currently, the F2T job is set to only include the remote location folders. My intention was that once I had backups of all the data, I would delete the head office data from the F2T volumes and set the F2T job at the volume level. I wanted to make sure I had a backup of all data before deleting it in case I mistyped the command and nuked the wrong data, but I haven't gotten to deleting the obsolete data and modifying the F2T job.

Further compounding the problem (maybe) is that I was heavily leaning towards having separate media sets for Backup to Tap and File to Tape, but my boss convinced me to use on media set and mix the job types.

Within the next few weeks, we'll be getting a 40% bandwidth increase, and we have not ruled out upgrading to Plus at the end of the summer so we can take advantage of the WAN accelerator. At that point, hopefully we won't need to monkey around with the File to Tape job.

I'm running tight on SAN storage, so I don't have the option of continually adding space to the Veeam server. If it ends up being a matter of coming in every other weekend to delete an obsolete tape and then shrinking the database, and then repeat that's what I'll have to do. I've expanded the C: drive from 50GB to 70GB so I've got 23GB free right now. Once my backup copy jobs finish in a couple hours, I'll try marking that tape as "Free" again. I no longer have the option to remove that tape from catalog, but depending on how much space I end up after this tape, I'll give that at try with another tape.

[mcsmithSOP]

I went to mark the tape in question as Free and almost didn't notice that Veeam used it for a Full backup on the weekend and just about nuked this week's Full tape backup. That explains why the Veeam server ran low on disk space Saturday when the first Full Backup to Tape job ran.

[mcsmithSOP]

So I've confirmed with Support that our only option is remove from catalog the tapes containing data we no longer need and then shrink the db and truncate the logs.

Not optimal, but unless a future release of VEB has the option to exclude certain volumes from copy jobs or the ability to run different jobs, that's the way it'll have to be. Once I get my increased bandwidth, that'll definitely help and I'll be able to move some of the data back to VEB instead of F2T.

[skrause]

May I ask the reason why your file server isn't virtualized?

[mcsmithSOP]

May I ask the reason why your file server isn't virtualized?

The physical file server was half the cost of another SAN node. I could've set it up as an iSCSI target, but I didn't have the extra 2012R2 license I would've needed for the VM guest OS.