Feature Request: Always copy the latest Full Bakcup to Tape

[staskorz]

Hello,

This is a follow-up topic for Support Case ID 02210537.

TL;DR;

We need a way to make Veeam Backup and Replication to be able to ALWAYS copy full VM backups to tapes. Currently, full backups are copied to tapes ONLY IF disk backup job starts and completes the same day, before 00:00. The Support Representative said it’s by design…

Full Version:

At first glance there was nothing special about our backup requirements, it's just a plain Backup-to-Disk-to-Tape scenario. We've created a backup job to backup some VMware VMs, and a Tape job to copy the latest full VM backups to tape. The Tape job is set to run after the source VM-to-Disk job completes.

The thing is we've noticed sometimes VMs were not copied to tapes, with a log message saying there were no new backups to process, which of course was not the case. Furthermore, the job didn’t fail, it completed successfully, but without actually copying anything to tapes. Opening a support case at Veeam resulted in a surprising reply saying it’s by design – and here is the explanation we’ve got: the tape job will copy full VM backup ONLY IF they were taken during the same daily cycle, which starts at 00:00.

There are several issues with such an approach:
1.Backups are usually scheduled to off-hours – and 00:00 is about in the middle of the off-hours period
2.It’s impossible to predict the duration of a backup job, as it can be affected by numerous factors, such as unexpected load on underlying production or backup infrastructure, a CBT reset (say after changing VM disk size) resulting in a full disk re-read, etc.
3.Global organizations might have different daily cycle boundary preference than the local 00:00 time of their backup server

Our current workaround, which is far from perfect, is to split the jobs so those starting before 00:00 will, with high probability (???), finish the same day. The thing is, one would expect a far more deterministic approach to her backup setup… I mean restoring from a backup is your last hope to gain your systems and data back would anything go seriously wrong – and as such it MUST be rock-solid, not depending on uncontrolled external factors. Another issue with our workaround is the backup duration – as described, we had to split the jobs: those that have high probability to finish before 00:00 are scheduled to start at evenings, leaving the longer ones to start after 00:00. As a result, the backup now takes far longer to complete and does not finish on time.

Here are the possible solutions I’ve proposed to the support representative:
1.Add an option to configure the daily cycle boundary, defaulting to 00:00 – this way nothing will change for those not interested
2.Add an option to force the tape copy job to ignore the daily cycle boundary, so it will always copy the latest full backup to tapes – same here, leaving this option off by default means nothing will change for those not interested

The second proposed solution seems to be better than the first one, because it IS deterministic, while the first one IS NOT. The thing is no one can tell for sure a job won’t take more than 24 hours to complete – say due to an unusual load on the underlying infrastructure or a planned change to sizes of several large VM disks – an operation causing CBT reset followed by a full disk re-read – and those are just two of many possible causes for a backup job to take way longer than usual to complete.


Thanks and Best Regards,

Stas.

[veremin]

Basically, you want to archive latest available full backup on everyday basis, regardless of whether it's been archived previously or not? What mode does a source backup job use? Reversed incremental, forward incremental with periodic full or forward forever incremental (which I believe is not the case, since you mentioned periodic full backup, but it's still worth double checking)? Thanks.

[staskorz]

As far as I understand, when Veeam needs to write a "Full Backup" to a tape, the tape copy job synthetically produces a full backup from the the latest incremental available in a backup chain (please correct me if I'm wrong) - which is exactly what we need.

Regarding the backup mode: we use forever forward incremental without periodic fulls. By "copy full backup to tape" I meant "synthetically produce a full backup and copy it to tape".

Thanks!

[veremin]

Got it, in this case all you need to do is to select all week days as dates for virtual full backup ("Schedule" button on "Full backup" settings page).

With such configuration whenever a backup to tape job runs, it will synthetically create a full backup directly on tapes.

[staskorz]

The thing it ONLY works IF the Backup-to-Disk job completes the same day (before 00:00). If, say, the job starts at 22:00 and completes at 00:05 it doesn't work - the Tape job says there are now new backups to process. You can take a look at the logs attached to Case ID 02210537. Thanks!

[veremin]

In this case why not to make backup to tape job run during the day (say, 3 pm or something) , instead of scheduling it after the source backup job? This way, whenever executed the backup to tape job will create virtual full backup without skipping anything. Thanks.

[staskorz]

The requirement is to take the Backup-to-Disk after 19:00 (after the day shift ends), and finish copying to tapes around noon, so a courier could pick up the backup tapes and ship them to an offsite location.

In any case, even if we do it as you say and start the backup job during the day (which does not meet our requirements), no one can tell for sure the backup-to-disk job will actually finish before 00:00 - say due to a CBT reset or because of additional load on underlying infrastructure.

Thanks.

[veremin]

I was talking about starting tape job during the day, not the backup one. Once executed, it will find restore point created on previous day (or on current day at 00-something - doesn't matter) and will use it as a source for virtual full backup.

The desired scenario will be achieved - one virtual full backup a day.

Thanks.

[staskorz]

As I said, the backup tape media is taken offsite. The courier comes to pick up the tapes before noon. The obvious goal is to handle the courier the freshest backup copy possible - if we start copying to tapes the day before, then the data on tapes sent to an offsite location is already over 30 hours old.

But generally speaking, I don't really get it: I mean Veeam Backup and Replication is a backup software, isn't it?? So it's supposed to backup. To ALWAYS backup. Not to think whether it should backup or not, but to always backup when instructed so. You guys try to find workarounds for a ridiculously simple classic scenario - just plain Backup-to-Disk-to-Tape. Why does the source backup start/completion times or duration can at all affect Veeam's decision whether to copy to tapes or not??? Sorry, but it's just ridiculous!

Really, it's not just me, when I tell other IT guys working for various organizations about this ridiculous limitation, their first reaction is shock - the second is laughter.

Veeam Backup and Replication is a great backup software, I just don't get it how the Product Management allowed such a limitation to slip in.

Sorry, it just frustrates me a lot.

Thanks.

[veremin]

As I said, the backup tape media is taken offsite. The courier comes to pick up the tapes before noon.

Then, start a backup to tape job before noon, say, at 5 am. The virtual full will be created, based on the latest available restore point.

Virtual full backup is created once a day, not frequently - this is currently by design behaviour. If don't you find that applicable to your scenario, consider switching backup mode.

However, we do appreciate your feedback, and will think how current experience might be improved in one of the next product versions.

Thanks.

[staskorz]

Then, start a backup to tape job before noon, say, at 5 am. The virtual full will be created, based on the latest available restore point.

Not enough time to complete copying everything to tapes before noon.

Thanks.

[Absalom]

Just want to say "me too". We only do monthly backups to tape but sometimes they fail because of a bad tape etc and it's really frustrating that I can't just tell the backup to run again because there are no new backups to process. To compound the issue, we don't copy to tape directly from backup - we do it from the backup copy. The backup copy only runs every few days so I have to wait a few days before retrying the backup. Would be really nice to have a tick box or something to force it to copy the latest backup to tape.

-Aaron

[veremin]

Monthly backups? Are you using GFS backup to tape job for that? If so, you can always make it create a gfs restore point immediately (right-click - active full - monthly). Thanks.

[Absalom]

Yes, we are using GFS. Wasn't aware of that feature so thanks for that

[veremin]

You're welcome. If this tasks needs to be automated, you can leverage our PowerShell snap-in for that:

Code: Select all

asnp VeeamPssnapin
Sync-VBRBackupToTapeJob -Job (Get-VBRTapeJob -Name "Name of your tape job") -Monthly

Thanks.

[Encrypted1024]

I had the same issue. The wording around the tape backups leads you to believe you should be able to easily do a synthetic full backup to tape each day, however it does not work consistently due to the factors mentioned above. The way I found to deal with this is to switch your backups to Reverse Incremental so your most recent backup is always a full backup. This will insure that a full backup is written to tape each night. According to Veeam support this is the recommended best practice when running full backups to tape.