Best practise: Making full backups to tape while NOT blocking source backup jobs in meantime

[wbauer84]

Dear Forum,

Circumstances:
Veaam B&R v12
very fast ReFS backup repository
2x LTO drives (8+9)
Per Machine backup files within a job: active

We want to have every day as much as possible full backups on tape, so that we are not depending on more than one tape for restoring a backup. For that we have 2 LTO drives, which we can write simultaneously, as our read/write throughput on our REFS backup repository is around 1-1,5 GByte/sec (very fast).
But LTO8/9 are not speeding up practically more than 280-300 MByte/sec
The problem now is, that tape jobs are blocking our source backup jobs over the productive time (8x5) too much, where we would like to make high backup intervals in meantime, every 30 mins around (as incremental backups of all our VMs take 10-15 mins max).
All what I experienced so far is, that tape jobs block the source backup job in general-
But is there probably an exception of that situation, like backing up just the latest synthetic full on tape (we would do synthetic fulls on the source job every day), while backup job is doing forever incremental, where he would not touch VBK, and just write another VIB, while tape job pulls the last synthetic full VBK to tape?

Or any other approach to that situation?
The only other idea I would have, is just so split up the jobs more, and pack more than one source backup to tape, to block less the source backup jobs

[david.domask]

Hi @wbauer84,

Can you clarify a little on what's happening? Are the source jobs stopping _with an error_ or just queuing behind the tape job? Also are the tapes backing up already existing full backups from the source job or are they synthesizing Virtual Fulls to tape? (you can check this in any of the Tape job reports; if it says "Synthetic Full" for the task, it's Virtual Full to tape, which is fastest way to get fulls to tape right now with Veeam)

If it's the former, you might be hitting a known issue; I don't have the issue ID handle right now, but if that is your situation, open a Support case and link your case number here, I'll check it and if it matches, there is a hot fix which may help. TruePerVM in v12 introduced a difference in locking mechanisms for the backups and in some situations might cause the tape job to make the source jobs abort with an error about concurrent locks, and the fix allows a mechanism to ignore this.

As far as speeds, 280-300 is pretty good for LTO8/9. It's a little slow, but I guess depends on the bottleneck.

If it's source, then it means reads from the source repository aren't fast enough (remember, writing to ReFS might be good, but random read IO might not perform the same).

If it's target, then probably there's something in the connection/driver side between tape server/tape hardware.

[wbauer84]

Dear David,

So far we experience no "errors". My question is way more a backup job design question, as our main goal is, to backup around 40-45 TB (effective in VBKs) on tape as much and as recent as possible, as a full backup, NOT splitting backups over several tapes or by full and increment tapes. We are aware, that based on let's say 250 MByte/sec LTO write speed, we will in theory hit max 43 TB/day, if we have 2 LTO's available, and that we will have to backup some things just every 2 days on LTO, but that's not a problem. LTO speed is limited by the LTO for sure, because on the backup repository we have constantly 1 - 1,5 GByte/sec speed, in read and write (based on Active Full jobs, pulling from SAN), which is awesome. Further, we experience similar LTO speeds at other installations.

The problem we would hit now, is that by running day and night LTO tape jobs, we block our backup source jobs, we also wanna run quite frequently (every 30mins) at least around 8x5 core working time.

But I asked myself, if I configure the backup jobs to do synthetic full every day (which he does at the first backup job, running on this day), the follow-up incremental jobs, he would write in a separate VIB file. If I let then a tape job start after the first synthetic full, he would lock the VBK (synthetic full), but could still go on with the backup job, and write VIB's, while corresponding tape job is running, and reading from the VBK.
I tested this, this weekend, and my assumptions seems to be true. Tape job was writing, and corresponding source backup job was able to run in parallel, without waiting for tape job to finish.

The thing is now, is that a supported/valid behaviour, and how this would work out, when tape job does not run right behind a active or synthetic full, and would run, after one or several incremental job? Would then be the last VIBs be locked, and source backup job could not run in parallel, because of this lock?

If the virtual full writing to tape is helpful with this locking VBK/VIB-thing, I'm not sure in the moment, because he just synthesizes VBK's and VIB's to a more recent VBK "on the fly", but that doesn't change anything behalf of locking source backup job files, and allow source backup job to run in parallel, right?

Case # 06045630

[david.domask]

> The problem we would hit now, is that by running day and night LTO tape jobs, we block our backup source jobs, we also wanna run quite frequently (every 30mins) at least around 8x5 core working time.

So the thing is, the locking happens at a storage level; when you enable the option "Prevent this job from being interrupted by source job" for the Backup to Tape job, this allows the tape job to retain its lock on the storage (backup file) and the source job will queue the task that requires that storage and continue with other tasks. Without the option enabled, the Tape job will fail with an error and surrender its lock to the source job.

The "Prevent this job from being interrupted" option doesn't stop the source jobs from running, it just means they need to wait their turn for the backup file. Note that there is an issue in v12 where with True PerMachine backups, the locking is not working quite as desired, but there is a fix + Registry value for this already.

So it is quite supported, and as above, the source job will just queue.

With regards to Virtual Fulls, the reason I asked is because for Virtual Full to Tape, the engine supports asynchronous reads; Virtual Fulls don't require extra space for a full _on disk_; the necessary files to make a full from an increment are read and the data is streamed to tape. So if your source jobs are Forever Forward Incremental (FFI), you don't need to worry about when the tape job runs vs the source job; just let them run and the scheduler will handle it

[bct44]

Hello,

I have the same behavior since v12 specially for many huge vbk (huge windows servers with agents) to copy to tape, before it was continued (i know it's not a best pratices but i have no choices) during many days. Now it's automatically interrupted when the backup jobs source is running.

The "Prevent this job from being interrupted" option doesn't stop the source jobs from running, it just means they need to wait their turn for the backup file.
This means that by increasing the value, it will return to the behavior as before in v11a? is this correct?

[david.domask]

Hi @b@bct44,

Well, it could be related to a known issue that appeared due to the addition of True PerMachine backups; the locking mechanism changed slightly, so if you're on v12 and have upgraded your backup chains, you might be affected by that.

Start with the option and see if it helps.

If it doesn't, if you're on v12 Cumulative Patch 2, try this registry value: (PM team, comes from issue 490040)
Cumulative Patch link: https://www.veeam.com/kb4420

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Name: TapeBackupLocksResolution
Type: DWORD
Value: 0

[bct44]

Hello @David,

I'm already in true-per vm because it comes Windows agents, currently in v12 CP2.
I will try the suggested solutions, i would give a feedback.

[bct44]

First Update,

"Prevent this job from being interrupted" puts on hold the source jobs even it's an incremental. It's not an attended behavior like it was in v11a, i will try the registry key.

@dima, Could you confirm there is un update from the engine behavior plz?

[bct44]

Hello,

Registry key has corrected the wrong behavior.

[david.domask]

Very glad to hear it @bct44.