SOBR offload with v10 issues

[PeterC]

I have been testing with Veeam B&R v10 capacity tier, offloading to Azure Blob storage. We want to use this in our live backup environment. At the moment i don't have a case id because i would like to no if i am doing something wrong

I built a Veeam B&R 9.5 update 4b environment with 1 HyperV host, 1 B&R server and 2 SOBR's (ReFS - Windows 2016).
I created 5 VM's and 2 backup jobs. 1 job contains 3 VM's and has 7 restorepoint retention, the other 1 VM and 14 restorepoints retention.
All jobs are forever forward incrementals, with no synthetic or periodic active fulls, also no healthcheck options.

I ran these jobs several times, all worked fine.

I then upgraded this to v10 RTM, added Azure blob storage and added this as capacity tier to the primary SOBR.
I checked the option "Copy backups to object storage as soon as they are created ".

The first SOBR offload job uploaded all existing *.vib and *.vbk files to the Azure Blob storage.
The first backupjob ran fine, after that 2 Offload jobs started, one for the *.vib files, the next one (after the merge was ready) for the *.vbk files.
What caught my attention is that the *.vib offload was only around 500 MB transferred, the *.vbk files was 24 GB!
The next run showed the same, it looks like the complete *.vbk is offloaded, not only the changed blocks!

In the Veeam B&R 10 HyperV guide it says;

Once the backup (or backup copy) job is complete, Veeam Backup & Replication initiates a new copy session which simply extracts data blocks and metadata from each new backup file (.vbk, .vib, .vrb) created on any of the extents of your scale-out backup repository and copies these blocks to object storage, thereby making an identical replica of your backup data.

This morning i tested this with the VM's shut down. The backups run quickly, the offload off the *.vib files is only a few MB, but the *.vbk is again around 24 GB.
I ran the backup several times (with the VM' being shut down) and this happens every time.
I updated this morning with the Day 0 update to the v10 GA version, but this didn't change anything.

Is this normal behaviour? This would mean that in our production environment we would need to offload 140 TB every night!
What could be going wrong here, my assumption is that this can not be normal behaviour.

[HannesK]

Hello,

Is this normal behaviour?

no, that's not normal. Please open a support case to check that. I assume you checked your Azure disk space consumption?

Please do also post the case number here for reference.

Best regards,
Hannes

[veremin]

What caught my attention is that the *.vib offload was only around 500 MB transferred, the *.vbk files was 24 GB! The next run showed the same, it looks like the complete *.vbk is offloaded, not only the changed blocks!

Can you tell us how exactly you understood that all 24 GB were transferred again? Some visuals might help. Thanks!

[PeterC]

Hello,

no, that's not normal. Please open a support case to check that. I assume you checked your Azure disk space consumption?

Please do also post the case number here for reference.

Best regards,
Hannes

Am i able to open a case when using a test environment with a temporary license?

[Gostev]

If you mean trial license, then yes.

[PeterC]

What i see in the job summary is the following (hope it is readable):

3/4/2020
Backupjob 1 (incremental)
Summary Data
Duration: 1:59 Processed: 66.7 GB (100%)
Processing rate:1 MB/s Read: 12.3 MB
Bottleneck: Source Transferred: 361.6 KB

BackupJob 1 Offload (*.vib)
Summary Data
Duration: 0:56 Processed: 63.6 MB (100%)
Processing rate:5 MB/s Read: 51.5 MB
Bottleneck: Source Transferred: 51.5 MB

BackupJob 1 Offload (*.vbk)
Summary Data
Duration: 17:55 Processed: 52 GB (100%)
Processing rate:24 MB/s Read: 23.1 GB
Bottleneck: Target Transferred: 23.1 GB

3/3/2020
Backupjob 1 (incremental)
Summary Data
Duration: 2:07 Processed: 66.7 GB (100%)
Processing rate:1 MB/s Read: 12.3 MB
Bottleneck: Source Transferred: 361.6 KB

BackupJob 1 Offload (*.vib)
Summary Data
Duration: 0:56 Processed: 63.6 MB (100%)
Processing rate:5 MB/s Read: 51.5 MB
Bottleneck: Source Transferred: 51.5 MB

BackupJob 1 Offload (*.vbk)
Summary Data
Duration: 19:05 Processed: 52 GB (100%)
Processing rate:24 MB/s Read: 23.1 GB
Bottleneck: Target Transferred: 23.1 GB

3/2/2020
Backupjob 1 (incremental)
Summary Data
Duration: 1:55 Processed: 66.7 GB (100%)
Processing rate:1 MB/s Read: 12.3 MB
Bottleneck: Source Transferred: 361.6 KB

BackupJob 1 Offload (*.vib)
Summary Data
Duration: 0:57 Processed: 63.6 MB (100%)
Processing rate:5 MB/s Read: 51.5 MB
Bottleneck: Source Transferred: 51.5 MB

BackupJob 1 Offload (*.vbk)
Summary Data
Duration: 18:10 Processed: 52.2 GB (100%)
Processing rate:24 MB/s Read: 23.2 GB
Bottleneck: Target Transferred: 23.2 GB

This is the output from 3 days, take in mind that the VM's that are in the backup are all shut down.
You can see that everytime the offload of the *.vbk is around 23 GB and takes quite a bit of time (the upload is capped at 250 Mbps).

[PeterC]

Case # 04045067

[HannesK]

thanks for the case number. can you please upload logs? I already talked to support to watch out for your case as it might be a bug. But without logs, they cannot do anything.

[PeterC]

I have just uploaded the logfiles.

[PeterC]

Just got back the following answer from our support engineer;

I managed to receive an answer from QA team. What you see is actual design. Due to the fact that is a forever forward incremental job - each time your full file changes, and Veeam offloads changed file again. That was just not yet mentioned in user guide, which I already reported.

To solve it - enable periodic full backups, either Active or Synth.

Is this really the issue? This would totally be no option for us, this could mean doubling our backup storage. And as soon as there would be a active or synthetic full backup it would offload 120 TB of new backup files to Azure blob.

[Gostev]

Not necessarily. Enabling synthetic fulls with ReFS or XFS backup repositories requires no extra storage.

That said, I disagree with the QA engineer opinion that "this is normal, just undocumented". Only the new metadata of the updated full backup file needs to be offloaded, and also the index updated. There's literally no point to offload the actual payload, because all the required blocks are already there in the cloud. I will review internally.

[dalbertson]

I agree with @Gostev. Looking at the case i am not sure that the correct info was conveyed.

[HannesK]

Is this really the issue?

after some more investigations: it's not by design and we are working on it. support will give updates.

[Gostev]

To elaborate on this a bit further: your debug logs do show an incorrect offload behavior that is definitely not by design, so clearly we're dealing with some bug here. We are working on reproducing the issue internally now, and are planning to create the hotfix ASAP. Thanks for bringing this up, and not just settling on the incorrect assessment of this situation by one of our engineers.

[PeterC]

Thank you @Gostev and @HannesK for your open mind and having a closer look at this problem. My experiences with Veeam support continue to be very positive.
Hope to see what is causing this and looking forward to test with the fix.

[mecki]

Just as a remark, I see this behavior also on offloading to S3 (Wasabi) on copy mode with an incremental forward forever backup chain from Windows Agent Backups. All blocks of the full will be uploaded when the merged full has been changed. IMHO the trigger for that is the changed vbk file to upload every block. There is the same behavior with periodic fulls, if there is a new full all blocks will be uploaded not only the missing to cloud. But this could be by design if there is no logic in the meta data to get the files sorted out of the block structure. If you loose the meta data you will loose all backup data. It will be helpful to get the format of object storge explained a little bit.

CU.

Martin

[dalbertson]

Thanks Martin,

Could you open a case as well so that we can track?

[veremin]

Currently we're investigating the issue with full backup being re-transferred to object storage in case of forever modes (forward forever incremental and reverse incremental ones). I will update the topic with the results of our findings.

That's said, it certainly would not harm to open your own support case; should help you to get a hot fix quickly, if the problem is confirmed by QA team.

Thanks!

[veremin]

The issue has been found and confirmed. Currently we're working on a hot fix. I will update the topic, once it's available. Thanks!

[veremin]

The fix is included in the v10 Cumulative Patch 1, so please install it to the solve the problem. Thanks!

[PeterC]

I have just installed the KB3127 update, the first tests look promising! Will inform you after the weekend.