Very slow VIBs, but fast VBKs. Extended periods of zero MB/s.

[chalkynz]

Recently reformatted/changed my storage holding my single repository, and am seeing a weird issue with GFS Daily Incrementals to tape - Veeam B&R seems to sit there for maybe 20 minutes or more, doing nothing, until the first activity to tape is seen. That activity will be a small burst of writing to tape, then it'll go idle again for ummm maybe 15 minutes or so, transfer rate drops to 0 MB/s. It's as if VBR is away trying to figure something out - like which VIBs are in scope, or something similar.

That daily incremental tape job used to take 3.5 hours, it now takes 19 or so! This doesn't happen with VBK full to tape (same GFS job, weekly) - they still run normal speed.

ReFS performance seems OK, as VIB copy jobs are normal speed, OS VIB file copies are normal speed, VBK full backup to tape is normal speed. The only way I can make it go slow is to run a VIB incremental backup in VBR.

Storage is 12 x drives presented to OS as JBOD. Changed from single Windows Dynamic RAID5 volume spanning all 12 drives formatted NTFS 4k blocks, to Windows Storage Spaces RAID5, ReFS 64k, thin-provisioned volume. Also enabled encryption on disk backup jobs, and set backup to disk block size to maximum. My 1 x LTO-7 tape drive has always been 1 MB block size.

Case ID is 03818765.

[HannesK]

Hello,
please understand that the forum does not replace support. So please continue working with them, as we can only guess here.

In general, your setup looks okay for me. The only thing I don't understand is "set backup to disk block size to maximum". Normally there is no reason to change the default block size. But anyway, this should not make tape that slow.

I did not read all case message: did you already do a tape performance test with the tape vendor tool (just to make sure that there were no driver chances that have negative impact)?

Best regards,
Hannes

[chalkynz]

Hi, no driver changes. VBK to tape is still fast, up to 291 MB/s. So it’s apparently not tape. On the incremental dailies, VBR just sits there, idle, no tape activity at all, zero MB/s, for 10 minutes, 15 minutes, quite odd. You can see in the logs when a job starts that VBR is maybe struggling to build a list of what to backup. I don’t get it.
I am working with support but sometimes they don’t call me for a day or more, despite it being a Sev 1. No idea what the SLA is, I believe that only covers initial contact.

[HannesK]

okay, good to hear about the speed with VBK

Normally they would reduce it from Sev 1 to Sev 3 or Sev 4 in your case (no impact on production or restores) . See https://www.veeam.com/veeam_software_su ... icy_ds.pdf for more details. Yes correct, the SLA is about it's initial contact.

[chalkynz]

On the contrary: massive impact on production, due to huge backup window, and elimination of restore window as a result.

Veeam response is that it is caused by ReFS. If I change back to NTFS then I lose my lovely fast merge. Urgh. Can't quite believe that the slow backup to tape with massive idle time is working as expected, but I am the only person to have experienced this.

[HannesK]

that makes no sense to me as there are no synthetic operations for a VIB-to-tape. let me check.

[chalkynz]

We're currently testing a hot fix for this.

Weird that our VBKs to tape are both fast (pushing max LTO speed currently), and quick to begin. VIBs to tape are fast, once they start, but take a long time to start, and we have significant delays (idle, 0 MB/s) after one VIB is pushed to tape, before the next one starts.