Backups to Tape Fail - Source is HPE StoreOnce 5100

[davemann]

I have been pulling my hair out trying to get some backup copies that I created at the end of 2017 off to tape.

Here's my setup:

• Veeam 9.5 Update 3
• VMWare ESX 6.0 Update 3
• HPE StoreOnce 5100 (w/1 Exp Tray - 96TB RAW) running v 3.16.5
• HPE MSL4048 w/2 LTO7 Drives (FC)
• HPE 3PAR 7200 2-Node (~176TB RAW Capacity)
• HPE Blades (BL460c G8 (13), BL460c G9 (1)) in c7000 chassis
• Blade Chassis Networking/Storage Backplane is HPE FlexFabric - 2x10GbE DAC to Core Networking, 4x8Gb FC to SAN Fabric Switches (2)
• Each Blade is connected through the same physical interconnects, and is zoned into SAN volumes required
• Backup Server - VM Instance on Cluster of Blades in HPE c7000
• Proxy To StoreOnce is Physical, Windows Server 2012 R2, BL460 G9, is also used for Storage Snapshot backups from 3PAR 7200. One port from each of the MSL4048 drives are zoned to this system.

My issue...

I create a Backups to Tape Job to duplicate the contents of a repository that contains the backups copies I ran at the end of 2017, and it runs for a very long time (20+ hours), until it ultimately fails with the following types of messages, and the files involved are not the same for each failure.

Code: Select all

4/11/2018 2:29:38 AM :: Full backup: BackupSynthesizedStorageToTape failed
OSCLT_ERR_SERVER_OFFLINE. Err: -1404
SeekToRead operation has failed. Object: '[storeonce-wpg-backup.notmydomain.ca] VeeamBackups:/Duplicate_Fileservers_Local_1/BRDFS01.vm-299D2017-12-30T030000.vib'. Distance to move: '4803975168'.
Unable to retrieve next block transmission command. Number of already processed blocks: [1070868].

OSCLT_ERR_SERVER_OFFLINE. Err: -1404  

Code: Select all

4/11/2018 8:02:13 AM :: Full backup: BackupSynthesizedStorageToTape failed
OSCLT_ERR_SERVER_OFFLINE. Err: -1404
Failed to read data to the object '[storeonce-wpg-backup.notmydomain.ca] VeeamBackups:/Duplicate_SharePoint_2013_Prod_Backups_Local/brdsp-sql01p.vm-44492D2018-01-01T000000_f8fc4fa8-a7a1-47bb-8637-ed395c51360a.vsb'.
Unable to retrieve next block transmission command. Number of already processed blocks: [977023].

OSCLT_ERR_SERVER_OFFLINE. Err: -1404  

I have restarted the Proxy/Gateway, Tape Library, StoreOnce units, etc... Run the Tape job several times, and some times it will manage to copy one of the files that previously failed, but ultimately fail again on another large file.

I have had tickets open with both Veeam support and with HPE, and I have received no helpful advice. Both would suggest that I have some type of networking issue, but I disagree.

I have been separately able to successfully restore the files the tape job was experiencing errors on without issue, to a volume presented on the same Veeam Proxy/Gateway that's being used on the Backups to Tape job. There are no errors being reported on any of the network or FC ports involved either. All of the devices involved are connected via the same backplanes, and through either 10GbE or FC connections.

I am at a loss as to what I should try next. The total amount of data that I'm attempting to get to tape is about 45TB, some machines are very large (6+TB) as they have several 2TB volumes attached. I realize this isn't best practice, but it's mostly what I've inherited and not much I can do in my current position to change it.

Any help out there? Anyone with the same issue writing out large files to tape from a dedupe appliance?

[nielsengelen]

I would suggest contacting support for this as there may be some tweaking that can be done to assist you here.

[csydas]

Hey Dave,

Just a thought, I know that we demoed a Storeonce VSA and we had a helluva time with the Virtual Fulls. I can see your jobs are doing the Virtual Fulls for tape (the vsb), but maybe you can try to schedule it so that it's only getting the Fulls and avoiding the synthetic operations. If I had a guess, the Storeonce just can't handle the pointer stuff with Virtual Fulls.

[jylb44]

Hello Dave,

Did you find a solution to this error ?

Thanks

Jeev

[PatrickT]

Hi,

I can confirm this. I have a customer environment that experiences the same issues:

- vSphere 6.5 U2
- StoreOnce 5100 with 3.16.7-1837.1
- Veeam B&R 9.5 U4b

Code: Select all

[08.12.2019 07:25:38] <14> Warning  [CTapeChainBackupUnit] Set completed with error for chain: Full:vm:z_Temp, task:Backup PROD - Daily
[08.12.2019 07:25:38] <14> Warning  [CTapeChainBackupUnit] OSCLT_ERR_SERVER_OFFLINE. Err: -1404
[08.12.2019 07:25:38] <14> Warning  Failed to read data to the object '[COFC-SO5100] Veeam:/Backup_PROD_-_Daily/servername.vm-101D2019-12-05T170347_DDE1_73cfc46c-9421-439c-8a41-66f0b1169332.vsb'.
[08.12.2019 07:25:38] <14> Warning  Unable to retrieve next block transmission command. Number of already processed blocks: [827402].
[08.12.2019 07:25:38] <14> Warning  OSCLT_ERR_SERVER_OFFLINE. Err: -1404

Pretty interesting... Veeam Support suggests a FC issue and/ or colliding Veeam Backup/ Copy-2-Tape Jobs. Bust the last crash (last weekend) clearly shows that this can't be the root cause:

- no concurrent jobs
- FC ports are fine and doesn't show any errors.

[Dima P.]

Hello PatrickT,

Can you please share your case ID so we could review thew case details? Thank you in advance!