Here's my setup:
- Veeam 9.5 Update 3
- VMWare ESX 6.0 Update 3
- HPE StoreOnce 5100 (w/1 Exp Tray - 96TB RAW) running v 3.16.5
- HPE MSL4048 w/2 LTO7 Drives (FC)
- HPE 3PAR 7200 2-Node (~176TB RAW Capacity)
- HPE Blades (BL460c G8 (13), BL460c G9 (1)) in c7000 chassis
- Blade Chassis Networking/Storage Backplane is HPE FlexFabric - 2x10GbE DAC to Core Networking, 4x8Gb FC to SAN Fabric Switches (2)
- Each Blade is connected through the same physical interconnects, and is zoned into SAN volumes required
- Backup Server - VM Instance on Cluster of Blades in HPE c7000
- Proxy To StoreOnce is Physical, Windows Server 2012 R2, BL460 G9, is also used for Storage Snapshot backups from 3PAR 7200. One port from each of the MSL4048 drives are zoned to this system.
I create a Backups to Tape Job to duplicate the contents of a repository that contains the backups copies I ran at the end of 2017, and it runs for a very long time (20+ hours), until it ultimately fails with the following types of messages, and the files involved are not the same for each failure.
Code: Select all
4/11/2018 2:29:38 AM :: Full backup: BackupSynthesizedStorageToTape failed OSCLT_ERR_SERVER_OFFLINE. Err: -1404 SeekToRead operation has failed. Object: '[storeonce-wpg-backup.notmydomain.ca] VeeamBackups:/Duplicate_Fileservers_Local_1/BRDFS01.vm-299D2017-12-30T030000.vib'. Distance to move: '4803975168'. Unable to retrieve next block transmission command. Number of already processed blocks: . OSCLT_ERR_SERVER_OFFLINE. Err: -1404
Code: Select all
4/11/2018 8:02:13 AM :: Full backup: BackupSynthesizedStorageToTape failed OSCLT_ERR_SERVER_OFFLINE. Err: -1404 Failed to read data to the object '[storeonce-wpg-backup.notmydomain.ca] VeeamBackups:/Duplicate_SharePoint_2013_Prod_Backups_Local/brdsp-sql01p.vm-44492D2018-01-01T000000_f8fc4fa8-a7a1-47bb-8637-ed395c51360a.vsb'. Unable to retrieve next block transmission command. Number of already processed blocks: . OSCLT_ERR_SERVER_OFFLINE. Err: -1404
I have had tickets open with both Veeam support and with HPE, and I have received no helpful advice. Both would suggest that I have some type of networking issue, but I disagree.
I have been separately able to successfully restore the files the tape job was experiencing errors on without issue, to a volume presented on the same Veeam Proxy/Gateway that's being used on the Backups to Tape job. There are no errors being reported on any of the network or FC ports involved either. All of the devices involved are connected via the same backplanes, and through either 10GbE or FC connections.
I am at a loss as to what I should try next. The total amount of data that I'm attempting to get to tape is about 45TB, some machines are very large (6+TB) as they have several 2TB volumes attached. I realize this isn't best practice, but it's mostly what I've inherited and not much I can do in my current position to change it.
Any help out there? Anyone with the same issue writing out large files to tape from a dedupe appliance?