Backups to Tape Fail - Source is HPE StoreOnce 5100

Discussions specific to tape backups

Backups to Tape Fail - Source is HPE StoreOnce 5100

Veeam Logoby davemann » Wed Apr 11, 2018 8:59 pm

I have been pulling my hair out trying to get some backup copies that I created at the end of 2017 off to tape.

Here's my setup:

  • Veeam 9.5 Update 3
  • VMWare ESX 6.0 Update 3
  • HPE StoreOnce 5100 (w/1 Exp Tray - 96TB RAW) running v 3.16.5
  • HPE MSL4048 w/2 LTO7 Drives (FC)
  • HPE 3PAR 7200 2-Node (~176TB RAW Capacity)
  • HPE Blades (BL460c G8 (13), BL460c G9 (1)) in c7000 chassis
  • Blade Chassis Networking/Storage Backplane is HPE FlexFabric - 2x10GbE DAC to Core Networking, 4x8Gb FC to SAN Fabric Switches (2)
  • Each Blade is connected through the same physical interconnects, and is zoned into SAN volumes required
  • Backup Server - VM Instance on Cluster of Blades in HPE c7000
  • Proxy To StoreOnce is Physical, Windows Server 2012 R2, BL460 G9, is also used for Storage Snapshot backups from 3PAR 7200. One port from each of the MSL4048 drives are zoned to this system.

My issue...

I create a Backups to Tape Job to duplicate the contents of a repository that contains the backups copies I ran at the end of 2017, and it runs for a very long time (20+ hours), until it ultimately fails with the following types of messages, and the files involved are not the same for each failure.

Code: Select all
4/11/2018 2:29:38 AM :: Full backup: BackupSynthesizedStorageToTape failed
OSCLT_ERR_SERVER_OFFLINE. Err: -1404
SeekToRead operation has failed. Object: '[storeonce-wpg-backup.notmydomain.ca] VeeamBackups:/Duplicate_Fileservers_Local_1/BRDFS01.vm-299D2017-12-30T030000.vib'. Distance to move: '4803975168'.
Unable to retrieve next block transmission command. Number of already processed blocks: [1070868].

OSCLT_ERR_SERVER_OFFLINE. Err: -1404 

Code: Select all
4/11/2018 8:02:13 AM :: Full backup: BackupSynthesizedStorageToTape failed
OSCLT_ERR_SERVER_OFFLINE. Err: -1404
Failed to read data to the object '[storeonce-wpg-backup.notmydomain.ca] VeeamBackups:/Duplicate_SharePoint_2013_Prod_Backups_Local/brdsp-sql01p.vm-44492D2018-01-01T000000_f8fc4fa8-a7a1-47bb-8637-ed395c51360a.vsb'.
Unable to retrieve next block transmission command. Number of already processed blocks: [977023].

OSCLT_ERR_SERVER_OFFLINE. Err: -1404 


I have restarted the Proxy/Gateway, Tape Library, StoreOnce units, etc... Run the Tape job several times, and some times it will manage to copy one of the files that previously failed, but ultimately fail again on another large file.

I have had tickets open with both Veeam support and with HPE, and I have received no helpful advice. Both would suggest that I have some type of networking issue, but I disagree.

I have been separately able to successfully restore the files the tape job was experiencing errors on without issue, to a volume presented on the same Veeam Proxy/Gateway that's being used on the Backups to Tape job. There are no errors being reported on any of the network or FC ports involved either. All of the devices involved are connected via the same backplanes, and through either 10GbE or FC connections.

I am at a loss as to what I should try next. The total amount of data that I'm attempting to get to tape is about 45TB, some machines are very large (6+TB) as they have several 2TB volumes attached. I realize this isn't best practice, but it's mostly what I've inherited and not much I can do in my current position to change it.

Any help out there? Anyone with the same issue writing out large files to tape from a dedupe appliance?
davemann
Novice
 
Posts: 3
Liked: 2 times
Joined: Fri Jul 15, 2016 8:40 pm
Full Name: Dave Reichert

Re: Backups to Tape Fail - Source is HPE StoreOnce 5100

Veeam Logoby vmniels » Thu Apr 12, 2018 8:10 am

I would suggest contacting support for this as there may be some tweaking that can be done to assist you here.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be
vmniels
Veeam Software
 
Posts: 1844
Liked: 398 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Backups to Tape Fail - Source is HPE StoreOnce 5100

Veeam Logoby csydas » Thu Apr 12, 2018 7:03 pm

Hey Dave,

Just a thought, I know that we demoed a Storeonce VSA and we had a helluva time with the Virtual Fulls. I can see your jobs are doing the Virtual Fulls for tape (the vsb), but maybe you can try to schedule it so that it's only getting the Fulls and avoiding the synthetic operations. If I had a guess, the Storeonce just can't handle the pointer stuff with Virtual Fulls.
csydas
Enthusiast
 
Posts: 70
Liked: 11 times
Joined: Tue Jan 16, 2018 5:14 pm
Full Name: Harvey Carel


Return to Tape



Who is online

Users browsing this forum: No registered users and 6 guests