HPE StoreOnce Catalyst Backup Copy Errors

Availability for the Always-On Enterprise

HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby KelKnowles » Wed Dec 28, 2016 2:36 pm

Hello,

As both an HPE and Veeam reseller I have been experiencing issues with various customers that run backup copy jobs to Catalyst stores on HPE StoreOnce. These problems are on 9.0 Update 2 as well as 9.5 of the Veeam Backup and Replication software. I have several installs with different types of appliances most of which are running version 3.15.1 firmware on StoreOnce 4500 and 3540 physical appliance as well as 3.13.3 for the VSA virtual appliance.

Due to networking issues or an as yet unresolved issue intermittently during a backup copy job occasionally one of the two timeout messages appears:

Error: ChannelError: TimedOut
OSCLT_ERR_SERVER_OFFLINE. Err: -1404

This is then followed up by a string repeated file lock messages:

OSCLT_ERR_OBJECT_LOCK_FAILED. Err: -1909

And these messages occasionally when there is a subsequent synthetic full job that is attempting to work on files created with these lock errors:

Failed to merge full backup file Error: Agent: Failed to process method {Transform.Patch}: There is an item with the specified name and another type.
Failed to generate points Error: Agent: Failed to process method {Transform.Patch}: There is an item with the specified name and another type.

HPE has told me that the locking of files on StoreOnce is supposed to be performed by the Veeam software. I would like to know if there would be a way to improve the software to recover from these file locks and complete successfully.
Also are other people seeing these errors? Some of these cases are 3 months old and were started on a previous version of the StoreOnce firmware. Due to the fact that it is showing up for My contention is that Veeam should be able to release file locks from previous failed backup sessions and recover from intermittent issues.

Also I am requesting that resources at Veeam be utilized to attempt to reproduce these errors and specifically the OSCLT_ERR_OBJECT_LOCK_FAILED. Err: -1909 error although it may be a symptom of a loss of connection it is specifically the fact that Veeam is not releasing the locks from previous sessions and relocking the files that I am attempting to shed some light on.

HPE Case number:
5311286838
Veeam Case numbers:
01902465 Backup copy: Fail to merge full backup
01907173 Error in Backup Copy to StoreOnce Catalyst
02017195 OSCLT_ERR_SERVER_OFFLINE
02017188 Failed to merge full backup file error
02023087 Errors when using backup copy to Catalyst

Anyone else with similar issues?
KelKnowles
Influencer
 
Posts: 14
Liked: never
Joined: Sat Sep 15, 2012 8:01 pm
Full Name: Kelly Knowles

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby foggy » Thu Dec 29, 2016 4:03 pm

Hi Kelly, the behavior seems expected in case of networking issues, since Veeam B&R cannot close the file (and release lock) in case connection is dropped. File will be automatically released after several hours (storage timeout).
foggy
Veeam Software
 
Posts: 14746
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby KelKnowles » Tue Jan 03, 2017 1:26 pm

Backup copy jobs that rely on these files to be created cannot simply wait hours for the files to unlock. The jobs will retry many times within a much shorter time and then just ultimately fail leaving an incomplete backup or worse yet a corrupted restore point. I am asking to see if it would be possible to have some better error recovery built into the software so that it can actually heal these locked files. Since the file was only ever touched by Veeam then Veeam should also be able to release the lock on the file if it drops a network connection and is able to reconnect. If this is not possible then Veeam should be able to create an alternative file for the incremental and use that instead of the old file name and at least prevent the incremental chain from being broken.

I know this seems to happen more on WAN connections but I have seen this on a LAN with no apparent bottleneck. Is it reasonable to expect that a backup copy job which technically can run 24/7 will never have an interruption in its connection?
KelKnowles
Influencer
 
Posts: 14
Liked: never
Joined: Sat Sep 15, 2012 8:01 pm
Full Name: Kelly Knowles

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby foggy » Tue Jan 03, 2017 2:24 pm

As you've mentioned, it performs a number of retries, so is able to resist minor network failures. However, in your case those seem to take longer than the period of retries, causing the job to fail. This, however, doesn't result in backup files corruption, you will not be able to restore from the latest restore point, but all previous ones are still ok and the next job run will repair the chain.

I'm not sure there's an ability to recognize and release these locks, will discuss with R&D once they get back from the winter holidays.
foggy
Veeam Software
 
Posts: 14746
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby KelKnowles » Wed Jan 04, 2017 3:05 pm

The drop in connectivity can be for only a few seconds which should be recoverable during the normal retries that Veeam B&R performs. The retries do successfully reestablish a connection to the StoreOnce appliance usually within one minute but the file lock on the StoreOnce prevents Veeam from doing anything with the files since they were locked from the previous session. I have several examples where if the backup copies are left on their own then forever forward incremental chain will not heal itself and the copies will just continuously fail unless a user manually initiates an active full. This also will break the weekly archive full creation process which prevents automatic GFS archives from being created reliably.
KelKnowles
Influencer
 
Posts: 14
Liked: never
Joined: Sat Sep 15, 2012 8:01 pm
Full Name: Kelly Knowles

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby Arnfinn » Fri Jan 06, 2017 7:28 am

This looks like what im struggeling with, in yesterdays backup copy jobs i had one job giving me a "OSCLT_ERR_OBJECT_LOCK_FAILED. Err: -1909" error when a job running in the same time did not report any problems, so im not sure this is only network related.
Arnfinn
Lurker
 
Posts: 2
Liked: never
Joined: Thu Jun 16, 2016 8:24 am
Full Name: Arnfinn H Grønberg

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby Dschano » Mon Jan 09, 2017 2:15 pm

Hi,

i am facing the same erros with my backup jobs (not backup copy, catalyst over fibre channel), also for newly created catalyst stores with no files that could be locked.

Veeam case id 02028650

veeam backup-server (physical) is working as the gateway-server located in the same server room as the storage and storeonce.

anyone else with this problem?
Dschano
Novice
 
Posts: 3
Liked: never
Joined: Mon Dec 19, 2016 10:13 am

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby Chris.B » Mon Jan 09, 2017 5:17 pm

We regularly received OSCLT_ERR_SERVER_OFFLINE and similar errors from both our StoreOnce systems (backup to primary then copy to secondary) until they were updated.
We're currently on 3.15.1-1636.1 which seems to have resolved the issues.

Chris
Chris.B
Lurker
 
Posts: 2
Liked: 1 time
Joined: Mon Jan 09, 2017 10:20 am
Location: UK
Full Name: Chris B

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby bulgie » Tue Jan 10, 2017 3:56 am

I am getting a couple of the same errors mentioned by the OP:
* Failed to merge full backup file Error: Agent: Failed to process method {Transform.Patch}: There is an item with the specified name and another type.
* Failed to generate points Error: Agent: Failed to process method {Transform.Patch}: There is an item with the specified name and another type.

But this is not with an HPE device. My repository is a Windows EC2 instance (server in Amazon cloud accessed via VPN).

It has failed for a few days now, not correcting itself. I believe the first errors were:
* Failed to merge full backup file Error: The process cannot access the file because it is being used by another process. Failed to open file [<path>\xxx.vbk] in readonly mode. Agent failed to process method {Stg.OpenRead}.
* Failed to generate points Error: The process cannot access the file because it is being used by another process. Failed to open file [<path>\xxx.vbk] in readonly mode. Agent failed to process method {Stg.OpenRead}.

Then the next day:
* Failed to merge full backup file Error: An existing connection was forcibly closed by the remote host
* Failed to create restore point <date> <time>
* Failed to generate points Error: An existing connection was forcibly closed by the remote host

Then each day since then, the two errors listed at the beginning of this post.

Veeam case # 02033654

-Mark
bulgie
Novice
 
Posts: 4
Liked: never
Joined: Thu Mar 24, 2011 5:14 pm
Full Name: Mark Bulgier

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby foggy » Tue Jan 10, 2017 4:22 pm

Let's not derail the thread with issues that just look similar. I appreciate you having opened support cases for each of them, so let our technical team deal with them.

As for the original OSCLT_ERR_OBJECT_LOCK_FAILED issue with StoreOnce, then the fact is that currently the file locked within one connection cannot be reopened in another one. Indeed, Veeam B&R is able to reconnect, but the files that were not closed in previous connection are locked and cannot be opened. We will look into this deeper for possible workarounds.
foggy
Veeam Software
 
Posts: 14746
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby Dschano » Wed Jan 11, 2017 2:13 pm

technical support closed my case with the statement "no possible workaround at the moment" which is not really cool.

Veeam B&R 9.5
Storeonce 3.15.1-1636.1

Error: OSCLT_ERR_SERVER_OFFLINE. Err: -1404

Backup via Catalyst over Network works fine, only Catalyst over Fibre Channel is not working
Dschano
Novice
 
Posts: 3
Liked: never
Joined: Mon Dec 19, 2016 10:13 am

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby foggy » Fri Jan 13, 2017 2:54 pm

Chris.B wrote:We regularly received OSCLT_ERR_SERVER_OFFLINE and similar errors from both our StoreOnce systems (backup to primary then copy to secondary) until they were updated.
We're currently on 3.15.1-1636.1 which seems to have resolved the issues.

I'd presume you are using Catalyst over network, then this can be explained by the fact that in StoreOnce 3.15.1 the timeout prior to closing the connection (and releasing the lock) was reduced to 20 minutes. In case of FC, it still lasts for several hours, though HPE have plans to reduce it in some of the future versions.

So, indeed, looks like there're no workarounds at the moment, other than addressing connectivity issues causing the discussed behavior.
foggy
Veeam Software
 
Posts: 14746
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby oberhofer » Mon Jan 16, 2017 9:11 am 1 person likes this post

Hello!

I can confirm: same problems here with veeam 9.0.x and 9.5 and HP StoreOnce 4500 (only FC-Catalyst-Stores) until last week, when we updated the StoreOnce-Aplliance to version 3.15.1-1636.1.

From this moment until now, no repository related errors (specified in the entry-post) occured again. So the error seems to be fixed in this software-release.
oberhofer
Novice
 
Posts: 3
Liked: 1 time
Joined: Mon Apr 18, 2016 9:54 am
Full Name: Kai

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby oberhofer » Mon Jan 16, 2017 9:33 am

Supplement:

Parallel to my veeam-case (Case # 02031171) I opened a case with HP. I got following statement of a level-2 engineer related to "Random backup failures on Veeam Backups with the following error: OSCLT_ERR_SERVER_OFFLINE. Err: -1404:"

HP: "The backup failures seen are a known issue that is fixed in 3.15.1 , so the recommendation is to upgrade the device firmware to the latest – 3.15.1 ." ...what we did last week. And it worked.
oberhofer
Novice
 
Posts: 3
Liked: 1 time
Joined: Mon Apr 18, 2016 9:54 am
Full Name: Kai

Re: HPE StoreOnce Catalyst Backup Copy Errors

Veeam Logoby foggy » Mon Jan 16, 2017 11:55 am

Thanks, Kai, much appreciated. Failures in your case, though, seem to be caused by some other (StoreOnce related) issue, not connectivity problems causing OSCLT_ERR_SERVER_OFFLINE (and further files locking) as in the original post.
foggy
Veeam Software
 
Posts: 14746
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: bspeerly and 58 guests