HPE StoreOnce Catalyst Backup Copy Errors

[KelKnowles]

Hello,

As both an HPE and Veeam reseller I have been experiencing issues with various customers that run backup copy jobs to Catalyst stores on HPE StoreOnce. These problems are on 9.0 Update 2 as well as 9.5 of the Veeam Backup and Replication software. I have several installs with different types of appliances most of which are running version 3.15.1 firmware on StoreOnce 4500 and 3540 physical appliance as well as 3.13.3 for the VSA virtual appliance.

Due to networking issues or an as yet unresolved issue intermittently during a backup copy job occasionally one of the two timeout messages appears:

Error: ChannelError: TimedOut
OSCLT_ERR_SERVER_OFFLINE. Err: -1404

This is then followed up by a string repeated file lock messages:

OSCLT_ERR_OBJECT_LOCK_FAILED. Err: -1909

And these messages occasionally when there is a subsequent synthetic full job that is attempting to work on files created with these lock errors:

Failed to merge full backup file Error: Agent: Failed to process method {Transform.Patch}: There is an item with the specified name and another type.
Failed to generate points Error: Agent: Failed to process method {Transform.Patch}: There is an item with the specified name and another type.

HPE has told me that the locking of files on StoreOnce is supposed to be performed by the Veeam software. I would like to know if there would be a way to improve the software to recover from these file locks and complete successfully.
Also are other people seeing these errors? Some of these cases are 3 months old and were started on a previous version of the StoreOnce firmware. Due to the fact that it is showing up for My contention is that Veeam should be able to release file locks from previous failed backup sessions and recover from intermittent issues.

Also I am requesting that resources at Veeam be utilized to attempt to reproduce these errors and specifically the OSCLT_ERR_OBJECT_LOCK_FAILED. Err: -1909 error although it may be a symptom of a loss of connection it is specifically the fact that Veeam is not releasing the locks from previous sessions and relocking the files that I am attempting to shed some light on.

HPE Case number:
5311286838
Veeam Case numbers:
01902465 Backup copy: Fail to merge full backup
01907173 Error in Backup Copy to StoreOnce Catalyst
02017195 OSCLT_ERR_SERVER_OFFLINE
02017188 Failed to merge full backup file error
02023087 Errors when using backup copy to Catalyst

Anyone else with similar issues?

[foggy]

Hi Kelly, the behavior seems expected in case of networking issues, since Veeam B&R cannot close the file (and release lock) in case connection is dropped. File will be automatically released after several hours (storage timeout).

[KelKnowles]

Backup copy jobs that rely on these files to be created cannot simply wait hours for the files to unlock. The jobs will retry many times within a much shorter time and then just ultimately fail leaving an incomplete backup or worse yet a corrupted restore point. I am asking to see if it would be possible to have some better error recovery built into the software so that it can actually heal these locked files. Since the file was only ever touched by Veeam then Veeam should also be able to release the lock on the file if it drops a network connection and is able to reconnect. If this is not possible then Veeam should be able to create an alternative file for the incremental and use that instead of the old file name and at least prevent the incremental chain from being broken.

I know this seems to happen more on WAN connections but I have seen this on a LAN with no apparent bottleneck. Is it reasonable to expect that a backup copy job which technically can run 24/7 will never have an interruption in its connection?

[foggy]

As you've mentioned, it performs a number of retries, so is able to resist minor network failures. However, in your case those seem to take longer than the period of retries, causing the job to fail. This, however, doesn't result in backup files corruption, you will not be able to restore from the latest restore point, but all previous ones are still ok and the next job run will repair the chain.

I'm not sure there's an ability to recognize and release these locks, will discuss with R&D once they get back from the winter holidays.

[KelKnowles]

The drop in connectivity can be for only a few seconds which should be recoverable during the normal retries that Veeam B&R performs. The retries do successfully reestablish a connection to the StoreOnce appliance usually within one minute but the file lock on the StoreOnce prevents Veeam from doing anything with the files since they were locked from the previous session. I have several examples where if the backup copies are left on their own then forever forward incremental chain will not heal itself and the copies will just continuously fail unless a user manually initiates an active full. This also will break the weekly archive full creation process which prevents automatic GFS archives from being created reliably.

[Arnfinn]

This looks like what im struggeling with, in yesterdays backup copy jobs i had one job giving me a "OSCLT_ERR_OBJECT_LOCK_FAILED. Err: -1909" error when a job running in the same time did not report any problems, so im not sure this is only network related.

[Dschano]

Hi,

i am facing the same erros with my backup jobs (not backup copy, catalyst over fibre channel), also for newly created catalyst stores with no files that could be locked.

Veeam case id 02028650

veeam backup-server (physical) is working as the gateway-server located in the same server room as the storage and storeonce.

anyone else with this problem?

[Chris.B]

We regularly received OSCLT_ERR_SERVER_OFFLINE and similar errors from both our StoreOnce systems (backup to primary then copy to secondary) until they were updated.
We're currently on 3.15.1-1636.1 which seems to have resolved the issues.

Chris

[bulgie]

I am getting a couple of the same errors mentioned by the OP:
* Failed to merge full backup file Error: Agent: Failed to process method {Transform.Patch}: There is an item with the specified name and another type.
* Failed to generate points Error: Agent: Failed to process method {Transform.Patch}: There is an item with the specified name and another type.

But this is not with an HPE device. My repository is a Windows EC2 instance (server in Amazon cloud accessed via VPN).

It has failed for a few days now, not correcting itself. I believe the first errors were:
* Failed to merge full backup file Error: The process cannot access the file because it is being used by another process. Failed to open file [<path>\xxx.vbk] in readonly mode. Agent failed to process method {Stg.OpenRead}.
* Failed to generate points Error: The process cannot access the file because it is being used by another process. Failed to open file [<path>\xxx.vbk] in readonly mode. Agent failed to process method {Stg.OpenRead}.

Then the next day:
* Failed to merge full backup file Error: An existing connection was forcibly closed by the remote host
* Failed to create restore point <date> <time>
* Failed to generate points Error: An existing connection was forcibly closed by the remote host

Then each day since then, the two errors listed at the beginning of this post.

Veeam case # 02033654

-Mark

[foggy]

Let's not derail the thread with issues that just look similar. I appreciate you having opened support cases for each of them, so let our technical team deal with them.

As for the original OSCLT_ERR_OBJECT_LOCK_FAILED issue with StoreOnce, then the fact is that currently the file locked within one connection cannot be reopened in another one. Indeed, Veeam B&R is able to reconnect, but the files that were not closed in previous connection are locked and cannot be opened. We will look into this deeper for possible workarounds.

[Dschano]

technical support closed my case with the statement "no possible workaround at the moment" which is not really cool.

Veeam B&R 9.5
Storeonce 3.15.1-1636.1

Error: OSCLT_ERR_SERVER_OFFLINE. Err: -1404

Backup via Catalyst over Network works fine, only Catalyst over Fibre Channel is not working

[foggy]

We regularly received OSCLT_ERR_SERVER_OFFLINE and similar errors from both our StoreOnce systems (backup to primary then copy to secondary) until they were updated.
We're currently on 3.15.1-1636.1 which seems to have resolved the issues.

I'd presume you are using Catalyst over network, then this can be explained by the fact that in StoreOnce 3.15.1 the timeout prior to closing the connection (and releasing the lock) was reduced to 20 minutes. In case of FC, it still lasts for several hours, though HPE have plans to reduce it in some of the future versions.

So, indeed, looks like there're no workarounds at the moment, other than addressing connectivity issues causing the discussed behavior.

[oberhofer]

Hello!

I can confirm: same problems here with veeam 9.0.x and 9.5 and HP StoreOnce 4500 (only FC-Catalyst-Stores) until last week, when we updated the StoreOnce-Aplliance to version 3.15.1-1636.1.

From this moment until now, no repository related errors (specified in the entry-post) occured again. So the error seems to be fixed in this software-release.

[oberhofer]

Supplement:

Parallel to my veeam-case (Case # 02031171) I opened a case with HP. I got following statement of a level-2 engineer related to "Random backup failures on Veeam Backups with the following error: OSCLT_ERR_SERVER_OFFLINE. Err: -1404:"

HP: "The backup failures seen are a known issue that is fixed in 3.15.1 , so the recommendation is to upgrade the device firmware to the latest – 3.15.1 ." ...what we did last week. And it worked.

[foggy]

Thanks, Kai, much appreciated. Failures in your case, though, seem to be caused by some other (StoreOnce related) issue, not connectivity problems causing OSCLT_ERR_SERVER_OFFLINE (and further files locking) as in the original post.

[davidburke84]

Supplement:

Parallel to my veeam-case (Case # 02031171) I opened a case with HP. I got following statement of a level-2 engineer related to "Random backup failures on Veeam Backups with the following error: OSCLT_ERR_SERVER_OFFLINE. Err: -1404:"

HP: "The backup failures seen are a known issue that is fixed in 3.15.1 , so the recommendation is to upgrade the device firmware to the latest – 3.15.1 ." ...what we did last week. And it worked.

Kai - how often are you getting the OSCLT error? I seem to get it maybe once every other week.

David

[oberhofer]

Hi, David. More often. I think average one time a day. Sometimes more. And this for at least 2-3 months. Now I have for 4 days clean logs. That's why I'm thinking, the issue is fixed. For your information: It concerned not only the copy-to-tape jobs, also backups to the storeonce, but not so often. There were not only the "OSCLT_ERR_SERVER_OFFLINE. Err: -1404" - failure, I found also things like:
- Error: The remote procedure call was cancelled RPC function call failed. Function name: [DoRpc]. Target machine: [IP xxx] or
- Backup file xxx.vbk will be excluded from the list of files to backup, because its extent <Repostitoryname xxx> is unavailable
- Unable to backup: Failed to set credentials.
Failed to call RPC function 'PluginsHost.StoreOnceSetCreds': The remote procedure call was cancelled. RPC function call failed. Function name: [DoRpc]. Target machine: [IP xxx].
The remote procedure call was cancelled

[richardw.wex]

Hi Kel

I'm getting the OSCLT_ERR_SERVER_OFFLINE error a lot from veeam 9.0 and storeonce 4500's firmwre 3.13.
This has been going on for a while. i had a case open with HP several months ago. They didnt help much.
3.13 was the current FW at the time.

We have 4 StoreOnce 4500's which are shared with HP dataProtector. We have mostly worked around the issue by distributing the jobs to different timeslots as it seems too many jobs running simultaneous;y is the cause of OSCLT_ERR_SERVER_OFFLINE.

This happens with backup jobs as well as backup copy jobs. HPDP gets similar errors

regards

Richard

[foggy]

Richard, looks like the storage firmware issue, haven't you thought of upgrading to the later one?