HPE StoreOnce Catalyst Repo Concurrent Tasks Limit

[david.buchanan]

Hi All,

Is there a best practice or a guideline as to how many tasks a StoreOnce VSA configured as a Catalyst repo can run?

Currently I have it configured with "Limit maximum concurrent tasks to" 12.

When I increase it I seem to run into errors like "maximum concurrent connections has been exceeded", however I dont know if this is Veeam or the VSA reporting this message.

Thanks,
David.

[foggy]

David, the recommended max number of concurrent tasks for repository residing on StoreOnce VSA running StoreOnce 3.15, I believe, is 8. The message you're seeing means that you're maxing out the number of allowed data streams on the storage.

[david.buchanan]

Thanks Foggy. I'll change it to 8 and see how it goes..

Regards,
David.

[foggy]

Basically, Veeam B&R v9.5 has a built-in scheduler that ensures that the maximum number of data sessions for the particular StoreOnce model is not exceeded (hence the messages), so this setting is not in fact mandatory and can be simply disabled.

[roscomputers]

Hello.
Lab description:
-Backup Server - vmware VM Windows Server 2019
-Backup Repository - vmware VM StoreOnce VSA 4.1.3
Limit maximum concurrent task is 8.

Backup to local repository completed succesfully.

Backup to StoreOnce VSA completed with error:

Code: Select all

[03.10.2019 16:38:18] <01> Error    Failed to create data session to file [[10.10.10.10] STOREONCE-STORE:/BACKUPSERVER-BKP/BACKUPSERVER-BKP.vbm_1_tmp]: maximum count of concurrent connections has been exceeded (Veeam.Backup.Common.CCppCOMException)
[03.10.2019 16:38:18] <01> Error       in c++: Failed to perform pipe command 'StoreOnceTruncateFile'. Xml: '<InputArguments><Link value="storeonce://10.10.10.10:STOREONCE-STORE@/BACKUPSERVER-BKP/BACKUPSERVER-BKP.vbm_1_tmp" /><Offset value="0" /></InputArguments>'.
[03.10.2019 16:38:18] <01> Error       in c++: Failed to execute external tool command 'StoreOnceTruncateFile'. Input xml: '<InputArguments><Link value="storeonce://10.10.10.10:STOREONCE-STORE@/BACKUPSERVER-BKP/BACKUPSERVER-BKP.vbm_1_tmp" /><Offset value="0" /></InputArguments>'. Tool name: 'VeeamPluginsHost'. Instance id: '{bede8b38-46e3-4c92-80bf-45e09efbe4ke}'.
[03.10.2019 16:38:18] <01> Error       in c++: Failed to complete RPC request. Method: [PluginsHost.StoreOnceTruncateFile], In XML: [<InputArguments><Link value="storeonce://10.10.10.10:STOREONCE-STORE@/BACKUPSERVER-BKP/BACKUPSERVER-BKP.vbm_1_tmp" /><Offset value="0" /></InputArguments>]
[03.10.2019 16:38:18] <01> Error       in c++: Failed to invoke RPC request
[03.10.2019 16:38:18] <01> Error       in c++: Failed to call DoRpc. CmdName: [DoSerialRpc] inParam: [<InputArguments/>].
[03.10.2019 16:38:18] <01> Error       in c++: Failed to call DoRpcWithBinary. Method: [DoSerialRpc], In xml: [<InputArguments/>]
[03.10.2019 16:38:18] <01> Error       in c++: Failed to call DoRpcWithBinary
[03.10.2019 16:38:18] <01> Error       in c++: Failed to do RPC call with serialized data, Strategy type: DefaultStrategy
[03.10.2019 16:38:18] <01> Error       in c++: Failed to call [DoSerialRpc], session id: {cf03d84f-61a2-49fc-b298-e49ecab72126}
[03.10.2019 16:38:18] <01> Error       in c++: Failed to call [PluginsHost.StoreOnceTruncateFile]
[03.10.2019 16:38:18] <01> Error    Failed to create data session to file [[10.10.10.10] STOREONCE-STORE:/BACKUPSERVER-BKP/BACKUPSERVER-BKP.vbm_1_tmp]: maximum count of concurrent connections has been exceeded (Veeam.Backup.Common.CCppCOMException)
[03.10.2019 16:38:18] <01> Error       in c++: Failed to perform pipe command 'StoreOnceTruncateFile'. Xml: '<InputArguments><Link value="storeonce://10.10.10.10:STOREONCE-STORE@/BACKUPSERVER-BKP/BACKUPSERVER-BKP.vbm_1_tmp" /><Offset value="0" /></InputArguments>'.
[03.10.2019 16:38:18] <01> Error       in c++: Failed to execute external tool command 'StoreOnceTruncateFile'. Input xml: '<InputArguments><Link value="storeonce://10.10.10.10:STOREONCE-STORE@/BACKUPSERVER-BKP/BACKUPSERVER-BKP.vbm_1_tmp" /><Offset value="0" /></InputArguments>'. Tool name: 'VeeamPluginsHost'. Instance id: '{bede8b38-46e3-4c92-80bf-45e09efbe4ke}'.
[03.10.2019 16:38:18] <01> Error       in c++: Failed to complete RPC request. Method: [PluginsHost.StoreOnceTruncateFile], In XML: [<InputArguments><Link value="storeonce://10.10.10.10:STOREONCE-STORE@/BACKUPSERVER-BKP/BACKUPSERVER-BKP.vbm_1_tmp" /><Offset value="0" /></InputArguments>]
[03.10.2019 16:38:18] <01> Error       in c++: Failed to invoke RPC request
[03.10.2019 16:38:18] <01> Error       in c++: Failed to call DoRpc. CmdName: [DoSerialRpc] inParam: [<InputArguments/>].
[03.10.2019 16:38:18] <01> Error       in c++: Failed to call DoRpcWithBinary. Method: [DoSerialRpc], In xml: [<InputArguments/>]
[03.10.2019 16:38:18] <01> Error       in c++: Failed to call DoRpcWithBinary
[03.10.2019 16:38:18] <01> Error       in c++: Failed to do RPC call with serialized data, Strategy type: DefaultStrategy
[03.10.2019 16:38:18] <01> Error       in c++: Failed to call [DoSerialRpc], session id: {cf03d84f-61a2-49fc-b298-e49ecab72126}
[03.10.2019 16:38:18] <01> Error       in c++: Failed to call [PluginsHost.StoreOnceTruncateFile]
[03.10.2019 16:38:19] <01> Error    Task 'TEST-BKP-01' was not completed. So it will be failed
[03.10.2019 16:38:19] <01> Info           Job progress: '0%', '0' of '75,161,927,680' bytes, object '1' of '1'
[03.10.2019 16:38:19] <01> Info     Task session [9269ff8d-6b9b-44a2-85dd-b5cdc41643d9] has been completed, status: Failed, 0 of 0 bytes, 0 of 0 objects, details: 
[03.10.2019 16:38:19] <01> Info     [BackupMetaUpdater] Disposing
[03.10.2019 16:38:20] <01> Info     [StorageDrsLocker] Unlocking remaining VMs
[03.10.2019 16:38:20] <01> Info     [SanSnapshotTransfer] Delete transfer resources
[03.10.2019 16:38:20] <01> Info     Job has been stopped successfully. Name: [TEST-BKP-01], JobId: [4e64d597-f8d6-48f2-b1df-279eeccc47c2]
[03.10.2019 16:38:20] <01> Info     [BackupJobPostActivity] JobSession '812078b6-621f-47fd-8659-c39f8212f814' PostActivity: 'AskService'
[03.10.2019 16:38:20] <01> Info     [BackupJobPostActivity] Session result is Failed. Checking if it is last retry
[03.10.2019 16:38:20] <01> Info     [BackupJobPostActivity] wouldBeRetried: 'False'. Need post activity: 'True'
[03.10.2019 16:38:20] <01> Info     Custom command execution (post script) disabled
[03.10.2019 16:38:20] <01> Info     [JobSession] Completing session '812078b6-621f-47fd-8659-c39f8212f814'
[03.10.2019 16:38:20] <01> Info     Backuped size: 0.0 B, total backuped size: 0.0 B
[03.10.2019 16:38:20] <01> Info     Job session '812078b6-621f-47fd-8659-c39f8212f814' has been completed, status: 'Failed', '0.0 B' of '0.0 B' bytes, '1' of '1' tasks, '0' successful, '1' failed, details: '', PointId: [00000000-0000-0000-0000-000000000000]
[03.10.2019 16:38:20] <01> Info     [BackupJobPostActivity] JobSession '812078b6-621f-47fd-8659-c39f8212f814' PostActivity: 'AskService'
[03.10.2019 16:38:20] <01> Info     [BackupJobPostActivity] Session result is Failed. Checking if it is last retry
[03.10.2019 16:38:20] <01> Info     [BackupJobPostActivity] wouldBeRetried: 'False'. Need post activity: 'True'
[03.10.2019 16:38:21] <01> Info     Email notification is disabled
[03.10.2019 16:38:21] <01> Info         Job event 'ready2finish' was created for session '812078b6-621f-47fd-8659-c39f8212f814'. Event full name: 'Global\ready2finishvmjobevent812078b6-621f-47fd-8659-c39f8212f814'
[03.10.2019 16:38:21] <01> Info     Job [TEST-BKP-01] is ready to finish. SessionId: [812078b6-621f-47fd-8659-c39f8212f814]
[03.10.2019 16:38:21] <01> Info         Job event 'ready2finish' was disposed. Session: '812078b6-621f-47fd-8659-c39f8212f814'.
[03.10.2019 16:38:21] <01> Info         Job event 'finished' was disposed. Session: '812078b6-621f-47fd-8659-c39f8212f814'.
[03.10.2019 16:38:21] <01> Info     Disposing SVeeamBackupService...
[03.10.2019 16:38:21] <01> Info         Unregistering TCP client channel [bstcp]
[03.10.2019 16:38:21] <01> Info         Channel successfully unregistered
[03.10.2019 16:38:21] <01> Info         Unregistering TCP client channel [bs_tcp]
[03.10.2019 16:38:21] <01> Info         Channel successfully unregistered
[03.10.2019 16:38:21] <01> Info         Unregistering TCP client channel [cat_tcp]
[03.10.2019 16:38:21] <01> Info         Channel successfully unregistered
[03.10.2019 16:38:21] <01> Info         Job process lock was disposed.
[03.10.2019 16:38:21] <01> Info         Job process lock was disposed.
[03.10.2019 16:38:21] <01> Info     [Ssh] Clearing connection cache
[03.10.2019 16:38:21] <01> Info     [Ssh] Connection cache cleared
[03.10.2019 16:38:21] <01> Info     Disposing SVeeamBrokerService...
[03.10.2019 16:38:21] <01> Info     Disposing SVeeamBackupService...
[03.10.2019 16:38:21] <01> Info     ------- Veeam Manager Stopped ------

[HannesK]

Hello,
as stated above, this should not happen.

has a built-in scheduler that ensures that the maximum number of data sessions for the particular StoreOnce model is not exceeded

Please open a case for further investigations and post the case number here for reference.

Best regards,
Hannes

[roscomputers]

Thank you Hannes, but i m use trial license. No support.

[HannesK]

trial licenses have "evaluation support" for exactly your situation

[roscomputers]

Im sorry, but I can not create a case on a https://my.veeam.com/#/open-case/step-1.

My license:
Status Valid
Expiration date 10/18/2019 (14 days left)
Type Evaluation
Edition Enterprise Plus Edition
Instances 1000 (1 used)
Support expiration date 10/18/2019 (14 days left)

[foggy]

What specifically prevents you from opening a case? Also, did you try calling our support instead?

[Regnor]

FYI: I've opened a case for my lab as I'm experiencing the same error with the VSA 4.1.3 (#03827323)

[Regnor]

Update:
I have deployed another StoreOnce VSA and access is working now without any problems; so probably it's not a Veeam topic.