We have recently setup a new Dell PowerScale and configured an SMB share to use as a backup repository with Veeam. We have configured the SMB: share with default settings and have successfully added it as a Backup Repository in Veeam.
All of our backup jobs have been running great; Domain Controller Windows Agent Backups, Hyper-V cluster Hyper-V Backups.
The problem is with our file server cluster Windows Agent Backup job. We are backing up one drive from the file server that consists of 10.8 TB of data to the Dell Powerscale SMB share. The job reaches exactly 50% - 51% and then fails with the error:
Code: Select all
[02.02.2024 23:25:06.141] < 5944> cli | Number of sessions: 10. Interval: 34764 sec.
[02.02.2024 23:25:14.891] < 9872> cli | WARN|Failed to set file size in CWinAsyncEngine::PrepareForWrite operation.
[02.02.2024 23:25:14.891] < 9872> cli | >> |There is not enough space on the disk.
[02.02.2024 23:25:14.891] < 9872> cli | >> |Unable to reset file size: trunk operation has failed. File: [\\rs-ps-smb\VeeamRepo\RS-FS F Drive\RS-FSC\RS-FS F Drive - RS-FSCD2024-02-02T134310_FC9C.vbk].
[02.02.2024 23:25:14.891] < 9872> cli | >> |--tr:Error code: 0x00000070
[02.02.2024 23:25:15.079] < 9872> cli | ERR |Command 'FibWrite(0x0000014D18BA8BA0)' has failed.
[02.02.2024 23:25:15.079] < 9872> cli | >> |There is not enough space on the disk.
[02.02.2024 23:25:15.079] < 9872> cli | >> |Asynchronous request operation has failed. [requestsize = 983040] [offset = 4398045532160]
[02.02.2024 23:25:15.079] < 9872> cli | >> |--tr:Error code: 0x00000070
[02.02.2024 23:25:15.079] < 9872> cli | >> |--tr:Failed to process de-duplicated block
[02.02.2024 23:25:15.079] < 9872> cli | >> |--tr:Failed to write block to blocks store.
[02.02.2024 23:25:15.079] < 9872> cli | >> |--tr:Failed to append block. FIB index: [5839302]. FIB: [b64c843a-6e7f-4158-b635-16227d30a9f2].
[02.02.2024 23:25:15.079] < 9872> cli | >> |--tr:Failed to process area [Data block. Start offset: [6122951933952], Length: [1048576], Area ID: [5083132].].
[02.02.2024 23:25:15.079] < 9872> cli | >> |An exception was thrown from thread [9872].
[02.02.2024 23:25:15.079] < 4900> cli | ERR |Send thread (channel) has failed.
[02.02.2024 23:25:15.079] < 4900> cli | >> |There is not enough space on the disk.
[02.02.2024 23:25:15.079] < 4900> cli | >> |Asynchronous request operation has failed. [requestsize = 983040] [offset = 4398045532160]
[02.02.2024 23:25:15.079] < 4900> cli | >> |--tr:Error code: 0x00000070
[02.02.2024 23:25:15.079] < 4900> cli | >> |--tr:Failed to process de-duplicated block
[02.02.2024 23:25:15.079] < 4900> cli | >> |--tr:Failed to write block to blocks store.
[02.02.2024 23:25:15.079] < 4900> cli | >> |--tr:Failed to append block. FIB index: [5839302]. FIB: [b64c843a-6e7f-4158-b635-16227d30a9f2].
[02.02.2024 23:25:15.079] < 4900> cli | >> |--tr:Failed to process area [Data block. Start offset: [6122951933952], Length: [1048576], Area ID: [5083132].].
[02.02.2024 23:25:15.079] < 4900> cli | >> |--tr:Failed to wait for recorder invoke.
[02.02.2024 23:25:15.079] < 4900> cli | >> |--tr:Failed to check queue and pending invoke status. Wait flag 'true'.
[02.02.2024 23:25:15.079] < 4900> cli | >> |--tr:FIB proxy was unable to process write operation, recorder '0x0000014D18BA8BA0', area offset '6123139629056', FIB 'Backup of the FIB b64c843a-6e7f-4158-b635-16227d30a9f2'.
[02.02.2024 23:25:15.079] < 4900> cli | >> |--tr:Unable to asynchronously write data block. Block identity: [Data block. Start offset: [6123139629056], Length: [1048576], Area ID: [5083311].].
[02.02.2024 23:25:15.079] < 4900> cli | >> |An exception was thrown from thread [9872].
[02.02.2024 23:25:15.079] < 7244> alg | Stopping processing threads....
[02.02.2024 23:25:15.079] < 4900> cli | Thread finished. Role: 'Area writer: Backup of the FIB b64c843a-6e7f-4158-b635-16227d30a9f2'.
[02.02.2024 23:25:15.079] < 9232> cli | Thread finished. Role: 'area reader : Receiver channel for b64c843a-6e7f-4158-b635-16227d30a9f2'.
[02.02.2024 23:25:15.079] < 7244> alg | Processing threads were stopped.
[02.02.2024 23:25:15.079] < 7244> alg | Processing conveyored task... Failed.
[02.02.2024 23:25:15.079] < 7244> alg | ERR |Transmission pipe error.
[02.02.2024 23:25:15.079] < 7244> alg | >> |There is not enough space on the disk.
[02.02.2024 23:25:15.079] < 7244> alg | >> |Asynchronous request operation has failed. [requestsize = 983040] [offset = 4398045532160]
[02.02.2024 23:25:15.079] < 7244> alg | >> |--tr:Error code: 0x00000070
[02.02.2024 23:25:15.079] < 7244> alg | >> |--tr:Failed to process de-duplicated block
[02.02.2024 23:25:15.079] < 7244> alg | >> |--tr:Failed to write block to blocks store.
[02.02.2024 23:25:15.079] < 7244> alg | >> |--tr:Failed to append block. FIB index: [5839302]. FIB: [b64c843a-6e7f-4158-b635-16227d30a9f2].
[02.02.2024 23:25:15.079] < 7244> alg | >> |--tr:Failed to process area [Data block. Start offset: [6122951933952], Length: [1048576], Area ID: [5083132].].
[02.02.2024 23:25:15.079] < 7244> alg | >> |--tr:Failed to wait for recorder invoke.
[02.02.2024 23:25:15.079] < 7244> alg | >> |--tr:Failed to check queue and pending invoke status. Wait flag 'true'.
[02.02.2024 23:25:15.079] < 7244> alg | >> |--tr:FIB proxy was unable to process write operation, recorder '0x0000014D18BA8BA0', area offset '6123139629056', FIB 'Backup of the FIB b64c843a-6e7f-4158-b635-16227d30a9f2'.
[02.02.2024 23:25:15.079] < 7244> alg | >> |--tr:Unable to asynchronously write data block. Block identity: [Data block. Start offset: [6123139629056], Length: [1048576], Area ID: [5083311].].
[02.02.2024 23:25:15.079] < 7244> alg | >> |--tr:Processing of asynchronous write requests has failed. Output file: [Backup of the FIB b64c843a-6e7f-4158-b635-16227d30a9f2].
[02.02.2024 23:25:15.079] < 7244> alg | >> |An exception was thrown from thread [9872].
[02.02.2024 23:25:15.079] < 7244> alg | Stopping recv. thread.
[02.02.2024 23:25:15.079] < 7244> alg | Stopping recv thread ('b64c843a-6e7f-4158-b635-16227d30a9f2'), timeout: 7200000 ms.
[02.02.2024 23:25:15.079] < 7244> alg | WARN|Generating terminated request for recv. thread.
[02.02.2024 23:25:15.079] < 7244> cli | Awaiting [60] sec for stage [reconn. receiver {55040d90-8e95-4e64-86f0-08551882eb72}] to complete.
[02.02.2024 23:25:15.079] < 6116> alg | Reconnectable stop request was sent on {55040d90-8e95-4e64-86f0-08551882eb72}.
[02.02.2024 23:25:15.110] < 5840> alg | Reconnectable protocol stop request confirmation was received on {55040d90-8e95-4e64-86f0-08551882eb72}.
[02.02.2024 23:25:15.110] < 5840> alg | Reconnectable protocol stop request was received on {55040d90-8e95-4e64-86f0-08551882eb72}.
[02.02.2024 23:25:15.110] < 5840> alg | Receiver: finishing on {55040d90-8e95-4e64-86f0-08551882eb72}.
[02.02.2024 23:25:15.110] < 5840> cli | Thread finished. Role: '(async) reconn. receiver {55040d90-8e95-4e64-86f0-08551882eb72}'.
[02.02.2024 23:25:15.110] < 6116> alg | Reconnectable stop confirmation was sent on {55040d90-8e95-4e64-86f0-08551882eb72}.
[02.02.2024 23:25:15.110] < 6116> alg | Sender stage is finishing.
[02.02.2024 23:25:15.110] < 7244> cli | Awaiting [60] sec for stage [reconn. sender {55040d90-8e95-4e64-86f0-08551882eb72}] to complete.
[02.02.2024 23:25:15.110] < 6116> cli | Thread finished. Role: '(async) reconn. sender {55040d90-8e95-4e64-86f0-08551882eb72}'.
[02.02.2024 23:25:15.110] < 10672> alg | WARN|Receive thread is terminated ('b64c843a-6e7f-4158-b635-16227d30a9f2').
[02.02.2024 23:25:15.110] < 10672> cli | Thread finished. Role: 'TCP receiver thread (b64c843a-6e7f-4158-b635-16227d30a9f2)'.
[02.02.2024 23:25:15.110] < 7244> alg | Recv. thread is stopped.
[02.02.2024 23:25:15.110] < 7244> alg | Stopping recv. thread.
[02.02.2024 23:25:15.110] < 7244> alg | Stopping recv thread ('b64c843a-6e7f-4158-b635-16227d30a9f2'), timeout: 7200000 ms.
[02.02.2024 23:25:15.110] < 7244> alg | WARN|Generating terminated request for recv. thread.
[02.02.2024 23:25:15.110] < 7244> cli | Awaiting [60] sec for stage [reconn. receiver {a1d79f5d-9889-4691-b01d-af3ce5b6455a}] to complete.
[02.02.2024 23:25:15.110] < 10032> alg | Reconnectable stop request was sent on {a1d79f5d-9889-4691-b01d-af3ce5b6455a}.
[02.02.2024 23:25:15.126] < 5812> alg | Reconnectable protocol stop request confirmation was received on {a1d79f5d-9889-4691-b01d-af3ce5b6455a}.
[02.02.2024 23:25:15.126] < 5812> alg | Reconnectable protocol stop request was received on {a1d79f5d-9889-4691-b01d-af3ce5b6455a}.
[02.02.2024 23:25:15.126] < 5812> alg | Receiver: finishing on {a1d79f5d-9889-4691-b01d-af3ce5b6455a}.
[02.02.2024 23:25:15.126] < 10032> alg | Reconnectable stop confirmation was sent on {a1d79f5d-9889-4691-b01d-af3ce5b6455a}.
[02.02.2024 23:25:15.126] < 10032> alg | Sender stage is finishing.
[02.02.2024 23:25:15.126] < 5812> cli | Thread finished. Role: '(async) reconn. receiver {a1d79f5d-9889-4691-b01d-af3ce5b6455a}'.
[02.02.2024 23:25:15.126] < 10032> cli | Thread finished. Role: '(async) reconn. sender {a1d79f5d-9889-4691-b01d-af3ce5b6455a}'.
[02.02.2024 23:25:15.126] < 7244> cli | Awaiting [60] sec for stage [reconn. sender {a1d79f5d-9889-4691-b01d-af3ce5b6455a}] to complete.
[02.02.2024 23:25:15.126] < 7360> alg | WARN|Receive thread is terminated ('b64c843a-6e7f-4158-b635-16227d30a9f2').
[02.02.2024 23:25:15.126] < 7360> cli | Thread finished. Role: 'TCP receiver thread (b64c843a-6e7f-4158-b635-16227d30a9f2)'.
[02.02.2024 23:25:15.126] < 7244> alg | Recv. thread is stopped.
[02.02.2024 23:25:15.126] < 7244> alg | Stopping recv. thread.
[02.02.2024 23:25:15.126] < 7244> alg | Stopping recv thread ('b64c843a-6e7f-4158-b635-16227d30a9f2'), timeout: 7200000 ms.
[02.02.2024 23:25:15.126] < 7244> alg | WARN|Generating terminated request for recv. thread.
[02.02.2024 23:25:15.126] < 7244> cli | Awaiting [60] sec for stage [reconn. receiver {78ded470-70a2-4bf0-bf39-b3d25a3ffdd0}] to complete.
[02.02.2024 23:25:15.126] < 10388> alg | Reconnectable stop request was sent on {78ded470-70a2-4bf0-bf39-b3d25a3ffdd0}.
[02.02.2024 23:25:15.141] < 1960> alg | Reconnectable protocol stop request confirmation was received on {78ded470-70a2-4bf0-bf39-b3d25a3ffdd0}.
[02.02.2024 23:25:15.141] < 1960> alg | Reconnectable protocol stop request was received on {78ded470-70a2-4bf0-bf39-b3d25a3ffdd0}.
[02.02.2024 23:25:15.141] < 1960> alg | Receiver: finishing on {78ded470-70a2-4bf0-bf39-b3d25a3ffdd0}.
[02.02.2024 23:25:15.141] < 10388> alg | Reconnectable stop confirmation was sent on {78ded470-70a2-4bf0-bf39-b3d25a3ffdd0}.
[02.02.2024 23:25:15.141] < 10388> alg | Sender stage is finishing.
[02.02.2024 23:25:15.141] < 1960> cli | Thread finished. Role: '(async) reconn. receiver {78ded470-70a2-4bf0-bf39-b3d25a3ffdd0}'.
[02.02.2024 23:25:15.141] < 10388> cli | Thread finished. Role: '(async) reconn. sender {78ded470-70a2-4bf0-bf39-b3d25a3ffdd0}'.
[02.02.2024 23:25:15.141] < 5808> alg | WARN|Receive thread is terminated ('b64c843a-6e7f-4158-b635-16227d30a9f2').
[02.02.2024 23:25:15.141] < 5808> cli | Thread finished. Role: 'TCP receiver thread (b64c843a-6e7f-4158-b635-16227d30a9f2)'.
[02.02.2024 23:25:15.141] < 7244> alg | Recv. thread is stopped.
[02.02.2024 23:25:15.141] < 7244> alg | Stopping recv. thread.
[02.02.2024 23:25:15.141] < 7244> alg | Stopping recv thread ('b64c843a-6e7f-4158-b635-16227d30a9f2'), timeout: 7200000 ms.
[02.02.2024 23:25:15.141] < 7244> alg | WARN|Generating terminated request for recv. thread.
[02.02.2024 23:25:15.141] < 7244> cli | Awaiting [60] sec for stage [reconn. receiver {2bcdc39a-819d-430c-84f3-9fa704c41e3e}] to complete.
[02.02.2024 23:25:15.141] < 9696> alg | Reconnectable stop request was sent on {2bcdc39a-819d-430c-84f3-9fa704c41e3e}.
[02.02.2024 23:25:15.141] < 504> alg | Reconnectable protocol stop request was received on {2bcdc39a-819d-430c-84f3-9fa704c41e3e}.
[02.02.2024 23:25:15.141] < 9696> alg | Reconnectable stop confirmation was sent on {2bcdc39a-819d-430c-84f3-9fa704c41e3e}.
[02.02.2024 23:25:15.157] < 5864> alg | Reconnectable protocol stop request was received on {05bf28bc-dd4e-4273-9a57-531e76be1390}.
[02.02.2024 23:25:15.157] < 5720> alg | Reconnectable stop confirmation was sent on {05bf28bc-dd4e-4273-9a57-531e76be1390}.
[02.02.2024 23:25:15.157] < 5720> alg | Reconnectable stop request was sent on {05bf28bc-dd4e-4273-9a57-531e76be1390}.
[02.02.2024 23:25:15.173] < 5864> alg | Reconnectable protocol stop request confirmation was received on {05bf28bc-dd4e-4273-9a57-531e76be1390}.
[02.02.2024 23:25:15.173] < 5864> alg | Receiver: finishing on {05bf28bc-dd4e-4273-9a57-531e76be1390}.
[02.02.2024 23:25:15.173] < 5864> cli | Thread finished. Role: '(async) reconn. receiver {05bf28bc-dd4e-4273-9a57-531e76be1390}'.
[02.02.2024 23:25:15.173] < 5720> alg | Sender stage is finishing.
[02.02.2024 23:25:15.173] < 5720> cli | Thread finished. Role: '(async) reconn. sender {05bf28bc-dd4e-4273-9a57-531e76be1390}'.
[02.02.2024 23:25:15.173] < 504> alg | Reconnectable protocol stop request confirmation was received on {2bcdc39a-819d-430c-84f3-9fa704c41e3e}.
[02.02.2024 23:25:15.173] < 504> alg | Receiver: finishing on {2bcdc39a-819d-430c-84f3-9fa704c41e3e}.
[02.02.2024 23:25:15.173] < 504> cli | Thread finished. Role: '(async) reconn. receiver {2bcdc39a-819d-430c-84f3-9fa704c41e3e}'.
[02.02.2024 23:25:15.173] < 9696> alg | Sender stage is finishing.
[02.02.2024 23:25:15.173] < 7244> cli | Awaiting [60] sec for stage [reconn. sender {2bcdc39a-819d-430c-84f3-9fa704c41e3e}] to complete.
[02.02.2024 23:25:15.173] < 9696> cli | Thread finished. Role: '(async) reconn. sender {2bcdc39a-819d-430c-84f3-9fa704c41e3e}'.
[02.02.2024 23:25:15.173] < 6284> alg | WARN|Receive thread is terminated ('b64c843a-6e7f-4158-b635-16227d30a9f2').
[02.02.2024 23:25:15.173] < 6284> cli | Thread finished. Role: 'TCP receiver thread (b64c843a-6e7f-4158-b635-16227d30a9f2)'.
[02.02.2024 23:25:15.173] < 7244> alg | Recv. thread is stopped.
[02.02.2024 23:25:15.173] < 7244> alg | Stopping recv. thread.
[02.02.2024 23:25:15.173] < 7244> alg | Stopping recv thread ('b64c843a-6e7f-4158-b635-16227d30a9f2'), timeout: 7200000 ms.
[02.02.2024 23:25:15.173] < 7244> alg | WARN|Generating terminated request for recv. thread.
[02.02.2024 23:25:15.173] < 7244> cli | Awaiting [60] sec for stage [reconn. receiver {a504d3f9-f8ce-421f-ba05-f408c0af1aef}] to complete.
[02.02.2024 23:25:15.173] < 9572> alg | Reconnectable stop request was sent on {a504d3f9-f8ce-421f-ba05-f408c0af1aef}.
[02.02.2024 23:25:15.173] < 10868> alg | Reconnectable protocol stop request was received on {a504d3f9-f8ce-421f-ba05-f408c0af1aef}.
[02.02.2024 23:25:15.173] < 9572> alg | Reconnectable stop confirmation was sent on {a504d3f9-f8ce-421f-ba05-f408c0af1aef}.
[02.02.2024 23:25:15.188] < 10868> alg | Reconnectable protocol stop request confirmation was received on {a504d3f9-f8ce-421f-ba05-f408c0af1aef}.
[02.02.2024 23:25:15.188] < 10868> alg | Receiver: finishing on {a504d3f9-f8ce-421f-ba05-f408c0af1aef}.
[02.02.2024 23:25:15.188] < 10868> cli | Thread finished. Role: '(async) reconn. receiver {a504d3f9-f8ce-421f-ba05-f408c0af1aef}'.
[02.02.2024 23:25:15.188] < 9572> alg | Sender stage is finishing.
[02.02.2024 23:25:15.188] < 7244> cli | Awaiting [60] sec for stage [reconn. sender {a504d3f9-f8ce-421f-ba05-f408c0af1aef}] to complete.
[02.02.2024 23:25:15.188] < 9572> cli | Thread finished. Role: '(async) reconn. sender {a504d3f9-f8ce-421f-ba05-f408c0af1aef}'.
[02.02.2024 23:25:15.188] < 11036> alg | WARN|Receive thread is terminated ('b64c843a-6e7f-4158-b635-16227d30a9f2').
[02.02.2024 23:25:15.188] < 11036> alg | All download streams are processed.
[02.02.2024 23:25:15.188] < 11036> cli | Thread finished. Role: 'TCP receiver thread (b64c843a-6e7f-4158-b635-16227d30a9f2)'.
[02.02.2024 23:25:15.188] < 7244> alg | Recv. thread is stopped.
[02.02.2024 23:25:15.188] < 7244> alg | Terminating recv. retry thread, timeout: 7200000 ms.
[02.02.2024 23:25:15.188] < 6336> alg | Terminating pending data resend commands
[02.02.2024 23:25:15.188] < 6336> alg | ERR |Recv. retry thread was terminated.
[02.02.2024 23:25:15.188] < 6336> cli | Thread finished. Role: 'Receive retry thread (b64c843a-6e7f-4158-b635-16227d30a9f2)'.
[02.02.2024 23:25:15.188] < 7244> alg | Recv. retry thread is finished.
[02.02.2024 23:25:15.219] < 7244> dt | Performing disk download. Failed.
[02.02.2024 23:25:15.219] < 7244> srv | Running disk data transfer ProtoEx dispatch loop. Failed.
[02.02.2024 23:25:15.219] < 7244> srv | ERR |There is not enough space on the disk.
[02.02.2024 23:25:15.219] < 7244> srv | >> |Asynchronous request operation has failed. [requestsize = 983040] [offset = 4398045532160]
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Error code: 0x00000070
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to process de-duplicated block
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to write block to blocks store.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to append block. FIB index: [5839302]. FIB: [b64c843a-6e7f-4158-b635-16227d30a9f2].
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to process area [Data block. Start offset: [6122951933952], Length: [1048576], Area ID: [5083132].].
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to wait for recorder invoke.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to check queue and pending invoke status. Wait flag 'true'.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:FIB proxy was unable to process write operation, recorder '0x0000014D18BA8BA0', area offset '6123139629056', FIB 'Backup of the FIB b64c843a-6e7f-4158-b635-16227d30a9f2'.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Unable to asynchronously write data block. Block identity: [Data block. Start offset: [6123139629056], Length: [1048576], Area ID: [5083311].].
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Processing of asynchronous write requests has failed. Output file: [Backup of the FIB b64c843a-6e7f-4158-b635-16227d30a9f2].
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to process conveyored task.
[02.02.2024 23:25:15.219] < 7244> srv | >> |Failed to download disk 'b64c843a-6e7f-4158-b635-16227d30a9f2'.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Disk download failed.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Unable to run ProtoEx server session.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to handle ProtoEx session.
[02.02.2024 23:25:15.219] < 11836> srv | Clearing finished srv. sessions.
[02.02.2024 23:25:15.219] < 7244> srv | ERR |Server session thread has failed.
[02.02.2024 23:25:15.219] < 7244> srv | >> |There is not enough space on the disk.
[02.02.2024 23:25:15.219] < 7244> srv | >> |Asynchronous request operation has failed. [requestsize = 983040] [offset = 4398045532160]
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Error code: 0x00000070
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to process de-duplicated block
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to write block to blocks store.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to append block. FIB index: [5839302]. FIB: [b64c843a-6e7f-4158-b635-16227d30a9f2].
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to process area [Data block. Start offset: [6122951933952], Length: [1048576], Area ID: [5083132].].
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to wait for recorder invoke.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to check queue and pending invoke status. Wait flag 'true'.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:FIB proxy was unable to process write operation, recorder '0x0000014D18BA8BA0', area offset '6123139629056', FIB 'Backup of the FIB b64c843a-6e7f-4158-b635-16227d30a9f2'.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Unable to asynchronously write data block. Block identity: [Data block. Start offset: [6123139629056], Length: [1048576], Area ID: [5083311].].
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Processing of asynchronous write requests has failed. Output file: [Backup of the FIB b64c843a-6e7f-4158-b635-16227d30a9f2].
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to process conveyored task.
[02.02.2024 23:25:15.219] < 7244> srv | >> |Failed to download disk 'b64c843a-6e7f-4158-b635-16227d30a9f2'.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Disk download failed.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Unable to run ProtoEx server session.
[02.02.2024 23:25:15.219] < 7244> srv | >> |--tr:Failed to handle ProtoEx session.
[02.02.2024 23:25:15.219] < 7244> srv | >> |An exception was thrown from thread [9872].
[02.02.2024 23:25:15.219] < 7244> srv | Closing server socket.
[02.02.2024 23:25:15.219] < 7244> srv | Socket is closed.
[02.02.2024 23:25:15.219] < 7244> alg | Traffic history max size: [15].
[02.02.2024 23:25:15.219] < 7244> alg | Traffic history max size: [3].
[02.02.2024 23:25:15.219] < 7244> alg | Traffic history max size: [73].
[02.02.2024 23:25:15.219] < 7244> alg | Traffic history max size: [3].
[02.02.2024 23:25:15.219] < 7244> alg | Traffic history max size: [2].
[02.02.2024 23:25:15.219] < 7244> alg | Traffic history max size: [2].
[02.02.2024 23:25:15.219] < 7244> net | AUX channel shutdown triggered.
[02.02.2024 23:25:15.219] < 7244> cli | Awaiting [60] sec for stage [AUX channel sender] to complete.
[02.02.2024 23:25:15.219] < 13008> net | AUX data sender: sending shutdown marker.
[02.02.2024 23:25:15.219] < 13008> cli | Awaiting [60] sec for stage [reconn. receiver {804d6962-1045-4c4e-bfe5-d5c5668ec5bf}] to complete.
[02.02.2024 23:25:15.219] < 4704> alg | Reconnectable stop request was sent on {804d6962-1045-4c4e-bfe5-d5c5668ec5bf}.
[02.02.2024 23:25:15.219] < 9912> alg | Reconnectable protocol stop request was received on {804d6962-1045-4c4e-bfe5-d5c5668ec5bf}.
[02.02.2024 23:25:15.219] < 4704> alg | Reconnectable stop confirmation was sent on {804d6962-1045-4c4e-bfe5-d5c5668ec5bf}.
[02.02.2024 23:25:15.219] < 9912> alg | Reconnectable protocol stop request confirmation was received on {804d6962-1045-4c4e-bfe5-d5c5668ec5bf}.
[02.02.2024 23:25:15.219] < 9912> alg | Receiver: finishing on {804d6962-1045-4c4e-bfe5-d5c5668ec5bf}.
[02.02.2024 23:25:15.219] < 9912> cli | Thread finished. Role: '(async) reconn. receiver {804d6962-1045-4c4e-bfe5-d5c5668ec5bf}'.
[02.02.2024 23:25:15.219] < 4704> alg | Sender stage is finishing.
[02.02.2024 23:25:15.219] < 13008> cli | Awaiting [60] sec for stage [reconn. sender {804d6962-1045-4c4e-bfe5-d5c5668ec5bf}] to complete.
[02.02.2024 23:25:15.219] < 4704> cli | Thread finished. Role: '(async) reconn. sender {804d6962-1045-4c4e-bfe5-d5c5668ec5bf}'.
[02.02.2024 23:25:15.219] < 13008> alg | Traffic history max size: [8].
[02.02.2024 23:25:15.219] < 13008> cli | Thread finished. Role: '(async) AUX channel sender'.
[02.02.2024 23:25:15.219] < 7244> cli | Awaiting [60] sec for stage [AUX channel receiver] to complete.
[02.02.2024 23:25:15.235] < 12000> net | AUX data receiver: shutdown marker received. Waiting for shutdown.
[02.02.2024 23:25:15.235] < 940> alg | Reconnectable protocol stop request was received on {588da429-9b0c-4dbf-8748-dcdca55b1f77}.
[02.02.2024 23:25:15.235] < 12000> cli | Awaiting [60] sec for stage [reconn. receiver {588da429-9b0c-4dbf-8748-dcdca55b1f77}] to complete.
[02.02.2024 23:25:15.235] < 1720> alg | Reconnectable stop confirmation was sent on {588da429-9b0c-4dbf-8748-dcdca55b1f77}.
[02.02.2024 23:25:15.235] < 1720> alg | Reconnectable stop request was sent on {588da429-9b0c-4dbf-8748-dcdca55b1f77}.
[02.02.2024 23:25:15.235] < 940> alg | Reconnectable protocol stop request confirmation was received on {588da429-9b0c-4dbf-8748-dcdca55b1f77}.
[02.02.2024 23:25:15.235] < 940> alg | Receiver: finishing on {588da429-9b0c-4dbf-8748-dcdca55b1f77}.
[02.02.2024 23:25:15.235] < 940> cli | Thread finished. Role: '(async) reconn. receiver {588da429-9b0c-4dbf-8748-dcdca55b1f77}'.
[02.02.2024 23:25:15.235] < 1720> alg | Sender stage is finishing.
[02.02.2024 23:25:15.235] < 12000> cli | Awaiting [60] sec for stage [reconn. sender {588da429-9b0c-4dbf-8748-dcdca55b1f77}] to complete.
[02.02.2024 23:25:15.235] < 1720> cli | Thread finished. Role: '(async) reconn. sender {588da429-9b0c-4dbf-8748-dcdca55b1f77}'.
[02.02.2024 23:25:15.235] < 12000> alg | Traffic history max size: [2].