Case #07125508 Restore to GCE takes long time - "Pipeline timeout"

[srlarsen]

We lose 30-45 minutes on restores to Google. The restores are successful but we should not be losing this amount of time in the process. Multiple lines in the log where the ProcessedSize = TotalSize. In this case it was approx 37 minutes.

Restore log

Code: Select all

[28.02.2024 13:59:31.741]    <01>    Info (3)    [CRestoreFilesProcessImpl] Job size updated. RestoreInfo: [CRestoreFilesProcessInfo] <9743886>: TotalFiles: 2, TotalFolders: 0, TotalSize: 128849018880, ProcessedFilesCount: 1, ProcessedSize: 128849018880, FileSize: 0, FilePercentProgress: 0, IsCalculationDone: Yes
................
[28.02.2024 14:36:46.194]    <01>    Info (3)    [CRestoreFilesProcessImpl] Job size updated. RestoreInfo: [CRestoreFilesProcessInfo] <9743886>: TotalFiles: 2, TotalFolders: 0, TotalSize: 128849018880, ProcessedFilesCount: 2, ProcessedSize: 128849018880, FileSize: 0, FilePercentProgress: 0, IsCalculationDone: Yes

Disk log

Code: Select all

[28.02.2024 13:59:27.272]    <67>    Info (3)    [CRestoreFilesProcessImpl] Job size updated. RestoreInfo: [CRestoreFilesProcessInfo] <9743886>: TotalFiles: 2, TotalFolders: 0, TotalSize: 128849018880, ProcessedFilesCount: 1, ProcessedSize: 67645734912, FileSize: 0, FilePercentProgress: 0, IsCalculationDone: Yes
[28.02.2024 14:27:57.239]    <28>    Info (3)    [AP] Waiting for completion. Id: [0x1396159], Time: 00:59:59.9995315
[28.02.2024 14:29:26.254]    <56>    Info (4)            [AP] (2555980d) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[28.02.2024 14:36:38.054]    <32>    Info (4)            [AP] (2cebc96b) output: <VCPCommandResult result="true" exception="" />
[28.02.2024 14:36:38.054]    <32>    Info (4)            [AP] (2cebc96b) output: <VCPCommandArgs />
[28.02.2024 14:36:38.054]    <32>    Info (4)            [AP] (2cebc96b) output: >
[28.02.2024 14:36:38.054]    <28>    Info (3)    [GoogleLocalAgentVhdUploader] Disk 4BA16F5EC5979F4B08806C07679F2C7C has been uploaded to gs://veeamrestorebucket-7ea996ec9ce3/disk-4ba16f5ec5979f4b08806c07679f2c7c.raw

We have done multiple troubleshooting steps.
On Prem PROD VBR -> PROD GCP project - delay/timeout
On Prem PROD VBR -> LAB GCP project - delay/timeout
DR GCP Project VBR -> DR GCP project - normal - no delay
On Prem TEST VBR ver12 -> DR GCP project - normal - no delay
On Prem TEST VBR upgraded to 12.1 -> DR GCP project - delay/timeout

At one point we were pretty sure it was some GCP project permission, then we thought network, we thought we ruled out VBR version with the GCP -> GCP test but then we had a successful On-Prem -> GCP test with version 12 but then upgraded to 12.1 and then got the delay/timeout. Support is working the ticket and transferring to Advanced Support but this is not adding up.

Any insight to offer us or the support team would be great. Thanks.

[Mildur]

Hello Stephen

The case should have been escalated to the next tier. But in my understanding, there wasn't any update for you since the escalation.
I asked Support management to have a look at the case.

Best,
Fabian