Problem with SQL Backup from Nimble Volume

[vgn0me]

All we have been experiencing some issues backing up our SQl Server via the Agent for Windows. This server has quite a few volumes attacked, but only two SAN volumes. We have been getting a standalone mode backup of the server however lately we cannot get past the Nimble attached volume. The job will run for about 10 hours then die. We have tried multiple methods as well as support cases with Veeam. We have tried a few KBs (I do not have them on hand) and they have not helped. We increased the network timeouts per a KB, SQL backup method per another KB (KB2463) all to no avail. At nearly exactly 10 hours into the job (no termination settings are set) the job dies.

I noticed tonight that it is always the Nimble volume. It processes 2.4TB out of 3.7 and dies right at the 10 hour marker.

Code: Select all

9/22/2018 12:56:26 PM :: DB (X:) (3.7 TB) 2.4 TB read at 134 MB/s 
9/22/2018 6:11:29 PM :: Error: Invalid Signature.
Failed to write data to the file [\\ALA-SYN-01\Backups\ALA-SQL-01 - ALA-SQL-01\ALA-SQL-01 - ALA-SQL-01D2018-09-22T081124.vbk].
Failed to download disk.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.

9/22/2018 6:12:32 PM :: Error: An unexpected network error occurred.
Failed to read data from the file [\\ALA-SYN-01\Backups\ALA-SQL-01 - ALA-SQL-01\ALA-SQL-01 - ALA-SQL-01D2018-09-22T081124.vbk].
Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{1f2c8289-3af2-426b-9f8a-faee379447af}]. CHMOD mask: [9].
Agent failed to process method {DataTransfer.RestoreText}.

Any other thoughts? I have a case opened with Nimble too.


Nimble Case Case #02474423 was submitted successfully

[Dima P.]

Hello Robert.

Please keep working with our support team and let us know how the investigation goes. Thanks!

[vgn0me]

I noticed that it dies on the same .VIB every attempt. I have started a new job with a new backup chain in the mean time.

Is there any way to rebuild a backup chain without needing to do a full again? A rebuild repository feature or something. Can we edit the .vbm or delete the bad .vib and re run the job. Each attempt takes 10-12 hours (lately dies right on 10).

[vgn0me]

So, now the issue has spread to all full or synthetic full jobs not just my SQL jobs. At the 10 hour mark I get the same error when trying to the .vib. @gostev?

[vgn0me]

So, in case anyone is curious. We may have figured out the issue on our own.

The 10 hour failure issue seems to be related to Kerberos tokens timing out for the SMB session back to the repository. We created a local account with access and set it as the credential on the repository. What is odd is our synthetic fulls had taken 13-14 hours for the last two years. So, why all of a sudden?

[Paulo Balau]

I was Backup Copying a 1,4TB Backup from a QNAP NAS to a USB disk attached to this very same NAS. It lasts about 12 hts to complete.
VBR is only one.
This backup regards to a file server having several disks.
When BC, I alway get this error and fails after 5 retries.
I always resync and veeam resumes by adding another incremental file with the remaining data.
But I'm concerned about the error because I need to be cheking all the time.
I am pretty sure NAS is available!
What may be causing this?

Code: Select all

11/12/2018 10:11:21 :: Error: Invalid Signature.
Failed to write data to the file [\\nasbak01\USB-BackupCopy-Week2\BckCp-FileServerOnly\BckCp-FileServerOnlyD2018-12-11T001000.vib].
Failed to download disk.
Shared memory connection was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.

(…)

11/12/2018 10:12:26 :: Error: An unexpected network error occurred.
Failed to read data from the file [\\nasbak01\USB-BackupCopy-Week2\BckCp-FileServerOnly\BckCp-FileServerOnlyD2018-12-10T001000.vbk].
Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{b7fed0c5-5c6c-4b87-a124-65af25d6711b}]. CHMOD mask: [176].
Agent failed to process method {DataTransfer.RestoreText}.
  
(...)

Again: I am pretty sure NAS is available and network is ok.
No other jobs were running.
After failing, after 5 retries a "Sync now" resumes the backup copy


Best Regards,
Paulo Balau

[PTide]

Paulo,

Please check the existing thread that describes a very similar case. If the resolution does not work for you, then kindly open a support case and post your case ID.

Thanks!

[Paulo Balau]

Hi All,

At Windows Group Policy Management \ Computer Configuration \Policies \ Windows Settings \ Security Settings \ Account Policies \ Kerberos Policy \ Maximum lifetime for user ticket = 10 hours!

Had someone tried to change with success?

BR
Paulo

[Dima P.]

Hi Paulo,

Are you using Kerberos authentication across your environment? Is it possible to switch to the regular NTLM authentication? Thanks!

[Paulo Balau]

Hi! Used an local account and it worked. Failed once at 10hrs but recovered.
Next week I'll place kerberos with +10hrs and check the behaviour.

Regarding NTLM, I really do not know I have to ask for.

[Dima P.]

Hello Paulo,

I was told that increasing the lifetime of the security token, so it should work. Please let us know how it goes. Cheers!

[Paulo Balau]

Deal!