Backup copy job interrupted: partial files any good?

[TheWaterbug]

I want to use the D: drive(s) on Backup4 as an offsite repository for the backups stored in Backup3:



I created a simple Backup copy job. Full backups are OFF.



Health check is ON:



Summary is as follows:

Code: Select all

Name: Backup3 to Backup4Swap1 
Target Path: D:\Backups
Type: Windows Agent Backup Copy
Source items:

I seeded the backup drives by running this Backup copy job while Backup4 was in the main office, over GbE, then copying the Backup folder (and the 1.2 TB vbk file) from ExtHDD1 to ExtHDD2. While still in the office I re-ran the Backup copy job while ExtHDD1 was mounted as D:, and then while ExtHDD2 was mounted as D:, and both worked fine, recognizing the rotated HDD and doing the right thing. Those two jobs resulted in .vib files of about 10 GB. I did _not_ copy this from ExtHDD1 to ExtHDD2; note the size, filenames and timestamps. These two 10 GB files were created by two runs of the Backup copy job.



I ran another job inside the office, to ExtHDD1, just to be see how big a file it would create, and it made a 3 GB file on 12/16 (see above).

So then I took Backup4 home, where I have a hardware IPSec tunnel back to the main office. Max throughput is only 20 Mbps right now. I clicked Sync on the Backup copy job, and it found the repository and started, but that where I started having problems:

1. For some reason it wants to start a full backup copy, not an incremental one.
2. It created a VBK file, and during the run it reported "1% complete" after it had processed ~30 GB of data. Transfer rate is reporting about ~1.5x.
3. This was after 4 hours, suggesting that this might take ~400 hours to complete.
4. And then, of course, the job got interrupted. My pfsense box at home crashed in the middle of the night, and had to be rebooted when I woke up.
5. I restarted the job, and this time it transferred 60 GB before the pfsense box crashed. But this time it's a VIB file.
6. I restarted it again, just now, and it's in the midst of created another VIB file (currently at ~3 GB, and reporting as 7% completed)

Here's the summary of the latest run:

Code: Select all

12/19/2019 8:29:06 AM :: Job started at 12/19/2019 8:29:04 AM  
12/19/2019 8:29:08 AM :: New copy interval started  
12/19/2019 8:29:32 AM :: Building list of machines to process  
12/19/2019 8:29:38 AM :: Total size: 3.1 TB  
12/19/2019 8:29:38 AM :: Processing Server2.MyCompany.tld Error: write: An existing connection was forcibly closed by the remote host
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
  
12/19/2019 8:29:38 AM :: Processing Server1.MyCompany.tld Error: write: An existing connection was forcibly closed by the remote host
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
  
12/20/2019 5:32:12 AM :: Total size: 3.1 TB  
12/20/2019 5:32:12 AM :: Processing Server2.MyCompany.tld Error: Failed to call RPC function 'CreateSessionTicket': The RPC server is unavailable. RPC function call failed. Function name: [DoRpc]. Target machine: [192.168.1.5:6162].  
12/20/2019 5:32:12 AM :: Processing Server1.MyCompany.tld Error: Failed to call RPC function 'CreateSessionTicket': The RPC server is unavailable. RPC function call failed. Function name: [DoRpc]. Target machine: [192.168.1.5:6162].  
12/20/2019 5:38:12 AM :: Total size: 3.1 TB  
12/20/2019 5:38:12 AM :: Processing Server2.MyCompany.tld Error: Failed to call RPC function 'CreateSessionTicket': The RPC server is unavailable. RPC function call failed. Function name: [DoRpc]. Target machine: [192.168.1.5:6162].  
12/20/2019 5:38:13 AM :: Processing Server1.MyCompany.tld Error: Failed to call RPC function 'CreateSessionTicket': The RPC server is unavailable. RPC function call failed. Function name: [DoRpc]. Target machine: [192.168.1.5:6162].  
12/20/2019 5:44:13 AM :: Total size: 3.1 TB  
12/20/2019 5:44:13 AM :: Processing Server2.MyCompany.tld Error: Failed to call RPC function 'CreateSessionTicket': The RPC server is unavailable. RPC function call failed. Function name: [DoRpc]. Target machine: [192.168.1.5:6162].  
12/20/2019 5:44:13 AM :: Processing Server1.MyCompany.tld Error: Failed to call RPC function 'CreateSessionTicket': The RPC server is unavailable. RPC function call failed. Function name: [DoRpc]. Target machine: [192.168.1.5:6162].  
12/20/2019 5:50:14 AM :: Total size: 3.1 TB  
12/20/2019 5:50:14 AM :: Processing Server2.MyCompany.tld Error: Failed to call RPC function 'CreateSessionTicket': The RPC server is unavailable. RPC function call failed. Function name: [DoRpc]. Target machine: [192.168.1.5:6162].  
12/20/2019 5:50:15 AM :: Processing Server1.MyCompany.tld Error: Failed to call RPC function 'CreateSessionTicket': The RPC server is unavailable. RPC function call failed. Function name: [DoRpc]. Target machine: [192.168.1.5:6162].  
12/20/2019 5:56:15 AM :: Total size: 252.8 GB  
12/20/2019 5:56:15 AM :: Processing Server1.MyCompany.tld Error: Failed to call RPC function 'CreateSessionTicket': The RPC server is unavailable. RPC function call failed. Function name: [DoRpc]. Target machine: [192.168.1.5:6162].  
12/20/2019 5:57:15 AM :: Total size: 2.8 TB  
12/20/2019 5:57:15 AM :: Processing Server2.MyCompany.tld Error: Failed to call RPC function 'CreateSessionTicket': The RPC server is unavailable. RPC function call failed. Function name: [DoRpc]. Target machine: [192.168.1.5:6162].  
12/20/2019 6:03:16 AM :: Waiting for copy interval expiration  
12/20/2019 7:38:16 AM :: Operation was canceled by user MyCompany\Administrator  
12/20/2019 7:38:19 AM :: Load: Source 0% > Proxy 7% > Network 99% > Target 0%  
12/20/2019 7:38:19 AM :: Primary bottleneck: Network  
12/20/2019 7:38:19 AM :: Job finished with error at 12/20/2019 6:03:16 AM  

So what does this all mean?

1. Is that 41 GB VBK file any good? It was a failed attempt at processing 3 TB.
2. If it's not good, does Veeam know it's not good?
3. Given that, is the 60 GB VIB file any good? Is it incremental to the 41 GB incomplete file or is it incremental to the 1.2 TB complete file?
4. Is that Storage Level Corruption Guard: Perform backup files health check

Or am I just doing this all wrong?

Thanks!

[HannesK]

Hello,
I did not find "backup4" in the picture, but I assume that it is the "Microserver Gen10 Win10/64 Pro" in the home office.

Heath check is not relevant for the backup copy process. So I will skip that one.

I seeded the backup drives by running this Backup copy job

Just to make sure: so your primary backup job is in reverse incremental mode and you tried this?

Because if you have a forward incremental chain (default), then you just need to copy the files to the target repository. Then you rescan the repository and you map the backup copy job to the existing files

For some reason it wants to start a full backup copy, not an incremental one.

I assume that you did not "map" the backup copy job.

Yes, Veeam knows if files are broken. For the initial transfer, the backup copy job can resume the transfer. You can find some more details here in the section after

If Veeam Backup & Replication fails to copy data for the full backup file during the first backup copy interval, it marks the job session as finished with the Warning status. During the next backup copy interval, Veeam Backup & Replication attempts to copy data for the full backup file in the following manner:

Best regards,
Hannes

[foggy]

Anyway, with that kind of errors we can see in the log excerpt, it is better to contact support and investigate the reasons of the failures. Thanks!

[TheWaterbug]

Ok. I'm going to kill this thread because I don't think this is the root cause of my overall problem. The overall problem is that my jobs are creating full backups instead of incremental backups. I'll start new thread(s) to address that.