Question on Backup Copy Job Intervals and Restarting After Error

[NerfHerder]

Hi,
I am troubleshooting a backup copy job that failed with a "Socket has been forcibly closed". message. The target repo is remote, so we are going over the Internet to it.

My question is, when this occurs, since the backup copy job interval is 3 days... that means that (since my jobs just started today) that it will wait for 3 days to try again? Is there a method to force it to start again sooner?

Thanks!

[HannesK]

Hello,
asking the other way round... what did you see after some time?

Is there a method to force it to start again sooner?

did you already try the "sync now" button? https://helpcenter.veeam.com/docs/backu ... ml?ver=100

Best regards,
Hannes

[foggy]

Also, what type of repository is it? Just in case it is CIFS, make sure you have a gateway server for it on the remote side.

[NerfHerder]

Hi,
Thanks for the replies.

We are running VBR 9.5u4.
In our local environment, all Veeam components are installed on a single backup server, and the storage / repo is locally attached disk.
Our Veeam partner provides us with a cloud repo that we are using for offsite and longer term retention storage.

I tried the "sync now" option about 30 minutes ago. It started everything over from scratch. I guess I wasn't sure if it would pick up where it left off as far as copying the restore points to the cloud repo. It appears not, it is starting all over again. We'll see how it goes.

I feel like I am still not fully understanding how this works and how to administer it though.
I have a (Windows agent, in this case) backup job that backs up to the local repo and I am retaining 3 restore points.
I have a backup copy job that uses the job above as the source, with an interval of 3 days.
The backup copy job is set to retain 14 restore points.
I set it this way (the interval) because of what I read here: https://helpcenter.veeam.com/archive/ba ... liary.html
Since the initial seed for this job would consist of about 400GB, I just figured the copy job could achieve that in its interval, so I didn't do anything else special to seed like the article describes... but the article is the only place I have come across so far that explained to be the requirements around how to set intervals for copy jobs and why.

So, on the very first run of this copy job, after an hour or two, the job fails (connection forcibly closed).
Now, if I wait for the copy job to hit its next Interval to try again, I will be waiting just about 3 days. This seems wrong to me... like I should intervene before that to get the job going and get the backup data offsite. However, when I force it to start again - using the sync button - all previous progress is discarded and the copy job starts from scratch.

Is there something that I am not seeing or understanding correctly?

Thanks!

[foggy]

There are the number of retries so it doesn't just drops the ball and waits. It also doesn't discard the previous progress - when the new interval is started (or when you click Sync Now) it creates a new restore point based on the latest VM state available on source. The blocks that has already been transferred will not be re-transferred again.

[NerfHerder]

Hi,
Thanks for the reply.
As I mentioned, I did try the "sync now" option this morning. The job ran for a while and then failed. It did 5 retries as well (as below). So since I started a new interval today (with sync now), and it performed 5 retries, and the job's interval is set to 3 days.... I would ultimately be waiting about 3 days for it to try again at this point? Is that correct?

Code: Select all

9/2/2020 8:59:56 AM :: Copying restore point 9/1/2020 9:00:41 PM from backup repository Backup Repository 1  
9/2/2020 8:59:56 AM :: Queued for processing at 9/2/2020 8:59:56 AM  
9/2/2020 8:59:59 AM :: Required backup infrastructure resources have been assigned  
9/2/2020 9:00:03 AM :: VM processing started at 9/2/2020 9:00:03 AM  
9/2/2020 9:00:03 AM :: VM size: 450.4 GB  
9/2/2020 9:00:10 AM :: Storage initialized  
9/2/2020 9:00:28 AM :: Network traffic will be encrypted  
9/2/2020 9:00:47 AM :: Saving digest_e2fefd68-b117-430c-9923-c446c7bc8f5f  
9/2/2020 9:00:49 AM :: Saving GuestMembers.xml  
9/2/2020 9:00:51 AM :: Saving BackupComponents.xml  
9/2/2020 9:01:03 AM :: Hard disk 1 (500.0 GB) 63.4 GB read at 6 MB/s 
9/2/2020 12:11:12 PM :: Error: Failed to obtain storage quota. Waiting has timed out.
Failed to download disk.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
  
9/2/2020 12:11:16 PM :: Busy: Source 2% > Proxy 0% > Network 99% > Target 0%  
9/2/2020 12:11:16 PM :: Primary bottleneck: Network  
9/2/2020 12:11:17 PM :: Network traffic verification detected no corrupted blocks  
9/2/2020 12:11:18 PM :: Processing will be retried (retry 1 out of 5)  
9/2/2020 12:12:10 PM :: Copying restore point 9/1/2020 9:00:41 PM from backup repository Backup Repository 1  
9/2/2020 12:12:10 PM :: Queued for processing at 9/2/2020 12:12:10 PM  
9/2/2020 12:12:15 PM :: Required backup infrastructure resources have been assigned  
9/2/2020 12:12:18 PM :: VM size: 450.4 GB  
9/2/2020 12:12:23 PM :: Error: Socket has been forcibly closed. Failed to recover connection.  
9/2/2020 12:12:26 PM :: Busy: Source 2% > Proxy 0% > Network 99% > Target 0%  
9/2/2020 12:12:27 PM :: Primary bottleneck: Network  
9/2/2020 12:12:28 PM :: Network traffic verification detected no corrupted blocks  
9/2/2020 12:12:29 PM :: Processing will be retried (retry 2 out of 5)  
9/2/2020 12:13:18 PM :: Copying restore point 9/1/2020 9:00:41 PM from backup repository Backup Repository 1  
9/2/2020 12:13:18 PM :: Queued for processing at 9/2/2020 12:13:18 PM  
9/2/2020 12:13:26 PM :: Required backup infrastructure resources have been assigned  
9/2/2020 12:13:32 PM :: VM size: 450.4 GB  
9/2/2020 12:13:37 PM :: Error: Socket has been forcibly closed. Failed to recover connection.  
9/2/2020 12:13:39 PM :: Busy: Source 2% > Proxy 0% > Network 99% > Target 0%  
9/2/2020 12:13:40 PM :: Primary bottleneck: Network  
9/2/2020 12:13:40 PM :: Network traffic verification detected no corrupted blocks  
9/2/2020 12:13:42 PM :: Processing will be retried (retry 3 out of 5)  
9/2/2020 12:14:24 PM :: Copying restore point 9/1/2020 9:00:41 PM from backup repository Backup Repository 1  
9/2/2020 12:14:24 PM :: Queued for processing at 9/2/2020 12:14:24 PM  
9/2/2020 12:14:35 PM :: Required backup infrastructure resources have been assigned  
9/2/2020 12:14:43 PM :: VM size: 450.4 GB  
9/2/2020 12:14:47 PM :: Error: Socket has been forcibly closed. Failed to recover connection.  
9/2/2020 12:14:49 PM :: Busy: Source 2% > Proxy 0% > Network 99% > Target 0%  
9/2/2020 12:14:49 PM :: Primary bottleneck: Network  
9/2/2020 12:14:50 PM :: Network traffic verification detected no corrupted blocks  
9/2/2020 12:14:51 PM :: Processing will be retried (retry 4 out of 5)  
9/2/2020 12:15:30 PM :: Copying restore point 9/1/2020 9:00:41 PM from backup repository Backup Repository 1  
9/2/2020 12:15:30 PM :: Queued for processing at 9/2/2020 12:15:30 PM  
9/2/2020 12:15:34 PM :: Required backup infrastructure resources have been assigned  
9/2/2020 12:15:38 PM :: VM size: 450.4 GB  
9/2/2020 12:15:42 PM :: Error: Socket has been forcibly closed. Failed to recover connection.  
9/2/2020 12:15:44 PM :: Busy: Source 2% > Proxy 0% > Network 99% > Target 0%  
9/2/2020 12:15:44 PM :: Primary bottleneck: Network  
9/2/2020 12:15:45 PM :: Network traffic verification detected no corrupted blocks  
9/2/2020 12:15:46 PM :: Processing will be retried (retry 5 out of 5)  
9/2/2020 12:16:36 PM :: Copying restore point 9/1/2020 9:00:41 PM from backup repository Backup Repository 1  
9/2/2020 12:16:36 PM :: Queued for processing at 9/2/2020 12:16:36 PM  
9/2/2020 12:16:38 PM :: Required backup infrastructure resources have been assigned  
9/2/2020 12:16:42 PM :: VM size: 450.4 GB  
9/2/2020 12:16:47 PM :: Error: Socket has been forcibly closed. Failed to recover connection.  
9/2/2020 12:16:49 PM :: Busy: Source 2% > Proxy 0% > Network 99% > Target 0%  
9/2/2020 12:16:49 PM :: Primary bottleneck: Network  
9/2/2020 12:16:50 PM :: Network traffic verification detected no corrupted blocks  
9/2/2020 12:16:50 PM :: Processing finished with errors at 9/2/2020 12:16:50 PM  
9/2/2020 12:16:52 PM :: Maximum retry count reached (5 out of 5)

[foggy]

If you try Sync Now once again, it will start a new interval. I would, however, rather investigate the connection issue with your partner first since seems just starting again wouldn't fix the original problem.