Replication fails, known issue

[FrancWest]

Case #04145650.

It appears that when for some reason a replication job (partially) fails, you can’t retry it in V10. You’ll get the error: ‘Restore point found, but is older than previously replicated one’. This is a known issue, but no hotfix is available yet. The only workaround currently known to me is by creating a new backup, after that the replication works again.

[foggy]

Hi Franc, just to clarify the scenario, are you talking about replication from backup?

[FrancWest]

Correct.

[foggy]

Thanks for the clarification. I can confirm this is a known behavior that is actually not specific to v10 and can be reproduced on earlier versions as well. This is considered as an improvement for future versions, without any particular ETA at the moment though.

[FrancWest]

Ow ok, never seen this before with 9.5... The error message is also a bit strange, how can an existing restorepoint be older than the replicated one. That would mean a backup has been removed...

Also, what the correct procedure from recovering from this apart from waiting for new backups to arrive? Since waiting for a new backup would mean we don't have a full replication of that day.

[foggy]

The current message wording might not reflect the situation, I agree. Seems that there's no other way of recovering from this other than either waiting for the new backup or triggering it manually (at least for the affected VMs).

[FrancWest]

Is this bug fixed in V11?

[foggy]

Hi Franc, no, this was not addressed in v11.

[cerberus]

We are running into similar issue.

Using Veeam for replicating from backup source, if the replication job fails for any reason (such as network blip), when job re-tries to run we get "Restore point found, but is older than previously replicated one".

The only way to get it to re-replicate the failed VM is to make another backup at source.

The job can't retry the incomplete replica. This is quite concerning when replicating coast to coast, sometimes there will be network blips and we need ot be able to re-try failed jobs.

I am surprised this does not work, being able to retry replication jobs should be part of the core functionality of the replication process.

Foggy, is there an ETA on when this will be fixed?

[foggy]

No ETA atm, but the more requests we get, the higher the priority of the issue is.

[FrancWest]

Agreed, this is core functionality which should be working. I can’t imagine we are the only two persons experiencing this issue.

[ks.any.cloud]

+1 on that error.
We're having some customers that is experiencing this error, which is very unfortunate for us as a SP.

[cerberus]

We are using VBR with WAN accelerators to replicate a bunch of VMware VMs from Production to D/R site.

Whenever there is a network blip that causes the job to fail with "Error: Source WAN accelerator error: An existing connection was forcibly closed by the remote host.
Error in __int64 __cdecl api::CSocketIP::Write(const char *,unsigned __int64)", the replication retry job will fail with "Restore point found, but is older than previously replicated one.

We have to wait for the next backup job to run, before the replication can be retried. This is a crippling limitation with replication.

We can't replicate from live source as replication jobs run for many hours, keeping active snapshots for extended period is no good.

Network maintenance blips are bound to happen, we need replication functionality to be able to retry failed runs.

Can we get some much needed attention from Veeam on this? Is this on the roadmap to be fixed soon?

Code: Select all

Replicating restore point 7/7/2021 7:54:51 PM from backup repository MD1420-SSD 	
Queued for processing at 7/7/2021 8:15:09 PM 	
Required backup infrastructure resources have been assigned 	01:21:33
VM processing started at 7/7/2021 9:36:50 PM 	
VM size: 6.1 TB (4.7 TB used) 	
Discovering replica VM 	00:00
Using source WAN Accelerator VBRAccel01 	
Using target WAN Accelerator VBRAccel02 	
Preparing replica VM 	00:04
Processing configuration 	00:15
Creating helper snapshot 	00:03
Using target proxy VBRProxy01 for disk Hard disk 3 [hotadd] 	
Hard disk 3 (1.9 TB) 83.8 GB read at 6 MB/s	04:13:12
10.1 GB transferred over network, 123.5 MB obtained from global cache, 3.4 GB obtained from target repository 	
Using target proxy VBRProxy01 for disk Hard disk 2 [hotadd] 	
Hard disk 2 (1.9 TB) 39.4 GB read at 8 MB/s	01:23:10
2.5 GB transferred over network, 83.8 MB obtained from global cache, 1.4 GB obtained from target repository 	
Error: Source WAN accelerator error: An existing connection was forcibly closed by the remote host. Error in __int64 __cdecl api::CSocketIP::Write(const char *,unsigned __int64)  	
Busy: Source 1% > Source WAN 98% > Network 0% > Target WAN 98% > Target 16% 	
Primary bottleneck: Target WAN 	
Network traffic verification detected no corrupted blocks 	
Processing finished with errors at 7/8/2021 3:15:46 AM 	

Restore point found, but is older than previously replicated one 	00:00
Processing finished with warnings at 7/8/2021 3:33:38 AM 	

[veremin]

Can you tell us your support case for this issue? Thanks!

[cerberus]

Case #04788730

[veremin]

Thank you, let's wait and see what support engineer suggests after debug log review. Thanks1

[foggy]

Looks similar to the above, so I'm merging the threads.

[cerberus]

Is there anything in the works for this feature request foggy?

This is crippling our replication efforts, sending data between two IPSec sites over the public internet is bound to have network hiccups, we just need the jobs to be able to re-try without having to wait for next backup window.

[foggy]

We see the increasing demand for addressing this issue so will prioritize it accordingly. Thanks!

[18436572]

Hey we have this issue maybe 1 in 25 replications. We run backups then immediate copy jobs (over a site-to-site VPN) to a remote repository. Then the copy job has a post job script to run the replica job at this remote site. It works pretty well.

So for us that means the replica (from copy) is local to our site. The occasional replica failure is annoying at worst... our normal schedule usually repeats the three jobs before I notice and need to intervene.

[SAFA_IT]

I agree this should be fixed. As a VCC customer, almost all cases are related to this issue.

[Moebius]

+1 for fixing this issue. I am being hit by this problem as well.

[Massamb]

+1 for addressing this issue. We also suffer from this problem and it is very annoying

[FrancWest]

Is this fixed in 11a?

[foggy]

Not in v11a but this is already addressed in another code branch so should get into the next version.

[FrancWest]

Ok, and by next version you mean v12 or an intermediate update of v11?

[foggy]

Likely v12.

[jtupeck]

Customer of mine is experiencing this same issue, which only began about 4 months ago around the time they moved to v11 and implemented a set of ExaGrid's units as their primary SOBR repo extents. Initially, they were still doing backups to multiple SOBRs, so the replication jobs pulled the source data from both, but I just made the change today to exclude the old source repository now that it is decommissioned. I am having them submit a support case to look at these errors (and some others that may be related, more below) and will post their support ticket number when it comes in.

This 'Restore point found, but is older than previously replicated one' seems to happen mostly after a failed replication job where they see a number of VMs error out with any one of these error messages:

Code: Select all

Error: Application is shutting down. Agent failed to process method {Signature.StartReversedSignatureUpdateSession} Exception from server: Application is shutting down. 

Code: Select all

Error: No connection could be made because the target machine actively refused it IPADDRESS:2503

note: the port number here varies

Code: Select all

Error: Application is shutting down. Failed to upload disk. Skipped arguments: [diskfilename-flat.vmdk] Agent failed to process method {DataTransfer.SyncDisk}

Hoping support can figure this one out for them.

[foggy]

Hi Jason, yes, looks like the same issue offhand, we'll be able to confirm once we have a case open. Thanks!

[jtupeck]

Updating with their Support Case # 05272034. Standing by in case anything can be gleaned from the logs. Currently we are being told to look at AV, which we have removed from all related Veeam infrastructure. This includes the VBR server and both Proxies at each end of the replication path - though only one Proxy had CrowdStrike installed in the first place.