Backup copy job stuck at some point

[danielchung]

I have Veeam B&R 7 R2, using a backup copy job to copy 1TB backup to offsite repository over a slow VPN link. Recently the link went unstable (perhaps the great firewall in China interfering the VPN), so I tweaked the number of retries from 5 to 150.

I hoped that increasing the number of retries would overcome the issue of the disconnection, but got the below issues:

1. The job stuck in 35% (14% of a particular VM) for unknown reasons, it doesn't look progressing even the whole copy interval ended and over 100 of retries.
2. The amount of backup data Read, Processed and Transferred shown in GUI keep growing but the actual backup file size doesn't grow relatively. Says 50GB transferred but only got 10GB VIB file. The file modified time will update when the job retry but the file size won't change.

I have successfully copied a vbk from the source backup locally using a backup copy job for offsite seeding, so I think there is no problem in the source side. I'm also wondering how the job handle retries? Does it resume reading/transferring data of the disk at the point where it fails? Says I have 10GB incremental change in the last restore point. The job has read and transferred 3GB data, then failed due to link disconnection, then it retries and resumes transferring the data right after the point of failure (3GB)? or just re-reading the disk at the beginning?

[dellock6]

Daniel, is it 3GB the limit or maybe 4 GB? Chances the destination folder is hosted on a FAT32 file system? This has a 4 GB limit in file size, happen exactly the same at a customer, with backup copy failing again and again everytime the file was reaching 4 GB....

Luca.

[danielchung]

Luca, the repository is a 20TB NTFS partition. It already hosted many backup files greater than 100GB.
Anyway, have opened a case "00497590" for this issue.

[foggy]

Indeed, the best way to address such issues is contacting support. They will be able to locate the particular operation the job is hanging on, via reviewing the logs.

I'm also wondering how the job handle retries?

Depends on the backup copy job mode you're using. I assume it is WAN accelerated job in your case, isn't it? In this case it should resume transferring from where it had stopped.

[danielchung]

The support reviewed the log and had webex session, suggested to re-copy the vm data from scratch and mentioned that the restries/resume is not "reliable" without the WAN accelerator over a slow link.

I hope I can have it but unfortunately, I'm using standard version which has no WAN accelerator.

[foggy]

Yep, the direct backup copy job mode is not that resistant to major connection drops as was designed primarily for usage on local networks.

[Gostev]

Backup Copy in direct mode does have retry+resume functionality, but it does not "fight" for too long trying to re-establish lost network connection, as it expects to be running over local network, where network down times are not typical (less likely than other errors) and very short when happen.

WAN accelerated mode is opposite to that.

[danielchung]

Thanks for the explanation. Actually the connection drop last for less than 10 seconds each time, have contacted network provider to check it anyway.

Still want to know why the data didn't get written into the backup file while the value of transferred data claimed it has.

[foggy]

Still want to know why the data didn't get written into the backup file while the value of transferred data claimed it has.

I suppose this is due to the data is sent (transferred) from the source side but does not reach the target in its entirety due to connection drop (data is written into VBK file in a transactional manner, by blocks).

[danielchung]

But when the problem occurs, the target repository server is receiving data from the source backup proxy server as shown in the task manager.