Comprehensive data protection for all workloads
Post Reply
danielchung
Enthusiast
Posts: 41
Liked: 1 time
Joined: Apr 24, 2012 3:50 am
Full Name: Daniel Chung
Contact:

Backup copy job stuck at some point

Post by danielchung »

I have Veeam B&R 7 R2, using a backup copy job to copy 1TB backup to offsite repository over a slow VPN link. Recently the link went unstable (perhaps the great firewall in China interfering the VPN), so I tweaked the number of retries from 5 to 150.

I hoped that increasing the number of retries would overcome the issue of the disconnection, but got the below issues:

1. The job stuck in 35% (14% of a particular VM) for unknown reasons, it doesn't look progressing even the whole copy interval ended and over 100 of retries.
2. The amount of backup data Read, Processed and Transferred shown in GUI keep growing but the actual backup file size doesn't grow relatively. Says 50GB transferred but only got 10GB VIB file. The file modified time will update when the job retry but the file size won't change.

I have successfully copied a vbk from the source backup locally using a backup copy job for offsite seeding, so I think there is no problem in the source side. I'm also wondering how the job handle retries? Does it resume reading/transferring data of the disk at the point where it fails? Says I have 10GB incremental change in the last restore point. The job has read and transferred 3GB data, then failed due to link disconnection, then it retries and resumes transferring the data right after the point of failure (3GB)? or just re-reading the disk at the beginning?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Backup copy job stuck at some point

Post by dellock6 »

Daniel, is it 3GB the limit or maybe 4 GB? Chances the destination folder is hosted on a FAT32 file system? This has a 4 GB limit in file size, happen exactly the same at a customer, with backup copy failing again and again everytime the file was reaching 4 GB....

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
danielchung
Enthusiast
Posts: 41
Liked: 1 time
Joined: Apr 24, 2012 3:50 am
Full Name: Daniel Chung
Contact:

Re: Backup copy job stuck at some point

Post by danielchung »

Luca, the repository is a 20TB NTFS partition. It already hosted many backup files greater than 100GB.
Anyway, have opened a case "00497590" for this issue.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup copy job stuck at some point

Post by foggy »

Indeed, the best way to address such issues is contacting support. They will be able to locate the particular operation the job is hanging on, via reviewing the logs.
danielchung wrote:I'm also wondering how the job handle retries?
Depends on the backup copy job mode you're using. I assume it is WAN accelerated job in your case, isn't it? In this case it should resume transferring from where it had stopped.
danielchung
Enthusiast
Posts: 41
Liked: 1 time
Joined: Apr 24, 2012 3:50 am
Full Name: Daniel Chung
Contact:

Re: Backup copy job stuck at some point

Post by danielchung »

The support reviewed the log and had webex session, suggested to re-copy the vm data from scratch and mentioned that the restries/resume is not "reliable" without the WAN accelerator over a slow link.

I hope I can have it but unfortunately, I'm using standard version which has no WAN accelerator. :(
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup copy job stuck at some point

Post by foggy »

Yep, the direct backup copy job mode is not that resistant to major connection drops as was designed primarily for usage on local networks.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup copy job stuck at some point

Post by Gostev »

Backup Copy in direct mode does have retry+resume functionality, but it does not "fight" for too long trying to re-establish lost network connection, as it expects to be running over local network, where network down times are not typical (less likely than other errors) and very short when happen.

WAN accelerated mode is opposite to that.
danielchung
Enthusiast
Posts: 41
Liked: 1 time
Joined: Apr 24, 2012 3:50 am
Full Name: Daniel Chung
Contact:

Re: Backup copy job stuck at some point

Post by danielchung »

Thanks for the explanation. Actually the connection drop last for less than 10 seconds each time, have contacted network provider to check it anyway.

Still want to know why the data didn't get written into the backup file while the value of transferred data claimed it has.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup copy job stuck at some point

Post by foggy »

danielchung wrote:Still want to know why the data didn't get written into the backup file while the value of transferred data claimed it has.
I suppose this is due to the data is sent (transferred) from the source side but does not reach the target in its entirety due to connection drop (data is written into VBK file in a transactional manner, by blocks).
danielchung
Enthusiast
Posts: 41
Liked: 1 time
Joined: Apr 24, 2012 3:50 am
Full Name: Daniel Chung
Contact:

Re: Backup copy job stuck at some point

Post by danielchung »

But when the problem occurs, the target repository server is receiving data from the source backup proxy server as shown in the task manager. :roll:
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 317 guests