Host-based backup of VMware vSphere VMs.
Post Reply
Dave-Departed
Expert
Posts: 167
Liked: 6 times
Joined: Feb 29, 2012 3:11 pm
Contact:

Backup Copy Jobs failing

Post by Dave-Departed »

Hi All,

Raised a support request also: 00466465

Most of our backup copy jobs are failing with the following error message:

Code: Select all

Error: End of file --tr:Cannot read data from the socket. Requested data size: [24]. --tr:Receive file failed --tr:Wan connection failed to run command. --tr:Wan connection failed to get target file '{32b511b1-f35c-4fd4-a29d-6548975ef031}' --tr:Sync global cache failed. --tr:Disk processor failed to init global dedup. --tr:Disk processor failed to process disk. --tr:Sync task failed to process disk.Arguments
Our backup copy job for our Exchange DAG server fails with the following error:

Code: Select all

Processing '<DAG server>' Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond [IP of remote Veeam server]:2500 Processing '<DAG Server>' Error: The remote procedure call failed RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [IP of remote Veeam server]. Processing '<DAG server>' Error: Data transfer is currently restricted Job has failed unexpectedly
I've tried rebooting all of our Veeam servers (1 x full VBR and 1 x proxy at each end, all of which are acting as WAN accelerators) just in case, but no change.

Thanks in advance.

Dave
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Copy Jobs failing

Post by veremin »

Hi, Dave.

Have these issues occurred only recently or you have been having them ever since you installed version 7? Did you seed target cache or let the backup copy jobs to populate it, instead?

Anyway, based on the first error description, it looks like a problem with target WAN Accelerator/cache. Have you tried to re-deploy target WAN Accelerator and/or global cache?

Thanks.
Dave-Departed
Expert
Posts: 167
Liked: 6 times
Joined: Feb 29, 2012 3:11 pm
Contact:

Re: Backup Copy Jobs failing

Post by Dave-Departed »

Hi Vladimir,

What I did was firstly have the local and 'remote' VBR servers/WAN accelerators actually all on the local LAN, and run the BCJ's that way, to let the target WAN accelerator cache be populated in that manner. I then moved the 'remote' VBR server and WAN accelerator to their actual home, our remote site! The jobs seem to run with intermittent success, but have been failing with the described errors for over a week now (I was on annual leave, so wasn't aware until yesterday).

I've not tried to redeploy target WAN accelerator or global cache (not sure how to do the global cache anyway), because our offsite link is very slow (10mbps), and I imagine it would take an absolute age to repopulate it (we have around 40-60GB worth of CBT changes per day on our local backup jobs)?

Thanks

Dave
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Copy Jobs failing

Post by foggy »

Dave-Departed wrote:I've not tried to redeploy target WAN accelerator or global cache (not sure how to do the global cache anyway)
Dave, you can re-seed the WAN global cache using this procedure.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Copy Jobs failing

Post by veremin »

Yep, there is no way to re-populate global cache, apart from deleting cache or re-seeding it again. As to target WAN Accelerator, you can try to delete it, create a new one, and map it to the already existed global cache.

Thanks.
Dave-Departed
Expert
Posts: 167
Liked: 6 times
Joined: Feb 29, 2012 3:11 pm
Contact:

Re: Backup Copy Jobs failing

Post by Dave-Departed »

foggy wrote: Dave, you can re-seed the WAN global cache using this procedure.
Hi Foggy,

Where do I copy the GlobalCache\trg folder from? Out of the 4 WAN accelerators, I only have 2 which contain the \trg folder (which I assume stands for 'target', as the local WAN accelerators contain the \src folder instead?) which are the remote/target WAN accelerators. The 2 \trg folders don't have the same content.

Thanks

Dave
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Copy Jobs failing

Post by foggy » 1 person likes this post

I was talking about creating the cache locally and then seeding it to remote site to avoid copying data over a slow link. You can set the backup copy job locally (as you did previously) and then just copy the cache folder from the target WAN accelerator of the local job to the remote WAN accelerator server.
Dave-Departed
Expert
Posts: 167
Liked: 6 times
Joined: Feb 29, 2012 3:11 pm
Contact:

Re: Backup Copy Jobs failing

Post by Dave-Departed »

Support is reporting that it appears to be down to a loss of network connectivity, which I am going to investigate, but seems unlikely, given that our connection to our remote location has been up, stable and connected for 16 days now.
Dave-Departed
Expert
Posts: 167
Liked: 6 times
Joined: Feb 29, 2012 3:11 pm
Contact:

Re: Backup Copy Jobs failing

Post by Dave-Departed »

Quick update. I have configured local Backup Copy Jobs to recreate the Global Cache, as described above, and I am just about ready to take the cache down to our remote site to reseed it. However, one of the new BCJ's still won't run/complete correctly. Any guesses which it is? That's right, it's the one which targets an Exchange 2010 DAG member!

I appear to have solved the RPC error reported in my first post on this thread, it looks like this was being caused by the network adapters being in the wrong priority within the guest OS (Server 2008 R2, FYI), ie the NIC connecting to our SAN network was top in the list, whereas the NIC connecting to our Production network was 2nd in the list. Since changing this, I haven't seen the RPC error, so I am hopeful.

However, now the BCJ relating to the Exchange VM described above, fails with "Error: Data transfer is currently restricted". It's the only one that's now failing. Anyone know what this means?

Thanks

Dave
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Copy Jobs failing

Post by foggy » 1 person likes this post

Just to check the first guess, are there any periods when the job must not transport data defined at the Schedule step of the wizard?
Dave-Departed
Expert
Posts: 167
Liked: 6 times
Joined: Feb 29, 2012 3:11 pm
Contact:

Re: Backup Copy Jobs failing

Post by Dave-Departed »

foggy wrote:Just to check the first guess, are there any periods when the job must not transport data defined at the Schedule step of the wizard?
Yes, upon further investigation, it would seem that I had specified a schedule (I must've done it on auto-pilot), and that, because it was the first run of the job, it was taking a while, and therefore hit the end of the scheduled period before completing the data transfer. I've removed the schedule, and am trying it again.
tomsmi
Influencer
Posts: 18
Liked: 1 time
Joined: Jun 07, 2011 8:26 am
Full Name: Tomasz S.
Contact:

[MERGED] Active Interval Not Found

Post by tomsmi »

Any one stumbled upon this kind of error?
Image
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup Copy Jobs failing

Post by PTide »

@tomsmi

Hi,

Kindly check foggy's guess and if that won't help please open a support case and post you case ID.

Thanks
tomsmi
Influencer
Posts: 18
Liked: 1 time
Joined: Jun 07, 2011 8:26 am
Full Name: Tomasz S.
Contact:

Re: Backup Copy Jobs failing

Post by tomsmi »

All 24h are available for transfering data. I'll take a deeper look and eventualy open a case.
Post Reply

Who is online

Users browsing this forum: divertisity and 76 guests