VAW 3.0 constantly timing out during restore

[mobamoba]

For me, VAW 3.0 has been a total disaster and I actually had to revert to the only version I had saved somewhere (2.1 because prior versions are no longer available on the Veeam website). The issue is the VAW is incapable of restoring, at least for large datasets. At some point in the process, a notice comes up saying VAW is not detecting any activity and is timing out in xx minutes - even though it's actually in the middle of restoring data at the time!!! There's no way to stop this process or disable the timeout and, eventually, VAW shuts itself down thus destroying the restore. This was for a 300gig restore so I'm sure you can imagine how frustrating it was to have VAW shut itself off for no reason and with no way that I could see to at least disable timeouts so it would keep restoring even if it "thought" that it wasn't doing anything.

No clue how to debug this as, as I mentioned, I reverted versions because I absolutely need my data restored right now and can't wait until this problem is solved. I'm mentioning it here in the hopes that someone will be able to solve it and release a patch to stave off restoring pain for other users. Thanks.

[Gostev]

Well, the problem is that we need to understand the issue before we are able to deliver a patch... and to understand the issue, we need someone to let us take look at the restore logs capturing the issue. It goes without saying that our QC tests restores a lot and we did not see this issue in our test labs, so this might be some environment-specific culprit in your case that we will never be able to reproduce internally. Meaning, it can potentially remain not fixed forever!

This is why we would highly appreciate that if you run into such issues in future, you submit a support case before rolling back to an earlier version, so that the debug logs that capture the issue are not lost. Thank you in advance!

[mobamoba]

Sorry - I'm sure you can imagine when restoring that much data, my prime issue was data recovery and bug reporting came after. At the very least: can you guys add an option to disable the timeout so it just keeps restoring even if it thinks it's idle?

[mobamoba]

Also: unless the uninstall and reinstall overwrote them, I probably still have the logs from the VAW 3.0 failure. I'm happy to email them or upload them somewhere if you think it would help debug.

[mobamoba]

Here's a screenshot of the timeout happening as I'm restoring (this on a different computer than the one before and using VAW 3).
https://photos.app.goo.gl/gfHobmuGPMeMmvYGA

[Gostev]

Actually, the screenshot helps a lot... am I right thinking that you used "Open in Explorer" functionality, and are copying those files using Windows Explorer, meaning that Backup Browser does not see any file restore activity through its interfaces?

Also, what happens if you click Cancel? This should stop the time out process.

[mobamoba]

Yes, exactly. But this means I have to sit here and nurse my restore by clicking "cancel" every 30 minutes meaning I can't let it run overnight or leave it open if I'm picking through particular files. I'm not sure what use-case requires a timeout shutting down Veeam when copying from the FLR, but in my use case, it makes the process incredibly frustrating. Is there a way to disable it or give me a link to the previous version which didn't have this feature? Thanks.

[chaycock]

I agree, most definitely, there should be an option to disable that behavior. I also don't really see the value in having it in the first place.

[Dima P.]

Hi folks,

Thank you for the provided details. We will investigate file level restore issue with RnD team. I'll update this thread once I have any details to share.

[Dima P.]

Hi folks,

Confirmed that the logic was not changed in latest version, so behavior should be the same for all versions. We will keep the investigation running and will try to find out the possible way to optimize the existing time out logic. Meanwhile, there a workaround to increase the File level restore timeout with the following key:

Code: Select all

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Endpoint Backup
InactiveFLRSessionTimeout DWORD Value 

Value is set in seconds.

[mobamoba]

Okay thanks. What's the maximum value I can enter?

[gonzo99]

I would assume zero for no timeout.

BTW, restoring using the "Copy to" feature works without nursing. Two days ago I restored 5,5tb data from a NAS to a 2016 server over night without any user action.

[Gostev]

Right, I was particularly puzzled why OP didn't just use native functionality, and instead went with a non-integrated 3rd party file management tool...

[mobamoba]

Because Windows File Copy has a number of nags when it comes to system files for example (asking if the user is sure they want to copy a system file) that stop the backup whereas Veeam's Overwrite feature does exactly without my having to pay attention. Well until the timeout thing. Why wouldn't I use the FLR that came with Veeam?

[Dima P.]

gonzo99,
Thanks for confirmation!

mobamoba,

What's the maximum value I can enter?

This value represents when to prompt for notification, so setting it to zero wont work (i.e. to set the timeout to 1 hour you should use 3600 value as it's calculated in seconds). No aware of any limit, so you can test it with any value you want. Cheers!

[ald.plan]

Hello,

We have Veeam Agent for Windows 3.0.0.748 installed on a File server (Windows Server 2008 R2). We backup the entire server and the size of the vbk file is about 4 TB.

When we try to restore one file or directory via the File Level Restore, it's running for more than an hour and a half and then stop with an error message saying the file level recovery session has been inactive for ~50 minutes and will close automatically in XX to remove locks from the backup files.

Is there any limit in the number of files contained in the backup files to be able to restore selected files or directories?
Should we split or backup job into several backup jobs of lower size?

Thanks in advance.

[PTide]

Hi,

Kindly try to use the registry key first. If that doesn't help, then please open a case with the support and post your case ID.

Thanks!

[elod]

I would assume zero for no timeout.

BTW, restoring using the "Copy to" feature works without nursing. Two days ago I restored 5,5tb data from a NAS to a 2016 server over night without any user action.

This is simply not the case.
I'm using the "Copy to" feature on Agent v3.0.2.1170. It simply times out in the middle of a restore.

[Dima P.]

Hello elod,

Can you please provide us more information about your issue? How much data you are trying to restore? When exactly operation fails? Any screenshots are appreciated. Cheers!

[cantsitstill]

Hello, we have same v 3.0.2.1170 and same problem with the time outs during file level restores. Registry key does not fix the problem. This makes the agent pretty useless for us and without a solution will need to find another product - is there a plan to fix this??

Thanks

[Dima P.]

Hello Stuart,

Do you have a case ID to share? Can you please describe your restore scenario in more details (i.e. size and average amount of files you are restoring, when specifically backup browser times out) Thank you in advance!

[Garf55]

Another month passes by since the last posting. Same problem for me - 3.0.2.1170 with the registry key added and about 35 minutes in it just stops, losing connectivity to the source "mounted" backup drive. I have to wonder how much testing went into this client, as this is a fundamentally broken product. This thread goes back to February this year.

[wishr]

Hi Garf,

Sad to hear you got into this situation.

Could you please share your support case ID with us?

Thanks

[jcm888]

Until there's a patch for this, you can use this autohotkey macro to cancel the popup:

Code: Select all

Loop
{
If WinExist("Session Inactive")
	{
		WinActivate
		ControlSend, Button2 , {space}, Session Inactive ; Press the 'Cancel' button
	}
}

[Staj]

Just got hit by this bug too; have terrabytes to restore over a non-ideal network link so now it has to be babysit the whole time over 24 continigious hours by a macro or person. The whole point of an Inactivity Timeout is to timeout after inactivity, is long process of restoring data not activity? Who coded this idiotic logic? We weren't aware of this issue so operator thought it just completed and closed, not knowing it never actually finshed restore, I'm glad our QA is better than yours or else we wouldn't have noticed.

[Dima P.]

Hello Staj,

have terrabytes to restore over a non-ideal network link

Can you please clarify what type of content you were restoring via File Level Recovery? Thanks!

[Egor Yakovlev]

This behaviour will be altered in next agent version and active restore sessions(data moves) will not timeout anymore.
Until then, please bear with Registry Key mentioned above.
/Thanks!

[ucnet]

hi Veeam-Team,
'altered in next agent version' ? how about 4.0.0.1811 ? is it fixed there ?`
TIA
Marc

[billrowland]

I have read through this thread and I do not see a successful resolution. I have not had to restore from backup since installing the software. Basic backup. Keeps timing out. Added registry entry as stated above and it went from 30 minutes to 10 Minutes now after it was almost fully restored (4.5 hours). I set the registry entry to "0". What is up with the software? Running version 3.0.0.1170

[Gostev]

What type of restore are you performing, and how do you perform the restore?

The issue discussed here so far is with using 3rd party tools to perform the restore, so it's not really an issue with the product per se (product functionality is NOT used, so the product thinks nothing is going and releases backup files to ensure backups can continue successfully).