Indexing Failures - Not reported in console

[adb98]

All I have been having an issue that we found only last week but am unsure of when it started. Long story short... When we are in Enterprise Manager and go to do an FLR we normally have indexing going so we don't have to mount the image (takes a while with Data Domain). Well we found that the indexing is not always working. When we click on a restore point at random times we get file not found and we only have the option to mount the image. This was weird to us as we don't get any errors for the job as in failed indexes or any failures for that matter.

Going thru it with support, we went down several rabbit holes so far.

1. The first thing they stated was that we needed to disable the linux indexing as this is windows. Did that though I doubted that was the error. Same issue.

I then started going thru the logs myself and found the issue but I am still very stuck on how to fix it. I thought as you will see in my support case that I had it fixed but it is still doing the same thing. The error I am seeing in the log is as follows:

Code: Select all

17.10.2016 21:09:25] <25> Error    Failed to copy file from proxy. Attempt 0 of 10
[17.10.2016 21:09:25] <25> Error    Access to the path '\\192.168.248.9\ADMIN$\veeam_{e61f83bd-0de6-4046-83b6-f3f052d3299d}' is denied. (System.UnauthorizedAccessException)
[17.10.2016 21:09:25] <25> Error       at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
[17.10.2016 21:09:25] <25> Error       at System.IO.FileSystemEnumerableIterator`1.CommonInit()
[17.10.2016 21:09:25] <25> Error       at System.IO.FileSystemEnumerableIterator`1..ctor(String path, String originalUserPath, String searchPattern, SearchOption searchOption, SearchResultHandler`1 resultHandler, Boolean checkHost)
[17.10.2016 21:09:25] <25> Error       at System.IO.Directory.GetFiles(String path)
[17.10.2016 21:09:25] <25> Error       at Veeam.Backup.VssProvider.CIndexingSessionRemote.IndexGetState(Boolean& bComplete, UInt32& completePercent)
[17.10.2016 21:09:26] <25> Error    Failed to copy file from proxy. Attempt 1 of 10
[17.10.2016 21:09:26] <25> Error    Access to the path '\\192.168.248.9\ADMIN$\veeam_{e61f83bd-0de6-4046-83b6-f3f052d3299d}' is denied. (System.UnauthorizedAccessException)
[17.10.2016 21:09:26] <25> Error       at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
[17.10.2016 21:09:26] <25> Error       at System.IO.FileSystemEnumerableIterator`1.CommonInit()
[17.10.2016 21:09:26] <25> Error       at System.IO.FileSystemEnumerableIterator`1..ctor(String path, String originalUserPath, String searchPattern, SearchOption searchOption, SearchResultHandler`1 resultHandler, Boolean checkHost)
[17.10.2016 21:09:26] <25> Error       at System.IO.Directory.GetFiles(String path)
[17.10.2016 21:09:26] <25> Error       at Veeam.Backup.VssProvider.CIndexingSessionRemote.IndexGetState(Boolean& bComplete, UInt32& completePercent)
[17.10.2016 21:09:26] <57> Info                   [AP] (1087) output: --pex:81;1335910989824;12721324032;1323185471488;12721324032;12717338656;83;7;89;11;6;94;131212265669310000
[17.10.2016 21:09:26] <57> Info           Task progress: '81%', '1,405,704,208,384' of '1,719,060,660,224' bytes, object '0' of '14'
[17.10.2016 21:09:26] <57> Info           Job progress: '96%', '10,113,060,438,016' of '10,426,416,889,856' bytes, object '4' of '6'
[17.10.2016 21:09:27] <25> Error    Failed to copy file from proxy. Attempt 2 of 10

Nasty part about this is there has to be a bug in the code as we don't get any failures in the job from the console. You only know you have an issue when you go to pull a restore point using FLR. It expects a file but it never got transfered so it errors saying file not found.

So this comes down to the Guest Interaction proxy. Here is the weird thing. I have several VMs use the same guest interaction proxy to process and 2 out of 3 will go thru and the 3rd will fail. Same proxy. VMs on the same network.

2. I validated that VBR and proxies can ping each other. I changed the proxies from IPs to DNS names that resolve to our external network. Had the same issue.

3. The last thing I thought was the fix was finding the AD account I was using was in an OU in which we had run removed. This removes the ability to UNC. I fixed that and ensured using that account, I could UNC from proxy to proxy and proxy to VBR. Well same issue. Kinda running out of ideas to fix this.

Ticket is still open with support but any help would be appreciated.


Case # 01924775

[PTide]

Hi,

Just to double-check - did you try to press the "Test Now" button at the "Guest Processing" step of the wizard?

Thanks

[adb98]

Yes and all was green and successful. Also in the jobs there is no errors and it says indexing was successful but clearly in the log it is not. I wonder how many people may be having this issue but it is not being reported and if you never have to do a FLR and you don't hit a restore point where it failed, you won't know its an issue.

[foggy]

Are you positive all 10 attempts to copy file from proxy failed (per the log file)?

[adb98]

Hi Foggy. Yes I am positive.

I have an update to this. I ended up changing the backup service to run as the service account user we are using for indexing and so far so good. Not going to say its fixed but so far the job I am testing with seems to be holding. We noticed this with our file server job that runs every two hours. After we found the issue we found that at least 1 VM would fail each time we backed up and the Guest Interaction proxy didn't matter. We could have 3 VMs use the same Guest Interaction proxy and all but 1 would fail with the file transfer error in the log and file not found when pulling up the restore point.

If changing the service fixes this then there has to be something not passing creds right. There is no UAC turned on as we turn this off at the policy level in the OU. Also UNC is working as I tested it by logging in as the service account on each proxy and the VBR.

[foggy]

Please let your engineer know about this, probably deeper look at the original issue is required.