How to speed up guest file indexing?

[bma-bs]

Hi,

when taking a backup from an w2008r2 file server then guest file system indexing take about an hour every time.

There are about 3 millon files on that server, indexing or windows search are not installed.

What can I do to speed up the indexing?

Best regards

Nico

[foggy]

Nico, typically, VM guest OS indexing is performed nearly instantly as index data is obtained from NTFS MFT directly (rather than by scanning the entire file system). Though it really depends on the number of files stored in the VM. If we are talking about millions of files, it could take a while to get, parse and write their descriptions. Nevertheless, I would suggest opening a case just to make sure that the delay is not caused by something else.

[chrisdearden]

I've seen big file servers take that sort of time before. Worth noting that the indexing process runs within the Guest itself , so make sure there is plenty of C: space free - its also worth checking that the VM isn't CPU starved and has an AV exception on the Veeam VSS process.

[Esmond]

I have a Windows Server 2008 VM which has around 800GB data (mainly are MS Office document, PDF and JPG files) running on ESXi 5.1, the datastore are running on IBM Storage connected by iSCSI. When we start the backup job, it take around 3-4 hrs for the Guest File Indexing process no matter in full or incremental backup. Is it normal to take that long for Indexing? If so, any method to shorten the index time?

Thanks in advance for any help.

Esmond

[veremin]

Hi, Esmond,

As mentioned above, the processing time typically depends on number of files stored inside VM. Processing might take a while, if this number is huge.

As to tricks that can shorten indexing time, you can check the space on disk C:\, the amount of CPU given to the VM in question, etc. If everything looks good, you can open a ticket with our support team and let them confirm your environment.

Thanks.

[Esmond]

Hi, Esmond,

As mentioned above, the processing time typically depends on number of files stored inside VM. Processing might take a while, if this number is huge.

As to tricks that can shorten indexing time, you can check the space on disk C:\, the amount of CPU given to the VM in question, etc. If everything looks good, you can open a ticket with our support team and let them confirm your environment.

Thanks.

The c drive has 12GB free space, and the CPU speed and utilization were low during the indexing (in both the guest level and host level). For the indexing, does it build a new set each time or merge to the index from previous backup session?

[veremin]

The new session creates new backup indexes that are stored in accordance to the retention settings. The retention settings of stored indexes depend on VB&R edition you're using. In case of Standard edition, indexes are kept for all backups currently present in backup repositories. In case of Enterprise edition, EM keeps track of indexes of archived backups, as well. (tapes, etc.). For more information kindly see this section of our Help Center.

Or you're asking whether or VB&R indexing is incremental or not?

Thanks.

[gveat]

The only thing holding me back from dumping DPM is the speed of the Guest Indexing.
On an incremental backup of a 1TB HDD, the backup takes less than 2 minutes. The guest indexing takes 22 hours.
There has to be a way to speed this thing up.

[PTide]

Hi,

Just to make sure we're on the same page - do you use DPM to backup Windows VM via VMware snapshot?

The guest indexing takes 22 hours.

Normally it should not take that long. Would you give an estimate of the number of files that you have on the system?

Thanks

[gveat]

do you use DPM to backup Windows VM via VMware snapshot?

I use DPM to do data protection. We take a snapshot via vss every 6 hrs.

Would you give an estimate of the number of files that you have on the system?

We have Trillions of files on this particular machine.

I also mistyped on my original post we are backing up 10TB not 1TB.

We do however have two servers that do incremental backups on 1TB every 6 hours that take almost an hour to index. (still in my opinion too long, especially since there is usually less than a 10GB of change in the data.

[EricJ]

We have a similar issue with two of our large file servers. We are a school district, and our student file servers were much slower to index than staff file servers, despite containing less data.

From combing through the logs, it appears that part of the guest indexing process includes reading the NTFS metadata file called "$Secure", which contains the SIDs of accounts who are owners of files on the volume. It appears that Windows never purges this file of old SIDs that no longer exist or own any files.

There are two things working against us in this situation:

1. Our student file servers see a large amount of turnover. Each school year, thousands of accounts are deleted, and thousands of new accounts are created - all of which are file/folder owners on these file servers. This means that our $Secure file still contains tens of thousands of SIDs that are unresolvable.

2. The scan of SIDs in the $Secure file appears to slow down as time goes on. At the start, the indexing process is looking up over 100 SIDs per second. After a few hours, the rate has dropped to just 1-2 SIDs per second.

To confirm this was the issue, I performed a test. I restored all of the files from one of our student file servers to a brand new NTFS volume on another server. Because this volume was new, the $Secure file was empty, and after restoring the files and permissions, only the current file owners were inserted in the $Secure file.

On our production file server, the guest indexing did 70,365 SID lookups, with 48,948 unresolvable, and the process took 10.5 hours. The guest indexing on the new volume (with identical files) did only 9,445 SID lookups, with 75 unresolved, in only 7 minutes. This is 90 times faster!

Support didn't have a good solution for this issue, and in the end we decided not to index student file servers. When we refresh the server OS and migrate data to a fresh volume I may try enabling indexing again. Just thought I'd tell our story in case it matches your situation.

[PTide]

Greg,

I suggest you to contact our support team directly, so they can look closer, and post your case ID.

Eric,

Would you post your case ID please so I could obtain the details from the case and push it to the dev team?

Thank you

[EricJ]

Here's the case info: [ID# 02031518] Guest File Indexing Very Slow

Thanks for forwarding it on!

[dwrice0]

Was a solution ever found for this? I'm seeing slow indexing on file servers as well.

For example, a 6.5TB file server (Windows 2012 R2) with low change rate (19.4 GB on this incremental backup) had a ~21 min backup time while the indexing is still going on after an hour.

I had another 4.7TB file server (Windows 2016 Server) with a very low change rate of 5GB on an incremental backup. It's guest index took around 91 minutes.

[PTide]

Hi,

If your problem is also related to old SIDs then there is no way to resolve it other than to perform a cleanup. I've managed to find a PS script and SubInACL tool that might help you with the task. Before doing a cleanup please contact our support team so they can confirm that the issue is indeed SID-related.

Thank you

[dwrice0]

Thanks for the update. I'll open a support case but was curious if there was an easy way to determine if the old SID issue is my problem? This is a brand new file server. However, the files were copied from an old file server using emcopy from EMC with the option passed on the command line to maintain security information from source to destination. So it wouldn't surprise me if the old SID info followed the files from the old server to the new. I just don't know the low level details about NTFS to make the determination. I want to be able to provide as much info up front to support.

[adrenaline_x]

Lucky for all of you.

Currently the indexing on our 2008 r2 file server with 8 tbs.

Indexing is taking over 24 hours. Support is blaming our Datadomain 6300 as the cause but seeing as its indexing within the VM how could this be.

[Mgamerz]

I just turned it off on our primary file server. With ~20TB of data and on-server data deduplication (windows dedupe) it would backup in like 15 mins (incremental) but take almost 3 hours to index. I don't use enterprise manager or the web ui's so it didn't really provide any benefit for me.

[TimWalker]

So, I've discovered a potential cause/resolution... which is fairly obvious if you look at the server in question using resource monitor.

The VeeamGuestIndexer.exe process runs at a 'background' IO priority, which is the very lowest possible. This is all well and good if you don't want to disrupt a live server, but if you want your backups to finish gracefully overnight it isn't so good!

In particular, we're using server 2012 r2 with deduplication turned on for our file server, the deduplication is set to the 'background' option for the volumes concerned (rather than running on a schedule) and it is completely crowding out the guest indexer, because it is running at 'Normal' IO priority!

Seems like a bit of a mistake by Microsoft to be honest!

Microsoft don't provide an easy way to change the IO priority of a process (changing the priority in task manager doesn't make any difference) I found a command line tool that will let you change it (and so potentially script the change) - https://sourceforge.net/projects/iopriority/

Command line is as follows:
IOPriorityV1.1.exe VeeamGuestIndexer 2

(the number is the priority - 0 is background, 1 is low, 2 is normal)

I've run that and I can see from resource monitor that the indexing is going much quicker. I suppose I could turn down the priority of the FSRM dedup job but would prefer not to muck around with core OS stuff.

Process hacker will apparently also let you change IO priority and has a nice GUI inspired by process explorer - https://processhacker.sourceforge.io/

Veeam, any chance of tweaking the way the indexer works or letting us set the priority per job?

[Gostev]

Yes, we reduced the priority of this process back a few versions ago due to actual impact on production workloads. May be we should provide a registry key to allow tweaking it.

[realsuperskye]

There is actually already a registry key to change the Windows Priority Level used for Veeam indexing.

Gostev, would you like me to share this registry key setting?

[adrenaline_x]

Name: VeeamGuestIndexerPriority
Type: REG_SZ
Default value: BACKGROUND
Description: Sets process priority for indexing process.
Available values: BACKGROUND, ABOVE_NORMAL, BELOW_NORMAL, HIGH, IDLE, NORMAL, REALTIME


But this did not change anything for us. indexing still runs 13 - 30 hours.

You can also open task manager while the indexing is running and assign it a higher priority, but it did not change anything. I'm thinking it the number of files/speed of the vmdks.

Its fustrating at this point at our Enviroment is rather robost with the host in a blade chasis with dual 40GBe links to the core switch stack with the Datadomains attached to the same switch with Dual 10Gbe links. I know the datadomains are slow to restore from, but indexing is not touching those, and our backup jobs are transferring at 300 Mb/s with Individual Vmdks backing up at 60-100Mb/s.

[bertdhont]

Do you add that key to the Veeam B&R Server or on the host you are indexing?

[foggy]

On the backup server (with further service restart).

[SBarrett847]

I've found a better Solution to control the indexer IO priority here --> http://geekswithblogs.net/akraus1/archi ... 53728.aspx

Create a reg key, on the Server being backed up, as below - this will raise the IO Priority from background to Normal. This method lets you control the IO Priority per server being backed up

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\VeeamGuestIndexer.exe\PerfOptions]
"IoPriority"=dword:00000002

EDIT:- Out of curiosity why doesn't Veeam just index from the Backed up Image files?

[SBarrett847]

Quick update - I was too hasty - The above works for any other EXE i throw at it, bar the VeeamGuestIndexer.exe - Ignore

[adrenaline_x]

I thought i would update here as well as i finally came back around to this after 2018 and Veeam support wasn't helpful at the time. I created a new ticket for this and the Current tech appears to have identified the problem.

UAC needs to be disabled on the servers you are indexing, based on the latest ticket i created with Veeam.

After disabling UAC and adding the registry key associated with it, indexing has dropped from 24-48 hours to 3 hours. We are using a specifically created veeam AD account that is pushed to the local admin groups on the servers as we don't use domain accounts within veeam.

"To fix that:

The account specified to connect to the Guest OS is the Built-in Local or Domain Administrator account
Note: This must be the original Built-in Administrator which has a SID that ends in -500, this user is unique and has the ability to bypass Windows User Account Controls. Even if the account was renamed, it will work.

or

To use any account other than the Built-in Local or Domain Administrator, Windows User Account Control (UAC) must be disabled on VM that is being protected.
For Server 2008 & 2008 R2, in the “Change User Account Control Settings”, move slider to Never Notify
Starting with Server 2012 the “EnableLUA” DWORD in HKLMSoftwareMicrosoftWindowsCurrentVersionpoliciessystem must be set to a value of '0' (zero)."

[Gostev]

Basically, what these approaches enable is to perform indexing by connecting directly to a guest OS over the network. Without this, we have to go through the virtualization hosts APIs, which are not particularly fast, to get to the guest.

[adrenaline_x]

Thanks for clarifying the reasoning behind the need to disable UAC.

We can finally use enterprise manager for searching for files instead of waiting for 4 to 18 TB servers to be mounted so we can check if the file required is present and keep repeating that process until we find it. it takes 10-40 minutes to mount the backup before FLR can browse the backups A bit sad it took use 3 years to finally resolve the issue but we gave up trying to resolve it and continued to use net backup for file level recoveries.