Comprehensive data protection for all workloads
Post Reply
bma-bs
Novice
Posts: 5
Liked: never
Joined: Mar 04, 2013 9:30 am
Contact:

How to speed up guest file indexing?

Post by bma-bs » Mar 04, 2013 1:42 pm

Hi,

when taking a backup from an w2008r2 file server then guest file system indexing take about an hour every time.

There are about 3 millon files on that server, indexing or windows search are not installed.

What can I do to speed up the indexing?

Best regards

Nico

foggy
Veeam Software
Posts: 18239
Liked: 1557 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How to speed up guest file indexing?

Post by foggy » Mar 04, 2013 3:33 pm

Nico, typically, VM guest OS indexing is performed nearly instantly as index data is obtained from NTFS MFT directly (rather than by scanning the entire file system). Though it really depends on the number of files stored in the VM. If we are talking about millions of files, it could take a while to get, parse and write their descriptions. Nevertheless, I would suggest opening a case just to make sure that the delay is not caused by something else.

chrisdearden
Expert
Posts: 1530
Liked: 225 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: How to speed up guest file indexing?

Post by chrisdearden » Mar 04, 2013 7:54 pm

I've seen big file servers take that sort of time before. Worth noting that the indexing process runs within the Guest itself , so make sure there is plenty of C: space free - its also worth checking that the VM isn't CPU starved and has an AV exception on the Veeam VSS process.

Esmond
Lurker
Posts: 2
Liked: never
Joined: May 10, 2012 1:32 pm
Full Name: Esmond Chow
Contact:

[MERGED] : Guest File Indexing take very long time to comple

Post by Esmond » May 19, 2014 7:58 am

I have a Windows Server 2008 VM which has around 800GB data (mainly are MS Office document, PDF and JPG files) running on ESXi 5.1, the datastore are running on IBM Storage connected by iSCSI. When we start the backup job, it take around 3-4 hrs for the Guest File Indexing process no matter in full or incremental backup. Is it normal to take that long for Indexing? If so, any method to shorten the index time?

Thanks in advance for any help.

Esmond

veremin
Product Manager
Posts: 16867
Liked: 1429 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: How to speed up guest file indexing?

Post by veremin » May 19, 2014 9:07 am

Hi, Esmond,

As mentioned above, the processing time typically depends on number of files stored inside VM. Processing might take a while, if this number is huge.

As to tricks that can shorten indexing time, you can check the space on disk C:\, the amount of CPU given to the VM in question, etc. If everything looks good, you can open a ticket with our support team and let them confirm your environment.

Thanks.

Esmond
Lurker
Posts: 2
Liked: never
Joined: May 10, 2012 1:32 pm
Full Name: Esmond Chow
Contact:

Re: How to speed up guest file indexing?

Post by Esmond » May 19, 2014 9:40 am

v.Eremin wrote:Hi, Esmond,

As mentioned above, the processing time typically depends on number of files stored inside VM. Processing might take a while, if this number is huge.

As to tricks that can shorten indexing time, you can check the space on disk C:\, the amount of CPU given to the VM in question, etc. If everything looks good, you can open a ticket with our support team and let them confirm your environment.

Thanks.
The c drive has 12GB free space, and the CPU speed and utilization were low during the indexing (in both the guest level and host level). For the indexing, does it build a new set each time or merge to the index from previous backup session?

veremin
Product Manager
Posts: 16867
Liked: 1429 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: How to speed up guest file indexing?

Post by veremin » May 19, 2014 2:08 pm

The new session creates new backup indexes that are stored in accordance to the retention settings. The retention settings of stored indexes depend on VB&R edition you're using. In case of Standard edition, indexes are kept for all backups currently present in backup repositories. In case of Enterprise edition, EM keeps track of indexes of archived backups, as well. (tapes, etc.). For more information kindly see this section of our Help Center.

Or you're asking whether or VB&R indexing is incremental or not?

Thanks.

gveat
Influencer
Posts: 23
Liked: 2 times
Joined: Sep 29, 2016 6:21 pm
Full Name: Greg Veater
Contact:

[MERGED] Need Faster Guest Indexing

Post by gveat » Dec 11, 2017 4:40 pm

The only thing holding me back from dumping DPM is the speed of the Guest Indexing.
On an incremental backup of a 1TB HDD, the backup takes less than 2 minutes. The guest indexing takes 22 hours.
There has to be a way to speed this thing up.

P.Tide
Product Manager
Posts: 5260
Liked: 459 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Need Faster Guest Indexing

Post by P.Tide » Dec 12, 2017 11:40 am

Hi,

Just to make sure we're on the same page - do you use DPM to backup Windows VM via VMware snapshot?
The guest indexing takes 22 hours.
Normally it should not take that long. Would you give an estimate of the number of files that you have on the system?

Thanks

gveat
Influencer
Posts: 23
Liked: 2 times
Joined: Sep 29, 2016 6:21 pm
Full Name: Greg Veater
Contact:

Re: How to speed up guest file indexing?

Post by gveat » Dec 13, 2017 3:28 pm

do you use DPM to backup Windows VM via VMware snapshot?
I use DPM to do data protection. We take a snapshot via vss every 6 hrs.
Would you give an estimate of the number of files that you have on the system?
We have Trillions of files on this particular machine.

I also mistyped on my original post we are backing up 10TB not 1TB.

We do however have two servers that do incremental backups on 1TB every 6 hours that take almost an hour to index. (still in my opinion too long, especially since there is usually less than a 10GB of change in the data.

EricJ
Influencer
Posts: 18
Liked: 4 times
Joined: Jan 12, 2017 7:06 pm
Contact:

Re: How to speed up guest file indexing?

Post by EricJ » Dec 14, 2017 7:17 pm 1 person likes this post

We have a similar issue with two of our large file servers. We are a school district, and our student file servers were much slower to index than staff file servers, despite containing less data.

From combing through the logs, it appears that part of the guest indexing process includes reading the NTFS metadata file called "$Secure", which contains the SIDs of accounts who are owners of files on the volume. It appears that Windows never purges this file of old SIDs that no longer exist or own any files.

There are two things working against us in this situation:

1. Our student file servers see a large amount of turnover. Each school year, thousands of accounts are deleted, and thousands of new accounts are created - all of which are file/folder owners on these file servers. This means that our $Secure file still contains tens of thousands of SIDs that are unresolvable.

2. The scan of SIDs in the $Secure file appears to slow down as time goes on. At the start, the indexing process is looking up over 100 SIDs per second. After a few hours, the rate has dropped to just 1-2 SIDs per second. :shock:

To confirm this was the issue, I performed a test. I restored all of the files from one of our student file servers to a brand new NTFS volume on another server. Because this volume was new, the $Secure file was empty, and after restoring the files and permissions, only the current file owners were inserted in the $Secure file.

On our production file server, the guest indexing did 70,365 SID lookups, with 48,948 unresolvable, and the process took 10.5 hours. The guest indexing on the new volume (with identical files) did only 9,445 SID lookups, with 75 unresolved, in only 7 minutes. This is 90 times faster!

Support didn't have a good solution for this issue, and in the end we decided not to index student file servers. When we refresh the server OS and migrate data to a fresh volume I may try enabling indexing again. Just thought I'd tell our story in case it matches your situation.

P.Tide
Product Manager
Posts: 5260
Liked: 459 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: How to speed up guest file indexing?

Post by P.Tide » Dec 15, 2017 12:50 pm

Greg,

I suggest you to contact our support team directly, so they can look closer, and post your case ID.

Eric,

Would you post your case ID please so I could obtain the details from the case and push it to the dev team?

Thank you

EricJ
Influencer
Posts: 18
Liked: 4 times
Joined: Jan 12, 2017 7:06 pm
Contact:

Re: How to speed up guest file indexing?

Post by EricJ » Dec 18, 2017 2:10 pm

Here's the case info: [ID# 02031518] Guest File Indexing Very Slow

Thanks for forwarding it on!

dwrice0
Novice
Posts: 5
Liked: 1 time
Joined: Jun 21, 2015 12:42 pm
Contact:

Re: How to speed up guest file indexing?

Post by dwrice0 » Mar 13, 2018 3:05 am

Was a solution ever found for this? I'm seeing slow indexing on file servers as well.

For example, a 6.5TB file server (Windows 2012 R2) with low change rate (19.4 GB on this incremental backup) had a ~21 min backup time while the indexing is still going on after an hour.

I had another 4.7TB file server (Windows 2016 Server) with a very low change rate of 5GB on an incremental backup. It's guest index took around 91 minutes.

P.Tide
Product Manager
Posts: 5260
Liked: 459 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: How to speed up guest file indexing?

Post by P.Tide » Mar 13, 2018 12:28 pm 1 person likes this post

Hi,

If your problem is also related to old SIDs then there is no way to resolve it other than to perform a cleanup. I've managed to find a PS script and SubInACL tool that might help you with the task. Before doing a cleanup please contact our support team so they can confirm that the issue is indeed SID-related.

Thank you

dwrice0
Novice
Posts: 5
Liked: 1 time
Joined: Jun 21, 2015 12:42 pm
Contact:

Re: How to speed up guest file indexing?

Post by dwrice0 » Mar 20, 2018 12:39 am 1 person likes this post

Thanks for the update. I'll open a support case but was curious if there was an easy way to determine if the old SID issue is my problem? This is a brand new file server. However, the files were copied from an old file server using emcopy from EMC with the option passed on the command line to maintain security information from source to destination. So it wouldn't surprise me if the old SID info followed the files from the old server to the new. I just don't know the low level details about NTFS to make the determination. I want to be able to provide as much info up front to support.

adrenaline_x
Influencer
Posts: 10
Liked: never
Joined: May 03, 2016 4:24 am
Full Name: Mike Fuller
Contact:

Re: How to speed up guest file indexing?

Post by adrenaline_x » Aug 24, 2018 2:48 pm

Lucky for all of you.

Currently the indexing on our 2008 r2 file server with 8 tbs.

Indexing is taking over 24 hours. Support is blaming our Datadomain 6300 as the cause but seeing as its indexing within the VM how could this be.

Mgamerz
Expert
Posts: 127
Liked: 21 times
Joined: Sep 29, 2017 8:07 pm
Contact:

Re: How to speed up guest file indexing?

Post by Mgamerz » Aug 24, 2018 8:55 pm

I just turned it off on our primary file server. With ~20TB of data and on-server data deduplication (windows dedupe) it would backup in like 15 mins (incremental) but take almost 3 hours to index. I don't use enterprise manager or the web ui's so it didn't really provide any benefit for me.

TimWalker
Novice
Posts: 8
Liked: never
Joined: Aug 25, 2018 11:53 am
Full Name: Tim Walker
Contact:

Re: How to speed up guest file indexing?

Post by TimWalker » Aug 25, 2018 12:08 pm

So, I've discovered a potential cause/resolution... which is fairly obvious if you look at the server in question using resource monitor.

The VeeamGuestIndexer.exe process runs at a 'background' IO priority, which is the very lowest possible. This is all well and good if you don't want to disrupt a live server, but if you want your backups to finish gracefully overnight it isn't so good!

In particular, we're using server 2012 r2 with deduplication turned on for our file server, the deduplication is set to the 'background' option for the volumes concerned (rather than running on a schedule) and it is completely crowding out the guest indexer, because it is running at 'Normal' IO priority!

Seems like a bit of a mistake by Microsoft to be honest!

Microsoft don't provide an easy way to change the IO priority of a process (changing the priority in task manager doesn't make any difference) I found a command line tool that will let you change it (and so potentially script the change) - https://sourceforge.net/projects/iopriority/

Command line is as follows:
IOPriorityV1.1.exe VeeamGuestIndexer 2

(the number is the priority - 0 is background, 1 is low, 2 is normal)

I've run that and I can see from resource monitor that the indexing is going much quicker. I suppose I could turn down the priority of the FSRM dedup job but would prefer not to muck around with core OS stuff.

Process hacker will apparently also let you change IO priority and has a nice GUI inspired by process explorer - https://processhacker.sourceforge.io/

Veeam, any chance of tweaking the way the indexer works or letting us set the priority per job?

Gostev
SVP, Product Management
Posts: 24785
Liked: 3513 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How to speed up guest file indexing?

Post by Gostev » Aug 25, 2018 3:23 pm

Yes, we reduced the priority of this process back a few versions ago due to actual impact on production workloads. May be we should provide a registry key to allow tweaking it.

realsuperskye
Veeam Software
Posts: 5
Liked: never
Joined: Aug 10, 2017 9:08 pm
Full Name: Caitlin Pack
Contact:

Re: How to speed up guest file indexing?

Post by realsuperskye » Sep 07, 2018 4:15 pm

There is actually already a registry key to change the Windows Priority Level used for Veeam indexing.

Gostev, would you like me to share this registry key setting?

adrenaline_x
Influencer
Posts: 10
Liked: never
Joined: May 03, 2016 4:24 am
Full Name: Mike Fuller
Contact:

Re: How to speed up guest file indexing?

Post by adrenaline_x » Sep 13, 2018 8:20 pm

Name: VeeamGuestIndexerPriority
Type: REG_SZ
Default value: BACKGROUND
Description: Sets process priority for indexing process.
Available values: BACKGROUND, ABOVE_NORMAL, BELOW_NORMAL, HIGH, IDLE, NORMAL, REALTIME


But this did not change anything for us. indexing still runs 13 - 30 hours.

You can also open task manager while the indexing is running and assign it a higher priority, but it did not change anything. I'm thinking it the number of files/speed of the vmdks.

Its fustrating at this point at our Enviroment is rather robost with the host in a blade chasis with dual 40GBe links to the core switch stack with the Datadomains attached to the same switch with Dual 10Gbe links. I know the datadomains are slow to restore from, but indexing is not touching those, and our backup jobs are transferring at 300 Mb/s with Individual Vmdks backing up at 60-100Mb/s.

bertdhont
Service Provider
Posts: 27
Liked: 3 times
Joined: Nov 08, 2013 2:53 pm
Full Name: Bert D'hont
Contact:

Re: How to speed up guest file indexing?

Post by bertdhont » Feb 06, 2019 3:07 pm

Do you add that key to the Veeam B&R Server or on the host you are indexing?

foggy
Veeam Software
Posts: 18239
Liked: 1557 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: How to speed up guest file indexing?

Post by foggy » Feb 08, 2019 12:43 pm

On the backup server (with further service restart).

SBarrett847
Service Provider
Posts: 308
Liked: 40 times
Joined: Feb 02, 2016 5:02 pm
Full Name: Stephen Barrett
Contact:

Re: How to speed up guest file indexing?

Post by SBarrett847 » Jul 11, 2019 9:34 am

I've found a better Solution to control the indexer IO priority here --> http://geekswithblogs.net/akraus1/archi ... 53728.aspx

Create a reg key, on the Server being backed up, as below - this will raise the IO Priority from background to Normal. This method lets you control the IO Priority per server being backed up

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\VeeamGuestIndexer.exe\PerfOptions]
"IoPriority"=dword:00000002

EDIT:- Out of curiosity why doesn't Veeam just index from the Backed up Image files?

SBarrett847
Service Provider
Posts: 308
Liked: 40 times
Joined: Feb 02, 2016 5:02 pm
Full Name: Stephen Barrett
Contact:

Re: How to speed up guest file indexing?

Post by SBarrett847 » Jul 11, 2019 2:57 pm

Quick update - I was too hasty - The above works for any other EXE i throw at it, bar the VeeamGuestIndexer.exe - Ignore

Post Reply

Who is online

Users browsing this forum: Bing [Bot], EIvanov, hoang92bn and 45 guests