How to speed up guest file indexing?

Availability for the Always-On Enterprise

How to speed up guest file indexing?

Veeam Logoby bma-bs » Mon Mar 04, 2013 1:42 pm

Hi,

when taking a backup from an w2008r2 file server then guest file system indexing take about an hour every time.

There are about 3 millon files on that server, indexing or windows search are not installed.

What can I do to speed up the indexing?

Best regards

Nico
bma-bs
Novice
 
Posts: 5
Liked: never
Joined: Mon Mar 04, 2013 9:30 am

Re: How to speed up guest file indexing?

Veeam Logoby foggy » Mon Mar 04, 2013 3:33 pm

Nico, typically, VM guest OS indexing is performed nearly instantly as index data is obtained from NTFS MFT directly (rather than by scanning the entire file system). Though it really depends on the number of files stored in the VM. If we are talking about millions of files, it could take a while to get, parse and write their descriptions. Nevertheless, I would suggest opening a case just to make sure that the delay is not caused by something else.
foggy
Veeam Software
 
Posts: 16008
Liked: 1223 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: How to speed up guest file indexing?

Veeam Logoby chrisdearden » Mon Mar 04, 2013 7:54 pm

I've seen big file servers take that sort of time before. Worth noting that the indexing process runs within the Guest itself , so make sure there is plenty of C: space free - its also worth checking that the VM isn't CPU starved and has an AV exception on the Veeam VSS process.
chrisdearden
Expert
 
Posts: 1530
Liked: 225 times
Joined: Wed Jul 21, 2010 9:47 am
Full Name: Chris Dearden

[MERGED] : Guest File Indexing take very long time to comple

Veeam Logoby Esmond » Mon May 19, 2014 7:58 am

I have a Windows Server 2008 VM which has around 800GB data (mainly are MS Office document, PDF and JPG files) running on ESXi 5.1, the datastore are running on IBM Storage connected by iSCSI. When we start the backup job, it take around 3-4 hrs for the Guest File Indexing process no matter in full or incremental backup. Is it normal to take that long for Indexing? If so, any method to shorten the index time?

Thanks in advance for any help.

Esmond
Esmond
Lurker
 
Posts: 2
Liked: never
Joined: Thu May 10, 2012 1:32 pm
Full Name: Esmond Chow

Re: How to speed up guest file indexing?

Veeam Logoby v.Eremin » Mon May 19, 2014 9:07 am

Hi, Esmond,

As mentioned above, the processing time typically depends on number of files stored inside VM. Processing might take a while, if this number is huge.

As to tricks that can shorten indexing time, you can check the space on disk C:\, the amount of CPU given to the VM in question, etc. If everything looks good, you can open a ticket with our support team and let them confirm your environment.

Thanks.
v.Eremin
Veeam Software
 
Posts: 14526
Liked: 1083 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: How to speed up guest file indexing?

Veeam Logoby Esmond » Mon May 19, 2014 9:40 am

v.Eremin wrote:Hi, Esmond,

As mentioned above, the processing time typically depends on number of files stored inside VM. Processing might take a while, if this number is huge.

As to tricks that can shorten indexing time, you can check the space on disk C:\, the amount of CPU given to the VM in question, etc. If everything looks good, you can open a ticket with our support team and let them confirm your environment.

Thanks.


The c drive has 12GB free space, and the CPU speed and utilization were low during the indexing (in both the guest level and host level). For the indexing, does it build a new set each time or merge to the index from previous backup session?
Esmond
Lurker
 
Posts: 2
Liked: never
Joined: Thu May 10, 2012 1:32 pm
Full Name: Esmond Chow

Re: How to speed up guest file indexing?

Veeam Logoby v.Eremin » Mon May 19, 2014 2:08 pm

The new session creates new backup indexes that are stored in accordance to the retention settings. The retention settings of stored indexes depend on VB&R edition you're using. In case of Standard edition, indexes are kept for all backups currently present in backup repositories. In case of Enterprise edition, EM keeps track of indexes of archived backups, as well. (tapes, etc.). For more information kindly see this section of our Help Center.

Or you're asking whether or VB&R indexing is incremental or not?

Thanks.
v.Eremin
Veeam Software
 
Posts: 14526
Liked: 1083 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

[MERGED] Need Faster Guest Indexing

Veeam Logoby gveat » Mon Dec 11, 2017 4:40 pm

The only thing holding me back from dumping DPM is the speed of the Guest Indexing.
On an incremental backup of a 1TB HDD, the backup takes less than 2 minutes. The guest indexing takes 22 hours.
There has to be a way to speed this thing up.
gveat
Influencer
 
Posts: 19
Liked: 2 times
Joined: Thu Sep 29, 2016 6:21 pm
Full Name: Greg Veater

Re: Need Faster Guest Indexing

Veeam Logoby PTide » Tue Dec 12, 2017 11:40 am

Hi,

Just to make sure we're on the same page - do you use DPM to backup Windows VM via VMware snapshot?

The guest indexing takes 22 hours.
Normally it should not take that long. Would you give an estimate of the number of files that you have on the system?

Thanks
PTide
Veeam Software
 
Posts: 3707
Liked: 308 times
Joined: Tue May 19, 2015 1:46 pm

Re: How to speed up guest file indexing?

Veeam Logoby gveat » Wed Dec 13, 2017 3:28 pm

do you use DPM to backup Windows VM via VMware snapshot?

I use DPM to do data protection. We take a snapshot via vss every 6 hrs.

Would you give an estimate of the number of files that you have on the system?

We have Trillions of files on this particular machine.

I also mistyped on my original post we are backing up 10TB not 1TB.

We do however have two servers that do incremental backups on 1TB every 6 hours that take almost an hour to index. (still in my opinion too long, especially since there is usually less than a 10GB of change in the data.
gveat
Influencer
 
Posts: 19
Liked: 2 times
Joined: Thu Sep 29, 2016 6:21 pm
Full Name: Greg Veater

Re: How to speed up guest file indexing?

Veeam Logoby EricJ » Thu Dec 14, 2017 7:17 pm 1 person likes this post

We have a similar issue with two of our large file servers. We are a school district, and our student file servers were much slower to index than staff file servers, despite containing less data.

From combing through the logs, it appears that part of the guest indexing process includes reading the NTFS metadata file called "$Secure", which contains the SIDs of accounts who are owners of files on the volume. It appears that Windows never purges this file of old SIDs that no longer exist or own any files.

There are two things working against us in this situation:

1. Our student file servers see a large amount of turnover. Each school year, thousands of accounts are deleted, and thousands of new accounts are created - all of which are file/folder owners on these file servers. This means that our $Secure file still contains tens of thousands of SIDs that are unresolvable.

2. The scan of SIDs in the $Secure file appears to slow down as time goes on. At the start, the indexing process is looking up over 100 SIDs per second. After a few hours, the rate has dropped to just 1-2 SIDs per second. :shock:

To confirm this was the issue, I performed a test. I restored all of the files from one of our student file servers to a brand new NTFS volume on another server. Because this volume was new, the $Secure file was empty, and after restoring the files and permissions, only the current file owners were inserted in the $Secure file.

On our production file server, the guest indexing did 70,365 SID lookups, with 48,948 unresolvable, and the process took 10.5 hours. The guest indexing on the new volume (with identical files) did only 9,445 SID lookups, with 75 unresolved, in only 7 minutes. This is 90 times faster!

Support didn't have a good solution for this issue, and in the end we decided not to index student file servers. When we refresh the server OS and migrate data to a fresh volume I may try enabling indexing again. Just thought I'd tell our story in case it matches your situation.
EricJ
Influencer
 
Posts: 17
Liked: 4 times
Joined: Thu Jan 12, 2017 7:06 pm

Re: How to speed up guest file indexing?

Veeam Logoby PTide » Fri Dec 15, 2017 12:50 pm

Greg,

I suggest you to contact our support team directly, so they can look closer, and post your case ID.

Eric,

Would you post your case ID please so I could obtain the details from the case and push it to the dev team?

Thank you
PTide
Veeam Software
 
Posts: 3707
Liked: 308 times
Joined: Tue May 19, 2015 1:46 pm

Re: How to speed up guest file indexing?

Veeam Logoby EricJ » Mon Dec 18, 2017 2:10 pm

Here's the case info: [ID# 02031518] Guest File Indexing Very Slow

Thanks for forwarding it on!
EricJ
Influencer
 
Posts: 17
Liked: 4 times
Joined: Thu Jan 12, 2017 7:06 pm

Re: How to speed up guest file indexing?

Veeam Logoby dwrice0 » Tue Mar 13, 2018 3:05 am

Was a solution ever found for this? I'm seeing slow indexing on file servers as well.

For example, a 6.5TB file server (Windows 2012 R2) with low change rate (19.4 GB on this incremental backup) had a ~21 min backup time while the indexing is still going on after an hour.

I had another 4.7TB file server (Windows 2016 Server) with a very low change rate of 5GB on an incremental backup. It's guest index took around 91 minutes.
dwrice0
Novice
 
Posts: 5
Liked: 1 time
Joined: Sun Jun 21, 2015 12:42 pm

Re: How to speed up guest file indexing?

Veeam Logoby PTide » Tue Mar 13, 2018 12:28 pm 1 person likes this post

Hi,

If your problem is also related to old SIDs then there is no way to resolve it other than to perform a cleanup. I've managed to find a PS script and SubInACL tool that might help you with the task. Before doing a cleanup please contact our support team so they can confirm that the issue is indeed SID-related.

Thank you
PTide
Veeam Software
 
Posts: 3707
Liked: 308 times
Joined: Tue May 19, 2015 1:46 pm

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 1 guest