We have a similar issue with two of our large file servers. We are a school district, and our student file servers were much slower to index than staff file servers, despite containing less data.
From combing through the logs, it appears that part of the guest indexing process includes reading the NTFS metadata file called "$Secure", which contains the SIDs of accounts who are owners of files on the volume. It appears that Windows never purges this file of old SIDs that no longer exist or own any files.
There are two things working against us in this situation:
1. Our student file servers see a large amount of turnover. Each school year, thousands of accounts are deleted, and thousands of new accounts are created - all of which are file/folder owners on these file servers. This means that our $Secure file still contains tens of thousands of SIDs that are unresolvable.
2. The scan of SIDs in the $Secure file appears to slow down as time goes on. At the start, the indexing process is looking up over 100 SIDs per second. After a few hours, the rate has dropped to just 1-2 SIDs per second.
To confirm this was the issue, I performed a test. I restored all of the files from one of our student file servers to a brand new NTFS volume on another server. Because this volume was new, the $Secure file was empty, and after restoring the files and permissions, only the current file owners were inserted in the $Secure file.
On our production file server, the guest indexing did 70,365 SID lookups, with 48,948 unresolvable, and the process took 10.5 hours. The guest indexing on the new volume (with identical files) did only 9,445 SID lookups, with 75 unresolved, in only 7 minutes. This is 90 times faster!
Support didn't have a good solution for this issue, and in the end we decided not to index student file servers. When we refresh the server OS and migrate data to a fresh volume I may try enabling indexing again. Just thought I'd tell our story in case it matches your situation.