Guest File System Indexing - Very slow, not sure how exclusions work

[Mgamerz]

I have a host with several million files (file server). It has data-deduplication on in the VM itself (would be like 80% larger if it wasn't). For a long time I have turned off guest file system indexing because it takes like 6 hours (on an 8 hour backup interval) which means it is constantly running and makes server maintenance difficult.

Some folders I don't care about indexing are rather large and contain ancient data that almost never changes so I don't think I should waste the time indexing them. I would like to use enterprise manager at some point so I can find specific files easier.

In the exclusion window for indexing (Edit job > Guest Processing > Indexing > Edit) I have the default item which is for the whole host, I have that disabled (there is one other VM in this job that runs linux, so I can't index it, nor do I want to). I added an indexing job specifically for this VM and chose the option to Index everything except... Not totally sure what I am supposed to put there for the filepath.



L:\ is a drive on the VM, but is that right? How do I know what's not being indexed? I would like to not index A:\ or L:\ drives. I looked at the indexing pages on the veeam help manuals (https://helpcenter.veeam.com/docs/backu ... l?ver=95u4) but they don't talk about this feature.

[Mgamerz]

Hmm... maybe not worth the effort now that I looked at enterprise manager. It only supports active directory username/password and requires administrative account. We don't allow username/password for administrative accounts on the domain, smartcard only. Guess that's probably dead then.

[Gostev]

Yes, you need to specify local paths on the VM in that dialog.

Restore Operators don't need to have administrative accounts, they can be regular domain users.

[Mgamerz]

Thanks Gostev. I looked it over and was having issues adding the backup server. Couldn't submit a ticket for some reason through the portal (submit button didn't do anything), called in instead (case #03629468) and it worked on the first try after that -_-

I learned that it would be wiser to put Enterprise Manager onto a server rather than a local laptop.

One thing I am noticing is that System Volume Information is not excluded from indexing in the defaults. I have deduplication on one of my large fileservers (10s of TBs), I wonder if that is significantly impacting performance (I will never FLR anything out of that directory). Windows Dedupe places blocks into that directory on the volume. (My backups take like 20 minute with 6 hours of indexing curently, which is far beyond acceptable for me).

[foggy]

Windows Deduplication metadata may indeed affect indexing performance. We've planned some tests in this regard, but I currently have no ETA for this - have you tried to exclude this folder from indexing and check yourself?

[Mgamerz]

I have excluded it, and it did improve it some (indexing now takes like 2-4 hours, which I guess... is okay...), however it is still slow - though this might just be how long it actually takes to index all of the files (doing a right click on some of the folders/properties can take hours as there is millions and millions of files).

I have a case open for indexing not actually producing a usable index file reliably (03633236), tech says it has to do with SMB signing but that doesn't seem like it would explain why it works sometimes but most times not. I am working on getting a policy in place that will let me disable our organizations requirement for it, just for testing.

The other solution was to use windows server 2012 R2, but I can't honestly see how that could even be posted as a solution.

[foggy]

I have excluded it, and it did improve it some (indexing now takes like 2-4 hours, which I guess... is okay...), however it is still slow - though this might just be how long it actually takes to index all of the files (doing a right click on some of the folders/properties can take hours as there is millions and millions of files).

Yes, indexing speed heavily depends on the number of files.

[Mgamerz]

Making a bit of progress on this issue - it seems that SMB signing is somehow making communication between the VM and the VM host (the interaction proxy in this instance) not work. It only seems to affect this combination of systems. SMB signing can't be turned off beyond just testing in our environment, however. It also seems that Veeam B&R does not log if the file fails to transfer to the interaction proxy, which makes debugging this a lot more work.

I have little to no idea why SMB signing would be an issue here though.

[halllowien]

Hello,

i have problems with the performance of the feature guest file system indexing.

It takes too long for my fileservers.
I use NTFS with Deduplication , i have around 5 TB on multiple disks with dedup savings about 30%.

The hardware under this VM is very quick and the server has no real load.
i look on the Performance and see 25% CPU 50% Memory and around 10-20MB Writing and reading (if i try i get around 1GB per second in both, thanks to SSD-SAN).

The Inventory takes around 90 - 120 Minutes.

Is this an expected time?
Is there a switch to change the priorities from Indexing to higher (Process is in Priority Low and IOP-Priority is Background).

Greets
halllowien

[david.domask]

Hi hallowien,

I've moved your post to an existing topic regarding a similar situation. Please check the responses here as regrettably Windows Deduplication has a non-trivial impact on Guest Indexing currently. You can try excluding the Windows Dedup metadata folder from indexing as Foggy recommends above? It should help a bit on the processing time.

[halllowien]

Hi David,

thank you, i hope somebody has some newer informations/ideas than 5 Years for this

Greets
halllowien