VBRCatalog with NTFS Compression

[tsightler]

Any reason not to enable NTFS compression on the VBRCatalog folder? I ask because our catalog has already grown to >20GB and it appears to contain mostly text file listings that compress very well. A quick test gave a 3:1 compression ratio which is a pretty big savings on the Veeam server.

[Gostev]

This might be a good idea. I will run this by developers.

[Gostev]

OK, so devs think this might be good idea, only worried if Search Server can still crawl such folder (pretty sure there will be no problems, but this was never explicitly tested).

[tsightler]

I'm pretty sure it will still crawl it with no issues but I'll have to admit that we're not even using the search server at this time, just the tree view.

[Gostev]

...which would not even be available in v5, if not your feedback during early beta funny how things work out sometimes.

[StanO]

How did this turn out?

Thanks,

Stan

[tsightler]

Seems to be working great for me and save a huge amount of space. I've experienced no issues.

[JoshF]

Has anyone tried this in v6? Have seen Gostev note that these should be 5x smaller in v6, but my experience to date is these are now 5x larger..

[J1mbo]

Have to say, I've long been a fan of NTFS compression and have used it wherever I can since P-Pro & NT days. For example file shares (less important now with docx etc being internally zipped), IIS log directories and so on are all easy targets. Technically the system works under the file cache, so cached content has no impact (rather the lazy writer and reads incur the penalty). I once read that the overhead was something like 30 clocks per byte on a P-Pro, which seemed to tally with throughput back then.

[JoshF]

Nice one.. this definately sounds like a go-er then - will log a change request to enable compression and let you know how this goes for us with v6.

[lp@albersdruck.de]

Hi!

Are there any downsides to use NTFS compression on the B&R catalog folder?
Compression works well on the *.ifd files, most of the time 50% or more.

Kind regards,
Lars

[foggy]

Lars, enabling NTFS compression on the VBRCatalog folder is a pretty common practice among our customers.

Also, probably the following topics will be useful:
cleaning up vbrcatalog folder
VBRCatalog folder growing in size

[lp@albersdruck.de]

Thanks foggy!
I did a brief search on compression and this (now merged) thread did not show up. Ah well the wonders of choosing the right terms to search for never cease to amaze

[Gostev]

No worries, Lars. This is exactly the reason why we merge topics - this adds more search terms to the topic due to adding additional ways of asking the same question. Thanks for trying the search before you post.

[lightsout]

Has anyone tried Windows Deduplication on the catalog?

[Gostev]

Has anyone tried Windows Deduplication on the catalog?

Yes and it's awesome. In fact, I recommended it in one of the weekly forum digests earlier this year

And to finish off for today, I wanted to discuss our integration with Microsoft Search Server (MSS), as it has been a subject to many questions lately. B&R have had MSS integration for over 5 years ago, back when we have introduced guest file system indexing in v4. Search was needed to enable our users to quickly find files in large catalogs of thousands of VMs, allowing for fast file level recovery when the location of the lost file was not known. Using MSS has always been optional, as we have also provided a built-in search engine that is used until you register at least one MSS with the Enterprise Manager. Since the built-in engine only scaled for up to a few hundreds of VMs, leveraging MSS integration was very popular at the time.

But, fast forward to 2015 – and a lot has changed since then! Over the past years, built-in search engine performance has been improved significantly, both due to optimizations in the code, and much faster storage hardware. On the other hand, through our support we learned that MSS falls apart performing searches against very large catalogs - while the built-in engine is able to query catalogs of virtually any size reliably (albeit not as fast). As the result, these days I don't know of many clients using MSS integration, and I do not recommend it anymore. We are also actively looking to replace it with the different search engine down the road.

But for now, best results can be achieved by optimizing built-in search engine performance, and for that I recommend simply placing the catalog on an SSD drive. And here we run into one associated issue that large customers often complain about: the disk space needed for the catalog. But, this one can be easily solved too! One of our Solution Architects has found that placing catalog on a Windows Server dedupe volume works very well and makes search even faster in some scenarios, likely because the number of total blocks that must be read from disk is significantly reduced. Bottom line, our current recommendation for customers looking to index and search a very large amount of VMs, is to: use built-in search engine, place the catalog on a fast, dedicated volume (preferably backed by an SSD drive), and enable Windows Server dedupe on that volume.