Comprehensive data protection for all workloads
Post Reply
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

VBRCatalog with NTFS Compression

Post by tsightler »

Any reason not to enable NTFS compression on the VBRCatalog folder? I ask because our catalog has already grown to >20GB and it appears to contain mostly text file listings that compress very well. A quick test gave a 3:1 compression ratio which is a pretty big savings on the Veeam server.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VBRCatalog with NTFS Compression

Post by Gostev »

This might be a good idea. I will run this by developers.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VBRCatalog with NTFS Compression

Post by Gostev »

OK, so devs think this might be good idea, only worried if Search Server can still crawl such folder (pretty sure there will be no problems, but this was never explicitly tested).
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: VBRCatalog with NTFS Compression

Post by tsightler »

I'm pretty sure it will still crawl it with no issues but I'll have to admit that we're not even using the search server at this time, just the tree view.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VBRCatalog with NTFS Compression

Post by Gostev »

...which would not even be available in v5, if not your feedback during early beta ;) funny how things work out sometimes.
StanO
Enthusiast
Posts: 36
Liked: 5 times
Joined: Jan 04, 2010 4:14 pm
Full Name: Stan
Contact:

Re: VBRCatalog with NTFS Compression

Post by StanO »

How did this turn out?

Thanks,

Stan
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: VBRCatalog with NTFS Compression

Post by tsightler »

Seems to be working great for me and save a huge amount of space. I've experienced no issues.
JoshF
Enthusiast
Posts: 27
Liked: 1 time
Joined: Sep 06, 2011 5:57 am
Full Name: Josh
Contact:

Re: VBRCatalog with NTFS Compression

Post by JoshF »

Has anyone tried this in v6? Have seen Gostev note that these should be 5x smaller in v6, but my experience to date is these are now 5x larger..
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: VBRCatalog with NTFS Compression

Post by J1mbo »

Have to say, I've long been a fan of NTFS compression and have used it wherever I can since P-Pro & NT days. For example file shares (less important now with docx etc being internally zipped), IIS log directories and so on are all easy targets. Technically the system works under the file cache, so cached content has no impact (rather the lazy writer and reads incur the penalty). I once read that the overhead was something like 30 clocks per byte on a P-Pro, which seemed to tally with throughput back then.
JoshF
Enthusiast
Posts: 27
Liked: 1 time
Joined: Sep 06, 2011 5:57 am
Full Name: Josh
Contact:

Re: VBRCatalog with NTFS Compression

Post by JoshF »

Nice one.. this definately sounds like a go-er then - will log a change request to enable compression and let you know how this goes for us with v6.
lp@albersdruck.de
Enthusiast
Posts: 82
Liked: 33 times
Joined: Mar 25, 2013 7:37 pm
Full Name: Lars Pisanec
Contact:

[MERGED] Compress B&R catalog folder

Post by lp@albersdruck.de »

Hi!

Are there any downsides to use NTFS compression on the B&R catalog folder?
Compression works well on the *.ifd files, most of the time 50% or more.

Kind regards,
Lars
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VBRCatalog with NTFS Compression

Post by foggy »

Lars, enabling NTFS compression on the VBRCatalog folder is a pretty common practice among our customers.

Also, probably the following topics will be useful:
cleaning up vbrcatalog folder
VBRCatalog folder growing in size
lp@albersdruck.de
Enthusiast
Posts: 82
Liked: 33 times
Joined: Mar 25, 2013 7:37 pm
Full Name: Lars Pisanec
Contact:

Re: VBRCatalog with NTFS Compression

Post by lp@albersdruck.de »

Thanks foggy!
I did a brief search on compression and this (now merged) thread did not show up. Ah well the wonders of choosing the right terms to search for never cease to amaze ;)
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VBRCatalog with NTFS Compression

Post by Gostev »

No worries, Lars. This is exactly the reason why we merge topics - this adds more search terms to the topic due to adding additional ways of asking the same question. Thanks for trying the search before you post.
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: VBRCatalog with NTFS Compression

Post by lightsout »

Has anyone tried Windows Deduplication on the catalog?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VBRCatalog with NTFS Compression

Post by Gostev »

lightsout wrote:Has anyone tried Windows Deduplication on the catalog?
Yes and it's awesome. In fact, I recommended it in one of the weekly forum digests earlier this year :)
gostev wrote:And to finish off for today, I wanted to discuss our integration with Microsoft Search Server (MSS), as it has been a subject to many questions lately. B&R have had MSS integration for over 5 years ago, back when we have introduced guest file system indexing in v4. Search was needed to enable our users to quickly find files in large catalogs of thousands of VMs, allowing for fast file level recovery when the location of the lost file was not known. Using MSS has always been optional, as we have also provided a built-in search engine that is used until you register at least one MSS with the Enterprise Manager. Since the built-in engine only scaled for up to a few hundreds of VMs, leveraging MSS integration was very popular at the time.

But, fast forward to 2015 – and a lot has changed since then! Over the past years, built-in search engine performance has been improved significantly, both due to optimizations in the code, and much faster storage hardware. On the other hand, through our support we learned that MSS falls apart performing searches against very large catalogs - while the built-in engine is able to query catalogs of virtually any size reliably (albeit not as fast). As the result, these days I don't know of many clients using MSS integration, and I do not recommend it anymore. We are also actively looking to replace it with the different search engine down the road.

But for now, best results can be achieved by optimizing built-in search engine performance, and for that I recommend simply placing the catalog on an SSD drive. And here we run into one associated issue that large customers often complain about: the disk space needed for the catalog. But, this one can be easily solved too! One of our Solution Architects has found that placing catalog on a Windows Server dedupe volume works very well and makes search even faster in some scenarios, likely because the number of total blocks that must be read from disk is significantly reduced. Bottom line, our current recommendation for customers looking to index and search a very large amount of VMs, is to: use built-in search engine, place the catalog on a fast, dedicated volume (preferably backed by an SSD drive), and enable Windows Server dedupe on that volume.
Post Reply

Who is online

Users browsing this forum: Google [Bot], Paul.Loewenkamp, ThomasIkoula and 180 guests