Lucene Related Blogs - Blogs at Near Infinity

An Introduction to Blur

Aaron Mccurry — Wed, 09 Nov 2011 15:00:00 -0500

Blur is a new Apache 2.0 licensed software project that provides a search capability built on top of Hadoop and Lucene. Elastic Search and Solr already exist so why build something new? While these projects work well, they didn't have a solid integration with the Hadoop ecosystem. Blur was built specifically for Big Data, taking scalability, redundancy, and performance into consideration from the very start, while leveraging all the goodness that already exists in the Hadoop stack.

A year and a half ago, my project began using Hadoop for data processing. Very early on, we were having networking issues that would make our HDFS cluster network connectivity spotty at best. Over one weekend in particular, we steadily lost network connection to 47 of the 90 data nodes in the cluster. When we came in on Monday morning, I noticed that the MapReduce system was a little sluggish but still working. When I checked HDFS I saw that our capacity had dropped by about 50%. After running an fsck on the cluster I was amazed to find that what seemed like a catastrophic failure over the weekend resulted in a still healthy file system. This experience left a lasting impression on me. It was then that I got the idea to somehow leverage the redundancy and fault tolerance of HDFS for the next version of a search system that I was just beginning to (re)write.

I had already written a custom sharded Lucene server that had been in a production system for a couple of years. Lucene worked really well and did everything that we needed for search. The issue that we faced was that it was running on big iron that was not redundant and could not be easily expanded. After seeing the resilient characteristics of Hadoop first hand, I decided to look into marrying the already mature and impressive feature set of Lucene with the built in redundancy and scalability of the Hadoop platform. From this experiment Blur was created.

The biggest technical issues/features that Blur solves:

Rapid mass indexing of entire datasets
Automatic Shard Server Failover
Near Real-time update compatibility via Lucene NRT
Compression of Lucene FDT files while maintaining random access performance
Lucene WAL (Write Ahead Log) to provide data reliability
Lucene R/W directly into HDFS (the seek on write problem)
Random access performance with block caching of the Lucene Directory

Data Model

Data in Blur is stored in Tables that contain Rows. Rows must have a unique row id and contain one or more Records. Records have a unique record id (unique within the Row) and a column family for grouping columns that logically make up a single record. Columns contain a name and a value, and a Record can contain multiple columns with the same name.

Architecture

Blur uses Hadoop's MapReduce framework for indexing data, and Hadoop's HDFS filesystem for storing indexes. Thrift is used for all inter-process communications and Zookeeper is used to know the state of the system and to store meta data. The Blur architecture is made up of two types of server processes:

Blur Controller Server
Blur Shard Server

The shard server, serves 0 or more shards from all the currently online tables. The calculation of the what shards are online in each shard server is done through the state information in Zookeeper. If a shard server goes down, through interaction with Zookeeper the remaining shard servers detect the failure and determine which if any of the missing shards they need to serve from HDFS.

The controller server provides a single point of entry (logically) to the cluster for spraying out queries, collecting the responses, and providing a single response. Both the controller and shard servers expose the same Thrift API which helps to ease debugging. It also allows developers to start a single shard server and interact with it the same way they would with a large cluster. Many controller servers can be (and should be) run for redundancy. The controllers act as gateways to all of the data that is being served by the shard servers.

Updating / Loading Data

Currently there are two ways to load and update data. The first is through a bulk load in MapReduce and the second is through mutation calls in Thrift.

Bulk Load MapReduce Example

Data Mutation Thrift Example

Searching Data

Any element in the Blur data model is searchable through the normal Lucene semantics: analyzers. Analyzers are defined per Blur table.

The standard Lucene query syntax is the default way to search Blur. If anything outside of the standard syntax is needed, you can create a Lucene query directly with Java objects, and submit them through the expert query API.

The column family grouping within Rows allows for results to be discovered across column families similar to what you would get with an inner join across two tables that share the same key (or in this case rowid). For complicated data models that have multiple column families, this makes for a very powerful search capability.

The following example searches for "value" as a full text search. If I had wanted to search for "value" in a single field like column "colA" in column family "famB" the query would look like "famB.colA:value".

Fetching Data

Fetches can be done by row or by record. This is done by creating a selector object in which you specify the rowid or recordid, and the specific column families or columns that you would like returned. When not specified, the entire Row or Record is returned.

Current State

Blur is nearing it's first release 0.1 and is relatively stable. The first release candidate should be available for download within the next few weeks. In the meantime you can check it out on github:

https://github.com/nearinfinity/blur

http://blur.io

Lucene Thrift and Ruby

Bill Bejeck — Wed, 18 May 2011 23:38:30 -0400

This post is going to demonstrate thrift usage by searching a Lucene index from Ruby.

Thrift In a Nutshell

Essentially thrift is a serialization and RPC framework that allows you to communicate between programs that are not necessarily written in the same language. Thrift is used by defining data types and services in a .thrift file. You then run the .thrift file against the thrift compiler which generates the stub code needed for clients and servers. Currently thrift will generate code for C++, C#, Erlang, Haskell, Java, Objective C/Cocoa, OCaml, Perl, PHP, Python, Ruby, and Squeak. For a more detailed description of thrift along with instructions on how to install thrift if needed, consult the thrift wiki

Generating the Lucene Index

Our first step is to generate a small, simple lucene index. To build our index, 50,000 fake person records were downloaded from the Fake Name Generator in a comma delimited file. Each person record will contain a first name, last name, address and email address. Our indexing code will be very simple and will not be using any of lucene's advanced features.

public class IndexBuilder {

    public static void main(String[] args) throws Exception {
        String namesFile = "names.csv";
        Document doc = new Document();
        Field[] fields = new Field[]{new Field("firstName", "", Field.Store.YES, Field.Index.ANALYZED_NO_NORMS),
                new Field("lastName", "", Field.Store.YES, Field.Index.ANALYZED_NO_NORMS),
                new Field("address", "", Field.Store.YES, Field.Index.ANALYZED_NO_NORMS),
                new Field("email", "", Field.Store.YES, Field.Index.ANALYZED_NO_NORMS)};
        addFieldsToDocument(doc, fields);

        BufferedReader reader = new BufferedReader(new FileReader(namesFile));

        IndexWriter indexWriter = new IndexWriter(FSDirectory.open(new File("blog-index")),new IndexWriterConfig(Version.LUCENE_31, new StandardAnalyzer(Version.LUCENE_31)));

        String line;
        while ((line = reader.readLine()) != null) {
            String[] personData = getPersonData(line);
            setFieldData(personData, fields);
            indexWriter.addDocument(doc);
        }
        indexWriter.optimize();
        indexWriter.close();
    }

    private static String[] getPersonData(String line) {
        return line.split(",");
    }

    private static void setFieldData(String[] data, Field[] fields) {
        int index = 0;
        for (Field field : fields) {
            field.setValue(data[index++]);
        }
    }

    private static void addFieldsToDocument(Document doc, Field[] fields) {
        for (Field field : fields) {
            doc.add(field);
        }
    }
}

Creating the .thrift File

The next step will be to define what objects and services we want in our .thrift file, which will be called lucene_search.thrift. The lucene_search.thrift file is intentionally very basic. For more details on the structure of .thrift files consult the thrift wiki tutorial

//all generated java code will have the following for package name
namespace java bbejeck.thrift.gen

//this is the person object 
struct Person {
  1: string firstName,
  2: string lastName,
  3: string address,
  4: string email
}

//exception used to send meaningful error messages back to user
exception LuceneSearchException {
  1: string message
}

//service definition used by client and server
service LuceneSearch { 
    list&lt;Person&gt; search(1: string query) throws (1:LuceneSearchException error) 
}

As you can see from the example above, the .thrift file format is completely language agnostic. Next we need to generate our java and ruby code. The following were run from the command line:

$ thrift --gen java lucene_search.thrift
$ thrift --gen rb lucene_search.thrift

The generated code ends up in two directories named gen-java/ and gen-rb/ respectively. The files generated for java are LuceneSearch.java, LuceneSearchException.java and Person.java. The generated ruby files are lucene_search.rb, lucene_search_types.rb and lucene_search_constants.rb. In our next step, we are going to use generated java code to write our thrift server.

Thrift Server - Java

Thrift generates all the stub code you need for a server to expose your service or program. The only code we will need to write is a class that implements the generated Iface interface (defined in the LuceneSearch class), which contains the search method defined in our .thrift file.

public class LuceneThriftServer {
    private static final int PORT = 9090;
    private static int numberThreads = 5;

    public static void main(String[] args) throws Exception {
        TServerSocket serverSocket = new TServerSocket(PORT, 100000);
        LuceneSearch.Processor searchProcessor = new LuceneSearch.Processor(new SearchHandler(args[0]));
        if (args.length > 1) {
            numberThreads = Integer.parseInt(args[1]);
        }
        TThreadPoolServer.Args serverArgs = new TThreadPoolServer.Args(serverSocket);
        serverArgs.maxWorkerThreads(numberThreads);
        TServer thriftServer = new TThreadPoolServer(serverArgs.processor(searchProcessor).protocolFactory(new TBinaryProtocol.Factory()));
        thriftServer.serve();
    }

Iface Implementation

The SearchHandler class actually does the work of searching the lucene index. One tradeoff made here is that any exception while searching is caught and re-thrown as a LuceneSearchException. While it's usually not a great idea to just re-throw an exception, in this case it makes sense to do so. Since the LuceneSearchException is defined in the lucene_search.thrift file, the generated client code will handle that exception. So instead of receiving a generic thrift exception when an error occurs, the client should receive a more meaningful error message.

public class SearchHandler implements LuceneSearch.Iface {
    private IndexSearcher searcher;
    private QueryParser queryParser;
    private static final int MAX_RESULTS = 1000;

    public SearchHandler(String indexPath) {
        try {
            searcher = new IndexSearcher(FSDirectory.open(new File(indexPath)), true);
            queryParser = new QueryParser(Version.LUCENE_31, null, new StandardAnalyzer(Version.LUCENE_31));
            queryParser.setAllowLeadingWildcard(true);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    public List&lt;Person&gt; search(String query) throws LuceneSearchException {
        List&lt;Person&gt; results = new ArrayList&lt;Person&gt;();
        try {
            Query q = queryParser.parse(query);
            TopDocs topDocs = searcher.search(q, MAX_RESULTS);
            for (ScoreDoc sd : topDocs.scoreDocs) {
                Document document = searcher.doc(sd.doc);
                results.add(getPersonFromDocument(document));
            }
        } catch (Exception e) {
            throw new LuceneSearchException(e.getMessage());
        }
        return results;
    }

    private Person getPersonFromDocument(Document document) {
        Person p = new Person();
        p.firstName = document.get("firstName");
        p.lastName = document.get("lastName");
        p.address = document.get("address");
        p.email = document.get("email");

        return p;
    }
}

The next step in our process is to write the client.

Thrift Client - Ruby

Writing the thrift ruby client is even easier than writing the server code. If you have not already done so, install the thrift gem by running "gem install thrift" to get the required thrift library code. All the code you need for your client is already generated by thrift. At this point we are only doing what is needed to get the client to communicate with the server.

module ThriftConnection

  class LuceneClient

    def initialize(host='localhost', port=9090)
      socket = Thrift::Socket.new(host, port)
      @transport = Thrift::BufferedTransport.new(socket)
      protocol_factory = ::Thrift::BinaryProtocolFactory.new
      protocol = protocol_factory.get_protocol(@transport)
      @transport.open
      @client = LuceneSearch::Client.new(protocol)
    end

    def search(query)
      @client.search(query)
    end

    def close
      @transport.close
    end

  end
end

Running With Scissors

This section has the odd title "Running With Scissors", because like actual running with scissors, what we are about to do may not be a great idea. In all the thrift generated code there is a warning at the top "DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING", obviously I don't, but I'm not going to let that stop me at this point (sometimes you just have to see if you can get something to work!) What we've done is implement method_missing in the generated Person class (found in lucene_search_types.rb) so we can specify searches ala ActiveRecord style. What we are going to do to accomplish this is use a regular expression to pull out what fields to search for and use the arguments passed in as the search values. The regular expression here is fairly simple and only aims to handle simple searches.

#added to translate from symbol to expected search format
  SEARCH_KEYS_MAPPING = {:first_name => 'firstName',
                                            :last_name => 'lastName',
                                            :email => 'email',
                                            :address => 'address'}


  def self.method_missing(method_name, *args)
    lucene_client = ThriftConnection::LuceneClient.new
    query = ""
        #handles find_by_first_name etc
    if method_name.to_s =~ /^find_by_([a-z]+_?[a-z]*)$/
      query = "#{SEARCH_KEYS_MAPPING[$1.to_sym]}:#{args[0]}"
        #handles find_by_first_name_or_last_name, find_by_first_name_and_email 
    elsif method_name.to_s =~/^find_by_([a-z]+_[a-z]+)_([a-z]+)_([a-z]+_?[a-z]*)$/
      query ="#{SEARCH_KEYS_MAPPING[$1.to_sym]}:#{args[0]} #{$2.upcase} #{SEARCH_KEYS_MAPPING[$3.to_sym]}:#{args[1]}"
    else
       raise ArgumentError.new("search method pattern #{method_name} not recognized")
    end

    results = lucene_client.search(query)
    lucene_client.close
    results
  end

As we'll see in the next section, this actually worked, but I still view this more as a useful experiment. First of all this was placed in generated code, so any time you make changes you would have to manually get the method_missing definition back into the Person class. Secondly, Lucene search syntax is really not all that hard to learn.

Testing

All of what we have done so far would not be worth much if we could not verify our work with some testing. Here is the unit test to verify that we are indeed able to search a Lucene index from Ruby. To get some names to search on I simply ran

head names.csv

and then used some of the information in various combinations to get counts of what searches should return. For example to get an idea of what searching for a first name of Elizabeth or last name of Krause would return I ran

cat names.csv | grep -iE 'elizabeth|krause' | wc -l

which returned a count of 289. So, first making sure that our thrift server was running in the background, here is the unit test that was run to verify our Ruby client searching against a Lucene index.

class SearchTest < Test::Unit::TestCase

  def setup
    @lucene_client = ThriftConnection::LuceneClient.new
  end


  def teardown
    @lucene_client.close
  end

  def test_search_client_first_name
    persons = @lucene_client.search("firstName:Tia")
    assert_equal(5, persons.length)

    persons.each do |person|
      assert_equal("Tia", person.firstName)
    end
  end

  def test_search_person_class_first_name
    persons = Person.find_by_first_name("Tia")
    assert_equal(5, persons.length)

    persons.each do |person|
      assert_equal("Tia", person.firstName)
    end
  end

  def test_search_client_first_name_email_domain
    persons = @lucene_client.search("+firstName:Elizabeth +email:*pookmail.com")
    assert_equal(59, persons.length)
  end

  def test_search_person_class_first_name_email_domain
    persons = Person.find_by_first_name_and_email("elizabeth", "*pookmail.com")
    assert_equal(59, persons.length)
  end

  def test_search_client_first_name_and_last_name
    persons = @lucene_client.search("+firstName:Elizabeth +lastName:Krause")
    assert_equal(1, persons.length)
    person = persons[0]

    assert_equal("Elizabeth", person.firstName)
    assert_equal("Krause", person.lastName)
  end

  def test_search_person_class_first_name_and_last_name
    persons = Person.find_by_first_name_and_last_name("elizabeth", "krause")
    assert_equal(1, persons.length)
    person = persons[0]

    assert_equal("Elizabeth", person.firstName)
    assert_equal("Krause", person.lastName)
  end

  def test_search_person_class_first_name_or_last_name
    persons = Person.find_by_first_name_or_last_name("elizabeth", "krause")
    assert_equal(289, persons.length)
  end

  def test_invalid_search
    assert_raises ArgumentError do
      Person.find_person_by_name("tia")
    end
  end

end

Conclusion

Thrift is a compelling alternative for RPC or message passing where one might otherwise be using either REST, Java RMI or middleware (JMS, AMQP). There is a great comparison of how thrift performs against other forms of RPC in this thrift tutorial from OCI found near the end of the article. It is hoped the reader was able to learn something useful. Thanks for your time

Resources

Full source for the blog including the generated code can be found on github. If you are interested in running the test you can download lucene-thrift-example.tar.gz extract the tar file and execute the runSearchTest.sh script. You do not need to have thrift installed to run the test.

For more information on thrift the thrift wiki is a great start
More information on Lucene can be found here

Lucene Compression

Aaron Mccurry — Thu, 21 Apr 2011 09:00:00 -0400

For a number of years I have used Lucene for both search and data storage. Meaning, I store all the data necessary to display search results in the Lucene index. This means that there is typically a lot of data in the FDT file, and compressing that data becomes a necessity as the indexes grow in size.

In version 3.0 support for field compression was dropped by Lucene. I could compress each field into a byte array and store the data in document, but if you have a lot of small fields in a document this doesn't work very well. It typically won't save you any disk space, and actually it might cost you some depending on the compression algorithm.

So I have built a block level compression for the FDT file in the form of a Lucene directory. It allows you to choose whatever compression algorithm you want and whatever block size makes sense for your data.

The block level compression allows you to compress the entire document (possibly multiple documents) into a single block and achieve a higher compress ratio than if you had compressed each field separately.

Plus you don't have to modify any of your Lucene code, other than wrapping your real directory with the compressed implementation.

NOTE: This only works with the compound files turned off.

https://github.com/nearinfinity/lucene-compression

Low Memory Patch For Lucene

Aaron Mccurry — Tue, 19 Jan 2010 21:25:35 -0500

While I'm working on getting my code into lucene. I have patched 3.0.0 and 2.9.1 with my low memory patch.

By default the small memory footprint is enabled to change back to the default implementation set the following system property.

-Dorg.apache.lucene.index.TermInfosReader=default

Have fun! If you have any problems or questions please let me know or add to this LUCENE-2205.

lucene-core-3.0.0-nic.jar

lucene-core-2.9.1-nic.jar

Lucene Hit Paging Made Easier

Aaron Mccurry — Mon, 18 Jan 2010 16:16:59 -0500

Lucene gives users the ability to search massive amounts of data in a very short amount of time. However allowing users to page through the entire result set of their search can be difficult and risky depending on how many users are performing searches and how many of those users are paging through 100's if not 1,000's of hits per page.

Problem Scenario:

Each of your indexes contains 100,000,000's documents.
You have 500 users on your system actively performing searches.
You have 100 search results per page.
And, your typical user pages through the first 10 pages of results. (Normal occurrence on some systems)

So for the 10th page you will have to collect 1,000 hits, at a cost of a float plus an int plus some object overhead per hit. So let's say 20 bytes per hit. So you have 500 users * 1,000 hits * 20 bytes = 10,000,000 bytes or 10M. Easy, no problem, right?

Well what if you also give the users an easy way to move to the end of the result set. Hmm... Well for a result set size of 10,000 it's no big deal. But what if you hand out result sets in the order of a 1,000,000 or even 10,000,000.

At this point you really just want to prevent the system from running out of memory. Because if you have 25 users getting 10,000,000 results each and they all click last page at the same time. That's going to cost you 5 Gig of heap! At least. Some might say that it won't ever happen, but in my experience, if it can happen, it will.

So I created a Paging Hit Collector, that windows the hits to the users. It's uses the last hit collected from the previous search pass, to feed the next search pass. So yes if a user clicks the last page, it might perform multiple searches but, the system won't run out of memory.

The user's will get there answer eventually, and if your system gives them some feedback as it searches and pages, they will probably sit and wait for it to come back. Instead of giving up and hitting cancel and search and cancel and search, and making the system worse and worse.

The Simple Example:

IndexSearcher searcher = new IndexSearcher(reader);<br />
TermQuery query = new TermQuery(new Term("f1", "value"));
IterablePaging paging = new IterablePaging(searcher, query, 100);<br />
for (ScoreDoc sd : paging.skipTo(90)) {
&nbsp;&nbsp;System.out.println("doc id [" + sd.doc + "] " + 
&nbsp;&nbsp;&nbsp;&nbsp;"score [" + sd.score + "]");
}

The More Advanced Example:

IndexSearcher searcher = new IndexSearcher(reader);<br />
TotalHitsRef totalHitsRef = new TotalHitsRef();
ProgressRef progressRef = new ProgressRef();<br />
TermQuery query = new TermQuery(new Term("f1", "value"));
IterablePaging paging = new IterablePaging(searcher, query, 100);<br />
for (ScoreDoc sd : paging.skipTo(90).
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;gather(20).
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;totalHits(totalHitsRef).
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;progress(progressRef)) {<br />
&nbsp;&nbsp;System.out.println("time [" + progressRef.queryTime() + "] " + 
&nbsp;&nbsp;&nbsp;&nbsp;"total hits [" + totalHitsRef.totalHits() + "] " + 
&nbsp;&nbsp;&nbsp;&nbsp;"searches [" + progressRef.searchesPerformed() + "] " + 
&nbsp;&nbsp;&nbsp;&nbsp;"position [" + progressRef.currentHitPosition() + "] " + 
&nbsp;&nbsp;&nbsp;&nbsp;"doc id [" + sd.doc + "] " + 
&nbsp;&nbsp;&nbsp;&nbsp;"score [" + sd.score + "]");
}

Here's a link to the code LUCENE-2215.

My First Lucene Patch - Making Lucene Do More With Less

Aaron Mccurry — Mon, 11 Jan 2010 20:06:21 -0500

I've been using Lucene for the better part of 2 years, from initial playing around, to prototyping to production application. It's an impressive library and it has come along way in the past couple of years.

When I first started playing around with it the version was 2.1 and the search times were so much faster than what we were trying to use at the time (Oracle Text). The first test was indexing a monster dataset and searching it quickly. It passed with flying colors!

Next was to add in record level access control. Easy and extremely fast.

Next was to add in all the other data needed for our application. That was a little bit harder, considering that we have close to 150 fields in our index and well into the billion record range (growing everyday).

The problem was that we needed more memory and there was no extra money for any more servers (or upgrades). So there we were, stuck. So I decided to start poking around using visualvm to see if there were any places in our application or in Lucene to save some memory.

We had already disabled norms on all our fields (we really didn't need norms for our data nor did we have the resources). Took a long look at all our fields that we were indexing to see if there were any we didn't need, but we really did need them all. Then I stumbled across the TermInfosReader class in Lucene.

This is where Lucene really gets it speed, but also uses quite a bit memory to do it. And this is where I wrote my first Lucene patch.

In TermInfosReader there is a bunch stuff but the big memory hogs are in three arrays.

Terms[]
TermInfos[]
long[]

Basically Lucene does a binary search across the Terms array (that by default contains every 128th Term in the index) with a given Term to find where on disk the exact Term needed lives. There's a little bit more going on in the class than that, but that's basically what it's doing.

So, I started this patch with the need to save memory. So how in the world do you do that in java when everything is already in basic arrays and everything is needed in memory. Well you have to save it another way, references. References are a hidden cost in Java, every single reference in 32-bit JVM costs you 4 bytes, and 64-bit JVM it's 8 (assuming that you don't have compressed references).

Let's count the references.

Terms[] length * 3, 1 reference for the Term and 2 references for the two Strings inside the Term
TermInfo[] length * 1
long[] = 1 reference total

So, let's talk numbers. If you have a billion terms in your index, that's 125 MB (1,000,000,000 / 128 * (3 + 1 references) * 4 bytes for every ref) bytes of memory for the references. In a 64-bit JVM that doubled 250 MB. Not to mention the object overhead for every one of those Term and TermInfos objects. Wow that's a lot!

So I decided to remove nearly all of those references by using a byte array and an int array as an offset index.

The results were impressive!

Given an index of 6.2 GB size 1,010,000 number of documents with 179,822,683 number of terms the default implementation uses 292,235,512 bytes to just get the index usable.

My no-ref implementation of the same index uses only 49,849,744 bytes get the index usable. That a 17% of the original size, that's an 83% savings!

And the best part is, that it loads the segments faster into memory. So those real-time updates will be online faster. The run-time performance is slightly faster as well. But the huge performance saving is in garbage collection. Over 7 times faster for full GC's on my Macbook Pro. Wow!

I think that the results speak for themselves, and I hope that the Lucene folks will accept my patch. That way I won't have to continue patching each version after the fact. Also removing references can be great, but the code required to do it, and maintain the same level of performance, is ugly! So don't try this at home!

LUCENE-2205

Tuning the IBM JVM for large heaps

Aaron Mccurry — Fri, 24 Apr 2009 20:53:36 -0400

Recently I have been rewriting a lot of our lucene search engine code for a web application that I'm currently supporting. Our physical environment is a little different than most, we have a single very large computer (a left over from a previous project) running several virtual machines. The virtual machine that we are currently using for our searching duties is a 64 cpu 128G box.

Before I go into the tuning saga, I would like to say that in a previous version of the search engine we ran many more (16), smaller jvms all on the same box. They work together through a main controller module that coordinates calls to all the searching jvms. After the rewrite of the searchers, there was a lot of pressure to get the new code out the door. The new code partitions the data and searching duties in a very similar way to the previous version, however I haven't had time to focus on getting all the parts broken up across all the jvms. It's very doable, but do to customer priorities it hasn't been developed. But I was in luck (kinda), our environment was a single computer, so I thought I will just run one large jvm. I knew going in that garbage collection (GC) was going to be a problem, I just didn't realize how much.

Currently, we are running the 64-bit IBM java 6 jvm, the IBM jvm is very different from the Sun jvm, but no more so than in the GC tuning department. In the Sun jvm you have all kinds of settings to tweak, algorithms to use etc. The IBM jvm just isn't as advanced, you have 4 different GC policies, and sizing for various parts of the internals. And that's it! Great, it should be simple right? Well sorta.

First try, bring the system online with a 50G (yes that's gigabytes) heap, with all the default settings. Run some load tests and see where we are.

Everything is running great right up to the first full GC, 25 seconds doesn't sound that long but when you are waiting for a computer to return results, it's an eternity. For those that don't know, when a full GC occurs (some of the newer Sun algorithms are different) it stops the world (STW). This means the JVM is frozen until the GC is complete. No good.

So the default policy is the optthruput policy (-Xgcpolicy:optthruput). After digging through the IBM documentation, this type of policy should be used for maximum throughput, but at the expense of pauses during GC. They also mention that this should be used for batch processing when pauses are really a problem.

Next I tried the optavgpause policy (-Xgcpolicy:optavgpause), this is suppose to smooth out the GCs by kicking off the mark phase early. I'm not going to talk about mark, sweep, and compaction, you can find it here (http://www.ibm.com/developerworks/ibm/library/i-incrcomp/). But basically it's suppose to run a concurrent parallel mark phase before the jvm runs out of heap space and performs a full GC that STW. This did help, got us down in the 10-15 second range on average, but the problem was that the mark phase was too slow to start under heavy load and didn't give us a whole lot of concurrent marking before the STW. I found this out by adding -verbose:gc, this adds a lot of debugging information to the standard out. You should just run this all the time, it provides a lot of useful information about your application.

Next I tried subpool (-Xgcpolicy:subpool), it worked fine but was plagued by the same problem, slow full GCs. Subpool is suppose to work better on large SMP machines like ours, but in the end it was more of the same. It's full GCs were in the 10-12 second range, plus the application seemed to run slower, by about 10%.

And last I tried gencon (-Xgcpolicy:gencon), the newest of all their policies. Gencon is suppose to be used on "transactional systems", systems that create a lot of short lived objects. Isn't that what most java applications do? When I started it up, it seemed to be faster, and our load test confirmed that, 30% more throughput. But the amazing thing was that the full GCs were fast, really fast, in the 200-400ms range, for a 50G heap! But wait, it wasn't using most of the heap, and it was GCing all the time. Back to the verbose:gc log, AHA! The nursery was too small, it was about 10% of the heap, and because our application is almost all short lived objects I decided to increase the size of the nursery. I gave it 50% of the heap, and the time between GCs slowed down, but the time to perform the GC was still in the 350ms range. Awesome!

I finally settled on 100G heap with 50% a nursery, and the full GCs are now in the 400-600ms range. I can live with that, because this gives us a huge ceiling for load, and capacity.

So to summarize, if you application needs to run a huge heap size, and you are using the IBM jvm, I would start by using the gencon policy. It seems to be the most modern of all their policies, and it seems to work the best.

Good luck!

A little tuning can go a long way...

Aaron Mccurry — Thu, 02 Apr 2009 21:50:52 -0400

The application that I'm current developing uses Lucene for searching and data retrieval. We recently had a rather special need to use one of the utility classes in Lucene, OpenBitSet. OpenBitSet is not unlike the regular BitSet provided by the java.util package. The biggest difference is that the class is NOT final. So for someone like me, that tinkers and optimizes to get every last bit of performance out of my application, final classes are a major PITA.

The problem:

OpenBitSet provides an optimized "find the next set bit starting here" method called nextSetBit(long index). However I needed to find the previous set bit, so naturally I started by using the built-in api. Iterate backwards over each bit starting at given position until a bit that it is set to true is found. Probably not the fastest way, but it works. I thought that if I wrote my own "find the previous set bit starting here" method it could be potently a lot faster. And because the class is not final and the internals are not private I can extend it, YAY!

So after some work and some tests I came up with this:

&nbsp;&nbsp;&nbsp; /**<br />&nbsp;&nbsp;&nbsp; &nbsp;* Finds the previous set bit in the bitset, start looking at the index position.<br />&nbsp;&nbsp;&nbsp; &nbsp;* @param index the index to start looking.<br />&nbsp;&nbsp;&nbsp; &nbsp;* @return the previous position in the bitset that is set.<br />&nbsp;&nbsp;&nbsp; &nbsp;*/<br />&nbsp;&nbsp;&nbsp; public long prevSetBit(long index) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; //get index of the word to start with<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; int i = (int) (index &gt;&gt;&gt; 6); //div by 64<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; if (i &gt;= wlen) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; //if the index requested is greater than the actual<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; //max word, just start with the last word<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; i = wlen - 1;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; //loop backwards over the bits[] until a non-zero word is found<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; while (i &gt;= 0) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; long word = bits[i];<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; if (word != 0) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; //find the position of the most significant bit<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; int pomsb = getPositionOfMostSigBit(word);<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; //multiply by 64<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; long l = ((long) i) &lt;&lt; 6;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; //add together to find the actual position<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; return l + pomsb;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; i--;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; return -1;<br />&nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp; <br />&nbsp;&nbsp;&nbsp; /**<br />&nbsp;&nbsp;&nbsp; &nbsp;* The word passed in, should not be 0.<br />&nbsp;&nbsp;&nbsp; &nbsp;* @param word the word to find the position of the most significant bit.<br />&nbsp;&nbsp;&nbsp; &nbsp;* @return the position in the word.<br />&nbsp;&nbsp;&nbsp; &nbsp;*/<br />&nbsp;&nbsp;&nbsp; private int getPositionOfMostSigBit(long word) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; if (word == 1l) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; return 0;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; int i = 0;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; while (word != 1) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; word = word &gt;&gt;&gt; 1;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; i++;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; return i;<br />&nbsp;&nbsp;&nbsp; }<br /><br />

And of course I had to write a performance test to make sure that my work had produced something useful. Here's the test:

&nbsp;&nbsp;&nbsp; public static void main(String[] args) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; CustomOpenBitSet bitSet = new CustomOpenBitSet(Integer.MAX_VALUE);<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; int maxCount = 1000000;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; fillBitSet(bitSet, maxCount);<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; <br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; for (int i = 0; i &lt; 10; i++) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; double s1 = System.nanoTime();<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; int slow = runSlowReverseSeeksToSetBits(bitSet, maxCount);<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; double e1 = System.nanoTime();<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; <br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; double s2 = System.nanoTime();<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; int fast = runFastReverseSeeksToSetBits(bitSet, maxCount);<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; double e2 = System.nanoTime();<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; <br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; assert (slow == fast);<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; <br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; System.out.println("The slow time took " + ((e1-s1) / ONE_BILLION) + " seconds");<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; System.out.println("The fast time took " + ((e2-s2) / ONE_BILLION) + " seconds");<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp; }<br /><br />&nbsp;&nbsp;&nbsp; private static int runFastReverseSeeksToSetBits(CustomOpenBitSet bitSet, int maxCount) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; int count = 0;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; long start = Integer.MAX_VALUE;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; while (count &lt; maxCount - 1) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; long position = bitSet.prevSetBit(start);<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; count++;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; start = position - 1;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; return count;<br />&nbsp;&nbsp;&nbsp; }<br /><br />&nbsp;&nbsp;&nbsp; private static int runSlowReverseSeeksToSetBits(CustomOpenBitSet bitSet, int maxCount) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; int count = 0;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; long start = Integer.MAX_VALUE;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; while (count &lt; maxCount - 1) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; long position = slowSeek(bitSet,start);<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; count++;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; start = position - 1;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; return count;<br />&nbsp;&nbsp;&nbsp; }<br /><br />&nbsp;&nbsp;&nbsp; private static long slowSeek(CustomOpenBitSet bitSet, long position) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; while (!bitSet.get(position)) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; position--;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; if (position &lt; 0) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; return -1;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; return position;<br />&nbsp;&nbsp;&nbsp; }<br /><br />&nbsp;&nbsp;&nbsp; private static void fillBitSet(CustomOpenBitSet bitSet, int numberOfBitsToSet) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; Random random = new Random();<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; int nextInt = -1;<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; for (int i = 0; i &lt; numberOfBitsToSet; i++) {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; do {<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; nextInt = random.nextInt(Integer.MAX_VALUE);<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; } while (bitSet.fastGet(nextInt));<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; bitSet.set(nextInt);<br />&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; }<br />&nbsp;&nbsp;&nbsp; }

The test creates a BitSet that is Integer.MAX_VALUE long and randomly populates it with 1 million bits set to true. Then starting at Integer.MAX_VALUE, find all the bits that are set to true, moving from the greatest position to the smallest position. Do this 10 times and print off the time taken to perform both algorithms.

Pass [0], the slow time took 7.867899136 seconds<br />Pass [0], the fast time took 0.059851008 seconds<br />Pass [1], the slow time took 7.84461184 seconds<br />Pass [1], the fast time took 0.051954944 seconds<br />Pass [2], the slow time took 7.820731904 seconds<br />Pass [2], the fast time took 0.07801088 seconds<br />Pass [3], the slow time took 7.81953408 seconds<br />Pass [3], the fast time took 0.074951168 seconds<br />Pass [4], the slow time took 7.798106112 seconds<br />Pass [4], the fast time took 0.07792384 seconds<br />Pass [5], the slow time took 7.742498048 seconds<br />Pass [5], the fast time took 0.067644928 seconds<br />Pass [6], the slow time took 7.664754944 seconds<br />Pass [6], the fast time took 0.07476992 seconds<br />Pass [7], the slow time took 7.841472768 seconds<br />Pass [7], the fast time took 0.073187072 seconds<br />Pass [8], the slow time took 7.927188992 seconds<br />Pass [8], the fast time took 0.07574912 seconds<br />Pass [9], the slow time took 7.85273088 seconds<br />Pass [9], the fast time took 0.076514048 seconds

So as you can see the new method is about 100 times faster than just iterating backwards over the bitset, given this sample data. I would assume that if the sample size of 1 million were to increase, the gap between the 2 methods would narrow.

How to implement row level access control in Lucene

Aaron Mccurry — Sat, 27 Sep 2008 21:08:42 -0400

Below I have written some fully functionally code that shows how you could implement row level access control in Lucene (2.3.2). Basically you have to index enough information to be able to search (in a single query) and find all documents that a given user has access to read.

In the below example there are two fields:

DATA: Which contains any data that you want your users to be able to search. NOTE: You can have as many data fields as you like.

ACL_FIELD: The field used to determine what users have access to this document. Note: You can have as many access control fields as you like.

All you have to do is built the access control query for each user and submit your user's query unchanged.

public class TestIndexerSearcher {

   public static void main(String[] args) throws Exception {
      Directory directory = new RAMDirectory();
      IndexWriter indexWriter = new IndexWriter(directory, new StandardAnalyzer());
      indexWriter.addDocument(buildDocument("DATA:sametoken","ACL_FIELD:access"));
      indexWriter.addDocument(buildDocument("DATA:sametoken","ACL_FIELD:noaccess"));
      indexWriter.optimize();
      indexWriter.close();

      IndexSearcher indexSearcher = new IndexSearcher(directory);

      QueryParser parser = new QueryParser("DATA", new StandardAnalyzer());
      Query query = parser.parse("sametoken");
		
      //This is all you have to add to your existing code.
      Filter aclFilter = applyAccessControl(new TermQuery(
         new Term("ACL_FIELD","access")));

      Hits hits = indexSearcher.search(query, aclFilter);
      System.out.println("Hits[" + hits.length() + "]");
      for (int i = 0; i &lt; hits.length(); i++) {
         Document doc = hits.doc(i);
         System.out.println("DATA [" + doc.get("DATA") + 
            "] ACL_FIELD [" + doc.get("ACL_FIELD") + "]");
      }
      indexSearcher.close();	
   }

   private static Filter applyAccessControl(Query aclQuery) {
      return new CachedQueryFilter(aclQuery.toString(), 
         new QueryWrapperFilter(aclQuery));
   }

   private static Document buildDocument(String... fieldInfo) {
      Document document = new Document();
      for (int i = 0; i &lt; fieldInfo.length; i++) {
         String[] split = fieldInfo[i].split(":");
         String fieldName = split[0];
         String fieldValue = split[1];
         document.add(new Field(fieldName,fieldValue,
            Field.Store.YES,Field.Index.TOKENIZED));
      }
      return document;
   }	
}

After you run this code, you will get a single hit, not the two that you would normally get if the access control filter wasn't in place.

public class CachedQueryFilter extends Filter {
   private static final long serialVersionUID = 6797293376134753695L;
   private Filter filter;
   private String key;
   private static transient Map&lt;String, BitSetCache&gt; filterCache = 
      new ConcurrentHashMap&lt;String, BitSetCache&gt;();

   public CachedQueryFilter(String key, Filter filter) {
      this.filter = filter;
      this.key = key;
   }

   public BitSet bits(IndexReader reader) throws IOException {
      BitSetCache cachedBitSet = (BitSetCache) filterCache.get(key);
      if (cachedBitSet != null) {
         BitSet bitSet = cachedBitSet.bitSet.get();
         if (bitSet != null &amp;&amp; cachedBitSet.indexReaderVersion == reader.getVersion()) {
            return bitSet;
         }
      }
      BitSet bits = filter.bits(reader);
      BitSetCache bitSetCache = new BitSetCache();
      bitSetCache.indexReaderVersion = reader.getVersion();
      bitSetCache.bitSet = new SoftReference&lt;BitSet&gt;(bits);
      filterCache.put(key, bitSetCache);
      return bits;
   }
	
   private class BitSetCache {
      long indexReaderVersion;
      SoftReference&lt;BitSet&gt; bitSet;
   }
}

There are two additional features that this query filter doesn't implements that you may want to consider.

1st - Provide per query locking around the bitset creation code. This would allow multiple bitset creation calls to occur at once, but the same access control query would block. Therefore we would only have to build it once, even if multiple user queries with the same access control hit the query filter at once.

2nd - Persist the bitsets. In the past I have used the same directory as the index, but you may want to use a database, or something else.

Lucene is a memory hog!

Aaron Mccurry — Wed, 03 Sep 2008 22:29:41 -0400

Let me start by saying that, I like Lucene, I have used it to solve many technical problems on my current project. But one aspect of Lucene that I have had issues with is with its memory footprint.

Currently we index 38 fields across 1.5 billion documents, and we have implemented a fair similarity object (see Lucene Scoring Documentation) for normalized scoring. We have no use for index time field boosting or for any type of Norms (see Lucene Scoring Documentation). However Lucene reads all of the Norms into memory for fast scoring.

So let's do the math:

1 byte per Norm value * 38 fields * 1,500,000,000 = 57,000,000,000

That's +/- 57 Gigs of heap space!

That's quiet a bit of memory usage for something that we don't even use. I have since patched Lucene, so that the indexed Norms have a much better memory footprint, something around 1.5 Gigs. Not great, but livable.

I havenÃ•t posted my patch, because itÃ•s all or nothing, I havenÃ•t implemented a way to turn it on and off. But if there is anyone else out there, with an application that is running out of memory as users use more and more indexed fields in Lucene, take a look at the SegmentReader class. There's a byte array in there that you should take a look at. Happy hunting!

What Happens To Lucene When You Go Big... Part 1

Aaron Mccurry — Wed, 30 Apr 2008 13:30:00 -0400

My current project has some unique searching requirements.

Requirements

Fuzzy searching is a must (Soundex, Levenshtein, etc.)
Has to be fast, a must with any searching solution
Has to provide access control
Full data load indexing needs to be completed in a reasonable amount of time
Scoring needs to be a custom implementation
Needs to run on a predetermined environment, meaning that new hardware purchases are not going to happen any time soon
And last but not least is ability do all these things on a dataset that exceeds a billion records

So we have had a lot of constraints to deal with, the hardest one by far is the last one.

The Data

1 billion plus records
Over 30 million unique terms

Indexing and Searching Server Specs

20 CPUs
32 Gig of ram
Dedicated SAN storage

First Searching Experiences

After getting the index built in multiple partitions, I fired up a simple Lucene console to do some simple searches with a Lucene multi searcher. Ran out of memory with 2 Gig heap, tried the maximum heap size for the 32 bit JVM we were using, 3.3 Gig, and that ran out of memory as well. So, initial tries to just run one search were unsuccessful.

Then we installed a 64-bit JVM and tried an 8 Gig heap, and it worked! I could run searches and after the first couple of warm up searches it was getting 20 - 80 ms responses on single term searches. Great, but then we tried a Fuzzy search, which uses a Levenshtein algorithm to calculate matches, 2 minutes 45 seconds, this was unacceptable.

Next we wrote our own Levenshtein Lucene query and got the 2 minutes plus search down to about one second. We found that the built in Lucene Fuzzy query was taking 85-95% of the time to find the terms to search. Then after those terms were found the actual search with those expanded terms only took a second to two depending on how many terms were found. So we replaced the built in Fuzzy query with a custom one that gets near instantaneous results on Levenshtein fuzzy matches. Problem solved.

Indexing Time

After our initial proof of concept was complete, we needed to improve the indexing time down to something more reasonable. The index creation from scratch was taking 36 - 48 hours to build with 20 CPUs running at 100% utilization. Which means that the machine was indexing about 9,000 records a second. Not bad for Lucene 2.2, but not that great.

First we stopped merging the indexes after we created them, that by itself was taking about 12 hours. At this point we also started searching these multiple indexes in parallel, and we are seeing modest increases in query performance.

Second, we upgraded to Lucene 2.3, this provided a huge increase in indexing speed. Our index creation time went from 36 - 48 hours (depending on if we merged indexes or not) down to 3-4 hours. The indexing process is now indexing around 125,000+ records a second. Huge improvement, if you haven't upgraded to 2.3, you should!

Current Development

We are in the process of adding access control to Lucene as well as adding new custom queries and scoring. So far Lucene has performed better than any of the competition that it has come up against, and with it's price point it seems to have won acceptance on our project.

In upcoming parts I will go into more details about the technical solutions that we have developed to solve these problems, as well others that I haven't mentioned yet.