Grant's model security is deliberately designed to force the developer to make conscious security decisions about what CRUD operations a user should be allowed to perform on your model objects. It doesn't care how you choose to authenticate and authorize your users to perform a CRUD operation, it only cares that you actually do it.

Rather than specify which operations are restricted, Grant restricts all CRUD operations unless they're explicitly granted to the user. It also restricts adding or removing items from has_many and has_and_belongs_to_many associations. Only allowing operations explicitly granted forces you to make conscious security decisions. While it obviously can't ensure you make the correct decisions, it should help ease the latent fear that you've inadvertently forgotten to secure something.

Enough talk, let me show you an example of how you might use it. To enable model security you simply include the Grant::ModelSecurity module in your model class. In this example you see three grant statements. The first grants find (aka read) permission to everyone. The second example grants create, update, and destroy permission when the passed block evaluates to true, which in this case happens when the model is editable by the current user. You can put any code you want in that block as long as it returns a boolean value. Similarly, the third grant statement permits additions and removals from the tags association when it's block evaluates to true. A Grant::ModelSecurityError is raised if any grant block evaluates to false or nil.

class EditablePage < ActiveRecord::Base
  include Grant::ModelSecurity
  has_many :tags

  grant(:find) { true }
  grant(:create, :update, :destroy) do |user, model| 
    model.editable_by_user? user 
  end
  grant(:add => :tags, :remove => :tags) do |user, model, associated_model| 
    model.editable_by_user? user 
  end

  def editable_by_user? user
    user.administrator?
  end
end

There's a lot more to the grant statement than shown in the above example. For instance, you can have multiple grant statements for the same action. Ultimate permission to perform the action will not be granted unless all grant blocks evaluate to true.

As you can see, Grant is pretty simple to use, but it's not going to do the dirty work for you. It's up to you to make the proper security decisions. Grant's just there to make sure you don't forget.

public class TestIndexerSearcher {

   public static void main(String[] args) throws Exception {
      Directory directory = new RAMDirectory();
      IndexWriter indexWriter = new IndexWriter(directory, new StandardAnalyzer());
      indexWriter.addDocument(buildDocument("DATA:sametoken","ACL_FIELD:access"));
      indexWriter.addDocument(buildDocument("DATA:sametoken","ACL_FIELD:noaccess"));
      indexWriter.optimize();
      indexWriter.close();

      IndexSearcher indexSearcher = new IndexSearcher(directory);

      QueryParser parser = new QueryParser("DATA", new StandardAnalyzer());
      Query query = parser.parse("sametoken");
		
      //This is all you have to add to your existing code.
      Filter aclFilter = applyAccessControl(new TermQuery(
         new Term("ACL_FIELD","access")));

      Hits hits = indexSearcher.search(query, aclFilter);
      System.out.println("Hits[" + hits.length() + "]");
      for (int i = 0; i < hits.length(); i++) {
         Document doc = hits.doc(i);
         System.out.println("DATA [" + doc.get("DATA") + 
            "] ACL_FIELD [" + doc.get("ACL_FIELD") + "]");
      }
      indexSearcher.close();	
   }

   private static Filter applyAccessControl(Query aclQuery) {
      return new CachedQueryFilter(aclQuery.toString(), 
         new QueryWrapperFilter(aclQuery));
   }

   private static Document buildDocument(String... fieldInfo) {
      Document document = new Document();
      for (int i = 0; i < fieldInfo.length; i++) {
         String[] split = fieldInfo[i].split(":");
         String fieldName = split[0];
         String fieldValue = split[1];
         document.add(new Field(fieldName,fieldValue,
            Field.Store.YES,Field.Index.TOKENIZED));
      }
      return document;
   }	
}

public class CachedQueryFilter extends Filter {
   private static final long serialVersionUID = 6797293376134753695L;
   private Filter filter;
   private String key;
   private static transient Map<String, BitSetCache> filterCache = 
      new ConcurrentHashMap<String, BitSetCache>();

   public CachedQueryFilter(String key, Filter filter) {
      this.filter = filter;
      this.key = key;
   }

   public BitSet bits(IndexReader reader) throws IOException {
      BitSetCache cachedBitSet = (BitSetCache) filterCache.get(key);
      if (cachedBitSet != null) {
         BitSet bitSet = cachedBitSet.bitSet.get();
         if (bitSet != null && cachedBitSet.indexReaderVersion == reader.getVersion()) {
            return bitSet;
         }
      }
      BitSet bits = filter.bits(reader);
      BitSetCache bitSetCache = new BitSetCache();
      bitSetCache.indexReaderVersion = reader.getVersion();
      bitSetCache.bitSet = new SoftReference<BitSet>(bits);
      filterCache.put(key, bitSetCache);
      return bits;
   }
	
   private class BitSetCache {
      long indexReaderVersion;
      SoftReference<BitSet> bitSet;
   }
}

Requirements

• Fuzzy searching is a must (Soundex, Levenshtein, etc.)

• Has to be fast, a must with any searching solution

• Has to provide access control

• Full data load indexing needs to be completed in a reasonable amount of time

• Scoring needs to be a custom implementation

• Needs to run on a predetermined environment, meaning that new hardware purchases are not going to happen any time soon

• And last but not least is ability do all these things on a dataset that exceeds a billion records

So we have had a lot of constraints to deal with, the hardest one by far is the last one.

The Data

• 1 billion plus records

• Over 30 million unique terms

Indexing and Searching Server Specs

• 20 CPUs

• 32 Gig of ram

• Dedicated SAN storage

First Searching Experiences

After getting the index built in multiple partitions, I fired up a simple Lucene console to do some simple searches with a Lucene multi searcher. Ran out of memory with 2 Gig heap, tried the maximum heap size for the 32 bit JVM we were using, 3.3 Gig, and that ran out of memory as well. So, initial tries to just run one search were unsuccessful.

Then we installed a 64-bit JVM and tried an 8 Gig heap, and it worked! I could run searches and after the first couple of warm up searches it was getting 20 - 80 ms responses on single term searches. Great, but then we tried a Fuzzy search, which uses a Levenshtein algorithm to calculate matches, 2 minutes 45 seconds, this was unacceptable.

Next we wrote our own Levenshtein Lucene query and got the 2 minutes plus search down to about one second. We found that the built in Lucene Fuzzy query was taking 85-95% of the time to find the terms to search. Then after those terms were found the actual search with those expanded terms only took a second to two depending on how many terms were found. So we replaced the built in Fuzzy query with a custom one that gets near instantaneous results on Levenshtein fuzzy matches. Problem solved.

Indexing Time

After our initial proof of concept was complete, we needed to improve the indexing time down to something more reasonable. The index creation from scratch was taking 36 - 48 hours to build with 20 CPUs running at 100% utilization. Which means that the machine was indexing about 9,000 records a second. Not bad for Lucene 2.2, but not that great.

First we stopped merging the indexes after we created them, that by itself was taking about 12 hours. At this point we also started searching these multiple indexes in parallel, and we are seeing modest increases in query performance.

Second, we upgraded to Lucene 2.3, this provided a huge increase in indexing speed. Our index creation time went from 36 - 48 hours (depending on if we merged indexes or not) down to 3-4 hours. The indexing process is now indexing around 125,000+ records a second. Huge improvement, if you haven't upgraded to 2.3, you should!

Current Development

We are in the process of adding access control to Lucene as well as adding new custom queries and scoring. So far Lucene has performed better than any of the competition that it has come up against, and with it's price point it seems to have won acceptance on our project.

In upcoming parts I will go into more details about the technical solutions that we have developed to solve these problems, as well others that I haven't mentioned yet.

Internal Session Id 1207429769869209087112251146956
User Session Id H3qJ4YGjTnTH1Sv0d4nVMBNhr2vdn77m4MKGQ3MT0SVVhQQvsQBk!1447771105!1207429769869
telprdB UserIntB12 java.lang.NullPointerException

That's a lot more information than should be given out to anyone, and is certainly not "user friendly." Do you think they are using Java? That NPE didn't give it away did it? Then again you can pretty much figure that out from the ".do" on the end of the URLs they use, which is one reason why web frameworks these days allow you to map things in a more REST-friendly and technology-agnostic manner using *.html or something like /my/site/person/1. Another rule of thumb: design URLs to be technology agnostic and generic, so that just from a URL it cannot be determined what technology you are using and in case you need or want to switch to a different technology you could theoretically use the same routing scheme in your URLs, which would allow bookmarks to keep working.

HTML filtering was never recommended because it was so difficult to get right, and with no proven libraries, trying to build a solution would almost certainly contain security holes. Thanks to Arshan Dabirsiaghi we finally have something to use. He has created the OWASP AntiSamy project to easily sanitize HTML input. AntiSamy is currently implemented as a Java 1.5 compatible library, but there are plans to support other platforms.

For the record, I don't believe there is any absolutely correct answer. There are valid arguments for each side of the debate. My perspective has mostly been shaped through experience developing custom software for clients. I have little experience developing commercial software and surmise that your thoughts on the subject may differ if that's your day job.

In my experience, most customers view their data as the most important aspect of any system, not the application. They want to be sure their data is safe at all times. What if someone accesses the database through a different application, a third-party data broker or directly via a SQL client? How do you protect the data in those situations if all the restrictions are resident in the application? I hear these questions all the time.

The rebuttal to this argument that I hear most often is that nobody will ever access the data outside of the application. This seems like an obviously shortsighted statement to me and I don't think anyone actually believes it. It's more of a synonym for saying that you'd rather not worry about security at the database layer because it's out of your control, too hard, etc. Pick your favorite excuse. I've probably used them all myself.

What it all boils down to is that the database is very likely to outlive your application. In fact, it'll probably outlive your application and several subsequent ones. Why would a customer want to pay someone to rebuild the security logic each time? It's tedious, expensive, and error prone. Additionally, you're application may not be the only one accessing the database. What are the odds that two development teams implement security constraints the same exact way in each application? Not good, I promise.

As I stated in the beginning, I've definitely changed my perspectives on application versus database-level security. I was certainly an application-level security bigot before, but I find it difficult to continue my unwavering ways when looking at the problem more critically. I just hope my fellow colleagues don't renounce me for siding with the DBAs on this one.

• Cross-site scripting (XSS)
• Cross-site-request forgery (CSRF)
• JSON Hi-jacking
• JavaScript portscanning
• JavaScript and CSS History "Go Fish"

1. Hardware-Encrypted Disk Drives
2. Solid State Disk Drives

I believe the first topic is a direct result of the disclosure of scores of high-profile data breaches resulting from stolen computers. The highest profile instance occurred in May of 2006 when a U.S. Department of Veteran's Affairs laptop containing the names, social security numbers, dates of birth, phone numbers, and addresses of 26.5 million American veterans was stolen. Unauthorized information disclosure has unfortunately become an all too common occurrence in today's world.

In response, vendors such as Seagate have introduced hardware-encrypted notebook hard drives to counter the unintended loss of sensitive information when a computer is stolen. The idea is that data stored on the hard drive is encrypted at all times, rendering the information practically useless to thieves. The encryption is done in specialized hardware to reduce the performance impact associated with the extra computation.

The second topic is the result of a continued increase in flash-based memory capacity to the point where it can now be considered as an alternative to traditional hard drives. Both Samsung and Sandisk have recently announced 64GB (Samsung) and 32GB (Sandisk) products targeted for laptop usage. Since there are no moving parts, flash-based memory consumes much less power than the spinning platters of a traditional hard disk, which is obviously a big benefit for laptop users. As a side benefit, these flash-based memory "drives" are rumored to be anywhere from 4-10x faster than a traditional hard drive, which brings me to the main point of this post.

How long will it be until we see solid-state "drives" such as the ones from Samsung and Sandisk coupled with dedicated hardware-based encryption as demonstrated on the new Seagate drive? A considerable downside to full-disk encryption of traditional drives is the performance impact. Since the new flash-based drives are several times faster, why not bake in full-drive encryption now when nobody will notice it. What you'd end up with are flash-based drives that provide full-disk encryption and still outperform traditional drives. Waiting to add full-disk encryption to later versions of flash-based drives will only introduce the performance debate that faces today's traditional drives, making the purchasing decision all that much more difficult. Why not avoid the confusion and help customers better secure their data in the process?

<table>
<c:forEach var="book" items="${books}">
<tr>
<td><c:out value="${book.title}"/></td>
<td><c:out value="${book.author}"/></td>
<td><c:out value="${book.isbn}"/></td>
</tr>
</c:forEach>
</table>

with this

<table>
<c:forEach var="book" items="${books}">
<tr>
<td>${book.title}</td>
<td>${book.author}</td>
<td>${book.isbn}</td>
</tr>
</c:forEach>
</table>

As you can see, the syntax in the second example is much cleaner. But, “with great power comes great responsibility.” The chink in the armor that every tutorial I’ve read (including Sun’s Java EE 5 Tutorial fails to mention is that the expression syntax does not escape HTML characters. Therefore, any web application using JSP Expressions to output unsanitized, user-entered data will be vulnerable to Cross Site Scripting (XSS) attacks.

Unfortunately, the cure for this XSS vulnerability leads us right back to our first example. As section 2.2.2 of the JSP 2.0 Specification reads, “In cases where escaping is desired (for example, to help prevent cross-site scripting attacks), the JSTL core tag c:out can be used.”

My advice is to use JSP Expressions only as tag library attribute values and stick to using JSTL’s c:out tag for writing text to a page. Deciding which instances of template text expression usage are safe and which are not is simply too error prone and the consequences of a mistake too dangerous.