Recently about Security
Near Infinity recently announced the release of Grant, a Ruby on Rails plugin for securing and auditing access to your Rails model objects, and I'm here to tell you a little bit about it. There are two primary pieces of Grant, model security and model audit. I'll be focusing on model security for this post and will address model audit in a later entry.
Grant's model security is deliberately designed to force the developer to make conscious security decisions about what CRUD operations a user should be allowed to perform on your model objects. It doesn't care how you choose to authenticate and authorize your users to perform a CRUD operation, it only cares that you actually do it.
Rather than specify which operations are restricted, Grant restricts all CRUD operations unless they're explicitly granted to the user. It also restricts adding or removing items from has_many and has_and_belongs_to_many associations. Only allowing operations explicitly granted forces you to make conscious security decisions. While it obviously can't ensure you make the correct decisions, it should help ease the latent fear that you've inadvertently forgotten to secure something.
Enough talk, let me show you an example of how you might use it. To enable model security you simply include the Grant::ModelSecurity module in your model class. In this example you see three grant statements. The first grants find (aka read) permission to everyone. The second example grants create, update, and destroy permission when the passed block evaluates to true, which in this case happens when the model is editable by the current user. You can put any code you want in that block as long as it returns a boolean value. Similarly, the third grant statement permits additions and removals from the tags association when it's block evaluates to true. A Grant::ModelSecurityError is raised if any grant block evaluates to false or nil.
class EditablePage < ActiveRecord::Base
include Grant::ModelSecurity
has_many :tags
grant(:find) { true }
grant(:create, :update, :destroy) do |user, model|
model.editable_by_user? user
end
grant(:add => :tags, :remove => :tags) do |user, model, associated_model|
model.editable_by_user? user
end
def editable_by_user? user
user.administrator?
end
end
There's a lot more to the grant statement than shown in the above example. For instance, you can have multiple grant statements for the same action. Ultimate permission to perform the action will not be granted unless all grant blocks evaluate to true.
As you can see, Grant is pretty simple to use, but it's not going to do the dirty work for you. It's up to you to make the proper security decisions. Grant's just there to make sure you don't forget.
In the below example there are two fields:
DATA: Which contains any data that you want your users to be able to search. NOTE: You can have as many data fields as you like.
ACL_FIELD: The field used to determine what users have access to this document. Note: You can have as many access control fields as you like.
All you have to do is built the access control query for each user and submit your user's query unchanged.
public class TestIndexerSearcher {
public static void main(String[] args) throws Exception {
Directory directory = new RAMDirectory();
IndexWriter indexWriter = new IndexWriter(directory, new StandardAnalyzer());
indexWriter.addDocument(buildDocument("DATA:sametoken","ACL_FIELD:access"));
indexWriter.addDocument(buildDocument("DATA:sametoken","ACL_FIELD:noaccess"));
indexWriter.optimize();
indexWriter.close();
IndexSearcher indexSearcher = new IndexSearcher(directory);
QueryParser parser = new QueryParser("DATA", new StandardAnalyzer());
Query query = parser.parse("sametoken");
//This is all you have to add to your existing code.
Filter aclFilter = applyAccessControl(new TermQuery(
new Term("ACL_FIELD","access")));
Hits hits = indexSearcher.search(query, aclFilter);
System.out.println("Hits[" + hits.length() + "]");
for (int i = 0; i < hits.length(); i++) {
Document doc = hits.doc(i);
System.out.println("DATA [" + doc.get("DATA") +
"] ACL_FIELD [" + doc.get("ACL_FIELD") + "]");
}
indexSearcher.close();
}
private static Filter applyAccessControl(Query aclQuery) {
return new CachedQueryFilter(aclQuery.toString(),
new QueryWrapperFilter(aclQuery));
}
private static Document buildDocument(String... fieldInfo) {
Document document = new Document();
for (int i = 0; i < fieldInfo.length; i++) {
String[] split = fieldInfo[i].split(":");
String fieldName = split[0];
String fieldValue = split[1];
document.add(new Field(fieldName,fieldValue,
Field.Store.YES,Field.Index.TOKENIZED));
}
return document;
}
}
After you run this code, you will get a single hit, not the two that you would normally get if the access control filter wasn't in place.
public class CachedQueryFilter extends Filter {
private static final long serialVersionUID = 6797293376134753695L;
private Filter filter;
private String key;
private static transient Map<String, BitSetCache> filterCache =
new ConcurrentHashMap<String, BitSetCache>();
public CachedQueryFilter(String key, Filter filter) {
this.filter = filter;
this.key = key;
}
public BitSet bits(IndexReader reader) throws IOException {
BitSetCache cachedBitSet = (BitSetCache) filterCache.get(key);
if (cachedBitSet != null) {
BitSet bitSet = cachedBitSet.bitSet.get();
if (bitSet != null && cachedBitSet.indexReaderVersion == reader.getVersion()) {
return bitSet;
}
}
BitSet bits = filter.bits(reader);
BitSetCache bitSetCache = new BitSetCache();
bitSetCache.indexReaderVersion = reader.getVersion();
bitSetCache.bitSet = new SoftReference<BitSet>(bits);
filterCache.put(key, bitSetCache);
return bits;
}
private class BitSetCache {
long indexReaderVersion;
SoftReference<BitSet> bitSet;
}
}
There are two additional features that this query filter doesn't implements that you may want to consider.1st - Provide per query locking around the bitset creation code. This would allow multiple bitset creation calls to occur at once, but the same access control query would block. Therefore we would only have to build it once, even if multiple user queries with the same access control hit the query filter at once.
2nd - Persist the bitsets. In the past I have used the same directory as the index, but you may want to use a database, or something else.
My current project has some unique searching requirements.
Requirements
Fuzzy searching is a must (Soundex, Levenshtein, etc.)
Has to be fast, a must with any searching solution
Has to provide access control
Full data load indexing needs to be completed in a reasonable amount of time
Scoring needs to be a custom implementation
Needs to run on a predetermined environment, meaning that new hardware purchases are not going to happen any time soon
And last but not least is ability do all these things on a dataset that exceeds a billion records
So we have had a lot of constraints to deal with, the hardest one by far is the last one.
The Data
1 billion plus records
Over 30 million unique terms
Indexing and Searching Server Specs
20 CPUs
32 Gig of ram
Dedicated SAN storage
First Searching Experiences
After getting the index built in multiple partitions, I fired up a simple Lucene console to do some simple searches with a Lucene multi searcher. Ran out of memory with 2 Gig heap, tried the maximum heap size for the 32 bit JVM we were using, 3.3 Gig, and that ran out of memory as well. So, initial tries to just run one search were unsuccessful.
Then we installed a 64-bit JVM and tried an 8 Gig heap, and it worked! I could run searches and after the first couple of warm up searches it was getting 20 - 80 ms responses on single term searches. Great, but then we tried a Fuzzy search, which uses a Levenshtein algorithm to calculate matches, 2 minutes 45 seconds, this was unacceptable.
Next we wrote our own Levenshtein Lucene query and got the 2 minutes plus search down to about one second. We found that the built in Lucene Fuzzy query was taking 85-95% of the time to find the terms to search. Then after those terms were found the actual search with those expanded terms only took a second to two depending on how many terms were found. So we replaced the built in Fuzzy query with a custom one that gets near instantaneous results on Levenshtein fuzzy matches. Problem solved.
Indexing Time
After our initial proof of concept was complete, we needed to improve the indexing time down to something more reasonable. The index creation from scratch was taking 36 - 48 hours to build with 20 CPUs running at 100% utilization. Which means that the machine was indexing about 9,000 records a second. Not bad for Lucene 2.2, but not that great.
First we stopped merging the indexes after we created them, that by itself was taking about 12 hours. At this point we also started searching these multiple indexes in parallel, and we are seeing modest increases in query performance.
Second, we upgraded to Lucene 2.3, this provided a huge increase in indexing speed. Our index creation time went from 36 - 48 hours (depending on if we merged indexes or not) down to 3-4 hours. The indexing process is now indexing around 125,000+ records a second. Huge improvement, if you haven't upgraded to 2.3, you should!
Current Development
We are in the process of adding access control to Lucene as well as adding new custom queries and scoring. So far Lucene has performed better than any of the competition that it has come up against, and with it's price point it seems to have won acceptance on our project.
In upcoming parts I will go into more details about the technical solutions that we have developed to solve these problems, as well others that I haven't mentioned yet.
Quick rule of thumb: Don't show users cryptic error messages. This one was an error I recently received at a major airline's web site while checking in online for a flight:
Internal Session Id 1207429769869209087112251146956 User Session Id H3qJ4YGjTnTH1Sv0d4nVMBNhr2vdn77m4MKGQ3MT0SVVhQQvsQBk!1447771105!1207429769869 telprdB UserIntB12 java.lang.NullPointerException
That's a lot more information than should be given out to anyone, and is certainly not "user friendly." Do you think they are using Java? That NPE didn't give it away did it? Then again you can pretty much figure that out from the ".do" on the end of the URLs they use, which is one reason why web frameworks these days allow you to map things in a more REST-friendly and technology-agnostic manner using *.html or something like /my/site/person/1. Another rule of thumb: design URLs to be technology agnostic and generic, so that just from a URL it cannot be determined what technology you are using and in case you need or want to switch to a different technology you could theoretically use the same routing scheme in your URLs, which would allow bookmarks to keep working.
I gave a JavaScript security talk last month, and one of the topics was HTML filtering. I gave examples of how MySpace tried to filter executable code, while still allowing HTML tags for formatting. MySpace, of course, failed to foresee every attack vector, and the Samy worm was born.
HTML filtering was never recommended because it was so difficult to get right, and with no proven libraries, trying to build a solution would almost certainly contain security holes. Thanks to Arshan Dabirsiaghi we finally have something to use. He has created the OWASP AntiSamy project to easily sanitize HTML input. AntiSamy is currently implemented as a Java 1.5 compatible library, but there are plans to support other platforms.
Here's a sample usage
AntiSamy sanitizer = new AntiSamy();
CleanResults results = sanitizer.scan(request.getParameter("html"));
String html = results.getCleanHTML();
if (!results.getErrorMessages().isEmpty()) {
log.warn("Input contains errors");
}
