I have been working on an application to be hosted by Google App Engine (GAE). Initially it was just an experiment, so I didn’t think about automated testing. As it morphed into a real application, I suddenly had the urge to write tests. A quick look at the GAE documentation did not reveal a built-in testing framework. I know about the Python unittest and doctest modules, and writing unit tests for self-contained classes is easy enough. But I really needed integration tests, too, where I can test all of the application code as if it were deployed to GAE. For those to work, I would have to somehow set up the testing environment to either simulate GAE or bootstrap it just enough for my entity and handler classes to operate within the GAE sandbox enforced by the SDK and the production environment.

I figured that somebody must have solved this problem already, so a Google search resulted in two interesting solutions. The first approach involves mocking most GAE objects using a record-and-playback-style library called Mocker (similar to EasyMock for Java). The second approach bootstraps the entire GAE SDK so the tests can be run from the command-line.

After examining these approaches, I was still unsatisfied. The first approach, mocking, is a sensible ways of testing in other platforms like Java, where we strive to avoid direct dependencies on the application server or database by introducing layers of abstraction that need to be mocked in tests. The second approach, bootstrapping, is a viable solution for some platforms (Rails, Grails). But for GAE, these approaches seem cumbersome to me. I want to run my tests in the real environment and not have to maintain many lines of mocking code, or bootstrap code that depends on Google’s internal SDK implementation. Call me lazy, but I want writing tests for my GAE application to be as painless as possible so I can focus on writing the application and testing real functionality.

Then it occurred to me: how about testing from within the GAE environment instead of trying to drive the tests externally? There is no reason to make the job harder when the GAE platform is so well-defined. The SDK exactly simulates the production environment, and it recompiles code changes immediately without needing a server restart. Testing for GAE becomes a no-brainer because the tests can simply be invoked by an HTTP request just like any other function of the GAE application.

First a URL must be mapped to a handler script in the app.yml configuration file, where all of the application’s handler are defined:

# This should only work for testing through the SDK.
- url: /test/?
  script: gaetest/handler.py

Adding this mapping should make many developers justifiably squeamish. After all, this will also be mapped in the production environment where users could invoke the tests, which could be very nasty. The test script will have to self-destruct in production - more on that later.

When I launch the GAE SDK server and visit http://localhost:8080/test in my browser, I want all of my tests to run, with the results displayed in the browser window (nothing fancy, text will do). Here’s the handler.py script, which is located in the gaetest directory under the application root:

import google.appengine.tools # causes an exception in production, as desired
import unittest
import sys
import wsgiref.handlers
from google.appengine.ext import webapp
from tests import * # this requires a tests/__init__.py to define the test modules

class TestSuiteHandler(webapp.RequestHandler):

  def get(self):
    self.response.headers['Content-Type'] = 'text/plain'
    self.response.out.write("=======\n Tests \n=======\n\n")
    modules = [sys.modules['tests.%s' % m] for m in dir(sys.modules['tests']) if m.endswith('_test')]
    loader = unittest.TestLoader()
    suite = unittest.TestSuite()
    for module in modules:
      suite.addTest(loader.loadTestsFromModule(module))
    runner = unittest.TextTestRunner(self.response.out)
    runner.run(suite)

def main():
  application = webapp.WSGIApplication([('/test/?', TestSuiteHandler)], debug=True)
  wsgiref.handlers.CGIHandler().run(application)

if __name__ == '__main__':
  main()

The handler looks for all modules in the tests package, which is a subdirectory under the application root. Modules must be explicitly defined in tests/__init__.py as such:

# List all modules here.
__all__ = ['model_test', 'another_test', 'no_tests_just_helpers']

Then the handler loads all test classes (subclasses of unittest.TestCase) from modules that end in _test, and adds them to the test suite. The tests are executed using the standard text-based test runner, and the output is rendered in the browser.

Notice that the first line of handler.py attempts to import google.appengine.tools. I chose this package because I’m fairly certain that it will not be available in production, and will therefore raise an exception if the handler is executed in that environment.

At this point, tests will run just fine in the environment provided by the GAE SDK server. Here’s an example test class:

import unittest
from google.appengine.ext import db

class MyEntity(db.Model):
  name = db.StringProperty()

class MyEntityTest(unittest.TestCase):

  def test_new_entity(self):
    entity = MyEntity(name='Foo')
    self.assertEqual('Foo', entity.name)

  def test_saved_enitity(self):
    entity = MyEntity(name='Foo')
    key = entity.put()
    self.assertEqual('Foo', db.get(key).name)

Can you spot the trouble with this test? If it I run the test over and over again as I actively develop and manually test the application, without restarting the server, then many MyEntity instances will pile up in my local datastore, which is a file-based stub used by the GAE SDK to simulate the production datastore. For this trivial example, that would simply be annoying. But in a complex suite of tests, it is undesirable to leave persistent objects in the datastore between runs. It is also nice to have a separate test datastore so development data doesn’t get in the way of asserting on objects stored by the tests. In other platforms, such as Ruby on Rails, not only do the tests get their own datastore, but the data is rolled back after each test by running them in a transaction.

The GAE datastore supports very limited transactions. Unfortunately the restrictions prevent them from being used to roll back all entities created by a test.

Let’s revisit the second approach that I had found to testing in GAE. It turns out that this example, which I had previously dismissed, contains the solution to the above problem. The SDK contains a number of “stub” service classes for API modules such as the datastore, users, urlfetch, and mail. The example recreates the SDK environment by registering these stubs through an API proxy class, which is used by the rest of the SDK code to lookup these services when needed. The datastore stub can be re-instantiated cleanly using the internal API, and the original datastore can be recovered so development entities are not lost. Since I run the tests from within the SDK server, the rest of the service stubs do not need to be replaced. Despite my misgivings about depending on the internal SDK implementation, the ability to reset the datastore state between tests is just too important to ignore.

In order to achieve this, I need to create a special base TestCase that installs the test datastore in setUp() and recovers the development datastore in tearDown(). All test classes that create entities, or that test code that creates entities, will need to inherit from the AppEngineTestCase, located in gaetest\base.py:

import unittest
from google.appengine.api import apiproxy_stub_map  
from google.appengine.api import datastore_file_stub

class AppEngineTestCase(unittest.TestCase):

  def setUp(self):
    # Preserve the current apiproxy for tearDown().
    self.original_apiproxy = apiproxy_stub_map.apiproxy

    # Create a new apiproxy and temporary datastore that will be used for this test.
    apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap() 
    temp_stub = datastore_file_stub.DatastoreFileStub('AppEngineTestCaseDataStore', None, None)  
    apiproxy_stub_map.apiproxy.RegisterStub('datastore_v3', temp_stub)

    # For convenience, the subclass can implement 'set_up' rather than overriding setUp()
    # and calling this base method.
    if hasattr(self, 'set_up'):
      self.set_up()

  def tearDown(self):
    # The subclass can optionally choose to implement 'tear_down'.
    if hasattr(self, 'tear_down'):
      self.tear_down()

    # Restore stubs for development.
    apiproxy_stub_map.apiproxy = self.original_apiproxy

The sample unit test class now extends the new base:

  import unittest
  from google.appengine.ext import db
  import gaetest.base

  from google.appengine.ext import db

  class MyEntity(db.Model):
    name = db.StringProperty()

  class MyEntityTest(gaetest.base.AppEngineTestCase):

    def set_up(self):
      # Populate test entities here along with other setup.
      entity = MyEntity(name='Bar')
      self.setup_key = entity.put()

    def tear_down(self):
      # Tear down here, but there is no need to delete test entities.
      pass

    def test_new_entity(self):
      entity = MyEntity(name='Foo')
      self.assertEqual('Foo', entity.name)

    def test_saved_enitity(self):
      entity = MyEntity(name='Foo')
      key = entity.put()
      self.assertEqual('Foo', db.get(key).name)

    def test_setup_entity(self):
      entity = db.get(self.setup_key)
      self.assertEqual('Bar', entity.name)

When the above test is run, two entities are created (with names ‘Foo’ from test_saved_enitity() and ‘Bar’ from test_setup_entity()), but will not be present in the development datastore.

That’s all for now. Feel free to download the example, packaged as a GAE application. Enjoy writing tests for Google App Engine!

Note: In the midst of writing this blog, I discovered that a very similar library, called GAEUnit, was recently added to Google Code. Check it out.

UPDATE: My above efforts will soon be incorporated into GAEUnit. The original GAEUnit author, George Lei, and I are now partnering to make it the testing framework of choice for Google App Engine developers.

I know this goes against the collective wisdom of Java developers everywhere, but I’ll say it anyway: Java Logging is not so bad. Granted, its API isn’t as simple as Apache Commons Logging or Log4J, and you may end up writing your own Formatter, but its core implementation is sound. Of course, if you depend on libraries such as Spring, they require you to include ‘commons-logging.jar’ in your classpath. In that case you can still choose to use the Java Logging API directly in your application and tell Commons to do the same by using a ‘commons-logging.properties’ file. There are numerous other logging utilities that you should also explore when deciding which library to commit to.

So let’s assume that you have chosen to use Java Logging, but you are not totally sold on the API. You could choose to use Commons Logging if you don’t mind the extra dependency. But with a minimal amount of effort, it’s not so difficult to write a thin wrapper around Java Logging that makes it more convenient to use than even the Apache offerings:

public final class Log {
  private Log() { }

  private static final Map<String, Logger> loggers = new HashMap<String, Logger>();

  public static void debug(String message, Object... params) {
    log(Level.FINE, message, params);
  }

  public static void info(String message, Object... params) {
    log(Level.INFO, message, params);
  }

  public static void warn(String message, Object... params) {
    log(Level.WARNING, message, params);
  }

  public static void error(String message, Object... params) {
    log(Level.SEVERE, message, params);
  }

  public static void log(Level level, String message, Object... params) {
    final int stackPositionOfCaller = 2;
    StackTraceElement caller = new Throwable().getStackTrace()[stackPositionOfCaller];
    String className = caller.getClassName();
    Logger logger;
    synchronized (loggers) {
      logger = loggers.get(className);
      if (logger == null) {
        logger = Logger.getLogger(className);
        loggers.put(className, logger);
      }
    }
    if (logger.isLoggable(level)) {
      String formattedMessage;
      Throwable thrown = null;
      if (params.length == 0) {
        formattedMessage = message;
      }
      else {
        Object last = params[params.length - 1];
        if (last instanceof Throwable) {
          Object[] subParams = new Object[params.length - 1];
          System.arraycopy(params, 0, subParams, 0, subParams.length);
          formattedMessage = String.format(message, subParams);
          thrown = (Throwable) last;
        }
        else {
          formattedMessage = String.format(message, params);
        }
      }
      LogRecord record = new LogRecord(level, formattedMessage);
      record.setLoggerName(logger.getName());
      record.setSourceClassName(className);
      record.setSourceMethodName(caller.getMethodName());
      record.setThrown(thrown);
      record.setParameters(params);
      logger.log(record);
    }
  }
}

This particular implementation provides the following advantages:

• The Logger name is automatically set to be the fully-qualified class name of the caller, by using stack introspection.
• Log messages are formatted using the String.format() patterns and variable arguments. The parameters are also assigned to the underlying LogRecord.
• If the last parameter to a method is a Throwable, it will be assigned to the LogRecord.
• The method names align more closely with the Log4j and Commons Logging method names.

All methods are static wrappers around a Java Logger. Each Logger is created as needed and then cached, one per calling class. Execution time is about the same when compared to direct calls to Logger.

You can download the code and use it as a starting point for your own logging utility. A JUnit test class is included which, in addition to testing the basic API, measures and compares execution times of the Java Logger and the Log class presented here.

The Purpose

I have never been impressed by the XML support supplied by Java. The DOM and SAX APIs seem overly complex for basic XML parsing. Even using XPath in Java is a chore. I have also used XSLT for certain applications, but that technology has its own complexity and readability issues. When I learned about Groovy’s GPath, I knew there were better days ahead.

Naturally, the question of performance comes up when talking about using Groovy. So I have put together a simple test which pits GPath against XPath. The results might convince you that Groovy can replace incumbent Java technologies for certain types of applications. When in doubt, measure it.

The Test

This test is not at all a comprehensive measure of all XML-parsing functionality; it doesn’t even come close to exercising the full query capabilities of XPath. The only question that is posed is How long does it take to search an XML document for matching attributes? The three implementations that will answer this question are Java using XPath, Groovy using XPath, and Groovy using GPath.

The XML document searched is four nodes deep (the root element, two descendent element levels, and a text node). Every element except for the root contains exactly the same attribute name and value. For example:

<root>
  <node1 aName='aValue'>
    <node1.1 aName='aValue'>1.1</node1.1>
    <node1.2 aName='aValue'>1.2</node1.2>
    ...
    <node1.N aName='aValue'>1.N</node1.N>
  </node1>
  <node2 aName='aValue'>
    <node2.1 aName='aValue'>2.1</node2.1>
    <node2.2 aName='aValue'>2.2</node2.2>
    ...
    <node2.N aName='aValue'>2.N</node2.N>
  </node2>
  ...
  <nodeN aName='aValue'>
    <nodeN.1 aName='aValue'>N.1</nodeN.1>
    <nodeN.2 aName='aValue'>N.2</nodeN.2>
    ...
    <nodeN.N aName='aValue'>N.N</nodeN.N>
  </nodeN>
</root>

The total number of elements with the ‘aName’ attribute is N * N + N. The test measures performance for increasingly larger values of N, from 1 to 340, so the node total ranges from 2 to 115940 (not counting the root node). Each search is run 10 times for each of the three implementations.

All Groovy code except for the test script itself was pre-compiled into Java class files before executing the test. In order to minimize the timing impact of nondeterministic JVM behaviors, the script was run with 512MB of Java heap memory allocated up front by first setting export JAVA_TOOL_OPTIONS="-Xms512m -Xmx512m", an attempt to garbage collect synchronously was made by calling System.gc() between iterations, and 10 iterations were executed with N=100 to “warm up” HotSpot. The machine used to test is a 2.16 GHz Intel Core 2 Duo MacBook Pro, running Mac OS X 10.5.1, with Java 1.5.0_13-119 and Groovy 1.5.0.

The Code

The code being tested consists of one interface and three different implementations whose performance characteristics will be compared.

Selector.groovy

The Selector interface, through its countNodesForAttribute() method, exposes the purpose of this test: to find all elements that contain a specific attribute name and value. Since the goal is to measure searching performance, matching elements are merely counted, not gathered and collected. countNodesForAttribute() is the only method whose execution time is recorded by the test script.

interface Selector {
  void setInputSource(org.xml.sax.InputSource xmlInput)
  Integer countNodesForAttribute(String attributeName, String attributeValue)
}

JavaXPathSelector.java

The first contender in this contest is a tried-and-true Java class. JavaXPathSelector uses the default XPath implementation built into Java 5. This code follows the recommended way to create its XML classes using factories. Because XPath expressions need to be compiled once, a cache is used to store compiled expressions. This has no impact on the test measurements because only one expression is used in this test, and it gets compiled during the initial iterations whose results are discarded.

Note that the XPath expression string is assembled using String.format(). Also notice that checked exceptions are caught and uncheck exceptions thrown instead. These are pointed out for contrast with the Groovy implementation that follows.

import org.w3c.dom.*;
import javax.xml.xpath.*;
import java.util.*;

public class JavaXPathSelector implements Selector {

  private Node document;
  private XPath xpath;
  private Map<String, XPathExpression> exprCache;

  public JavaXPathSelector() { }

  public void setInputSource(org.xml.sax.InputSource xmlInput) {
    try {
      exprCache = new HashMap<String, XPathExpression>();
      xpath = XPathFactory.newInstance().newXPath();
      document = javax.xml.parsers.DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(xmlInput);   
    }
    catch (javax.xml.parsers.ParserConfigurationException e) { throw new RuntimeException(e); }
    catch (org.xml.sax.SAXException e) { throw new RuntimeException(e); }
    catch (java.io.IOException e) { throw new RuntimeException(e); }
  }

  public Integer countNodesForAttribute(String attributeName, String attributeValue) {
    try {
      XPathExpression xpe = getXPathExpression(String.format("//*[@%s='%s']", attributeName, attributeValue));
      NodeList nodes = (NodeList) xpe.evaluate(document, XPathConstants.NODESET);
      return new Integer(nodes.getLength());
    }
    catch (XPathExpressionException e) { throw new RuntimeException(e); }
  }

  private XPathExpression getXPathExpression(String expression) {
    try {
      XPathExpression xpe = exprCache.get(expression);
      if (xpe == null) {
        xpe = xpath.compile(expression);
        exprCache.put(expression, xpe);
      }
      return xpe;
    }
    catch (XPathExpressionException e) { throw new RuntimeException(e); }
  }
}

GroovyXPathSelector.groovy

This implementation is nearly identical to JavaXPathSelector, but with some Groovy enhancements: the semicolons are gone, the public keyword is dropped, bracket notation is used for the map, and checked exceptions do not need to be handled. Also, note that the use of Java’s String.format() has been replaced with a concise GString for creating the XPath expression string. I chose not to use the def keyword because I prefer the clarity of declaring the types.

I included this implementation because I was curious to know whether any overhead cost was incurred by using Groovy rather than Java.

import org.w3c.dom.*
import javax.xml.xpath.*

class GroovyXPathSelector implements Selector {

  private Node document
  private XPath xpath
  private Map<String, XPathExpression> exprCache

  GroovyXPathSelector() { }

  void setInputSource(org.xml.sax.InputSource xmlInput) {
    exprCache = new HashMap<String, XPathExpression>()
    xpath = XPathFactory.newInstance().newXPath()
    document = javax.xml.parsers.DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(xmlInput)    
  }

  Integer countNodesForAttribute(String attributeName, String attributeValue) {
    XPathExpression xpe = getXPathExpression("//*[@${attributeName}='${attributeValue}']")
    NodeList nodes = xpe.evaluate(document, XPathConstants.NODESET)
    return nodes.length
  }

  private XPathExpression getXPathExpression(String expression) {
    XPathExpression xpe = exprCache[expression]
    if (!xpe) {
      xpe = xpath.compile(expression)
      exprCache[expression] = xpe
    }
    return xpe
  }
}

GroovyGPathSelector.groovy

Finally, here is the GPath challenger. Astute readers will notice that there is no real equivalent to an XPath expression to be compiled and executed. GPath doesn’t work that way. Rather, GPath expressions are Groovy expressions. The structure of the XML document is mapped to a graph of objects that provide dynamic access to the XML nodes through properties and special methods that take great advantage of closures. The way the attributes are searched in this case is by calling findAll() on the iterator returned by document.depthFirst(), with a closure that evaluates a boolean GPath expression looking for an attribute name/value match.

class GroovyGPathSelector implements Selector {

  private groovy.util.slurpersupport.GPathResult document

  GroovyGPathSelector() { }

  void setInputSource(org.xml.sax.InputSource xmlInput) {
    document = new groovy.util.XmlSlurper().parse(xmlInput)
  }

  Integer countNodesForAttribute(String attributeName, String attributeValue) {
    return document.depthFirst().findAll { it["@${attributeName}"] == attributeValue }.size
  }
}

The Results

The performance of each Selector implementation was tested with XML documents containing the following numbers of elements (minus the root element): 2, 12, 20, 30, 110, 420, 930, 2070, 4970, 10100, 22650, 50850, 115940. The recorded times were converted to elements/milliseconds so the vertical scale of the chart could be kept within a reasonable range.

Without further ado, here are the results:

The first thing you might notice is that Java XPath and Groovy XPath perform identically. That makes sense since the majority of the processing is happening in pure Java code via the javax.xml.xpath classes. Because of the similarity, we can talk simply about XPath versus GPath.

GPath readily beats XPath for documents under 100 elements, and appears to hold up well for documents greater than 50000 elements. I don’t think it’s worthwhile to test larger documents since many of us are working with document sizes on the lower end of the scale.

The real surprise for me was the XPath performance curve. XPath beats GPath for documents sized from approximately 400 to 20000 elements. But I never expected XPath to perform quite so poorly for smaller documents (considering the compiled expression was cached), nor did I expect such a sharp decline for very large documents. Perhaps that is due to my ignorance of how XPath really works.

The Conclusions

• GPath performs much better that XPath when searching for attributes in XML documents containing less than 100 elements. It seems likely that Groovy and GPath is a good choice for applications that must quickly search documents of this size.

• For XML documents with hundreds to several thousand elements, XPath moderately out-performs GPath. For applications that must quickly search documents sized within this range, XPath is a good choice (but it may be feasible to implement the wrapping code in Groovy.)

• When a majority of the processing time is spent inside of Java-compiled classes, overhead introduced by Groovy wrapper/glue code is not discernable.

Personally, I feel more comfortable about using Groovy and GPath for searching XML now that I have some conclusive performance numbers. This test measured a very specific type of search, but it wouldn’t be too difficult to write tests to compare GPath vs. DOM vs. SAX for general parsing.

The Download

Feel free to download the code and run the tests on your own. It includes the RunComparison.groovy test script that I did not delve into.