Thanks to Near Infinity's generous training budget, I had the opportunity last week to spend 3 days with several members of the Object Mentors group. These guys have an enormous amount of experience, especially "Uncle" Bob Martin. So I thought I would share a few of the pearls of wisdom they dropped along the way:

A recent project started out life as an all-Java project that used Maven as the build tool. Initially we used Atlassian Clover to measure unit test coverage. Clover is a great product for Java code, but unfortunately it only works with Java code because it works at the Java source level. (This was the case as of Spring 2009, and I haven't checked since then.) As we started migrating existing code from Java to Groovy and writing new code in Groovy, we started to lose data about unit test coverage because Clover does not understand Groovy code. To remedy this problem we switched from Clover to Cobertura, which instruments at the bytecode level and thus works with Groovy code. Theoretically it would also work with any JVM-based language but I'm not sure whether or not it could handle something like Clojure or not.

In any case, we only cared about Groovy so Cobertura was a good choice. With the Cobertura Maven plugin we quickly found a problem, which was that even though the code coverage was running, the reports only showed coverage for Java code, not Groovy. This blog shows you how to display coverage on Groovy code when using Maven and the Cobertura plugin. In other words, I'll show how to get Cobertura reports to link to the real Groovy source code in Maven, so you can navigate Cobertura reports as you normally would.

The core problem is pretty simple, though it took me a while to figure out how to fix it. Seems to be pretty standard in Maven: I know what I want to do, but finding out how to do it is the really hard part. The only thing you need to do is tell Maven about the Groovy source code and where it lives. The way I did this is to use the Codehaus build-helper-maven-plugin which has an add-source goal. The add-source goal does just what you would expect; it adds a specified directory (or directories) as a source directory in your Maven build. Here's how you use it in your Maven pom.xml file:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <executions>
        <execution>
            <phase>generate-sources</phase>
            <goals>
                <goal>add-source</goal>
            </goals>
            <configuration>
                <sources>
                    <source>src/main/groovy</source>
                </sources>
            </configuration>
        </execution>
    </executions>
</plugin>

In the above code snippet, we're using the "build-helper-maven-plugin" to add the src/main/groovy directory. That's pretty much it. Run Cobertura as normal, view the reports, and you should now see coverage on Groovy source code as well as Java.

"We're missing some coverage..."

While creating coverage reports for a fairly new JRuby on Rails project, we noticed that our coverage numbers weren't quite right: certain classes were missing from the coverage reports. Rcov doesn't know about classes unless they are required: not a problem for models, but we were missing tests for some controllers and libraries.

Oops.

To properly correct this problem, I wrote the coverage_helper (to live alongside the test_helper). Basically, this causes all of the classes to be required so that rcov knows about them.

test\coverage_helper.rb

require 'test/unit'
require 'test_helper'

class CoverageHelper < Test::Unit::TestCase
  def test_coverage
    ['app', 'lib'].each {|path| Dir.glob("#{path}/**/*.rb") {|file| 
      require File.expand_path(file.chomp('.rb'))
    }}
    assert true
  end
end

Simply include this test in your rcov builds and the problem is solved.

Because of how rcov counts lines and the way Ruby class loading works, you'll never see files with 0% coverage. However, at least you will see all of your classes listed and those that aren't covered will have a low percentage.

Is this really necessary?

First, the File.expand_path makes sure that your files are only required once. I hate random warning messages because constants are initialized twice (among other issues).

Second, no, I didn't need to make this a test, but it just seemed nicer to. I added the assert true simply because I didn't feel right about not asserting something in the test.

Third, as long as one uses the Rails scripts to generate the skeletons for your code, this scenario should never happen (because Rails will create all of the appropriate tests). However, there is the tendency not to use the generated scripts when they don't output what you want, which is what we have discovered (Rails and Legacy Database Schemas aren't a perfect fit). Also, sometimes I just forget to use them.

What if you can't run rcov?

One minor glitch of running JRuby on Windows is that the File.separator is technically incorrect (it's '/' instead of '\'). This usually isn't an issue... except when using rcov. Since rcov executes from the shell, the arguments requiring file names and/or directories won't work because the separator is the wrong direction from what Windows is expecting.

The fix is to add a couple of methods to the File class to address the problem.

Windows Separator Fix

class File
  @@is_windows = ENV['OS'] && 
    (not ENV['OS'].downcase.match(/^windows/).nil?)

  def self.fix_name(name)
    @@is_windows ? name.tr(File::SEPARATOR, File::ALT_SEPARATOR) : name
  end

  def self.fixed_join(*files)
    self.fix_name(self.join files)
  end
end

The reason to do the ENV['OS'] truth check first is that in JRuby on Solaris (where our CI is), that property doesn't exist. We couldn't use the RUBY_PLATFORM variable either, as in JRuby it's always assigned to 'java'.

I should note that I've only use these fixed separator methods when necessary (in my rcov rake task). The 'normal' separator has worked in every other situation I've run into.

I have been working on an application to be hosted by Google App Engine (GAE). Initially it was just an experiment, so I didn't think about automated testing. As it morphed into a real application, I suddenly had the urge to write tests. A quick look at the GAE documentation did not reveal a built-in testing framework. I know about the Python unittest and doctest modules, and writing unit tests for self-contained classes is easy enough. But I really needed integration tests, too, where I can test all of the application code as if it were deployed to GAE. For those to work, I would have to somehow set up the testing environment to either simulate GAE or bootstrap it just enough for my entity and handler classes to operate within the GAE sandbox enforced by the SDK and the production environment.

I figured that somebody must have solved this problem already, so a Google search resulted in two interesting solutions. The first approach involves mocking most GAE objects using a record-and-playback-style library called Mocker (similar to EasyMock for Java). The second approach bootstraps the entire GAE SDK so the tests can be run from the command-line.

After examining these approaches, I was still unsatisfied. The first approach, mocking, is a sensible ways of testing in other platforms like Java, where we strive to avoid direct dependencies on the application server or database by introducing layers of abstraction that need to be mocked in tests. The second approach, bootstrapping, is a viable solution for some platforms (Rails, Grails). But for GAE, these approaches seem cumbersome to me. I want to run my tests in the real environment and not have to maintain many lines of mocking code, or bootstrap code that depends on Google's internal SDK implementation. Call me lazy, but I want writing tests for my GAE application to be as painless as possible so I can focus on writing the application and testing real functionality.

Then it occurred to me: how about testing from within the GAE environment instead of trying to drive the tests externally? There is no reason to make the job harder when the GAE platform is so well-defined. The SDK exactly simulates the production environment, and it recompiles code changes immediately without needing a server restart. Testing for GAE becomes a no-brainer because the tests can simply be invoked by an HTTP request just like any other function of the GAE application.

First a URL must be mapped to a handler script in the app.yml configuration file, where all of the application's handler are defined:

# This should only work for testing through the SDK.
- url: /test/?
  script: gaetest/handler.py

Adding this mapping should make many developers justifiably squeamish. After all, this will also be mapped in the production environment where users could invoke the tests, which could be very nasty. The test script will have to self-destruct in production - more on that later.

When I launch the GAE SDK server and visit http://localhost:8080/test in my browser, I want all of my tests to run, with the results displayed in the browser window (nothing fancy, text will do). Here's the handler.py script, which is located in the gaetest directory under the application root:

import google.appengine.tools # causes an exception in production, as desired
import unittest
import sys
import wsgiref.handlers
from google.appengine.ext import webapp
from tests import * # this requires a tests/__init__.py to define the test modules

class TestSuiteHandler(webapp.RequestHandler):

  def get(self):
    self.response.headers['Content-Type'] = 'text/plain'
    self.response.out.write("=======\n Tests \n=======\n\n")
    modules = [sys.modules['tests.%s' % m] for m in dir(sys.modules['tests']) if m.endswith('_test')]
    loader = unittest.TestLoader()
    suite = unittest.TestSuite()
    for module in modules:
      suite.addTest(loader.loadTestsFromModule(module))
    runner = unittest.TextTestRunner(self.response.out)
    runner.run(suite)

def main():
  application = webapp.WSGIApplication([('/test/?', TestSuiteHandler)], debug=True)
  wsgiref.handlers.CGIHandler().run(application)

if __name__ == '__main__':
  main()

The handler looks for all modules in the tests package, which is a subdirectory under the application root. Modules must be explicitly defined in tests/__init__.py as such:

# List all modules here.
__all__ = ['model_test', 'another_test', 'no_tests_just_helpers']

Then the handler loads all test classes (subclasses of unittest.TestCase) from modules that end in _test, and adds them to the test suite. The tests are executed using the standard text-based test runner, and the output is rendered in the browser.

Notice that the first line of handler.py attempts to import google.appengine.tools. I chose this package because I'm fairly certain that it will not be available in production, and will therefore raise an exception if the handler is executed in that environment.

At this point, tests will run just fine in the environment provided by the GAE SDK server. Here's an example test class:

import unittest
from google.appengine.ext import db

class MyEntity(db.Model):
  name = db.StringProperty()

class MyEntityTest(unittest.TestCase):

  def test_new_entity(self):
    entity = MyEntity(name='Foo')
    self.assertEqual('Foo', entity.name)

  def test_saved_enitity(self):
    entity = MyEntity(name='Foo')
    key = entity.put()
    self.assertEqual('Foo', db.get(key).name)

Can you spot the trouble with this test? If it I run the test over and over again as I actively develop and manually test the application, without restarting the server, then many MyEntity instances will pile up in my local datastore, which is a file-based stub used by the GAE SDK to simulate the production datastore. For this trivial example, that would simply be annoying. But in a complex suite of tests, it is undesirable to leave persistent objects in the datastore between runs. It is also nice to have a separate test datastore so development data doesn't get in the way of asserting on objects stored by the tests. In other platforms, such as Ruby on Rails, not only do the tests get their own datastore, but the data is rolled back after each test by running them in a transaction.

The GAE datastore supports very limited transactions. Unfortunately the restrictions prevent them from being used to roll back all entities created by a test.

Let's revisit the second approach that I had found to testing in GAE. It turns out that this example, which I had previously dismissed, contains the solution to the above problem. The SDK contains a number of "stub" service classes for API modules such as the datastore, users, urlfetch, and mail. The example recreates the SDK environment by registering these stubs through an API proxy class, which is used by the rest of the SDK code to lookup these services when needed. The datastore stub can be re-instantiated cleanly using the internal API, and the original datastore can be recovered so development entities are not lost. Since I run the tests from within the SDK server, the rest of the service stubs do not need to be replaced. Despite my misgivings about depending on the internal SDK implementation, the ability to reset the datastore state between tests is just too important to ignore.

In order to achieve this, I need to create a special base TestCase that installs the test datastore in setUp() and recovers the development datastore in tearDown(). All test classes that create entities, or that test code that creates entities, will need to inherit from the AppEngineTestCase, located in gaetest\base.py:

import unittest
from google.appengine.api import apiproxy_stub_map  
from google.appengine.api import datastore_file_stub

class AppEngineTestCase(unittest.TestCase):

  def setUp(self):
    # Preserve the current apiproxy for tearDown().
    self.original_apiproxy = apiproxy_stub_map.apiproxy

    # Create a new apiproxy and temporary datastore that will be used for this test.
    apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap() 
    temp_stub = datastore_file_stub.DatastoreFileStub('AppEngineTestCaseDataStore', None, None)  
    apiproxy_stub_map.apiproxy.RegisterStub('datastore_v3', temp_stub)

    # For convenience, the subclass can implement 'set_up' rather than overriding setUp()
    # and calling this base method.
    if hasattr(self, 'set_up'):
      self.set_up()

  def tearDown(self):
    # The subclass can optionally choose to implement 'tear_down'.
    if hasattr(self, 'tear_down'):
      self.tear_down()

    # Restore stubs for development.
    apiproxy_stub_map.apiproxy = self.original_apiproxy

The sample unit test class now extends the new base:

  import unittest
  from google.appengine.ext import db
  import gaetest.base

  from google.appengine.ext import db

  class MyEntity(db.Model):
    name = db.StringProperty()

  class MyEntityTest(gaetest.base.AppEngineTestCase):

    def set_up(self):
      # Populate test entities here along with other setup.
      entity = MyEntity(name='Bar')
      self.setup_key = entity.put()

    def tear_down(self):
      # Tear down here, but there is no need to delete test entities.
      pass

    def test_new_entity(self):
      entity = MyEntity(name='Foo')
      self.assertEqual('Foo', entity.name)

    def test_saved_enitity(self):
      entity = MyEntity(name='Foo')
      key = entity.put()
      self.assertEqual('Foo', db.get(key).name)

    def test_setup_entity(self):
      entity = db.get(self.setup_key)
      self.assertEqual('Bar', entity.name)

When the above test is run, two entities are created (with names 'Foo' from test_saved_enitity() and 'Bar' from test_setup_entity()), but will not be present in the development datastore.

That's all for now. Feel free to download the example, packaged as a GAE application. Enjoy writing tests for Google App Engine!

Note: In the midst of writing this blog, I discovered that a very similar library, called GAEUnit, was recently added to Google Code. Check it out.

UPDATE: My above efforts will soon be incorporated into GAEUnit. The original GAEUnit author, George Lei, and I are now partnering to make it the testing framework of choice for Google App Engine developers.

Often when writing unit tests I use EasyMock to mock dependencies of the class under test. And many times I need to test that a certain type of exception is thrown during a test. Sometimes I need both, for example I am using a mock to simulate a dependency that throws an exception and I want my test to verify the appropriate exception was indeed thrown and that the mock was called properly. In those cases, I use a simple little trick: use a try/finally block along with a JUnit "@Test(expected = FooException.class" annotation on my test method. Voila! Now you can verify mock behavior and verify the exception was thrown at the same time. This is cleaner than using a catch block and asserting the correct exception was thrown, since you rely on JUnit to verify the exception using the "expected" attribute of the @Test annotation. For example:

@Test(expected = ThrottleException.class)
public void testSomethingWithDependencies() {
    // Create mock
    Rocket mainBooster = createMock(Rocket.class);
	
    // Record behavior
    mainBooster.ignite();
    
    // Simluate an invalid call to 'throttleUp'
    expect(rocket.throttleUp(-10000L)).andThrow(ThrottleException.class);
    
    // Replay mock
    replay(rocket);
    
    // Create object with dependency on mainBooster
    SpaceShuttle shuttle = new SpaceShuttle(rocket);    
    
    // Now try to perform a 'blast off' and
    // verify the mock behavior was as expected
    try {
        shuttle.blastOff();
    }
    finally {
        verify(rocket);
    }
}

If instead of writing tests in Java you write them in Groovy, the above code could be a little bit Groovier, though not much. (In an earlier post I showed how you can write unit tests in Groovy and still use JUnit instead of GroovyTestCase if you like.)

@Test(expected = ThrottleException)
void testSomethingWithDependencies() {
    // Create mock
    def mainBooster = createMock(Rocket)
    
    // Record behavior
    mainBooster.ignite()
    
    // Simluate an invalid call to 'throttleUp'
    expect(rocket.throttleUp(-10000L)).andThrow(ThrottleException.class)
    
    // Replay mock
    replay rocket
    
    // Create object with dependency on mainBooster
    def shuttle = new SpaceShuttle(rocket)
    
    // Now try to perform a 'blast off' and
    // verify the mock behavior was as expected
    try {
        shuttle.blastOff()
    }
    finally {
        verify rocket
    }
}

Even more Groovy in this case would be to extend GroovyTestCase and use its shouldFail method, like so:

void testSomethingWithDependencies() {
    // Create mock
    def mainBooster = createMock(Rocket)
	
    // Record behavior
    mainBooster.ignite()
	
    // Simluate an invalid call to 'throttleUp'
    expect(rocket.throttleUp(-10000L)).andThrow(ThrottleException.class)
    
    // Replay mock
    replay rocket
    
    // Create object with dependency on mainBooster
    def shuttle = new SpaceShuttle(rocket)
    
    // Now try to perform a 'blast off' and
    // verify the mock behavior was as expected
    shouldFail(ThrottleException) {
        shuttle.blastOff()
    }
    verify rocket
}

The GroovyTestCase version is probably the cleanest of the above three options. If you're not into Groovy, you can still use the JUnit "@Test(expected = BarException.class)" together with a try/finally/verify in plain Java unit tests for a cleaner test experience.