In order to load Groovy scripts from a database, I showed how to implement custom ScriptFactoryPostProcessor and ScriptSource classes. The CustomScriptFactoryPostProcessor extends the default Spring ScriptFactoryPostProcessor and overrides the convertToScriptSource method to recognize a database-based script, e.g. you could specify a script source of database:com/nearinfinity/demo/GroovyPdfGenerator.groovy. There is also DatabaseScriptSource that implements the ScriptSource interface and which knows how to load Groovy scripts from a database.

In order to put these pieces together, you need to do a bit of configuration. In the articles I used Spring 2.5.x which was current at the time in early 2009. The configuration looked like this:

<bean id="dataSource"
  class="org.springframework.jdbc.datasource.DriverManagerDataSource">
    <!-- set data source props, e.g. driverClassName, url, username, password... -->
</bean>

<bean id="scriptFactoryPostProcessor"
  class="com.nearinfinity.spring.scripting.support.CustomScriptFactoryPostProcessor">
    <property name="dataSource" ref="dataSource"/>
</bean>

<lang:groovy id="pdfGenerator"
  script-source="database:com/nearinfinity/demo/DemoGroovyPdfGenerator.groovy">
    <lang:property name="companyName" value="Database Groovy Bookstore"/>
</lang:groovy>

In Spring 2.5.x this works because the <lang:groovy> tag looks for a Spring bean with id "scriptFactoryPostProcessor" and if one exists it uses it, if not it creates it. In the above configuration we created our own "scriptFactoryPostProcessor" bean for <lang:groovy> tags to utilize. So all's well...until you move to Spring 3.x at which point the above configuration no longer works. This was pointed out to me by João from Brazil who tried the sample code in the articles with Spring 3.x, and it did not work. After trying a bunch of things, we eventually determined that in Spring 3.x the <lang:groovy> tag looks for a ScriptFactoryPostProcessor bean whose id is "org.springframework.scripting.config.scriptFactoryPostProcessor" not just "scriptFactoryPostProcessor." So once you figure this out, it is easy to change the above configuration to:

<bean id="org.springframework.scripting.config.scriptFactoryPostProcessor"
  class="com.nearinfinity.spring.scripting.support.CustomScriptFactoryPostProcessor">
    <property name="dataSource" ref="dataSource"/>
</bean>

<lang:groovy id="pdfGenerator"
  script-source="database:com/nearinfinity/demo/DemoGroovyPdfGenerator.groovy">
    <lang:property name="companyName" value="Database Groovy Bookstore"/>
</lang:groovy>

Then, everything works as expected and the Groovy scripts can reside in your database and be automatically reloaded when you change them. So if you download the article sample code as-is, it will work since the bundled Spring version is 2.5.4, but if you update to Spring 3.x then you'll need to modify the configuration in applicationContext.xml for example #7 (EX #7) as shown above to change the "scriptFactoryPostProcessor" bean to be "org.springframework.scripting.config.scriptFactoryPostProcessor." Note there is a scheduled JIRA issue SPR-5106 that will make the ScriptFactoryPostProcessor mechanism pluggable, so that you won't need to extend the default ScriptFactoryPostProcessor and replace the default bean, etc. But until then, this hack continues to work pretty well.

Selenium.prototype.setupProtoypeJS = function() {
    id = selenium.browserbot.getCurrentWindow().$;
    css = selenium.browserbot.getCurrentWindow().$$;
}

public void waitForPage() {
    selenium.waitForPage('60000')
    proc.doCommand('setupPrototypeJS', [])
}

selenium.click("dom=id('foo')")
selenium.click("dom=css('.bar')")
selenium.click("dom=css('span.foo a.baz')")

selenium = new DefaultSelenium(...)
selenium.setExtensionJs('...')
selenium.start()

selenium.setExtensionJs('''
        id = function(value) {
            return selenium.browserbot.getCurrentWindow().$(value);
        } 

        css = function(value) {
            return selenium.browserbot.getCurrentWindow().$$(value);
        }
''');

selenium.click("dom=id('foo')")
selenium.click("dom=css('.bar')")
selenium.click("dom=css('span.foo a.baz')")

In any case, we only cared about Groovy so Cobertura was a good choice. With the Cobertura Maven plugin we quickly found a problem, which was that even though the code coverage was running, the reports only showed coverage for Java code, not Groovy. This blog shows you how to display coverage on Groovy code when using Maven and the Cobertura plugin. In other words, I'll show how to get Cobertura reports to link to the real Groovy source code in Maven, so you can navigate Cobertura reports as you normally would.

The core problem is pretty simple, though it took me a while to figure out how to fix it. Seems to be pretty standard in Maven: I know what I want to do, but finding out how to do it is the really hard part. The only thing you need to do is tell Maven about the Groovy source code and where it lives. The way I did this is to use the Codehaus build-helper-maven-plugin which has an add-source goal. The add-source goal does just what you would expect; it adds a specified directory (or directories) as a source directory in your Maven build. Here's how you use it in your Maven pom.xml file:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <executions>
        <execution>
            <phase>generate-sources</phase>
            <goals>
                <goal>add-source</goal>
            </goals>
            <configuration>
                <sources>
                    <source>src/main/groovy</source>
                </sources>
            </configuration>
        </execution>
    </executions>
</plugin>

In the above code snippet, we're using the "build-helper-maven-plugin" to add the src/main/groovy directory. That's pretty much it. Run Cobertura as normal, view the reports, and you should now see coverage on Groovy source code as well as Java.

In one of the methods we had some logic similar to the following:

if (listOfThings && listOfThings.size < 0) {
    //Do my logic
}


def list = [1,2,3] as List<Integer>
println list

def list = [1,2,3] as List<Integer>;
println list

{-> assert GroovyClosureTest == owner.getClass() }()
{-> assert GroovyClosureTest == delegate.getClass() }()

{-> assert GroovyClosureTest == owner.getClass() }();
{-> assert GroovyClosureTest == delegate.getClass() }()

groovification. noun. the process of converting java source code into groovy source code (usually done to make development more fun)

On my main day-to-day project, we've been writing unit tests in Groovy for quite a while now, and recently we decided to start implementing new code in Groovy rather than Java. The reason for doing this is to gain more flexibility in development, to make testing easier (i.e. in terms of the ability to mock dependencies in a trivial fashion), to eliminate a lot of Java boilerplate code and thus write less code, and of course to make developing more fun. It's not that I hate Java so much as I feel Java simply isn't innovating anymore and hasn't for a while, and isn't adding features that I simply don't want to live without anymore such as closures and the ability to do metaprogramming when I need to. In addition, it isn't removing features that I don't want, such as checked exceptions. If I know, for a fact, that I can handle an exception, I'll handle it appropriately. Otherwise, when there's nothing I can do anyway, I want to let the damn thing propagate up and just show a generic error message to the user, log the error, and send the admin team an email with the problem details.

This being, for better or worse, a Maven project, we've had some interesting issues with mixed compilation of Java and Groovy code. The GMaven plugin is easy to install and works well but currently has some outstanding issues related to Groovy stub generation, specifically it cannot handle generics or enums properly right now. (Maybe someone will be less lazy than me and help them fix it instead of complaining about it.) Since many of our classes use generics, e.g. in service classes that return domain objects, we currently are not generating stubs. We'll convert existing classes and any other necessary dependencies to Groovy as we make updates to Java classes, and we are implementing new code in Groovy. Especially in the web controller code, this becomes trivial since the controllers generally depend on other Java and/or Groovy code, but no other classes depend on the controllers. So starting in the web tier seems to be a good choice. Groovy combined with implementing controllers using the Spring @MVC annotation-based controller configuration style (i.e. no XML configuration), is making the controllers really thin, lightweight, and easy to read, implement, and test.

I estimate it will take a while to fully convert all the existing Java code to Groovy code. The point here is that we are doing it piecemeal rather than trying to do it all at once. Also, whenever we convert a Java file to a Groovy one, there are a few basics to make the classes Groovier without going totally overboard and spending loads of time. First, once you've used IntelliJ's move refactoring to move the .java file to the Groovy source tree (since we have src/main/java and src/main/groovy) you can then use IntelliJ's handy-dandy "Rename to Groovy" refactoring. In IntelliJ 8.1 you need to use the "Search - Find Action" menu option or keystroke and type "Rename to..." and select "Rename to Groovy" since they goofed in version 8 and that option was left off a menu somehow. Once that's done you can do a few simple things to make the class a bit more groovy. First, get rid of all the semi-colons. Next, replace getter/setter code with direct property access. Third, replace for loops with "each"-style internal iterators when you don't need the loop index and "eachWithIndex" where you do. You can also get rid of some of the redundant modifiers like "public class" since that is the Groovy default. That's not too much at once, doesn't take long, and makes your code Groovier. Over time you can do more groovification if you like.

The most common gotchas I've found have to do with code that uses anonymous or inner classes since Groovy doesn't support those Java language features. In that case you can either make a non-public named class (and it's OK to put it in the same Groovy file unlike Java as long as it's not public) or you can refactor the code some other way (using your creativity and expertise since we are not monkeys, right?). This can sometimes be a pain, especially if you are using a lot of them. So it goes. (And yes, that is a Slaughterhouse Five reference.)

Happy groovification!

class Book {
    String title
    Author author
}

class Author {
    String firstName
    String lastName
}

def author = new Author(firstName:'Jason')
def book = new Book(title:'Say no to NPEs', author:author)

println book.author?.lastName

Did you see that ?. operator on the last line of the code sample? That's the safe navigation operator, and it's the equivalent of writing the following if statement (although it's much more readable).

if (book.author != null) {
    println book.author.lastName
}

The safe navigation operator essentially checks to see if the object you're about to invoke a method on is null. If it is null, it simply skips the method invocation and returns null. If it isn't null, it proceeds as usual. It works for method chaining too, so if you're worried about book being null, you could write

println book?.author?.lastName

This way, if either book or author are null, you simply print out null instead of causing a NullPointerException.

Since learning about the safe navigation operator, I've used it successfully in dozens of classes without any problem. Until last week. Consider the following line of code .

println book?.author?.firstName?.trim().concat(" is great.")

Thanks to the safe navigation operator we can write this concatenation of firstName and a sentence fragment without worrying about NullPointerException(s) and ugly if blocks. Or so I thought.

Looking at this line of code, I thought for sure I was safe from any sneaky NullPointerException. If book, author or firstName are null I'll simply end up printing null and not have to worry about the concat() method. After all, if the trim() method succeeds, there's no sense in guarding it's result for null. And that's where I was wrong.

I naively assumed the method chain would stop once the safe navigation operator encountered a null when in fact it does not. While the trim() method is safe from being called on a null firstName object, it will return null, upon which the concat() method is called. Oops!

void testSearch() {
  def service = createMock(FooService)
  def searcher = new FooSearcher(fooService: service, maxAllowedResults: 10)
  def criteria = new FooSearchCriteria()
  def sortCriteria = ["bar", "baz"] as String[]
  def expectedResult = [new Foo(), new Foo()]
  expect(service.findFoos(criteria, 10, sortCriteria)).andReturn(expectedResult)
  replay service
  def result = searcher.search(criteria, "bar", "baz")
  assertSame expectedResult, result
  verify service
}

The above test fails with the following error message:

java.lang.AssertionError: 
  Unexpected method call findFoos(com.acme.FooSearchCriteria@ea443f, 10, [Ljava.lang.String;@e41d4a):
    findFoos(com.acme.FooSearchCriteria@ea443f, 10, [Ljava.lang.String;@268cc6): expected: 1, actual: 0
	at org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:29)
	at org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:45)
	at $Proxy0.findFoos(Unknown Source)
	at com.acme.FooSearcher.search(FooSearcher.java:19)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    ...

We expected a call to findFoos which takes a FooSearchCriteria, an integer, and a string array describing the sort conditions. But from the error message, EasyMock told us that the expected method was not called and so verification of the mock behavior failed. What happened? Well, basically the string array that was expected was not the string array actually passed as the argument. Look back at the stack trace and specifically the array arguments: the actual argument was [Ljava.lang.String;@e41d4a while the expected argument was [Ljava.lang.String;@268cc6. The FooSearcher.search method's signature is List<Foo> FooSearchCriteria criteria, String... sortBy) - the varargs that are passed to FooSearcher.search are getting packed into a new array when called and that new array is subsequently passed into the FooService which is what causes the difference between the expected and actual array arguments!

To make sure that arguments such as arrays and other complex objects are matched properly by the mock object, EasyMock provides IArgumentMatcher to compare the expected and actual arguments to method calls. Essentially, it is like performing a logical "assertEquals" on the arguments. One of the matchers EasyMock provides is obtained via the static aryEq method in the EasyMock class. So for example if you had a method that took a single array argument, you could make an expectation of mock behavior like this:

def myArray = ["foo", "bar", "baz"] as String[]
expect(someObject.someMethod(EasyMock.aryEq(myArray)).andReturn(anotherObject)

Here you tell EasyMock to expect a call to someMethod on someObject with myArray as the sole argument, and to return anotherObject. Cool, so let's try to fix the failing test above using EasyMock.aryEq (which was imported statically using import static):

void testSearch() {
  def service = createMock(FooService)
  def searcher = new FooSearcher(fooService: service, maxAllowedResults: 10)
  def criteria = new FooSearchCriteria()
  def sortCriteria = ["bar", "baz"] as String[]
  def expectedResult = [new Foo(), new Foo()]
  // Try to use EasyMock's aryEq() to ensure the expected array argument equals the actual argument...
  expect(service.findFoos(criteria, 10, aryEq(sortCriteria))).andReturn(expectedResult)
  replay service
  def result = searcher.search(criteria, "bar", "baz")
  assertSame expectedResult, result
  verify service
}

This test also fails with the following error message:

java.lang.IllegalStateException: 3 matchers expected, 1 recorded.
	at org.easymock.internal.ExpectedInvocation.createMissingMatchers(ExpectedInvocation.java:41)
	at org.easymock.internal.ExpectedInvocation.(ExpectedInvocation.java:33)
	at org.easymock.internal.ExpectedInvocation.(ExpectedInvocation.java:26)
	at org.easymock.internal.RecordState.invoke(RecordState.java:64)
	at org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:24)
	at org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:45)
	at $Proxy0.findFoos(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    ...

Wait, shouldn't that have made EasyMock ensure that the supplied argument was verified using an IArgumentMatcher, specifically an ArrayEquals matcher? Well, sort of. And this is where I always forget what the "N matchers expected, M recorded" error message means and fumble around for a few minutes while I remember. In short, the rule is this:

If you use an argument matcher for one argument, you must use an argument matcher for all the arguments.

So in the above example, we recorded one matcher via the call to aryEq. Now EasyMock will expect an argument matcher for all the arguments in the expectation, and there are three arguments. Now this makes sense. We need to add argument matchers for the other arguments as well. So let's now fix the test:

void testSearch() {
  def service = createMock(FooService)
  def maxResults = 10
  def searcher = new FooSearcher(fooService: service, maxAllowedResults: maxResults)
  def criteria = new FooSearchCriteria()
  def sortCriteria = ["bar", "baz"] as String[]
  def expectedResult = [new Foo(), new Foo()]
  // If you define one matcher for an expected argument, you need to define them for all the arguments!
  expect(service.findFoos(isA(FooSearchCriteria), eq(maxResults), aryEq(sortCriteria))).andReturn(expectedResult)
  replay service
  def result = searcher.search(criteria, "bar", "baz")
  assertSame expectedResult, result
  verify service
}

Now the test passes as we expect it to. We used several other common types of argument matchers here via the static isA and eq argument matchers. The isA matcher ensures the argument is an instance of the specified class, while the eq matcher checks that the actual argument equals the expected argument via the normal Java equality check, i.e. expected.equals(actual). So in summary, if you ever receive the dreaded "N matchers expected, M recorded" error message from EasyMock, you know you need to ensure that all arguments to an expectation use a matcher. And, if you got this far and were dying to mention that if you're using Groovy to test Java code there are easier ways in Groovy to test than using a framework like EasyMock, you're right for the most part. There are still some things you cannot do when testing Java code using Groovy. I plan to go into that more in a future blog post.

groovy:000> r = 0.0..10.0
===> 0.0..10.0
groovy:000> r.contains(5.6)
===> false

WTF? The number 5.6 is not contained in the range 0.0 to 10.0? I beg to differ. So what's actually going on here? Using the shell we can do some more digging, interrogating the range object to see what its bounds are, what values it contains if you iterate it, and so on:

groovy:000> r = 0.0..10.0
===> 0.0..10.0
groovy:000> r.class
===> class groovy.lang.ObjectRange
groovy:000> r.from.class
===> class java.math.BigDecimal
groovy:000> r.to.class
===> class java.math.BigDecimal
groovy:000> r.each { print " $it " }  
 0.0  1.0  2.0  3.0  4.0  5.0  6.0  7.0  8.0  9.0  10.0 ===> 0.0..10.0
groovy:000> r.contains 5 
===> true
groovy:000> r.contains 5.0
===> true
groovy:000> r.contains 5.6
===> false

So what did we learn? First, the range is an ObjectRange whose bounds are BigDecimals. Second, that iterating by default iterates in increments of one. And third that the contains method does not quite do what I would expect by default. Looking at Groovy's ObjectRange class makes it clear exactly what's going on, so let's look:

public boolean contains(Object value) {
  Iterator it = iterator();
  if (value == null) return false;
  while (it.hasNext()) {
    try {
      if (DefaultTypeTransformation.compareEqual(value, it.next())) return true;
    } catch (ClassCastException e) {
      return false;
    }
  }
  return false;
}

The Groovy ObjectRange's contains method defines containment as whether a value is contained within the values generated by its iterator. By now many of my many millions of readers are about to post a comment telling me the problem, so I'll preempt that temptation by adding a few more lines of interaction with the Groovy shell:

groovy:000> r.containsWithinBounds 5.0
===> true
groovy:000> r.containsWithinBounds 5.6
===> true

Aha! So contains doesn't do what you might think it should, but containsWithinBounds does. Its JavaDoc says "Checks whether a value is between the from and to values of a Range." Conspicuously there is no JavaDoc on the contains method to tell me that what it actually does is check whether a value is contained within the discrete values generated by the iterator. Let's try more more thing:

groovy:000> r.containsWithinBounds 5
ERROR java.lang.ClassCastException: java.lang.Integer
        at groovysh_evaluate.run (groovysh_evaluate:2)
        ...

Oops! Not only do you need to call containsWithinBounds rather than contains, you also need to call it with the correct type, as there is no coercion going on since it uses Comparable.compareTo() under the covers.

Notwithstanding all the recent activity regarding all the Ruby security flaws recently discovered, how does Ruby handle inclusion of a number within a range? Here's some irb output:

>> r = 0.0..10.0
=> 0.0..10.0
>> r.class
=> Range
>> r.begin.class
=> Float
>> r.end.class
=> Float
>> r.each { |item| print " #{item} " }
TypeError: can't iterate from Float
        from (irb):53:in `each'
        from (irb):53
>> r.include? 5
=> true
>> r.include? 5.0
=> true
>> r.include? 5.6
=> true

To me this is more semantically and intuitively correct behavior. First, while I can create a range with float bounds, I cannot iterate that range - for non-integral numbers, how exactly can you define the next item after 0.0 for example? 0.1, 0.01, 0.001, and so on till infinity. Second, the include? method behaves as I would expect no matter what type of argument I pass. I am able to iterate ranges of integral numbers, however, which could arguably also be confusing since the behavior of the method depends on the type. Then again, that's pretty much what polymorphic method behavior is about I suppose.

>> r = 0..10
=> 0..10
>> r.each { |item| print " #{item} " }
 0  1  2  3  4  5  6  7  8  9  10 => 0..10

In the case of integers Ruby uses an increment of one by default. You could use the step method to get a different increment, e.g.:

>> r.step(2) { |item| print " #{item} " }
 0  2  4  6  8  10 => 0..10

So what's the point of all this? That Ruby is better than Groovy? Nope. I really like both languages. I think there are a couple of points that were reinforced to me:

First, RTFM (or source code --> RTFC). Even though Groovy's contains method doesn't behave how I think it should, there is the method I was looking for with containsWithinBounds.

Second, having a shell to play around with short snippets of code is really, really useful, without needing to create a class with a main method just to play around with code.

Third, documentation and semantics matter. If something doesn't feel intuitively correct based on how similar things act, it is more likely to cause confusion and errors. In this case, since my unit test immediately caught the error, I was able to figure the problem out in a few minutes.

Finally, following on from the last point, unit tests continue to remain valuable. Of course anyone who knows me would roll their eyes over my anal-ness (which Mac's dictionary is telling me is not really a word but I don't care at the moment) expecting me to get something about unit testing in somehow.

@Test(expected = ThrottleException.class)
public void testSomethingWithDependencies() {
    // Create mock
    Rocket mainBooster = createMock(Rocket.class);
	
    // Record behavior
    mainBooster.ignite();
    
    // Simluate an invalid call to 'throttleUp'
    expect(rocket.throttleUp(-10000L)).andThrow(ThrottleException.class);
    
    // Replay mock
    replay(rocket);
    
    // Create object with dependency on mainBooster
    SpaceShuttle shuttle = new SpaceShuttle(rocket);    
    
    // Now try to perform a 'blast off' and
    // verify the mock behavior was as expected
    try {
        shuttle.blastOff();
    }
    finally {
        verify(rocket);
    }
}

If instead of writing tests in Java you write them in Groovy, the above code could be a little bit Groovier, though not much. (In an earlier post I showed how you can write unit tests in Groovy and still use JUnit instead of GroovyTestCase if you like.)

@Test(expected = ThrottleException)
void testSomethingWithDependencies() {
    // Create mock
    def mainBooster = createMock(Rocket)
    
    // Record behavior
    mainBooster.ignite()
    
    // Simluate an invalid call to 'throttleUp'
    expect(rocket.throttleUp(-10000L)).andThrow(ThrottleException.class)
    
    // Replay mock
    replay rocket
    
    // Create object with dependency on mainBooster
    def shuttle = new SpaceShuttle(rocket)
    
    // Now try to perform a 'blast off' and
    // verify the mock behavior was as expected
    try {
        shuttle.blastOff()
    }
    finally {
        verify rocket
    }
}

Even more Groovy in this case would be to extend GroovyTestCase and use its shouldFail method, like so:

void testSomethingWithDependencies() {
    // Create mock
    def mainBooster = createMock(Rocket)
	
    // Record behavior
    mainBooster.ignite()
	
    // Simluate an invalid call to 'throttleUp'
    expect(rocket.throttleUp(-10000L)).andThrow(ThrottleException.class)
    
    // Replay mock
    replay rocket
    
    // Create object with dependency on mainBooster
    def shuttle = new SpaceShuttle(rocket)
    
    // Now try to perform a 'blast off' and
    // verify the mock behavior was as expected
    shouldFail(ThrottleException) {
        shuttle.blastOff()
    }
    verify rocket
}

The GroovyTestCase version is probably the cleanest of the above three options. If you're not into Groovy, you can still use the JUnit "@Test(expected = BarException.class)" together with a try/finally/verify in plain Java unit tests for a cleaner test experience.

<project>

  <modelversion>4.0.0</modelversion>
  <groupid>com.trial.easyb</groupid>
  <artifactid>easyb-trial</artifactid>
  <version>0.1</version>

  <dependencies>
    <dependency>
        <groupid>org.easyb</groupid>
        <artifactid>easyb</artifactid>
        <version>0.8</version>
    </dependency>
    <dependency>
        <groupid>junit</groupid>
        <artifactid>junit</artifactid>
        <version>4.4</version>
    </dependency>
  </dependencies>

  <repositories>
    <repository>
      <id>easyb</id>
      <url>http://www.easyb.org/repo/</url>
    </repository>
  </repositories>
  <pluginrepositories>
    <pluginrepository>
      <id>easyb</id>
      <url>http://www.easyb.org/repo/</url>
    </pluginrepository>
  </pluginrepositories>

  <build>
    <plugins>
      <plugin>
        <groupid>org.easyb</groupid>
        <artifactid>maven-easyb-plugin</artifactid>
        <version>0.8</version>
        <executions>
          <execution>
            <goals>
              <goal>test</goal>
              <goal>storyReport</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <easybtestdirectory>${basedir}/src/test/stories</easybtestdirectory>
          <storyreport>${project.build.directory}/easyb-stories.txt</storyreport>
          <xmlreport>${project.build.directory}/easyb/report.xml</xmlreport>
        </configuration>
      </plugin>
    </plugins>
  </build>  

</project>

My blog post about the first problem in Project Euler went strange places. I was just trying to learn Python by solving a barely non-trivial problem. Maybe it was a full moon, not sure, but it wound up in a tangle of gcc, groovyc, and .elc. Happily, some good performance improvements were suggested and I've tried to incorporate them below.  

This is a really long post. Here's the skeleton:

1. Runtime environment
2. Results
3. Code (alphabetically by language)
   1. C
   2. Groovy
   3. Lisp
   4. Perl
   5. Python
   6. Ruby
4. Big O(rly)?
5. Wrap up
6. Extras

Runtime environment

• Machine: Powerbook Pro, 10.5.2, 2x 2.16 GHz Core 2 Duo
• Userspace programs: Finder, Terminal
• Battery conservation mode: Better Performance (plugged in)
• C: GCC 4.0.1 (Apple Inc. build 5465)
• Groovy: 1.5.5 JVM: 1.5.0_13-119
• Lisp: GNU Emacs 22.1.1
• Perl: v5.8.8, built for darwin-thread-multi-2level
• Python: 2.5.1
• Ruby: 1.8.6 (2007-09-24 patchlevel 111) [universal-darwin9.0]

Results

No cheesy graphs this time, just better organized numbers. Each row has one implementation of the problem. The 10**x columns show how many milliseconds it took the implementation to sum the multiples of three and five less than that number. Each implementation was run seven times for each column, and the average runtime is shown below. A test harness was written to execute the scripts automatically; I disabled the screensaver, started the tests, and went to bed.

Code

C

#include <stdio.h>
#include <math.h>

#define multiple_of(a, b) ((a == 0 || b == 0) ? 0 : a % b == 0)

int main(int argc, char** argv)
{
    int exp = atoi(argv[1]);
    double sum = 0;
    int i;

    for (i = 3; i < pow(10, exp); i++)
        if (multiple_of(i, 3) || multiple_of(i, 5))
            sum += i;

    printf("%0.0f\n", sum);

    return 0;
}

Groovy

// first.groovy
exp = Integer.parseInt(args[0])

mof = { val, test -> val && test && test % val == 0 }
mof3 = mof.curry(3)
mof5 = mof.curry(5)

sum = 0 as BigDecimal

(3..<(10**exp)).each {
    if (mof3(it) || mof5(it)) {
        sum += it
    }
}

println sum

// second.groovy
def exp = Integer.parseInt(args[0])

def mof = { val, test -> val && test && test % val == 0 }
def mof3 = mof.curry(3)
def mof5 = mof.curry(5)

def sum = 0 as BigInteger

(3..<(10**exp)).each {
    if (mof3(it) || mof5(it)) {
        sum += it
    }
}

println sum

// third.groovy
def exp = Integer.parseInt(args[0])

def sum = 0 as BigInteger

for (int i = 3; i < 10 ** exp; i++)
    if (i % 3 == 0 || i % 5 == 0)
        sum += i

println sum

Lisp

; first.el
(defun multiple_of (a b)
  (= 0 (% a b))
)

(defun val-to-add (val)
  (cond
   ((multiple_of val 3) val)
   ((multiple_of val 5) val)
   (0)
   )
  )

(defun sum-of-3or5-multiples (exp)
  (let ((num 3) (sum 0.0))
    (while (< num (expt 10 exp))
      (setq sum (+ sum (val-to-add num)))
      (setq num (1+ num))
      )
    sum
    )
  )

(sum-of-3or5-multiples 7)

Perl

#!/usr/bin/env perl -w
# first.pl
use strict;

my $exp = $ARGV[0];
my $sum = 0;

sub multiple_of {
    my ($val, $tgt) = @_;
    return $val % $tgt == 0;
}

for my $i (3 ... 10 ** $exp - 1) {
    if (multiple_of($i, 3) || multiple_of($i, 5)) {
        $sum += $i
    }
}

print "$sum\n"

#!/usr/bin/env perl -w
# second.pl

use strict;

my $exp = $ARGV[0];
my $sum = 0;

for my $i (3 ... 10 ** $exp - 1) {
    if ($i % 3 == 0 || $i % 5 == 0) {
        $sum += $i
    }
}

print "$sum\n"

Python

# first.py
import sys

def multiple_checker(val):
    return lambda test: test and val and test % val == 0

mof3 = multiple_checker(3)
mof5 = multiple_checker(5)

sum = 0

for i in range(10 ** int(sys.argv[1])):
    if mof3(i) or mof5(i):
        sum = sum + i

print sum

# second.py
import sys

exp = int(sys.argv[1])
sum = 0

for mof3or5 in set(range(3, 10**exp, 3)) | set(range(5, 10**exp, 5)):
    sum = sum + mof3or5

print sum

# third.py
import sys

exp = int(sys.argv[1])
sum = 0

for mof3or5 in set(xrange(3, 10**exp, 3)) | set(xrange(5, 10**exp, 5)):
    sum = sum + mof3or5

print sum

# fourth.py
import sys

exp = int(sys.argv[1])
total = sum(xrange(3, 10**exp, 3)) + sum(xrange(5, 10**exp, 5)) - sum(xrange(15, 10**exp, 15))

print total

Ruby

# first.rb
exp = ARGV[0].to_i
sum = 0

def is_mult(large, small)
  large % small == 0
end

for i in 3..(10 ** exp - 1)
  next unless is_mult(i, 3) or is_mult(i, 5)
  sum = sum + i
end

puts sum

# second.rb
exp = ARGV[0].to_i
sum = 0

for i in 3..(10 ** exp - 1)
  next unless i % 3 == 0 or i % 5 == 0
  sum = sum + i
end

puts sum

Big O(rly)?

Well... sigh... all of the code above works correctly but is slow and naïve. All of the approaches above take more time as the largest multiple increases. I was ignorantly, happily tweaking code down this road until fortunately Bob clued me in. The best approach takes the same amount of time as the largest multiple increases. Why? There's a simple equation for this, calculating an arithmetic series.  

# instant.rb
ceil = 10 ** ARGV[0].to_i - 1

def series_of_multiples(val, ceiling)
  diff = val
  vals = ceiling / val
  last_val = val + diff * (vals - 1)
  (val + last_val) / 2.0 * vals
end

def som(v, c) series_of_multiples(v, c).to_i end

puts som(3, ceil) + som(5, ceil) - som(15, ceil)

Wrap up

• I got a lot more out of this than a quick glance at Python. It was good to calculate a series or two, but much better to build a little test harness.
• Groovy performance tested out well here. I would avoid using some of its features in tight loops. I'm confident that Monsieur LaForge et son amis will optimize that concern away eventually.
• I thought I could love it, I really wanted to love it, it's loved by many for many good reasons, but Python started to chafe me. It's so hard to nail down exactly why... it's chock full of goodness, but just seems to need more Kool-Aid than Groovy and especially Ruby. Perl's Kool-Aid threshold is a well-known constant, at least until the Parrot changes things drastically.
• Cosine similarity + arithmetic series = it's good to work with a friendly, helpful PhD. in math. Thanks Bob :-)

Extras

The Python test harness. Probably needs tweaking for Wintel boxes.

import sys
import re
from datetime import timedelta, datetime
from os import system

#
# prefix used to run scripts. Trailing whitespace matters!
#
vms = { 
        'el':           'emacs --script ',
        'elc':          'emacs --script ',
        'groovy':       'groovy ',
        'out':          './',
        'pl':           'perl ',
        'py':           'python ',
        'rb':           'ruby '
        }

# Configurable parameters
# 
DEFAULT_MIN = 1
DEFAULT_MAX = 7
min = DEFAULT_MIN       # min "max" val = 10 ** min
max = DEFAULT_MAX       # max "max" val = 10 ** max
reps = 7                # times to repeat each test, then average the results


# Helper method
def td_to_ms(delta):
    val = delta.microseconds / 1000.0
    val = val + delta.seconds * 1000
    val = val + delta.days * 24 * 60 * 60 * 1000
    return val

#
# "main"
#
for script in sys.argv[1:]:
    ext = script.split('.').pop()
    if not ext in vms:
        print 'ignoring', script, 'unrecognized extension'
        continue

    vm = vms[ext]

    # the lisp files hardcode the max values :(
    if re.search('^el', ext):
        max = min
    else: # reset to defaults, needed when running multiple types of tests
        max = DEFAULT_MAX
        min = DEFAULT_MIN

    # gotta like list comprehensions
    runs = [0 for x in range(max - min + 1)]

    for rep in xrange(reps):
        for exp in xrange(min, max + 1):
            start = datetime.now()
            system('%s%s %s >/dev/null' % (vm, script, exp))
            end = datetime.now()
            runs[exp - min] = runs[exp - min] + td_to_ms(end - start)

    for i in xrange(len(runs)):
        runs[i] = int(runs[i] / reps)

    print script.split('/').pop(), runs

    # see, isn't it strange how python leaves you hanging at the end
    # of the last scope in the file?

groovy:000> test = []
===> []
groovy:000> (1..10).each { test << "Item $it" }
===> 1..10
groovy:000> test                               
===> [Item 1, Item 2, Item 3, Item 4, Item 5, Item 6, Item 7, Item 8, Item 9, Item 10]
groovy:000> test.contains("Item 1")
===> false

Um...what? We started by adding items to a list, e.g. "Item 1", "Item 2", and so on. Then we simply ask whether the list contains an item we know (or think) is there. But no, it prints false! How can test possibly not contain "Item 1?" First, what's going on here? Look at this second example:

groovy:000> test = []
===> []
groovy:000> (1..10).each { test << "Item $it" }
===> 1..10
groovy:000> test
===> [Item 1, Item 2, Item 3, Item 4, Item 5, Item 6, Item 7, Item 8, Item 9, Item 10]
groovy:000> println test[0].class 
class org.codehaus.groovy.runtime.GStringImpl
===> null

Ah ha! The items in the list aren't strings. So what? Look at this code:

groovy:000> a = "Item 2"
===> Item 2
groovy:000> a.class
===> class java.lang.String
groovy:000> n = 2
===> 2
groovy:000> b = "Item $n"
===> Item 2
groovy:000> b.class
===> class org.codehaus.groovy.runtime.GStringImpl
groovy:000> a == b
===> true
groovy:000> b == a
===> true
groovy:000> a.equals(b)
===> false
groovy:000> b.equals(a)
===> false

a is a normal Java String, whereas b is a Groovy GString. Even more interesting (or perhaps confusing), a == comparison yields true, but a comparison using equals() is false! So back in the original example where we checked if the 'test' list contained 'Item 1', the contains() method (which comes from Java not Groovy) uses equals() to compare the values and responds that in fact, no, the list does not contain the String "Item 1." Without getting too much more detailed, how to fix this? Look at this example:

groovy:000> test = []
===> []
groovy:000> (1..10).each { test << "Item $it".toString() }
===> 1..10
groovy:000> test
===> [Item 1, Item 2, Item 3, Item 4, Item 5, Item 6, Item 7, Item 8, Item 9, Item 10]
groovy:000> test.contains('Item 1')
===> true

Now it returns true, since in the above example we explicitly forced evaluation of the GStrings in the list by calling toString(). The Groovy web site explains about GString lazy evaluation stating that "GString uses lazy evaluation so its not until the toString() method is invoked that the GString is evaluated."

Ok, so we now understand why the observed behavior happens. The second question is whether or not this is something likely to confuse the You-Know-What out of developers and whether it should behave differently. To me, this could easily lead to some subtle and difficult to find bugs and I think it should work like a developer expects without requiring an understanding of whether GStrings are lazily-evaluated or not. Lazy evaluation always seems to come at a price, for example in ORM frameworks it can easily result in the "n + 1 selects" problem.

For comparison, let's take a look at similar Ruby example in an irb session:

>> test = []
=> []
>> (1..10).each { |n| test << "Item #{n}" }
=> 1..10
>> test
=> ["Item 1", "Item 2", "Item 3", "Item 4", "Item 5", "Item 6", "Item 7", "Item 8", "Item 9", "Item 10"]
>> test.include? "Item 1"
=> true

Ruby "does the right thing" from just looking at the code. Ruby 1, Groovy 0 on this one. I don't actually know whether Ruby strings evaluate the expressions immediately or not, and should not need to care. Code that looks simple should not require understanding the implementation strategy of a particular feature. Otherwise it is broken in my opinion.