Easy Grails hosting

I started a new Grails-based project, and went out looking for an easy to set up Grails hosting option. I currently have a virtual server hosted with Rackspace (originally Slicehost). For a couple reasons, I want to keep this new application on its own server. I could get an additional virtual server from Rackspace, and start setting up SSH, Apache Web Server, Tomcat, MySQL, MongoDB, and whatever else came up, which I’ve done many times, but the setup and administration takes a significant amount of time, which could instead be spent on developing the app.

My requirements are:

  • Free or cheap to get started
  • No vendor lock-in. If I want to move to another provider, no code changes necessary.
  • Ability to scale up the number of instances and amount of memory easily
  • Minimal effort to set up a Grails application
  • Support for MySQL or PostgreSQL, and MongoDB
  • HTTPS support

Ideally, I’d also like to be able to:

  • Download my server image and run it locally via VirtualBox or similar
  • Run my server image on a different cloud provider

As a reference point on pricing, Rackspace currently offers the following Cloud Server configurations:

  • Instance with 512MB memory, 20GB storage: $16/mo
  • Instance with 1GB memory, 40GB storage: $44/mo

Interestingly, Rackspace is transitioning to an Openstack-based platform.

Looking at the PaaS (platform-as-a-service) providers, I’ve been hearing about Heroku for a while. Heroku looks very easy to get started with, and the add-on services that are available look impressive.

Kudos to the Heroku website, very slick and inviting. On to the substance..

As far as pricing,

  • 1 web dyno, up all the time with a dev database (10K rows) is free
  • 1 web dyno, up all the time with a “crane” production database (400 MB cache, 1TB storage) is $50/mo
  • 2 web dyno, up all the time with a “crane” production database (400 MB cache, 1TB storage) is $85/mo

Basically, each additional web dyno is approximately $35/month, and each doubling of production database cache doubles the price, starting with $50 for the “crane” database.

But what exactly is a web dyno or a “crane” database?

According to the Heroku website, “A web dyno runs your code and responds to HTTP requests. More dynos provide more concurrency” Sounds like a thread, which would be really bad in relation to the pricing.

But drilling into the details, A dyno is defined as “the basic unit of composition on Heroku, is a lightweight container running a single user-specified command. You can think of it as a virtualized unix container”. Which sounds more like a virtual server instance, and is more reasonable.

The standard Heroku database offerings are based on PostgreSQL, with different pricing based on the amount of cache you need, “crane” being the lowest.

What about HTTPS/SSL support? This add-on provides SSL support at $20/month. On further digging, SSL does work out of the box using a shared certificate and a heroku.com-based domain name. The $20/mo add-on lets you use your own certificate for your own domain.

So Heroku looks to be a bit pricier for a small production instance than setting up your own virtual server, but with less setup and administration work, and an easier path to scale up.

But the show-stopper for me was the deployment model. In Heroku, deployment means pushing your source code to Heroku, which they then build and deploy themselves. I’m aware of the argument that if someone has access to your deployable artifacts, and therefore the bytescode, they can decompile source code. But decompiled source code isn’t the same as the original, and it feels wrong to make it so darn easy for someone to steal your intellectual property if there’s a security breach.

In my next post, I’ll explore CloudForge.

4/14/2013 Update:
Heroku added support for WAR deployment via their Heroku Enterprise for Java offering. Looks like pricing information isn’t publicly available though.

Posted in Grails | Tagged , | 4 Comments

Making Eclipse run configurations portable

I’m using Eclipse, GWT, and Maven on a couple projects, and have several customized Eclipse run configurations I use. There’s a bit of work and specific settings required to make these work well, and they have to be set up for each new workspace, so I’d like to set them up once and share them with the team.

Before being able to check these into source control, and make them available to the rest of the team, there’s a couple things in these run configurations that have platform-specific references (e.g. the full path to files on my hard drive) that have to be fixed. Fortunately, if you know where to look it’s not hard to change.

First, the GWT launch configuration which is generated by the Google Eclipse Plugin needs to be fixed. As described in this stackoverflow question, the classpath used by the launcher contains a hard-coded full path to the GWT development jar on your computer. The solution is to change the path to be relative to a classpath variable which is defined at the workspace level. In my case, I need to use the M2_REPO classpath variable which points to my local Maven repository.

To change this, open the Eclipse Run Configuration for the GWT launcher in question, select the Classpath tab, click the “Advanced” button on the right to add a new classpath entry, choose “Classpath Variable” in the dialog that comes up, then choose the classpath variable you want to use. Assuming the jar file you need to select is in a subdirectory below the classpath variable, click on “Extend” to navigate to the jar file relative to the classpath variable.

Once your new classpath entry is in place, you can remove the old one with the hard-coded path.

For the Maven run configurations, the only thing that was platform-specific was the base directory setting on the first tab, which by default is the full directory path to the project.

To make this portable, you can make this location relative to the named project that’s loaded in the current workspace. Click “Browse Workspace…” and select the project that the Maven build should run from. You’ll end up with a location like ${workspace_loc:/my-project}.

For both of these run configurations, the last thing that needs to be done is to save them as a file within the project. Select the last tab of the run configuration, “Common”, select “Shared File” under “Save as”, and choose a directory for the run configuration to be saved to.

. You can now check these files into source control and load them into another workspace.

Posted in Tools | Tagged | 1 Comment

Sculptor lately

The latest version, 2.1.0, of Sculptor came out and it has some great features including JPA 2.0 support, custom finder methods generated from the DSL, and a fluent domain object builders feature I contributed.

Other than the builders feature, I haven’t been able to work with the new features as I’m working on a Sculptor GWT GUI branch based on the 2.0 release. At some point I’ll migrate my branch to the latest and get to use these goodies.

On the horizon, there’s work being done on migrating Sculptor to the latest versions of XText and XTend which look like great advancements, and Hibernate 4.X support has just been added to trunk by Oliver Ringel.

Not tied to a particular release, but Torsten Juergeleit has also done some great work with a new all-in-one Maven Sculptor plugin which does the work of what used to be 3 separate plugins.

Lots of exciting developments for Sculptor. I’ll write more on the Sculptor GWT GUI work in a future post.

Posted in MDSD | Tagged | Leave a comment

Java multiple dispatch / multimethod libraries

Multiple dispatch (aka multimethods) is a method dispatch feature of some languages whereby the method that is dispatched to is based on the runtime type of the method arguments, not just the runtime type of the class being called (single dispatch polymorphism), and compile-time type of the method arguments.

Java doesn’t support multiple dispatch at a language level, but fortunately, it can be implemented in a library.

A simple test program to show Java behavior:

public class Mapper {

	static abstract class Content {}

	static class Link extends Content {}
	
	static class Message extends Content {}
	
	void map(Content c) {
		System.out.println("shouldn't get here!");
	}
	
	void map(Link l) {
		System.out.println("It's a Link");
	}
	
	void map(Message m) {
		System.out.println("It's a message");
	}
	
	public static void main(String[] args) {
		Mapper mapper = new Mapper();
		Content c1 = new Link();
		Content c2 = new Message();
		mapper.map(c1);
		mapper.map(c2);
	}
}

The output for the above test program is:

shouldn't get here!
shouldn't get here!

Java is dispatching to the overloaded map method based on the compile-time type of c1 and c2, which is the Content class. The above example is a bit contrived, but it shows that when working with base class references, for instance if you are working with a List of Content objects, Java won’t dispatch based on the runtime type.

I set out looking for a library that would implement multiple dispatch. I was a bit surprised the Google Guava library didn’t implement it. This seems to be right up their alley. What I did find was the following implementation: jmultimethod

Nice and simple, multiple dispatch using annotations and reflection.

Using jmultimethod, our test program becomes:

package org.sculptor.mapprovisions.domain;

import jmultimethod.Multi;
import jmultimethod.Multimethod;

public class Mapper {

	// Object responsible for doing the dispatching
    protected static Multimethod mapMM =
        new Multimethod("map", Mapper.class);

	static abstract class Content {}

	static class Link extends Content {}
	
	static class Message extends Content {}
	
    @Multi("map")
	public void map(Content c) {
		System.out.println("shouldn't get here!");
	}
	
    @Multi("map")
	public void map(Link l) {
		System.out.println("It's a Link");
	}
	
    @Multi("map")
	public void map(Message m) {
		System.out.println("It's a Message");
	}
	
	public static void main(String[] args) {
		Mapper mapper = new Mapper();
		Content c1 = new Link();
		Content c2 = new Message();
		
		// Invoke the multiple-dispatch method "map" on mapper, passing c1 as an argument
		mapMM.invoke(mapper, c1);
		
		// Invoke the multiple-dispatch method "map" on mapper, passing c2 as an argument
		mapMM.invoke(mapper, c2);
	}
}

The new output is:

It's a Link
It's a Message

I’d still prefer to have it directly supported by Java, but it’s good to have this solution when needed.

Posted in Java | Leave a comment

Web and mobile UI wireframes

People think visually. A picture is worth a thousand words as the saying goes.

We’ve recently been using Balsamiq to put together wireframes we use to work out user experience concepts, and to work with stakeholders. Balsamiq is an Adobe AIR-based desktop application that can be used to quickly put together web and mobile UI wireframes. It’s also relatively easy to make clickable demos in Balsamiq by stringing together screen wireframes.

What I probably like most about Balsamiq, is the wireframes have a rough and unfinished look to them. They get the point across, but purposefully look informal so people focus on the general concepts and information presented on the UI, and not the color or exact positioning of buttons.

If you’re an OmniGraffle user, there’s some stencils that are also a good option, among them Konigi, and Yahoo Design Stencils (also available for Visio and in other formats).

Posted in General | Leave a comment

Customizing Sculptor to generate Groovy classes

I’ve been using the Sculptor DSL/tool to build Java EE applications based on DDD and EDA principles. This has worked out great, and Sculptor allows me to express the intent at a higher level and more succinctly than I could with straight Java or another GPL.

When it comes to expressing the behavior of your domain, that’s done outside of the Sculptor DSL, using Java. The Sculptor DSL is largely about the solution structure; the structure and relationship between classes, and what standard behaviors they support, such as CRUD repository methods. If you need to implement complex logic for your credit score calculator class, Sculptor is used to define the interface into your method, but it’s up to you to implement the logic for that method in Java. After a while, I found this was significantly slowing me down. Java is so verbose compared to languages such as Groovy or Scala, in particular when it comes to working with object graphs and functional programming.

Fortunately, Sculptor and the XText tools it is based on support customizing each step of the generation process. This post outlines what I did to customize Sculptor to allow me to implement my domain and service classes in Groovy. This only applies to the concrete implementation classes you would customize when using the ‘gap’ generation option. The base classes that are regenerated each time are still Java, as there was no benefit to using a different language for them.

The following changes should be made to the business tier project.

Maven POM updates

First, the Groovy libraries and a Groovy compiler need to be added to the Maven build Traditionally this was done using the GMaven plugin, but I came across this post that describes using the eclipse-groovy plugin to jointly compile Java and Groovy together in a single pass. Much better.

To use this, modify the pom.xml to add the springsource maven dependency and plugin repositories:

	<repositories>
		<repository>
			<id>springsource</id>
			<url>http://maven.springframework.org/snapshot</url>
			<releases>
				<enabled>true</enabled>
			</releases>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</repository>
	</repositories>

	<pluginRepositories>
		<pluginRepository>
			<id>springsource</id>
			<url>http://maven.springframework.org/snapshot</url>
		</pluginRepository>
	</pluginRepositories>

Then update the maven-compiler plugin in the build/plugins section to use the groovy-ecilpse-compiler:

			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<version>2.3.1</version>
				<configuration>
					<encoding>ISO-8859-1</encoding>
					<source>1.6</source>
					<target>1.6</target>
					<compilerId>groovy-eclipse-compiler</compilerId>
					<verbose>true</verbose>
				</configuration>
				<dependencies>
					<dependency>
						<groupId>org.codehaus.groovy</groupId>
						<artifactId>groovy-eclipse-compiler</artifactId>
						<version>0.0.1-SNAPSHOT</version>
					</dependency>
				</dependencies>
			</plugin>

Finally, add groovy to your dependencies:

		<dependency>
			<groupId>org.codehaus.groovy</groupId>
			<artifactId>groovy-all</artifactId>
			<version>1.7.5</version>
			<type>jar</type>
		</dependency>

Xpand templates

The next step is to create Xpand templates that will generate Groovy class files instead of Java. I’m using the Sculptor ‘hint’ feature to control whether a class will get generated as Java or Groovy. If a Service or Domain class is annotated with hint=”gapImpl=groovy”, Groovy will be generated, otherwise it’ll fall back to the default of Java.

The code below defines an aspect for the generation of the DomainObject and Service sub-classes. Whenever these templates are invoked within Sculptor, the aspect below will be evaluated, deciding whether to generate Groovy code or to proceed with the normal Sculptor templates.

Add to src/main/resources/templates/SpecialCases.xpt:

«AROUND *::domainObjectSubclass FOR DomainObject»
        «IF getHint("gapImpl") == "groovy"»
                «EXPAND Groovy::Groovy»         
        «ELSE»
            «targetDef.proceed()»
        «ENDIF»
«ENDAROUND»

«AROUND *::serviceImplSubclass FOR Service»
        «IF getHint("gapImpl") == "groovy"»
                «EXPAND Groovy::GroovyService»      
        «ELSE»
            «targetDef.proceed()»
        «ENDIF»
«ENDAROUND»

Next, define the templates that will generate the actual Groovy code. Since Groovy syntax is practically a superset of Java, I was able to copy the code from the existing Sculptor templates (DomainObject.xpt and Service.xpt). The one key difference is where the code gets generated to – a new helper function groovyFileName returns a .groovy file name.

Create src/main/resources/templates/Groovy.xpt:

«REM»
Templates to generate Sculptor implementation classes as Groovy
«ENDREM»

«IMPORT sculptormetamodel»
«EXTENSION extensions::helper»
«EXTENSION extensions::groovyhelper»
«EXTENSION extensions::properties»

«DEFINE Groovy FOR DomainObject»
«FILE groovyFileName(getDomainPackage() + "." + name) TO_SRC»
«javaHeader()»
package «getDomainPackage()»;

«IF formatJavaDoc() == "" -»
/**
 * Data transfer object for «name». Properties and associations are
 * implemented in the generated base class {@link «getDomainPackage()».«name»Base}.
 */
«ELSE -»
«formatJavaDoc()»
«ENDIF -»
«EXPAND templates::DomainObject::domainObjectSubclassAnnotations»
public «getAbstractLitteral()»class «name» extends «name»Base {
        «EXPAND templates::DomainObject::serialVersionUID»
    «IF getLimitedConstructorParameters().isEmpty || getMinimumConstructorParameters().isEmpty»public«ELSE»protected«ENDIF» «name»() {
    }

    «EXPAND templates::DomainObject::propertyConstructorSubclass»
    «EXPAND templates::DomainObject::limitedConstructor»
    «EXPAND templates::DomainObject::minimumConstructor»

}
«ENDFILE»
«ENDDEFINE»


«DEFINE GroovyService FOR Service»
	«FILE groovyFileName(getServiceimplPackage() + "." + name + "Impl") TO_SRC»
«javaHeader()»
package «getServiceimplPackage()»;

/**
 * Implementation of «name».
 */
«IF isSpringAnnotationToBeGenerated()»
@org.springframework.stereotype.Service("«name.toFirstLower()»")
«ENDIF»
«IF webService»
«EXPAND webServiceAnnotations»
«ENDIF»
public class «name»Impl extends «name»ImplBase {

    public «name»Impl() {
    }
	«EXPAND templates::Service::otherDependencies»

    «EXPAND templates::Service::implMethod FOREACH operations.select(op | op.delegate == null && op.serviceDelegate == null)
    	.reject(e|e.hasHint("EventSourcingScaffold")) »
}
	«ENDFILE»
«ENDDEFINE»

Xtend helper function

The last step is to define the helper function used above to return the groovy file name.

Create src/main/resources/extensions/groovyhelper.ext:

import sculptormetamodel;

String groovyFileName(String name) :
    name.replaceAll("\\.", "/") + ".groovy";

Model file

Finally, to use this, add the hint to whatever Service or Domain classes you need it for in your model file:

		Service MyService {
			gap
			hint="gapImpl=groovy"
			...
		}
Posted in MDSD | Tagged , | Leave a comment

Fornax Sculptor DSL tool: Switching an existing app to MongoDB

Fornax Sculptor is a DSL, Eclipse-based editor, and set of code generators used to build Java/JEE applications.  You define your model and other parts of your application in the Sculptor textual DSL, and the Sculptor model transformations and code generators generate Java code, Spring configuration, Hibernate mappings, and UI code.  Of course, Sculptor only takes you so far, which is where you supply your own code via extension points and composition/delegation, but Sculptor goes a long way to minimize repetitive code, and keeping you working closer to the actual problem domain.  You can also customize and extend Sculptor via the excellent Eclipse MDSD tools (Xtext, MWE, Xpand, Xtend, Check) that Sculptor is based on.

I’ve been building a JEE application using Sculptor, based on JSF2, Hibernate, MySQL, and Spring.  The latest release of Sculptor added support for MongoDB and Events.  I’ve been looking to eventually migrate this application to a distributed & schema-less database, and more of an Event-Driven architecture, so I decided to try converting it.  All in all it went very smoothly.  Following are my notes on switching over a Sculptor application from Hibernate to MongoDB.

Updating the Maven POM and Sculptor configuration

First, remove mysql, hibernate, spring-jdbc and related dependencies from the business tier project pom, then add the following dependency:

<dependency>
    <groupId>org.fornax.cartridges</groupId>
    <artifactId>fornax-cartridges-sculptor-mongodb</artifactId>
    <version>${sculptor.version}</version>
</dependency>

Then modify sculptor-generator.properties to switch the persistence tier from Hibernate to MongoDB.  Comment out:

#db.product=mysql

and add:

nosql.provider=mongoDb

Run a build:

mvn clean install

Fix domain concrete classes

If you have domain classes with gap classes, as I had, you’ll have some compile errors due to JPA-isms that are in the domain concrete classes. Since these concrete domain classes are generated only once, they don’t get overwritten on the next build and still have the JPA-specific code in them. Edit each concrete domain class that has a gap class, and remove the JPA-isms:

import javax.persistence.Entity;
import javax.persistence.Table;

@Entity
@Table(name = "LINK")
public class Link extends LinkBase {

The domain base classes don’t have to be modified, because they’re regenerated each build.

Fix references to entity unique IDs in your custom code

Another difference with the MongoDB persistence layer is that the entity unique IDs are Strings rather than Longs. Any place in your hand-written code that uses entity IDs, you’ll need to make this change.

Fix service and repository unit tests

When using the JPA-based persistence tier, the service tests  are based on dbunit, which won’t work with MongoDB.  I did a quick search and didn’t find a dbunit-equivalent for MongoDB.  The Sculptor MongoDB test samples use a technique of setting up test data using Java code and the repository classes:


package my.application.serviceapi;

...

/**
 * Spring based test with MongoDB.
 */
@RunWith(org.springframework.test.context.junit4.SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations =  {
    "classpath:applicationContext-test.xml"}
)
public class LinkServiceTest
    extends AbstractDependencyInjectionSpringContextTests
    implements LinkServiceTestBase {
    @Autowired
    private DbManager dbManager;
    @Autowired
    private LinkService linkService;

    @Before
    public void initTestData() {
        // Set up your test fixtures here:
        Link link1 = new Link("http://some.com", "Some title");
        linkService.save(link1);
    }

    @Before
    public void initDbManagerThreadInstance() throws Exception {
        // to be able to do lazy loading of associations inside test class
        DbManager.setThreadInstance(dbManager);
    }

    @After
    public void dropDatabase() {
        Set<String> names = dbManager.getDB().getCollectionNames();
        for (String each : names) {
            if (!each.startsWith("system")) {
                dbManager.getDB().getCollection(each).drop();
            }
        }

        // dbManager.getDB().dropDatabase();
    }

    private int countRowsInDBCollection(String name) {
        return (int) dbManager.getDBCollection(name).getCount();
    }

    @Test
    public void testFindById() throws Exception {
        // Your test code and assertions here...
    }

}

Simply set up your test fixtures using your service/repository classes or the MongoDB DbManager within the initTestData() method.

No support for findByQuery repository method

When building, I got errors such as:

.../LinkAccessFactoryImpl.java:[21,66] cannot find symbol
symbol  : class MongoDbFindByQueryAccessImpl
location: package org.fornax.cartridges.sculptor.framework.accessimpl.mongodb

This is odd since LinkAccessFactoryImpl is a generated class. The cause of this error was that I have findByQuery as one of my scaffolding operations that is generated for every repository. But this type of method isn’t supported by the MongoDb cartridge. In the future it’d be nice if there was a model validation that checked for this and generated an error before generating code. To fix this, remove findByQuery from the list of default scaffolding operations in scupltor-generator.properties, if you have it there at all:

#scaffold.operations=findById,findAll,save,delete,findByQuery,findByKey,findByCondition
scaffold.operations=findById,findAll,save,delete,findByKey,findByCondition

Fix bidirectional associations between aggregate roots

I had a bidirectional association between domain aggregate roots that caused problems. I had something along the lines of:

    Entity A { 
    aggregateRoot 
    
    - Set<@A> children <-> parent 
    
    - @A parent nullable <-> children 
    } 

This resulted in an endless loop in the generated data access code, because it was following both the parent and children associations when mapping the objects to a MongoDB document.

In hindsight, I think this should not be supported within Sculptor when using the MongoDB cartridge. It may make sense for relational databases, but since MongoDB doesn’t support transactions across collections (ditto for most other NoSQL databases), it cannot be guaranteed that both sides of the association will be updated as a unit of work. This also makes distributing your database across many nodes much more complicated. I changed my model to make this a unidirectional association, problem solved.

An interesting point is that with MongoDB and similar databases, the association reference is stored on the “one” side of the association, rather than the “many” side as with relational databases. The document at the “one” side of the association contains a list of IDs of the “many” instances.

Tip: To see how Sculptor maps entities to MongoDB documents, see the “mapper” package within your app.

Add DbManagerFilter to web.xml

Lastly, I had to add the DbManagerFilter servlet filter to the web.xml file. This filter registers a DbManager instance in a thread local so that it may be used to lazy-load associations as needed within your JSF managed bean/Facelets code.


  <filter>
    <filter-name>dbManagerFilter</filter-name>
    <filter-class>org.fornax.cartridges.sculptor.framework.accessimpl.mongodb.DbManagerFilter</filter-class>
  </filter>
  <filter-mapping>
    <filter-name>dbManagerFilter</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>

Overall the switch to MongoDB went very smoothly, and there were very few changes I had to make, largely thanks to Sculptor. The most significant change was switching the unit tests from using dbunit to using Java test fixtures.

Posted in MDSD | Leave a comment

ActiveJDBC: an implementation of the Active Record pattern for Java

Igor Polevoy, who I’ve had the pleasure of working with on a couple projects, has just posted his ActiveJDBC project to Google code.  If you work on Java applications that access relational databases, you owe it to yourself to check it out.

ActiveJDBC implements the Active Record pattern, initially documented by Martin Fowler in his book Patterns of Enterprise Application Architecture and popularized by Ruby on Rails.

What’s interesting about ActiveJDBC is how little baggage it brings with it, and how quickly one can get started using it.  It’s pure Java, requires few other libraries (no Hibernate, JPA, etc), and requires no configuration (no mapping, etc).  All metadata is inferred from the database schema.

Some examples of ActiveJDBC in action:

Persistent Book class definition:

import activejdbc.Model;

public class Book extends Model {}

Adding a new Book:

Book b = new Book();
b.set("isbn", "0321127420")
  .set("title", "Patterns of Enterprise Application Architecture")
  .set("author", "Martin Fowler")
  .set("publisher", "Addison-Wesley Professional");
b.saveIt();

Paging through Addison-Wesley books in sorted order

List<Book> books = Book.where("publisher = ?", ""Addison-Wesley Professional");
  .offset(11)
  .limit(10)
  .orderBy("author asc");

It looks like this framework strikes a good balance between a simple framework that works as you expect, and providing an elegant DSL for database access.  This is particularly impressive given Java as a language doesn’t lend itself to implementing DSLs.

One thing I was curious about but didn’t see an example of, was working with associations, which is listed on the features page.

For more info, see the Getting Started Guide on Google Code, Igor’s blog, and the Productive Edge Java blog.

Posted in Java | 1 Comment

Using Scala for something I’d normally using a “scripting” language for

From time to time, some repetitive task comes up that I can do quicker by writing a script to do it than to do it manually. Especially if it’s something I may be needing to do again in the future.

Usually I’d turn to a “scripting” language like Python, Groovy, or back in the day, Perl for this type of thing.

Today such a need came up, and I decided to try tackling it with Scala since it has many of the features that make the above dynamic languages good for this:

  • first-class support for map and list data structures
  • an interactive shell
  • minimal overhead to write a program, compile, and run it
  • support for functional programming
  • good regular expression support

The problem:
A small performance test program ran a large number of tests, and measured the elapsed time for each execution. The program output a line like “request completed in 10451 msecs” for each test. I needed to parse the output, collect the elapsed time measurements, and get some basic statistics on them; simple average, minimum, and maximum.

I used a Scala 2.8 snapshot, and fleshed out the code using the Scala interactive shell. First, define a value with the raw output to be processed:

scala> val rawData = """request completed in 10288 msecs
     | request completed in 10321 msecs
     | request completed in 10347 msecs
     | request completed in 10451 msecs
     | request completed in 10953 msecs
     | request completed in 11122 msecs
... hundreds of lines ...
     | request completed in 11672 msecs"""

The above uses Scala’s support for multi-line string literals.

The next thing I needed to do was parse the above output, using a regular expression to extract just the milliseconds. There’s several ways to create a regular expression in Scala. This is the one I like:

val ReqCompletedRE = """\s*request completed in (\d+) msecs"""r

There’s a bit of magic in how the above string literal actually ends up becoming a regular expression. There’s an implicit conversion in the Scala Predef object which turns a Java String into a RichString. RichString provides a ‘r’ method that returns a regular expression object. The members of the Predef object are automatically imported into every Scala module, so the Scala compiler will attempt to apply any conversions it finds in Predef when trying to resolve the ‘r’ method. So the above expression is creating a RichString from a String via an implicit conversion, then calling the ‘r’ method on it, which returns the regular expression.

To apply the regular expression to a line of the output and to extract the milliseconds, we can use an expression like:

scala> val ReqCompletedRE(msecs) = " request completed in 10451 msecs"
msecs: String = 10451

msecs gets bound to the first group in the regular expression (the part that matches (\d+)). This takes place via the Scala extractors feature – the scala regular expression class defines an extractor which extracts the grouping results.

The next step is to iterate over the lines of the output, extract the milliseconds, and turn the results into a list.

scala> val msecsVals = rawData.lines.map { line => val ReqCompletedRE(msecs) = line; Integer.parseInt(msecs);} toList
res11: List[Int] = List(10288, 10321, 10347, 10451, 10953, 11122, ..., 11672)

The above code is using the RichString lines, Iterator.map method, along with Scala closures.

Finally, to get the simple statistics:

scala> (msecsVals.sum / msecsVals.length, msecsVals.min, msecsVals.max)
res21: (Int, Int, Int) = (10736,10288,11672)

Putting the whole script together:

val ReqCompletedRE = """\s*request completed in (\d+) msecs"""r
val msecsVals = rawData.lines.map { line => val ReqCompletedRE(msecs) = line; Integer.parseInt(msecs);} toList
(msecsVals.sum / msecsVals.length, msecsVals.min, msecsVals.max)
Posted in Scala | 2 Comments

New change management tool: Redmine

I was starting a new project, needed a change management tool, and wasn’t really satisfied with the tools available last time I looked, so I took a look at what was currently available.  One option I came across that hadn’t been on my radar before was Redmine.

Redmine has many of the features I like from Trac, such as the Roadmap view, version control integration, an Activity view, and built in Wiki.  It also has a bunch of plugins, one of which looks like it’ll generate burndown charts, which I’m planning to use.

It looked like Redmine was fairly well supported on Windows (box I have available) so I decided to try it out.  Redmine is based on Ruby on Rails, so the first step to installation is to get Ruby and Rails installed.

I started off trying to install Ruby and Rails under cygwin, but ran into problems with building mysql from source (no cygwin port for this!).

So instead I went down the route of installing everything outside of cygwin.  Here’s a summary of the steps:

  • Install the Windows Ruby binary, along with Gem package system.
  • Install the Rails 2.1.2 gem, which Redmine 0.8.X depends on
  • Download and unpack the Redmine distribution
  • Setup the database (running on MySQL in my case) and edit database.yml per instructions.
  • I had to install older MySQL client library due to some Rails/MySQL library incompatibility (described here)
  • Create the database schema for Redmine via:
    rake redmine:load_default_data RAILS_ENV=&amp;quot;production&amp;quot;
  • Populate a default configuration via:
    rake redmine:load_default_data RAILS_ENV=&amp;quot;production&amp;quot;
  • You can then run Redmine via:
    ruby script/server webrick -e production
Posted in Software Development, Tools | Leave a comment