Here comes buildr: yet another Java build tool. Hopefully I, or one of my other cohorts will check this out in detail soon. But, with my experience working with all manner of build tools, with 100 companies and many more development teams, I can already make a few observations.

First of all, why another build tool for Java? I am occasionally told that Maven or Ant is a perfect tool, but clearly the people behind buildr don’t think so. The choice of JRuby as the vehicle for delivering this tool, I think is probably a good one. JRuby is a scripting language in the same vain as Perl, which is used by Meister.

Doing software builds is an ugly business involving lots of file and operating system interaction. This is not where Java shines, but scripting languages can. As long as operating systems are written in C and not Java, C-like tools will be better and faster at interacting with them. Plain Ruby itself is C-based, and JRuby no doubt inherits C-like operating traits. Calling out to a Java compiler from Perl or JRuby, though it has its own JVM, does not represent a significant overhead compared with the file system operations and the compilation/translation itself.

Both Maven and Ant are relatively difficult to extend compared most other build tools, and I’ll be buildr beats them here. If you have all the Maven plug-ins and Ant tasks you need, then good for you. If not, then you have to start developing in Java and it becomes too much of an investment to sink into a build system. It is much cheaper to extend in JRuby or Perl. My frequently cited example is the XMLBeans compile step in Meister, written in Perl, which is only 40 lines of real code. The Maven plug-in is 60 pages of Java code and no one can tell me really what it is doing (I asked on all the forums). Less code is usually more transparent, which is also good for build audits.

I am a little disappointed to see them try to placate the Maven and Ant users by promising it is a drop-in replacement for Maven and they have all the Ant tasks covered. Both tools have their drawbacks and I don’t want to see another tool with the same deficiencies. They should have the cajones (or coñejos) to apply all their resources to what they think is a better tool (with its own unique benefits and deficiencies). I imagine offering Ant task equivalents is pretty easy because of the ease of coding in JRuby compared with Java.

They also don’t mention who is supposed to use the tool. Is it for individuals, small development teams, the enterprise? Maven falls short because it is only appropriate for development teams and not for stable, controlled, enterprise builds. Ant is not even tool, but a means to create some tools for small teams. I don’t think Meister will fear buildr either.

Well, since buildr is only in incubation status with Apache, I’m not sure how much time I’ll be able to spend on it, but I am curious and I’ll let you know if I find out more.

As a follow up to my article on automating XML updates, I’d like to report that I did use Excel and Perl’s XML::Twig to successfully generate XML descriptors for my web service consumer, and it was a lot easier than I thought. I’m using XFire 1.2.6 web services stack running under JBoss and using MyEclipse IDE 5.0. I’m happy to say I went from blank spreadsheet and no plan to generated XML files from spreadsheet values in one and half hours. The implementation is of course expandable and reusable. This implementation should work for WebSphere and .NET as well.

I needed to create different configurations for my web application so that the service request went to different endpoints for different environments. The endpoint is at an enterprise service bus (ESB) and there is a different ESB for each environment. I need to have my ‘dev’ instance of the consumer hit the ‘dev’ instance of the ESB, the ‘qa’ instance of my web app hit the ‘qa’ instance of the ESB, etc. We’ve set up Meister to pick up the correct XML file for the target environment for the build of the WAR.

I started by setting up the spreadsheet as follows. I had an unnecessary column for Host indicating JBoss, but I hope to include WebSphere and maybe .NET as well some day. My web app actually connects to two services a.k.a. providers, so there is a column there. And, next is the configuration label for my web app with the name corresponding to the environment it is designed for. So, the first three columns of the spreadsheet look like:

 

Then I needed a way to indicate the resource that would change. Right now I only have XML files, but I chose to stick with a generic URL for that. Unlike Maven or Ant generators, we start with an XML file that actually works and has been tested - not some hacked up parameterized version that takes additional effort to create. The fourth column of the spreadsheet looks like the following (with repeated entries omitted):

 

Next, I needed a way to specify a target location to change within the XML file. Now, I know I’m going to use XPath, but I’ll want this to one day work for properties files as well, so I came up with a URL-like thing called a Universal Datum Locator (UDL) which pre-pends the method of locating the datum to change on to a method-specific locator. It could be a property name, an XPath or a Perl regex, for example. In this case it is XPath and then the last column contains the replacement value for the datum indicated by the UDL. XPath is also very intuitive and easier to construct than it may look.

The value for the UDL looks like:

xpath://beans/bean[@factory-bean=’xfireProxyFactory’]/ constructor-arg[@index=’1′]/value

So the fifth column contains the UDL’s, which in my case is always the same XPath expression. The final column of the spreadsheet contains the replacement value of the datum indicated by the UDL:

 

My nifty Perl script is only about 80 lines of real code and because XML::Twig is nearly the best thing in the world, I pass the entire XPath in as a hash key to modify the source XML file:

my $twig = XML::Twig->new(

pretty_print => ‘indented’,

twig_handlers => {

“$xpath” => sub {

$_->set_text($new_datum);

}

}

);

Here, “$xpath” is directly from the “UDL” column of the spreadsheet with only the ‘xpath://’ stripped off and “$new_datum” is directly from the “Value” column. That’s a pretty useful one line subroutine if you ask me. I had the new XML files each generated into a different folder (dev/,int/, etc). Then, I checked them into version control (CA Harvest) and built each of them with Meister. If you want the full code, let me know and I’ll post it somewhere.

I did find working with the Excel 2003 XML Spreadsheet format a tiny bit awkward. You have to keep track of the column and row indices, but not bad other than that. I see Microsoft Word 2007 allows you to save as an XML document directly, but you apparently have to define bindings. I’ll have to check that out.

I’ll also be attending Ignite - Chicago. This is not the music festival, but a series of geeky 5 minute talks with about 20 slides. This is just my style, the length of the talk matching the length of my patience. My talk is entitled “Controlling Java with Perl”, and I’ll be covering many of the same topics related to build management I cover in my blog. I see O’Reilly Publishing’s name on the web page, so maybe they have something to do with it. Already there are 98 people planning to attend. Like the Windy City Hackathon, if you are planning to attend, please drop me a line – I’ll buy you a beer.

Ivy seems to be trying to resurrect Ant, when it should stay dead in my opinion (unless you are a lone developer on the simplest project). Maven is a more proper build tool with some workflow functionality that falls short outside of a small team of developers with no audit or control requirements. Both are vying for prominence as the open source build tool, yet trying to settle into compatible niches like two brothers hopeful for the throne.

Here is what Ivy says about Maven: http://ant.apache.org/ivy/m2comparison.html

And here is Maven’s side of the story: http://docs.codehaus.org/display/MAVEN/Feature+Comparisons

I think it is funny that Electric Cloud is saying that Perl is and old, outdated language (presumably to position themselves against OpenMake and Meister) when they are based on make, which truly was designed for C and is not very extensible (but works for ad hoc build scripting). I can add FTP or email support to a Meister build method in less than 10 lines of Perl code - not so easy to do with make alone. I can even update my Facebook page with the latest build result through a Perl API.

Most developers know Perl has a recognized place in most companies. There are trends in software languages, and Perl may not be the latest, coolest thing on the street (like Ruby or Python) but Perl is incredibly dynamic and robust. It is possibly the greatest single example of successful open source development. Neither Ruby, Python nor anything else besides the open source Java community even comes close to offering the breadth of functionality that Perl does. Perl even does  automatic release dependency management - something unique for its scope in the open source world, although the Maven project is attempting to do something similar with Java.

Java developers often think Java is the only thing in the world, but in comparing Maven (an all Java build tool for Java builds only) and OpenMake, it took 60 pages of Java code to do the same thing (see my previous blog on XMLBeans) as 1.5 pages of perl code as used in OpenMake.

Perl works great at moving files around and working with the operating system – Java is not so good at that. Meister and Mojo use Java for the server which works well, but Meister uses Perl where it can be used best (parsing texts and interacting with the operating system) and C is in turn used where is best (C can’t be beat for speed). We also use make for what it is does well - the rules engine and dependency ordering. Combining the C based make engine with Perl for executing commands really gives enormous power to Meister.

An important part of a build system is transparency. How easy is it for some to tell how the gory mechanics of a build are done? In part 2, I continue with the comparison of the perl build method of Meister and the Java plug-in of Maven to compile XML schema’s to Java classes via the XMLBeans compiler. The lack of transparency in the Maven build was costly to developers.

As I mentioned in part 1, creating the perl stub, or build method, was pretty straightforward for Meister. Using the predefined perl objects from the Openmake::File and Openmake::FileList classes allowed a straightforward call of the XMLBeans compiler, via the XMLBeans scomp wrapper, took less than a page and a half of perl code. There was one parameter that was required to generate 1.5 Java source code, and we got it working, but there was some question about extra classes in the straightforward compile by Meister compared with the JAR file created with Maven using the Maven XMLBeans compile plug-in (here). The straight Meister build also had some classes in different folders with names that looked suspiciously like signatures. This made some people nervous about the integrity of the Meister build.

So, I asked how Maven was doing the compile. I had set the Meister build method to just call the compiler with one parameter and the source files. It’s not like it was the GNU C++ compiler with its 400 or so flags! There was literally no other way to do the build. The Meister build was done exactly the way Apache said to do it. I didn’t get any answers.

I should also point out that my developer contact was worried about differences in the byte counts for like named classes. But, in most modern compilers, including Java, source file system paths and/or time stamps are often embedded in the binary or byte code, so it is actually more common than not that byte counts will not match when doing builds with two different build systems.

I had the fortune of working with Selvi at the time, and she looked at the extra classes in the Meister scomp compile and said they looked like temporary template classes, because of their naming convention. They all ended in “List” – also common in C++. A little research indicated that was likely the case. Was Maven cleaning these up?

OK, so I couldn’t get any answers from anyone. I went and downloaded the source code for the Maven plug-in for XMLBeans. I printed it out and got a big shock: it was 60 pages of Java source code printed! I went through the code, and while I understand Java and could basically understand what it was doing, the details were not obvious. Running Maven through a Java debugger was not going to happen. I then posted a question on the XMLBeans developers’ forum (can’t remember the URL) and got no response, despite the fact that there were a number of build questions answered there.

By this time, the developers had already deployed the Meister-built JAR and it had tested fine, and there was no more nervousness about differences in the fine details of the differently built JAR files. It was time to move on.

Apparently none of the developers either in house or responsible for XMLBeans at Apache could explain the differences, which means they could not explain how Maven was doing the build. I’m sure the one guy somewhere in the world who wrote the Maven plug-in probably knows, but I didn’t try too hard to find him.

The lack of transparency in the Maven build process probably cost the organization about 20 hours of work through unnecessary research and validation. Usually the lack of transparency in the build process is mainly the concern of auditors, but this is a case where it was costly for the developers.

An important part of a build system is transparency. How easy is it for some to tell how the gory mechanics of a build are done? Sean looks at his experience writing a Meister build service and method to compile XMLBeans and compares it with Maven.

I was working on a build team that managed QA and Production builds using OpenMake (previous name for Meister). In this particular shop, the developers were free to use what they wanted on their desktops, but OpenMake was used for the QA and Production builds.

It was determined that XMLBeans was going to be used as part of one application build. XMLBeans (http://xmlbeans.apache.org/ ) binds Java objects to XML schema designs (XSD). It has a simple compile process that takes XSD and an XSD config file and produces a JAR. The developers were using Maven with a plug-in to compile the XMLBeans.

How a company can effectively adopt new technologies deserves a book in itself, and I am not suggesting this is the best process to take, but this is how it happened. I took on the task of creating the build service (rules) and build method (perl stub) to allow OpenMake to build the XMLBeans JAR. This JAR was then deployed in a WAR under WEB-INF/lib. That was pretty much the only specs for this build service I had.

The simple approach is to find out from the vendor, Apache in this case, their recommended approach for compiling and stick closely to that. The documentation was sparse, but there was a command line wrapper, called scomp, for the Java compiler class available that loaded the necessary libraries and constructed the arguments for the Java class. My impression looking at the wrapper (ksh for *NIX, bat for Windows) was that it was a little too simple, but it met all the specs so I went forward with having the build method perl stub call the scomp wrapper using OpenMake’s perl object methods for both the dependencies and the output JAR file name in the scomp command line. The perl object for dependencies simply contains a list of the XSD and XSD config files as determined by the C engine by searching for dependencies along the search path. The name of the JAR file is determined by the target defintion.

Using these objects is pretty simple and familiar to most Perl programmers. If I want to use the name of the expected JAR file name, the target name, I use $Target->get. That’s it. (It’s not a proprietary language as some competitors suggest!) It was pretty straightforward, and I turned it back to the developers.