Here comes buildr: yet another Java build tool. Hopefully I, or one of my other cohorts will check this out in detail soon. But, with my experience working with all manner of build tools, with 100 companies and many more development teams, I can already make a few observations.

First of all, why another build tool for Java? I am occasionally told that Maven or Ant is a perfect tool, but clearly the people behind buildr don’t think so. The choice of JRuby as the vehicle for delivering this tool, I think is probably a good one. JRuby is a scripting language in the same vain as Perl, which is used by Meister.

Doing software builds is an ugly business involving lots of file and operating system interaction. This is not where Java shines, but scripting languages can. As long as operating systems are written in C and not Java, C-like tools will be better and faster at interacting with them. Plain Ruby itself is C-based, and JRuby no doubt inherits C-like operating traits. Calling out to a Java compiler from Perl or JRuby, though it has its own JVM, does not represent a significant overhead compared with the file system operations and the compilation/translation itself.

Both Maven and Ant are relatively difficult to extend compared most other build tools, and I’ll be buildr beats them here. If you have all the Maven plug-ins and Ant tasks you need, then good for you. If not, then you have to start developing in Java and it becomes too much of an investment to sink into a build system. It is much cheaper to extend in JRuby or Perl. My frequently cited example is the XMLBeans compile step in Meister, written in Perl, which is only 40 lines of real code. The Maven plug-in is 60 pages of Java code and no one can tell me really what it is doing (I asked on all the forums). Less code is usually more transparent, which is also good for build audits.

I am a little disappointed to see them try to placate the Maven and Ant users by promising it is a drop-in replacement for Maven and they have all the Ant tasks covered. Both tools have their drawbacks and I don’t want to see another tool with the same deficiencies. They should have the cajones (or coñejos) to apply all their resources to what they think is a better tool (with its own unique benefits and deficiencies). I imagine offering Ant task equivalents is pretty easy because of the ease of coding in JRuby compared with Java.

They also don’t mention who is supposed to use the tool. Is it for individuals, small development teams, the enterprise? Maven falls short because it is only appropriate for development teams and not for stable, controlled, enterprise builds. Ant is not even tool, but a means to create some tools for small teams. I don’t think Meister will fear buildr either.

Well, since buildr is only in incubation status with Apache, I’m not sure how much time I’ll be able to spend on it, but I am curious and I’ll let you know if I find out more.

As a follow up to my article on automating XML updates, I’d like to report that I did use Excel and Perl’s XML::Twig to successfully generate XML descriptors for my web service consumer, and it was a lot easier than I thought. I’m using XFire 1.2.6 web services stack running under JBoss and using MyEclipse IDE 5.0. I’m happy to say I went from blank spreadsheet and no plan to generated XML files from spreadsheet values in one and half hours. The implementation is of course expandable and reusable. This implementation should work for WebSphere and .NET as well.

I needed to create different configurations for my web application so that the service request went to different endpoints for different environments. The endpoint is at an enterprise service bus (ESB) and there is a different ESB for each environment. I need to have my ‘dev’ instance of the consumer hit the ‘dev’ instance of the ESB, the ‘qa’ instance of my web app hit the ‘qa’ instance of the ESB, etc. We’ve set up Meister to pick up the correct XML file for the target environment for the build of the WAR.

I started by setting up the spreadsheet as follows. I had an unnecessary column for Host indicating JBoss, but I hope to include WebSphere and maybe .NET as well some day. My web app actually connects to two services a.k.a. providers, so there is a column there. And, next is the configuration label for my web app with the name corresponding to the environment it is designed for. So, the first three columns of the spreadsheet look like:

Then I needed a way to indicate the resource that would change. Right now I only have XML files, but I chose to stick with a generic URL for that. Unlike Maven or Ant generators, we start with an XML file that actually works and has been tested - not some hacked up parameterized version that takes additional effort to create. The fourth column of the spreadsheet looks like the following (with repeated entries omitted):

Next, I needed a way to specify a target location to change within the XML file. Now, I know I’m going to use XPath, but I’ll want this to one day work for properties files as well, so I came up with a URL-like thing called a Universal Datum Locator (UDL) which pre-pends the method of locating the datum to change on to a method-specific locator. It could be a property name, an XPath or a Perl regex, for example. In this case it is XPath and then the last column contains the replacement value for the datum indicated by the UDL. XPath is also very intuitive and easier to construct than it may look.

The value for the UDL looks like:

xpath://beans/bean[@factory-bean=’xfireProxyFactory’]/ constructor-arg[@index=’1′]/value

So the fifth column contains the UDL’s, which in my case is always the same XPath expression. The final column of the spreadsheet contains the replacement value of the datum indicated by the UDL:

My nifty Perl script is only about 80 lines of real code and because XML::Twig is nearly the best thing in the world, I pass the entire XPath in as a hash key to modify the source XML file:

my $twig = XML::Twig->new(

pretty_print => ‘indented’,

twig_handlers => {

“$xpath” => sub {

$_->set_text($new_datum);

}

}

);

Here, “$xpath” is directly from the “UDL” column of the spreadsheet with only the ‘xpath://’ stripped off and “$new_datum” is directly from the “Value” column. That’s a pretty useful one line subroutine if you ask me. I had the new XML files each generated into a different folder (dev/,int/, etc). Then, I checked them into version control (CA Harvest) and built each of them with Meister. If you want the full code, let me know and I’ll post it somewhere.

I did find working with the Excel 2003 XML Spreadsheet format a tiny bit awkward. You have to keep track of the column and row indices, but not bad other than that. I see Microsoft Word 2007 allows you to save as an XML document directly, but you apparently have to define bindings. I’ll have to check that out.

In developing Java applications for multiple server environments (e.g. dev, test and prod) there is a common pain-point of having to manage deployment descriptor or configuration files specific to each server. For example, you may have an XML log4j configuration file with some parameters different for different server environments. You may want to turn on debug messaging for the development server, but turn it off for production. At the same time, the Java source code will (eventually) be the same in production as it was in development. A similar situation applies for .NET application development.

Like many build management tasks, managing these environment-specific files is generally left to either manual or some type of scripting. This is really something that needs to have a high level of automation applied. Particularly in larger environments, much like scripted build management solutions, existing tactics fall short. This situation is in a far worse state than even the compile part of build management. It is not enough to simply have a script that can spit out some files. One of the biggest problems is information management and the fact that parameter values in the configuration files may be determined by different teams! How do a production engineering team and an application developer both feed inputs into the same XML file?

I’ve worked on this problem for several years and with a number of companies. The critical functionality can be broken down into two different items – information management and a processing engine. In an effort come up with something better, I’ve done a review of what’s out there and here is what I came up with:

• Ant ‘filter‘ task: As with many Ant tasks, this works great if you are an individual with a few items that need updating. It is a nightmare if you are working in a multi-team enterprise with multiple server environments. The main problem is that you have to constantly take working copies of XML files and insert a token for Ant to later re-replace. This leads to a management nightmare to synchronize parameterized copies of XML files with their working copies from the desktop environment. The advantage is that it works for any file type so you can use it for properties files as well as XML files.
• OOPS Consultancy Ant ‘xmltask‘: This is a good engine for specifying and performing changes to the XML and has a full feature set. In fact, we use this in some of the Meister build services. The problem is that it is only for Ant and therefore you have all the reuse, standardization and hard coding issues. Xmltask can provide part of the solution we are looking for, but we still have an information management problem to deal with.
• Maven: Maven has what is essentially the Ant filter task. The specifications are abstracted in the pom files, which is better than Ant, but it encourages templating of configuration files leading to all the problems associated with that (synchronizing templates with working files, testing templates, etc.)
• XML:DB XUpdate: This is a working draft of a specification to encode XML update instructions into an XML document. There is a Java implementation of XUpdate listed on the site called ‘Lexus’, but I couldn’t find anything on it. Since the build management task requires us to generate XML files, I’m not keen on generating XML files using xupdate tags that will allow me to generate other XML files.
• Perl XML::Twig: This has worked wonderfully for me on a back-end web services security effort and I could not be more happy with such a precise, elegant and brief XML library, which includes XPath. This is not a solution for Java or .NET developers, but it could serve as an engine to mimic xmltask or implement the XUpdate specification.
• Excel. Yes, I’ve seen Excel used effectively as the information management front-end to updating the XML. It is a convenient format to share among teams, it is centralized source of information, it can be checked into version control and it can be saved as an XML file itself for processing by another engine. In a large environment, you may have 5 or more server environments, lots of different components to configure, so you could have literally hundreds of parameters to manage. Excel gives you a nicely transparent way to view those values.

I heard this from some developers today who confirmed the existence, by indirect evidence, of willful tampering of Java archives by gremlins using Ant. There is speculation that they live inside their own plug-in inside WebSphere Studio Application Developer.

Developer A, we’ll call him Nick (because that’s his name), came into our room to work out some tweaks with Openmake targets. He told us all that they’d been working on some problems with runtime errors getting the dreaded Major-Minor version error: “Unsupported major.minor version 49.0.” This usually indicates that you have compiled something with a 1.5 JDK but are running it under a 1.4 JVM.

The archive they were deploying was created wholly inside a workspace managed by WebSphere Studio Application Developer (WSAD) but built by Ant using IBM JDK 1.4.2. Now, WSAD by default launches under a 1.3 JVM, so there is no hint of a simple conflict of JVM’s. They actually traced the issue down to a few class files in the deployed archive that had a 49.0 class versions by using a byte editor. So only a subset of the classes compiled by Ant were affected! But, if they created their own Jar file at the command line through the Windows command prompt using the same exact files, the Major-Minor problem didn’t appear. This means the classes were altered at the packaging step - NOT the compile step!

Harking back to my pain with multiple versions of Java over the last decade, I interrupted this fine tale and asked with a haughty tone “Is a 1.5 JDK installed anywhere on the machine?”

“Yes”, he answered, and the problem, he said, was resolved by uninstalling the 1.5 JDK. We all then had a nice knowing chuckle. Then we questioned Sun’s use of the registry in Java and how the use of the registry changed with each release, forgetting, in fact that the problem happened under an IBM JDK. We also talked about the practice of dumping of java.exe under the windows/system32 (%SYSTEM_ROOT%) directory and how that could trip you up. You never knew if it was a Microsoft java.exe or a Sun one, and if it was Sun, which one was it?

Has anyone heard of this happening before? Is it a bug with Ant? Meister uses Ant under the covers for some things, but not for the Java compile. We’ve never heard of this problem before. We are, in fact, working with Nick to roll out Meister (and the Java build services used by thousands) to Eclipse at the desktop so mysterious problems like this won’t happen in the future.

I received a call from one of our development build meisters … there is a build issue that needs to be resolved for a complex application - this application was building successfully for years without any problems and suddenly it stopped working today. I looked at the error snippet that he had emailed me from the build log…

The first thought that went through my head was “I know how to resolve this - I have seen it before”! I knew that the build tool uses Ant under the cover for Java compiles. All I have to do is increase the memory using ANT_OPTS.

This problem was happening on an Unix machine – this was one of our controlled build machines for our enterprise java builds. Before I changed the environment variable on the build machine, I went through the usual drill…

I asked if anything had changed significantly with the application today - I heard that there was no change other than a few java files were added. I asked him if his other applications were building – he said that everything else has been building successfully except this one application. I asked him if we are able to reproduce this “out of memory” problem - he tried the build for the same application and the compile failed at the same step.

Just to be sure, I requested the on call unix administrator to see if there was any memory problems reported at the system level for that machine. The machine didn’t report any errors of being out of memory or high swap at all that day.

It was time to set the ANT_OPTS environment variable! I added “export ANT_OPTS=-Xmx512m” to the .profile of the system user that is used for all controlled builds on that machine. The development build meister tried the build again - it passed the compile step where it had failed consistently today! The build was successful once again for this complex application. Problem solved!