How do you set up an automated build for EJB client JAR’s from the IBM Rational Software Delivery 7 development environment for WebSphere 6?

This question came up recently in my work for a major insurance company. When one extends the EJB client class, that is all a developer has to do as far as RAD 7 is concerned. When the developer deploys the JAR to the server, RAD 7 quietly generates stub source Java classes, compiles them and includes them in the JAR file.

An automated build in this context means that all the code the developer created in RAD 7 and checked into version control, is checked out of version control without RAD 7 and built exactly the way the developer intended. This is what OpenMake Meister is for.

One developer I was working with was concerned with how to generate those same source files in the automated build, which in his case was using OpenMake. He was familiar with how OpenMake uses the ejbdeploy command for building EAR’s with EJB server-side code and expected some equivalent for the EJB client.

Mercifully RAD 7 actually leaves the generated source files behind in the Eclipse project, in the standard source location. This means that we get the source code for free and there is really no need to regenerate it. All one has to do is check in the generated source to version control along with the developer coded source and build a normal JAR file in the automated build.

For the developer, this means:

1. Check in all the Java source code in the project. This is easy – better than picking and choosing which Java source to check in as the developer thought he had to do. This would be ultra-high risk for making a mistake and breaking the team’s automated build.
2. Assign the “Default Java JAR” configuration mapping for the EJB client project using the OpenMake Target Generator Eclipse plug-in.

A lesson to learn from this is that not all technologies or technology variants will have an impact to the build process. The developer was considering an idealist approach to reproduce every minute step of RAD 7, but the best solution was something practical and simple. Build management is part art and part dirty science. Having a “generate” step for the EJB client Java classes in the automated build only introduces an additional point of possible failure, and we build-meisters know we don’t need any more of those!

As a follow up to my article on automating XML updates, I’d like to report that I did use Excel and Perl’s XML::Twig to successfully generate XML descriptors for my web service consumer, and it was a lot easier than I thought. I’m using XFire 1.2.6 web services stack running under JBoss and using MyEclipse IDE 5.0. I’m happy to say I went from blank spreadsheet and no plan to generated XML files from spreadsheet values in one and half hours. The implementation is of course expandable and reusable. This implementation should work for WebSphere and .NET as well.

I needed to create different configurations for my web application so that the service request went to different endpoints for different environments. The endpoint is at an enterprise service bus (ESB) and there is a different ESB for each environment. I need to have my ‘dev’ instance of the consumer hit the ‘dev’ instance of the ESB, the ‘qa’ instance of my web app hit the ‘qa’ instance of the ESB, etc. We’ve set up Meister to pick up the correct XML file for the target environment for the build of the WAR.

I started by setting up the spreadsheet as follows. I had an unnecessary column for Host indicating JBoss, but I hope to include WebSphere and maybe .NET as well some day. My web app actually connects to two services a.k.a. providers, so there is a column there. And, next is the configuration label for my web app with the name corresponding to the environment it is designed for. So, the first three columns of the spreadsheet look like:

 

Then I needed a way to indicate the resource that would change. Right now I only have XML files, but I chose to stick with a generic URL for that. Unlike Maven or Ant generators, we start with an XML file that actually works and has been tested - not some hacked up parameterized version that takes additional effort to create. The fourth column of the spreadsheet looks like the following (with repeated entries omitted):

 

Next, I needed a way to specify a target location to change within the XML file. Now, I know I’m going to use XPath, but I’ll want this to one day work for properties files as well, so I came up with a URL-like thing called a Universal Datum Locator (UDL) which pre-pends the method of locating the datum to change on to a method-specific locator. It could be a property name, an XPath or a Perl regex, for example. In this case it is XPath and then the last column contains the replacement value for the datum indicated by the UDL. XPath is also very intuitive and easier to construct than it may look.

The value for the UDL looks like:

xpath://beans/bean[@factory-bean=’xfireProxyFactory’]/ constructor-arg[@index=’1′]/value

So the fifth column contains the UDL’s, which in my case is always the same XPath expression. The final column of the spreadsheet contains the replacement value of the datum indicated by the UDL:

 

My nifty Perl script is only about 80 lines of real code and because XML::Twig is nearly the best thing in the world, I pass the entire XPath in as a hash key to modify the source XML file:

my $twig = XML::Twig->new(

pretty_print => ‘indented’,

twig_handlers => {

“$xpath” => sub {

$_->set_text($new_datum);

}

}

);

Here, “$xpath” is directly from the “UDL” column of the spreadsheet with only the ‘xpath://’ stripped off and “$new_datum” is directly from the “Value” column. That’s a pretty useful one line subroutine if you ask me. I had the new XML files each generated into a different folder (dev/,int/, etc). Then, I checked them into version control (CA Harvest) and built each of them with Meister. If you want the full code, let me know and I’ll post it somewhere.

I did find working with the Excel 2003 XML Spreadsheet format a tiny bit awkward. You have to keep track of the column and row indices, but not bad other than that. I see Microsoft Word 2007 allows you to save as an XML document directly, but you apparently have to define bindings. I’ll have to check that out.

When you work with a locking-type version control tool like CA Harvest, your Meister build project will appear in your Eclipse workspace as read-only when you check out an existing workspace. I’ve been using Eclipse for WebSphere development (WebSphere Studio Application Developer) and for JBoss via MyEclipse IDE. If you want to regenerate your Java targets, you first have to check out the Meister build project so that the files are writable.

Since this can lock the targets exclusively and prevent others from updating the target, you may not want to check out the build project, but you may still want to develop freely and update your local targets for Meister to build it. For this situation I recommend creating a separate build project that you may never check in to version control. It will be writable and it allows you great freedom for a maximally agile development environment. The ‘official’ build project may reference all the built archives in the workspace, but having your own local build project can allow you to focus for a unit build. For example, my workspace may contain an EAR project, a WAR project and one or more JAR projects. If I am principally working only on one of the JAR projects, my local build project can reference only that one JAR project.

When it’s time to release your JAR code updates to the system build and test environments, synchronize your workspace and check out the VC build project. Generate your targets, do a local system build and then check everything in. Your team system build will work fine!

I wanted to share a specific benefit I enjoyed while using Meister for Java development. As part of my role to help develop an automated JBoss build and deploy system, I ended up taking on a developer role for a web services security project for both JBoss and WebSphere. While the project involved about 1000 lines of Perl, it also got me writing simple web services and consumers for JBoss and WebSphere and building them using Meister and its Eclipse plug-in.

Believe it or not, I am still using WebSphere Studio Application Developer 5.1. While my specific tale involves that IDE, it is equally applicable to MyEclipse and Rational Application Developer set of Eclipse IDE’s. In my environment, CA Harvest is the version control/SCM tool and Meister is the build tool. After code is checked in from my desktop using the CA Harvest eclipse plug-in, the code is replicated out to a Linux server, where Meister performs the official system build that is sanctioned for deployment to the application server. There is also a Meister Eclipse plug-in that scans the WSAD workspace for build targets and dependencies. Meister stores this information in one XML file per build target and those files are also checked in to CA Harvest right along side the source code.

Working intensely within the WSAD Eclipse environment as the project manager cracked the whip, I worked with a consumer application and updated it according to the changes in the service WSDL and service endpoint URL’s. One thing I learned is that if one of the parameters for the consumer is tweaked, don’t bother tweaking the XML or generated code, just regenerate the whole client. WSAD will even check out the files before if they need to be. So everything looked good on my desktop with the service and consumer deployed to two separate WebSphere servers on ports 9080 and 9081. Now to get it into the enterprise ‘dev’ environment…

Using the ‘Generate Target Definitions’ feature of the Meister plug-in I updated the Meister build target XML definition files and checked in all my code. I then promoted the code in CA Harvest which automatically kicked off a ‘dev’ build in the Linux environment. I got an error back from Meister saying ‘jdmpview.jar’ doesn’t exist.

Since I knew my consumer app and its elementary nature, I knew that jdmpview.jar wasn’t one of my JAR’s and it must be one of WebSphere’s. Given that 200 other Java apps use the same build environment with the same standards, I probably didn’t use some new feature of WebSphere that no one else is using. Therefore, it must a problem on my local desktop with the version of JVM I was using.

Sure enough, the consumer app was using the base_v51 WebSphere runtime instead of the ee_v51. (I did inherit the initial version of the app from someone else!) And, oddly enough, there is an extra JAR in the base that is missing in the more fully featured Enterprise Edition. Meister correctly forced the runtime environment to be EE for the Linux build, overriding the developer selection. I switched the runtime in the Java build path properties, regenerated the Meister target definitions, checked them in and promoted them to a successful ‘dev’ build. Regenerating the target definitions had the effective of switching out the list of JAR files in the library path from the base_v51 set to the ee_v51 set. The whole thing including one bad and one good build took about 4 minutes.

The great benefit for me was the balance between developer and SCM functions. We could have applied more controls at the desktop level, but from my perspective, I prefer an Agile environment with more freedom even if it means occasionally hanging myself with my own rope. In this scenario I let the tools dot the I’s and cross the T’s and it took no more time than say, waiting for Outlook over VPN.

I am working on a project to secure Java web services running under JBoss and WebSphere application servers. As part of this effort, I’ve created some ‘Hello World’ type web services and consumers to use for the team’s integration testing and implementation verification. This has allowed me to launch two identical projects: one using MyEclipse targeting the JBoss application server and the other using WebSphere Studio Application Developer 5.1 (WSAD) targeting the WebSphere 5 application server environment. By doing these two Java development projects side by side, I’ve been able to compare the two development environments on equal grounds for one very specific type of development.

Initial Setup of Eclipse Projects and Local Servers

Furthermore, the applications for JBoss used the XFire SOAP implementation for web services. This is an external component, but MyEclipse supports this and has a simple wizard to generate a web service. I found resources on the web and within MyEclipse that helped me struggle through writing my first web service and I learned a lot. It was harder to learn how to do get it to work in the MyEclipse environment than it was to learn how it to configure it, but I appreciate the features in MyEclipse. MyEclipse’s support for JBoss is also nice and it is really easy to integrate with a JBoss server, which must be installed and configured separately. One-click server start/stop and one-click deploy from your Eclipse project was really nice.

Setting up the WebSphere applications was a different experience as everything was done within the WSAD IDE. You used wizards (Eclipse plug-ins) to generate both web services and clients, and when you are done, you also get JSP’s and Java classes that allow you to test them. I admit I had some more experienced help in setting this environment up, but it was real point-and-click. And, so was the server configuration. Everything about the server is in a ’server’ Eclipse project and you can control parameters within WSAD. In fact, it was so easy that we decided to run two WebSphere servers listening on different ports and deploy the service to one and the consumer to the other. To do the same for JBoss, you have to define a separate server using the configuration files and MyEclipse will happily support it, but I don’t really have the time to learn about details of JBoss server administration to do that. WSAD wins there, but everything else in setting up the applications was pretty equivalent.

Iterative Development

Going forward to create the applications I wanted, with naming standards and just the right Hello World message, I first had to contend with port conflicts between the JBoss server and the Windows operating system. It seems that a lot of the port numbers for JBoss are around the 1000’s and that’s just where Windows starts giving away port numbers. This required us to learn how to reserve ports by manually modifying the registry. OK, maybe JBoss is not to blame there.

Finally we get JBoss working properly and the second annoying thing arises. After deploying your web application to the JBoss server (a nice one-click that physically copies files from the Eclipse workspace to the deploy folder on the JBoss server) it works fine, but redeploying or removing the application is problematic. The Windows operating system again intrudes and locks at least some JAR files. In order to redeploy your entire application, you have to stop the JBoss server, redeploy and restart the server. Restarting takes awhile. If you forget to shut down the server first, you will get a nasty gram and a half-deployed application. To be fair, files are copied to the server when you save them in your workspace preventing the need to constantly redeploy, but starting up the project required a lot of redeploying and it was very tedious.

Here is where IBM rockets ahead. With two WebSphere servers running on different ports, it was pretty simple to associate different projects with the different servers for one-click everything. Because of the inevitable mis-starts (like using the wrong JDK version), we had to redefine and redeploy the applications a number of times at the beginning of the project. This procedure was definitely less painful than the JBoss experience and we had more capabilities in the end with two servers. Also, the server configurations are stored in an Eclipse project, which are shared via version control to other developers. One thing that helps architecturally is that the WebSphere servers use your actual Java Eclipse project folders as the deployed application area. This prevents the need to copy files around. There is also a nice server management view that allows you to look at the different logs and perform server start/stops.

Summary

I like the more lightweight MyEclipse/JBoss combination that seems to be more simple and understandable, and also gives me the freedom to screw up my own project. The painful redeployment process and the external configuration of JBoss were drawbacks. For a multi-developer environment, a number of standards would need to be defined to get everyone working on the same page. IBM’s WSAD did most of the configuration for me for the servers and it also generated web services applications that were complete and robust, but it was impossible to understand the inner workings (for me). Changing any of the major parameters of your web service required you to regenerate it completely, but it worked great and it was easy. I also ended up with more capabilities by having two servers. The version of WSAD I used, 5.1, is quite old, but I will soon be able to go through the same exercise with Rational Application Developer for WebSphere Software version 7. I can’t wait!