How do you set up an automated build for EJB client JAR’s from the IBM Rational Software Delivery 7 development environment for WebSphere 6?

This question came up recently in my work for a major insurance company. When one extends the EJB client class, that is all a developer has to do as far as RAD 7 is concerned. When the developer deploys the JAR to the server, RAD 7 quietly generates stub source Java classes, compiles them and includes them in the JAR file.

An automated build in this context means that all the code the developer created in RAD 7 and checked into version control, is checked out of version control without RAD 7 and built exactly the way the developer intended. This is what OpenMake Meister is for.

One developer I was working with was concerned with how to generate those same source files in the automated build, which in his case was using OpenMake. He was familiar with how OpenMake uses the ejbdeploy command for building EAR’s with EJB server-side code and expected some equivalent for the EJB client.

Mercifully RAD 7 actually leaves the generated source files behind in the Eclipse project, in the standard source location. This means that we get the source code for free and there is really no need to regenerate it. All one has to do is check in the generated source to version control along with the developer coded source and build a normal JAR file in the automated build.

For the developer, this means:

1. Check in all the Java source code in the project. This is easy – better than picking and choosing which Java source to check in as the developer thought he had to do. This would be ultra-high risk for making a mistake and breaking the team’s automated build.
2. Assign the “Default Java JAR” configuration mapping for the EJB client project using the OpenMake Target Generator Eclipse plug-in.

A lesson to learn from this is that not all technologies or technology variants will have an impact to the build process. The developer was considering an idealist approach to reproduce every minute step of RAD 7, but the best solution was something practical and simple. Build management is part art and part dirty science. Having a “generate” step for the EJB client Java classes in the automated build only introduces an additional point of possible failure, and we build-meisters know we don’t need any more of those!

If you are a hard-core open source programmer, you probably use Git for project code change management instead of Subversion (I chose those words carefully). There is a lot of passion from Git advocates and, while it is not a very mature solution, it has a lot of momentum to push it forward. Merely being conceived of and written by Linus Torvalds and being used on a few large open source projects, such as the very Linux kernel itself, is enough to garner wide support.

A great place to learn about Git is Sam Vilain’s Tutorial. He goes into a lot of detail on the benefits and how-to’s of using Git. Some of the highlights include repository space savings of over 90% and local-to-repository sync times dropping from hours in Subversion to minutes with Git. The real power of Git is in the highly distributed repositories and the ease and control of moving and accepting changes between repositories. For an open source project with a large number of developers it seems Git will really shine. Git has fine control over branching, merging and accepting or not accepting project changes according to various criteria.

A popular way to use Git is to have Git pull from a public Subversion or CVS repository with convenient integration with those tools to a local Git repository and work from there. Friends working on the same project can easily pass changes between each other with Git and later commit back to the centralized CVS or Subversion repository. GitHub provides a simple Git repository hosting service. Doing a lot of Java work with JBoss and WebSphere, I am naturally interested in an Eclipse plug-in for Git and indeed one exists. It looks like a newborn infant, but I will check it out.

I also have a Perl open source project that is currently pretty anemic, but I hope to revitalize it soon. I really hate the fact that I’m locked into using Subversion on SourceForge and I never came to like Subversion. I’m eager to explore moving the project to GitHub, even though I’ll probably be the only committer for awhile. Since I’m a hardcore software management person and robust Perl developer, I think Git might be my tool. I’ll let you know.

Here comes buildr: yet another Java build tool. Hopefully I, or one of my other cohorts will check this out in detail soon. But, with my experience working with all manner of build tools, with 100 companies and many more development teams, I can already make a few observations.

First of all, why another build tool for Java? I am occasionally told that Maven or Ant is a perfect tool, but clearly the people behind buildr don’t think so. The choice of JRuby as the vehicle for delivering this tool, I think is probably a good one. JRuby is a scripting language in the same vain as Perl, which is used by Meister.

Doing software builds is an ugly business involving lots of file and operating system interaction. This is not where Java shines, but scripting languages can. As long as operating systems are written in C and not Java, C-like tools will be better and faster at interacting with them. Plain Ruby itself is C-based, and JRuby no doubt inherits C-like operating traits. Calling out to a Java compiler from Perl or JRuby, though it has its own JVM, does not represent a significant overhead compared with the file system operations and the compilation/translation itself.

Both Maven and Ant are relatively difficult to extend compared most other build tools, and I’ll be buildr beats them here. If you have all the Maven plug-ins and Ant tasks you need, then good for you. If not, then you have to start developing in Java and it becomes too much of an investment to sink into a build system. It is much cheaper to extend in JRuby or Perl. My frequently cited example is the XMLBeans compile step in Meister, written in Perl, which is only 40 lines of real code. The Maven plug-in is 60 pages of Java code and no one can tell me really what it is doing (I asked on all the forums). Less code is usually more transparent, which is also good for build audits.

I am a little disappointed to see them try to placate the Maven and Ant users by promising it is a drop-in replacement for Maven and they have all the Ant tasks covered. Both tools have their drawbacks and I don’t want to see another tool with the same deficiencies. They should have the cajones (or coñejos) to apply all their resources to what they think is a better tool (with its own unique benefits and deficiencies). I imagine offering Ant task equivalents is pretty easy because of the ease of coding in JRuby compared with Java.

They also don’t mention who is supposed to use the tool. Is it for individuals, small development teams, the enterprise? Maven falls short because it is only appropriate for development teams and not for stable, controlled, enterprise builds. Ant is not even tool, but a means to create some tools for small teams. I don’t think Meister will fear buildr either.

Well, since buildr is only in incubation status with Apache, I’m not sure how much time I’ll be able to spend on it, but I am curious and I’ll let you know if I find out more.

First, let me say how nice it is to have the Mojo workflow engine that allows us to manage the compliance checks, deploy to multiple machines in parallel and validate deployment. This makes our lives a lot easier and provides clear benefits for deployment via the parallelization, dependency management, scalability, logging and reporting. Underneath the covers, and for those of you who don’t have the luxury to use this almost-free product, there are some important low-level tools that are critical to the development, testing and operation of the Mojo JBoss deployment system on Linux.

With the most important listed first, they are:

1. JBoss support
2. The Perl executable (5.6-5.10) and base language
3. Perl’s Test::Simple or Test::More modules
4. Perl’s Test::Harness module
5. The JBoss twiddle.sh script or command equivalent
6. Perl’s XML::Twig
7. Perl’s Archive::Zip
8. vi
9. ssh
10. xterm

JBoss support wins hands down due to the number of bugs and critically important undocumented features. On a scale of 1 to 10 where 10 is the best documentation, I give JBoss about a 3 or 4. Googling doesn’t even help that much for deployment issues.

You may be surprised at the prominence of Perl, but if you think about what you are really doing and what the best tool for the job is, it makes sense. You are really moving an archive (a ZIP format file), copying XML files, creating directories, changing permissions, extracting the archive to the file system perhaps. Where did I mention Java? Nowhere. The twiddle.sh command comes in handy if you get the secret commands from JBoss support that tell you if the application you deployed has actually started correctly. Notice that this is a shell script suggesting we’re not the first to use non-Java tools to manage deployment.

Particularly on the testing side, I can’t think of a viable alternative to Perl testing. We need to test that we created this directory, changed that permission, updated that file timestamp, etc. We have about 300 test cases encoded in Perl that are run with every change to the deployment system. It takes about 20 seconds to write and run a simple test case in Perl.

Lessons? Use JBoss support early and often and use Perl.

As a follow up to my article on automating XML updates, I’d like to report that I did use Excel and Perl’s XML::Twig to successfully generate XML descriptors for my web service consumer, and it was a lot easier than I thought. I’m using XFire 1.2.6 web services stack running under JBoss and using MyEclipse IDE 5.0. I’m happy to say I went from blank spreadsheet and no plan to generated XML files from spreadsheet values in one and half hours. The implementation is of course expandable and reusable. This implementation should work for WebSphere and .NET as well.

I needed to create different configurations for my web application so that the service request went to different endpoints for different environments. The endpoint is at an enterprise service bus (ESB) and there is a different ESB for each environment. I need to have my ‘dev’ instance of the consumer hit the ‘dev’ instance of the ESB, the ‘qa’ instance of my web app hit the ‘qa’ instance of the ESB, etc. We’ve set up Meister to pick up the correct XML file for the target environment for the build of the WAR.

I started by setting up the spreadsheet as follows. I had an unnecessary column for Host indicating JBoss, but I hope to include WebSphere and maybe .NET as well some day. My web app actually connects to two services a.k.a. providers, so there is a column there. And, next is the configuration label for my web app with the name corresponding to the environment it is designed for. So, the first three columns of the spreadsheet look like:

Then I needed a way to indicate the resource that would change. Right now I only have XML files, but I chose to stick with a generic URL for that. Unlike Maven or Ant generators, we start with an XML file that actually works and has been tested - not some hacked up parameterized version that takes additional effort to create. The fourth column of the spreadsheet looks like the following (with repeated entries omitted):

Next, I needed a way to specify a target location to change within the XML file. Now, I know I’m going to use XPath, but I’ll want this to one day work for properties files as well, so I came up with a URL-like thing called a Universal Datum Locator (UDL) which pre-pends the method of locating the datum to change on to a method-specific locator. It could be a property name, an XPath or a Perl regex, for example. In this case it is XPath and then the last column contains the replacement value for the datum indicated by the UDL. XPath is also very intuitive and easier to construct than it may look.

The value for the UDL looks like:

xpath://beans/bean[@factory-bean=’xfireProxyFactory’]/ constructor-arg[@index=’1′]/value

So the fifth column contains the UDL’s, which in my case is always the same XPath expression. The final column of the spreadsheet contains the replacement value of the datum indicated by the UDL:

My nifty Perl script is only about 80 lines of real code and because XML::Twig is nearly the best thing in the world, I pass the entire XPath in as a hash key to modify the source XML file:

my $twig = XML::Twig->new(

pretty_print => ‘indented’,

twig_handlers => {

“$xpath” => sub {

$_->set_text($new_datum);

}

}

);

Here, “$xpath” is directly from the “UDL” column of the spreadsheet with only the ‘xpath://’ stripped off and “$new_datum” is directly from the “Value” column. That’s a pretty useful one line subroutine if you ask me. I had the new XML files each generated into a different folder (dev/,int/, etc). Then, I checked them into version control (CA Harvest) and built each of them with Meister. If you want the full code, let me know and I’ll post it somewhere.

I did find working with the Excel 2003 XML Spreadsheet format a tiny bit awkward. You have to keep track of the column and row indices, but not bad other than that. I see Microsoft Word 2007 allows you to save as an XML document directly, but you apparently have to define bindings. I’ll have to check that out.

JBoss checks for certain watch files when handling deploying or undeploying an application. The watch files are certain key files germane to the object you are deploying. For an EAR, the watch file is the application.xml and the optional jboss-app.xml files. For a web application archive, the watch files are the web.xml and jboss-web.xml files. For single-file XML resources, such as datasources, the watch file is the XML file itself. In this article, I am dealing with archives that are deployed in unextracted (unzipped) form.

The first check is made for the existence or non-existence of a watch file. If a previously unknown watch file is found, the appropriate deployer is started and the file modification timestamp is stored in memory. If a known watch file is found to be missing, the appropriate undeployer is launched.

If a known watch file is found on a subsequent pass of checking watch files, its timestamp is checked against the time that was stored in memory by the deploy process. If the deployed watch file is newer, the appropriate deployer is launched which apparently first dumps the associated resources and then reloads the object as if it were newly found.

This leaves a hole that can lead to the horrifying result of having files deployed to the server, but not having the changes reflected in the running application.

The issue has to do with completely replacing a running application with a new version. You might first delete the application completely from the runtime area leaving the server to undeploy it. Then you replace the object with a new version of itself. The window of time between checks of the watch files is finite and I’ve found it is possible to remove and replace the archive within that window so that the JBoss server does not detect that the watch file was missing and so it is not unloaded from memory. The server does check the watch file timestamps, but if you have changed files other than the watch files and have not updated the timestamps of the watch files themselves, the server will happily ignore the new version of the archive while running the old one.

If you use this deployment strategy, then this issue is essentially a random process, and a deployment failure due to this reason happened in our case on only a few percent of all deployments. When you are running a few hundred deployments a week, or it happens for a production deployment it becomes a big problem – especially when people don’t know what the problem is. A simple resolution is to always update the timestamps of the watch files when changing anything for a deployed application. This will take care of everything but possibly compiled JSP’s. (Possibly more on that later.)

This also points to a “restart” mechanism for JBoss – simply ‘touch’ the watch files of a running application to change their timestamps to the current time. This will trigger the dump-and-reload on the next watch file check. This can be useful when the application has not changed, but an associated XML resource has.

When you work with a locking-type version control tool like CA Harvest, your Meister build project will appear in your Eclipse workspace as read-only when you check out an existing workspace. I’ve been using Eclipse for WebSphere development (WebSphere Studio Application Developer) and for JBoss via MyEclipse IDE. If you want to regenerate your Java targets, you first have to check out the Meister build project so that the files are writable.

Since this can lock the targets exclusively and prevent others from updating the target, you may not want to check out the build project, but you may still want to develop freely and update your local targets for Meister to build it. For this situation I recommend creating a separate build project that you may never check in to version control. It will be writable and it allows you great freedom for a maximally agile development environment. The ‘official’ build project may reference all the built archives in the workspace, but having your own local build project can allow you to focus for a unit build. For example, my workspace may contain an EAR project, a WAR project and one or more JAR projects. If I am principally working only on one of the JAR projects, my local build project can reference only that one JAR project.

When it’s time to release your JAR code updates to the system build and test environments, synchronize your workspace and check out the VC build project. Generate your targets, do a local system build and then check everything in. Your team system build will work fine!

I wanted to share a specific benefit I enjoyed while using Meister for Java development. As part of my role to help develop an automated JBoss build and deploy system, I ended up taking on a developer role for a web services security project for both JBoss and WebSphere. While the project involved about 1000 lines of Perl, it also got me writing simple web services and consumers for JBoss and WebSphere and building them using Meister and its Eclipse plug-in.

Believe it or not, I am still using WebSphere Studio Application Developer 5.1. While my specific tale involves that IDE, it is equally applicable to MyEclipse and Rational Application Developer set of Eclipse IDE’s. In my environment, CA Harvest is the version control/SCM tool and Meister is the build tool. After code is checked in from my desktop using the CA Harvest eclipse plug-in, the code is replicated out to a Linux server, where Meister performs the official system build that is sanctioned for deployment to the application server. There is also a Meister Eclipse plug-in that scans the WSAD workspace for build targets and dependencies. Meister stores this information in one XML file per build target and those files are also checked in to CA Harvest right along side the source code.

Working intensely within the WSAD Eclipse environment as the project manager cracked the whip, I worked with a consumer application and updated it according to the changes in the service WSDL and service endpoint URL’s. One thing I learned is that if one of the parameters for the consumer is tweaked, don’t bother tweaking the XML or generated code, just regenerate the whole client. WSAD will even check out the files before if they need to be. So everything looked good on my desktop with the service and consumer deployed to two separate WebSphere servers on ports 9080 and 9081. Now to get it into the enterprise ‘dev’ environment…

Using the ‘Generate Target Definitions’ feature of the Meister plug-in I updated the Meister build target XML definition files and checked in all my code. I then promoted the code in CA Harvest which automatically kicked off a ‘dev’ build in the Linux environment. I got an error back from Meister saying ‘jdmpview.jar’ doesn’t exist.

Since I knew my consumer app and its elementary nature, I knew that jdmpview.jar wasn’t one of my JAR’s and it must be one of WebSphere’s. Given that 200 other Java apps use the same build environment with the same standards, I probably didn’t use some new feature of WebSphere that no one else is using. Therefore, it must a problem on my local desktop with the version of JVM I was using.

Sure enough, the consumer app was using the base_v51 WebSphere runtime instead of the ee_v51. (I did inherit the initial version of the app from someone else!) And, oddly enough, there is an extra JAR in the base that is missing in the more fully featured Enterprise Edition. Meister correctly forced the runtime environment to be EE for the Linux build, overriding the developer selection. I switched the runtime in the Java build path properties, regenerated the Meister target definitions, checked them in and promoted them to a successful ‘dev’ build. Regenerating the target definitions had the effective of switching out the list of JAR files in the library path from the base_v51 set to the ee_v51 set. The whole thing including one bad and one good build took about 4 minutes.

The great benefit for me was the balance between developer and SCM functions. We could have applied more controls at the desktop level, but from my perspective, I prefer an Agile environment with more freedom even if it means occasionally hanging myself with my own rope. In this scenario I let the tools dot the I’s and cross the T’s and it took no more time than say, waiting for Outlook over VPN.

In developing Java applications for multiple server environments (e.g. dev, test and prod) there is a common pain-point of having to manage deployment descriptor or configuration files specific to each server. For example, you may have an XML log4j configuration file with some parameters different for different server environments. You may want to turn on debug messaging for the development server, but turn it off for production. At the same time, the Java source code will (eventually) be the same in production as it was in development. A similar situation applies for .NET application development.

Like many build management tasks, managing these environment-specific files is generally left to either manual or some type of scripting. This is really something that needs to have a high level of automation applied. Particularly in larger environments, much like scripted build management solutions, existing tactics fall short. This situation is in a far worse state than even the compile part of build management. It is not enough to simply have a script that can spit out some files. One of the biggest problems is information management and the fact that parameter values in the configuration files may be determined by different teams! How do a production engineering team and an application developer both feed inputs into the same XML file?

I’ve worked on this problem for several years and with a number of companies. The critical functionality can be broken down into two different items – information management and a processing engine. In an effort come up with something better, I’ve done a review of what’s out there and here is what I came up with:

• Ant ‘filter‘ task: As with many Ant tasks, this works great if you are an individual with a few items that need updating. It is a nightmare if you are working in a multi-team enterprise with multiple server environments. The main problem is that you have to constantly take working copies of XML files and insert a token for Ant to later re-replace. This leads to a management nightmare to synchronize parameterized copies of XML files with their working copies from the desktop environment. The advantage is that it works for any file type so you can use it for properties files as well as XML files.
• OOPS Consultancy Ant ‘xmltask‘: This is a good engine for specifying and performing changes to the XML and has a full feature set. In fact, we use this in some of the Meister build services. The problem is that it is only for Ant and therefore you have all the reuse, standardization and hard coding issues. Xmltask can provide part of the solution we are looking for, but we still have an information management problem to deal with.
• Maven: Maven has what is essentially the Ant filter task. The specifications are abstracted in the pom files, which is better than Ant, but it encourages templating of configuration files leading to all the problems associated with that (synchronizing templates with working files, testing templates, etc.)
• XML:DB XUpdate: This is a working draft of a specification to encode XML update instructions into an XML document. There is a Java implementation of XUpdate listed on the site called ‘Lexus’, but I couldn’t find anything on it. Since the build management task requires us to generate XML files, I’m not keen on generating XML files using xupdate tags that will allow me to generate other XML files.
• Perl XML::Twig: This has worked wonderfully for me on a back-end web services security effort and I could not be more happy with such a precise, elegant and brief XML library, which includes XPath. This is not a solution for Java or .NET developers, but it could serve as an engine to mimic xmltask or implement the XUpdate specification.
• Excel. Yes, I’ve seen Excel used effectively as the information management front-end to updating the XML. It is a convenient format to share among teams, it is centralized source of information, it can be checked into version control and it can be saved as an XML file itself for processing by another engine. In a large environment, you may have 5 or more server environments, lots of different components to configure, so you could have literally hundreds of parameters to manage. Excel gives you a nicely transparent way to view those values.

I’ve found the multi-threaded capabilities of Mojo and Meister workflows to be very valuable for builds and deployment. The chief benefit I’ve received is in saving time as you might expect. I’ve been working with a workflow that deploys a Java application to up to 24 servers. Let’s ignore the sequential part of the workflow and examine the time difference of running parallel deployments versus one where each of the 24 machines is updated in sequence. The deployment process takes about 5 seconds per machine. Sequentially, that’s 24 x 5 seconds, or 2 minutes. In parallel, well it’s not quite 5 seconds, but closer to about 20 seconds because of limitations of the Linux machine it is running on. Still, that’s a tremendous 100 second savings.

In addition to using the parallel workflow to cater to impatience and improve productivity, I want the Java application to hit all of the servers in the cluster close to the same time. In this particular strategy, only 3 machines out of the 24 are in the cluster. The rest are to support dynamic resource allocation and disaster recovery. Running the deploys in parallel allows me to hit all machines, and therefore all the machines in a cluster at close to the same time without having to figure out some ordering so that the cluster servers are hit first and then the rest. This ends up saving a lot of coding, testing and possibly debugging. Great stuff.

There are times in build management that you need to encrypt something – often a password. In the last blog, I gave an overview of the encryption process. Now, I’ll show how you can accomplish something.

Besides just having an encryption algorithm, there are a number of important details to be minded: key, block management algorithm, initialization vector, binary-to-text encoding. Here is what I ended up doing. The encrypted text ended up in the text field of an element in XML and it was successfully decrypted on the other end in pure Java.

First, you need your basic cipher We’ll use the Rijndael algorithm specified by AES. I used a 128-bit key generated with help from the Crypt::Random module:

use Crypt::Rijndael;

my $base_cipher = Crypt::Rijndael->new(
 $key,
 Crypt::Rijndael::MODE_CBC( )
);

Next you use this cipher within a block algorithm:

use Crypt::CBC;  #--Cipher block chaining

my $block_cipher = Crypt::CBC->new(
  -cipher => $base_cipher,
  -header => 'none',
  -iv    =>    $iv,
  -padding => 'space'
);

Even though the initialization vector, $iv, does not need to be secret, I enjoyed making it “randomy” with the ultra-cool Data::Random module. Also note that the padding strategy, adding spaces, is not binary safe so it works for encrypting text, but not for binary format files. Now you just encrypt:

my $encrypted_raw_binary =
               $block_cipher->encrypt( $plain_text );

use MIME::Base64;

my $encrypted_text_string  = encode_base64(
  $encrypted_raw_binary,
  ''
);

#-- empty 2nd arg means “don’t break up long lines”

The last step is necessary to give you something you can easily manipulate as a string to read and write into files.

So that’s it. To decrypt you just do the reverse.

I wanted to pass along what I learned about a new area for me: encryption. I’m working on a build management project for securing Java web services and I’ve enjoyed learning about encryption methods. There are a couple of key concepts to learn and Wikipedia has some informative and entertaining pages. I recommend the “The Code Book” by Simon Singh for a great history of the subject.

One concept strange to newbies is that the encryption algorithm should be widely known and public. The key (**ahem**) is that the encryption key remains secret. If there is a problem discovered with the algorithm, you want to be the first to know. The commonly used algorithms are so robust that there is little advantage to be gained by understanding how they work, as long as the encryption key remains secret.

The U.S. government held a competition for an encryption algorithm to be the Advanced Encryption Standard (AES). The algorithm chosen for this is called Rijndael and it replaced the Data Encryption Algorithm (DEA) of the Data Encryption Standard (DES). The triple form of DEA is still commonly used and is incorrectly but widely known as Triple DES. So yes, the encryption algorithm for the U.S. government’s most top secret data is widely known.

AES specifies not only that Rijndael be used, but that it be used with a 128-bit key. Rijndael also encrypts only 16 bytes. What?! Yes, so basically you have to chop up your message into 16 byte blocks and encrypt each one separately.

If you were encrypting a long message or a lot of messages, you would be encrypting similar words over and over and a lot of your 16 byte blocks might look similar or even identical. This makes you susceptible to a form of frequency analysis attack (see “The Code Book”). So, another algorithm is tacked on to obfuscate the 16 byte blocks after encryption. A commonly used block algorithm, Cipher Block Chaining (CBC), makes the text of a block depend on the encrypted text of the preceding block as well as its own encrypted value. The first block in the message is seeded with an initialization vector (starter value of 16 bytes of text) that interestingly does NOT need to be secret. That doesn’t quite make sense to me, but I trust the experts.

If your message is not exactly a multiple of 16 bytes, you will have to pad it with something that you agree on with the decrypter. The padding characters have implications for what is “binary safe” so be careful. (See Crypt::CBC for a great rundown of commonly used padding techniques.)

The last thing you need to know is that when you encrypt your blocks and obfuscate with a good block algorithm, you end up with raw binary data. I certainly don’t recommend it for XML. This encrypted data, however, is commonly encoded using the MIME format Base 64. This, from the early use.net days, converts raw binary into alphanumeric characters plus ‘-’, ‘=’ and ‘/’. And, yes, that’s 65 characters. You will also need to decide if you will break up the lines with a carriage return after so many characters or not.

So, to get your encrypted value into XML, you can 1) choose a known encryption algorithm, 2) generate a key, 3) use a block management algorithm, 4) decide how to pad your last block, 5) generate an initialization vector (for CBC), and 6) convert it to Base 64 for suitability for text files. To decrypt, do the reverse. Enjoy!

OK, this topic might be a snoozer, but if we’re going to do build management for our RDBMS (Oracle, SQL Server, etc.) in a revolutionary new way, we need to compare what’s going on with database source code changes and builds and compare that with what we already know.

We said that when we make a runtime change to a database, we are only applying changes on top of what we already have, but in J2EE for Java, we replace the entire running application with another instance of the entire application. This is not an incremental deployment in any sense.

If we compare with the case for C/C++, our source code change might result in replacing one of the application’s executables with a new one. OK, this is a more incremental, but maybe I changed one C source file, resulting in one object file change. I still have to rebuild a new executable with possibly many additional unchanged object files.

For both Java and C, traditional build management technologies allow for incremental builds. That means, if I only change a subset of the source code, a build can be done that re-compiles the minimum number of files, taking into account the full impact of each file change. (Many applications have lost the ability to do incremental builds, but Meister can get it back.) So, for Java and C, there should be the ability to do an incremental build, but when that build step is complete, the runtime environment remains unchanged. A separate deployment step needs to happen which is less incremental to some degree.

For the database changes, there is only a single step combining both build and “deployment” and it is always incremental. When you do the build, the runtime environment is changed immediately. So, as I mentioned earlier, there are differences in build management for RDBMS’s and operating system/JVM applications. Again, let that not deter us from bringing those changes under the umbrella of a common build management system.

I’ll also be attending Ignite - Chicago. This is not the music festival, but a series of geeky 5 minute talks with about 20 slides. This is just my style, the length of the talk matching the length of my patience. My talk is entitled “Controlling Java with Perl”, and I’ll be covering many of the same topics related to build management I cover in my blog. I see O’Reilly Publishing’s name on the web page, so maybe they have something to do with it. Already there are 98 people planning to attend. Like the Windy City Hackathon, if you are planning to attend, please drop me a line – I’ll buy you a beer.

Ivy seems to be trying to resurrect Ant, when it should stay dead in my opinion (unless you are a lone developer on the simplest project). Maven is a more proper build tool with some workflow functionality that falls short outside of a small team of developers with no audit or control requirements. Both are vying for prominence as the open source build tool, yet trying to settle into compatible niches like two brothers hopeful for the throne.

Here is what Ivy says about Maven: http://ant.apache.org/ivy/m2comparison.html

And here is Maven’s side of the story: http://docs.codehaus.org/display/MAVEN/Feature+Comparisons

I am working on a project to secure Java web services running under JBoss and WebSphere application servers. As part of this effort, I’ve created some ‘Hello World’ type web services and consumers to use for the team’s integration testing and implementation verification. This has allowed me to launch two identical projects: one using MyEclipse targeting the JBoss application server and the other using WebSphere Studio Application Developer 5.1 (WSAD) targeting the WebSphere 5 application server environment. By doing these two Java development projects side by side, I’ve been able to compare the two development environments on equal grounds for one very specific type of development.

Initial Setup of Eclipse Projects and Local Servers

Furthermore, the applications for JBoss used the XFire SOAP implementation for web services. This is an external component, but MyEclipse supports this and has a simple wizard to generate a web service. I found resources on the web and within MyEclipse that helped me struggle through writing my first web service and I learned a lot. It was harder to learn how to do get it to work in the MyEclipse environment than it was to learn how it to configure it, but I appreciate the features in MyEclipse. MyEclipse’s support for JBoss is also nice and it is really easy to integrate with a JBoss server, which must be installed and configured separately. One-click server start/stop and one-click deploy from your Eclipse project was really nice.

Setting up the WebSphere applications was a different experience as everything was done within the WSAD IDE. You used wizards (Eclipse plug-ins) to generate both web services and clients, and when you are done, you also get JSP’s and Java classes that allow you to test them. I admit I had some more experienced help in setting this environment up, but it was real point-and-click. And, so was the server configuration. Everything about the server is in a ’server’ Eclipse project and you can control parameters within WSAD. In fact, it was so easy that we decided to run two WebSphere servers listening on different ports and deploy the service to one and the consumer to the other. To do the same for JBoss, you have to define a separate server using the configuration files and MyEclipse will happily support it, but I don’t really have the time to learn about details of JBoss server administration to do that. WSAD wins there, but everything else in setting up the applications was pretty equivalent.

Iterative Development

Going forward to create the applications I wanted, with naming standards and just the right Hello World message, I first had to contend with port conflicts between the JBoss server and the Windows operating system. It seems that a lot of the port numbers for JBoss are around the 1000’s and that’s just where Windows starts giving away port numbers. This required us to learn how to reserve ports by manually modifying the registry. OK, maybe JBoss is not to blame there.

Finally we get JBoss working properly and the second annoying thing arises. After deploying your web application to the JBoss server (a nice one-click that physically copies files from the Eclipse workspace to the deploy folder on the JBoss server) it works fine, but redeploying or removing the application is problematic. The Windows operating system again intrudes and locks at least some JAR files. In order to redeploy your entire application, you have to stop the JBoss server, redeploy and restart the server. Restarting takes awhile. If you forget to shut down the server first, you will get a nasty gram and a half-deployed application. To be fair, files are copied to the server when you save them in your workspace preventing the need to constantly redeploy, but starting up the project required a lot of redeploying and it was very tedious.

Here is where IBM rockets ahead. With two WebSphere servers running on different ports, it was pretty simple to associate different projects with the different servers for one-click everything. Because of the inevitable mis-starts (like using the wrong JDK version), we had to redefine and redeploy the applications a number of times at the beginning of the project. This procedure was definitely less painful than the JBoss experience and we had more capabilities in the end with two servers. Also, the server configurations are stored in an Eclipse project, which are shared via version control to other developers. One thing that helps architecturally is that the WebSphere servers use your actual Java Eclipse project folders as the deployed application area. This prevents the need to copy files around. There is also a nice server management view that allows you to look at the different logs and perform server start/stops.

Summary

I like the more lightweight MyEclipse/JBoss combination that seems to be more simple and understandable, and also gives me the freedom to screw up my own project. The painful redeployment process and the external configuration of JBoss were drawbacks. For a multi-developer environment, a number of standards would need to be defined to get everyone working on the same page. IBM’s WSAD did most of the configuration for me for the servers and it also generated web services applications that were complete and robust, but it was impossible to understand the inner workings (for me). Changing any of the major parameters of your web service required you to regenerate it completely, but it worked great and it was easy. I also ended up with more capabilities by having two servers. The version of WSAD I used, 5.1, is quite old, but I will soon be able to go through the same exercise with Rational Application Developer for WebSphere Software version 7. I can’t wait!

As an open source product (”owned” by Red Hat) JBoss is a robust application server. From a pure product purchase standpoint, it is clearly much less expensive than having WebSphere development and runtime environments. So why not switch?

I’ve been working with both and there is a clear difference in development and implementation philosophies. IBM’s Rational Application Developer for WebSphere Software is the development IDE IBM sell’s for developers to write applications for the WebSphere runtime environment. It is regarded by most as expensive, but it also comes with a full set of features from security to a bundled web services implementation and access to IBM’s extensive business component library. Many features are implemented through wizards using code generator plug-ins to the Eclipse platform. The generated code can be complex, but it is robust and works but not necessarily designed to be tinkered with.

The WebSphere application server likewise has a full set of features including an automated deployment manager and extensive administrative tools. It even includes a stripped down version of Eclipse to run some of the same code generators found in the Rational Application Developer products.

JBoss, on the other hand, has no real interactive development environment, but MyEclipse offers one choice that has a lot of support for JBoss and is very reasonable priced. While MyEclipse has some excellent features supporting development for JBoss, there are many choices of components available to use for things such as web services that require much more hand coding and configuration relative to IBM. IBM gives you a few options for doing things and a few ways of doing things providing a very structured and highly standardized environment. With JBoss, any standards-oriented company will have to regular meet to define standards around such things as security, resources and web service configuration.

Standards are important to support automation for things such as build and deploy and companies weighing JBoss and WebSphere as choices for their application server will have to weigh the additional man hours required to support JBoss against the features and price of WebSphere.

I’m here in Izmir (formerly Smyrna), Republic of Turkey, and this is one of the regions that is home to the birthplace of Western philosophy. On a previous trip, I visited Assos - the site of Aristotle’s first school. Now, my sister-in-law is studying philosophy at Aegean University and is a big fan of Kant. Last year, I had helped her with understanding some of the advanced English usage in her philosophy text books and picked up some basic understanding of Kant and Leibniz and how they decide and define what any particular object is.

Meanwhile, at OM Software and as part of my consulting in Madison, WI, we are thinking of the new possibilities that configuration management database products offer. We were thinking about how to track deployments of Java applications and how “deployment” meant different things to different groups. To the SCM team, a deployment was considered successful when the files were transferred to the runtime environments and in the context of JBoss, loaded by the application server. To the application development team, a successful deployment meant the planned changes were working as expected. Two different perspectives on the same thing. So, if we are to track Java application deployments, the company has to agree on what a deployment is by agreeing on what attributes it has and will be tracked in the CMDB. Since this will undoubtedly be a social process involving perception by different groups in the company (and not just a set of cut and dried technical parameters) this kind of decision making hearkened back to my introduction to Kant and Leibniz.

So,  that begs the question: Is there a practical application of philosophy that can help more effectively implement a  CMDB? I’ve proposed to her that we explore co-authoring an academic paper on the subject. Hopefully I am not being too cynical when I suggest that finding any modern, practical application of philosophy will make waves in academia…

I heard this from some developers today who confirmed the existence, by indirect evidence, of willful tampering of Java archives by gremlins using Ant. There is speculation that they live inside their own plug-in inside WebSphere Studio Application Developer.

Developer A, we’ll call him Nick (because that’s his name), came into our room to work out some tweaks with Openmake targets. He told us all that they’d been working on some problems with runtime errors getting the dreaded Major-Minor version error: “Unsupported major.minor version 49.0.” This usually indicates that you have compiled something with a 1.5 JDK but are running it under a 1.4 JVM.

The archive they were deploying was created wholly inside a workspace managed by WebSphere Studio Application Developer (WSAD) but built by Ant using IBM JDK 1.4.2. Now, WSAD by default launches under a 1.3 JVM, so there is no hint of a simple conflict of JVM’s. They actually traced the issue down to a few class files in the deployed archive that had a 49.0 class versions by using a byte editor. So only a subset of the classes compiled by Ant were affected! But, if they created their own Jar file at the command line through the Windows command prompt using the same exact files, the Major-Minor problem didn’t appear. This means the classes were altered at the packaging step - NOT the compile step!

Harking back to my pain with multiple versions of Java over the last decade, I interrupted this fine tale and asked with a haughty tone “Is a 1.5 JDK installed anywhere on the machine?”

“Yes”, he answered, and the problem, he said, was resolved by uninstalling the 1.5 JDK. We all then had a nice knowing chuckle. Then we questioned Sun’s use of the registry in Java and how the use of the registry changed with each release, forgetting, in fact that the problem happened under an IBM JDK. We also talked about the practice of dumping of java.exe under the windows/system32 (%SYSTEM_ROOT%) directory and how that could trip you up. You never knew if it was a Microsoft java.exe or a Sun one, and if it was Sun, which one was it?

Has anyone heard of this happening before? Is it a bug with Ant? Meister uses Ant under the covers for some things, but not for the Java compile. We’ve never heard of this problem before. We are, in fact, working with Nick to roll out Meister (and the Java build services used by thousands) to Eclipse at the desktop so mysterious problems like this won’t happen in the future.

Since there was some interest in this topic, I’m diving deeper and showing some code snippets.

As I mentioned before, I’ve found the Archive::Zip and XML::Twig modules invaluable in validating Java deployment descriptors. Post-build validation is important to ensure company standards are adhered to and prevent mishaps, such as deploying an app with a context root already used by another app, or one with no context root defined. It is far easier to fail prior to attempting deployment than to allow deployment to continue, waste valuable time and resources and possibly collide or upset the jvm, costing more time for more people.

I’ve pared down the code I’ve used to the bare essentials for just taking an archive, finding an XML file and extracting it to memory. Next time I’ll get to the XML parsing bit with XML::Twig. Enjoy