JBoss checks for certain watch files when handling deploying or undeploying an application. The watch files are certain key files germane to the object you are deploying. For an EAR, the watch file is the application.xml and the optional jboss-app.xml files. For a web application archive, the watch files are the web.xml and jboss-web.xml files. For single-file XML resources, such as datasources, the watch file is the XML file itself. In this article, I am dealing with archives that are deployed in unextracted (unzipped) form.

The first check is made for the existence or non-existence of a watch file. If a previously unknown watch file is found, the appropriate deployer is started and the file modification timestamp is stored in memory. If a known watch file is found to be missing, the appropriate undeployer is launched.

If a known watch file is found on a subsequent pass of checking watch files, its timestamp is checked against the time that was stored in memory by the deploy process. If the deployed watch file is newer, the appropriate deployer is launched which apparently first dumps the associated resources and then reloads the object as if it were newly found.

This leaves a hole that can lead to the horrifying result of having files deployed to the server, but not having the changes reflected in the running application.

The issue has to do with completely replacing a running application with a new version. You might first delete the application completely from the runtime area leaving the server to undeploy it. Then you replace the object with a new version of itself. The window of time between checks of the watch files is finite and I’ve found it is possible to remove and replace the archive within that window so that the JBoss server does not detect that the watch file was missing and so it is not unloaded from memory. The server does check the watch file timestamps, but if you have changed files other than the watch files and have not updated the timestamps of the watch files themselves, the server will happily ignore the new version of the archive while running the old one.

If you use this deployment strategy, then this issue is essentially a random process, and a deployment failure due to this reason happened in our case on only a few percent of all deployments. When you are running a few hundred deployments a week, or it happens for a production deployment it becomes a big problem – especially when people don’t know what the problem is. A simple resolution is to always update the timestamps of the watch files when changing anything for a deployed application. This will take care of everything but possibly compiled JSP’s. (Possibly more on that later.)

This also points to a “restart” mechanism for JBoss – simply ‘touch’ the watch files of a running application to change their timestamps to the current time. This will trigger the dump-and-reload on the next watch file check. This can be useful when the application has not changed, but an associated XML resource has.

I wanted to share a specific benefit I enjoyed while using Meister for Java development. As part of my role to help develop an automated JBoss build and deploy system, I ended up taking on a developer role for a web services security project for both JBoss and WebSphere. While the project involved about 1000 lines of Perl, it also got me writing simple web services and consumers for JBoss and WebSphere and building them using Meister and its Eclipse plug-in.

Believe it or not, I am still using WebSphere Studio Application Developer 5.1. While my specific tale involves that IDE, it is equally applicable to MyEclipse and Rational Application Developer set of Eclipse IDE’s. In my environment, CA Harvest is the version control/SCM tool and Meister is the build tool. After code is checked in from my desktop using the CA Harvest eclipse plug-in, the code is replicated out to a Linux server, where Meister performs the official system build that is sanctioned for deployment to the application server. There is also a Meister Eclipse plug-in that scans the WSAD workspace for build targets and dependencies. Meister stores this information in one XML file per build target and those files are also checked in to CA Harvest right along side the source code.

Working intensely within the WSAD Eclipse environment as the project manager cracked the whip, I worked with a consumer application and updated it according to the changes in the service WSDL and service endpoint URL’s. One thing I learned is that if one of the parameters for the consumer is tweaked, don’t bother tweaking the XML or generated code, just regenerate the whole client. WSAD will even check out the files before if they need to be. So everything looked good on my desktop with the service and consumer deployed to two separate WebSphere servers on ports 9080 and 9081. Now to get it into the enterprise ‘dev’ environment…

Using the ‘Generate Target Definitions’ feature of the Meister plug-in I updated the Meister build target XML definition files and checked in all my code. I then promoted the code in CA Harvest which automatically kicked off a ‘dev’ build in the Linux environment. I got an error back from Meister saying ‘jdmpview.jar’ doesn’t exist.

Since I knew my consumer app and its elementary nature, I knew that jdmpview.jar wasn’t one of my JAR’s and it must be one of WebSphere’s. Given that 200 other Java apps use the same build environment with the same standards, I probably didn’t use some new feature of WebSphere that no one else is using. Therefore, it must a problem on my local desktop with the version of JVM I was using.

Sure enough, the consumer app was using the base_v51 WebSphere runtime instead of the ee_v51. (I did inherit the initial version of the app from someone else!) And, oddly enough, there is an extra JAR in the base that is missing in the more fully featured Enterprise Edition. Meister correctly forced the runtime environment to be EE for the Linux build, overriding the developer selection. I switched the runtime in the Java build path properties, regenerated the Meister target definitions, checked them in and promoted them to a successful ‘dev’ build. Regenerating the target definitions had the effective of switching out the list of JAR files in the library path from the base_v51 set to the ee_v51 set. The whole thing including one bad and one good build took about 4 minutes.

The great benefit for me was the balance between developer and SCM functions. We could have applied more controls at the desktop level, but from my perspective, I prefer an Agile environment with more freedom even if it means occasionally hanging myself with my own rope. In this scenario I let the tools dot the I’s and cross the T’s and it took no more time than say, waiting for Outlook over VPN.

In developing Java applications for multiple server environments (e.g. dev, test and prod) there is a common pain-point of having to manage deployment descriptor or configuration files specific to each server. For example, you may have an XML log4j configuration file with some parameters different for different server environments. You may want to turn on debug messaging for the development server, but turn it off for production. At the same time, the Java source code will (eventually) be the same in production as it was in development. A similar situation applies for .NET application development.

Like many build management tasks, managing these environment-specific files is generally left to either manual or some type of scripting. This is really something that needs to have a high level of automation applied. Particularly in larger environments, much like scripted build management solutions, existing tactics fall short. This situation is in a far worse state than even the compile part of build management. It is not enough to simply have a script that can spit out some files. One of the biggest problems is information management and the fact that parameter values in the configuration files may be determined by different teams! How do a production engineering team and an application developer both feed inputs into the same XML file?

I’ve worked on this problem for several years and with a number of companies. The critical functionality can be broken down into two different items – information management and a processing engine. In an effort come up with something better, I’ve done a review of what’s out there and here is what I came up with:

• Ant ‘filter‘ task: As with many Ant tasks, this works great if you are an individual with a few items that need updating. It is a nightmare if you are working in a multi-team enterprise with multiple server environments. The main problem is that you have to constantly take working copies of XML files and insert a token for Ant to later re-replace. This leads to a management nightmare to synchronize parameterized copies of XML files with their working copies from the desktop environment. The advantage is that it works for any file type so you can use it for properties files as well as XML files.
• OOPS Consultancy Ant ‘xmltask‘: This is a good engine for specifying and performing changes to the XML and has a full feature set. In fact, we use this in some of the Meister build services. The problem is that it is only for Ant and therefore you have all the reuse, standardization and hard coding issues. Xmltask can provide part of the solution we are looking for, but we still have an information management problem to deal with.
• Maven: Maven has what is essentially the Ant filter task. The specifications are abstracted in the pom files, which is better than Ant, but it encourages templating of configuration files leading to all the problems associated with that (synchronizing templates with working files, testing templates, etc.)
• XML:DB XUpdate: This is a working draft of a specification to encode XML update instructions into an XML document. There is a Java implementation of XUpdate listed on the site called ‘Lexus’, but I couldn’t find anything on it. Since the build management task requires us to generate XML files, I’m not keen on generating XML files using xupdate tags that will allow me to generate other XML files.
• Perl XML::Twig: This has worked wonderfully for me on a back-end web services security effort and I could not be more happy with such a precise, elegant and brief XML library, which includes XPath. This is not a solution for Java or .NET developers, but it could serve as an engine to mimic xmltask or implement the XUpdate specification.
• Excel. Yes, I’ve seen Excel used effectively as the information management front-end to updating the XML. It is a convenient format to share among teams, it is centralized source of information, it can be checked into version control and it can be saved as an XML file itself for processing by another engine. In a large environment, you may have 5 or more server environments, lots of different components to configure, so you could have literally hundreds of parameters to manage. Excel gives you a nicely transparent way to view those values.

I’ve found the multi-threaded capabilities of Mojo and Meister workflows to be very valuable for builds and deployment. The chief benefit I’ve received is in saving time as you might expect. I’ve been working with a workflow that deploys a Java application to up to 24 servers. Let’s ignore the sequential part of the workflow and examine the time difference of running parallel deployments versus one where each of the 24 machines is updated in sequence. The deployment process takes about 5 seconds per machine. Sequentially, that’s 24 x 5 seconds, or 2 minutes. In parallel, well it’s not quite 5 seconds, but closer to about 20 seconds because of limitations of the Linux machine it is running on. Still, that’s a tremendous 100 second savings.

In addition to using the parallel workflow to cater to impatience and improve productivity, I want the Java application to hit all of the servers in the cluster close to the same time. In this particular strategy, only 3 machines out of the 24 are in the cluster. The rest are to support dynamic resource allocation and disaster recovery. Running the deploys in parallel allows me to hit all machines, and therefore all the machines in a cluster at close to the same time without having to figure out some ordering so that the cluster servers are hit first and then the rest. This ends up saving a lot of coding, testing and possibly debugging. Great stuff.

We are building a Wiki of development, build and release terms and as I contribute starter definitions, I am blogging them. Feel free to comment, or wait for the Wiki. Release management is one of the more challenging parts of application development, largely because it is more social than other aspects. Particularly in large organizations, more teams must come together to coordinate a release. Release management includes all the activities surrounding application production changes. It may involve ensuring requirements and change requests are met, reporting and coordination of multiple application teams to test and simultaneously change production. It is often the latter coordination that is the most difficult as it is a coordination and leadership effort requiring consensus as opposed to something entirely electronic or technical.

Release management in the Perl open source community: There is a version naming convention in the Perl community where each contributor of a Perl module cites the versions of other Perl modules it is dependent on. The automated install program that comes with Perl (cpan for CPAN distributions and ppm for ActivePerl) cross checks the user’s version of dependent Perl modules and updates them if necessary. Thus when one installs  a new perl module, they may automatically get updates to a number of existing Perl modules to satisfy the release requirements.

Release management in the Java open source community: There is none, in general. Eclipse has coordinates the simultaneous release of multiple projects in the Europe release. For the wider community, only the Maven Apache project has attempted to record interdependencies between open source Java projects. Maven has built in automation to walk the dependency lists in a manner similar to Perl by reading the dependency lists on http://ibiblio.org/maven

In the first part of this series, I gave an overview of what this series is about. I want to step back and describe the problem I have with part of my development process that is so conveniently resolved by a Mojo/Meister workflow using perl scripts. This results in a continuous integration for perl development.

My environment currently is a build and deployment system that uses Meister for build and workflow management, CA Harvest for version control and high level application lifecycle workflow management and about 100 perl scripts used for deploying all sorts of Java applications and performing validations of various sorts. This system supports several hundred Java applications. Because of the large numbers of applications, automation can only be achieved by adhering to standards and naming conventions, much to the dismay of developers.

I was planning to dive in with the deployment and validation scripts, but I’ve been slowly building up a change control and testing system for the perl scripts themselves and I was confronted with a more simple problem that is perfect for starters in this series. I’ve been improving the testing of the perl scripts by implementing unit test scripts using the Test::More perl module and relatives. We have three environments for perl development, testing and runtime. I’ll call them dev, qa and prod, for short.

CA Harvest is a big database storing every version ever created in the company. Although it is mostly used for Java code, we the build team are using it to manage our perl scripts. At each state in the lifecycle: dev, qa and prod, there is an associated file sytem area that is synchronized with what is in the database. This synchornization is done via a perl script so that when we ‘promote’ code in Harvest from dev to qa, the qa file system area is synchronzied with the qa view in Harvest of our source code. (One of the nice things about Harvest is its ability to trigger back end automation of your favorite script on certain actions, like ‘promote’.) My goal is to trigger the automated testing of those perl scripts after synchronization with a script using Test::Harness.

All perl scripts executed by Harvest are executed on the Harvest server itself. The first problem I had is that some of our perl scripts are executed on the Harvest server, and another set is executed on the physically different build server. I needed to run two sets of perl scripts on two different machines. This is just for the qa environment. I also wanted to do post-production deployment implementation verification (testing what you just put into production) and there are currently two build servers, meaning the scripts need to run on two build machines plus the server. Since the build machines are geographically separate (different network connections) and they could have slightly different shell profiles, I don’t want to assume the test results would be the same.

The problem I have with Harvest is that Harvest will execute a series of post action scripts, but 1) it will execute them in series, 2) if any preceding step fails, it will not execute the following steps.

Currently my synchronization script actually fails, due to a few files that shouldn’t be overwritten. These files are configuration files that must be different between dev, qa and prod, and it is my TODO to clean that up. Rather than tweak the perl synchronization script to make it pass, I need to reorganize the files in Harvest - no small task. So I want they syncrhonization script to fail, but I still want the unit tests to run. That is problem A. Harvest will detect the failure in the synch and exit without running the tests.

The second problem is (B) I am extremely impatient, and why shouldn’t I simultaneously execute my perl tests on the two or three different machines instead of waiting for them to finish, one-by-one. The Meister workflow (same as Mojo) allows me to A) ignore the return code of the synchronization script and B) execute all of the unit tests in parallel on the different machines (via ssh if necessary). For gravy, I had the workflow wait until all the tests were complete on all the machines and then send an email with the HTML log URL.

So, I had Harvest call a single perl script that launched a Meister worflow, which in turn executed the synchronziation, ignored its return code, and the split into parallel unit tests, wait for all the unit tests to complete on all the machines and then fire off an email.

This really saved a lot of tedious repetition in my development process and I am excited about using it. We are likely to grow both the number of build machines and number of perl scripts as time goes on this system is nicely extensible.