How do you set up an automated build for EJB client JAR’s from the IBM Rational Software Delivery 7 development environment for WebSphere 6?

This question came up recently in my work for a major insurance company. When one extends the EJB client class, that is all a developer has to do as far as RAD 7 is concerned. When the developer deploys the JAR to the server, RAD 7 quietly generates stub source Java classes, compiles them and includes them in the JAR file.

An automated build in this context means that all the code the developer created in RAD 7 and checked into version control, is checked out of version control without RAD 7 and built exactly the way the developer intended. This is what OpenMake Meister is for.

One developer I was working with was concerned with how to generate those same source files in the automated build, which in his case was using OpenMake. He was familiar with how OpenMake uses the ejbdeploy command for building EAR’s with EJB server-side code and expected some equivalent for the EJB client.

Mercifully RAD 7 actually leaves the generated source files behind in the Eclipse project, in the standard source location. This means that we get the source code for free and there is really no need to regenerate it. All one has to do is check in the generated source to version control along with the developer coded source and build a normal JAR file in the automated build.

For the developer, this means:

1. Check in all the Java source code in the project. This is easy – better than picking and choosing which Java source to check in as the developer thought he had to do. This would be ultra-high risk for making a mistake and breaking the team’s automated build.
2. Assign the “Default Java JAR” configuration mapping for the EJB client project using the OpenMake Target Generator Eclipse plug-in.

A lesson to learn from this is that not all technologies or technology variants will have an impact to the build process. The developer was considering an idealist approach to reproduce every minute step of RAD 7, but the best solution was something practical and simple. Build management is part art and part dirty science. Having a “generate” step for the EJB client Java classes in the automated build only introduces an additional point of possible failure, and we build-meisters know we don’t need any more of those!

Here comes buildr: yet another Java build tool. Hopefully I, or one of my other cohorts will check this out in detail soon. But, with my experience working with all manner of build tools, with 100 companies and many more development teams, I can already make a few observations.

First of all, why another build tool for Java? I am occasionally told that Maven or Ant is a perfect tool, but clearly the people behind buildr don’t think so. The choice of JRuby as the vehicle for delivering this tool, I think is probably a good one. JRuby is a scripting language in the same vain as Perl, which is used by Meister.

Doing software builds is an ugly business involving lots of file and operating system interaction. This is not where Java shines, but scripting languages can. As long as operating systems are written in C and not Java, C-like tools will be better and faster at interacting with them. Plain Ruby itself is C-based, and JRuby no doubt inherits C-like operating traits. Calling out to a Java compiler from Perl or JRuby, though it has its own JVM, does not represent a significant overhead compared with the file system operations and the compilation/translation itself.

Both Maven and Ant are relatively difficult to extend compared most other build tools, and I’ll be buildr beats them here. If you have all the Maven plug-ins and Ant tasks you need, then good for you. If not, then you have to start developing in Java and it becomes too much of an investment to sink into a build system. It is much cheaper to extend in JRuby or Perl. My frequently cited example is the XMLBeans compile step in Meister, written in Perl, which is only 40 lines of real code. The Maven plug-in is 60 pages of Java code and no one can tell me really what it is doing (I asked on all the forums). Less code is usually more transparent, which is also good for build audits.

I am a little disappointed to see them try to placate the Maven and Ant users by promising it is a drop-in replacement for Maven and they have all the Ant tasks covered. Both tools have their drawbacks and I don’t want to see another tool with the same deficiencies. They should have the cajones (or coñejos) to apply all their resources to what they think is a better tool (with its own unique benefits and deficiencies). I imagine offering Ant task equivalents is pretty easy because of the ease of coding in JRuby compared with Java.

They also don’t mention who is supposed to use the tool. Is it for individuals, small development teams, the enterprise? Maven falls short because it is only appropriate for development teams and not for stable, controlled, enterprise builds. Ant is not even tool, but a means to create some tools for small teams. I don’t think Meister will fear buildr either.

Well, since buildr is only in incubation status with Apache, I’m not sure how much time I’ll be able to spend on it, but I am curious and I’ll let you know if I find out more.

The first rule for Bash/C/Korn shell scripts in a Perl program environment is to re-write them all in Perl. If your Perl environment has any sophistication, you will have common code, standardized logging (perhaps with Log::Log4perl), testing with Test::More, etc. and your shell scripts just can’t keep pace.

If you share the environment with any non-Perl applications, however, you will still have to deal with the environment profile(s). I also have some legacy shell scripts that we can’t justify converting to Perl unless they have another reason to change. (Don’t change tested code in my house ~~ head bobble + finger wave ~~, nuh-uh!)

There are two ways I know of that you can extend the benefits of your Perl implementation towards your legacy and profile shell scripts. The first is through Bahut’s excellent tip on embedding POD documentation in shell script. This solves my problem of generating HTML documentation from POD in Perl scripts and having upsetting holes where the shell scripts are. I also have some controls for the Perl scripts that run podchecker before committing to version control, which fails if no documentation is found. Now, I can extend this control to the shell scripts.

The second Perl tool you can extend is the testing functionality. I’ve found the functionality in Test::More to be useful for validating that the changes to the shell environment profiles are correct and do not introduce defects. Profiles can be notoriously tricky to change when they get fat and you have variables depending on other variables. Mostly the profiles in my case are used to set environment variables that control the version control and build system, and these can be easily validated in a test script called profiles.t via checks like:

ok( $ENV{CODE_ROOT} eq ‘/opt/code’, “CODE_ROOT set to ‘/opt/code’”);

You then just rattle off tests for all the variables that are set and you have a great way to validate that everything will still work after the profile change. For a legacy script, you may not be able to have a crack at the internals, but you can at least check the return code and maybe some external effect it has somewhere, such as a file timestamp change.

eval { `legacy_script.sh`};

ok( !$?, “legacy_script”); #– $? is zero if script executes successfully

Profile.t and any other test scripts used to test legacy shell code can be bundled with all the other Perl tests via Test::Harness for a single test suite that really tests everything shell and Perl.

As a follow up to my article on automating XML updates, I’d like to report that I did use Excel and Perl’s XML::Twig to successfully generate XML descriptors for my web service consumer, and it was a lot easier than I thought. I’m using XFire 1.2.6 web services stack running under JBoss and using MyEclipse IDE 5.0. I’m happy to say I went from blank spreadsheet and no plan to generated XML files from spreadsheet values in one and half hours. The implementation is of course expandable and reusable. This implementation should work for WebSphere and .NET as well.

I needed to create different configurations for my web application so that the service request went to different endpoints for different environments. The endpoint is at an enterprise service bus (ESB) and there is a different ESB for each environment. I need to have my ‘dev’ instance of the consumer hit the ‘dev’ instance of the ESB, the ‘qa’ instance of my web app hit the ‘qa’ instance of the ESB, etc. We’ve set up Meister to pick up the correct XML file for the target environment for the build of the WAR.

I started by setting up the spreadsheet as follows. I had an unnecessary column for Host indicating JBoss, but I hope to include WebSphere and maybe .NET as well some day. My web app actually connects to two services a.k.a. providers, so there is a column there. And, next is the configuration label for my web app with the name corresponding to the environment it is designed for. So, the first three columns of the spreadsheet look like:

Then I needed a way to indicate the resource that would change. Right now I only have XML files, but I chose to stick with a generic URL for that. Unlike Maven or Ant generators, we start with an XML file that actually works and has been tested - not some hacked up parameterized version that takes additional effort to create. The fourth column of the spreadsheet looks like the following (with repeated entries omitted):

Next, I needed a way to specify a target location to change within the XML file. Now, I know I’m going to use XPath, but I’ll want this to one day work for properties files as well, so I came up with a URL-like thing called a Universal Datum Locator (UDL) which pre-pends the method of locating the datum to change on to a method-specific locator. It could be a property name, an XPath or a Perl regex, for example. In this case it is XPath and then the last column contains the replacement value for the datum indicated by the UDL. XPath is also very intuitive and easier to construct than it may look.

The value for the UDL looks like:

xpath://beans/bean[@factory-bean=’xfireProxyFactory’]/ constructor-arg[@index=’1′]/value

So the fifth column contains the UDL’s, which in my case is always the same XPath expression. The final column of the spreadsheet contains the replacement value of the datum indicated by the UDL:

My nifty Perl script is only about 80 lines of real code and because XML::Twig is nearly the best thing in the world, I pass the entire XPath in as a hash key to modify the source XML file:

my $twig = XML::Twig->new(

pretty_print => ‘indented’,

twig_handlers => {

“$xpath” => sub {

$_->set_text($new_datum);

}

}

);

Here, “$xpath” is directly from the “UDL” column of the spreadsheet with only the ‘xpath://’ stripped off and “$new_datum” is directly from the “Value” column. That’s a pretty useful one line subroutine if you ask me. I had the new XML files each generated into a different folder (dev/,int/, etc). Then, I checked them into version control (CA Harvest) and built each of them with Meister. If you want the full code, let me know and I’ll post it somewhere.

I did find working with the Excel 2003 XML Spreadsheet format a tiny bit awkward. You have to keep track of the column and row indices, but not bad other than that. I see Microsoft Word 2007 allows you to save as an XML document directly, but you apparently have to define bindings. I’ll have to check that out.

When you work with a locking-type version control tool like CA Harvest, your Meister build project will appear in your Eclipse workspace as read-only when you check out an existing workspace. I’ve been using Eclipse for WebSphere development (WebSphere Studio Application Developer) and for JBoss via MyEclipse IDE. If you want to regenerate your Java targets, you first have to check out the Meister build project so that the files are writable.

Since this can lock the targets exclusively and prevent others from updating the target, you may not want to check out the build project, but you may still want to develop freely and update your local targets for Meister to build it. For this situation I recommend creating a separate build project that you may never check in to version control. It will be writable and it allows you great freedom for a maximally agile development environment. The ‘official’ build project may reference all the built archives in the workspace, but having your own local build project can allow you to focus for a unit build. For example, my workspace may contain an EAR project, a WAR project and one or more JAR projects. If I am principally working only on one of the JAR projects, my local build project can reference only that one JAR project.

When it’s time to release your JAR code updates to the system build and test environments, synchronize your workspace and check out the VC build project. Generate your targets, do a local system build and then check everything in. Your team system build will work fine!

I wanted to share a specific benefit I enjoyed while using Meister for Java development. As part of my role to help develop an automated JBoss build and deploy system, I ended up taking on a developer role for a web services security project for both JBoss and WebSphere. While the project involved about 1000 lines of Perl, it also got me writing simple web services and consumers for JBoss and WebSphere and building them using Meister and its Eclipse plug-in.

Believe it or not, I am still using WebSphere Studio Application Developer 5.1. While my specific tale involves that IDE, it is equally applicable to MyEclipse and Rational Application Developer set of Eclipse IDE’s. In my environment, CA Harvest is the version control/SCM tool and Meister is the build tool. After code is checked in from my desktop using the CA Harvest eclipse plug-in, the code is replicated out to a Linux server, where Meister performs the official system build that is sanctioned for deployment to the application server. There is also a Meister Eclipse plug-in that scans the WSAD workspace for build targets and dependencies. Meister stores this information in one XML file per build target and those files are also checked in to CA Harvest right along side the source code.

Working intensely within the WSAD Eclipse environment as the project manager cracked the whip, I worked with a consumer application and updated it according to the changes in the service WSDL and service endpoint URL’s. One thing I learned is that if one of the parameters for the consumer is tweaked, don’t bother tweaking the XML or generated code, just regenerate the whole client. WSAD will even check out the files before if they need to be. So everything looked good on my desktop with the service and consumer deployed to two separate WebSphere servers on ports 9080 and 9081. Now to get it into the enterprise ‘dev’ environment…

Using the ‘Generate Target Definitions’ feature of the Meister plug-in I updated the Meister build target XML definition files and checked in all my code. I then promoted the code in CA Harvest which automatically kicked off a ‘dev’ build in the Linux environment. I got an error back from Meister saying ‘jdmpview.jar’ doesn’t exist.

Since I knew my consumer app and its elementary nature, I knew that jdmpview.jar wasn’t one of my JAR’s and it must be one of WebSphere’s. Given that 200 other Java apps use the same build environment with the same standards, I probably didn’t use some new feature of WebSphere that no one else is using. Therefore, it must a problem on my local desktop with the version of JVM I was using.

Sure enough, the consumer app was using the base_v51 WebSphere runtime instead of the ee_v51. (I did inherit the initial version of the app from someone else!) And, oddly enough, there is an extra JAR in the base that is missing in the more fully featured Enterprise Edition. Meister correctly forced the runtime environment to be EE for the Linux build, overriding the developer selection. I switched the runtime in the Java build path properties, regenerated the Meister target definitions, checked them in and promoted them to a successful ‘dev’ build. Regenerating the target definitions had the effective of switching out the list of JAR files in the library path from the base_v51 set to the ee_v51 set. The whole thing including one bad and one good build took about 4 minutes.

The great benefit for me was the balance between developer and SCM functions. We could have applied more controls at the desktop level, but from my perspective, I prefer an Agile environment with more freedom even if it means occasionally hanging myself with my own rope. In this scenario I let the tools dot the I’s and cross the T’s and it took no more time than say, waiting for Outlook over VPN.

In developing Java applications for multiple server environments (e.g. dev, test and prod) there is a common pain-point of having to manage deployment descriptor or configuration files specific to each server. For example, you may have an XML log4j configuration file with some parameters different for different server environments. You may want to turn on debug messaging for the development server, but turn it off for production. At the same time, the Java source code will (eventually) be the same in production as it was in development. A similar situation applies for .NET application development.

Like many build management tasks, managing these environment-specific files is generally left to either manual or some type of scripting. This is really something that needs to have a high level of automation applied. Particularly in larger environments, much like scripted build management solutions, existing tactics fall short. This situation is in a far worse state than even the compile part of build management. It is not enough to simply have a script that can spit out some files. One of the biggest problems is information management and the fact that parameter values in the configuration files may be determined by different teams! How do a production engineering team and an application developer both feed inputs into the same XML file?

I’ve worked on this problem for several years and with a number of companies. The critical functionality can be broken down into two different items – information management and a processing engine. In an effort come up with something better, I’ve done a review of what’s out there and here is what I came up with:

• Ant ‘filter‘ task: As with many Ant tasks, this works great if you are an individual with a few items that need updating. It is a nightmare if you are working in a multi-team enterprise with multiple server environments. The main problem is that you have to constantly take working copies of XML files and insert a token for Ant to later re-replace. This leads to a management nightmare to synchronize parameterized copies of XML files with their working copies from the desktop environment. The advantage is that it works for any file type so you can use it for properties files as well as XML files.
• OOPS Consultancy Ant ‘xmltask‘: This is a good engine for specifying and performing changes to the XML and has a full feature set. In fact, we use this in some of the Meister build services. The problem is that it is only for Ant and therefore you have all the reuse, standardization and hard coding issues. Xmltask can provide part of the solution we are looking for, but we still have an information management problem to deal with.
• Maven: Maven has what is essentially the Ant filter task. The specifications are abstracted in the pom files, which is better than Ant, but it encourages templating of configuration files leading to all the problems associated with that (synchronizing templates with working files, testing templates, etc.)
• XML:DB XUpdate: This is a working draft of a specification to encode XML update instructions into an XML document. There is a Java implementation of XUpdate listed on the site called ‘Lexus’, but I couldn’t find anything on it. Since the build management task requires us to generate XML files, I’m not keen on generating XML files using xupdate tags that will allow me to generate other XML files.
• Perl XML::Twig: This has worked wonderfully for me on a back-end web services security effort and I could not be more happy with such a precise, elegant and brief XML library, which includes XPath. This is not a solution for Java or .NET developers, but it could serve as an engine to mimic xmltask or implement the XUpdate specification.
• Excel. Yes, I’ve seen Excel used effectively as the information management front-end to updating the XML. It is a convenient format to share among teams, it is centralized source of information, it can be checked into version control and it can be saved as an XML file itself for processing by another engine. In a large environment, you may have 5 or more server environments, lots of different components to configure, so you could have literally hundreds of parameters to manage. Excel gives you a nicely transparent way to view those values.

There are times in build management that you need to encrypt something – often a password. In the last blog, I gave an overview of the encryption process. Now, I’ll show how you can accomplish something.

Besides just having an encryption algorithm, there are a number of important details to be minded: key, block management algorithm, initialization vector, binary-to-text encoding. Here is what I ended up doing. The encrypted text ended up in the text field of an element in XML and it was successfully decrypted on the other end in pure Java.

First, you need your basic cipher We’ll use the Rijndael algorithm specified by AES. I used a 128-bit key generated with help from the Crypt::Random module:

use Crypt::Rijndael;

my $base_cipher = Crypt::Rijndael->new(
 $key,
 Crypt::Rijndael::MODE_CBC( )
);

Next you use this cipher within a block algorithm:

use Crypt::CBC;  #--Cipher block chaining

my $block_cipher = Crypt::CBC->new(
  -cipher => $base_cipher,
  -header => 'none',
  -iv    =>    $iv,
  -padding => 'space'
);

Even though the initialization vector, $iv, does not need to be secret, I enjoyed making it “randomy” with the ultra-cool Data::Random module. Also note that the padding strategy, adding spaces, is not binary safe so it works for encrypting text, but not for binary format files. Now you just encrypt:

my $encrypted_raw_binary =
               $block_cipher->encrypt( $plain_text );

use MIME::Base64;

my $encrypted_text_string  = encode_base64(
  $encrypted_raw_binary,
  ''
);

#-- empty 2nd arg means “don’t break up long lines”

The last step is necessary to give you something you can easily manipulate as a string to read and write into files.

So that’s it. To decrypt you just do the reverse.

Jim Graham pointed me to Max Kanat’s VCI Project that is providing an abstracted interface to version control functions. This is similar to my Perl SCM project. VCI has support for CVS, Subversion and a number of version control tools I confess I’ve never heard of: Mercurial, Git and Bazaar.

I haven’t had proper time to devote to Perl SCM and so it is withering in its quasi-alpha state. VCI appears to work, so I’ve been thinking of contributing to it. One thing perl-scm does have is adapters to Harvest, Openmake and the Lawson ERP tool. The perl-scm project and its uncommitted updates ** ahem ** do have a robust interface to CA Harvest, so I would probably start there to contribute to VCI. (A customer is paying for work that includes updates to the perl-scm module and they own the code so I can’t commit it ‘as-is’ and need to re-shape it to commit it back to the repository)

Using the Perl SCM module for this client has reduced coding on build management utilities by 90% so I know the VCI project is a useful one. CA Harvest, for example, does not have a persistent command line connection or context and particularly benefits from the abstraction. It allows me to do $hctx->get(@files) instead of construction a very long command line: “hco –b myserver –eh usrpwd –en myproject –st dev –vp \myproject\source –cp /home/sean –p mychange_001 –r –uk –op pc @files” (no guarantees that command is even right). With Harvest you tend to want to use a number of different sequential commands, each of which reuses two-thirds of the same arguments. This was an object waiting to happen.

OK, this topic might be a snoozer, but if we’re going to do build management for our RDBMS (Oracle, SQL Server, etc.) in a revolutionary new way, we need to compare what’s going on with database source code changes and builds and compare that with what we already know.

We said that when we make a runtime change to a database, we are only applying changes on top of what we already have, but in J2EE for Java, we replace the entire running application with another instance of the entire application. This is not an incremental deployment in any sense.

If we compare with the case for C/C++, our source code change might result in replacing one of the application’s executables with a new one. OK, this is a more incremental, but maybe I changed one C source file, resulting in one object file change. I still have to rebuild a new executable with possibly many additional unchanged object files.

For both Java and C, traditional build management technologies allow for incremental builds. That means, if I only change a subset of the source code, a build can be done that re-compiles the minimum number of files, taking into account the full impact of each file change. (Many applications have lost the ability to do incremental builds, but Meister can get it back.) So, for Java and C, there should be the ability to do an incremental build, but when that build step is complete, the runtime environment remains unchanged. A separate deployment step needs to happen which is less incremental to some degree.

For the database changes, there is only a single step combining both build and “deployment” and it is always incremental. When you do the build, the runtime environment is changed immediately. So, as I mentioned earlier, there are differences in build management for RDBMS’s and operating system/JVM applications. Again, let that not deter us from bringing those changes under the umbrella of a common build management system.

The majority of the time database changes (regardless of whether you are using Oracle, SQLServer, Sybase, DB2, MySQL or something else) coincide with application changes in a typical business application. For example, an application change request indicates that the Java application needs to start using a new column in an existing database. So, in order for the new version of the application to function correctly, you need to alter the existing database table to add the new column. There are some fundamental differences in how the two changes are manifested.

For the Java application, deployed to WebSphere, JBoss or other app server, let’s say you started out with a single Java class, foo.java, and this application change requires you to add a second class, bar.java. Typically, I would recompile both classes and bundle them together in an archive (a ZIP file) and ship the archive out to the runtime environment. This has the effect of replacing foo.java, whether or not it has changed along with adding bar.java. For the database change, however, you can only apply the ALTER statement to add the column to the table. You do not re-issue a command to recreate that table. Even if you issued commands to drop the table and recreate it, before applying the ALTER, you would lose all the data in the table (unless you dumped the data before hand and re-imported it).

Following J2EE standards for Java development, the whole application is completely replaced every single time it is changed, while for the database, only the changes needed are applied to the existing configuration. In other language, for Java, the existing runtime configuration is completely replaced, while for the database only a configuration delta is applied.

So, there are some significant differences between Java and database changes, but that doesn’t stop you from managing database changes effectively.

In many ways, databases are runtime systems comparable to other programmable environments such as operating systems and application servers based on Java virtual machines. All environments typically have their own specialty engineering support teams in larger companies and their own preferred programming languages for making changes: Java for application server environments such as WebSphere and JBoss, C/C++/.NET for the Linux, UNIX and Windows operating systems; and, PL/SQL/DDL for database changes.

While there are all sorts of tools that allow directly changing databases through an IDE (Interactive Development Environment), and even versioning changes, one trend has become clear in corporate environments over the last decade. That is the desire to manage database changes within their existing change and configuration management infrastructure. Going forward in this blog, I’ll address some of the challenges both organizationally and technically with doing this. The good news is that, yes it can be and is being done successfully. I will also address how Meister contributes to database change control and integrates database changes with the organization’s overall build management practice.

I had a great time at the Ignite Chicago talks last night. I was actually the first speaker of eighteen with no idea what to expect. It was pretty challenging to write a 5 minute talk at 15 seconds per slide. It forced me to focus on the subject matter and also helped me hone my rehearsal skills. Feedback was that it was entertaining but a bit of a tour de force as I explained the problem of build management in corporate America and also presented a solution with a particular technical architecture involving tools and Perl scripting. I used some examples from this blog.

I met some neat people who are now friends on Facebook. There was a big U. Chicago contingent and I’m pretty sure I must have met Erin McKean before. Her talk on her work with the Oxford American Dictionary gets my nod for the best talk.

Hats off to Jesper Andersen and Sean Harper for organizing it. I liked it so much I volunteered to help organize the next one.

I’ll also be attending Ignite - Chicago. This is not the music festival, but a series of geeky 5 minute talks with about 20 slides. This is just my style, the length of the talk matching the length of my patience. My talk is entitled “Controlling Java with Perl”, and I’ll be covering many of the same topics related to build management I cover in my blog. I see O’Reilly Publishing’s name on the web page, so maybe they have something to do with it. Already there are 98 people planning to attend. Like the Windy City Hackathon, if you are planning to attend, please drop me a line – I’ll buy you a beer.

I’ll be attending the Windy City Hackathon in Chicago on Saturday, December 15. The Hackathon is for Perl programmers to get together and help each other solve problems, learn new techniques and get to know one another. The full Hackathon lasts from Friday the 14^th through Sunday, the 16^th. This is yet another great benefit of the Perl community (YAGBPC). If you didn’t get the parenthetical humor, then you are not close enough to the Perl community and you need to attend.

If you will be attending, drop me a line – I’d love to meet you.

Ivy seems to be trying to resurrect Ant, when it should stay dead in my opinion (unless you are a lone developer on the simplest project). Maven is a more proper build tool with some workflow functionality that falls short outside of a small team of developers with no audit or control requirements. Both are vying for prominence as the open source build tool, yet trying to settle into compatible niches like two brothers hopeful for the throne.

Here is what Ivy says about Maven: http://ant.apache.org/ivy/m2comparison.html

And here is Maven’s side of the story: http://docs.codehaus.org/display/MAVEN/Feature+Comparisons

The key to success in build management is to have a standard build solution that manages builds in all the stages of an application development lifecycle in a consistent manner. Some of the features that must be considered when developing a standard build solution are as follows:

* Provide a standard, consistent build process from desktops all the way up to production
* Directory structure standards
* Standard build tool
* User and Group Management
* Ability to run inside or outside the IDE
* Handle configuration files for the various stages in a consistent way.
* Runs in an automated fashion without manual intervention
* Provides Bill of Material reports which shows the matching of the run time objects (executables) to the source code (in a version control tool) for audits.
* Provides impact analysis (dependency) reports
* Has the ability to perform pre-processing (such as check out from standard version control tool) and post-processing (such as automated testing and reporting).
* Repeatable
* Portable
* Flexible
* Documentation and Training

By using the standard build solution from the developer desktops all the way up to production, the organization can ensure that the same build process is used in each stage of the lifecycle and is consistent, repeatable and auditable. This would minimize finding build issues at later stages of the lifecycle and also reduce the time it takes to release the code to production.

By incorporating standards and best practices (ITIL, CMMI, SOX…) in the build solution, high quality build management services can be achieved. This facilitates organizations to minimize risk and achieve both high quality and value in IT operations.

Stay tuned for our next topic…

Since there was some interest in this topic, I’m diving deeper and showing some code snippets.

As I mentioned before, I’ve found the Archive::Zip and XML::Twig modules invaluable in validating Java deployment descriptors. Post-build validation is important to ensure company standards are adhered to and prevent mishaps, such as deploying an app with a context root already used by another app, or one with no context root defined. It is far easier to fail prior to attempting deployment than to allow deployment to continue, waste valuable time and resources and possibly collide or upset the jvm, costing more time for more people.

I’ve pared down the code I’ve used to the bare essentials for just taking an archive, finding an XML file and extracting it to memory. Next time I’ll get to the XML parsing bit with XML::Twig. Enjoy

Another Wiki Starter. This time on Build Management…
Build management encompasses the control of builds and all of the requirements an organization may have related to builds. Security, version configuration control, logging, audit, impact analysis, reporting, standardization, compliance, verification, deployment, roles, responsibilities and automation may all be involved in build management.

Build management is often cited as an essential part of software configuration management. While many version control tools claim the ability to provide configuration management, they usually only provide application source code configuration management whereas it is the build that translates the source code configuration into the runtime configuration. Without effective build management, source configuration management can not be trusted to match the testing and production runtime environments.

An important part of a build system is transparency. How easy is it for some to tell how the gory mechanics of a build are done? In part 2, I continue with the comparison of the perl build method of Meister and the Java plug-in of Maven to compile XML schema’s to Java classes via the XMLBeans compiler. The lack of transparency in the Maven build was costly to developers.

As I mentioned in part 1, creating the perl stub, or build method, was pretty straightforward for Meister. Using the predefined perl objects from the Openmake::File and Openmake::FileList classes allowed a straightforward call of the XMLBeans compiler, via the XMLBeans scomp wrapper, took less than a page and a half of perl code. There was one parameter that was required to generate 1.5 Java source code, and we got it working, but there was some question about extra classes in the straightforward compile by Meister compared with the JAR file created with Maven using the Maven XMLBeans compile plug-in (here). The straight Meister build also had some classes in different folders with names that looked suspiciously like signatures. This made some people nervous about the integrity of the Meister build.

So, I asked how Maven was doing the compile. I had set the Meister build method to just call the compiler with one parameter and the source files. It’s not like it was the GNU C++ compiler with its 400 or so flags! There was literally no other way to do the build. The Meister build was done exactly the way Apache said to do it. I didn’t get any answers.

I should also point out that my developer contact was worried about differences in the byte counts for like named classes. But, in most modern compilers, including Java, source file system paths and/or time stamps are often embedded in the binary or byte code, so it is actually more common than not that byte counts will not match when doing builds with two different build systems.

I had the fortune of working with Selvi at the time, and she looked at the extra classes in the Meister scomp compile and said they looked like temporary template classes, because of their naming convention. They all ended in “List” – also common in C++. A little research indicated that was likely the case. Was Maven cleaning these up?

OK, so I couldn’t get any answers from anyone. I went and downloaded the source code for the Maven plug-in for XMLBeans. I printed it out and got a big shock: it was 60 pages of Java source code printed! I went through the code, and while I understand Java and could basically understand what it was doing, the details were not obvious. Running Maven through a Java debugger was not going to happen. I then posted a question on the XMLBeans developers’ forum (can’t remember the URL) and got no response, despite the fact that there were a number of build questions answered there.

By this time, the developers had already deployed the Meister-built JAR and it had tested fine, and there was no more nervousness about differences in the fine details of the differently built JAR files. It was time to move on.

Apparently none of the developers either in house or responsible for XMLBeans at Apache could explain the differences, which means they could not explain how Maven was doing the build. I’m sure the one guy somewhere in the world who wrote the Maven plug-in probably knows, but I didn’t try too hard to find him.

The lack of transparency in the Maven build process probably cost the organization about 20 hours of work through unnecessary research and validation. Usually the lack of transparency in the build process is mainly the concern of auditors, but this is a case where it was costly for the developers.