Since there was some interest in this topic, I’m diving deeper and showing some code snippets.

As I mentioned before, I’ve found the Archive::Zip and XML::Twig modules invaluable in validating Java deployment descriptors. Post-build validation is important to ensure company standards are adhered to and prevent mishaps, such as deploying an app with a context root already used by another app, or one with no context root defined. It is far easier to fail prior to attempting deployment than to allow deployment to continue, waste valuable time and resources and possibly collide or upset the jvm, costing more time for more people.

I’ve pared down the code I’ve used to the bare essentials for just taking an archive, finding an XML file and extracting it to memory. Next time I’ll get to the XML parsing bit with XML::Twig. Enjoy

Another Wiki Starter. This time on Build Management…
Build management encompasses the control of builds and all of the requirements an organization may have related to builds. Security, version configuration control, logging, audit, impact analysis, reporting, standardization, compliance, verification, deployment, roles, responsibilities and automation may all be involved in build management.

Build management is often cited as an essential part of software configuration management. While many version control tools claim the ability to provide configuration management, they usually only provide application source code configuration management whereas it is the build that translates the source code configuration into the runtime configuration. Without effective build management, source configuration management can not be trusted to match the testing and production runtime environments.

We are building a Wiki of development, build and release terms and as I contribute starter definitions, I am blogging them. Feel free to comment, or wait for the Wiki. Release management is one of the more challenging parts of application development, largely because it is more social than other aspects. Particularly in large organizations, more teams must come together to coordinate a release. Release management includes all the activities surrounding application production changes. It may involve ensuring requirements and change requests are met, reporting and coordination of multiple application teams to test and simultaneously change production. It is often the latter coordination that is the most difficult as it is a coordination and leadership effort requiring consensus as opposed to something entirely electronic or technical.

Release management in the Perl open source community: There is a version naming convention in the Perl community where each contributor of a Perl module cites the versions of other Perl modules it is dependent on. The automated install program that comes with Perl (cpan for CPAN distributions and ppm for ActivePerl) cross checks the user’s version of dependent Perl modules and updates them if necessary. Thus when one installs  a new perl module, they may automatically get updates to a number of existing Perl modules to satisfy the release requirements.

Release management in the Java open source community: There is none, in general. Eclipse has coordinates the simultaneous release of multiple projects in the Europe release. For the wider community, only the Maven Apache project has attempted to record interdependencies between open source Java projects. Maven has built in automation to walk the dependency lists in a manner similar to Perl by reading the dependency lists on http://ibiblio.org/maven

When compiling code in any language, it’s important to remember
that the versions of your compiler and libraries must match. In the
Java world, which I am most familar with, a common compiler error
claims an “unsupport major minor version.” This is usually followed
by some number. Any time you see this error, it usually means your
compiler and libraries are out of synch. Check to make sure your
compiler home variable (in Java, JAVA_HOME is standard) lines up
with the compiler being found in the PATH environment variable. One
way to ensure this is to have the PATH dynamically set the
compiler’s bin directory based on the compiler home
variable. For example, for Windows Java, your variables might
be set as:

This will guarantee that if you change your JAVA_HOME variable,
the PATH will also dynamically change to include the bin directory
from that location. If setting in a shell or batch script, remember
to append the %PATH% portion so that you don’t wipe out the other
contents of your existing PATH settings!

This same strategy can be applied for other languages as
well.

- Adam

In the first part of this series, I gave an overview of what this series is about. I want to step back and describe the problem I have with part of my development process that is so conveniently resolved by a Mojo/Meister workflow using perl scripts. This results in a continuous integration for perl development.

My environment currently is a build and deployment system that uses Meister for build and workflow management, CA Harvest for version control and high level application lifecycle workflow management and about 100 perl scripts used for deploying all sorts of Java applications and performing validations of various sorts. This system supports several hundred Java applications. Because of the large numbers of applications, automation can only be achieved by adhering to standards and naming conventions, much to the dismay of developers.

I was planning to dive in with the deployment and validation scripts, but I’ve been slowly building up a change control and testing system for the perl scripts themselves and I was confronted with a more simple problem that is perfect for starters in this series. I’ve been improving the testing of the perl scripts by implementing unit test scripts using the Test::More perl module and relatives. We have three environments for perl development, testing and runtime. I’ll call them dev, qa and prod, for short.

CA Harvest is a big database storing every version ever created in the company. Although it is mostly used for Java code, we the build team are using it to manage our perl scripts. At each state in the lifecycle: dev, qa and prod, there is an associated file sytem area that is synchronized with what is in the database. This synchornization is done via a perl script so that when we ‘promote’ code in Harvest from dev to qa, the qa file system area is synchronzied with the qa view in Harvest of our source code. (One of the nice things about Harvest is its ability to trigger back end automation of your favorite script on certain actions, like ‘promote’.) My goal is to trigger the automated testing of those perl scripts after synchronization with a script using Test::Harness.

All perl scripts executed by Harvest are executed on the Harvest server itself. The first problem I had is that some of our perl scripts are executed on the Harvest server, and another set is executed on the physically different build server. I needed to run two sets of perl scripts on two different machines. This is just for the qa environment. I also wanted to do post-production deployment implementation verification (testing what you just put into production) and there are currently two build servers, meaning the scripts need to run on two build machines plus the server. Since the build machines are geographically separate (different network connections) and they could have slightly different shell profiles, I don’t want to assume the test results would be the same.

The problem I have with Harvest is that Harvest will execute a series of post action scripts, but 1) it will execute them in series, 2) if any preceding step fails, it will not execute the following steps.

Currently my synchronization script actually fails, due to a few files that shouldn’t be overwritten. These files are configuration files that must be different between dev, qa and prod, and it is my TODO to clean that up. Rather than tweak the perl synchronization script to make it pass, I need to reorganize the files in Harvest - no small task. So I want they syncrhonization script to fail, but I still want the unit tests to run. That is problem A. Harvest will detect the failure in the synch and exit without running the tests.

The second problem is (B) I am extremely impatient, and why shouldn’t I simultaneously execute my perl tests on the two or three different machines instead of waiting for them to finish, one-by-one. The Meister workflow (same as Mojo) allows me to A) ignore the return code of the synchronization script and B) execute all of the unit tests in parallel on the different machines (via ssh if necessary). For gravy, I had the workflow wait until all the tests were complete on all the machines and then send an email with the HTML log URL.

So, I had Harvest call a single perl script that launched a Meister worflow, which in turn executed the synchronziation, ignored its return code, and the split into parallel unit tests, wait for all the unit tests to complete on all the machines and then fire off an email.

This really saved a lot of tedious repetition in my development process and I am excited about using it. We are likely to grow both the number of build machines and number of perl scripts as time goes on this system is nicely extensible.

This article is mainly dealing with Haskell, but if you read on ’til the middle of page 2, there is a good description of make from a language perspective.

Adam Turoff article on ONLamp.com

Most build systems have superficial build forensics: “This code was checked out and I ran this build, so the build must have used the code that was checked out.” This type of report usually can’t tell what items that were checked out were not actually used, and it often misses items that were used on the file system, but not checked out of version control, such as compiler libraries and extensions. Having this type of report can easily lead to a false sense that builds are reproducible. Build forensics means identifying after a build all the information needed to tell you exactly how that build was done. It is not enough to say “it is a Java build from this version control project.” What is meant by “build forensics” is all the information needed to precisely reproduce a build. This means not only every version number of every source AND library file, but also all of the options and parameters passed to the compiler and all the environmental information such as compiler version.
Meister records environment information and records source, resource and library items as they are used in the build. It also tells you if the item came from version control or not. This is an accurate picture of the build and allows forensic analysis after the build which is useful not only for audit purposes, but also for debugging.

I received a call from one of our development build meisters … there is a build issue that needs to be resolved for a complex application - this application was building successfully for years without any problems and suddenly it stopped working today. I looked at the error snippet that he had emailed me from the build log…

The first thought that went through my head was “I know how to resolve this - I have seen it before”! I knew that the build tool uses Ant under the cover for Java compiles. All I have to do is increase the memory using ANT_OPTS.

This problem was happening on an Unix machine – this was one of our controlled build machines for our enterprise java builds. Before I changed the environment variable on the build machine, I went through the usual drill…

I asked if anything had changed significantly with the application today - I heard that there was no change other than a few java files were added. I asked him if his other applications were building – he said that everything else has been building successfully except this one application. I asked him if we are able to reproduce this “out of memory” problem - he tried the build for the same application and the compile failed at the same step.

Just to be sure, I requested the on call unix administrator to see if there was any memory problems reported at the system level for that machine. The machine didn’t report any errors of being out of memory or high swap at all that day.

It was time to set the ANT_OPTS environment variable! I added “export ANT_OPTS=-Xmx512m” to the .profile of the system user that is used for all controlled builds on that machine. The development build meister tried the build again - it passed the compile step where it had failed consistently today! The build was successful once again for this complex application. Problem solved!

An important part of a build system is transparency. How easy is it for some to tell how the gory mechanics of a build are done? In part 2, I continue with the comparison of the perl build method of Meister and the Java plug-in of Maven to compile XML schema’s to Java classes via the XMLBeans compiler. The lack of transparency in the Maven build was costly to developers.

As I mentioned in part 1, creating the perl stub, or build method, was pretty straightforward for Meister. Using the predefined perl objects from the Openmake::File and Openmake::FileList classes allowed a straightforward call of the XMLBeans compiler, via the XMLBeans scomp wrapper, took less than a page and a half of perl code. There was one parameter that was required to generate 1.5 Java source code, and we got it working, but there was some question about extra classes in the straightforward compile by Meister compared with the JAR file created with Maven using the Maven XMLBeans compile plug-in (here). The straight Meister build also had some classes in different folders with names that looked suspiciously like signatures. This made some people nervous about the integrity of the Meister build.

So, I asked how Maven was doing the compile. I had set the Meister build method to just call the compiler with one parameter and the source files. It’s not like it was the GNU C++ compiler with its 400 or so flags! There was literally no other way to do the build. The Meister build was done exactly the way Apache said to do it. I didn’t get any answers.

I should also point out that my developer contact was worried about differences in the byte counts for like named classes. But, in most modern compilers, including Java, source file system paths and/or time stamps are often embedded in the binary or byte code, so it is actually more common than not that byte counts will not match when doing builds with two different build systems.

I had the fortune of working with Selvi at the time, and she looked at the extra classes in the Meister scomp compile and said they looked like temporary template classes, because of their naming convention. They all ended in “List” – also common in C++. A little research indicated that was likely the case. Was Maven cleaning these up?

OK, so I couldn’t get any answers from anyone. I went and downloaded the source code for the Maven plug-in for XMLBeans. I printed it out and got a big shock: it was 60 pages of Java source code printed! I went through the code, and while I understand Java and could basically understand what it was doing, the details were not obvious. Running Maven through a Java debugger was not going to happen. I then posted a question on the XMLBeans developers’ forum (can’t remember the URL) and got no response, despite the fact that there were a number of build questions answered there.

By this time, the developers had already deployed the Meister-built JAR and it had tested fine, and there was no more nervousness about differences in the fine details of the differently built JAR files. It was time to move on.

Apparently none of the developers either in house or responsible for XMLBeans at Apache could explain the differences, which means they could not explain how Maven was doing the build. I’m sure the one guy somewhere in the world who wrote the Maven plug-in probably knows, but I didn’t try too hard to find him.

The lack of transparency in the Maven build process probably cost the organization about 20 hours of work through unnecessary research and validation. Usually the lack of transparency in the build process is mainly the concern of auditors, but this is a case where it was costly for the developers.

An important part of a build system is transparency. How easy is it for some to tell how the gory mechanics of a build are done? Sean looks at his experience writing a Meister build service and method to compile XMLBeans and compares it with Maven.

I was working on a build team that managed QA and Production builds using OpenMake (previous name for Meister). In this particular shop, the developers were free to use what they wanted on their desktops, but OpenMake was used for the QA and Production builds.

It was determined that XMLBeans was going to be used as part of one application build. XMLBeans (http://xmlbeans.apache.org/ ) binds Java objects to XML schema designs (XSD). It has a simple compile process that takes XSD and an XSD config file and produces a JAR. The developers were using Maven with a plug-in to compile the XMLBeans.

How a company can effectively adopt new technologies deserves a book in itself, and I am not suggesting this is the best process to take, but this is how it happened. I took on the task of creating the build service (rules) and build method (perl stub) to allow OpenMake to build the XMLBeans JAR. This JAR was then deployed in a WAR under WEB-INF/lib. That was pretty much the only specs for this build service I had.

The simple approach is to find out from the vendor, Apache in this case, their recommended approach for compiling and stick closely to that. The documentation was sparse, but there was a command line wrapper, called scomp, for the Java compiler class available that loaded the necessary libraries and constructed the arguments for the Java class. My impression looking at the wrapper (ksh for *NIX, bat for Windows) was that it was a little too simple, but it met all the specs so I went forward with having the build method perl stub call the scomp wrapper using OpenMake’s perl object methods for both the dependencies and the output JAR file name in the scomp command line. The perl object for dependencies simply contains a list of the XSD and XSD config files as determined by the C engine by searching for dependencies along the search path. The name of the JAR file is determined by the target defintion.

Using these objects is pretty simple and familiar to most Perl programmers. If I want to use the name of the expected JAR file name, the target name, I use $Target->get. That’s it. (It’s not a proprietary language as some competitors suggest!) It was pretty straightforward, and I turned it back to the developers.

I just wanted to mention how much I love the XML::Twig and Archive::Zip perl modules. And, why perl, anyway?

I’ve working on some custom perl scripts to validate deployment descriptors and I’ve found that using Archive::Zip to identify and extract XML files and then using XML::Twig to parse them is vastly superior to having perl call shell commands (yuck!). I’m a serious minimalist when it comes to designig and writing scripts and I love the simplicity and brevity these modules allow. These scripts are called as workflow activities in Meister and it is a pretty sweet setup.

Some developers I’ve worked with have asked “Why not Java instead of perl?” Briefly, perl lends itself very well to doing the simple, dirty work of pushing files around and reading, writing and parsing text. Since perl is compiled at runtime, the scripts are transparent, easy to change and you don’t have to compile it and deploy it somewhere. This translates into greatly shorter development times for the build system. This is important because it’s the software engineers writing the business applications that get the big bucks - no one wants to pay for infrastructure improvements. Java has its place too in Mojo and Meister and so does C/C++. All three languages are cross-platform and are used for what they are best at in the tools.

When I made the transition from physics to build meistering and scm/build management consulting, I was surprised by my first glimpse into the firey pit of GNU make build management at a local Chicago financial services company.

I knew shortly after entering graduate school that I didn’t want to be scientist. Yes, better to find that out before, but it was interesting and I stuck it out, painful though it was. I heard a WWII fighter pilot say something on the TV show “Dogfights” (one of my favorites) that rang true for me. “I wouldn’t trade the experience for a million dollars, but I wouldn’t give one red cent to do it again.”
>
> I ran into the co-founders of OpenMake Software at a trade show in Chicago over 10 years ago and it was an easy transition for me to go from small lab of five to a small company of three, as the first hire. Steve Taylor and Tracy Ragan were still doing consulting at a Chicago credit card company and one day I was in the “office”, when the phone rang. Before I knew it, Steve and I were in the car heading over to help out with a critical build issue.
>
> I met Selvi Dayabaran, who was the poor sole whose responsibility this problem was. This company didn’t have OpenMake 3.0 or whatever it was at the time, but most of their application development was in C/C++ and COBOL on the UNIX side with GNU make to manage the builds. There was also a “makedepends” process which scanned all the C code and appended the list of headers to the makefile so their timestamps could be checked for incremental builds. Before I knew it, Steve a.ka. “F1″, was running GNU make through a debugger.
>
> “Every undergrad at MIT who does a thesis on builds adds his own code to GNU make and the code is really a mess!” Steve said. Steve found the problem, updated the local copy of the source code, they rebuilt GNU make and ran the build successfully.
>
> Having used make for awhile, I was surprised that the bug was enough to prevent a build from happening. I was also surprised by the magnitude of the software application. Hundreds of header files were processed by makedepends. The early versions of OpenMake actually used GNU make, but one day a year or so later Steve said he was writing his own dependency engine because “it is easier to fix the bugs.” I thought back about debugging GNU make and it made a lot of sense.
>
> Most importantly, I met Selvi, who became hire #2 at OpenMake Software. We’ve now worked together for over eight years and I’ve learned more from her than from GNU make.
>

SCM team wins President Award using Openmake
The Discover Card  SCM team has won the President’s Award for best IT project implementation. The project combined a Harvest to ClearCase migration, WebSphere 4 to 5 upgrade, an Openmake 6.1 to 6.3 upgrade and a number of other smaller technologies.

My name is Selvi Dayabaran and welcome to my blog series “Investing in Build Management”.  Many companies are in the situation where they must evaluate their current build management strategy or develop a new one.  This series is designed to help you understand build management and key areas that must be addressed when developing a software configuration management strategy for your organization.

Build management is not often considered important to companies, but in this day and age, if you have been audited or challenged to comply with any regulatory demands (Sarbanes-Oxley, ITIL, CMMI, FDA, HIPAA…) in your industry, you have already seen how critical it is to the success of your organization.

Not having a standard process to build the same way in all the stages of the development lifecycle could lead to discovering problems at the later stages of the lifecycle becoming a costly endeavor and hindering the on-time delivery to customers. When companies experience mergers, acquisitions, or simply grow, IT infrastructures must be optimized and grow too. The tools available for build management, version control and process management have a range of approaches and so it is important to choose the tool(s) that suit the needs of your organization.

Upfront analysis keeping in mind the industry, business processes and competition will make your investment pay off and avoid those that won’t.  By being an educated consumer, you can embrace the ideas that work, accomplish the goals that you have set for the configuration management, and begin to measure the ROI within your organization.

Stay tuned for our next topic…

My name is Sean Blanton and welcome to the beginning of my OpenMake Mavericks blog series. This blog series is designed to teach powerful, field-developed techniques to create an agile, repeatable development to release environment. The series is designed not only to help effectively use the Mojo and Meister products, but also to address software change and configuration management and build management in general. For the first of my series, I’ll be taking you through how to create workflows in Mojo and Meister to get the greatest impact for your development and release tasks.

Mojo and Meister workflows can be easily applied to actions such as Java archive deployment. A series of workflow activities that perform the essential deployment steps can be constructed where each activity executes one of a series of copy commands or perl or ant scripts to accomplish the goal.

By writing simple scripts that each perform a specific task and having a workflow activity execute it, a great savings in lines of code and reduction of complexity can be achieved. By reducing the lines of code and complexity, the resources required for testing can be reduced as well. Let the reporting and logging of Meister do the work for you instead of coding into your script calls to logging functions. Using the dependency management of Meister frees you from coding flow control logic into your script. Constructed in this way, the Meister workflow monitor will report back live the success or failure of each essential step and log it.

In my next few posts, I’ll jump into more detail on the following aspects of creating effective workflows:

* Introduction Mojo and Meister workflows
* Letting Mojo/Meister control the workflow path
* Passing information between activities
* Real life workflows for Java deployment
* Designing your workflow for a complex task

I look forward to bringing you proven best practices from the field . I plan to post about once a week.