While we have managed Jboss and WebSphere deployments using Meister and Mojo and we frequently manage resources for different target server environments through build, here is a company that focuses on both of those things.

I posted a comment about the reluctance of script writers to give up their scripting on their blog here.

It is just plain fun to run parallel workflows and builds and watch the activities and build steps light up the workflow monitor in real time like a Christmas tree. See this flash demo to see what I mean.

As customers go to machines with more and more cores, fewer machines are needed in the application lifecycle infrastructure, particularly for builds and code retrievals - the most resource intensive functions. This is helping to simplify the infrastructure, reduce maintenance and administration and drive down costs.

Several of our customers are running around 5000 builds and non-build workflows per month on two machines. The primary reason for two machines, in fact, is for disaster recovery, and the goal is to run both machines at less than half capacity so that in the event that one machine (or datacenter) fails, all the current capacity can be run as a contingency on the one machine that’s left.

Thread control is very simple with Meister and Mojo. Both use the omsubmit dependency manager program to handle this. Meister’s om program translates build events into workflow steps using omsubmit. The OMSUBMIT_MAX_USER_PROC value sets the maximum allowed number of threads.

You might think that if you are running dual, quad-core build machines that you should set the max threads at 8. However, Meister posts build operations to one thread and the associated logging operation to another. Compile operations notoriously use a lot of memory and CPU resources, but the logging operation posts to a server and waits for the operation to complete. There is really no disadvantage to setting the max threads higher than 16 in this case, so go ahead and do it.

As a non-build workflow example, I worked on JBoss deployments to 48 Linux machines. The workflow was parallelized into 48 activities each of which deployed to a single machine in parallel. The deployment activity was largely a remote execute operation that extracted archives on the remote machine. The extraction took about a second for a medium sized application. Again, this is a waiting situation where machine resources are essentially idle while the thread is in use, so use more threads. The machine was a dual, dual-core build machine and we set OMSUBMIT_MAX_USER_PROC to 50.

Watching the workflow monitor as the deployment ran, we could see roughly half of the machines light up (meaning actively running) at any one time and the entire deployment process synchronized all 48 machines in a little over two seconds.

So, don’t simply match your machine’s CPU threading capabilities - overclock! Aim high for max threads and try to determine where your performance is optimized. I’d love to provide you with some metrics as a function of thread count, but usually once something is working it’s on to the next project. I barely have enough time to blog!

One of OpenMake Software’s product strategies is to keep things simple. Build management is one of the most complex operations in all of the IT world, and one of our key benefits is to simplify, organize and automate the build process for development, testing and production.

We’ve seen a trend among our customers to simplify their build management infrastructure by going to fewer build machines with more CPU cores. Builds in particular use relatively more CPU resources than other resources as code is interpreted and compiled in memory and then finally written to disk. By reducing the total number of machines, rack space, procurement, administration and other IT overhead costs are reduced at great cost savings per machine eliminated.

Recently, I was at one of the big chip makers where they used dual quad-core CPU Linux machines for their development and builds. They had two machines and were able to control access to allow separate areas for development, testing and release builds in keeping with best practices. Having all the horsepower of 8 CPU cores on a single machine kept them from needing more machines.

Another customer does 6000 builds per month with Meister on just two build machines.

IBM, when selling BuildForge, likes to talk about big build server farms, because their tool does remote execution on multiple machines, as does Meister. However, BuildForge does not do builds at all. It can remotely execute your existing build scripts, but there is little real value add to that. BuildForge is also famously expensive. What happens over the next few years to the high investment in multi-machine remote execution software as the number of machines declines, perhaps dramatically?

A similar argument can be for Electric Cloud’s Electric Accelerator product. It’s possible in some cases, for C/C++ builds to gain an edge by pushing a compile operation to another machine, and then bringing it back. You would only do this to gain access to additional CPU resources. In the past, you might have 8 build machines that Electric Accelerator would farm operations out to. Now, you can pull all those operations into a single machine and there is no need for that functionality. Also, you are stuck with converting your GNU makefiles into other GNU makefiles.

Meister is optimized for multi-core CPU build machines and offers multi-threaded capability to both build events and non-build workflow events. You know where your build is and there are fewer dependencies on network resources. Both BuildForge and Electric Accelerator add additional overhead to build administration to coordinate across multiple machines - a dying practice, that no organization wants to invest in. Meister is the best bet for a future with fewer build machines with more horsepower.

We’ve been testing Mojo and Meister 7.2.1 and getting ready for their release on December 15. This is a maintenance release with bug fixes from users who’ve started running builds and workflows with Meister 7.2 and using the free workflow automation of Mojo and putting those releases through their paces. It also contains a lot of UI and documentation improvements.

Existing users of the 7.2 version of either Mojo or Meister will be able to upgrade via the update sites, http://www.openmakesoftware.com/mojo/update_site and http://www.openmakesoftware.com/meister/update_site, respectively.

Users interested in getting Mojo, the free workflow automation tool, and Meister, the industry leading build automation tool, can find download instructions on our website, http:///www.openmakesoftware.com.

I’ll take that last one about ensuring a drive is mapped and make a Mojo activity template out of it so you only have to enter the drive letter and the share name. Here’s another one…

I have a workflow that can only run on a specific machine because it assumes certain software is installed (Git in this case). To fail a workflow that is not running on the correct machine, run this shell command at the beginning of the workflow:

if not “%COMPUTERNAME%”==”ASLAN” exit 1

I’ll make an activity template out of that one also, so then you only have to enter the computername. Note that this is case sensitive!

It’s true we can make one out of Perl that is platform independent.

In order to verify that a drive is mapped on Windows, and if not, map it, and then have the command complete with a successful return code, use this command:

dir O: > nul || net use O: \\computer\myshare

It should only return a fail code when it is unable to map a drive. I’m using this in one of my Mojo workflow activities…

First, let me say how nice it is to have the Mojo workflow engine that allows us to manage the compliance checks, deploy to multiple machines in parallel and validate deployment. This makes our lives a lot easier and provides clear benefits for deployment via the parallelization, dependency management, scalability, logging and reporting. Underneath the covers, and for those of you who don’t have the luxury to use this almost-free product, there are some important low-level tools that are critical to the development, testing and operation of the Mojo JBoss deployment system on Linux.

With the most important listed first, they are:

1. JBoss support
2. The Perl executable (5.6-5.10) and base language
3. Perl’s Test::Simple or Test::More modules
4. Perl’s Test::Harness module
5. The JBoss twiddle.sh script or command equivalent
6. Perl’s XML::Twig
7. Perl’s Archive::Zip
8. vi
9. ssh
10. xterm

JBoss support wins hands down due to the number of bugs and critically important undocumented features. On a scale of 1 to 10 where 10 is the best documentation, I give JBoss about a 3 or 4. Googling doesn’t even help that much for deployment issues.

You may be surprised at the prominence of Perl, but if you think about what you are really doing and what the best tool for the job is, it makes sense. You are really moving an archive (a ZIP format file), copying XML files, creating directories, changing permissions, extracting the archive to the file system perhaps. Where did I mention Java? Nowhere. The twiddle.sh command comes in handy if you get the secret commands from JBoss support that tell you if the application you deployed has actually started correctly. Notice that this is a shell script suggesting we’re not the first to use non-Java tools to manage deployment.

Particularly on the testing side, I can’t think of a viable alternative to Perl testing. We need to test that we created this directory, changed that permission, updated that file timestamp, etc. We have about 300 test cases encoded in Perl that are run with every change to the deployment system. It takes about 20 seconds to write and run a simple test case in Perl.

Lessons? Use JBoss support early and often and use Perl.

I’ve worked out a good system putting Meister build management metadata under version control. Why put it under version control when you can do a backup, you may ask? Well, there are times when you may want to develop a reusable workflow or update a Java build method script and test in an isolated test environment. For this you would use a separate test instance of the Meister server. Putting at least some of these files under control will help ensure that you move the known version of your tested server metadata file into your production environment.

There are a few other challenges to be aware of. You may want projects and dependency directories to be able to be created an edited in production while you simultaneously modify a workflow, for example. You don’t want to overwrite a dependency directory definition in production with an older one you are migrating from the test environment.

I’ve got a few other challenges as I’m using CA Harvest for version control and its excellent control over changes extends version locking to the file system by removing write access to files that are not locked by someone for change. Meister requires that most metadata files be writable, so if you check in files to CA Harvest directly, you will leave Meister metadata files read-only and that will cause problems.

So, here are a few tips from my now slick Meister metadata version control system:

• Use the …/meister/kbserver/tomcat/webapps/openmake.ear/openmake.war/ directory as the root of the file tree under version control. Don’t version anything outside of that tree, but do version everything under it for a simple boundary to your project.
• Before checking in/committing changes, copy files from the Meister runtime directory to an alternate workspace or temporary directory tree. Check in/commit from there, not the Meister runtime directory. This ensures that you don’t write into the runtime environment, possibly corrupting something.
• Before taking updates from version control, check them out to an alternate workspace, reference directory or other temporary directory. This is again to avoid corrupting the runtime environment. For CA Harvest, you can use the automated reference directories to copy from. Be careful not to preserve the read-only attribute when you copy from there. I used File::Copy::Recursive and its option not to preserve permissions for a simple, streamlined copy.
• Manage the check in’s and deploys (copy’s) according to the major subdivisions of the Meister metadata. My check in and deploy commands work by taking a directory relative to the openmake.war/ directory as an argument.
• Use the awesome Perl SCM project by yours truly to reduce your coding by 90% if you are using CA Harvest. CA Harvest also requires a number of other steps, such as creating a change package, locking files you plan to change to the package before checking them in, and breaking up the locking and check in commands into manageable chunks. The Perl SCM project helps with all that.
• If you can manage your changes incrementally, all the better. OpenMake Software’s HarRefresh and CA Harvest’s HRefresh manage reference directories with incremental changes. This gives you a workspace or reference directory with only the changes relative to a baseline, not the full baseline. This means if you changed only one workflow, then it is easy to copy just the one changed file to the production environment, instead of the whole baseline with only one changed file. You will have a very simple and transparent production change procedure with this – very nice.

I’ve found the multi-threaded capabilities of Mojo and Meister workflows to be very valuable for builds and deployment. The chief benefit I’ve received is in saving time as you might expect. I’ve been working with a workflow that deploys a Java application to up to 24 servers. Let’s ignore the sequential part of the workflow and examine the time difference of running parallel deployments versus one where each of the 24 machines is updated in sequence. The deployment process takes about 5 seconds per machine. Sequentially, that’s 24 x 5 seconds, or 2 minutes. In parallel, well it’s not quite 5 seconds, but closer to about 20 seconds because of limitations of the Linux machine it is running on. Still, that’s a tremendous 100 second savings.

In addition to using the parallel workflow to cater to impatience and improve productivity, I want the Java application to hit all of the servers in the cluster close to the same time. In this particular strategy, only 3 machines out of the 24 are in the cluster. The rest are to support dynamic resource allocation and disaster recovery. Running the deploys in parallel allows me to hit all machines, and therefore all the machines in a cluster at close to the same time without having to figure out some ordering so that the cluster servers are hit first and then the rest. This ends up saving a lot of coding, testing and possibly debugging. Great stuff.

There are times in build management that you need to encrypt something – often a password. In the last blog, I gave an overview of the encryption process. Now, I’ll show how you can accomplish something.

Besides just having an encryption algorithm, there are a number of important details to be minded: key, block management algorithm, initialization vector, binary-to-text encoding. Here is what I ended up doing. The encrypted text ended up in the text field of an element in XML and it was successfully decrypted on the other end in pure Java.

First, you need your basic cipher We’ll use the Rijndael algorithm specified by AES. I used a 128-bit key generated with help from the Crypt::Random module:

use Crypt::Rijndael;

my $base_cipher = Crypt::Rijndael->new(
 $key,
 Crypt::Rijndael::MODE_CBC( )
);

Next you use this cipher within a block algorithm:

use Crypt::CBC;  #--Cipher block chaining

my $block_cipher = Crypt::CBC->new(
  -cipher => $base_cipher,
  -header => 'none',
  -iv    =>    $iv,
  -padding => 'space'
);

Even though the initialization vector, $iv, does not need to be secret, I enjoyed making it “randomy” with the ultra-cool Data::Random module. Also note that the padding strategy, adding spaces, is not binary safe so it works for encrypting text, but not for binary format files. Now you just encrypt:

my $encrypted_raw_binary =
               $block_cipher->encrypt( $plain_text );

use MIME::Base64;

my $encrypted_text_string  = encode_base64(
  $encrypted_raw_binary,
  ''
);

#-- empty 2nd arg means "don't break up long lines"

The last step is necessary to give you something you can easily manipulate as a string to read and write into files.

So that’s it. To decrypt you just do the reverse.

I wanted to pass along what I learned about a new area for me: encryption. I’m working on a build management project for securing Java web services and I’ve enjoyed learning about encryption methods. There are a couple of key concepts to learn and Wikipedia has some informative and entertaining pages. I recommend the “The Code Book” by Simon Singh for a great history of the subject.

One concept strange to newbies is that the encryption algorithm should be widely known and public. The key (**ahem**) is that the encryption key remains secret. If there is a problem discovered with the algorithm, you want to be the first to know. The commonly used algorithms are so robust that there is little advantage to be gained by understanding how they work, as long as the encryption key remains secret.

The U.S. government held a competition for an encryption algorithm to be the Advanced Encryption Standard (AES). The algorithm chosen for this is called Rijndael and it replaced the Data Encryption Algorithm (DEA) of the Data Encryption Standard (DES). The triple form of DEA is still commonly used and is incorrectly but widely known as Triple DES. So yes, the encryption algorithm for the U.S. government’s most top secret data is widely known.

AES specifies not only that Rijndael be used, but that it be used with a 128-bit key. Rijndael also encrypts only 16 bytes. What?! Yes, so basically you have to chop up your message into 16 byte blocks and encrypt each one separately.

If you were encrypting a long message or a lot of messages, you would be encrypting similar words over and over and a lot of your 16 byte blocks might look similar or even identical. This makes you susceptible to a form of frequency analysis attack (see “The Code Book”). So, another algorithm is tacked on to obfuscate the 16 byte blocks after encryption. A commonly used block algorithm, Cipher Block Chaining (CBC), makes the text of a block depend on the encrypted text of the preceding block as well as its own encrypted value. The first block in the message is seeded with an initialization vector (starter value of 16 bytes of text) that interestingly does NOT need to be secret. That doesn’t quite make sense to me, but I trust the experts.

If your message is not exactly a multiple of 16 bytes, you will have to pad it with something that you agree on with the decrypter. The padding characters have implications for what is “binary safe” so be careful. (See Crypt::CBC for a great rundown of commonly used padding techniques.)

The last thing you need to know is that when you encrypt your blocks and obfuscate with a good block algorithm, you end up with raw binary data. I certainly don’t recommend it for XML. This encrypted data, however, is commonly encoded using the MIME format Base 64. This, from the early use.net days, converts raw binary into alphanumeric characters plus ‘-’, ‘=’ and ‘/’. And, yes, that’s 65 characters. You will also need to decide if you will break up the lines with a carriage return after so many characters or not.

So, to get your encrypted value into XML, you can 1) choose a known encryption algorithm, 2) generate a key, 3) use a block management algorithm, 4) decide how to pad your last block, 5) generate an initialization vector (for CBC), and 6) convert it to Base 64 for suitability for text files. To decrypt, do the reverse. Enjoy!

In my previous blog I showed how to test that a subroutine fails properly when given incorrect arguments. I generally use a similar method to test command line Perl programs that are run as Meister activities.

use Test::More qw(no_plan); 

my $program = 'actvity_01.pl';
 my $rc = eval {`$program`} ; 

ok( !$?,  "Runs fine or not: $rc");

I take it a bit farther and test calling the script with different inputs to make sure it fails with missing args and succeeds in a situation when it should. When running scripts with Meister and Mojo, it is important to make sure you end with the correct return code so you have the proper handling in the workflow.

I am aware of the Test::Exception and just now really looked at it. It has some attractive functions like ‘throws_ok’. This function tests for a failure and that the error message matches a regex. That is nice and compact. Still, for what I am doing these days, I don’t have standard error messages so it is overkill. I’ll keep it in mind for the future.

A common problem in the UNIX and Linux worlds is managing the shell profile. The profile is used by the shell to set basic parameters such as environment variables that all programs running under the shell would have access to. An example is the OPENMAKE_SERVER environment variable that must have as its value the URL of the KB server servlet.

Profiles tend to grow and may contain a number of different settings important to different programs. It is very common to hear about changes to the profile causing some unintended effect to one or more settings, which in turn causes some program or other to stop working. For example, if I have an OpenMake client installed on a Linux box, and then I want to use a COBOL compiler, I may need to set a number of new variables like COBDIR in my profile. A common practice would be to copy the profile from one of the developers who regularly compiles COBOL, on top of your profile. This way you ensure your compiles work. However, by copying you just overwrote all the setting you had before.

Now, of course, you wouldn’t intentionally lose all your settings you careful defined, but maybe the sys admin copied it for you, trying to be helpful. This very simplistic example is easy to handle, but with lots of settings and lots of programs depending on them, you could easily have a situation where some parameter change is not detected until something crashes in production. So, what do you do? You want to test your changes, but the profiles are usually in some shell script language and are outside the common application lifecycle processes.

It turns out Perl is ideal for this task. All you have to do (for a login-type profile) is to check the environment variables. This effectively gives you both functional and regression tests for your profile.

use Test::More;

ok( exists $ENV{COBDIR}, "COBDIR exists");

ok($ENV{COBDIR} eq '/usr/cob', "COBDIR set correctly");

Because it is so easy to write these tests (vi: yy j p dw <your change>), I like to make them pretty granular. In this example, I will know from the two tests if COBDIR is set incorrectly, or not set at all. That can tell me a lot about what caused the problem. This is an incredibly small investment to ensure your profile changes don’t cause unintended production failures.

I think it is funny that Electric Cloud is saying that Perl is and old, outdated language (presumably to position themselves against OpenMake and Meister) when they are based on make, which truly was designed for C and is not very extensible (but works for ad hoc build scripting). I can add FTP or email support to a Meister build method in less than 10 lines of Perl code - not so easy to do with make alone. I can even update my Facebook page with the latest build result through a Perl API.

Most developers know Perl has a recognized place in most companies. There are trends in software languages, and Perl may not be the latest, coolest thing on the street (like Ruby or Python) but Perl is incredibly dynamic and robust. It is possibly the greatest single example of successful open source development. Neither Ruby, Python nor anything else besides the open source Java community even comes close to offering the breadth of functionality that Perl does. Perl even does  automatic release dependency management - something unique for its scope in the open source world, although the Maven project is attempting to do something similar with Java.

Java developers often think Java is the only thing in the world, but in comparing Maven (an all Java build tool for Java builds only) and OpenMake, it took 60 pages of Java code to do the same thing (see my previous blog on XMLBeans) as 1.5 pages of perl code as used in OpenMake.

Perl works great at moving files around and working with the operating system – Java is not so good at that. Meister and Mojo use Java for the server which works well, but Meister uses Perl where it can be used best (parsing texts and interacting with the operating system) and C is in turn used where is best (C can’t be beat for speed). We also use make for what it is does well - the rules engine and dependency ordering. Combining the C based make engine with Perl for executing commands really gives enormous power to Meister.

How did Cruise Control and Maven lead to Mojo and Meister? We listened to developers’ struggles with CC and Maven and the result so far is Mojo and Meister. Read on for details…

I was working with some developers on standardizing their builds on Openmake (precursor to Meister) and asked them about their experiences with Cruise Control and Maven for Java builds. They had been using Cruise Control to call Maven. While there are a lot of different bits of functionality in these tools, I want to focus on the repeatable processes they are set up to control. Cruise Control (CC from now on) was set up to check out files from CVS and initiate Maven which did the build and ran JUnits.

So CC had a repeatable build loop of 1. execute according to schedule, 2. run a CVS check out, 3. initiate Maven, 4. report results.

Maven itself had its own, more specialized build loop of 3a. compile, 3b. run Junits, 3c. “package” the Java, or create the deployable archive. (There were a few more steps, but I’ll skip it to maintain focus)

The resulting integration involved nested build loops. CC ran one build loop, and one of the steps in the CC build loop was a multistep Maven build loop. This cumbersome “integration” resulted in a typical build log. If you’ve never read a build log, then maybe you are reading the wrong Blog - they are typically very long, somewhat cryptic and it is usually tedious to identify specifics of commands and errors when there is a problem.

So, how can we do better? Well, we can “flatten” the nested build loop into a single repeatable build loop where every step is reported equally and then some. Mojo is roughly the equivalent of CC with a graphical build monitor to flag in bright red steps that fail in an event summary view in Eclipse or a in a standalone Eclipse RCP client. Meister adds to Mojo the Maven bit. The difference is that Meister includes Mojo, as if CC and Maven were one fully seemless product. With Meister, all steps in the build loop are treated equally. This includes the CVS check out as well as the compile step for a particular JAR. When an error is flagged in red on the Workflow Monitor the developer clicks on the red line and the (HTML formatted) build log is pulled up with the focus precisely on the error location. This provides the transparency the developers were looking for in the build loop and it saved them a lot of time poring through tedious logs.

This is what we mean by community-developed. We work closely with developers in the field and find out what is a productivity killer (and annoying) and listen to what they say about what they would like to see improved. We take their feedback and directly put it back into product development. For me it is really rewarding to help improve the build process for developers. Developers tell me they appreciate someone listening and working to help them be more productive. I wonder if it is a kind of social work for builds?

Since there was some interest in this topic, I’m diving deeper and showing some code snippets.

As I mentioned before, I’ve found the Archive::Zip and XML::Twig modules invaluable in validating Java deployment descriptors. Post-build validation is important to ensure company standards are adhered to and prevent mishaps, such as deploying an app with a context root already used by another app, or one with no context root defined. It is far easier to fail prior to attempting deployment than to allow deployment to continue, waste valuable time and resources and possibly collide or upset the jvm, costing more time for more people.

I’ve pared down the code I’ve used to the bare essentials for just taking an archive, finding an XML file and extracting it to memory. Next time I’ll get to the XML parsing bit with XML::Twig. Enjoy
[cc lang="xml" tab_size="2" lines="40"]
# Load modules…

use Archive::Zip qw( :ERROR_CODES :CONSTANTS );
use strict;

my $az_ear = Archive::Zip->new(); # Define new zip object

unless ( $az_ear->read( ‘foo.ear’ ) == AZ_OK ) { # Read it
die ‘read error’; # hopefully

}

# Look for a file
my ($azm_app_xml) = $az_ear->membersMatching(’META-INF/application.xml’);
print “Found: ” . $azm_app_xml->fileName . “\n”;

# Extract it to memory

my $app_xml = $az_ear->contents($azm_app_xml);
print “Its contents are:\n$app_xml\n”;

[/cc]

In the first part of this series, I gave an overview of what this series is about. I want to step back and describe the problem I have with part of my development process that is so conveniently resolved by a Mojo/Meister workflow using perl scripts. This results in a continuous integration for perl development.

My environment currently is a build and deployment system that uses Meister for build and workflow management, CA Harvest for version control and high level application lifecycle workflow management and about 100 perl scripts used for deploying all sorts of Java applications and performing validations of various sorts. This system supports several hundred Java applications. Because of the large numbers of applications, automation can only be achieved by adhering to standards and naming conventions, much to the dismay of developers.

I was planning to dive in with the deployment and validation scripts, but I’ve been slowly building up a change control and testing system for the perl scripts themselves and I was confronted with a more simple problem that is perfect for starters in this series. I’ve been improving the testing of the perl scripts by implementing unit test scripts using the Test::More perl module and relatives. We have three environments for perl development, testing and runtime. I’ll call them dev, qa and prod, for short.

CA Harvest is a big database storing every version ever created in the company. Although it is mostly used for Java code, we the build team are using it to manage our perl scripts. At each state in the lifecycle: dev, qa and prod, there is an associated file sytem area that is synchronized with what is in the database. This synchornization is done via a perl script so that when we ‘promote’ code in Harvest from dev to qa, the qa file system area is synchronzied with the qa view in Harvest of our source code. (One of the nice things about Harvest is its ability to trigger back end automation of your favorite script on certain actions, like ‘promote’.) My goal is to trigger the automated testing of those perl scripts after synchronization with a script using Test::Harness.

All perl scripts executed by Harvest are executed on the Harvest server itself. The first problem I had is that some of our perl scripts are executed on the Harvest server, and another set is executed on the physically different build server. I needed to run two sets of perl scripts on two different machines. This is just for the qa environment. I also wanted to do post-production deployment implementation verification (testing what you just put into production) and there are currently two build servers, meaning the scripts need to run on two build machines plus the server. Since the build machines are geographically separate (different network connections) and they could have slightly different shell profiles, I don’t want to assume the test results would be the same.

The problem I have with Harvest is that Harvest will execute a series of post action scripts, but 1) it will execute them in series, 2) if any preceding step fails, it will not execute the following steps.

Currently my synchronization script actually fails, due to a few files that shouldn’t be overwritten. These files are configuration files that must be different between dev, qa and prod, and it is my TODO to clean that up. Rather than tweak the perl synchronization script to make it pass, I need to reorganize the files in Harvest - no small task. So I want they syncrhonization script to fail, but I still want the unit tests to run. That is problem A. Harvest will detect the failure in the synch and exit without running the tests.

The second problem is (B) I am extremely impatient, and why shouldn’t I simultaneously execute my perl tests on the two or three different machines instead of waiting for them to finish, one-by-one. The Meister workflow (same as Mojo) allows me to A) ignore the return code of the synchronization script and B) execute all of the unit tests in parallel on the different machines (via ssh if necessary). For gravy, I had the workflow wait until all the tests were complete on all the machines and then send an email with the HTML log URL.

So, I had Harvest call a single perl script that launched a Meister worflow, which in turn executed the synchronziation, ignored its return code, and the split into parallel unit tests, wait for all the unit tests to complete on all the machines and then fire off an email.

This really saved a lot of tedious repetition in my development process and I am excited about using it. We are likely to grow both the number of build machines and number of perl scripts as time goes on this system is nicely extensible.

I just wanted to mention how much I love the XML::Twig and Archive::Zip perl modules. And, why perl, anyway?

I’ve working on some custom perl scripts to validate deployment descriptors and I’ve found that using Archive::Zip to identify and extract XML files and then using XML::Twig to parse them is vastly superior to having perl call shell commands (yuck!). I’m a serious minimalist when it comes to designig and writing scripts and I love the simplicity and brevity these modules allow. These scripts are called as workflow activities in Meister and it is a pretty sweet setup.

Some developers I’ve worked with have asked “Why not Java instead of perl?” Briefly, perl lends itself very well to doing the simple, dirty work of pushing files around and reading, writing and parsing text. Since perl is compiled at runtime, the scripts are transparent, easy to change and you don’t have to compile it and deploy it somewhere. This translates into greatly shorter development times for the build system. This is important because it’s the software engineers writing the business applications that get the big bucks - no one wants to pay for infrastructure improvements. Java has its place too in Mojo and Meister and so does C/C++. All three languages are cross-platform and are used for what they are best at in the tools.