There has been some talk around using physical, colored, lamps to notify build results (pragmatic automation, Carlos Sanchez, Richard Durnall, Alberto Savoia).

The idea comes from Lean Manufacturing principles developed by Toyota applied to software development. An “andon” paper lamp was a way to let other workers know there was a problem on the assembly line. The idea for builds is that while a build is running, an amber light flashes, letting other developers know not to run a build, a green light signifies the last build was completed successfully, and a red light means the build is broken and needs to be fixed.

While I’m sure it is an entertaining exercise to set this up, some simple software automation could achieve the same results, and it has obvious faults, like what if I am working from home?

Email is the de facto standard for build notification. There is a hardcoded mailing list somewhere and you get the email, even if you don’t want it, and you have to go and check your mail to receive it.

Twitter, on the other hand, puts the user in control of the notifications in true Web 2.0 fashion. Tweets are also ideal for receiving notifications via cell phone, where email is generally not. Consider the following cases:

• I can follow the build when I want, and unfollow when I want. No going to an admin to remove me from the list.
• I can turn on device updates so I get an SMS message about the result (maybe an important release is coming up), or I can turn off device updates, say, when I’m on vacation.
• I can retweet (forward) the build notification.
• You can put links in the message. Meister uploads all build logs in HTML format to a web server and we would include the link to that.

About a year ago, I used Meister with a post-build activity to Tweet the build result after a build. The activity was a simple Perl script using Net::Twitter. Let’s just say this experiment sort of fell on deaf ears.

Now I’m ready to roll with this again and you can follow my progress. I had created a Twitter account, @builds, for this purpose. In a day or two, you’ll see some tweets from that guy from one of my builds.

I’m interested in what other people have to say. Meister already uses Eclipse RCP as it’s front end, but most other build tools are still command-line and in the 1970’s. Let’s bring build management tools into the 21st century with Web 2.0 features.

With our third nomination, we won the Jolt Award for product excellences in the change and configuration management category. You can view the announcement here.

It is just plain fun to run parallel workflows and builds and watch the activities and build steps light up the workflow monitor in real time like a Christmas tree. See this flash demo to see what I mean.

As customers go to machines with more and more cores, fewer machines are needed in the application lifecycle infrastructure, particularly for builds and code retrievals - the most resource intensive functions. This is helping to simplify the infrastructure, reduce maintenance and administration and drive down costs.

Several of our customers are running around 5000 builds and non-build workflows per month on two machines. The primary reason for two machines, in fact, is for disaster recovery, and the goal is to run both machines at less than half capacity so that in the event that one machine (or datacenter) fails, all the current capacity can be run as a contingency on the one machine that’s left.

Thread control is very simple with Meister and Mojo. Both use the omsubmit dependency manager program to handle this. Meister’s om program translates build events into workflow steps using omsubmit. The OMSUBMIT_MAX_USER_PROC value sets the maximum allowed number of threads.

You might think that if you are running dual, quad-core build machines that you should set the max threads at 8. However, Meister posts build operations to one thread and the associated logging operation to another. Compile operations notoriously use a lot of memory and CPU resources, but the logging operation posts to a server and waits for the operation to complete. There is really no disadvantage to setting the max threads higher than 16 in this case, so go ahead and do it.

As a non-build workflow example, I worked on JBoss deployments to 48 Linux machines. The workflow was parallelized into 48 activities each of which deployed to a single machine in parallel. The deployment activity was largely a remote execute operation that extracted archives on the remote machine. The extraction took about a second for a medium sized application. Again, this is a waiting situation where machine resources are essentially idle while the thread is in use, so use more threads. The machine was a dual, dual-core build machine and we set OMSUBMIT_MAX_USER_PROC to 50.

Watching the workflow monitor as the deployment ran, we could see roughly half of the machines light up (meaning actively running) at any one time and the entire deployment process synchronized all 48 machines in a little over two seconds.

So, don’t simply match your machine’s CPU threading capabilities - overclock! Aim high for max threads and try to determine where your performance is optimized. I’d love to provide you with some metrics as a function of thread count, but usually once something is working it’s on to the next project. I barely have enough time to blog!

It’s very common to have a code check-out step be part of an integration build. Far better it is to not check out code before a build. What? How is that possible?

Let me explain, Fred. The simple approach most of us take (and have to take when getting things started) has developers commit, commit, commit, and when it is time to deploy, check out the code, do a build, and then deploy the application. There is room here for both problems and optimizations. Doing a full check out of the code tree is more costly in terms of time than checking out only what has changed. Updating the code tree with a single commit is less costly than updating with a large number of commits.

You may be limited by the technology in-hand and how much you’ve invested in learning the technology and possibly customizing it. For example, if your file control tool can only do a full check out of a source tree, or that’s the only command you had time to implement in order to meet the deadline, or you don’t trust your tool to do incremental updates, then you are basically running the longest builds possible.

On the other hand, if you could update the code tree every time a developer does a commit with only the changed files, then you are ready to execute a build at any moment. This requires some deft manipulation of your file control tool, and that’s why you don’t see it more often.

You might think “continuous integration” will take care of this. Developer commits, update checked out, build is run. However, you may end up with a build, test execution and deployment that takes longer than the typical time between developer commits. You still have to do incremental updates and it only solves the problem in cases with very low developer activity.

I’d like to point out one tool that does an excellent job of post-commit code checkout, CA Software Change Manager for Distributed. CA SCM (for short) is the tool, formerly known as Harvest, from the company formerly known as Computer Associates. CA SCM is a highly scalable (1000’s of developers) file control tool with a great lifecycle process model. We at OpenMake Software still have our very first customer still using OpenMake/Meister with CA Harvest/SCM after 11 years. While we have a reseller arrangement with CA, our partnership with CA in services has extended to 14 years.

About 10 years ago, OpenMake Software developed an integration with the then, Platinum Technologies’ Harvest product, modeled after the now dead Computer Associates product, Endevor Workstation, that had an excellent post-action code tree update. (Endevor for z/OS, a.k.a CA SCM for Z/OS is still very popular and has a similar functionality called ‘output libraries’ - following all this?) Our integration had the horrific name, ‘Har-refresh’.

As product partners, we finally transitioned Har-refresh from an external add-on to CA who have turned it into a core functionality of the product, called Hrefresh (a better name.) Rather than simply a post-commit check out, HRefresh updates the code tree after any action that updates a dynamic code view. This includes, renames, deletions, commits and code promotions and demotions. We like this because CA SCM does all the work and we cherry-pick sets of up-to-date code trees to build up an application source code stack for a build. We align Meister dependency directories with HRefresh-managed file system directories for a tight SCM (software configuration management) build.

This mechanism distributes the resource load for checking out code to times when builds are not required. It’s true that often times people want to build as soon as their code is checked in (or promoted), but on average it is a very big net win reducing build times.

This is just an example of the type of sophistication that is out there to prevent pre-build code check outs and save time on your builds.

One of OpenMake Software’s product strategies is to keep things simple. Build management is one of the most complex operations in all of the IT world, and one of our key benefits is to simplify, organize and automate the build process for development, testing and production.

We’ve seen a trend among our customers to simplify their build management infrastructure by going to fewer build machines with more CPU cores. Builds in particular use relatively more CPU resources than other resources as code is interpreted and compiled in memory and then finally written to disk. By reducing the total number of machines, rack space, procurement, administration and other IT overhead costs are reduced at great cost savings per machine eliminated.

Recently, I was at one of the big chip makers where they used dual quad-core CPU Linux machines for their development and builds. They had two machines and were able to control access to allow separate areas for development, testing and release builds in keeping with best practices. Having all the horsepower of 8 CPU cores on a single machine kept them from needing more machines.

Another customer does 6000 builds per month with Meister on just two build machines.

IBM, when selling BuildForge, likes to talk about big build server farms, because their tool does remote execution on multiple machines, as does Meister. However, BuildForge does not do builds at all. It can remotely execute your existing build scripts, but there is little real value add to that. BuildForge is also famously expensive. What happens over the next few years to the high investment in multi-machine remote execution software as the number of machines declines, perhaps dramatically?

A similar argument can be for Electric Cloud’s Electric Accelerator product. It’s possible in some cases, for C/C++ builds to gain an edge by pushing a compile operation to another machine, and then bringing it back. You would only do this to gain access to additional CPU resources. In the past, you might have 8 build machines that Electric Accelerator would farm operations out to. Now, you can pull all those operations into a single machine and there is no need for that functionality. Also, you are stuck with converting your GNU makefiles into other GNU makefiles.

Meister is optimized for multi-core CPU build machines and offers multi-threaded capability to both build events and non-build workflow events. You know where your build is and there are fewer dependencies on network resources. Both BuildForge and Electric Accelerator add additional overhead to build administration to coordinate across multiple machines - a dying practice, that no organization wants to invest in. Meister is the best bet for a future with fewer build machines with more horsepower.

I’ve recently been learning the Ruby on Rails framework for web development. It’s become a quite popular framework for getting database connected websites up and running relatively quickly. One way it is easier to get a site started than with other frameworks is because of the Convention over Configuration mantra that it lives by. Instead of requiring loads of configuration files to build a basic site (that can grow and become quite complex by the way) it has a feature called scaffolding which automatically builds your Model, View and Controller classes based on tables it finds in the database. It can do this by making assumptions based on standard conventions about interacting with a database from a website and naming and using classes in a standard way.

Although I am still a rookie when it comes to understanding the many facets of Ruby on Rails, I have really been trying to emulate the Convention over Configuration way of doing things in my various build/release projects at customer sites. One problem I inevitably encounter in most organizations is that the development of build and release methodologies has been left to the various development teams and not been thought about holistically using a centrally managed approach - this leads to little to no standards and Convention over Configuration is chucked aside. Not only is this inefficient from an organizational standpoint - why reinvent the wheel over and over again for each team when they are essentially tackling the same sets of problems, but it also makes for a nightmarish audit trail that could get you into trouble.

One reason this happens so frequently is that the managers that are supposed to be in charge of standards for building and releasing applications are often not privy to the kind of technical requirements that the various development units have when it comes to putting together and delivering their applications. And when developers try to explain the requirements, the standards people may get lost because they can’t possibly understand the nuts and bolts of every application.

To pull off real centralized management of builds and deployments, the standards people need to take a deep breath and rethink their objectives - start looking for the commonalities, not the differences between applications. In doing this, they will find that that problem that that developer told you was so unique and must be solved a certain way is probably very similar to the problem the other developer told you about last week - or just look on the web and see how many thousands of external developers have this same “unique” problem. It turns out that most applications can be constructed in the same type of way. Just because the source files are different between applications doesn’t mean that the paths they take to their target executable, dll, Jar, War or Ear file is very different at all. And when those paths are essentially the same - create a reusable process that the various teams can share. Use Convention over Configuration as your guide - standardize and centralize the common processes and externalize the technical specifications using highly modularized control files.

Here’s a simple task for you to try. This assumes all of your application teams have their code checked into a central repository - if not, you have bigger problems than standardizing builds and releases and should address those first. Look at your various technologies, whether it’s .Net, Java or some other and try to identify where the code tree’s start under the root of the project. You’d be amazed at how many teams check their .Net solutions into different levels of a code tree for no good reason, or Java teams that have their their source packages buried some place in the code tree. Next, look for the common root starting point for all these application types and try to come up with a simple standard based on this information. Finally, notify those that are not following that standard that you would like to move their code up and over to this new location (its usually up and not down) - it should actually be pretty easy to do. After this has been done, you can now have all build and deploy scripts use a standard root variable to find dependencies (think something like SOURCE_ROOT).

It always amazes me how many teams don’t standardize simple things like code tree start points in their source projects - it equally amazes me how much mileage you can get just out of making simple path standardization adjustments. After you’ve worked on the source tree, try doing the same with your common libraries. This isn’t rocket science - just remember, Convention over Configuration makes everything easier.

Adam

Finding the blog Enterprise Maven made me decide to go back to the basics, today. This blog is from 2006, but the best practices of production control ignored here go back decades. I’d like to point out that Oleg Gusakov, the author, wrote the blog in a very good spirit and seems like a nice guy. He just seems to be a bit naive about what’s been happening with software development in the enterprise.

In the first section, he assumes that the only enterprise build and deploy solution is one that is customized, while OpenMake Meister has been serving that role now for 12 years. He does correctly conclude that all the enterprises in the world should not be independently investing in the same type of build and deploy solution. It is a costly investment and this functionality should be productized. That’s exactly why we did it and why that is still one of our chief selling points.

He is right that developing a product that should be commoditized is a drain on the business. However, the converse, having a commercial product provide the functionality at a greatly reduced cost compared with one homegrown, provides a competitive advantage over those companies who don’t have such a product.

Through the middle of the article, again, I think Oleg is unaware of the heavy horse SCM products out there that provide a lot of the expected functionality. Tools like CA Harvest, Serena Dimensions and others are very complex and sophisticated n-tier products. They nevertheless do not provide build support, so by combining an enterprise file control tool with an enterprise build and workflow tool, Meister, you canvas the required functionality.

Lastly, regarding the enterprise development lifecycle, he is right it is an oversimplification. I like his phrase that he hopes to “grow the meat.” At OM Services, we have “fully grown meat” and the enterprise lifecycle documents that we develop with our customers and clients are typically 50-80 pages in length. Here is where I review the generally accepted best practices, going back to the seventies with mainframe development. (NO, distributed platforms are not somehow different in the high level process!)

1. It all starts with production control. Developers do not have access to production due to a fundamental conflict of interest. Maintaining business continuity trumps developers’ ease of delivery to production.
2. Since someone else puts the code into production (or operates a tool which does so), this is the basis for separation of roles and responsibilities in the enterprise software development lifecycle.
3. To ensure integrity of the production environment, the production build must be done by a group representing the business, not development. Developers do not do production builds.
4. Working backwards, if you want your test environment to be as close as possible to production, you lock this down and prevent access to developers. This is usually the QA testing environment.
5. Again, to avoid a conflict of interest, the QA testers should be working for the business, not the application development team.
6. And it follows that the build for the QA environment is done by the business.
7. The developers job from this perspective is delivering source code to the business, which wants retain the source code and the ability to use it (meaning they can build it).
8. The process of developers transferring source code to the QA build people was called “throwing it over the wall”. Now, the heavy horse SCM tools and Meister workflow make it easy to do this and allows variations of iterative development involving the QA environment.

Any type of continuous integration or agile development practice typically happens before the QA environment. Any develop methodology for the enterprise must take into account the fundamental conflict of interest between software change delivery and business continuity or ignore it and remain entirely in front of QA.

If you are a developer, you can think of this as a loss of privilege, or you can be elated that other people are doing the dirty work for you and you can focus on the art and science of engineering business solutions. If you are really depressed, maybe you should be on the other side of the wall!

We’ve been testing Mojo and Meister 7.2.1 and getting ready for their release on December 15. This is a maintenance release with bug fixes from users who’ve started running builds and workflows with Meister 7.2 and using the free workflow automation of Mojo and putting those releases through their paces. It also contains a lot of UI and documentation improvements.

Existing users of the 7.2 version of either Mojo or Meister will be able to upgrade via the update sites, http://www.openmakesoftware.com/mojo/update_site and http://www.openmakesoftware.com/meister/update_site, respectively.

Users interested in getting Mojo, the free workflow automation tool, and Meister, the industry leading build automation tool, can find download instructions on our website, http:///www.openmakesoftware.com.

We’ve heard from a number of companies that they are having problems with their WebSphere Eclipse headless builds and they are looking to us for a solution.

A headless build is an Eclipse function (and therefore an IBM WSAD/RAD IDE function) that allows you to execute Eclipse and some of it’s plug-ins at the command line. The GUI does not launch, but the main machinery does, and it makes use of the workspace metadata.

In theory this allows you to do the same build at the command line that was done in the GUI. It’s a nice theory that we subscribe to ourselves.

In practice, these companies do not feel they are successful for a number of reasons:

1. If it works it is slow. A slow build is almost as bad as a broken build.
2. Unexplained crashes in the middle of the build due to workspace corruption.
3. Crashes may leave the .lock file in the .metadata folder of the workspace. This prevents Eclipse from executing in the same location until it is cleaned up.
4. It’s a black-box build with a large number of operation swept

Meister takes a different approach and records the files and project types and then applies a traditional build with a mixture of Ant tasks and command line calls. This eliminates the need to even install RAD 7 on the build server, but you still need the runtime libraries.

I’ve worked with the IBM WebSphere EJBDeploy plug-ins and integrated Meister to actually call an Eclipse headless build via the WSAD/RAD ejbdeploy.bat file. As a rule, we always stick with the vendor-recommended approach, so we don’t have a choice. However, we don’t do the compiling and archive this way - just the EJBDeploy step.

By compiling and archiving the Java parts in a traditional way, we at least minimize the risk of a broken headless build and maximize the speed. And, yes the ejbdeploy step is usually the longest step in the build. The fact that IBM supplies the ejbdeploy.bat file at all, which mimics a command-line compiler, means we’re not the only ones taking this approach.

On a related note, people are still having trouble with make. Here’s a note about case preservation problems when doing cross-platform builds. Well, the problem is implied, given that this is a post describing a solution.

Given that this functionality has been missing from make for decades of doing cross-platform builds, I have to question if this is really necessary. Is the root cause of the problem something else?

Here comes buildr: yet another Java build tool. Hopefully I, or one of my other cohorts will check this out in detail soon. But, with my experience working with all manner of build tools, with 100 companies and many more development teams, I can already make a few observations.

First of all, why another build tool for Java? I am occasionally told that Maven or Ant is a perfect tool, but clearly the people behind buildr don’t think so. The choice of JRuby as the vehicle for delivering this tool, I think is probably a good one. JRuby is a scripting language in the same vain as Perl, which is used by Meister.

Doing software builds is an ugly business involving lots of file and operating system interaction. This is not where Java shines, but scripting languages can. As long as operating systems are written in C and not Java, C-like tools will be better and faster at interacting with them. Plain Ruby itself is C-based, and JRuby no doubt inherits C-like operating traits. Calling out to a Java compiler from Perl or JRuby, though it has its own JVM, does not represent a significant overhead compared with the file system operations and the compilation/translation itself.

Both Maven and Ant are relatively difficult to extend compared most other build tools, and I’ll be buildr beats them here. If you have all the Maven plug-ins and Ant tasks you need, then good for you. If not, then you have to start developing in Java and it becomes too much of an investment to sink into a build system. It is much cheaper to extend in JRuby or Perl. My frequently cited example is the XMLBeans compile step in Meister, written in Perl, which is only 40 lines of real code. The Maven plug-in is 60 pages of Java code and no one can tell me really what it is doing (I asked on all the forums). Less code is usually more transparent, which is also good for build audits.

I am a little disappointed to see them try to placate the Maven and Ant users by promising it is a drop-in replacement for Maven and they have all the Ant tasks covered. Both tools have their drawbacks and I don’t want to see another tool with the same deficiencies. They should have the cajones (or coñejos) to apply all their resources to what they think is a better tool (with its own unique benefits and deficiencies). I imagine offering Ant task equivalents is pretty easy because of the ease of coding in JRuby compared with Java.

They also don’t mention who is supposed to use the tool. Is it for individuals, small development teams, the enterprise? Maven falls short because it is only appropriate for development teams and not for stable, controlled, enterprise builds. Ant is not even tool, but a means to create some tools for small teams. I don’t think Meister will fear buildr either.

Well, since buildr is only in incubation status with Apache, I’m not sure how much time I’ll be able to spend on it, but I am curious and I’ll let you know if I find out more.

As a follow up to my article on automating XML updates, I’d like to report that I did use Excel and Perl’s XML::Twig to successfully generate XML descriptors for my web service consumer, and it was a lot easier than I thought. I’m using XFire 1.2.6 web services stack running under JBoss and using MyEclipse IDE 5.0. I’m happy to say I went from blank spreadsheet and no plan to generated XML files from spreadsheet values in one and half hours. The implementation is of course expandable and reusable. This implementation should work for WebSphere and .NET as well.

I needed to create different configurations for my web application so that the service request went to different endpoints for different environments. The endpoint is at an enterprise service bus (ESB) and there is a different ESB for each environment. I need to have my ‘dev’ instance of the consumer hit the ‘dev’ instance of the ESB, the ‘qa’ instance of my web app hit the ‘qa’ instance of the ESB, etc. We’ve set up Meister to pick up the correct XML file for the target environment for the build of the WAR.

I started by setting up the spreadsheet as follows. I had an unnecessary column for Host indicating JBoss, but I hope to include WebSphere and maybe .NET as well some day. My web app actually connects to two services a.k.a. providers, so there is a column there. And, next is the configuration label for my web app with the name corresponding to the environment it is designed for. So, the first three columns of the spreadsheet look like:

Then I needed a way to indicate the resource that would change. Right now I only have XML files, but I chose to stick with a generic URL for that. Unlike Maven or Ant generators, we start with an XML file that actually works and has been tested - not some hacked up parameterized version that takes additional effort to create. The fourth column of the spreadsheet looks like the following (with repeated entries omitted):

Next, I needed a way to specify a target location to change within the XML file. Now, I know I’m going to use XPath, but I’ll want this to one day work for properties files as well, so I came up with a URL-like thing called a Universal Datum Locator (UDL) which pre-pends the method of locating the datum to change on to a method-specific locator. It could be a property name, an XPath or a Perl regex, for example. In this case it is XPath and then the last column contains the replacement value for the datum indicated by the UDL. XPath is also very intuitive and easier to construct than it may look.

The value for the UDL looks like:

xpath://beans/bean[@factory-bean='xfireProxyFactory']/ constructor-arg[@index='1']/value

So the fifth column contains the UDL’s, which in my case is always the same XPath expression. The final column of the spreadsheet contains the replacement value of the datum indicated by the UDL:

My nifty Perl script is only about 80 lines of real code and because XML::Twig is nearly the best thing in the world, I pass the entire XPath in as a hash key to modify the source XML file:

my $twig = XML::Twig->new(

pretty_print => ‘indented’,

twig_handlers => {

“$xpath” => sub {

$_->set_text($new_datum);

}

}

);

Here, “$xpath” is directly from the “UDL” column of the spreadsheet with only the ‘xpath://’ stripped off and “$new_datum” is directly from the “Value” column. That’s a pretty useful one line subroutine if you ask me. I had the new XML files each generated into a different folder (dev/,int/, etc). Then, I checked them into version control (CA Harvest) and built each of them with Meister. If you want the full code, let me know and I’ll post it somewhere.

I did find working with the Excel 2003 XML Spreadsheet format a tiny bit awkward. You have to keep track of the column and row indices, but not bad other than that. I see Microsoft Word 2007 allows you to save as an XML document directly, but you apparently have to define bindings. I’ll have to check that out.

In developing Java applications for multiple server environments (e.g. dev, test and prod) there is a common pain-point of having to manage deployment descriptor or configuration files specific to each server. For example, you may have an XML log4j configuration file with some parameters different for different server environments. You may want to turn on debug messaging for the development server, but turn it off for production. At the same time, the Java source code will (eventually) be the same in production as it was in development. A similar situation applies for .NET application development.

Like many build management tasks, managing these environment-specific files is generally left to either manual or some type of scripting. This is really something that needs to have a high level of automation applied. Particularly in larger environments, much like scripted build management solutions, existing tactics fall short. This situation is in a far worse state than even the compile part of build management. It is not enough to simply have a script that can spit out some files. One of the biggest problems is information management and the fact that parameter values in the configuration files may be determined by different teams! How do a production engineering team and an application developer both feed inputs into the same XML file?

I’ve worked on this problem for several years and with a number of companies. The critical functionality can be broken down into two different items – information management and a processing engine. In an effort come up with something better, I’ve done a review of what’s out there and here is what I came up with:

• Ant ‘filter‘ task: As with many Ant tasks, this works great if you are an individual with a few items that need updating. It is a nightmare if you are working in a multi-team enterprise with multiple server environments. The main problem is that you have to constantly take working copies of XML files and insert a token for Ant to later re-replace. This leads to a management nightmare to synchronize parameterized copies of XML files with their working copies from the desktop environment. The advantage is that it works for any file type so you can use it for properties files as well as XML files.
• OOPS Consultancy Ant ‘xmltask‘: This is a good engine for specifying and performing changes to the XML and has a full feature set. In fact, we use this in some of the Meister build services. The problem is that it is only for Ant and therefore you have all the reuse, standardization and hard coding issues. Xmltask can provide part of the solution we are looking for, but we still have an information management problem to deal with.
• Maven: Maven has what is essentially the Ant filter task. The specifications are abstracted in the pom files, which is better than Ant, but it encourages templating of configuration files leading to all the problems associated with that (synchronizing templates with working files, testing templates, etc.)
• XML:DB XUpdate: This is a working draft of a specification to encode XML update instructions into an XML document. There is a Java implementation of XUpdate listed on the site called ‘Lexus’, but I couldn’t find anything on it. Since the build management task requires us to generate XML files, I’m not keen on generating XML files using xupdate tags that will allow me to generate other XML files.
• Perl XML::Twig: This has worked wonderfully for me on a back-end web services security effort and I could not be more happy with such a precise, elegant and brief XML library, which includes XPath. This is not a solution for Java or .NET developers, but it could serve as an engine to mimic xmltask or implement the XUpdate specification.
• Excel. Yes, I’ve seen Excel used effectively as the information management front-end to updating the XML. It is a convenient format to share among teams, it is centralized source of information, it can be checked into version control and it can be saved as an XML file itself for processing by another engine. In a large environment, you may have 5 or more server environments, lots of different components to configure, so you could have literally hundreds of parameters to manage. Excel gives you a nicely transparent way to view those values.

Ivy seems to be trying to resurrect Ant, when it should stay dead in my opinion (unless you are a lone developer on the simplest project). Maven is a more proper build tool with some workflow functionality that falls short outside of a small team of developers with no audit or control requirements. Both are vying for prominence as the open source build tool, yet trying to settle into compatible niches like two brothers hopeful for the throne.

Here is what Ivy says about Maven: http://ant.apache.org/ivy/m2comparison.html

And here is Maven’s side of the story: http://docs.codehaus.org/display/MAVEN/Feature+Comparisons

I think it is funny that Electric Cloud is saying that Perl is and old, outdated language (presumably to position themselves against OpenMake and Meister) when they are based on make, which truly was designed for C and is not very extensible (but works for ad hoc build scripting). I can add FTP or email support to a Meister build method in less than 10 lines of Perl code - not so easy to do with make alone. I can even update my Facebook page with the latest build result through a Perl API.

Most developers know Perl has a recognized place in most companies. There are trends in software languages, and Perl may not be the latest, coolest thing on the street (like Ruby or Python) but Perl is incredibly dynamic and robust. It is possibly the greatest single example of successful open source development. Neither Ruby, Python nor anything else besides the open source Java community even comes close to offering the breadth of functionality that Perl does. Perl even does  automatic release dependency management - something unique for its scope in the open source world, although the Maven project is attempting to do something similar with Java.

Java developers often think Java is the only thing in the world, but in comparing Maven (an all Java build tool for Java builds only) and OpenMake, it took 60 pages of Java code to do the same thing (see my previous blog on XMLBeans) as 1.5 pages of perl code as used in OpenMake.

Perl works great at moving files around and working with the operating system – Java is not so good at that. Meister and Mojo use Java for the server which works well, but Meister uses Perl where it can be used best (parsing texts and interacting with the operating system) and C is in turn used where is best (C can’t be beat for speed). We also use make for what it is does well - the rules engine and dependency ordering. Combining the C based make engine with Perl for executing commands really gives enormous power to Meister.

I heard this from some developers today who confirmed the existence, by indirect evidence, of willful tampering of Java archives by gremlins using Ant. There is speculation that they live inside their own plug-in inside WebSphere Studio Application Developer.

Developer A, we’ll call him Nick (because that’s his name), came into our room to work out some tweaks with Openmake targets. He told us all that they’d been working on some problems with runtime errors getting the dreaded Major-Minor version error: “Unsupported major.minor version 49.0.” This usually indicates that you have compiled something with a 1.5 JDK but are running it under a 1.4 JVM.

The archive they were deploying was created wholly inside a workspace managed by WebSphere Studio Application Developer (WSAD) but built by Ant using IBM JDK 1.4.2. Now, WSAD by default launches under a 1.3 JVM, so there is no hint of a simple conflict of JVM’s. They actually traced the issue down to a few class files in the deployed archive that had a 49.0 class versions by using a byte editor. So only a subset of the classes compiled by Ant were affected! But, if they created their own Jar file at the command line through the Windows command prompt using the same exact files, the Major-Minor problem didn’t appear. This means the classes were altered at the packaging step - NOT the compile step!

Harking back to my pain with multiple versions of Java over the last decade, I interrupted this fine tale and asked with a haughty tone “Is a 1.5 JDK installed anywhere on the machine?”

“Yes”, he answered, and the problem, he said, was resolved by uninstalling the 1.5 JDK. We all then had a nice knowing chuckle. Then we questioned Sun’s use of the registry in Java and how the use of the registry changed with each release, forgetting, in fact that the problem happened under an IBM JDK. We also talked about the practice of dumping of java.exe under the windows/system32 (%SYSTEM_ROOT%) directory and how that could trip you up. You never knew if it was a Microsoft java.exe or a Sun one, and if it was Sun, which one was it?

Has anyone heard of this happening before? Is it a bug with Ant? Meister uses Ant under the covers for some things, but not for the Java compile. We’ve never heard of this problem before. We are, in fact, working with Nick to roll out Meister (and the Java build services used by thousands) to Eclipse at the desktop so mysterious problems like this won’t happen in the future.

This article is mainly dealing with Haskell, but if you read on ’til the middle of page 2, there is a good description of make from a language perspective.

Adam Turoff article on ONLamp.com

I received a call from one of our development build meisters … there is a build issue that needs to be resolved for a complex application - this application was building successfully for years without any problems and suddenly it stopped working today. I looked at the error snippet that he had emailed me from the build log…

[cc lang="java" tab_size="2" lines="10"]

compile: [javac] The system is out of resources.

[javac] Consult the following stack trace for details.
[javac] java.lang.OutOfMemoryError

[/cc]
The first thought that went through my head was “I know how to resolve this - I have seen it before”! I knew that the build tool uses Ant under the cover for Java compiles. All I have to do is increase the memory using ANT_OPTS.

This problem was happening on an Unix machine – this was one of our controlled build machines for our enterprise java builds. Before I changed the environment variable on the build machine, I went through the usual drill…

I asked if anything had changed significantly with the application today - I heard that there was no change other than a few java files were added. I asked him if his other applications were building – he said that everything else has been building successfully except this one application. I asked him if we are able to reproduce this “out of memory” problem - he tried the build for the same application and the compile failed at the same step.

Just to be sure, I requested the on call unix administrator to see if there was any memory problems reported at the system level for that machine. The machine didn’t report any errors of being out of memory or high swap at all that day.

It was time to set the ANT_OPTS environment variable! I added “export ANT_OPTS=-Xmx512m” to the .profile of the system user that is used for all controlled builds on that machine. The development build meister tried the build again - it passed the compile step where it had failed consistently today! The build was successful once again for this complex application. Problem solved!

When I made the transition from physics to build meistering and scm/build management consulting, I was surprised by my first glimpse into the firey pit of GNU make build management at a local Chicago financial services company.

I knew shortly after entering graduate school that I didn’t want to be scientist. Yes, better to find that out before, but it was interesting and I stuck it out, painful though it was. I heard a WWII fighter pilot say something on the TV show “Dogfights” (one of my favorites) that rang true for me. “I wouldn’t trade the experience for a million dollars, but I wouldn’t give one red cent to do it again.”
>
> I ran into the co-founders of OpenMake Software at a trade show in Chicago over 10 years ago and it was an easy transition for me to go from small lab of five to a small company of three, as the first hire. Steve Taylor and Tracy Ragan were still doing consulting at a Chicago credit card company and one day I was in the “office”, when the phone rang. Before I knew it, Steve and I were in the car heading over to help out with a critical build issue.
>
> I met Selvi Dayabaran, who was the poor sole whose responsibility this problem was. This company didn’t have OpenMake 3.0 or whatever it was at the time, but most of their application development was in C/C++ and COBOL on the UNIX side with GNU make to manage the builds. There was also a “makedepends” process which scanned all the C code and appended the list of headers to the makefile so their timestamps could be checked for incremental builds. Before I knew it, Steve a.ka. “F1″, was running GNU make through a debugger.
>
> “Every undergrad at MIT who does a thesis on builds adds his own code to GNU make and the code is really a mess!” Steve said. Steve found the problem, updated the local copy of the source code, they rebuilt GNU make and ran the build successfully.
>
> Having used make for awhile, I was surprised that the bug was enough to prevent a build from happening. I was also surprised by the magnitude of the software application. Hundreds of header files were processed by makedepends. The early versions of OpenMake actually used GNU make, but one day a year or so later Steve said he was writing his own dependency engine because “it is easier to fix the bugs.” I thought back about debugging GNU make and it made a lot of sense.
>
> Most importantly, I met Selvi, who became hire #2 at OpenMake Software. We’ve now worked together for over eight years and I’ve learned more from her than from GNU make.
>