In March of this year, OpenMake Meister won the 2009 Jolt award in the change and configuration management category.  Last week at TechEd, OpenMake Mojo Max won the 2009 Code Project Member’s Choice award in the ALM category.  This gives OpenMake Software two important developer based awards for 2009.  Looks like more development teams are starting to understand the importance of managing both workflow and build automation as part of their development process.

OK, I often see the term “distributed builds” used to refer to processing the workload of a workflow management tool.  So lets get some terms sorted out.  First, a distributed build is one that takes the calls to the compilers and linkers, sorts out the dependency relationships and then builds them across multiple machines.  A workflow management tool cannot do this - and most of the so called build management tools on the market are only workflow management tools even though they claim to speed up your builds by distributing them across multiple machines.

A workflow management tool is only as intelligent as the scripts it is being asked to execute.  Build management tools that function like a job scheduler simply takes a command an executes it on a particular machine.  If that command is a make, maven or ant script, it is going to send that script to be executed on a particular machine.  That is NOT Distributing the build. It is distributing the build script.  Build scripts are rarely written to support distributed build processing, so your workflow solution cannot magically turn that script into an intelligent process that can be distributed.  In fact most scripts have no understanding of dependencies, so there is no way for them to know what order to build the application, much less distribute those calls out to external machines.

OpenMake Meister does understand dependencies and does have the intelligence to know the order in which to build artifacts.  In fact, out of the box, it does this for your builds, on a single machine using multi-core processors calling the compilers in multi-threaded mode.  So if you are looking for speed and need to distribute builds across multiple machines, look no farther than Meister.  This is what Build Automation is.

It seems that the “agile development” methodology is getting more and more press.  Some of that press is asking the question “does agile really work”.  I certainly cannot answer that question, but I know for one thing, Agile processes have brought the build state of the development cycle to front and center, where it should be.  Even if Agile development methodologies do not improve the requirements gathering effort, or improve testing results, I can say one thing is for sure - Agile has improved the build process because it is uncovering how unsophisticated the build process is on the distributed platform.  My prediction is that as the Agile process blooms and grows, the build process will bloom and grow as well.  Developers will begin to see how important it is to automatically synchronize IDE Builds with external team builds in order to accomplish consistent and continuous integration.  As this occurs, the answer to the question “has Agile has really improved development” will be clear - an emphatic YES!

As a vendor of build management tools, it gets a bit frustrating when I read articles that glorify the transparency of any SCM tool.  An SCM tool cannot provide an audit of the build that shows all of the artifacts used in the build. PERIOD!!  Yes, you can run a bill of material report, but that is simply a report of what was checked into the SCM tool at the time of the build.  The build scripts that execute the build can point to code anywhere on your network, so a BOM report is at best a guess.  I’m sort of surprised that as sophisticated we have become in our development practices that this simple fact is so misrepresented and misunderstood.  The build, meaning the creation of the pre-production binaries, is the decider when it comes to assembling the binaries.  The SCM tool is just a filing cabinet where code may or may not reside and referenced during the build.  Just because the code is in the local build directory does not mean that the build used the local code.  In many cases it does not.  How many developers check out rt.jar into the local build directory and then writes a script that ensures that rt.jar is found there.  I know this sound like bits and bytes and why does it matter - well it does, and without a real audit of the build by a process that monitors the calling of build engines, the SCM tool is nothing more than a repository of changes, that may or may not have been used when the build executed.

Because we have been doing build management for the last 14 years, we seem to have the honor of defining new terms to describe what we do only to have our competitors, borrow and redefine the term.  In a recent competitors webinar, they began talking about “building services”, around SOA.  Now this competitor does not do Builds, they simply manage workflows and perform job scheduling, they build nothing and rely on developers to write the build scripts that actually do builds.  So for clarification, Build Services is the process of creating reuse around the execution of  the Build engines. Build engines, such as Ant, Make and MSBuild, call compiler and linkers that assemble the application.  A Build Service does builds - so don’t be fooled when you hear build tools say they offer build services when they do not even attempt to manage the build engines.

Going through a review of build products that support Continuous Integration builds, it became very clear that few of them support iterative builds, meaning executing builds that re-builds only the changes, not doing full builds every time.  I find this ironic.  Continuous Integration is intended to support agile development processes where change is encouraged.  But if a build is taking several hours, than the builds are really not continuous.  They are just done more often than once every two weeks - which is certainly a big improvement, but just not enough.  Continuous Integration builds are only “continuous” when they can support incremental builds.  A build that understands how to re-build your application incrementally takes agile development to where it needs to be – integration on a continuous basis, meaning everytime source code changes, it can be built and tested immediately. 

The build tool market has gone insane.  Our competitors who in the past criticized us for providing a system that replaces ad-hoc scripts, now brag that it is one of the features they provide.  They say this but still do not address the ad hoc build scripts that actually run the builds.  So think twice before you drink their cool-aid.  If you are looking at a build workflow systems to run your build automation between check-in, check-out, make, ant, maven, testing and deployment - consider using free tools such as OpenMake Mojo or CruiseControl.  When looking at build systems, you need to look at tools that integrate into your Development IDEs, support refactoring, builds from the Project IDEs without you having to write manual ad-hoc scripts, and can do build avoidance to support incremental continuous integration builds.  Workflow is important so workflow should be included as part of the solution, but should not be the only solution.  In other words, if a build management tool still relies on you writing your build system, then is it really a build management solution or is it simply a job scheduler?

In the November issue of RedmondDeveloper, Peter Varhol wrote an article called “Code Dependency Analysis”.  In this article he states “It appears that tools to analyze and correct dependencies are few and far between”.  Well Peter, you may be correct when it comes to standard debuggers and code analysis tools; however, the place to look for these types of features is in build management tools.  Unlike debuggers and code analysis tools, a build management tool must perform this level of dependency analysis - otherwise it cannot create the binaries.  Maybe this is the reason why so many of our competitors, calling themselves build management tools, really don’t do builds.  Instead they have the developers write scripts that the so called build management tool can call to execute the real build process. And unfortunately, these scripts do not have the scanners capable of performing the required level of dependency analysis.So yes, build management is where to look for code-dependency analysis tools. And if your build management tool is not offering you these features, maybe it is not managing binaries and instead just executing the scripts that the developers have provided.  Don’t be fooled by tools that call themselves build management tools but cannot create a binary without the heavy lifting performed by the  developers custom scripts.

Build Management, Build Automation, dependency scanning - I hear these terms tossed around by vendors who sell “build tools”, but no one uses the terms in a standard manner. This inconsistency is a reflection of the build space in general. Builds are so “black box” that most people really don’t know what “build management” means. 

Having worked in the field for close to 15 years, I feel I have earned the right to comment on what these terms mean:

Build Management - The ability to manage the creation of binaries including the enforcement of source file locations, third party library usage and compile flags.  It also must include the ability to track all artifacts used to create the binary, not just show what was checked into a SCM tool. 

Build Automation - The ability to run automatic builds, without the need to baby sit ad hoc scripts.  Build Automation is not the process of scheduling scripts or executing scripts from some sort of workflow tool.  This is just “script scheduling”  - not build automation.  There is nothing automatic about writing build scripts, or babysitting build scripts.

Dependency Scanning - This is the process of watching the compiler and scanning source code for nested references to source code and libraries.  It is not running a “check-out” report against a SCM tool to determine what was checked out to the local build directory prior to a build, referred to as a Bill of Material (BOM) report.  A BOM report cannot tell you what it does not know, and many artifacts there are used to create your binary come from places other than the local build directory.  Dependency Analysis is just that, a thorough review of what occurred in the build, even if an SCM tool is not used.  So if your build relies on an SCM tool for “dependency scanning”, it is not doing dependency analysis.

This list can go on for some length, but I will stop here.  The point is that the Build Management market can be as confusing as  builds themselves.  This is proof that we,  as an IT community,  have not spent enough time discussing, improving and standardizing this critical component of application development.

Yes, I have not blogged in a bit - but I’m back.  I’ve had some time to think about the question people most commonly ask me about builds, and my personal mission to minimize ad hoc build scripts.  Well the answer is really simple.  The practice of writing build scripts sprang from the traditional waterfall model of software development.  Developers worked on their unique .exe or .dll file. They used a makefile that compiled their individual “Unit” binary.  When they were done, the turned over their source and makefile to the Build Meister.  The Build Meister than incorporated that makefile into a “System” level build.

Well guess what - developers don’t really work that way anymore, but they continue to rely on this waterfall approach to builds.  What developers do is work within IDEs where their builds are nicely managed for them.  They then try to setup a continuous integration build, or nightly build process, and find themselves falling back into the waterfall process model and calling on a build expert to create an Ant, Maven or Make script.  If you are trying to improve your process, automate the creation of your build script, in the same way that you use IDEs to perform automatic code creation.  You cannot do agile development without streamlining the build step.  Developers should be able to build inside or outside the IDE without the overhead of a manual scripted process designed for a waterfall approach to application development.  I’m not saying scripting is bad, just that done ad hoc, it is highly redundant and forces agile development into a waterfall phase of build management.  Just some thoughts to chew on.