I attended a lunchtime set of lightning talks at ThoughtWorks yesterday (11/11/09) in downtown Chicago. It was great, but surprisingly they had trouble nailing down compelling arguments for adopting Agile practices. I wanted to help them by adding a business perspective to “trust me, you’ll like it better,” and summarizing a few other topics of the day interpreting them in terms of my own experience.

They also dispelled the belief by some audience members that Agile is a radical new way of doing things, and it correctly came out that many practices were developed previously, even on the mainframe. I am not taking anything away from the leaders in the Agile community who have popularized many best practices, encouraged collective adoption and developed specific ways of incorporating them into coherent software engineering frameworks.

I realized I myself ended up doing a variant of extreme programming (XP) without knowing anything about it through application of common sense optimizations and best practices. Hopefully my notes here will help people understand why certain engineering practices are advocated by XP. One day I ended up being a development lead with no knowledge and a blank slate and this is what happened…

Code Quality Beyond – “It Works”

Erik Dornenberg mentioned that code should be easy to change and ThoughtWorks CEO Becky Parsons mentioned that you show pay attention to testing when writing code. My own mantra is “good quality code should be easy to change, easy to test and easy to debug.” The relative priorities of each is determined by business needs.

Test Driven Development

In my environment, I often was given requirements that were simple technical specifications. For example, I had to ensure that some value followed a naming convention. The best way to test that the code met the specification was to encode the specifications into the unit tests and then write the code so that it passes the tests. How else would you do it? The optimization here is that I didn’t want to waste my time writing code that didn’t pass a unit test, so I wrote the test first. Possibly I found this out by trial-and-error.

Automated Testing

Duh – I had 300 unit tests for my codebase. Am I going to run each of them manually? We did have sufficient complexity that I worried that one change might break something seemingly unrelated, so I made sure all unit tests were run before promoting the code for deployment. So, no changes could be deployed without all unit tests being run against the code.

Continuous Integration

I’ve also done a lot of consulting in the application lifecycle management space and, we didn’t get too deep into developer practices. But, since we were implementing version control with automated build, we always introduced continuous integration – we just didn’t have that nice tidy term. One day, I hope to thank Martin Fowler (who gave one of the lightning talks) for popularizing/coining the term so I don’t have to fumble with a longer explanation. Unfortunately, when I now say “continuous integration,” I’m met with a blank or embarrassed stare (“I feel I should know what that is, but don’t”) so I still have to go through the full explanation.

On my own team, I wanted to know about integration problems as soon as possible and I wanted to ensure unit tests were written for code changes by other developers. So, I had unit tests automatically run prior to code check-in. A failing or missing unit test meant the code could not be checked in. Well, I lost that battle and had to relax the enforcement to mere warnings. I even had a unit test generator to make it easy to pass.

BTW, this was all Perl and you can have integration issues there just as in C# or Java. Integration problems often have to do with namespace and interface changes, and those issues come up in Perl also. Perl also gets compiled prior to runtime, so you can also break the Perl build. There is just not a separately executed build step (ok, ‘perl –c’ – maybe I’ll try that next time).

Pair Programming

We sort of half-implemented pair programming and code reviews because the engineers were not familiar with Perl best practice coding. They were also not familiar with the architecture and standard patterns I invented and had in my head alone. Yes, another nasty real world situation. Had I introduced actual “pair programming” it would have been better than me looking over someone’s shoulder and trying to convince them to use best practices to no avail.

Had I introduced actual pair programming, I would have had to immediately answer to the project manager who would want to know why it was taking two people to make each change instead of one (sure there are twice as many changes, but…). This was the hottest topic of the day at ThoughtWorks, but no one in the audience articulated the question that succinctly.

Neal Ford in his talk, told us that, with pair programming, it took 15% longer to code the changes, but it resulted in 15% fewer defects. Then came “trust us, that’s better.” You were almost there Neal. We need cost and timeline justifications for the management to accept pair programming. The one thing I’ll add to the below analysis is that probably most of the 15% defects would get caught in pre-release testing for non-pair programming, but let’s say one would get through to production.

Cost Justification for Pair Programming

Find out the cost of a production defect. At the low end, it is a few $10k’s, at the high end it’s millions. Let’s say with pair programming, we prevent one defect that would cost $50k plus a lot of stress.

Then you have on top of that the cost of finding the other defects in QA, repairing them and possibly impacting your timeline. That’s got to be worth at least $50k for a typical business app.

Now compare with the 15% extra time it takes to make the changes in pair programming. That’s 15% extra time for two programmers over let’s say a take a two-week iteration. One of our customers uses $65/hr to estimate the cost of salary AND all the IT resources consumed by a developer, so let’s use that. The cost of extra development time is:

80 hrs/developer * 2 developers * 15% * $65/hr = $1560

That is nothing compared with $100k worth of problems and sill worth it if I’ve wildly overestimated the cost of the defects.

Timeline Justification for Pair Programming

To calculate the cost of the extra test and repair work, you really needed to know how long all that would take and what the extra effort is because time is money. Al l you have to do is take 15% of 80 hours (12) and compare that to the time it takes to test, repair and re-test. It is probably worth investing the extra time time for pair programming.

Interleaving Design and Coding

When I became a dev lead, I was supposed to do design and let the junior members code, but I could not do one without the other. I couldn’t articulate why, but I did both and it worked out. Now, we hear that is a best practice, and to wait as long as possible before doing the design and design while coding. I’d still like a well-articulated argument about why it works.

Effective Communication

Communication breakdowns are a common reason for project failures. Features of Agile methodologies such as the stand-up meeting, well-defined artifacts and having testers and business analysts interacting very closely with developers are ways to facilitate communication. Discover (the credit card company) has a Discover Lean initiative that puts business analysts and developers in the same room, accomplishing the same thing.

Summary

I was also happy to hear the experienced ThoughtWorks folks espouse an adopt-what-works methodology rather than follow a specific methodology religiously. That is very good advice in my view. After all, I was very successful with IXP – Ignorant Extreme Programming.

I was working on C language builds using Meister with the Eclipse CDT (C Development Tools). The way libraries are managed in CDT compared with the JDT (Java Development Tools) is very different. In fact, there is really no concept of libraries in CDT. Linking is configured by supplying –l (dash little-el) and –L (dash big-el) options for the linker to use.

The way libraries are acknowledged to exist in JDT and have a build path is a great implementation. To manage your libraries in CDT, you are taken out of the Eclipse environment of abstracted functionality for creating solutions and forced to focus on dirty command-line compiler options.

We also have this great code in our plug-in that uses Eclipse API’s in JDT, but can’t be re-used for equivalent functionality in CDT. It’s not surprising since Java developers usually don’t think about the command line and C developers can’t think of anything else.

Mainly it shows there is no overall build management strategy for Eclipse across compiled languages. This is a huge opportunity loss for plug-in developers who don’t have a common API and each has to develop standard patterns of managing builds – if they even know what those are.

Continuing my last post about using a service such as Twitter (or Yammer or equivalent) to create “pull” notifications that serve the individual, I’ve set up a Meister workflow to update the status of the Twitter handle @builds after a successful build.

I’ve set up a JBoss build in Meister involving a web service and consumer. This was originally developed in MyEclipse, but the Meister Eclipse plug-in allows the native Eclipse build to be externalized (a better approach than Maven). So, I am just dealing with that, which simulates a development continuous integration or QA build.

I have a workflow that has the two main Meister build tasks, “Generate Build Control File”, and “Build Targets”. The first gathers local and server metadata consolidating that into something like a makefile or Ant script depending on your perspective. The second is similar to running make or Ant.

This is a simple workflow to illustrate our messaging technique. Normally, we’d use an email notification task, but I’ve replaced that with an activity called “Tweet”. This calls a simple Perl script, called omtweet.pl that uses Net::Twitter. Here is why we use Perl a lot – the code is simple to update your Twitter status:

1
2
3
4
5
6
7

use Net::Twitter;
 
my $twit = Net::Twitter->new(
  {username => "myuser", password => "mypass"; });
 
my $result = $twit->update(
  {status => "My current Status"});

I created a slightly more elaborate script that takes arguments and passes in a “Build Successful” message. If Java is your thing, look at Davanum Srinivas’ post and for .NET, there is Twitteroo.

Next, I run the build and we see the steps complete in real time on the workflow monitor.

The key is the last step, of course:

And, the proof is in the pudding:

I used the $(JOB_NUMBER) macro in Meister to get an incremental build number passed to omtweet.pl. Of course, following @builds and choosing device updates, I got a text message on my cell phone. I tried to get a picture of my cell phone with the message, but I haven’t worked out how to do that yet and it turned out terrible.

Well, this is a simple example, with a lot of possibilities. Meister wraps all build output in HTML and pushes to a web server. I’d like to include a shortened URL for that in the update. Currently we don’t have anything outside of a firewall, maybe I’ll try Yammer for that.

If you have any other suggestions, let me know. @seanblanton

There has been some talk around using physical, colored, lamps to notify build results (pragmatic automation, Carlos Sanchez, Richard Durnall, Alberto Savoia).

The idea comes from Lean Manufacturing principles developed by Toyota applied to software development. An “andon” paper lamp was a way to let other workers know there was a problem on the assembly line. The idea for builds is that while a build is running, an amber light flashes, letting other developers know not to run a build, a green light signifies the last build was completed successfully, and a red light means the build is broken and needs to be fixed.

While I’m sure it is an entertaining exercise to set this up, some simple software automation could achieve the same results, and it has obvious faults, like what if I am working from home?

Email is the de facto standard for build notification. There is a hardcoded mailing list somewhere and you get the email, even if you don’t want it, and you have to go and check your mail to receive it.

Twitter, on the other hand, puts the user in control of the notifications in true Web 2.0 fashion. Tweets are also ideal for receiving notifications via cell phone, where email is generally not. Consider the following cases:

• I can follow the build when I want, and unfollow when I want. No going to an admin to remove me from the list.
• I can turn on device updates so I get an SMS message about the result (maybe an important release is coming up), or I can turn off device updates, say, when I’m on vacation.
• I can retweet (forward) the build notification.
• You can put links in the message. Meister uploads all build logs in HTML format to a web server and we would include the link to that.

About a year ago, I used Meister with a post-build activity to Tweet the build result after a build. The activity was a simple Perl script using Net::Twitter. Let’s just say this experiment sort of fell on deaf ears.

Now I’m ready to roll with this again and you can follow my progress. I had created a Twitter account, @builds, for this purpose. In a day or two, you’ll see some tweets from that guy from one of my builds.

I’m interested in what other people have to say. Meister already uses Eclipse RCP as it’s front end, but most other build tools are still command-line and in the 1970’s. Let’s bring build management tools into the 21st century with Web 2.0 features.

In the Agile Scrum framework, a ScrumBut is a “reason why [you] can’t take full advantage of Scrum to solve the problems and realize the benefits.” Clearly, build problems are traditionally one of the biggest ScrumButs. Meister removes the scrum butt by taking variables out of the hands of operators managing the builds.

Ask for a build and get a build. Ask for continuous integration and test automation and coalescence and get it with Meister. Product owners should not be losing cycles to debugging builds, tweaking parameters to get different types of builds or in resolving confusion about what did or did not happen in a build or continuous integration workflow.

The client-server architecture and highly reusable build and workflow metadata extend the ScrumBut elimination to all developer machines and continuous integration build servers, creating a highly consistent build. Team members working on these different machines know exactly what to expect.

Since they know what to expect from the build, they can focus on resolving actual integration issues and testing rather than trying to make the build work in a consistent manner.

With our third nomination, we won the Jolt Award for product excellences in the change and configuration management category. You can view the announcement here.

While we have managed Jboss and WebSphere deployments using Meister and Mojo and we frequently manage resources for different target server environments through build, here is a company that focuses on both of those things.

I posted a comment about the reluctance of script writers to give up their scripting on their blog here.

It is just plain fun to run parallel workflows and builds and watch the activities and build steps light up the workflow monitor in real time like a Christmas tree. See this flash demo to see what I mean.

As customers go to machines with more and more cores, fewer machines are needed in the application lifecycle infrastructure, particularly for builds and code retrievals – the most resource intensive functions. This is helping to simplify the infrastructure, reduce maintenance and administration and drive down costs.

Several of our customers are running around 5000 builds and non-build workflows per month on two machines. The primary reason for two machines, in fact, is for disaster recovery, and the goal is to run both machines at less than half capacity so that in the event that one machine (or datacenter) fails, all the current capacity can be run as a contingency on the one machine that’s left.

Thread control is very simple with Meister and Mojo. Both use the omsubmit dependency manager program to handle this. Meister’s om program translates build events into workflow steps using omsubmit. The OMSUBMIT_MAX_USER_PROC value sets the maximum allowed number of threads.

You might think that if you are running dual, quad-core build machines that you should set the max threads at 8. However, Meister posts build operations to one thread and the associated logging operation to another. Compile operations notoriously use a lot of memory and CPU resources, but the logging operation posts to a server and waits for the operation to complete. There is really no disadvantage to setting the max threads higher than 16 in this case, so go ahead and do it.

As a non-build workflow example, I worked on JBoss deployments to 48 Linux machines. The workflow was parallelized into 48 activities each of which deployed to a single machine in parallel. The deployment activity was largely a remote execute operation that extracted archives on the remote machine. The extraction took about a second for a medium sized application. Again, this is a waiting situation where machine resources are essentially idle while the thread is in use, so use more threads. The machine was a dual, dual-core build machine and we set OMSUBMIT_MAX_USER_PROC to 50.

Watching the workflow monitor as the deployment ran, we could see roughly half of the machines light up (meaning actively running) at any one time and the entire deployment process synchronized all 48 machines in a little over two seconds.

So, don’t simply match your machine’s CPU threading capabilities – overclock! Aim high for max threads and try to determine where your performance is optimized. I’d love to provide you with some metrics as a function of thread count, but usually once something is working it’s on to the next project. I barely have enough time to blog!

It’s very common to have a code check-out step be part of an integration build. Far better it is to not check out code before a build. What? How is that possible?

Let me explain, Fred. The simple approach most of us take (and have to take when getting things started) has developers commit, commit, commit, and when it is time to deploy, check out the code, do a build, and then deploy the application. There is room here for both problems and optimizations. Doing a full check out of the code tree is more costly in terms of time than checking out only what has changed. Updating the code tree with a single commit is less costly than updating with a large number of commits.

You may be limited by the technology in-hand and how much you’ve invested in learning the technology and possibly customizing it. For example, if your file control tool can only do a full check out of a source tree, or that’s the only command you had time to implement in order to meet the deadline, or you don’t trust your tool to do incremental updates, then you are basically running the longest builds possible.

On the other hand, if you could update the code tree every time a developer does a commit with only the changed files, then you are ready to execute a build at any moment. This requires some deft manipulation of your file control tool, and that’s why you don’t see it more often.

You might think “continuous integration” will take care of this. Developer commits, update checked out, build is run. However, you may end up with a build, test execution and deployment that takes longer than the typical time between developer commits. You still have to do incremental updates and it only solves the problem in cases with very low developer activity.

I’d like to point out one tool that does an excellent job of post-commit code checkout, CA Software Change Manager for Distributed. CA SCM (for short) is the tool, formerly known as Harvest, from the company formerly known as Computer Associates. CA SCM is a highly scalable (1000’s of developers) file control tool with a great lifecycle process model. We at OpenMake Software still have our very first customer still using OpenMake/Meister with CA Harvest/SCM after 11 years. While we have a reseller arrangement with CA, our partnership with CA in services has extended to 14 years.

About 10 years ago, OpenMake Software developed an integration with the then, Platinum Technologies’ Harvest product, modeled after the now dead Computer Associates product, Endevor Workstation, that had an excellent post-action code tree update. (Endevor for z/OS, a.k.a CA SCM for Z/OS is still very popular and has a similar functionality called ‘output libraries’ – following all this?) Our integration had the horrific name, ‘Har-refresh’.

As product partners, we finally transitioned Har-refresh from an external add-on to CA who have turned it into a core functionality of the product, called Hrefresh (a better name.) Rather than simply a post-commit check out, HRefresh updates the code tree after any action that updates a dynamic code view. This includes, renames, deletions, commits and code promotions and demotions. We like this because CA SCM does all the work and we cherry-pick sets of up-to-date code trees to build up an application source code stack for a build. We align Meister dependency directories with HRefresh-managed file system directories for a tight SCM (software configuration management) build.

This mechanism distributes the resource load for checking out code to times when builds are not required. It’s true that often times people want to build as soon as their code is checked in (or promoted), but on average it is a very big net win reducing build times.

This is just an example of the type of sophistication that is out there to prevent pre-build code check outs and save time on your builds.

One of OpenMake Software’s product strategies is to keep things simple. Build management is one of the most complex operations in all of the IT world, and one of our key benefits is to simplify, organize and automate the build process for development, testing and production.

We’ve seen a trend among our customers to simplify their build management infrastructure by going to fewer build machines with more CPU cores. Builds in particular use relatively more CPU resources than other resources as code is interpreted and compiled in memory and then finally written to disk. By reducing the total number of machines, rack space, procurement, administration and other IT overhead costs are reduced at great cost savings per machine eliminated.

Recently, I was at one of the big chip makers where they used dual quad-core CPU Linux machines for their development and builds. They had two machines and were able to control access to allow separate areas for development, testing and release builds in keeping with best practices. Having all the horsepower of 8 CPU cores on a single machine kept them from needing more machines.

Another customer does 6000 builds per month with Meister on just two build machines.

IBM, when selling BuildForge, likes to talk about big build server farms, because their tool does remote execution on multiple machines, as does Meister. However, BuildForge does not do builds at all. It can remotely execute your existing build scripts, but there is little real value add to that. BuildForge is also famously expensive. What happens over the next few years to the high investment in multi-machine remote execution software as the number of machines declines, perhaps dramatically?

A similar argument can be for Electric Cloud’s Electric Accelerator product. It’s possible in some cases, for C/C++ builds to gain an edge by pushing a compile operation to another machine, and then bringing it back. You would only do this to gain access to additional CPU resources. In the past, you might have 8 build machines that Electric Accelerator would farm operations out to. Now, you can pull all those operations into a single machine and there is no need for that functionality. Also, you are stuck with converting your GNU makefiles into other GNU makefiles.

Meister is optimized for multi-core CPU build machines and offers multi-threaded capability to both build events and non-build workflow events. You know where your build is and there are fewer dependencies on network resources. Both BuildForge and Electric Accelerator add additional overhead to build administration to coordinate across multiple machines – a dying practice, that no organization wants to invest in. Meister is the best bet for a future with fewer build machines with more horsepower.