Don’t forget my blog has moved!

Check out my new post on comparing build management in Eclipse C and Java development tools, here:

Comparing Build Management in Eclipse CDT and JDT

This blog was technically for all the OM Services folk to blog, but since I did 95% of the posts, I went ahead and took it over. Selvi Dayabaran, Adam Gabor and my blogs are now splitting off from this site to the following respective URL’s:

Selvi’s Blog

Adam Gabor’s Blog

Sean Blanton’s Blog

The content up to this point will appear in both places for convenience. Please update your RSS feeds if you are using them. See you there!

Sean

Continuing my last post about using a service such as Twitter (or Yammer or equivalent) to create “pull” notifications that serve the individual, I’ve set up a Meister workflow to update the status of the Twitter handle @builds after a successful build.

I’ve set up a JBoss build in Meister involving a web service and consumer. This was originally developed in MyEclipse, but the Meister Eclipse plug-in allows the native Eclipse build to be externalized (a better approach than Maven). So, I am just dealing with that, which simulates a development continuous integration or QA build.

I have a workflow that has the two main Meister build tasks, “Generate Build Control File”, and “Build Targets”. The first gathers local and server metadata consolidating that into something like a makefile or Ant script depending on your perspective. The second is similar to running make or Ant.

This is a simple workflow to illustrate our messaging technique. Normally, we’d use an email notification task, but I’ve replaced that with an activity called “Tweet”. This calls a simple Perl script, called omtweet.pl that uses Net::Twitter. Here is why we use Perl a lot - the code is simple to update your Twitter status:

1
2
3
4
5
6
7

use Net::Twitter;
 
my $twit = Net::Twitter->new(
  {username => "myuser", password => "mypass"; }); 
 
my $result = $twit->update(
  {status => "My current Status"});

I created a slightly more elaborate script that takes arguments and passes in a “Build Successful” message. If Java is your thing, look at Davanum Srinivas’ post and for .NET, there is Twitteroo.

Next, I run the build and we see the steps complete in real time on the workflow monitor.

The key is the last step, of course:

And, the proof is in the pudding:

I used the $(JOB_NUMBER) macro in Meister to get an incremental build number passed to omtweet.pl. Of course, following @builds and choosing device updates, I got a text message on my cell phone. I tried to get a picture of my cell phone with the message, but I haven’t worked out how to do that yet and it turned out terrible.

Well, this is a simple example, with a lot of possibilities. Meister wraps all build output in HTML and pushes to a web server. I’d like to include a shortened URL for that in the update. Currently we don’t have anything outside of a firewall, maybe I’ll try Yammer for that.

If you have any other suggestions, let me know. @seanblanton

There has been some talk around using physical, colored, lamps to notify build results (pragmatic automation, Carlos Sanchez, Richard Durnall, Alberto Savoia).

The idea comes from Lean Manufacturing principles developed by Toyota applied to software development. An “andon” paper lamp was a way to let other workers know there was a problem on the assembly line. The idea for builds is that while a build is running, an amber light flashes, letting other developers know not to run a build, a green light signifies the last build was completed successfully, and a red light means the build is broken and needs to be fixed.

While I’m sure it is an entertaining exercise to set this up, some simple software automation could achieve the same results, and it has obvious faults, like what if I am working from home?

Email is the de facto standard for build notification. There is a hardcoded mailing list somewhere and you get the email, even if you don’t want it, and you have to go and check your mail to receive it.

Twitter, on the other hand, puts the user in control of the notifications in true Web 2.0 fashion. Tweets are also ideal for receiving notifications via cell phone, where email is generally not. Consider the following cases:

• I can follow the build when I want, and unfollow when I want. No going to an admin to remove me from the list.
• I can turn on device updates so I get an SMS message about the result (maybe an important release is coming up), or I can turn off device updates, say, when I’m on vacation.
• I can retweet (forward) the build notification.
• You can put links in the message. Meister uploads all build logs in HTML format to a web server and we would include the link to that.

About a year ago, I used Meister with a post-build activity to Tweet the build result after a build. The activity was a simple Perl script using Net::Twitter. Let’s just say this experiment sort of fell on deaf ears.

Now I’m ready to roll with this again and you can follow my progress. I had created a Twitter account, @builds, for this purpose. In a day or two, you’ll see some tweets from that guy from one of my builds.

I’m interested in what other people have to say. Meister already uses Eclipse RCP as it’s front end, but most other build tools are still command-line and in the 1970’s. Let’s bring build management tools into the 21st century with Web 2.0 features.

In the Agile Scrum framework, a ScrumBut is a “reason why [you] can’t take full advantage of Scrum to solve the problems and realize the benefits.” Clearly, build problems are traditionally one of the biggest ScrumButs. Meister removes the scrum butt by taking variables out of the hands of operators managing the builds.

Ask for a build and get a build. Ask for continuous integration and test automation and coalescence and get it with Meister. Product owners should not be losing cycles to debugging builds, tweaking parameters to get different types of builds or in resolving confusion about what did or did not happen in a build or continuous integration workflow.

The client-server architecture and highly reusable build and workflow metadata extend the ScrumBut elimination to all developer machines and continuous integration build servers, creating a highly consistent build. Team members working on these different machines know exactly what to expect.

Since they know what to expect from the build, they can focus on resolving actual integration issues and testing rather than trying to make the build work in a consistent manner.

With our third nomination, we won the Jolt Award for product excellences in the change and configuration management category. You can view the announcement here.

While we have managed Jboss and WebSphere deployments using Meister and Mojo and we frequently manage resources for different target server environments through build, here is a company that focuses on both of those things.

I posted a comment about the reluctance of script writers to give up their scripting on their blog here.

It is just plain fun to run parallel workflows and builds and watch the activities and build steps light up the workflow monitor in real time like a Christmas tree. See this flash demo to see what I mean.

As customers go to machines with more and more cores, fewer machines are needed in the application lifecycle infrastructure, particularly for builds and code retrievals - the most resource intensive functions. This is helping to simplify the infrastructure, reduce maintenance and administration and drive down costs.

Several of our customers are running around 5000 builds and non-build workflows per month on two machines. The primary reason for two machines, in fact, is for disaster recovery, and the goal is to run both machines at less than half capacity so that in the event that one machine (or datacenter) fails, all the current capacity can be run as a contingency on the one machine that’s left.

Thread control is very simple with Meister and Mojo. Both use the omsubmit dependency manager program to handle this. Meister’s om program translates build events into workflow steps using omsubmit. The OMSUBMIT_MAX_USER_PROC value sets the maximum allowed number of threads.

You might think that if you are running dual, quad-core build machines that you should set the max threads at 8. However, Meister posts build operations to one thread and the associated logging operation to another. Compile operations notoriously use a lot of memory and CPU resources, but the logging operation posts to a server and waits for the operation to complete. There is really no disadvantage to setting the max threads higher than 16 in this case, so go ahead and do it.

As a non-build workflow example, I worked on JBoss deployments to 48 Linux machines. The workflow was parallelized into 48 activities each of which deployed to a single machine in parallel. The deployment activity was largely a remote execute operation that extracted archives on the remote machine. The extraction took about a second for a medium sized application. Again, this is a waiting situation where machine resources are essentially idle while the thread is in use, so use more threads. The machine was a dual, dual-core build machine and we set OMSUBMIT_MAX_USER_PROC to 50.

Watching the workflow monitor as the deployment ran, we could see roughly half of the machines light up (meaning actively running) at any one time and the entire deployment process synchronized all 48 machines in a little over two seconds.

So, don’t simply match your machine’s CPU threading capabilities - overclock! Aim high for max threads and try to determine where your performance is optimized. I’d love to provide you with some metrics as a function of thread count, but usually once something is working it’s on to the next project. I barely have enough time to blog!

It’s very common to have a code check-out step be part of an integration build. Far better it is to not check out code before a build. What? How is that possible?

Let me explain, Fred. The simple approach most of us take (and have to take when getting things started) has developers commit, commit, commit, and when it is time to deploy, check out the code, do a build, and then deploy the application. There is room here for both problems and optimizations. Doing a full check out of the code tree is more costly in terms of time than checking out only what has changed. Updating the code tree with a single commit is less costly than updating with a large number of commits.

You may be limited by the technology in-hand and how much you’ve invested in learning the technology and possibly customizing it. For example, if your file control tool can only do a full check out of a source tree, or that’s the only command you had time to implement in order to meet the deadline, or you don’t trust your tool to do incremental updates, then you are basically running the longest builds possible.

On the other hand, if you could update the code tree every time a developer does a commit with only the changed files, then you are ready to execute a build at any moment. This requires some deft manipulation of your file control tool, and that’s why you don’t see it more often.

You might think “continuous integration” will take care of this. Developer commits, update checked out, build is run. However, you may end up with a build, test execution and deployment that takes longer than the typical time between developer commits. You still have to do incremental updates and it only solves the problem in cases with very low developer activity.

I’d like to point out one tool that does an excellent job of post-commit code checkout, CA Software Change Manager for Distributed. CA SCM (for short) is the tool, formerly known as Harvest, from the company formerly known as Computer Associates. CA SCM is a highly scalable (1000’s of developers) file control tool with a great lifecycle process model. We at OpenMake Software still have our very first customer still using OpenMake/Meister with CA Harvest/SCM after 11 years. While we have a reseller arrangement with CA, our partnership with CA in services has extended to 14 years.

About 10 years ago, OpenMake Software developed an integration with the then, Platinum Technologies’ Harvest product, modeled after the now dead Computer Associates product, Endevor Workstation, that had an excellent post-action code tree update. (Endevor for z/OS, a.k.a CA SCM for Z/OS is still very popular and has a similar functionality called ‘output libraries’ - following all this?) Our integration had the horrific name, ‘Har-refresh’.

As product partners, we finally transitioned Har-refresh from an external add-on to CA who have turned it into a core functionality of the product, called Hrefresh (a better name.) Rather than simply a post-commit check out, HRefresh updates the code tree after any action that updates a dynamic code view. This includes, renames, deletions, commits and code promotions and demotions. We like this because CA SCM does all the work and we cherry-pick sets of up-to-date code trees to build up an application source code stack for a build. We align Meister dependency directories with HRefresh-managed file system directories for a tight SCM (software configuration management) build.

This mechanism distributes the resource load for checking out code to times when builds are not required. It’s true that often times people want to build as soon as their code is checked in (or promoted), but on average it is a very big net win reducing build times.

This is just an example of the type of sophistication that is out there to prevent pre-build code check outs and save time on your builds.

One of OpenMake Software’s product strategies is to keep things simple. Build management is one of the most complex operations in all of the IT world, and one of our key benefits is to simplify, organize and automate the build process for development, testing and production.

We’ve seen a trend among our customers to simplify their build management infrastructure by going to fewer build machines with more CPU cores. Builds in particular use relatively more CPU resources than other resources as code is interpreted and compiled in memory and then finally written to disk. By reducing the total number of machines, rack space, procurement, administration and other IT overhead costs are reduced at great cost savings per machine eliminated.

Recently, I was at one of the big chip makers where they used dual quad-core CPU Linux machines for their development and builds. They had two machines and were able to control access to allow separate areas for development, testing and release builds in keeping with best practices. Having all the horsepower of 8 CPU cores on a single machine kept them from needing more machines.

Another customer does 6000 builds per month with Meister on just two build machines.

IBM, when selling BuildForge, likes to talk about big build server farms, because their tool does remote execution on multiple machines, as does Meister. However, BuildForge does not do builds at all. It can remotely execute your existing build scripts, but there is little real value add to that. BuildForge is also famously expensive. What happens over the next few years to the high investment in multi-machine remote execution software as the number of machines declines, perhaps dramatically?

A similar argument can be for Electric Cloud’s Electric Accelerator product. It’s possible in some cases, for C/C++ builds to gain an edge by pushing a compile operation to another machine, and then bringing it back. You would only do this to gain access to additional CPU resources. In the past, you might have 8 build machines that Electric Accelerator would farm operations out to. Now, you can pull all those operations into a single machine and there is no need for that functionality. Also, you are stuck with converting your GNU makefiles into other GNU makefiles.

Meister is optimized for multi-core CPU build machines and offers multi-threaded capability to both build events and non-build workflow events. You know where your build is and there are fewer dependencies on network resources. Both BuildForge and Electric Accelerator add additional overhead to build administration to coordinate across multiple machines - a dying practice, that no organization wants to invest in. Meister is the best bet for a future with fewer build machines with more horsepower.

I’ve recently been learning the Ruby on Rails framework for web development. It’s become a quite popular framework for getting database connected websites up and running relatively quickly. One way it is easier to get a site started than with other frameworks is because of the Convention over Configuration mantra that it lives by. Instead of requiring loads of configuration files to build a basic site (that can grow and become quite complex by the way) it has a feature called scaffolding which automatically builds your Model, View and Controller classes based on tables it finds in the database. It can do this by making assumptions based on standard conventions about interacting with a database from a website and naming and using classes in a standard way.

Although I am still a rookie when it comes to understanding the many facets of Ruby on Rails, I have really been trying to emulate the Convention over Configuration way of doing things in my various build/release projects at customer sites. One problem I inevitably encounter in most organizations is that the development of build and release methodologies has been left to the various development teams and not been thought about holistically using a centrally managed approach - this leads to little to no standards and Convention over Configuration is chucked aside. Not only is this inefficient from an organizational standpoint - why reinvent the wheel over and over again for each team when they are essentially tackling the same sets of problems, but it also makes for a nightmarish audit trail that could get you into trouble.

One reason this happens so frequently is that the managers that are supposed to be in charge of standards for building and releasing applications are often not privy to the kind of technical requirements that the various development units have when it comes to putting together and delivering their applications. And when developers try to explain the requirements, the standards people may get lost because they can’t possibly understand the nuts and bolts of every application.

To pull off real centralized management of builds and deployments, the standards people need to take a deep breath and rethink their objectives - start looking for the commonalities, not the differences between applications. In doing this, they will find that that problem that that developer told you was so unique and must be solved a certain way is probably very similar to the problem the other developer told you about last week - or just look on the web and see how many thousands of external developers have this same “unique” problem. It turns out that most applications can be constructed in the same type of way. Just because the source files are different between applications doesn’t mean that the paths they take to their target executable, dll, Jar, War or Ear file is very different at all. And when those paths are essentially the same - create a reusable process that the various teams can share. Use Convention over Configuration as your guide - standardize and centralize the common processes and externalize the technical specifications using highly modularized control files.

Here’s a simple task for you to try. This assumes all of your application teams have their code checked into a central repository - if not, you have bigger problems than standardizing builds and releases and should address those first. Look at your various technologies, whether it’s .Net, Java or some other and try to identify where the code tree’s start under the root of the project. You’d be amazed at how many teams check their .Net solutions into different levels of a code tree for no good reason, or Java teams that have their their source packages buried some place in the code tree. Next, look for the common root starting point for all these application types and try to come up with a simple standard based on this information. Finally, notify those that are not following that standard that you would like to move their code up and over to this new location (its usually up and not down) - it should actually be pretty easy to do. After this has been done, you can now have all build and deploy scripts use a standard root variable to find dependencies (think something like SOURCE_ROOT).

It always amazes me how many teams don’t standardize simple things like code tree start points in their source projects - it equally amazes me how much mileage you can get just out of making simple path standardization adjustments. After you’ve worked on the source tree, try doing the same with your common libraries. This isn’t rocket science - just remember, Convention over Configuration makes everything easier.

Adam

Recently on an assignment I come across a problem that seemingly should have been easy to solve but turned out to be a bit of a stumper. I was trying to execute a remote process using an agent that had to start as a service under the local system account. My requirement was that the process started by that service had to execute a build using a specific user’s profile. It was not enough just start the build process as another user, because I needed all of the user’s environment variables, such as APPDATA and USERPROFILE and its registry settings - the equivalent of logging in that user. In Unix, I would just ’su’ to the new user and source the profile - not so easy, as it turned out, on Windows.

After a bit of research I learned about the Windows RunAs program that allows you to do essentially what I needed. The only problem was that it couldn’t be executed in batch mode since it opened another command prompt to ask for the user’s password. Looking online, I found some people who had written all sorts of wrapper processes to try to get RunAs to work programatically. This included such circuitous methods as using a VB Script wrapper that detected typed keystrokes to pass them to the password login command prompt. Not a glamorous solution to be sure and not one that I could trust to be reliable for widespread enterprise use. I knew there had to be a Windows API for this - after some trial and error I finally found the perfect one to do what I wanted and its supposed to be compatible with 2000 - Vista versions of Windows. It involved using the C# ProcessStartInfo class under the System.Diagnostics package. And, like anytime you find the right tool for the job, once I figured out how to use it, the problem actually became a very quick fix. The class requires five arguments to Start a new process: executable file name, arguments to executable, user name, domain name and password. Note, the password needs to be converted into a  SecureString for PSI to work. For my purposes, I didn’t want to pass in an encrypted password, so I just converted it in my script.

Here is the critical code snippet to handle command line arguments passed into the Main method.

ProcessStartInfo psi = new ProcessStartInfo();
psi.FileName = args[0];
Console.WriteLine(”Execution File: ” + args[0]);
psi.Arguments = args[1];
Console.WriteLine(”Execution Arguments: ” + args[1]);
psi.UseShellExecute = false;
psi.Domain = args[2];
Console.WriteLine(”Domain: ” + args[2]);
psi.UserName = args[3];
Console.WriteLine(”User: ” + args[3]);
String pw = args[4];
Char[] pwChars = pw.ToCharArray();
SecureString ss = new SecureString();
foreach (Char pwChar in pwChars)
{
ss.AppendChar(pwChar);
}
psi.Password = ss;
psi.WorkingDirectory = Directory.GetCurrentDirectory(); //start the process in the current working dir
psi.LoadUserProfile = true;
Process p = new Process();
p.StartInfo = psi;
p.Start();

That’s it! I actually wrapped the Start() method in a try, catch and attempted to login the user name to a local machine account of the same name in cases where the domain login failed in order to handle some machines that were not on the same domain, but you probably won’t need that. Hope this helps!

Adam

OM Services Senior Consultant Adam Gabor will be contributing to this blog this week. I’ve been the primary blogger for awhile now and am eager to have our other world-class consultants contributing their expertise to the community.

I took over the name of this blog, but now that we will be expecting more contributions from the others, we may have to reorg the blog a bit. I’ll keep you posted.

I enjoyed the Windy City Perl Mongers meeting last night. I met a lot of neat people and heard a great talk on CouchDB and one on an AI aggregator. A diverse group of people showed up with some serious technical knowledge and skills. Iraq veteran and author of many Perl books, Brian D Foy was there. They were appreciative of OpenMake Software’s sponsorship of YAPC NA last year.

Josh McAdams organized and hosted the meeting at the Chicago Google offices, where free beer, wine and milk were available in the fridge. Afterwards, we went to Rock Bottom Brewery and lasted until after midnight.

Finding the blog Enterprise Maven made me decide to go back to the basics, today. This blog is from 2006, but the best practices of production control ignored here go back decades. I’d like to point out that Oleg Gusakov, the author, wrote the blog in a very good spirit and seems like a nice guy. He just seems to be a bit naive about what’s been happening with software development in the enterprise.

In the first section, he assumes that the only enterprise build and deploy solution is one that is customized, while OpenMake Meister has been serving that role now for 12 years. He does correctly conclude that all the enterprises in the world should not be independently investing in the same type of build and deploy solution. It is a costly investment and this functionality should be productized. That’s exactly why we did it and why that is still one of our chief selling points.

He is right that developing a product that should be commoditized is a drain on the business. However, the converse, having a commercial product provide the functionality at a greatly reduced cost compared with one homegrown, provides a competitive advantage over those companies who don’t have such a product.

Through the middle of the article, again, I think Oleg is unaware of the heavy horse SCM products out there that provide a lot of the expected functionality. Tools like CA Harvest, Serena Dimensions and others are very complex and sophisticated n-tier products. They nevertheless do not provide build support, so by combining an enterprise file control tool with an enterprise build and workflow tool, Meister, you canvas the required functionality.

Lastly, regarding the enterprise development lifecycle, he is right it is an oversimplification. I like his phrase that he hopes to “grow the meat.” At OM Services, we have “fully grown meat” and the enterprise lifecycle documents that we develop with our customers and clients are typically 50-80 pages in length. Here is where I review the generally accepted best practices, going back to the seventies with mainframe development. (NO, distributed platforms are not somehow different in the high level process!)

1. It all starts with production control. Developers do not have access to production due to a fundamental conflict of interest. Maintaining business continuity trumps developers’ ease of delivery to production.
2. Since someone else puts the code into production (or operates a tool which does so), this is the basis for separation of roles and responsibilities in the enterprise software development lifecycle.
3. To ensure integrity of the production environment, the production build must be done by a group representing the business, not development. Developers do not do production builds.
4. Working backwards, if you want your test environment to be as close as possible to production, you lock this down and prevent access to developers. This is usually the QA testing environment.
5. Again, to avoid a conflict of interest, the QA testers should be working for the business, not the application development team.
6. And it follows that the build for the QA environment is done by the business.
7. The developers job from this perspective is delivering source code to the business, which wants retain the source code and the ability to use it (meaning they can build it).
8. The process of developers transferring source code to the QA build people was called “throwing it over the wall”. Now, the heavy horse SCM tools and Meister workflow make it easy to do this and allows variations of iterative development involving the QA environment.

Any type of continuous integration or agile development practice typically happens before the QA environment. Any develop methodology for the enterprise must take into account the fundamental conflict of interest between software change delivery and business continuity or ignore it and remain entirely in front of QA.

If you are a developer, you can think of this as a loss of privilege, or you can be elated that other people are doing the dirty work for you and you can focus on the art and science of engineering business solutions. If you are really depressed, maybe you should be on the other side of the wall!

As one who has done many version control tool A to version control tool B conversions, I know how difficult such a task is. That’s why I am all the more impressed that 20+ years of Perl history from multiple repositories have been converted to a single Git repository.

I can’t add much more about the benefits than the announcement itself:

http://use.perl.org/article.pl?sid=08/12/22/0830205

I’ve been involved with software procurement that involve RFP’s (Request-for-Proposal) on both sides of the fence - as part of the purchasing organization, and more frequently as a software vendor. I’ve seen the mistakes people make in sending out an RFP and then making a purchasing decision based on the results and I’m filing those items away should the time come for me to head up a software purchase myself.

RFP’s work best when you need a product that is strongly commoditized. For example, you need a software package to manage your purchasing. Or, you need software to do perform all of your HR (Human Resource) functions. HR functions are pretty much the same at most companies - sure larger companies might have needs for scalability and breadth of functionality that smaller companies don’t, but its all HR.

Before I get too far into my experience, let me mention that this is not a sour grapes article. Meister does great in sales with RFP’s and we invariably win or come in a close second. However, in some of the “close second’s” we’ve lost to a product that we would not regard as a competitor and we can see that the purchaser has not satisfied some of their stated key requirements at the beginning of the process that got us involved. It has made me wonder and this article is the result.

The first way to screw up an RFP is to make it a democracy - keep it an oligarchy of stakeholders. If you start out with a need for an HR solution and you solicit requirements from everyone in your company, you might get requirements like “needs to do purchasing” and “needs to do supply-chain management”. If you then invite more people from the purchasing department to participate and then have everyone score according to the sum-total requirements, you may very well end up with a purchasing solution when your original goal was an HR solution. In this case, there was a lack of weighting for HR requirements and HR stakeholders’ votes.

Some old fashioned leadership can work here, where the key stakeholder makes the final decision and is accountable for it, taking into account everyone’s scorecard. Sure, this is an extreme example, but it illustrates my point about requirement dilution clearly.

The second way to screw up an RFP is to limit the value you can get from a procurement. Let’s say you send out an RFP for an HR solution and one of the vendors says they can do financials as well. You could say, well, we are only looking for an HR solution (that could very well be the case, but let’s say there is no solution for financials in place). You could talk to the guy in charge of financials and let him know there is an opportunity. My first point is relevant here because really the software products are no longer commodities. Either you should open the RFP up to vendors who can do HR AND financials or make a decision based on thorough investigation of the functionality with management consensus about the overall benefit of each product to the organization. In this case, it’s more opportunity lost, but finding opportunities and bringing them forward is how people win leadership awards (or keep their jobs in a rough economy).

To summarize, have clearly defined business needs and requirements and stick to those when making your decision. If you find you are trying to choose between apples and oranges, step back and regroup with management to determine the overall value of each product to the organization. In our industry, single tools that provide functionality in SCM, development and IT infrastructure are hardly commoditized today and have small overlap with one another.

I suppose it boils down to having clear requirements, stakeholder involvement and effective leadership. But, isn’t that always the case?

We’ve been testing Mojo and Meister 7.2.1 and getting ready for their release on December 15. This is a maintenance release with bug fixes from users who’ve started running builds and workflows with Meister 7.2 and using the free workflow automation of Mojo and putting those releases through their paces. It also contains a lot of UI and documentation improvements.

Existing users of the 7.2 version of either Mojo or Meister will be able to upgrade via the update sites, http://www.openmakesoftware.com/mojo/update_site and http://www.openmakesoftware.com/meister/update_site, respectively.

Users interested in getting Mojo, the free workflow automation tool, and Meister, the industry leading build automation tool, can find download instructions on our website, http:///www.openmakesoftware.com.

With the recent financial meltdown, I couldn’t help but notice a trend among my clients. I’ve worked with over one hundred companies in one capacity or another that has given me an insight into how they develop software.

Among these companies, there were two particularly frustrating companies where I was on site and a third that I assisted with a very difficult proof-of-concept. I actually compiled software applications for each of these companies, each of them failed to implement the enterprise software process and automation improvements I was helping them with and none of these three companies exists today - victims of risky investment practices and high-profile failures in the 2008 financial crisis.

To be sure, I’ve had many frustrations at many other companies because implementing centralized software development management practices is extremely difficult, involving almost every department in IT. But the other companies ultimately gained the consensus, management backing and financial support to implement real change to lower the risk of software delivery and improve business continuity.

The three software management failures that ultimately turned business failures were particularly sore spots for me. And, as a trained physicist, when I see three software management failures and three business failures and they are the SAME three out of a hundred, well, I know there’s a very high probability for a relationship.

An anecdote: one of the three, a super large bank liked to grow by acquiring other banks. Word on the street is that the OCC, stepped in and said you have to improve your software management practices before you can acquire more banks. So, the bank implemented a software management improvement program including first centralized version control and later more proper configuration management (always surprising to me that they can deliver binaries and manage versions but not know if the two are related in any way). I was brought in for the centralized build management part. Then the OCC said something like “We see you’ve implemented some version control. OK, you can go ahead an buy more banks.” POOF! The software improvement projects were all massively scaled back and there was no more enterprise build management to work on. How about that?

When a company implements software development and delivery improvements they are lowering the risk of proprietary software changes which in turn lowers business risk by decreasing interruptions of services and ensuring on-time delivery of new features. At this stage of industry maturity, a company that does not have control over software delivery is accepting a business risk that fewer and fewer competitors accept.

So its reasonable to believe that a company that takes large financial risks will take risks across the board - even with their software management practices.

We’ve got a couple of things going on in the next two weeks. Quinn and Dan and I were at a rousing TechCocktail Chicago event last week and met a lot of new people as well as some old friends.

Chicago Rational User Group - Presentation on IBM’s Jazz. This is at TransUnion, 555 W Adams, Wed Nov 12. Quinn Bailey will be attending this one, starts at 5pm. Registration at: https://secure.rational-ug.org/check_user.php?type=meeting&meetid=873

WindyCity.pm Perl User Group. At the Google offices, Mon, Nov 17. Topic: http://www.frdcsa.org/ . Registration is usually via a Google Doc form - not available yet. Yours truly will be attending this one.

Eclipse Demo Camp hosted by OpenMake Software. At one of our favorite restaurant/bars, The Kerryman, 70 W. Erie St., Nov 19, 6pm. Yours truly will demo “Desktop Workflow Automation in Eclipse with Mojo Plug-Ins”. Free beer - need another reason?