Also like how Facebook or Twitter allows you to specify a person’s name in lieu of the communication protocol identifier (email address or web page), Git uses aliases for long repository locations so you have a more direct, natural language and human feel to what you are doing: “git fetch linus” will pull changes from Linus’ repository, which you have only had to define once.

Here is a scenario where Steve and I are working on a part of the Linux file system to provide information useful for build management and dependency tracking, which Meister and other tools can take advantage of. Steve started by cloning the master Linux repository and started working away making changes. Steve asked me to work on another part of this project, so I cloned his repository, allowing me to pick up all his changes. I am now automatically following (Git calls it remote-tracking) Steve’s “master” branch of his repository since I started my repository by cloning his. The “master” branch is a.k.a. the “trunk” code stream. I can pick up his updates periodically with:

Now, I may also want to get updates directly from the master Linux repository, but it has a complicated URL that I won’t remember and only want to look up once. So, as a one-time command I do:

Forever after:

The “fetch” command doesn’t put the master Linux changes directly into my workspace, but off to the side for me to examine first (very nice). If I want, I can accept the changes into my local work tree. To tell me which repositories I am following (which friends), I do:

“origin/master” is my own trunk. I could also get the full repository information associated with the short names, but as long as it works, I don’t want to know what it is. For me, this type of friendly and fluid interaction with repositories is one of the major advantages over CVS and Subversion.

A great place to learn about Git is Sam Vilain’s Tutorial. He goes into a lot of detail on the benefits and how-to’s of using Git. Some of the highlights include repository space savings of over 90% and local-to-repository sync times dropping from hours in Subversion to minutes with Git. The real power of Git is in the highly distributed repositories and the ease and control of moving and accepting changes between repositories. For an open source project with a large number of developers it seems Git will really shine. Git has fine control over branching, merging and accepting or not accepting project changes according to various criteria.

A popular way to use Git is to have Git pull from a public Subversion or CVS repository with convenient integration with those tools to a local Git repository and work from there. Friends working on the same project can easily pass changes between each other with Git and later commit back to the centralized CVS or Subversion repository. GitHub provides a simple Git repository hosting service. Doing a lot of Java work with JBoss and WebSphere, I am naturally interested in an Eclipse plug-in for Git and indeed one exists. It looks like a newborn infant, but I will check it out.

I also have a Perl open source project that is currently pretty anemic, but I hope to revitalize it soon. I really hate the fact that I’m locked into using Subversion on SourceForge and I never came to like Subversion. I’m eager to explore moving the project to GitHub, even though I’ll probably be the only committer for awhile. Since I’m a hardcore software management person and robust Perl developer, I think Git might be my tool. I’ll let you know.

So, until I find a new spam filter plug-in for Word Press (and figure out how to use Word Press) I’ll have to require registration.

Thanks,

Sean

First of all, why another build tool for Java? I am occasionally told that Maven or Ant is a perfect tool, but clearly the people behind buildr don’t think so. The choice of JRuby as the vehicle for delivering this tool, I think is probably a good one. JRuby is a scripting language in the same vain as Perl, which is used by Meister.

Doing software builds is an ugly business involving lots of file and operating system interaction. This is not where Java shines, but scripting languages can. As long as operating systems are written in C and not Java, C-like tools will be better and faster at interacting with them. Plain Ruby itself is C-based, and JRuby no doubt inherits C-like operating traits. Calling out to a Java compiler from Perl or JRuby, though it has its own JVM, does not represent a significant overhead compared with the file system operations and the compilation/translation itself.

Both Maven and Ant are relatively difficult to extend compared most other build tools, and I’ll be buildr beats them here. If you have all the Maven plug-ins and Ant tasks you need, then good for you. If not, then you have to start developing in Java and it becomes too much of an investment to sink into a build system. It is much cheaper to extend in JRuby or Perl. My frequently cited example is the XMLBeans compile step in Meister, written in Perl, which is only 40 lines of real code. The Maven plug-in is 60 pages of Java code and no one can tell me really what it is doing (I asked on all the forums). Less code is usually more transparent, which is also good for build audits.

I am a little disappointed to see them try to placate the Maven and Ant users by promising it is a drop-in replacement for Maven and they have all the Ant tasks covered. Both tools have their drawbacks and I don’t want to see another tool with the same deficiencies. They should have the cajones (or coñejos) to apply all their resources to what they think is a better tool (with its own unique benefits and deficiencies). I imagine offering Ant task equivalents is pretty easy because of the ease of coding in JRuby compared with Java.

They also don’t mention who is supposed to use the tool. Is it for individuals, small development teams, the enterprise? Maven falls short because it is only appropriate for development teams and not for stable, controlled, enterprise builds. Ant is not even tool, but a means to create some tools for small teams. I don’t think Meister will fear buildr either.

Well, since buildr is only in incubation status with Apache, I’m not sure how much time I’ll be able to spend on it, but I am curious and I’ll let you know if I find out more.

Enables Greater Social Interaction

This is especially true with people that you would not socialize with frequently such as people you don’t see in the office every day. My social interaction with these people went from ‘never’ to ’sometimes’. This includes those of my friends and relatives who are a great physical distance away. As people move, my laptops explode, and email clients change, I lose track of people’s email addresses, so I miss out on sending a holiday greeting or the latest round of baby pictures.

Between Facebook and LinkedIn, I pretty much have all of my high school and college friends somewhere and that can be beneficial professionally as well as socially.

Saves Time

Some of my friends say that Facebook uses up too much of their time, but I find that Facebook in particular saves me a lot of time. I can click on anyone’s face in the Entourage application and send them a quick note. Or, I can take advantage of Facebook’s slick auto-complete where I only need to type one or two characters to uniquely identify someone. Time savings would be #1 on my list if it were not for the fact that this greater facilitation leads me to slightly more interaction. I do not consider the greater interaction to be a distraction, but it in fact slightly improves my overall quality of life.

LinkedIn also saves me time professionally. Forwarding someone’s LinkedIn profile is much faster than attaching a resume document and writing an introductory email.

Facilitates Special Interests

Facebook also helps me share my interest in less popular topics with like-minded enthusiasts. I can easily send soccer highlight videos or punk rock news to those few friends of mine who are interested. While I could create an email list to do the same, Facebook makes it far simpler and much faster to manage these types of lists. Or, I can post something, and those who spot the post on their mini-feed and are interested can click on it to learn more.

I suppose baby pictures fall into this category. Facebook in particular has awesome picture sharing functionality. The ability to ‘tag’ people in photos and have Facebook collect them is priceless. For example, I get can all the pictures taken by many different people where I am tagged and Facebook will collect them for me into a single slide show.

If you share the environment with any non-Perl applications, however, you will still have to deal with the environment profile(s). I also have some legacy shell scripts that we can’t justify converting to Perl unless they have another reason to change. (Don’t change tested code in my house ~~ head bobble + finger wave ~~, nuh-uh!)

There are two ways I know of that you can extend the benefits of your Perl implementation towards your legacy and profile shell scripts. The first is through Bahut’s excellent tip on embedding POD documentation in shell script. This solves my problem of generating HTML documentation from POD in Perl scripts and having upsetting holes where the shell scripts are. I also have some controls for the Perl scripts that run podchecker before committing to version control, which fails if no documentation is found. Now, I can extend this control to the shell scripts.

The second Perl tool you can extend is the testing functionality. I’ve found the functionality in Test::More to be useful for validating that the changes to the shell environment profiles are correct and do not introduce defects. Profiles can be notoriously tricky to change when they get fat and you have variables depending on other variables. Mostly the profiles in my case are used to set environment variables that control the version control and build system, and these can be easily validated in a test script called profiles.t via checks like:

ok( $ENV{CODE_ROOT} eq ‘/opt/code’, “CODE_ROOT set to ‘/opt/code’”);

You then just rattle off tests for all the variables that are set and you have a great way to validate that everything will still work after the profile change. For a legacy script, you may not be able to have a crack at the internals, but you can at least check the return code and maybe some external effect it has somewhere, such as a file timestamp change.

eval { `legacy_script.sh`};

ok( !$?, “legacy_script”); #– $? is zero if script executes successfully

Profile.t and any other test scripts used to test legacy shell code can be bundled with all the other Perl tests via Test::Harness for a single test suite that really tests everything shell and Perl.

With the most important listed first, they are:

1. JBoss support
2. The Perl executable (5.6-5.10) and base language
3. Perl’s Test::Simple or Test::More modules
4. Perl’s Test::Harness module
5. The JBoss twiddle.sh script or command equivalent
6. Perl’s XML::Twig
7. Perl’s Archive::Zip
8. vi
9. ssh
10. xterm

JBoss support wins hands down due to the number of bugs and critically important undocumented features. On a scale of 1 to 10 where 10 is the best documentation, I give JBoss about a 3 or 4. Googling doesn’t even help that much for deployment issues.

You may be surprised at the prominence of Perl, but if you think about what you are really doing and what the best tool for the job is, it makes sense. You are really moving an archive (a ZIP format file), copying XML files, creating directories, changing permissions, extracting the archive to the file system perhaps. Where did I mention Java? Nowhere. The twiddle.sh command comes in handy if you get the secret commands from JBoss support that tell you if the application you deployed has actually started correctly. Notice that this is a shell script suggesting we’re not the first to use non-Java tools to manage deployment.

Particularly on the testing side, I can’t think of a viable alternative to Perl testing. We need to test that we created this directory, changed that permission, updated that file timestamp, etc. We have about 300 test cases encoded in Perl that are run with every change to the deployment system. It takes about 20 seconds to write and run a simple test case in Perl.

Lessons? Use JBoss support early and often and use Perl.

I needed to create different configurations for my web application so that the service request went to different endpoints for different environments. The endpoint is at an enterprise service bus (ESB) and there is a different ESB for each environment. I need to have my ‘dev’ instance of the consumer hit the ‘dev’ instance of the ESB, the ‘qa’ instance of my web app hit the ‘qa’ instance of the ESB, etc. We’ve set up Meister to pick up the correct XML file for the target environment for the build of the WAR.

I started by setting up the spreadsheet as follows. I had an unnecessary column for Host indicating JBoss, but I hope to include WebSphere and maybe .NET as well some day. My web app actually connects to two services a.k.a. providers, so there is a column there. And, next is the configuration label for my web app with the name corresponding to the environment it is designed for. So, the first three columns of the spreadsheet look like:

Then I needed a way to indicate the resource that would change. Right now I only have XML files, but I chose to stick with a generic URL for that. Unlike Maven or Ant generators, we start with an XML file that actually works and has been tested - not some hacked up parameterized version that takes additional effort to create. The fourth column of the spreadsheet looks like the following (with repeated entries omitted):

Next, I needed a way to specify a target location to change within the XML file. Now, I know I’m going to use XPath, but I’ll want this to one day work for properties files as well, so I came up with a URL-like thing called a Universal Datum Locator (UDL) which pre-pends the method of locating the datum to change on to a method-specific locator. It could be a property name, an XPath or a Perl regex, for example. In this case it is XPath and then the last column contains the replacement value for the datum indicated by the UDL. XPath is also very intuitive and easier to construct than it may look.

The value for the UDL looks like:

xpath://beans/bean[@factory-bean=’xfireProxyFactory’]/ constructor-arg[@index=’1′]/value

So the fifth column contains the UDL’s, which in my case is always the same XPath expression. The final column of the spreadsheet contains the replacement value of the datum indicated by the UDL:

My nifty Perl script is only about 80 lines of real code and because XML::Twig is nearly the best thing in the world, I pass the entire XPath in as a hash key to modify the source XML file:

my $twig = XML::Twig->new(

pretty_print => ‘indented’,

twig_handlers => {

“$xpath” => sub {

$_->set_text($new_datum);

}

}

);

Here, “$xpath” is directly from the “UDL” column of the spreadsheet with only the ‘xpath://’ stripped off and “$new_datum” is directly from the “Value” column. That’s a pretty useful one line subroutine if you ask me. I had the new XML files each generated into a different folder (dev/,int/, etc). Then, I checked them into version control (CA Harvest) and built each of them with Meister. If you want the full code, let me know and I’ll post it somewhere.

I did find working with the Excel 2003 XML Spreadsheet format a tiny bit awkward. You have to keep track of the column and row indices, but not bad other than that. I see Microsoft Word 2007 allows you to save as an XML document directly, but you apparently have to define bindings. I’ll have to check that out.

The first check is made for the existence or non-existence of a watch file. If a previously unknown watch file is found, the appropriate deployer is started and the file modification timestamp is stored in memory. If a known watch file is found to be missing, the appropriate undeployer is launched.

If a known watch file is found on a subsequent pass of checking watch files, its timestamp is checked against the time that was stored in memory by the deploy process. If the deployed watch file is newer, the appropriate deployer is launched which apparently first dumps the associated resources and then reloads the object as if it were newly found.

This leaves a hole that can lead to the horrifying result of having files deployed to the server, but not having the changes reflected in the running application.

The issue has to do with completely replacing a running application with a new version. You might first delete the application completely from the runtime area leaving the server to undeploy it. Then you replace the object with a new version of itself. The window of time between checks of the watch files is finite and I’ve found it is possible to remove and replace the archive within that window so that the JBoss server does not detect that the watch file was missing and so it is not unloaded from memory. The server does check the watch file timestamps, but if you have changed files other than the watch files and have not updated the timestamps of the watch files themselves, the server will happily ignore the new version of the archive while running the old one.

If you use this deployment strategy, then this issue is essentially a random process, and a deployment failure due to this reason happened in our case on only a few percent of all deployments. When you are running a few hundred deployments a week, or it happens for a production deployment it becomes a big problem – especially when people don’t know what the problem is. A simple resolution is to always update the timestamps of the watch files when changing anything for a deployed application. This will take care of everything but possibly compiled JSP’s. (Possibly more on that later.)

This also points to a “restart” mechanism for JBoss – simply ‘touch’ the watch files of a running application to change their timestamps to the current time. This will trigger the dump-and-reload on the next watch file check. This can be useful when the application has not changed, but an associated XML resource has.

Since this can lock the targets exclusively and prevent others from updating the target, you may not want to check out the build project, but you may still want to develop freely and update your local targets for Meister to build it. For this situation I recommend creating a separate build project that you may never check in to version control. It will be writable and it allows you great freedom for a maximally agile development environment. The ‘official’ build project may reference all the built archives in the workspace, but having your own local build project can allow you to focus for a unit build. For example, my workspace may contain an EAR project, a WAR project and one or more JAR projects. If I am principally working only on one of the JAR projects, my local build project can reference only that one JAR project.

When it’s time to release your JAR code updates to the system build and test environments, synchronize your workspace and check out the VC build project. Generate your targets, do a local system build and then check everything in. Your team system build will work fine!