As Steve wrote last year, we’ve been using SQLite as a backend for some of our datatypes in our omsubmit and om command line executables. And it seems we’re in good company: here are a few products that are also using SQLite

• It’s the Ruby on Rails default DB
• Adobe Air
• Mozilla Firefox
• Mac OS X

It’s such a popular tool, that its creator, Dr. D. Richard Hipp, won a 2005 Open Source Award from Google and O’Reilly. About a year later, Dr. Hipp gave a Google Tech Talk introducing SQLite to Google. In that talk, he remarked on a number of features and benefits of SQLite:

• SQLite is designed to be embedded, it can be less than 250 KB in size.
• Some uncommon SQLite uses:
  • stand-in for client-server DBMS during testing/debugging.
  • Local database caching.
  • Implementing complex data structures.
  • Sorting large amounts of data.
  • Configuration files.
  • IPC via database.
  • Application file formats.
• SQLite is very convenient to use as a tool to teach basics of SQL, as it just works.

Of the uncommon uses for SQLite, I can speak to at least two of them, because that’s how we use it internally. Originally, I was looking for a good way to persist tree data to disk, since I was having timing issues with reading and writing linked-list data to disk. I literally googled “persist binary trees to disk” and one of the hits said “use SQLite”. Steve took this idea further and used the database as a way to do interprocess communication between the main omsubmit daemon and the child processes that run each activity. This allowed us to move away from shared memory and message queues that seemed to be a bit flaky.

As for the second point, SQLite is a good tool to teach SQL. But more importantly, it gives a really good insight into how to use a database. With the big “enterprise” databases, you read and write and go on your way. With SQLite embedded in your program, you have to actually think! For example, if you do multithreaded design, you can’t have two threads updating the database or you’ll get a deadlock. Furthermore, if you do a big SELECT with a bunch of processing for each row on a worker thread, you’ll get writer starvation on the main thread. We had to come up to speed pretty quickly in terms of good design to make sure we didn’t introduce new timing issues using SQLite, since our goal was to reduce timing issues.

We’ve also used SQLite in our om build engine executable to do more efficient wildcard matching and we’re seeing really good speed improvements there. That improvement will be available in our upcoming release.

Here are some tools to investigate SQLite:

• Perl DBI module. Great for prototyping code and databases, especially from the debugger.
• GUIs
  • SQLite Browser
  • SQLite Spy
  • SQLite Manager, which is a Firefox Add-on!

Hope this helps you get into using SQLite.

- Jim

I’ve been trying to add support to our Visual Studio Add in to create target definition files (TGTs) for Web Deployment Projects. I need to get meta-data information about the .wdproj file from within Visual Studio. Within the Visual Studio Extensibility object model, these items are marked as

project.Kind == EnvDTE.Constants.vsProjectKindUnmodeled;

meaning that the object model only provides limited information about the project. As far as I can tell, only project.Name and project.Type are valid properties; the rest throw System.NotImplementedException.

This causes a number of problems, as other Visual Studio projects also use the “unmodeled” project type. For example, database projects return the same project Kind, and I’m not interested in them at all. I needed a way to differentiate between sub-project types where the overall project type was “unmodeled”.

A quick and dirty way to do this would be to look at the

project.FileName

property and see what extension it has. A “.dbp” extension would be a database project and a “.wdproj” extension would be a web deployment project. There are two things wrong with this approach. It’s frail — what if Microsoft updates database projects to be .dbproj files using MSBuild (they may have already done this in VS 2008)? More importantly, both database projects and web deployment projects do not implement the FileName property (it throws a NotImplemented exception).

The correct way to do this is to find out the project’s subtype GUID. Looking into the XML of the .wdproj file, and looking at the registry under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\8.0\Projects, I know the Project Type GUIDS are:

• Database : {4F174C21-8C12-11D0-8340-0000F80270F8}
• WebDeploy : {2CFEAB61-6A3B-4EB8-B523-560B4BEEF521}

You can also see this in a solution file. Each project is listed in this format:

Project("<Project Type GUID>") = "<Project Name>", "<Location>", "<Individual Project GUID>"

So, I needed a way for the project object inside of Visual Studio to return its project type GUID, so I could determine what type of project I’m dealing with.

At this point, I was at the limit of the Extensibility API. A quick Googling revealed that I would have to use the VSPackage interfaces, the underlying API of Visual Studio. Originally, I had thought that you could not do this without downloading super-secret APIs from the Visual Studio Integration Partner (VSIP) program, but according to Craig Skibo, the SDK is now free. You just have to access the Visual Studio SDK.

The MZ-Tools HOW-TO series has a nice article on how to access the project GUID for a project throught the IVsHeirarchy and IVsAggregatableProject interfaces. The two problems are that it relies on using the project.UniqueName property, which throws a NotImplemented exception in my case; and that some projects won’t implement the IVsAggregatableProject interface. I added a quick work-around for it, which is not perfect, but seems to do the trick. (Sorry for the weird formatting).

private static String getProjectTypeGuids( EnvDTE.Project proj )
{
    String sProjectTypeGuids = "";
    String name;
    Object objService;
    EnvDTE.Solution sln = proj.DTE.Solution;
    Microsoft.VisualStudio.Shell.Interop.IVsSolution objIVsSolution;
    Microsoft.VisualStudio.Shell.Interop.IVsHierarchy objIVsHierarchy = null;
    Microsoft.VisualStudio.Shell.Interop.IVsAggregatableProject objIVsAggregratableProject = null;
    int iResult;

    objService = _getService(proj.DTE, typeof(Microsoft.VisualStudio.Shell.Interop.IVsSolution));
    objIVsSolution = (Microsoft.VisualStudio.Shell.Interop.IVsSolution) objService;
    try
    {
	name = proj.UniqueName;
	iResult = objIVsSolution.GetProjectOfUniqueName(name, out objIVsHierarchy);
	if (iResult == 0)
	{
	    objIVsAggregratableProject = objIVsHierarchy as Microsoft.VisualStudio.Shell.Interop.IVsAggregatableProject;
	    iResult = objIVsAggregratableProject.GetAggregateProjectTypeGuids(out sProjectTypeGuids);
	}
    }
    catch
    {
	// Since UniqueName is not implemented, try looping over the
	// project files in the solution to see if we get a match on the
	// project.Name. It's not ideal, as the project.Name might not
	// be part of the project Filename.

	// call once to get size
	uint nProjects;
	iResult = objIVsSolution.GetProjectFilesInSolution( (uint)
	    Microsoft.VisualStudio.Shell.Interop.__VSGETPROJFILESFLAGS.GPFF_SKIPUNLOADEDPROJECTS,
	    (uint) 0, null, out nProjects);

	// Call again once we know the size
	string[] projectFiles = new string[nProjects];
	iResult = objIVsSolution.GetProjectFilesInSolution(
	    (uint) Microsoft.VisualStudio.Shell.Interop.__VSGETPROJFILESFLAGS.GPFF_SKIPUNLOADEDPROJECTS,
	    nProjects, projectFiles, out nProjects);

	name = proj.Name;
	foreach ( string projectFile in projectFiles)
	{
	    if ( ! String.IsNullOrEmpty(projectFile) && projectFile.Contains( proj.Name) )
	    {
		name = projectFile.Remove( 0, sln.FullName.LastIndexOf("\")+1);
		break;
	    }
	}

	iResult = objIVsSolution.GetProjectOfUniqueName(name, out objIVsHierarchy);
	if (iResult == 0)
	{
	    Microsoft.VisualStudio.Shell.Interop.IVsProject objIVsProject = objIVsHierarchy as Microsoft.VisualStudio.Shell.Interop.IVsProject;
	    if (objIVsProject != null)
	    {
	        // cast the project to get its Moniker Name
		string projFile;
		iResult = objIVsProject.GetMkDocument(Microsoft.VisualStudio.VSConstants.VSITEMID_ROOT, out projFile);
		if (iResult == 0)
		{
		    uint dwReserved = 0;
		    Guid pGuidProjectType;
		    iResult = objIVsSolution.GetProjectTypeGuid(dwReserved, projFile, out pGuidProjectType);
		    if (iResult == 0)
		    {
			sProjectTypeGuids = pGuidProjectType.ToString().ToUpperInvariant();
		    }
		}
	    }
	}
    }
    return sProjectTypeGuids;
}

private static Object _getService( Object serviceProvider, System.Type type)
{
   return _getService(serviceProvider, type.GUID);
}
private static Object _getService(Object serviceProvider, System.Guid guid)
{
    Object objService = null;
    Microsoft.VisualStudio.OLE.Interop.IServiceProvider objIServiceProvider;
    IntPtr objIntPtr;
    int hr;
    Guid objSIDGuid;
    Guid objIIDGuid;

    objSIDGuid = objIIDGuid = guid;
    objIServiceProvider = (Microsoft.VisualStudio.OLE.Interop.IServiceProvider) serviceProvider;
    hr = objIServiceProvider.QueryService(ref objSIDGuid, ref objIIDGuid, out objIntPtr);
    if (hr != 0)
    {
	System.Runtime.InteropServices.Marshal.ThrowExceptionForHR(hr);
    }
    else if (!objIntPtr.Equals(IntPtr.Zero))
    {
	objService = System.Runtime.InteropServices.Marshal.GetObjectForIUnknown(objIntPtr);
	System.Runtime.InteropServices.Marshal.Release(objIntPtr);
    }

    return objService;
}

So far, so good. I can now identify and differentiate database projects from web deployment projects based on GUIDs. The next step is to figure out how to instantiate the sub APIs for the web deployment projects so I can get at the real data I need.

On Friday I tried to upgrade from Team Foundation Server 2008 Beta 2 to TFS Release to Market (RTM) Workgroup edition on a virtual machine. I wanted to get a running RTM version of TFS Workgroup, for which we are licensed through MSDN. But why on the Virtual PC image from the Beta 2 period, which expired in February? Two reasons:

• Due to a bad VPC image, the Windows 2003 R2 Server license on the VPC expired early. We had to burn one of our own W2k3 Licenses just to keep the VPC running. Since I understand as much Microsoft licensing as much as the next guy (i.e. not at all), I wanted to upgrade on our own licensed W2K3 machine.
• I wanted to see how hard it was to upgrade.

Well, it turns out, it’s really hard. Even without updating SQL Server or Sharepoint, and keeping the application tier and the data tier on the same machine, and upgrading Workgroup to Workgroup, it still didn’t work.

First, I had to uninstall all of these products. That’s a long list. The last install was the actual server install. And of course, it failed. The failure seems to be related to the user account doing the uninstall, but after three hours, I couldn’t get it to work, so I now have a junked install. I can’t uninstall the old version, and cannot install the new version without uninstalling the old one.

So, I’m now pulling down the new Virtual PC image for TFS RTM that runs until Dec 31, 2008 (the ever-helpful Martin Woodward has a list of all the URLs if you use a download manager). We can evaluate the RTM there.

- Jim

Small post here. I’ve been tweaking some Eclipse settings and wanted to pass them on.

Show Annotated

I don’t know how long this has been in the Eclipse Editor, but turning on the Editor Presentation action set and then selecting the Show Annotated is really cool. You can highlight a function or method and then only work on that section of code in the editor.

AnyEdit

The AnyEdit plugin adds some really useful features to the Editor window for coding, like

• Convert tabs to spaces and back,
• Convert CamelCase to underscores,
• Convert to and from HTML,
• etc, etc

It’s already part of most of the EasyEclipse installs, but I just didn’t know about it. Here it is in action

Compare With Outline

I never understood why the Eclipse Compare view always created the “structured compare” panel in the top of the compare editor. It seemed to just waste a lot of real estate, especially since there was the Outline view just sitting there waiting to be utilized. I even thought about opening a bug on it … Well, don’t I feel dumb; it’s a preference

Hope these help.

– Jim

I’ve been working on our command line client (CLC) for a long time now, and I used to build it with the old 6.3 release of the OpenMake product. Now that we have the really useful Build Workflows and Build Activities setup in Meister 7, we can start to push out our build and test process.

For unit testing, we’ve added two simple post build steps to our CLC builds: C Unit tests and .NET Unit tests.

For .NET, we use the extremely easy NUnit framework. This framework is really trivial to set up and use, and we can easily use it inside the Visual Studio IDE (before checkins) and during the build process. This post, towards the end (point #12), shows how to set up an “External Tool” in Visual Studio to run NUnit tests. You can also go with TestDriven.NET, (I think it’s free for personal use and $95 for a professional copy) which will integrate and consolidate NUnit, MSTest, NCover, NCoverExplorer and all the great unit-testing frameworks out there. We could also run MSTest, the Microsoft testing frameworks, but that’s not easily integrated into Visual Studio 2005 Professional. Since we’re just ramping up, and getting the unit tests up and running is my first goal, NCover will come later, as we push our unit testing coverage up to a respectable number.

For our C and C++ development, picking a unit testing framework was harder, as there is no clear leader in the space. After some Googling, I found this blog post by Noel Llopis from a few years back that compared the leaders in the space. One of his highest recommendations was for CxxTest, an incredibly simple unit test framework that only uses C++ header files. All the tests are written as header files, and are extremely easy to set up. Trust me, if I can do it, anyone can. It requires Perl or Python to auto-generate the .cpp code from your test headers, but since Openmake requires Perl, not much of a problem. It was so easy that I added a CxxTest rule to our standard MSVC Console Build Service so that everything is built and run in the same workflow.

– Jim

In my last post, I wrote about putting Windows XP on my MacBook Pro Bootcamp partition. After that, I had to install some software and get it running!

My XP partition isn’t going to be a full step-up. I am mostly going to run virtual machine images for our demos, and maybe do a bit of coding on the side. So here’s a list of what I put on the machine, and how I did it.

The most important thing I wanted to do was to set it up in a safe manner. As Jeff Atwood points out, the single biggest issue with Windows security is running as an Adminstrator. On my desktop, I run as an Admin, and I’m too far gone in terms of switching back to a regular user. But on the laptop, I wanted to see if I could run only as a regular user.

First I created two users, “jim” and “root”. “Root” is an admin account, and “jim” is my day to day account. I needed the admin account so that I could install the .NET 2.0 framework before installing the next piece.

The most critical piece was installing Sudowin, a sudo clone for Windows. This allows you to run certain programs (or all programs, if configured that way) as an Administrator, without being an admin. In the Sudowin About page, there is a long discussion why this is a different, and better, solution to running with elevated privileges than the native “Run as ..” or Vista UACs. This install allowed me to install the remainder of my software while still running in my local account. It also has a command-line “sudo” tool.

The remaining software that I installed was

• Microsoft Virtual PC and VMWare VM Player — for our demos,
• A text editor (I’m switching between Textpad and Activestate’s Komodo Edit) — for simple text editing,
• 7-Zip — free zip/unzip,
• Perl — needed for day-to-day scripting,
• CutePDF — for printing to PDFs,
• Microsoft Visual Studio 2005 and 2008, and NUnit, FxCop, and Reflector — for when I want to actually write code,
• Firefox,
• Microsoft PowerToys for XP– to tweak the UI the way I like it,
• Sharp Keys — to tweak the keyboard the way I like it,
• Sysinternals Process Explorer — to see what’s running,
• Winpatrol — to keep tabs on anything that might get surreptitiously installed or run,
• PuTTY — to SSH into the VM Ware Linux images.

So far, so good. I’ve been really impressed with my laptop as a Windows box, and more so with running the VPC and VM images for our demos. See you at VS Live!

– Jim

I am in the process of preparing for a trip to VS Live at the end of the month. Part of that trip will be demonstrating our Team Foundation integrations to OpenMake, and I’ve had to prepare a laptop for that.

I moved to a Macbook Pro as my laptop about six months ago, and I really enjoy it. I also like the fact that I’m working in multiple environments. Mac OSX on the laptop, Windows XP on the desktop, and Linux on our servers.

But for our Microsoft demos, I have to run Windows on the laptop. Fortunately, some have said a MacbookPro is some of the best hardware to run Windows! My plan was to use Boot Camp to create a separate Windows partition (because I will want to run our demos at native speed), and then use VMWare Fusion to run off that partition if I want to run OSX and Windows simultaneously.

Preparing for the partition was difficult. Because I have been using my laptop for six months, I already have lots of kruft on the disk. The folklore is that Mac drives don’t get fragmented, but this is just not true. Boot Camp could not create the partition, and on occasion would go into kernel panic mode and bomb out. This thread discusses some ideas why I couldn’t partition. Based on their suggestions, I downloaded the demo version of iDefrag to have a peek at my HD. Files were scattered over the drive, and I needed a good 1/3 of the disk free at the end. I tried lots of manual moving of files, deleting unnecessary files, and running Disk Utility and fsck -f from single user mode. Quite a pain. After a few hours late Tuesday night, I decided my time was worth $35 and purchased the full iDefrag. Good thing I did — it ran for 26 hours compacting my disk! But it all worked out, and Boot Camp worked like a charm after that.

I elected to go with a 32Gb NTFS partition. NTFS is a bit quicker and more robust than FAT32, and the downside of not being able to write to the NTFS partition from the Mac didn’t seem like a big deal, because I have external drives that I use for big files.

Funny to be back to the dual-boot days of when I ran Windows 2000 and Red Hat on my desktop. But the VMWare Fusion technology is great — it’s the best of both worlds. In my next post I’ll detail how I set up the Windows machine, as I wanted to try to keep it clean.

– Jim

As I’ve posted before, we’re working hard on getting a tight integration into Team Foundation Server’s Team Build, so that one can execute Meister Workflows from Team Build on OpenMake remote build servers. This means you can do Java builds on Linux from within Team Foundation and have all the goodness of your work items updated and preserving your regular process. To see it in action, take a look at this Flash demo I created.

We’re obviously not the only ones interested in a thriving Team Foundation cross-platform community. Microsoft wants Team Foundation Server to be an SCM tool for all shops, not just .NET shops. Teamprise got the ball rolling by creating a cross-platform client for Team Foundation, so that one can to check-ins and check-outs from TFS (kinda important). They are also getting interested on the build front, too. Martin Woodard posted an extension for running Ant Builds from TFS. This is a important first step — it allows you to reuse existing scripts from within TFS on a Windows machine. We go further; we support builds on multiple OSs and all kinds of legacy builds and other compilers as well as Java, as we should given that it’s our primary focus, whereas SCM is Teamprise’s primary focus.

The next release of Teamprise (3.0, now in preview), allows the Teamprise clients to access the Team Build management functionality of TFS, what was previously only available in Visual Studio Team Suite. I’m looking forward to accessing this functionality to see how we can plug into it and leverage our plugins to automate adding the OpenMake integration to Team Build.

– Jim

We’ve been working hard on our cross-platform builds for Microsoft Team Foundation Server, as I’ve blogged about previously. I’ve been doing the bulk of the development work with our Team Build and Team Foundation Server integrations, and Mark and the rest of the management team are working in depth with some folks at Microsoft. It’s a really exciting topic to be working on, and I’ve got to interact with some smart folks; for example Jim Lamb at Microsoft, and Martin Woodward and Ben Pryor at Teamprise.

Our hard work is starting to get noticed. Brian Harry blogged about our “X-plat builds”. Feels good to get noticed at that level.

- Jim

As I’ve said in other posts, I tend to use Eclipse for the bulk of my C development, along with Java and Perl. There are two places where Visual Studio is, for me, an absolute necessity.

1. Running the debugger: we compile the Windows versions of our apps using the native Microsoft compilers, which Eclipse CDT debugging doesn’t yet support. So to run the debugger, I switch into Visual Studio. This is the one place where I wish the Eclipse debugger keyboard shortcuts matched Visual Studio’s (yes, I know I can remap them, just haven’t done it yet!).
2. C# development. In the last year or so, we’ve been working hard on our Microsoft integrations, through the MSBuild API on the client side, and a Visual Studio Addin. Here it would be crazy to attempt to use Eclipse.

However, I still need to check into CVS, which is trivial to do from Eclipse, but not natively supported in Visual Studio. I was back to the bad old days of switching out of Visual Studio to Windows Explorer and firing up TortoiseCVS or WinCVS … until I found TamTam CVS.

TamTam CVS is a great little ($10) add-on to Visual Studio that provides, as Dave says:

… a Microsoft Source Code Control (MSSCCI) compliant implementation that seamlessly integrates Concurrent Versions System NT (CVSNT) client with Microsoft Visual Studio / Visual Studio.NET 2002/2003/2005, MatLab, FoxPro, 3D Studio Max and other MSSCCI compliant applications.

I’m also using it in Visual Studio 2008.

TamTam “just worked” when I installed it, giving me 95% of the functionality I wanted right away. I felt there was one feature missing (the ability to store “template” commit comments, which we use to link commits back to the issues that we are fixing), and I asked Dave if that feature were possible. He added it in under a week! (See version 1.2.1). Thanks Dave!

For $10, less than you might spend on lunch, you’re avoid tons of seemingly minor annoyances that really add up during the day. And the support was excellent. I haven’t used TamTam SVN, but I bet it’s of similar quality.

– Jim