Teaching mar to Use Manifests

05/24/2010

While looking through Planet the other day, I noticed that my name had a red line under it, which was both surprising¹, and a gave me a good kick in the pants to write about some Mozilla-stuff we’ve been working on lately².

One of Mozilla’s most underrated contributions to the body of open source, I think, is its cross platform update system. Back in the day, I had always wanted to work on making it more generic for other projects to make use of³.

Being a Mozilla-derived project, Songbird uses these update mechanisms provided by the platform, and for the most part, they’ve served us well.

Being a XULRunner-based application, with a large number of XPCOM components, one problem we did run into when generating updates⁴ was the method in which the mar tool determines which lists of files it should pack into the mar.

The mar tool determines this list of files by having them passed in the command line to the mar tool itself. This is problematic for a couple of reasons:

• On Win32, the maximum length of a command line is laughably short⁵
• Because the paths are passed in via the command line, you have a shell involved; and when you have a shell involved with filenames, you always have quoting issues⁶

We’re apparently not the only in XULRunner App Land to have this problem⁷.

We fixed this with a series of patches in two areas:

• A patch to the mar tool itself to add the ability to specify a manifest file instead of passing the files in on the command line
• A couple of patches to the (bash shell!) packaging tools⁸ to coerce them to use this option.

The contextual details of both of those fixes are in the two (Songbird) bugs. We’ve been using the patches for a couple of releases without any problems.

Now, if other XULrunner-based projects run into this problem (or Firefox becomes big enough that it has enough files that Things Stop Working(tm) because some magical, under-documented limit was hit), they’ll be able to, at-best, yank our solution, or at-worst at least know there is a known workaround to base a solution on.

_________________
¹ It’s really been that long?! I have been writing, but I guess it’s all been about other random things…
² Lots of changes in Mozilla-land lately, so there’s plenty to say!
³ But, alas, never found the time; it’s a shame really
⁴ Of course, at release time, so this was a fun fire drill to fix!
⁵ Compared to other modern operating systems
⁶ Preed’s 43rd Law Of Build Engineering
⁷ Although, we did apparently solve it oh-so-slightly differently
⁸ Which are (being?) rewritten in python, apparently