Posts Tagged ‘tools’
Friday, April 27th, 2012
Just a quick note before your Friday happy hour get started: QuickRelease 0.14 has shipped.
This release focuses on a request we’ve had from many people: more documentation and examples.
0.14 sports:
We didn’t get to generated tarballs for releases in this release; that will be in 0.15. For now, you can grab a tarball from github as always.
We hope the documentation helps! If not, hit up the QuickRelease Google Group, and we’ll see how we can help!
Now, on to that happy hour!
process, quickrelease, tools
Releases, tipstricks | No Comments »
V1093
Tuesday, April 10th, 2012
Just a quick announcement that the newest version of QuickRelease has shipped!
(Lucky!) 0.13, most notably, has the following updates/improvements:
- Entries from your release configuration files can now be coerced by the ConfigSpec class into dictionaries; you can do this be defining an item in the configuration file with the syntax [key1 value1] [key2 value2]
- You can now define command-line overrides for variables; you must set allow_config_overrides to true in your config file to allow this behavior, but after doing so, you can redefine variables from the commandline, like so: quickrelease -p SomeProcess -c myconfig.cfg -DoverrideVariable=overrideValue -DanotherOverride=somethingElse. This is especially useful for processes that are used daily, and thus need minor configuration changes.
- Addition of a chdir() utility method that follows shell semantics; this is a bit of a random feature, but some programs (Perforce’s p4 command line utility, most notably) rely on a correctly-set PWD environment variable instead of getcwd(3) to know which directory they’re operating in; this function provides a compatibility layer for such programs.
- You can now pass either a filename or an input stream to RunShellCommand(), and it will hook this up to the calling process. This is useful for some programs (again, notably, p4) where you can script their behavior by giving them commands via STDIN.
Packages can be grabbed directly from github. (We’ll be publishing “proper” packages for future releases.)
As always, if you run into any problems, don’t hesitate to let us know
And if you’re interested in the project, feel free to follow/fork us on github!
The focus on 0.14 will be something many have been asking for: more documentation, and examples, including a sample set of processes, steps, and configuration to build Firefox!
open source, quickrelease, Releases, tools
Releases | No Comments »
V1055
Monday, December 12th, 2011
A few weeks ago, I tweeted about a great blog post on a technique for managing the massive amounts of email many of us navigate daily.
I’d become frustrated with the state of my own inbox1 and had been on the prowl for a new method to manage the email torrent.
Having used it for a little over a week now, I can say: it works wonders!
One caveat: the author uses Postbox, which has some core features that are the cornerstone of the technique.
Now, I’ve been a longtime proponent of Postbox, and I use it as my work client. But for a variety of historical reasons, my personal email-life is centered around the venerable Mutt mail client2. So, the technique described in the article works perfectly if you use Postbox.
This is how I successfully re-created the work flow with Mutt.
Labeling
The technique relies heavily on the ability to label mail, as Gmail and Postbox both easily and handily support.
Mutt calls this “tagging,” and while there’s been support for reading the X-Label header since 1.4, support to edit labels isn’t in a currently-released Mutt.
The good news: there’s a patch to support label/tag editing.
The bad news: the patch is against a pretty old version of Mutt3.
I’m a Gentoo user, so I updated the patch4 to apply to [Gentoo's] 1.5.205,6 and also wrote an ebuild8 you can add into a local overlay to make it easier to install9.
After that, add the following to your muttrc to bind label-editing to keys and display labels:
# Add X-Label to your unignored headers
unignore From: To: Subject: Date: Cc: Reply-To: X-Label:
color header brightwhite default '^X-Label:'
macro index \Cy "
~y " "Limit view to label"
The patch binds the ‘y’ key to adding, deleting, or changing a message label in both Mutt’s message and index views; the above config binds Ctrl-y to limiting the view of the folder based on a specific label.
When editing labels, they are space-separated.
Search, Don’t Sort
The “Search, don’t sort”-mantra isn’t new; but when all you have is grep, you still tend to sort your email as an initial step to being able to easily find it later.
Enter mairix10.
Setup is relatively easy: configure it with an rc file, and run it. I run it from cron every four hours to keep the index fresh, and reset the entire index every month11
Since mairix provides its results as an mbox or Maildir folder, I wrote a simple wrapper that only launches Mutt if there are results to my search and makes the folder read-only, to keep the conceptual model—these are search results, and I shouldn’t edit them—consistent.
As for sorting, I still do sort some email, mostly Twitter, Facebook, and other automated notifications, as well as mail from important people that tends to fit neatly into boundaries (my parents, certain friends, etc.)
Keeping Folders Tidy
Now that my mail is searchable, where we save it isn’t as important. Now I can move to a model of just saving everything to mutt’s default saved-messages and going from there.
Only problem is, won’t that folder balloon over time? Probably.
Cron comes to our rescue again; I have the following entry:
5 0 1 * * MAILBOX="/home/preed/mail/saved-messages-archive/saved-messages-$(/bin/date +\%Y-\%m)" && /usr/bin/find /home/preed/mail/ -maxdepth 1 -type l -name 'saved-messages' -exec rm {} \; && /bin/touch $MAILBOX && /bin/ln -s $MAILBOX /home/preed/mail/saved-messages
This makes “saved-messages” a symlink that automatically gets updated at 12:05 am every month to point to a folder with the year and month.
And mairix updates its index with these new folders the index gets reset every month.
I use this same technique for my sent-messages, spamassassin-populated spam folder, and deleted-messages folder; the last two are auto-deleted by cron six months later.
Filtering Assaults On Your Inbox
The last tenent of this email management technique is probably the most important: if you don’t want it in your inbox again, set up a filter immediately. Don’t wait.
I’d been getting spam mails from Ticketmaster forevar. I didn’t set up a filter for them until I started using this technique.
It’s a bit costly at first, but it pays off and quicker than you think.
To help with this, I set up an easy way to test procmail rules without running them on my live inbox.
I stole the scripts from this page and slightly modified them to fit paths on my system.
Inbox Zero?!!
After using this method for about a week, I was able to dump most of my inbox into a saved-messages-archive folder, and begin using the dated saved-messages folder.
I’ve been able to find any message I’ve needed to find, and my inbox now hovers around 50 messages.
Finally, people emailing me are getting responses on the order of 72 hours; before, if their message got lost in my inbox, it was hit and miss; I just restarted a four month old thread, because I’d lost track of it, and couldn’t tag it “todo.”
Have I reached Inbox Zero? No. But as the original post points out: “I don’t care about ‘Inbox Zero’ because there’ll be more [email] tomorrow as soon as the sun comes up.”
But using this technique, I’ve been able to turn my inbox into something manageable, without the fear12 that I’ll lose a message.
All in all, I’ve been very impressed with how it’s been working so far. Hats off to Mr. Ignition for the inspiration!
_______________
1 Over 2,000 messages in 75 megabytes before I started this project
2 Having used them since I was 13, my customized Mutt key bindings, cribbed from Pine, are hopelessly burned into my muscle memory
3 It’s against 1.5.1; the current released version is 1.5.21
4 PGP sig
5 Which I’m using because of an attachment handling bug in 1.5.21
6 I also hacked up patches against pristine-1.5.20 (PGP sig) and pristine-1.5.217 (PGP sig)
7 This patch doesn’t include the documentation changes; that’s left as an exercise for the reader
8 I just added one to the ebuild revision; I’d love to hear from a Gentoo-dev on whether this is the correct way to do this, because I’m betting it’s not…
9 There’s also a method to allow editing of labels using an external script, but I didn’t use this script, so I can’t vouch for how well it works
10 Which I have @scanlime to thank for turning me on to
11 A full index of my 1.6 gigabytes of mail takes 97 seconds; reindexes only look at new messages, so it’s pretty low cost
12 Which was probably largely irrational anyway
lessons, process, tips, tools
blahblahblah, personal, tipstricks | No Comments »
V919
Saturday, June 11th, 2011
I started using Perforce in 2003, for my first post-collegiate job.
Coming from a CVS-centric world1 it was weird and confusing to me.
But as I used it, I began to really like its workflow and the tool’s conceptual consistency.
Fast forward eight years, and I’m using Perforce again3. I convinced the Powers That Be of the utility in sending our team to this year’s the Perforce Conference. I was very interested to see what had changed with Perforce in a market that’s seen surprising interest over that time.
The 2011 conference, “Roadmap to Innovation,” certainly fulfilled that desire. For me, I found the following stops on the highway most interesting:
- A new mantra of “Version Everything.” Putting everything in version control is old hat to configuration management engineers4. What’s interesting is that Perforce is making this a new cornerstone of both their product and their marketing message, thus reinventing themselves as a platform to do versioning.
You can see this with their new Chronicle offering. It’s a great move. So many tools5 have weak versioning functionality that seems tacked on as an afterthought6, so to take a tool that has proven itself time and again to bring rich, mature versioning functionality to other markets is a great direction to see Perforce moving in.
- As soon as I heard about it, I tweeted “Perforce Streams + Sandbox is distributed version control that doesn’t bite developers.”
I can’t count the number of times I’ve heard developers complaining their fancy DVCS of choice acted in a way they didn’t intend, destroying hours’7 worth of their work8. These two features feel like a culmination of Perforce stripping away the DVCS hype and looking at the features developers really use in those systems… and then building that, with Perforce’s famous attention to detail.
Sandbox brings offline operations to the Perforce world; there were some good questions asked at the presentation, and there are some corner cases to still be worked out, but it brings the common workflows developers care about to Perforce, with a usage model they’re used to9.
Streams, on the other hand, brings CM patterns many of us have been performing by hand—sparse branching and merge-down/copy-up branch patterns—to the P4V UI, and available directly to developers, so you don’t need to involve your overworked-build team.
Frankly, I’d been worried about Perforce with respect to the movement around DVCS tools for awhile, but the combination of these features makes me confident not only that Perforce has an answer, but that it’s the right one.
- The diversity in attendees at the conference demonstrated Perforce continued flexibility in the configuration management space: the myriad use cases, environments, artifacts versioned, and installation sizes illustrate the ways in which Perforce performs well against all sorts of workloads, in all sorts of environments: everything from Google versioning its world to Pixar versioning its production assets to the NYSE versioning entire machine configurations10.
And for every huge customer that we see presentations for11, there are hundreds of Perforce customers that have installations humming along every day with minimal administration and effort, and they perform wonderfully.
In short, the conference’s more than delivered on a “Roadmap to Innovation.” As someone who’s watched the Perforce ecosystem grow since 2003, I’m very pleased to see it’s thriving today.
I’m already looking forward to next year’s conference, if only to remind Michael Shields that we have a bet… a friendly, fun side effect of one-too-many p4 obliterates…
_______________
1 Unlike many college students, I’d been using source control for all of my school projects2
2 This, after an incident where I came home smashed from a party and thought that was a perfect time to “finish up” a mostly-working project
3 This detail was, in fact, one of the reasons I looked forward to starting at SYMC
4 Today’s developers mostly seem to be on board that train too
5 Wikis especially
6 The lack of annotate ability in modern wikis is both amazing and rage-inducing…
7 Sometimes days’
8 The fact that they often continue to blindly espouse these tools has prompted me to call them “The Ike Turner of version control systems
9 No more git resets eating code
10 SEC audits are, apparently, how do you say… “a bitch”
11 Google, famously, has its repository still on one server
Perforce, release engineering, tools
Preed on Build/Release Engineering, Releng Machinery, Reviews | No Comments »
V652
Saturday, December 4th, 2010
Well, that apparently struck a nerve1.
Much has been said about those who argue on the Internet2, so I’ll state the obvious: I’m pretty sure my chance of convincing Buildbot’s developers that the tool they work on suffers from ongoingly poor engineering design decisions is about the same of them convincing me to be an advocate for their tool3.
If you don’t have anything fanboy-riffic to say… |
Let me start by saying I’m a bit disappointed in the Buildbot community: Amber Yust starts by characterizing my post as a “rant,” presumably to cast doubt on its credibility, before providing a single counterpoint.
The project’s current maintainer, Dustin Mitchell, called me a “curmudgeon” and purported to possess some definitive knowledge into when I last used Buildbot; I can guess where he got this notion, but it’s incorrect4.
All this after Buildbot developers called me “incompetent“5, and responded to points I never made, but to anyone who had assumed I had made them, it sure sounded like I was suggesting absurdities.
After the name calling was out of the way, every person who said anything6 did, at some point, intimate “Well, this guy has a point.”7
So before I respond to some of the specific points, it’s important to note that the Buildbot community’s response to criticism they eventually agreed with at least parts of was to level personal mischaracterizations and verbally throw sand? I wonder if there’s a coincidence between this sort of apparently-acceptable behavior and the “Developer Recruitment” agenda line item on their upcoming summit…
That said:
- I am using an older version of Buildbot8; I haven’t invested much time looking into future revisions, since core design failures—the always-connected-slave problem, for instance—have yet to be addressed; right now, if I had that time, it would be utilized to replace Buildbot.
- With respect to features that are supposedly in later versions, I was very careful about the missing features I chose; I understand all software is a “work in progress,” but I picked those that I felt their lack of inclusion in any shipping version of a capable continuous integration system to be egregious.
- My complaint about properties was poorly explained; I’ll use one of my patented footnotes, in what may be the longest yet in the history of this blog, to rectify this9.
- Yust claims that Buildbot does not “force” you to describe a build process in its venerable master.cfgs; she’s right, and I didn’t claim otherwise. I said it happily leads you to its “encode your entire build process in my non-portable master.cfg“-trough and dares you to drink. The sample code nudges you in this direction, and one need only look at Mozilla Corporation’s current11 effort to undo this costly mistake. I have yet to see a Buildbot deployment that hasn’t initially made this mistake, which makes sense: people copy the sample code and modify it.
- The suggestion to run Buildbot on port 80 largely misses the point that dealing with large, corporate IT departments often involves working with rules that don’t make immediate sense. Requiring custom ports running unaudited software to be open to the world is usually a non-starter.12,13
- Regarding the list of “It’s On the List”s14, frankly “I don’t care.”
Why?
The amount of time organizations and companies have wasted re-running builds that had to be restarted due to this crazy assumption of a bulletproof network connection is a design error of Challenger-esque proportions.
For any engineer to say with a straight face “Well yah, it’s a problem,” and in the same breath say “it’s on a list,” and then not have any movement on it IN SEVEN YEARS is completely inexcusable from any reasonable engineering perspective.
The fact that there is such a list of what are, in some respects, similar calibers of engineering design failures, that are apparently on some list somewhere to eventually correct someday when maybe there’s time lends credence to my point that Buildbot advertising itself as a production continuous integration system should be reconsidered.
It is Yust’s conclusion, however, that really gets to the heart of the matter: “Buildbot is not a just-add-water CI server.”
I wholeheartedly agree.
The problem is: either through the (original?) authors’ claims, the verbiage on the website, the documentation, the way the community positions the tool, or some other factor, this is not the notion would-be users walk away with.
Buildbot was originally sold to me as “A better Tinderbox, with a bustling community.” This was right before Brian Warner all but abandoned it, which was fitting, since the other statement didn’t turn out to be particularly true either.
As a build/release engineer who has plenty of other things to be doing to support his organization’s software development and QA teams I have ZERO INTEREST in customizing, tweaking, and effectively dumbing down a “distributed build manager” to be, in Yust’s words, “just a [continuous integration] server,” which is all I need it to be.
Compound that with these whacky, mostly-unstated requirements of a perfect network under ideal intra-company/project group relationships, and Buildbot is a poor choice that wastes a bunch of time that many release engineering teams, mine included, do not have.
If the Buildbot community’s position is “Buildbot is not a continuous integration server that usable for standard software projects, in the real world, under real networking and systems conditions,” and is intended to be some redesigned distcc offering, just say that. Stop letting engineering managers and VPs think otherwise15.
And if your answers to your users’ complaints are always “Well, Buildbot wasn’t designed to do that,” as Brian Warner reportedly stated, then you, Buildbot community, need to do a better job of clarifying what, exactly, Buildbot is designed to be competent at.
(Bonus points for attempting that without the name calling or wholesale mischaracterizations.)
In either event, as a stressed, time-constrained release engineer, I need a “set it and forget it”-solution17 that gets my builds out the door without requiring me to deal with a lot of weird software dependencies, mysterious, unexplained assumptions, and long-lingering, still-uncorrected, painful design decisions.
And based on Yust’s own statements, one thing we apparently agree on: the system best suited for that purpose is not Buildbot.
_______________
1 I had planned to link to two blog posts, since it was implied another would be written, but I haven’t seen it appear anywhere yet…
2 Googling that phrase is left as an exercise for the reader
3 At least at this point in time
4 And irrelevant to any cogent argument
5 Points go to “exarkun” for the money shot quotation: “The guy is probably incompetent, most guys are.”
6 Save Dustin Mitchell
7 Which point people agreed I had, however, wasn’t consistent, which is fine; I’ll take what I can get…
8 0.7.8, to be exact
9 I’m really conflating runtime evaluation of code and properties; my complaint is that properties shouldn’t be necessary, but are if you want to control certain elements of the build process. This requires adding a step, and often the contents of that step is echoing the output of a command, which is also somewhat asinine. My particular problem was I wanted to set a port value that a tool picks up from the environment; I had code that changed this port based on the type of build and some other attributes, and set it in the environment for the step by providing a hash; unfortunately, the port number never changed, and it took a few hours to figure out that Buildbot doesn’t support this common use case without setting properties; this is a poor, counterintuitive design when you’re allowed to write and integrate your own Python modules into Buildbot.10
10 While I’m correcting errors, I kept referring to “twistd,” which is Twisted’s cute name for its daemon; everywhere I wrote twistd, I was referring to Twisted.
11 Quite painful, from what I hear
12 If you ever wondered why Mozilla Corporation’s Buildbot masters aren’t open to the world, that’s why; incidentally, that was one of the last deployment discussions I was involved in, and at the time, everyone agreed with me.
13 And no Mook, I don’t think the transport layer should be SMTP, though I always did admire that resiliency of that particular Tinderbox data transport design decision; I would be happy with what the rest of the world uses these days: HTTP.
14 Or “the points I made that they discovered they maybe sorta kinda agreed with after getting the personal attacks out of the way and reading what I had written
15 Assuming, of course, they’re not planning on hiring Buildbot’s project maintainers16
16 Who are apparently already spoken for…
17 Does RonCo do software?
buildbot, continuous integration, tools
Preed on Build/Release Engineering, Releng Machinery | 8 Comments »
V589
« Older Entries