Open Source Release Engineering at Scale:
Keeping Firefox Running on Millions of Desktops
J. Paul Reed
Senior Build/Release Monkey
Release Engineering? Wuzzuh?
- The Web 2.0 Answer
- Release engineering attempts to bring consistency, reproducibility, and verifiability to the build and release cycle.
Buzzword Bingo
- Other buzzwords
- Software configuration management
- Build automation
- Continuous integration
- Change management
- Release management
- Packaging & deployment
- Porting
But in practice, that means?
- Different organizations... uhh... organize it differently
- For our purposes...
- Someone who "polishes bits"
- Help the community figure out/implement release processes
- Steward of the "official bits"
But... in practice... that means??!
- Maintain production build farm/systems
- Maintain production release infrastructure (ftp.m.o)
- Responsible for turning CVS checkins into builds for regular testing/QA consumption
Sounds Like Closed Source!
- "Nightly testers" == thousands of people
- Build process, logs: completely open
- Community-shared release infrastructure
- ftp.m.o
- Automatic updates
- Feedback from all angles
diff -uN8 open.src closed.src
- "Supported" platform/environment doesn't really mean anything
- Extra coordination...
- External library maintainers
- Community members
- Between timezones
- Between projects you may not even know about
- Translation: "Extra overhead?"
- Releases are driven by more transparent factors
The Browser is Different?
- Disclaimer: I <3 Apache and the Kernel
- It's not just posting source tarballs
- "Finalizing" the source
- Binary builds, including Win32!
- So, just ./configure && make && tar -cf && ftp?
Just a bit more than make install...
- 40+ localized builds
- Build, verify, publish automatic updates
- Sign builds as appropriate
- Support/develop infrastructure to handle releases
Weird Effects
- Updates affect many people
- "Oops, you broke The Web (tm)"
- Security firedrills can ruin your weekend
- Community actions have a direct impact on users
- The Anti-release?
Coping Mechanisms
- Reference Platforms
- More methodical (anal?) configuration management
- Version numbers (pre, rcN)
- More CVS tags per release than any other open source project!
- Generally, a more standardized process
It Ain't Easy Being Preed
- Distinction of "Project Asshole"
- Teaching "bad Build words"
- "Oh, you wanted a life with that?"
- All eyes on you!
Pretty Rewarding, Though...
- Last person to see the release go out the door
- Announcing the release
- Spotlight is a double-edged sword
A Picture is Worth 1,000 Releases
Parts I'm Going to Gloss Over
- The Internets (they're not a big truck)
- Users' Computers
- Tinderbox
- cvs.m.o
- And svn.m.o...
- Aaand hg.m.o...
- The Mirror Farm
Fitting It All Together
Fitting It All Together Over Time
Kicking Connections Around
- Bouncer written in 2005
- Provides Mirror Management
- Verifies file integrity
- Allows us to help mirrors who are helping us out
Bits To and Fro
Handing Out Upgrades Like Candy
- "Automated Update Service"
- Has two components: client/server
- XML-based
- Updates are...
- Relatively simple format that's evolved
- Described by "snippets"
- Simplicity is both a blessing and a curse
- Newfound respect for the "automatic update" problem
One Core Component Missing
- Volunteers!
- Hard to participate in Build
- Try to solve with...
