Tour of the Chromium Continuous Integration Console

Chromium uses LUCI to run continuous builds and tests. The most useful is the Main Console. From the top section you'll see other consoles linked. Each can display its own console view. The Chromium Console shows the status of many tests. That Console shows a lot of information that can be hard to understand at first, so let's take a quick tour. 
The LUCI service watches the git repository, telling machines in Swarming to start building and testing new revisions, and serves the "console" page that shows all the results.  Every time a new revision is discovered, the scheduler triggers builders as each serve a different purpose. "Testing" builders are triggered when a "building" builder doing compilation archives its build. For example, all "XP Tests (dbg)(N)" and "Vista Tests (dbg)(N)" are triggered when "Chromium Builder (dbg)" is done archiving its build.




TODO: Everything below here is outdated.

"Memory" builders run tests using ASAN to check memory correctness, "Perf" builders are running performance tests, and "GPU" builders run tests related to GPU usage. "Tree closers" automatically close the tree (preventing people from committing new changes) when they fail; "FYI" testers are considered less important, so failures there shouldn't close the tree.

Click on one of the platform names next to a row of colored boxes in the big grey box at the top, and then on "waterfall" at the left, to see the testing bots working on that platform, for example Windows. The set of builders changes pretty often, so any list we could put here would go out of date quickly, but the names are generally pretty good at describing what kinds of machines they are and what kinds of tests they're running. A "dbg" builder is running a Debug build rather than a Release build. Some machines both build and test, but "builder" machines only build the executables, then upload them to "tests" machines to run the tests.

Build Steps 

Now look down one of the columns, starting at the grey box with the builder's name. Each box shows one step in that builder's build/test sequence, with the oldest one at the bottom and the current or latest one just below the name box. Just above the name box is the builder's current activity, and above that is the final result of the last build/test sequence that finished. 


update scripts gclient sync Update the internal build scripts on the builder. N/A svn server failure; contact trooper
compile Build the executables N/A A bad change; revert
extract build Extract an archive build on a tester from the corresponding "builder" Failed to fetch the url requested. The last archived build is used instead. Failed to fetch any build; the bot probably needs to be restarted, contact a trooper



Builders vs Testers



Each time someone commits a change to the repository, the system notifies builders to start their build and test sequences. (If they're already busy, the change is queued, which means that more than one change is included in a single run if they're coming in faster than the builders can test.) The "changes" column at the left of the waterfall shows who committed a patch and when. If you hover over that link, you'll see a summary of the change; click on it to see a little more information. The times at the very left are in Pacific time. 


Tree state


Seeing More and Seeing Less


Banner and Box


Below that is an oval box. It has a number of handy links at the left -- try them to see where they go. It also has four rows of colored boxes, which show the pass/fail status of the last completed runs for several categories of builders. If you click on one of those category names, you'll go to a partial waterfall view that shows only the related builders. Hovering over a colored box shows you the name of the builder it's summarizing, and clicking on the box will go to a waterfall view with only that one builder. 




Most of the source for Chromium's builder setup is found in the chromium/tools/build Git repo. 


Since every additional configuration that closes the tree has a cost for all the developers to maintain it, there are some guidelines for adding new bots to the main waterfall and commit queue:
  • tests should use the existing test harnesses
  • cycle time needs to be under 40 minutes for trybots
In addition to the above requirements, configurations need to catch enough failures to be worth adding to the commit queue, because running builds on every CL is expensive. If a configuration only fails once every couple of weeks on the waterfalls, it's probably not worth it to add it to the commit queue.

If a configuration has tests that depend on code in other repositories, then it could have its own LUCI project. The team responsible for a separate configuration would sheriff that separate tree. Teams would generally find it more maintainable to have a configuration on the main Chromium tree that observes the above guidelines, even if it only contains a subset of the full code and tests. Whenever a regression is found that only reproduces on the separate team's LUCI project, the separate team's goal should be to write a regression test that runs on the main Chromium waterfall to catch those failures in Chromium, too.

Please email to sign off on new build configurations.


Builder setup


Other views



  • bot: an actual machine (often a VM) connected to a builder. In the case of the try server, multiple bots are connected to one builder. In general there is a 1:1 mapping between a bot and a builder so they can usually be used interchangeably.
  • "a tester": a builder that only runs tests, it gets its binaries from the extract step.
  • "a builder": a builder has a double-meaning, it can be any builder but also only builder that compiles code but does not run tests.