It’s Q4, and at Mozilla that means it’s planning season. There’s a lot of work happening to define a Vision, Strategy and Roadmap for all of the projects that Engineering Productivity is working on; I’ll share progress on that over the next couple of updates.
Higlights
Build System: Work is starting on a comprehensive revamp of the build system, which should make it modern, fast, and flexible. A few bits of this are underway (like migration of remaining Makefiles to moz.build); more substantial progress is being planned for Q1 and the rest of 2016.
Bugzilla: Duo 2FA support is coming soon! The necessary Bugzilla changes has landed, we’re just waiting for some licensing details to be sorted out.
Treeherder: Improvements have been made to the way that sheriffs can backfill jobs in order to bisect a regression. Meanwhile, lots of work continues on backend and frontend support for automatic starring.
Perfherder and Performance Testing: Some optimizations were made to Perfherder which has made it more performant – no one wants a slow performance monitoring dashboard! jmaher and bc are getting close to being able to run Talos on real devices via Autophone; some experimental runs are already showing up on Treeherder.
MozReview and Autoland: It’s no longer necessary to have an LDAP account in order to push commits to MozReview; all that’s needed is a Bugzilla account. This opens the door to contributors using the system. Testing of Autoland is underway on MozReview’s dev instance – expect it to be available in production soon.
TaskCluster Migration: OSX cross-compiled builds are now running in TaskCluster and appearing in Treeherder as Tier-2 jobs, for debug and static checking. The TC static checking build with likely become the official build soon (and the buildbot build retired); the debug build won’t become official until work is done to enable existing test jobs to consume the TC build.
Work is progressing on enabling TaskCluster test jobs for linux64-debug; our goal is to have these all running side-by-side the buildbot jobs this quarter, so we can compare failure rates before turning off the corresponding buildbot jobs in Q1. Moving these jobs to TaskCluster enables us to chunk them to a much greater degree, which will offer some additional flexibility in automation and improve end-to-end times for these tests significantly.
Mobile Automation: All Android test suites that show in Treeherder can now be run easily using mach.
Dev Workflow: It’s now easier to create new web-platform-tests, thanks to a new |mach web-platform-tests-create| command.
e10s Support: web-platform-tests are now running in e10s mode on linux and OSX platforms. We want to turn these and other tests in e10s mode on for Windows, but have hardware capacity problems. Discussions are underway on how to resolve this in the short-term; longer-term plans include an increase in hardware capacity.
Test Harnesses: run-by-dir is now applied to all mochitest jobs on desktop. This improves test isolation and paves the way for chunking changes which we will use to improve end-to-end times and make bisection turnaround faster. Structured logging has been rolled out to Android reftests; Firefox OS reftests still to come.
ActiveData: Work is in progress to build out a model of our test jobs running in CI, so that we can identify pieces of job setup and teardown which are too slow and targets of possible optimization, and so that we can begin to predict the effects of changes to jobs and hardware capacities.
hg.mozilla.org: Mercurial 3.6 will have built-in support for seeding clones from pre-generated bundle files, and will have improved performance for cloning, especially on Windows.
Marionette and WebDriver: Message sequencing is being added to Marionette; this will help prevent synchronization issues where the client mixes up responses. Client-side work is being done in both Python and node.js. ato wrote an article making a case against visibility checks in WebDriver.
Details
bugzilla.mozilla.org
- bug 1199089 – support for Duo 2FA has landed. It isn’t available just yet as we’re waiting on the licensing situation to be sorted
- lots of tweaks to the experimental UI
Treeherder
- Fix backfilling and fix missing jobs on a revision
Perfherder/Performance Testing
- Getting closer to talos on autophone. Test porting is done and core tooling is in place, first sanity check ran this weekend: https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&filter-searchStr=autophone&fromchange=8b40fc14199c&tochange=ce75fb8d5a8b
- Perfherder loads faster due to optimized storage and apis
- compare view loads all data at once (e10s and counters)
MozReview/Autoland
- An LDAP / SSH account is no longer required to create new review requests (http://gregoryszorc.com/blog/2015/10/14/lowering-the-barrier-to-pushing-to-mozreview/)
- Fixed Unicode in commit messages
- Autoland code has been reviewed and committed; testing is underway on our dev instance.
TaskCluster Support
- Landed all patches for cross-mac builds, running fine on inbound/central!
- [ahal] Got some linux64 tests running (various flavours of mochitest, reftest and xpcshell), though not yet green.
Mobile Automation
- [gbrown] mach reftest|crashtest|jstestbrowser now supports Firefox for Android (all Android test suites run on treeherder can now be run from mach)
Dev Workflow
- [jgraham] Added a |mach web-platform-tests-create| target to help with the workflow of creating new web-platform-tests.
Firefox and Media Automation
- Netflix bandwidth limiting tests blocked because of a problem on Netflix side.
- Web platform media-source directory no longer being run on our Jenkins since all platforms of web platform tests now run as part of release.
- We’ve established a roadmap that coordinates moving ui-tests and media-tests in-tree, updating the Marionette test runner and moving media jobs into mozmill-ci
General Automation
- [jgraham] web-platform-tests-e10s now running across all trees on Mac/Linux (Windows has capacity problems)
- SETA updated to support new android debug tests
- run-by-dir is enabled for all desktop mochitests.
- [ahal] reftest structured logging working on desktop and android (b2g still left to do)
ActiveData
- Continuous importing of all known test text logs. These detail the various builder steps, and mozharness steps and their timings, and can be found in the `jobs.action.timings` table. There is still more to do to clean up the data.
- The above is incomplete though. Also working on importing the buildbot json logs, for a complete picture, and to fill in the ?few? missing logs.
hg.mozilla.org
- Mercurial 3.6 will have built-in support for seeding clones from pre-generated, externally-hosted bundle files (i.e. the bundleclone extension)
- Mercurial 3.6 features significant performance improvements for cloning, especially on Windows.
WebDriver
- ato wrote an article making a case against visibility checks in WebDriver
Marionette
- [ato] Support for screen capture of SVG documents, and generally more resilience when things go wrong with captures
- [ato] Ongoing work to provide message sequencing. This will help prevent synchronicity issues where the client mixes up responses. This entails work in both the Python and Node.js clients.