Engineering Productivity Update, August 26, 2015

It’s PTO season and many people have taken a few days or week off.  While they’re away, the team continues making progress on a variety of fronts.  Planning also continues for GoFaster and addon-signing, which will both likely be significant projects for the team in Q4.

Highlights

Treeherder: camd rolled out a change which collapses chunked jobs on Treeherder, reducing visual noise.  In the future, we plan on significantly increasing the number of chunks of many jobs in order to reduce runtimes, so this change makes that work more practical.  See camd’s blog post.  emorley has landed a change which allows TaskCluster job errors that occur outside of mozharness to be properly handled by Treeherder.

Automatic Starring: jgraham has developed a basic backend which supports recognizing simple intermittent failures, and is working on integrating that into Treeherder; mdoglio is landing some related database changes. ekyle has received sheriff training from RyanVM, and plans to use this to help improve the automated failure recognition algorithm.

Perfherder and Performance Testing: Datazilla has finally been decommissioned (R.I.P.), in favor of our newest performance analysis tool, Perfherder.  A lot of Talos documentation updates have been made at https://wiki.mozilla.org/Buildbot/Talos, including details about how we perform calculations on data produced by Talos.  wlach performed a useful post-mortem of Eideticker, with several takeaways which should be applicable to many other projects.

MozReview and Autoland: There’s a MozReview meetup underway, so expect some cool updates next time!

TaskCluster Support: ted has made a successful cross-compiled OSX build using TaskCluster!  Take it for a spin.  More work is needed before we can move OSX builds from the mac mini builders to the cloud.

Mobile Automation: gbrown continues to make improvements on the new |mach emulator| command which makes running Android tests locally on emulator very simple.

General Automation: run-by-dir is live on opt mochitest-plain; debug and ASAN coming soon.  This reduces test “bleed-through” and makes it easier to change chunking.  adusca, our Outreachy intern, is working to integrate the try extender into Treeherder.  And ahal has merged the mozharness “in-tree” configs with the regular mozharness config files, now that mozharness lives in the tree.

Firefox Automation: YouTube ad detection has been improved for firefox-media-tests by maja, which fixes the source of the top intermittent failure in this suite.

Bughunter: bc has got asan-opt builds running in production, and is working on gtk3 support.

hg.mozilla.org: gps has enabled syntax highlighting in hgweb, and has added a new JSON API as well.  See gps’ blog post.

The Details

bugzilla.mozilla.org

• start of changes to modernise bugzilla’s security practices (session visibility, ability to force a password reset)
• as always, https://wiki.mozilla.org/BMO/Recent_Changes has the full list

Treeherder

• Camd’s job counts landed and have deployed to production https://bugzil.la/1163064
• [emorley] log parser taskcluster support has landed https://bugzil.la/1188444

Perfherder/Performance Testing

• talos cleanup and preparation to move in-tree
• perfherder database cleanup in progress for simpler and more optimized queries. This is mainly preparatory work for making perfherder capable of managing/starring performance alerts, but as a bonus perfherder compare view should load virtually instantly once this is finished. 
• most talos wiki docs are updated: https://wiki.mozilla.org/Buildbot/Talos
  • in particular details on how we perform calculations on data produced by talos tests are here: https://wiki.mozilla.org/Buildbot/Talos/Data , indexed via the existing documentation on talos tests here https://wiki.mozilla.org/Buildbot/Talos/Test

TaskCluster Support

• Successful mac cross build + package on Try! https://treeherder.mozilla.org/#/jobs?repo=try&revision=6177892c36cd https://queue.taskcluster.net/v1/task/z9RBJJS4RSKcRHz1HANfyw/runs/0/artifacts/public/build/target.mac64.dmg (build shows as failing, TaskCluster build script is failing to handle split test zips)

Mobile Automation

•  [gbrown] Working on “mach emulator” support: wip can download and run 2.3, 4.3, or x86 emulator images. Integrating with other mach commands like “install” and “mochitest”.
•  [gbrown] Updated mochitest manifests to run most dom/media mochitests on Android 4.3 (under review, bug 1189784)

Firefox and Media Automation

• [maja_zf] Improved ad detection on YouTube for firefox-media-tests, which fixes our top intermittent failure for long-running playback tests.

General Automation

•  run-by-dir is live for mochitest-plain (opt only); debug is coming soon, followed by ASAN.
• Mozilla CI tools is moving from using BuildAPI as the scheduling entry point to use TaskCluster’ scheduling. This work will allow us to schedule a graph of buildbot jobs and their dependencies in one shot. https://bugzil.la/1194264
• adusca is integrating into treeherder the ability to extend the jobs run for any push. This is based on the http://try-extender.herokuapp.com prototype. Follow along in https://bugzil.la/1194830
• Git was deployed to the test machines. This is necessary to make the Firefox UI update tests work on them.
• [ahal] merge mozharness in-tree configs with the main mozharness configs

ActiveData

• Bug fixes to the ETL – fix bad lookups on hg repo, mostly l10n builds 
• More error reporting on ETL – Structured logging has changed a little, handle the new variations, and be more elegant when it comes to unknowns, an complain when there is non-conformance.
• Some work on adding hg `repo` table – acts as a cache for ETL, but can be used to calculate ‘per push’ statistics on OrangeFactor data.
• Added Talos to the `perf` table – used the old Datazilla ETL code to fill the ES cluster.  This may speed up extracting the replicates, for exploring the behaviour of a test.
• Enable deep queries – Effectively performing SQL join on Elasticsearch – first attempt did too much refactoring.  Second attempt is simpler, but still slogging through all the resulting test breakage

hg.mozilla.org

• json style deployed. Read http://gregoryszorc.com/blog/2015/08/18/json-apis-on-hg.mozilla.org/
• All server processes are now running Python 2.7 and in virtualenvs (before we were using system Python 2.6)
• highlight extension enabled (syntax highlighting in file and annotate views)

WebDriver

• Updated 

Marionette

• [ahal] helped review and finish contributor patch for switching marionette_client from optparse to argparse
• Corrected UUID used for session ID and element IDs
• Updated dispatching of various marionette calls in Gecko

bughunter

• [bc] Have asan-opt builds running in production. Finalizing patch. Still need to build gtk3 for rhel6 32bit in order to stop using custom builds and support opt in addition to debug.

charts.mozilla.org

• Updated the hierarchical burndowns to EPM’s important metabugs that track features 
• More config changes