JGriffin's Blog

Is this thing on?

Monthly Archives: September 2015

Engineering Productivity Update, Sept 10, 2015


Bugzilla: The BMO has been busy implementing security enhancements, and as a result, BMO now supports two-factor authentication.  Setting this up is easy through BMO’s Preferences page.

Treeherder: The size of the Treeherder database dropped from ~700G to around ~80G thanks to a bunch of improvements in the way we store data.  Jonathan French is working on improvements to the Sheriff Panel.  And Treeherder is now ingesting data that will be used to support Automatic Starring, a feature we expect to be live in Q4.

Perfherder and Performance: Will Lachance has published a roadmap for Perfherder, and has landed some changes that should improve Perfherder’s performance.  Talos tests on OSX 10.10 have been hidden in Perfherder because the numbers are very noisy; the reason for this is not currently known.  Meanwhile, Talos has finally landed in mozilla-central, which should make it easier to iterate on.  Thanks to our contributor Julien Pagès for making this happen!  Joel Maher has posted a Talos update on dev.platform with many more details.

MozReview and Autoland: The web UI now uses BMO API keys; this should make logins smoother and eliminate random logouts.  Several UI improvements have been made; see full list in the “Details” section below.

Mobile Automation: Geoff Brown has landed the much-requested |mach android-emulator| command, which makes it much easier to run tests locally with an Android emulator.  Meanwhile, we’re getting closer to moving the last Talos Android tests (currently running on panda boards) to Autophone.

Developer Workflow: Our summer intern, Vaibhav Agrawal, landed support for an initial version of |mach find-test-chunk|, which can tell you which chunk a test gets run in.  This initial version supports desktop mochitest only.  Vaibhav gave an intern presentation this week, “Increasing Productivity with Automation”.  Check it out!

General Automation: James Graham has enabled web-platform-tests-e10s on try, but they’re hidden pending investigation of tests which are problematic with e10s enabled.  Joel Maher and Kim Moir in Release Engineering have tweaked our SETA coalescing, so that lower prioritized jobs are run at least every 7th push, or every hour; further increasing the coalescing window will wait until we have better automatic backfililng in place.  Meanwhile, the number of chunks of mochitest-browser-chrome has been increased from 3 to 7, with mochitest-devtools-chrome soon to follow.  This will make backfilling faster, as well as improving turnaround times on our infrastructure.

hg.mozilla.org: The bzexport and bzpost extensions have been updated to support BMO API keys.

WebDriver and Marionette: Several changes were made to the WebDriver specification, including new chapters on screen capture and user prompts and modal handling.

Bughunter: Our platform coverage now includes opt and debug builds of linux32, linux64, opt-asan-linux64, OSX 10.6, 10.8, 10.9, and windows7 32- and 64-bit.

The Details

  • A number of optimizations have reduced Treeherder’s db size from ~700GB to ~80GB!
  • For those with Sheriff access, an improved layout for the Sheriff panel will appear soon on production https://tojonmz.wordpress.com/2015/09/04/layout-improvements-to-the-sheriff-panel/ with similar work planned soon for the Filter panel
  • Among other fixes and improvements, Logviewer UI more gracefully handles additional incomplete log states: unknown log steps (1193222) and expired jobs (1193222)
  • A new Help menu has been added with useful links for all users (1199078)
  • We are now storing the data required for the autostarring project. That means storing every single crash/test failure/log error line from the structured log (1182464).
Perfherder/Performance Testing
  • Autoland VCS interaction performance improvements
  • MozReview web UI now using BMO API keys
  • Login in smoother and faster and random logouts should be a thing of the past
  • MozReview Mercurial client extension now requires BMO API keys
  • No more defining password and/or cookie in plaintext config files
  • “Ship It” is now determined by reviewer status so random people can’t grant Ship It
  • Messaging during pushing is more clear about what is happening
  • “commitid” hidden metadata is now preserved across `hg rebase`
  • Review buttons and text links in web interface are now consolidated to reduce confusion
  • Empty Try syntax is now rejected properly
  • Try trigger button has been moved to an “Automation” drop-down
  • pull and import commands are now displayed
  • Bugzilla’s commit list is now properly ordered
TaskCluster Support
  • armenzg – Work is underway to support running Buildbot test jobs through TaskCluster
  • ted – successful Mac cross build with Breakpad last week, landing patches and fixing some fallout from Linux TC build switch to CentOS 6
Mobile Automation
Dev Workflow
  • vaibhav1994 – A basic version of find-test-chunk has landed. This will help in determining on which chunk a particular test is present in production. It works for mochitest for desktop platforms, see various options with ‘./mach find-test-chunk’
  • vaibhav1994 – –rebuild-talos option now present in trigger-bot to trigger only talos jobs a certain number of times on try.
Firefox and Media Automation
  • sydpolk – Network bandwidth limiting tests have been written; working to deploy them to Jenkins.
  • sydpolk – Streamlined Jenkins project generation based on Jenkins python API (found out about this at the Jenkins conference last week)
  • sydpolk – Migration of hardware out of MTV2 QA lab won’t happen this quarter because Network Ops people are shutting down the Phoenix data center.
  • maja_zf – mozharness script for firefox-media-tests has been refactored into scripts for running the tests in buildbot and our Jenkins instance
General Automation
  • chmanchester – psutil 3.1.1 is now installed on all test slaves as a part of running desktop unit tests. This will help our test harnesses manages subprocesses of the browser, and particularly kill them to get stacks after a hang.
  • armenzg – Firefox UI tests can now be called through a python wrapper instead of only through a python binary. This is very important since it was causing Windows UAC prompts on Release Engineering’s Windows test machines. The tests now execute well on all test platforms.
  • jgraham – web-platform-tests-e10s now running on try, but still hidden pending some investigation of tests that are only unstable with e10s active
  • SETA work is ongoing to support new platforms, tests, and jobs.  
  • [ekyle] Queries into nested documents pass tests, but do not scale on the large cluster; startup time is unacceptable.  Moving work to separate thread for quick startup, with the hope a complicated query will not arrive until the metadata is finished collecting
  • [ekyle] Added auto-restart on ETL machines that simply stop working (using CloudWatch); probably caused by unexpected data, which must be looked into later.
  • [ekyle] SpotManager config change for r3.* instances
  • Add times for replication events on push
  • Reformat pushlog messages on push to be less eye bleedy
  • bzexport and bzpost extensions now support Bugzilla API Keys
  • [ato] Specified element interaction commands
  • [ato] New chapter on user prompts and modal handling
  • [ato] New chapter on screen capture
  • [ato] Cookies retrieval bug fixes
  • [ato] Review of normative dependencies
  • [ato] Security chapter review
  • Wires 0.4 has been released.
  • [ato] Deployed Marionette protocol changes, bringing Marionette closer to the WebDriver standard
  • [ato] Assorted content space commands converted to use new dispatching technique
  • [jgraham] Updated wires for protocol changes
Now running opt, debug tinderbox builds for Linux 32 bit, 64 bit; OSX 10.6, 10.8, 10.9; Windows 7 32 bit, 64 bit; opt asan tinderbox builds for Linux 64 bit.
  • bug 1180749 Sisyphus – Django 1.8.2 support
  • bug 1185497 Sisyphus – Bughunter – use ASAN builds for Linux 64 bit workers
  • bug 1192646 Sisyphus – Bughunter – use crashloader.py to upload urls to be resubmitted
  • bug 1194290 Sisyphus – Bughunter – install gtk3 on rhel6 vms