JGriffin's Blog
Is this thing on?
Monthly Archives: August 2015
Engineering Productivity Update, August 26, 2015
August 26, 2015
Posted by on It’s PTO season and many people have taken a few days or week off. While they’re away, the team continues making progress on a variety of fronts. Planning also continues for GoFaster and addon-signing, which will both likely be significant projects for the team in Q4.
Highlights
Treeherder: camd rolled out a change which collapses chunked jobs on Treeherder, reducing visual noise. In the future, we plan on significantly increasing the number of chunks of many jobs in order to reduce runtimes, so this change makes that work more practical. See camd’s blog post. emorley has landed a change which allows TaskCluster job errors that occur outside of mozharness to be properly handled by Treeherder.
Automatic Starring: jgraham has developed a basic backend which supports recognizing simple intermittent failures, and is working on integrating that into Treeherder; mdoglio is landing some related database changes. ekyle has received sheriff training from RyanVM, and plans to use this to help improve the automated failure recognition algorithm.
Perfherder and Performance Testing: Datazilla has finally been decommissioned (R.I.P.), in favor of our newest performance analysis tool, Perfherder. A lot of Talos documentation updates have been made at https://wiki.mozilla.org/Buildbot/Talos, including details about how we perform calculations on data produced by Talos. wlach performed a useful post-mortem of Eideticker, with several takeaways which should be applicable to many other projects.
MozReview and Autoland: There’s a MozReview meetup underway, so expect some cool updates next time!
TaskCluster Support: ted has made a successful cross-compiled OSX build using TaskCluster! Take it for a spin. More work is needed before we can move OSX builds from the mac mini builders to the cloud.
Mobile Automation: gbrown continues to make improvements on the new |mach emulator| command which makes running Android tests locally on emulator very simple.
General Automation: run-by-dir is live on opt mochitest-plain; debug and ASAN coming soon. This reduces test “bleed-through” and makes it easier to change chunking. adusca, our Outreachy intern, is working to integrate the try extender into Treeherder. And ahal has merged the mozharness “in-tree” configs with the regular mozharness config files, now that mozharness lives in the tree.
Firefox Automation: YouTube ad detection has been improved for firefox-media-tests by maja, which fixes the source of the top intermittent failure in this suite.
Bughunter: bc has got asan-opt builds running in production, and is working on gtk3 support.
hg.mozilla.org: gps has enabled syntax highlighting in hgweb, and has added a new JSON API as well. See gps’ blog post.
The Details
bugzilla.mozilla.org
Treeherder
Perfherder/Performance Testing
TaskCluster Support
Mobile Automation
Firefox and Media Automation
General Automation
ActiveData
hg.mozilla.org
WebDriver
Marionette
Engineering Productivity Update, August 13, 2015
August 13, 2015
Posted by on From Automation and Tools to Engineering Productivity
“Automation and Tools” has been our name for a long time, but it is a catch-all name which can mean anything, everything, or nothing, depending on the context. Furthermore, it’s often unclear to others which “Automation” we should own or help with.
For these reasons, we are adopting the name “Engineering Productivity”. This name embodies the diverse range of work we do, reinforces our mission (https://wiki.mozilla.org/Auto-tools#Our_Mission), promotes immediate recognition of the value we provide to the organization, and encourages a re-commitment to the reason this team was originally created—to help developers move faster and be more effective through automation.
The “A-Team” nickname will very much still live on, even though our official name no longer begins with an “A”; the “get it done” spirit associated with that nickname remains a core part of our identity and culture, so you’ll still find us in #ateam, brainstorming and implementing ways to make the lives of Mozilla’s developers better.
Highlights
Treeherder: Most of the backend work to support automatic starring of intermittent failures has been done. On the front end, several features were added to make it easier for sheriffs and others to retrigger jobs to assist with bisection: the ability to fill in all missing jobs for a particular push, the ability to trigger Talos jobs N times, the ability to backfill all the coalesced jobs of a specific type, and the ability to retrigger all pinned jobs. These changes should make bug hunting much easier. Several improvements were made to the Logviewer as well, which should increase its usefulness.
Perfherder and performance testing: Lots of Perfherder improvements have landed in the last couple of weeks. See details at wlach’s blog post. Meanwhile, lots of Talos cleanup is underway in preparation for moving it into the tree.
MozReview: Some upcoming auth changes are explained in mcote’s blog post.
Mobile automation: gbrown has converted a set of robocop tests to the newly enabled mochitest-chrome on Android. This is a much more efficient harness and converting just 20 tests has resulted in a reduction of 30 minutes of machine time per push.
Developer workflow: chmanchester is working on building annotations into moz.build files that will automatically select or prioritize tests based on files changed in a commit. See his blog post for more details. Meanwhile, armenzg and adusca have implemented an initial version of a Try Extender app, which allows people to add more jobs on an existing try push. Additional improvements for this are planned.
Firefox automation: whimboo has written a Q2 Firefox Automation Report detailing recent work on Firefox Update and UI tests. Maja has improved the integration of Firefox media tests with Treeherder so that they now officially support all the Tier 2 job requirements.
WebDriver and Marionette: WebDriver is now officially a living standard. Congratulations to David Burns, Andreas Tolfsen, and James Graham who have contributed to this standard. dburns has created some documentation which describes which WebDriver endpoints are implemented in Marionette.
Version control: The ability to read and extra metadata from moz.build files has been added to hg.mozilla.org. This opens the door to cool future features, like the ability auto file bugs in the proper component and automatically selecting appropriate reviewers when pushing to MozReview. gps has also blogged about some operational changes to hg.mozilla.org which enables easier end-to-end testing of new features, among other things.
The Details
bugzilla.mozilla.org
Treeherder/Automatic Starring
Treeherder/Front End
Perfherder/Performance Testing
MozReview/Autoland
Mobile Automation
Dev Workflow
Media Automation
General Automation
ActiveData