JGriffin's Blog
Is this thing on?
Tag Archives: mozilla
Engineering Productivity Update, Sept 10, 2015
Posted by on September 10, 2015
Highlights
Bugzilla: The BMO has been busy implementing security enhancements, and as a result, BMO now supports two-factor authentication. Setting this up is easy through BMO’s Preferences page.
Treeherder: The size of the Treeherder database dropped from ~700G to around ~80G thanks to a bunch of improvements in the way we store data. Jonathan French is working on improvements to the Sheriff Panel. And Treeherder is now ingesting data that will be used to support Automatic Starring, a feature we expect to be live in Q4.
Perfherder and Performance: Will Lachance has published a roadmap for Perfherder, and has landed some changes that should improve Perfherder’s performance. Talos tests on OSX 10.10 have been hidden in Perfherder because the numbers are very noisy; the reason for this is not currently known. Meanwhile, Talos has finally landed in mozilla-central, which should make it easier to iterate on. Thanks to our contributor Julien Pagès for making this happen! Joel Maher has posted a Talos update on dev.platform with many more details.
MozReview and Autoland: The web UI now uses BMO API keys; this should make logins smoother and eliminate random logouts. Several UI improvements have been made; see full list in the “Details” section below.
Mobile Automation: Geoff Brown has landed the much-requested |mach android-emulator| command, which makes it much easier to run tests locally with an Android emulator. Meanwhile, we’re getting closer to moving the last Talos Android tests (currently running on panda boards) to Autophone.
Developer Workflow: Our summer intern, Vaibhav Agrawal, landed support for an initial version of |mach find-test-chunk|, which can tell you which chunk a test gets run in. This initial version supports desktop mochitest only. Vaibhav gave an intern presentation this week, “Increasing Productivity with Automation”. Check it out!
General Automation: James Graham has enabled web-platform-tests-e10s on try, but they’re hidden pending investigation of tests which are problematic with e10s enabled. Joel Maher and Kim Moir in Release Engineering have tweaked our SETA coalescing, so that lower prioritized jobs are run at least every 7th push, or every hour; further increasing the coalescing window will wait until we have better automatic backfililng in place. Meanwhile, the number of chunks of mochitest-browser-chrome has been increased from 3 to 7, with mochitest-devtools-chrome soon to follow. This will make backfilling faster, as well as improving turnaround times on our infrastructure.
hg.mozilla.org: The bzexport and bzpost extensions have been updated to support BMO API keys.
WebDriver and Marionette: Several changes were made to the WebDriver specification, including new chapters on screen capture and user prompts and modal handling.
Bughunter: Our platform coverage now includes opt and debug builds of linux32, linux64, opt-asan-linux64, OSX 10.6, 10.8, 10.9, and windows7 32- and 64-bit.
The Details
bugzilla.mozilla.org
Treeherder
Perfherder/Performance Testing
MozReview/Autoland
TaskCluster Support
Mobile Automation
Dev Workflow
Firefox and Media Automation
General Automation
ActiveData
hg.mozilla.org
WebDriver
Marionette
bughunter
Automation and Tools Team Update, July 16 2015
Posted by on July 16, 2015
The Automation and Tools Team (the A-Team, for short) is a large team that oversees a diverse set of services, tools and test harnesses used by nearly everyone at Mozilla. We’re borrowing a page from Release Engineering and publishing a series of updates to inform people about what we’re up to, in the hopes of fostering better visibility and inter-team coordination.
Highlights
Treeherder and Automatic Starring: Our focus for Treeherder in Q3 will be improving the signal-to-noise ratio for dealing with intermittent oranges. An overall design has been agreed to for the “automatic starring” project, and work has begun; final rollout is likely in Q4. This quarter, we’ll also stop spamming Bugzilla with comments for each intermittent, but we will put in place an alternate notification system for people who rely on Bugzilla orange comments to determine when an intermittent needs attention. We’ve also agreed on a redesign for the Logviewer that should result in a more useful and intuitive interface.
MozReview and Autoland: MozReview now offers to publish review requests when you push, so it isn’t necessary to visit the MozReview’s UI. Work has started on adding support for autoland-to-inbound, which will allow developers to push changes to inbound directly from MozReview… no more battling tree closures!
Performance: Work continues on Perfherder’s “comparison mode”, a view that compares Talos performance data between two revisions. See wlach’s blog post for more details.
TaskCluster Support: We’re helping Release Engineering migrate from Buildbot to TaskCluster; this quarter we’re standing up Linux tests in TaskCluster and getting OS X cross-compilation to work so that we can move those builds to the cloud.
BMO now has tests running in continuous integration using TaskCluster and reporting to Treeherder.
Mobile Automation: mochitest-chrome for Android is now live! Work is also underway to enable debug reftests on Android emulators, and significant reliability improvements have been landed in Autophone.
Desktop Automation: Work is in progress to get Thread Sanitizer (TSan) builds running on try and to split gTest into its own test chunk. We’re also working towards applying –run-by-dir to mochitest-plain, in order to improve isolation and enable smarter chunking in CI.
Developer Workflow: We’re adding test-selection flexibility to the reftest harness as a prelude to making ‘mach try’ work with more test types.
The Details
Bugzilla.mozilla.org
- BMO now has CI, running on Treeherder: https://treeherder.mozilla.org/#/jobs?repo=bmo-master
- see https://wiki.mozilla.org/BMO/Recent_Changes for a full list
Treeherder/Automatic Starring
- Work has started on backend work needed to support automatic starring, including db simplification, and db unification (so each tree doesn’t have its own database). Bug 1179263 tracks this work. As a side effect of this work, Treeherder code should become less complex and easier to maintain.
- Work has started on identifying what needs to happen in order to turn off Bugzilla comments for intermittents, and to create an alternative notification mechanism instead. Bug 1179310 tracks this work.
Treeherder/Front End
- New shortcuts for Logviewer, Delete Classification plus improved classification save
- Agreed on new design for Logviewer, bug 1182178
- Design work is in progress for collapsing chunks in Treeherder in order to reduce visual noise in bug 1163064
Perfherder/Performance Testing
- Evaluating alerts generated from PerfHerder
- Improvements to compare chooser and viewer inside of PerfHerder
- Work towards building a new tab switching test (bug 1166132)
- wlach did a blog post summarizing recent notable developments in perfherder: http://wrla.ch/blog/2015/07/perfherder-update/
MozReview/Autoland
- Automatic publishing of reviews upon pushing
- Known bug: people using cookie auth may experience bug 1181886
- Better error message when MozReview’s Bugzilla session has expired (bug 1178811)
- Pruned user database to improve user searching (bug 1171274)
- Fix automatic reviewer selection (bug 1177454)
- Work is progressing on autoland-to-inbound (bug 1128039)
TaskCluster Support
- Ability to schedule Linux64 tests on try (tests not running yet due to a couple blockers) – bug 1171033
- Working on OSX cross-compilation, which will allow us to move OSX builds to the cloud; this will make OSX builds much faster in CI.
Mobile Automation
- Autophone detects USB lock-ups and gracefully restarts. This is a huge improvement in system reliability.
- Continued work on getting Android Talos tests ported to Autophone (bug 1170685)
- Updated manifests and mozharness configs for mochitest-chrome (bug 1026290)
- Determined total-chunks requirements for Android 4.3 Debug reftests (bug 1140471)
- Re-wrote robocop harness to significantly improve run-time efficiency (bug 1179981)
Dev Workflow
- Helped RelEng resolve some problems that were preventing them from landing mozharness in the tree. This opens the door to a lot of future dev workflow improvements, including better unification of the ways we run automated tests in continuous integration and locally. We’ve wanted this for years and it’s great to see it finally happen.
- Did some work on top of jgraham’s patch to make mach use mozlog structured logging
Platform QA
- We had to respond to the breakup of .tests.zip into several files to keep our Jenkins instance running.
- Getting firefox-media-tests to satisfy Tier-2 Treeherder visibility requirements involves changing how Treeherder accommodates non-buildbot jobs (e.g bug 1182299)
General Automation
- Working on running multiple tests/manifests through reftests harness as a prelude for supporting |mach try| for more test types.
- Created patch to move mozlog.structured to the top level package (and what was previously there to mozlog.unstructured)
- Figured out the series of steps needed to produce a usable Thread Sanitizer enabled linux build on our infra
- Separating out gTest into a separate job in CI – bug 1179955.
- Support has been added to mozregression for downloading inbound builds from S3 – bug 1177923. More work is needed on both mozregression and mozdownload to fully adapt them to the migration of builds off the netapp.
- Work is underway to allow the Python mozillapulse client to consume from multiple exchanges – bug 1180897.
- mozregression 0.37 has been released.
- mozci 0.8.2 now allows you to use Treeherder as a source of job data.
ActiveData
- More memory optimizations (motivation: releng query for Chris Atlee: query slow tests)
- run staging environment as stability test for production
- change etl procedure so pushing changes to prod are easier (moving toward standard procedure)
- import treeherder data markup to active data (motivation: characterizing test failures
- ateam query: summary of test failures, stars and resolutions (bug 1161268, bug 1172048)
- subtests are too large for download of more than one day – working on code to only pull what’s required
Writing WebAPI tests for B2G using Marionette
Posted by on May 4, 2012
At Mozilla, we have many different testing frameworks, each of which fills a different niche (although there is definitely some degree of overlap among them). For testing WebAPIs in B2G, some of these existing frameworks can be utilized, depending on the API. For example, mozSettings and mozContacts can be tested using mochitests, since there isn’t much, if anything, that’s device-specific to them. (We’re not currently running mochitests on B2G devices, but will be soon.)
But there are many other WebAPIs which are not testable using any of our standard frameworks, because tests for them need to interact with hardware in interesting ways, and most of our frameworks are designed to operate entirely within a gecko context, and thus have no ability to directly access hardware.
Malini Das and I have been working on a new framework called Marionette which can help. Marionette is a remote test driver, so it can remotely execute test steps within a gecko process while retaining the ability to interact with the outside world, including devices running B2G. When this is combined with the B2G emulator’s ability to query and set hardware state, we have a solution for testing a number of WebAPIs that would be difficult or impossible to test otherwise.
To illustrate how this works, I’m going to walk through the entire process of writing WebAPI tests for mozBattery and mozTelephony, to be run on B2G emulators. We already have such tests running in continuous integration, reporting to autolog. If developers add new Marionette WebAPI tests, they will be run and reported here as well. Eventually, they will likely be migrated over to TBPL.
Building the emulator
These tests will be run on the emulator, so you’ll have to build the B2G Ice Cream Sandwich emulator first, if you don’t have one already. You’ll need to do this on linux, preferably Ubuntu. Make sure to install the build prerequisites before you begin, if you haven’t built B2G before.
git clone https://github.com/andreasgal/B2G cd B2G make sync (get a cup of coffee, this takes quite a while) make config-qemu-ics (get another cup of coffee) make gonk (get another drink, but I think you've had enough coffee by now) make
You should now have an emulator, which can you launch using:
./emu-ics.sh
After you’ve verified the emulator is working, close it again.
Running a Marionette sanity test
Now we’ll run a single Marionette test to verify that everything is working as expected. First, ensure that you have Python 2.7 on your system. Then, install some prerequisites:
pip install (or easy_install) manifestdestiny pip install (or easy_install) mozhttpd pip install (or easy_install) mozprocess
Now, from the directory where you cloned the B2G repo:
cd gecko/testing/marionette/client/marionette python runtests.py --emulator --homedir /path/to/B2G/repo \ tests/unit/test_simpletest_sanity.py
If everything has gone well, you should see something like the following:
TEST-START test_simpletest_sanity.py test_is (test_simpletest_sanity.SimpletestSanityTest) ... ok test_isnot (test_simpletest_sanity.SimpletestSanityTest) ... ok test_ok (test_simpletest_sanity.SimpletestSanityTest) ... ok ---------------------------------------------------------------------- Ran 3 tests in 2.952s OK SUMMARY ------- passed: 3 failed: 0 todo: 0
Writing a battery test
The B2G emulator allows you to arbitrarily set the battery level and charging state, by telnetting into the emulator’s console port and issuing certain commands. Marionette has an EmulatorBattery class which abstracts these operations, and allows you to interact with the emulator’s battery using a very simple API.
A simple example is given in the EmulatorBattery documentation on MDN. Save this example to a file named test_battery_example.py, and run this command:
python runtests.py --emulator --homedir /path/to/B2G/repo /path/to/test_battery_example.py
Marionette should launch an emulator and run the test; when it’s done you should see:
TEST-START test_battery_example.py test_level (test_battery_example.TestBatteryLevel) ... ok ---------------------------------------------------------------------- Ran 1 test in 0.391s OK SUMMARY ------- passed: 1 failed: 0 todo: 0
How it works
This test, like all Marionette Python tests, is written using Python’s unittest framework, which provides the assert methods used in the test. Other methods used by the test are provided by the Marionette and EmulatorBattery classes.
When the test executes this line:
self.marionette.emulator.battery.level = 0.25
the EmulatorBattery class telnets into the emulator and sets the battery’s level. We then read the level back (which invokes another telnet command) to verify that the emulator’s battery state was updated as expected. And finally, we execute a snippet of JavaScript inside gecko:
moz_level = self.marionette.execute_script("return navigator.mozBattery.level;")
and verify that it returns the same battery level as the emulator is reporting directly.
More tests with hardware interaction
In addition to battery interaction, the B2G emulator allows you to query and set the state of other properties normally set by hardware, like GPS location, network status, and various sensors. Tests for all these could be written in a similar way. It probably makes sense to make classes for these similar to EmulatorBattery which abstract the details of getting and setting the state of the underlying hardware. I would encourage WebAPI developers to add as many WebAPI tests as possible; if you would like us to add convenience classes, please ping us on IRC (jgriffin and mdas, on #ateam or #b2g) or file a bug under Testing:Marionette.
Multi-emulator tests
There are some WebAPIs which cannot be completely tested using a single device or emulator, like telephony and SMS. Marionette can help with these too, as Marionette can be used to manipulate two emulator instances which are capable of communicating with each other.
In any tests run with the --emulator switch, Marionette launches an emulator before running the tests, and this emulator is associated with an instance of the Marionette class available to the test as self.marionette. Tests can invoke a second emulator instance using self.get_new_emulator(), and these emulator instances can call and text each other using their port numbers as their phone numbers.
To illustrate how this works, Malini has written an example test in which one emulator is used to dial another, and the caller’s number is verified on the receiver. See this example at https://developer.mozilla.org/en/Marionette/Marionette_Python_Tests/Emulator_Integrated_Tests#Manage_Multiple_Emulators.
If you save this example to test_dial_example.py and run the command:
python runtests.py --emulator --homedir /path/to/B2G/repo /path/to/test_dial_example.py
you should see Marionette launch one emulator, and then after it starts execution of the test, you should see a second emulator instance launch. After the test is done, you should see a successful report, similar to the one shown for the battery test.
We currently have a few tests for mozTelephony, but many more could be added, and new tests should be added for SMS/MMS as well.
Adding new tests to the B2G continuous integration
When new test are ready to be added to the CI, they should be checked into gecko under their dom component, e.g., dom/telephony/test/marionette. They should be added to the manifest.ini file in the same directory, and then for new manifest.ini files, the path to the .ini file should be added to the master manifest at http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit-tests.ini. After this is done, it should be picked up by the B2G CI, after the gecko fork of B2G is updated, where it will be reported along with the other tests to autolog.
Caveats, provisos, and miscellanea
B2G builds go to sleep after 60 seconds of inactivity. In the emulator, this “sleep” will completely lock up Marionette if it occurs while a test is running. This is very inconvenient while testing. See bug 739476. Until some better mechanism of handling this is available, I usually edit gecko/b2g/apps/b2g.js to increase the value of the power.screen.timeout pref before building, to prevent the emulator from going to sleep.
The current test failures in autolog are being tracked as bug 751403 and bug 751406.
Network access in the emulator currently doesn’t seem to work (see https://github.com/andreasgal/B2G/issues/287). This prevents some parts of Gaia from working correctly but doesn’t interfere with the above style of WebAPI tests, none of which rely on Gaia or network access.
Building the emulator is very time-consuming, mostly due to the time required to sync all the various repos needed by B2G. We hope to be able to post emulator builds for download soon, after a few details are worked out.
More reading
Please contribute tests
There are many WebAPIs which are less tested than they could be. Please help us expand test coverage by contributing tests in areas similar to those described above. If you need help, contact :jgriffin or :mdas on IRC, or file a bug under Testing:Marionette.
