Is this thing on?
It’s the first week of November, and because of the December all-hands and the end-of-year holidays, this essentially means the quarter is half over. You can see what the team is up to and how we’re tracking against our deliverables with this spreadsheet.
hg.mozilla.org: gps did some interesting work investigating ways to increase cloning performance on Windows; it turns out closing files which have been appended is a very expensive process there. He also helped roll out bundle-related cloning improvements in Mercurial 3.6.
Community: jmaher has posted details about our newest Quarter of Contribution. One of our former Outreachy interns, adusca, has blogged about what she gets out of contributing to open source software.
MozReview and Autoland: dminor blogged about the most recent MozReview work week in Toronto. Meanwhile, mcote is busy trying to design a more intuitive way to deal with parent-child review requests. And glob, who is jumping in to help out with MozReview, has created a high-level diagram sketching out MozReview’s primary components and dependencies.
Autoland has been enabled for the version-control-tools repo and is being dogfooded by the team. We hope to have it turned on for landings to mozilla-inbound within a couple of weeks.
Treeherder: the team is in London this week working on the automatic starring project. They should be rolling out an experimental UI soon for feedback from sheriffs and others. armenzg has fixed several issues with automatic backfilling so it should be more useful.
Perfherder: wlach has blogged about recent improvements to Perfherder, including the ability to track the size of the Firefox installer.
Developer Workflows: gbrown has enabled |mach run| to work with Android.
TaskCluster Support: the mochitest-gl job on linux64-debug is now running in TaskCluster side-by-side with buildbot. Work is ongoing to green up other suites in TaskCluster. A few other problems (like failure to upload structured logs) need to be fixed before we can turn off the corresponding buildbot jobs and make the TaskCluster jobs “official”.
e10s Support: we are planning to turn on e10s tests on Windows 7 as they are greened up; the first job which will be added is the e10s version of mochitest-gl, and the next is likely mochitest-devtools-chrome. To help mitigate capacity impacts, we’ve turned off Windows XP tests by default on try in order to allow us to move some machines from the Windows XP pool to the Windows 7 pool, and some machines have already been moved from the Linux 64 pool (which only runs Talos and Android x86 tests) to the Windows 7 pool. Combined with some changes recently made by Releng, Windows wait times are currently not problematic.
WebDriver: ato, jgraham and dburns recently went to Japan to attend W3C TPAC to discuss the WebDriver specification. They will be extending the charter of the working group to get it through to CR. This will mean certain parts of the specification need to finished as soon as possible to start getting feedback.
Firefox and Media Automation
It’s Q4, and at Mozilla that means it’s planning season. There’s a lot of work happening to define a Vision, Strategy and Roadmap for all of the projects that Engineering Productivity is working on; I’ll share progress on that over the next couple of updates.
Build System: Work is starting on a comprehensive revamp of the build system, which should make it modern, fast, and flexible. A few bits of this are underway (like migration of remaining Makefiles to moz.build); more substantial progress is being planned for Q1 and the rest of 2016.
Bugzilla: Duo 2FA support is coming soon! The necessary Bugzilla changes has landed, we’re just waiting for some licensing details to be sorted out.
Treeherder: Improvements have been made to the way that sheriffs can backfill jobs in order to bisect a regression. Meanwhile, lots of work continues on backend and frontend support for automatic starring.
Perfherder and Performance Testing: Some optimizations were made to Perfherder which has made it more performant – no one wants a slow performance monitoring dashboard! jmaher and bc are getting close to being able to run Talos on real devices via Autophone; some experimental runs are already showing up on Treeherder.
MozReview and Autoland: It’s no longer necessary to have an LDAP account in order to push commits to MozReview; all that’s needed is a Bugzilla account. This opens the door to contributors using the system. Testing of Autoland is underway on MozReview’s dev instance – expect it to be available in production soon.
TaskCluster Migration: OSX cross-compiled builds are now running in TaskCluster and appearing in Treeherder as Tier-2 jobs, for debug and static checking. The TC static checking build with likely become the official build soon (and the buildbot build retired); the debug build won’t become official until work is done to enable existing test jobs to consume the TC build.
Work is progressing on enabling TaskCluster test jobs for linux64-debug; our goal is to have these all running side-by-side the buildbot jobs this quarter, so we can compare failure rates before turning off the corresponding buildbot jobs in Q1. Moving these jobs to TaskCluster enables us to chunk them to a much greater degree, which will offer some additional flexibility in automation and improve end-to-end times for these tests significantly.
Mobile Automation: All Android test suites that show in Treeherder can now be run easily using mach.
Dev Workflow: It’s now easier to create new web-platform-tests, thanks to a new |mach web-platform-tests-create| command.
e10s Support: web-platform-tests are now running in e10s mode on linux and OSX platforms. We want to turn these and other tests in e10s mode on for Windows, but have hardware capacity problems. Discussions are underway on how to resolve this in the short-term; longer-term plans include an increase in hardware capacity.
Test Harnesses: run-by-dir is now applied to all mochitest jobs on desktop. This improves test isolation and paves the way for chunking changes which we will use to improve end-to-end times and make bisection turnaround faster. Structured logging has been rolled out to Android reftests; Firefox OS reftests still to come.
ActiveData: Work is in progress to build out a model of our test jobs running in CI, so that we can identify pieces of job setup and teardown which are too slow and targets of possible optimization, and so that we can begin to predict the effects of changes to jobs and hardware capacities.
hg.mozilla.org: Mercurial 3.6 will have built-in support for seeding clones from pre-generated bundle files, and will have improved performance for cloning, especially on Windows.
Marionette and WebDriver: Message sequencing is being added to Marionette; this will help prevent synchronization issues where the client mixes up responses. Client-side work is being done in both Python and node.js. ato wrote an article making a case against visibility checks in WebDriver.
Firefox and Media Automation
We’ve said good-bye to Q3, and are moving on to Q4. Planning for Q4 goals and deliverables is well underway; I’ll post a link to the final versions next update.
Last week, a group of 8-10 people from Engineering Productivity gathered in Toronto to discuss approaches to several aspects of developer workflow. You can look at the notes we took; next up is articulating a formal Vision and Roadmap for 2016, which incorporates both this work as well as other planning which is ongoing separately for things like MozReview and Treeherder.
Bugzilla: Support for 2FA has been enhanced.
Perfherder and Performance Automation:
MozReview and Autoland: mcote posted a blog post detailing some of the rough edges in MozReview, and explaining how the team intends on tackling these. dminor blogged about the state of autoland; in short, we’re getting close to rolling out an initial implementation which will work similarly to the current “checkin-needed” mechanism, except, of course, it will be entirely automated. May you never have to worry about closed trees again!
Mobile Automation: gbrown made some additional improvements to mach commands on Android; bc has been busy with a lot of Autophone fixes and enhancements.
Firefox Automation: maja_zf has enabled MSE playback tests on trunk, running per-commit. They will go live at the next buildbot reconfig.
Developer Workflow: numerous enhancements have been made to |mach try|; see list below in the Details section. run-by-dir has been applied to mochitest-plain on most platforms, and to mochitest-chrome-opt, by kaustabh93, one of team’s contributors. This reduces test bleedthrough, a source of intermittent failures, as well as improves our ability to change job chunking without breaking tests.
Build System: gps has improved test package generation, which results in significantly faster builds – a savings of about 5 minutes per build on OSX and Windows in automation; about 90s on linux.
TaskCluster Migration: linux64 debug builds are now running, so ahal is unblocked on getting linux64 debug tests running in TaskCluster. armenzg has landed mozharness code to support running buildbot jobs via TaskCluster scheduling, via buildbot bridge.
Bugzilla: The BMO has been busy implementing security enhancements, and as a result, BMO now supports two-factor authentication. Setting this up is easy through BMO’s Preferences page.
Treeherder: The size of the Treeherder database dropped from ~700G to around ~80G thanks to a bunch of improvements in the way we store data. Jonathan French is working on improvements to the Sheriff Panel. And Treeherder is now ingesting data that will be used to support Automatic Starring, a feature we expect to be live in Q4.
Perfherder and Performance: Will Lachance has published a roadmap for Perfherder, and has landed some changes that should improve Perfherder’s performance. Talos tests on OSX 10.10 have been hidden in Perfherder because the numbers are very noisy; the reason for this is not currently known. Meanwhile, Talos has finally landed in mozilla-central, which should make it easier to iterate on. Thanks to our contributor Julien Pagès for making this happen! Joel Maher has posted a Talos update on dev.platform with many more details.
MozReview and Autoland: The web UI now uses BMO API keys; this should make logins smoother and eliminate random logouts. Several UI improvements have been made; see full list in the “Details” section below.
Mobile Automation: Geoff Brown has landed the much-requested |mach android-emulator| command, which makes it much easier to run tests locally with an Android emulator. Meanwhile, we’re getting closer to moving the last Talos Android tests (currently running on panda boards) to Autophone.
Developer Workflow: Our summer intern, Vaibhav Agrawal, landed support for an initial version of |mach find-test-chunk|, which can tell you which chunk a test gets run in. This initial version supports desktop mochitest only. Vaibhav gave an intern presentation this week, “Increasing Productivity with Automation”. Check it out!
General Automation: James Graham has enabled web-platform-tests-e10s on try, but they’re hidden pending investigation of tests which are problematic with e10s enabled. Joel Maher and Kim Moir in Release Engineering have tweaked our SETA coalescing, so that lower prioritized jobs are run at least every 7th push, or every hour; further increasing the coalescing window will wait until we have better automatic backfililng in place. Meanwhile, the number of chunks of mochitest-browser-chrome has been increased from 3 to 7, with mochitest-devtools-chrome soon to follow. This will make backfilling faster, as well as improving turnaround times on our infrastructure.
hg.mozilla.org: The bzexport and bzpost extensions have been updated to support BMO API keys.
Bughunter: Our platform coverage now includes opt and debug builds of linux32, linux64, opt-asan-linux64, OSX 10.6, 10.8, 10.9, and windows7 32- and 64-bit.
It’s PTO season and many people have taken a few days or week off. While they’re away, the team continues making progress on a variety of fronts. Planning also continues for GoFaster and addon-signing, which will both likely be significant projects for the team in Q4.
Treeherder: camd rolled out a change which collapses chunked jobs on Treeherder, reducing visual noise. In the future, we plan on significantly increasing the number of chunks of many jobs in order to reduce runtimes, so this change makes that work more practical. See camd’s blog post. emorley has landed a change which allows TaskCluster job errors that occur outside of mozharness to be properly handled by Treeherder.
Automatic Starring: jgraham has developed a basic backend which supports recognizing simple intermittent failures, and is working on integrating that into Treeherder; mdoglio is landing some related database changes. ekyle has received sheriff training from RyanVM, and plans to use this to help improve the automated failure recognition algorithm.
Perfherder and Performance Testing: Datazilla has finally been decommissioned (R.I.P.), in favor of our newest performance analysis tool, Perfherder. A lot of Talos documentation updates have been made at https://wiki.mozilla.org/Buildbot/Talos, including details about how we perform calculations on data produced by Talos. wlach performed a useful post-mortem of Eideticker, with several takeaways which should be applicable to many other projects.
MozReview and Autoland: There’s a MozReview meetup underway, so expect some cool updates next time!
TaskCluster Support: ted has made a successful cross-compiled OSX build using TaskCluster! Take it for a spin. More work is needed before we can move OSX builds from the mac mini builders to the cloud.
Mobile Automation: gbrown continues to make improvements on the new |mach emulator| command which makes running Android tests locally on emulator very simple.
General Automation: run-by-dir is live on opt mochitest-plain; debug and ASAN coming soon. This reduces test “bleed-through” and makes it easier to change chunking. adusca, our Outreachy intern, is working to integrate the try extender into Treeherder. And ahal has merged the mozharness “in-tree” configs with the regular mozharness config files, now that mozharness lives in the tree.
Firefox Automation: YouTube ad detection has been improved for firefox-media-tests by maja, which fixes the source of the top intermittent failure in this suite.
Bughunter: bc has got asan-opt builds running in production, and is working on gtk3 support.
hg.mozilla.org: gps has enabled syntax highlighting in hgweb, and has added a new JSON API as well. See gps’ blog post.
“Automation and Tools” has been our name for a long time, but it is a catch-all name which can mean anything, everything, or nothing, depending on the context. Furthermore, it’s often unclear to others which “Automation” we should own or help with.
For these reasons, we are adopting the name “Engineering Productivity”. This name embodies the diverse range of work we do, reinforces our mission (https://wiki.mozilla.org/Auto-tools#Our_Mission), promotes immediate recognition of the value we provide to the organization, and encourages a re-commitment to the reason this team was originally created—to help developers move faster and be more effective through automation.
The “A-Team” nickname will very much still live on, even though our official name no longer begins with an “A”; the “get it done” spirit associated with that nickname remains a core part of our identity and culture, so you’ll still find us in #ateam, brainstorming and implementing ways to make the lives of Mozilla’s developers better.
Treeherder: Most of the backend work to support automatic starring of intermittent failures has been done. On the front end, several features were added to make it easier for sheriffs and others to retrigger jobs to assist with bisection: the ability to fill in all missing jobs for a particular push, the ability to trigger Talos jobs N times, the ability to backfill all the coalesced jobs of a specific type, and the ability to retrigger all pinned jobs. These changes should make bug hunting much easier. Several improvements were made to the Logviewer as well, which should increase its usefulness.
Perfherder and performance testing: Lots of Perfherder improvements have landed in the last couple of weeks. See details at wlach’s blog post. Meanwhile, lots of Talos cleanup is underway in preparation for moving it into the tree.
MozReview: Some upcoming auth changes are explained in mcote’s blog post.
Mobile automation: gbrown has converted a set of robocop tests to the newly enabled mochitest-chrome on Android. This is a much more efficient harness and converting just 20 tests has resulted in a reduction of 30 minutes of machine time per push.
Developer workflow: chmanchester is working on building annotations into moz.build files that will automatically select or prioritize tests based on files changed in a commit. See his blog post for more details. Meanwhile, armenzg and adusca have implemented an initial version of a Try Extender app, which allows people to add more jobs on an existing try push. Additional improvements for this are planned.
Firefox automation: whimboo has written a Q2 Firefox Automation Report detailing recent work on Firefox Update and UI tests. Maja has improved the integration of Firefox media tests with Treeherder so that they now officially support all the Tier 2 job requirements.
WebDriver and Marionette: WebDriver is now officially a living standard. Congratulations to David Burns, Andreas Tolfsen, and James Graham who have contributed to this standard. dburns has created some documentation which describes which WebDriver endpoints are implemented in Marionette.
Version control: The ability to read and extra metadata from moz.build files has been added to hg.mozilla.org. This opens the door to cool future features, like the ability auto file bugs in the proper component and automatically selecting appropriate reviewers when pushing to MozReview. gps has also blogged about some operational changes to hg.mozilla.org which enables easier end-to-end testing of new features, among other things.
Treeherder: We’ve added to mozlog the ability to create error summaries which will be used as the basis for automatic starring. The Treeherder team is working on implementing database changes which will make it easier to add support for that. On the front end, there’s now a “What’s Deployed” link in the footer of the help page, to make it easier to see what commits have been applied to staging and production. Job details are now shown in the Logviewer, and a mockup has been created of additional Logviewer enhancements; see bug 1183872.
MozReview and Autoland: Work continues to allow autoland to work on inbound; MozReview has been changed to carry forward r+ on revised commits.
Bugzilla: The ability to search attachments by content has been turned off; BMO documentation has been started at https://bmo.readthedocs.org.
Perfherder/Performance Testing: We’re working towards landing Talos in-tree. A new Talos test measuring tab-switching performance has been created (TPS, or Talos Page Switch); e10s Talos has been enabled on all platforms for PGO builds on mozilla-central. Some usability improvements have been made to Perfherder – https://treeherder.mozilla.org/perf.html#/graphs.
TaskCluster: Successful OSX cross-compilation has been achieved; working on the ability to trigger these on Try and sorting out details related to packaging and symbols. Work on porting Linux tests to TaskCluster is blocked due to problems with the builds.
Marionette: The Marionette-WebDriver proxy now works on Windows. Documentation on using this has been added at https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/WebDriver.
Developer Workflow: A kill_and_get_minidump method has been added to mozcrash, which allows us to get stack traces out of Windows mochitests in more situations, particularly plugin hangs. Linux xpcshell debug tests have been split into two chunks in buildbot in order to reduce E2E times, and chunks of mochitest-browser-chrome and mochitest-devtools-chrome have been re-normalized by runtime across all platforms. Now that mozharness lives in the tree, we’re planning on removing the “in-tree configs”, and consolidating them with the previously out-of-tree mozharness configs (bug 1181261).
Tools: We’re testing an auto-backfill tool which will automatically retrigger coalesced jobs in Treeherder that precede a failing job. The goal is to reduce the turnaround time required for this currently manual process, which should in turn reduce tree closure times related to test failures
This post describes the status of the various pieces of B2G test automation.
We use a Jenkins instance to run continuous integration tests for B2G, using B2G emulators. Unfortunately, this has been unable to run any tests for several weeks due to incompatibilities between the emulator and the headless Amazon AWS linux VM’s we have been running the CI on, which have arisen due to the work on hardware acceleration in B2G. Michael Wu has identified a new VM configuration which does work (Ubuntu 12.04 + Xorg + xorg-video-dummy), and I’m busy switching our CI over to new VM’s of this configuration. The WebAPI tests are already running again, and the rest will be soon.
As soon as tests are rolling again normally, those of us most closely involved in B2G test automation (myself, Malini Das, Andrew Halberstadt, and Geo Mealer) will institute some informal sheriffing on Autolog (a TBPL look-alike) to help keep track of test failures. If you’d like to help with this effort, let me know.
Our B2G test automation has gone down for weeks at a time on several occasions over the past few months. Typically this has one of two causes:
Mochitest: will be running soon. We’re currently only running the subset of tests that used to be run by Fennec. We know we want to run all of them, but running all of them results in so many timeouts that the harness aborts. We’ll need to spend some time triaging these. We also know we want to change the way we run mochitests so that we can run them out-of-process: bug 777871.
XPCShell tests: running locally with good results, thanks to Mihnea Balaur, an A-Team intern. We will add them to the CI after mochitests.
Reftests: Andrew Halberstadt has these running locally and is working to understand test failures (bug 773842). He will get them running on a daily basis on a linux desktop with an Nvidia GPU, reporting to the same Autolog instance used by our Jenkins CI. If we need more frequent coverage and running them on the Amazon VM’s would provide useful data, we can do that. The reftest runner also needs to be modified so that it runs tests out-of-process: bug 778072.
Eideticker: Malini Das is working to adapt William Lachance’s Eideticker harness to B2G. This will be used to generate frame-rate data for inter- and intra-app transitions. The testing will be performed on panda boards. See bug 769167.
Other performance tests: There are no plans at this time to port talos to B2G. Malini has written a simple performance test using Marionette, which tracks the amount of time needed to launch each of the Gaia apps on an emulator. This has suffered from the same emulator problems described above, and needs to be moved to a new VM. This test currently reports to a new graphserver system called Datazilla, which isn’t in production yet. Once it goes live, we’ll be able to analyze the data and see whether the current test provides useful data, and what other tests would be useful to write.
Gaia integration tests: James Lal has recently added these. I’ll hook these up to CI soon.
The emulator is not an ideal test platform for several reasons, most notably poor performance and the fact that it doesn’t provide the real hardware environment that we care about. But actual phones are often not good automation targets either; they tend to suffer from problems relating to networking, power consumption, and rebooting that make them a nightmare to deal with in large-scale automation. Because of this, we’re going to target panda boards for test automation on real hardware. This is the same platform that will be used for Fennec automation, so we can leverage a lot of that team’s work.
There are several things needed for this to happen; see bug 777530. First, we need to get B2G panda builds in production using buildbot; we need to figure out how to flash B2G on pandas remotely; we need to adapt all the testrunners to work with the panda boards; and we need to write mozharness scripts for B2G unit tests, to allow them to work in rel-eng’s infrastructure.
For reftests, we also need to figure out “the resolution problem”: the fact that we can’t set the pandas to a resolution that would allow the reftest window to be exactly 800×1000, which is the resolution that test authors assume when writing reftests. Running reftests at other resolutions is possible, but we don’t know how many false passes we might be seeing, and analyzing the tests to try and determine this is laborious.
There are a lot of dependencies here, so I don’t have a very good ETA. But when this work is done, we will transition all of testing to pandas on rel-eng infrastructure, except for the WebAPI tests which have been written specifically for the emulator. This means the tests will show up on TBPL; they’ll be available on try; they will benefit from formal sheriffing. The emulator WebAPI tests will eventually be transitioned to rel-eng as well, if/when rel-eng starts making emulator builds.
At Mozilla, we have many different testing frameworks, each of which fills a different niche (although there is definitely some degree of overlap among them). For testing WebAPIs in B2G, some of these existing frameworks can be utilized, depending on the API. For example, mozSettings and mozContacts can be tested using mochitests, since there isn’t much, if anything, that’s device-specific to them. (We’re not currently running mochitests on B2G devices, but will be soon.)
But there are many other WebAPIs which are not testable using any of our standard frameworks, because tests for them need to interact with hardware in interesting ways, and most of our frameworks are designed to operate entirely within a gecko context, and thus have no ability to directly access hardware.
Malini Das and I have been working on a new framework called Marionette which can help. Marionette is a remote test driver, so it can remotely execute test steps within a gecko process while retaining the ability to interact with the outside world, including devices running B2G. When this is combined with the B2G emulator’s ability to query and set hardware state, we have a solution for testing a number of WebAPIs that would be difficult or impossible to test otherwise.
To illustrate how this works, I’m going to walk through the entire process of writing WebAPI tests for mozBattery and mozTelephony, to be run on B2G emulators. We already have such tests running in continuous integration, reporting to autolog. If developers add new Marionette WebAPI tests, they will be run and reported here as well. Eventually, they will likely be migrated over to TBPL.
These tests will be run on the emulator, so you’ll have to build the B2G Ice Cream Sandwich emulator first, if you don’t have one already. You’ll need to do this on linux, preferably Ubuntu. Make sure to install the build prerequisites before you begin, if you haven’t built B2G before.
git clone https://github.com/andreasgal/B2G cd B2G make sync (get a cup of coffee, this takes quite a while) make config-qemu-ics (get another cup of coffee) make gonk (get another drink, but I think you've had enough coffee by now) make
You should now have an emulator, which can you launch using:
After you’ve verified the emulator is working, close it again.
Now we’ll run a single Marionette test to verify that everything is working as expected. First, ensure that you have Python 2.7 on your system. Then, install some prerequisites:
pip install (or easy_install) manifestdestiny pip install (or easy_install) mozhttpd pip install (or easy_install) mozprocess
Now, from the directory where you cloned the B2G repo:
cd gecko/testing/marionette/client/marionette python runtests.py --emulator --homedir /path/to/B2G/repo \ tests/unit/test_simpletest_sanity.py
If everything has gone well, you should see something like the following:
TEST-START test_simpletest_sanity.py test_is (test_simpletest_sanity.SimpletestSanityTest) ... ok test_isnot (test_simpletest_sanity.SimpletestSanityTest) ... ok test_ok (test_simpletest_sanity.SimpletestSanityTest) ... ok ---------------------------------------------------------------------- Ran 3 tests in 2.952s OK SUMMARY ------- passed: 3 failed: 0 todo: 0
The B2G emulator allows you to arbitrarily set the battery level and charging state, by telnetting into the emulator’s console port and issuing certain commands. Marionette has an EmulatorBattery class which abstracts these operations, and allows you to interact with the emulator’s battery using a very simple API.
A simple example is given in the EmulatorBattery documentation on MDN. Save this example to a file named test_battery_example.py, and run this command:
python runtests.py --emulator --homedir /path/to/B2G/repo /path/to/test_battery_example.py
Marionette should launch an emulator and run the test; when it’s done you should see:
TEST-START test_battery_example.py test_level (test_battery_example.TestBatteryLevel) ... ok ---------------------------------------------------------------------- Ran 1 test in 0.391s OK SUMMARY ------- passed: 1 failed: 0 todo: 0
This test, like all Marionette Python tests, is written using Python’s unittest framework, which provides the assert methods used in the test. Other methods used by the test are provided by the Marionette and EmulatorBattery classes.
When the test executes this line:
self.marionette.emulator.battery.level = 0.25
moz_level = self.marionette.execute_script("return navigator.mozBattery.level;")
and verify that it returns the same battery level as the emulator is reporting directly.
In addition to battery interaction, the B2G emulator allows you to query and set the state of other properties normally set by hardware, like GPS location, network status, and various sensors. Tests for all these could be written in a similar way. It probably makes sense to make classes for these similar to EmulatorBattery which abstract the details of getting and setting the state of the underlying hardware. I would encourage WebAPI developers to add as many WebAPI tests as possible; if you would like us to add convenience classes, please ping us on IRC (jgriffin and mdas, on #ateam or #b2g) or file a bug under Testing:Marionette.
There are some WebAPIs which cannot be completely tested using a single device or emulator, like telephony and SMS. Marionette can help with these too, as Marionette can be used to manipulate two emulator instances which are capable of communicating with each other.
In any tests run with the
--emulator switch, Marionette launches an emulator before running the tests, and this emulator is associated with an instance of the
Marionette class available to the test as
self.marionette. Tests can invoke a second emulator instance using
self.get_new_emulator(), and these emulator instances can call and text each other using their port numbers as their phone numbers.
To illustrate how this works, Malini has written an example test in which one emulator is used to dial another, and the caller’s number is verified on the receiver. See this example at https://developer.mozilla.org/en/Marionette/Marionette_Python_Tests/Emulator_Integrated_Tests#Manage_Multiple_Emulators.
If you save this example to test_dial_example.py and run the command:
python runtests.py --emulator --homedir /path/to/B2G/repo /path/to/test_dial_example.py
you should see Marionette launch one emulator, and then after it starts execution of the test, you should see a second emulator instance launch. After the test is done, you should see a successful report, similar to the one shown for the battery test.
We currently have a few tests for mozTelephony, but many more could be added, and new tests should be added for SMS/MMS as well.
When new test are ready to be added to the CI, they should be checked into gecko under their dom component, e.g.,
dom/telephony/test/marionette. They should be added to the
manifest.ini file in the same directory, and then for new manifest.ini files, the path to the .ini file should be added to the master manifest at http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit-tests.ini. After this is done, it should be picked up by the B2G CI, after the gecko fork of B2G is updated, where it will be reported along with the other tests to autolog.
B2G builds go to sleep after 60 seconds of inactivity. In the emulator, this “sleep” will completely lock up Marionette if it occurs while a test is running. This is very inconvenient while testing. See bug 739476. Until some better mechanism of handling this is available, I usually edit
gecko/b2g/apps/b2g.js to increase the value of the
power.screen.timeout pref before building, to prevent the emulator from going to sleep.
Network access in the emulator currently doesn’t seem to work (see https://github.com/andreasgal/B2G/issues/287). This prevents some parts of Gaia from working correctly but doesn’t interfere with the above style of WebAPI tests, none of which rely on Gaia or network access.
Building the emulator is very time-consuming, mostly due to the time required to sync all the various repos needed by B2G. We hope to be able to post emulator builds for download soon, after a few details are worked out.
There are many WebAPIs which are less tested than they could be. Please help us expand test coverage by contributing tests in areas similar to those described above. If you need help, contact :jgriffin or :mdas on IRC, or file a bug under Testing:Marionette.