JGriffin's Blog

Is this thing on?

Writing WebAPI tests for B2G using Marionette

At Mozilla, we have many different testing frameworks, each of which fills a different niche (although there is definitely some degree of overlap among them). For testing WebAPIs in B2G, some of these existing frameworks can be utilized, depending on the API. For example, mozSettings and mozContacts can be tested using mochitests, since there isn’t much, if anything, that’s device-specific to them. (We’re not currently running mochitests on B2G devices, but will be soon.)

But there are many other WebAPIs which are not testable using any of our standard frameworks, because tests for them need to interact with hardware in interesting ways, and most of our frameworks are designed to operate entirely within a gecko context, and thus have no ability to directly access hardware.

Malini Das and I have been working on a new framework called Marionette which can help. Marionette is a remote test driver, so it can remotely execute test steps within a gecko process while retaining the ability to interact with the outside world, including devices running B2G. When this is combined with the B2G emulator’s ability to query and set hardware state, we have a solution for testing a number of WebAPIs that would be difficult or impossible to test otherwise.

To illustrate how this works, I’m going to walk through the entire process of writing WebAPI tests for mozBattery and mozTelephony, to be run on B2G emulators. We already have such tests running in continuous integration, reporting to autolog. If developers add new Marionette WebAPI tests, they will be run and reported here as well. Eventually, they will likely be migrated over to TBPL.

Building the emulator

These tests will be run on the emulator, so you’ll have to build the B2G Ice Cream Sandwich emulator first, if you don’t have one already.  You’ll need to do this on linux, preferably Ubuntu.  Make sure to install the build prerequisites before you begin, if you haven’t built B2G before.

git clone https://github.com/andreasgal/B2G
cd B2G
make sync (get a cup of coffee, this takes quite a while)
make config-qemu-ics (get another cup of coffee)
make gonk (get another drink, but I think you've had enough coffee by now)
make

You should now have an emulator, which can you launch using:

./emu-ics.sh

After you’ve verified the emulator is working, close it again.

Running a Marionette sanity test

Now we’ll run a single Marionette test to verify that everything is working as expected.   First, ensure that you have Python 2.7 on your system.  Then, install some prerequisites:

pip install (or easy_install) manifestdestiny
pip install (or easy_install) mozhttpd
pip install (or easy_install) mozprocess

Now, from the directory where you cloned the B2G repo:

cd gecko/testing/marionette/client/marionette
python runtests.py --emulator --homedir /path/to/B2G/repo \
  tests/unit/test_simpletest_sanity.py

If everything has gone well, you should see something like the following:

TEST-START test_simpletest_sanity.py
test_is (test_simpletest_sanity.SimpletestSanityTest) ... ok
test_isnot (test_simpletest_sanity.SimpletestSanityTest) ... ok
test_ok (test_simpletest_sanity.SimpletestSanityTest) ... ok

----------------------------------------------------------------------
Ran 3 tests in 2.952s

OK

SUMMARY
-------
passed: 3
failed: 0
todo: 0

Writing a battery test

The B2G emulator allows you to arbitrarily set the battery level and charging state, by telnetting into the emulator’s console port and issuing certain commands.  Marionette has an EmulatorBattery class which abstracts these operations, and allows you to interact with the emulator’s battery using a very simple API.

A simple example is given in the EmulatorBattery documentation on MDN.  Save this example to a file named test_battery_example.py, and run this command:

python runtests.py --emulator --homedir /path/to/B2G/repo /path/to/test_battery_example.py

Marionette should launch an emulator and run the test; when it’s done you should see:

TEST-START test_battery_example.py
test_level (test_battery_example.TestBatteryLevel) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.391s

OK

SUMMARY
-------
passed: 1
failed: 0
todo: 0

How it works

This test, like all Marionette Python tests, is written using Python’s unittest framework, which provides the assert methods used in the test.  Other methods used by the test are provided by the Marionette and EmulatorBattery classes.

When the test executes this line:

self.marionette.emulator.battery.level = 0.25

the EmulatorBattery class telnets into the emulator and sets the battery’s level.  We then read the level back (which invokes another telnet command) to verify that the emulator’s battery state was updated as expected.  And finally, we execute a snippet of JavaScript inside gecko:

moz_level = self.marionette.execute_script("return navigator.mozBattery.level;")

and verify that it returns the same battery level as the emulator is reporting directly.

More tests with hardware interaction

In addition to battery interaction, the B2G emulator allows you to query and set the state of other properties normally set by hardware, like GPS location, network status, and various sensors.  Tests for all these could be written in a similar way.  It probably makes sense to make classes for these similar to EmulatorBattery which abstract the details of getting and setting the state of the underlying hardware.  I would encourage WebAPI developers to add as many WebAPI tests as possible; if you would like us to add convenience classes, please ping us on IRC (jgriffin and mdas, on #ateam or #b2g) or file a bug under Testing:Marionette.

Multi-emulator tests

There are some WebAPIs which cannot be completely tested using  a single device or emulator, like telephony and SMS.  Marionette can help with these too, as Marionette can be used to manipulate two emulator instances which are capable of communicating with each other.

In any tests run with the --emulator switch, Marionette launches an emulator before running the tests, and this emulator is associated with an instance of the Marionette class available to the test as self.marionette. Tests can invoke a second emulator instance using self.get_new_emulator(), and these emulator instances can call and text each other using their port numbers as their phone numbers.

To illustrate how this works, Malini has written an example test in which one emulator is used to dial another, and the caller’s number is verified on the receiver. See this example at https://developer.mozilla.org/en/Marionette/Marionette_Python_Tests/Emulator_Integrated_Tests#Manage_Multiple_Emulators.

If you save this example to test_dial_example.py and run the command:

python runtests.py --emulator --homedir /path/to/B2G/repo /path/to/test_dial_example.py

you should see Marionette launch one emulator, and then after it starts execution of the test, you should see a second emulator instance launch. After the test is done, you should see a successful report, similar to the one shown for the battery test.

We currently have a few tests for mozTelephony, but many more could be added, and new tests should be added for SMS/MMS as well.

Adding new tests to the B2G continuous integration

When new test are ready to be added to the CI, they should be checked into gecko under their dom component, e.g., dom/telephony/test/marionette. They should be added to the manifest.ini file in the same directory, and then for new manifest.ini files, the path to the .ini file should be added to the master manifest at http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit-tests.ini. After this is done, it should be picked up by the B2G CI, after the gecko fork of B2G is updated, where it will be reported along with the other tests to autolog.

Caveats, provisos, and miscellanea

B2G builds go to sleep after 60 seconds of inactivity.  In the emulator, this “sleep” will completely lock up Marionette if it occurs while a test is running.  This is very inconvenient while testing.  See bug 739476. Until some better mechanism of handling this is available, I usually edit gecko/b2g/apps/b2g.js to increase the value of the power.screen.timeout pref before building, to prevent the emulator from going to sleep.

The current test failures in autolog are being tracked as bug 751403 and bug 751406.

Network access in the emulator currently doesn’t seem to work (see https://github.com/andreasgal/B2G/issues/287).  This prevents some parts of Gaia from working correctly but doesn’t interfere with the above style of WebAPI tests, none of which rely on Gaia or network access.

Building the emulator is very time-consuming, mostly due to the time required to sync all the various repos needed by B2G.  We hope to be able to post emulator builds for download soon, after a few details are worked out.

More reading

What is Marionette

Marionette Python tests

Marionette Emulator tests

the Marionette class

the Emulator class

Please contribute tests

There are many WebAPIs which are less tested than they could be.  Please help us expand test coverage by contributing tests in areas similar to those described above.    If you need help, contact :jgriffin or :mdas on IRC, or file a bug under Testing:Marionette.

About these ads

11 responses to “Writing WebAPI tests for B2G using Marionette

  1. mw22 August 6, 2012 at 8:43 am

    Nowadays, you need to use ./run-emulator.sh instead of ./emu-ics.sh.

    I had to install pip: http://www.pip-installer.org/en/latest/installing.html#prerequisites
    I also had to do this:
    pip install datazilla
    pip install mozrunner

    Then I issued this:
    python runtests.py –emulator arm –homedir /Users/mw22/B2G/ tests/unit/test_simpletest_sanity.py

    That works, but sometimes times out, it seems. A typical run usually takes something around 35s.

    • jagriffin August 6, 2012 at 9:35 am

      What kind of timeout errors were you seeing? There are a few different ways a test run could time out, and they indicate different problems.

      • mw22 August 13, 2012 at 3:38 pm

        This is the error message that I’m seeing:
        Firefox:marionette mw22$ python runtests.py –emulator arm –homedir
        /Users/mw22/B2G tests/unit/test_simpletest_sanity.py
        starting httpd
        running webserver on http://192.168.1.102:59699/
        Traceback (most recent call last):
        File “runtests.py”, line 621, in
        runner.run_tests(tests, testtype=options.type)
        File “runtests.py”, line 295, in run_tests
        self.run_test(test, testtype)
        File “runtests.py”, line 328, in run_test
        self.start_marionette()
        File “runtests.py”, line 244, in start_marionette
        logcat_dir=self.logcat_dir)
        File “/Users/mw22/B2G/gecko/testing/marionette/client/marionette/marionette.py”,
        line 121, in __init__
        self.emulator.start()
        File “/Users/mw22/B2G/gecko/testing/marionette/client/marionette/emulator.py”,
        line 274, in start
        raise Exception(‘timed out waiting for emulator to start’)
        Exception: timed out waiting for emulator to start
        Firefox:marionette mw22$

        I seem to be getting this error messag every time now, though.

  2. Luuwen Day August 8, 2012 at 1:27 pm

    I got the following errors when ‘make sync':

    —————————————————————–
    Submodule ‘toolchains’ () registered for path ‘toolchains’
    fatal: Needed a single revision
    Unable to find current revision in submodule path ‘gecko’
    make: *** [sync] Error 1

    • Luuwen Day August 8, 2012 at 1:43 pm

      Got the following error when running ‘make gonk':

      ——————————————————————-
      device/samsung/crespo/device_base.mk:200: frameworks/base/build/phone-hdpi-512-dalvik-heap.mk: No such file or directory
      build/core/product_config.mk:193: *** _nic.PRODUCTS.[[device/samsung/maguro/full_maguro.mk]]: “frameworks/base/build/phone-xhdpi-1024-dalvik-heap.mk” does not exist. Stop.

      ** Don’t have a product spec for: ‘generic’
      ** Do you have the right repo manifest?

      make: *** [gonk] Error 1

  3. Luuwen Day August 9, 2012 at 9:10 am

    I got the following seg fault when I run emulator. I built on Ubuntu 64-bit VM using https://github.com/mozilla-b2g/B2G. I try to identify where the problem is, some 32-bit libraries I installed (libX11.so, libGL.so, libOSMesa.so, etc), config files, or parameters setting?

    ————————————————————————————————
    ./run-emulator.sh: line 40: 23359 Segmentation fault (core dumped) ${DBG_CMD} $EMULATOR -kernel $KERNEL -sysdir $B2G_HOME/out/target/product/$DEVICE/ -data $B2G_HOME/out/target/product/$DEVICE/userdata.img -memory 512 -partition-size 512 -skindir $B2G_HOME/development/tools/emulator/skins -skin HVGA -verbose -gpu on -qemu $TAIL_ARGS

    • Luuwen Day August 9, 2012 at 9:11 am

      The whole command should be:

      /data/Firefox.Mobile.OS.2/B2G/out/host/linux-x86/bin/emulator64-arm -kernel /data/Firefox.Mobile.OS.2/B2G/prebuilts/qemu-kernel/arm/kernel-qemu-armv7 -sysdir /data/Firefox.Mobile.OS.2/B2G/out/target/product/generic/ -data /data/Firefox.Mobile.OS.2/B2G/out/target/product/generic/userdata.img -memory 512 -partition-size 512 -skindir /data/Firefox.Mobile.OS.2/B2G/development/tools/emulator/skins -skin HVGA -verbose -gpu on -qemu -cpu cortex-a8

    • jagriffin August 9, 2012 at 11:12 am

      I’m not sure why this would happen; posting to the mozilla.dev.b2g newsgroup would probably be your best bet in getting help for this.

  4. largeBIRDtalons May 4, 2014 at 4:01 am

    I can run some pretty interesting tests thanks to Marionette.
    thanks

    I’m building a bunch of API’s to take advantage of it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 35 other followers

%d bloggers like this: