At Mozilla, we have many different testing frameworks, each of which fills a different niche (although there is definitely some degree of overlap among them). For testing WebAPIs in B2G, some of these existing frameworks can be utilized, depending on the API. For example, mozSettings and mozContacts can be tested using mochitests, since there isn’t much, if anything, that’s device-specific to them. (We’re not currently running mochitests on B2G devices, but will be soon.)
But there are many other WebAPIs which are not testable using any of our standard frameworks, because tests for them need to interact with hardware in interesting ways, and most of our frameworks are designed to operate entirely within a gecko context, and thus have no ability to directly access hardware.
Malini Das and I have been working on a new framework called Marionette which can help. Marionette is a remote test driver, so it can remotely execute test steps within a gecko process while retaining the ability to interact with the outside world, including devices running B2G. When this is combined with the B2G emulator’s ability to query and set hardware state, we have a solution for testing a number of WebAPIs that would be difficult or impossible to test otherwise.
To illustrate how this works, I’m going to walk through the entire process of writing WebAPI tests for mozBattery and mozTelephony, to be run on B2G emulators. We already have such tests running in continuous integration, reporting to autolog. If developers add new Marionette WebAPI tests, they will be run and reported here as well. Eventually, they will likely be migrated over to TBPL.
Building the emulator
These tests will be run on the emulator, so you’ll have to build the B2G Ice Cream Sandwich emulator first, if you don’t have one already. You’ll need to do this on linux, preferably Ubuntu. Make sure to install the build prerequisites before you begin, if you haven’t built B2G before.
git clone https://github.com/andreasgal/B2G cd B2G make sync (get a cup of coffee, this takes quite a while) make config-qemu-ics (get another cup of coffee) make gonk (get another drink, but I think you've had enough coffee by now) make
You should now have an emulator, which can you launch using:
./emu-ics.sh
After you’ve verified the emulator is working, close it again.
Running a Marionette sanity test
Now we’ll run a single Marionette test to verify that everything is working as expected. First, ensure that you have Python 2.7 on your system. Then, install some prerequisites:
pip install (or easy_install) manifestdestiny pip install (or easy_install) mozhttpd pip install (or easy_install) mozprocess
Now, from the directory where you cloned the B2G repo:
cd gecko/testing/marionette/client/marionette python runtests.py --emulator --homedir /path/to/B2G/repo \ tests/unit/test_simpletest_sanity.py
If everything has gone well, you should see something like the following:
TEST-START test_simpletest_sanity.py test_is (test_simpletest_sanity.SimpletestSanityTest) ... ok test_isnot (test_simpletest_sanity.SimpletestSanityTest) ... ok test_ok (test_simpletest_sanity.SimpletestSanityTest) ... ok ---------------------------------------------------------------------- Ran 3 tests in 2.952s OK SUMMARY ------- passed: 3 failed: 0 todo: 0
Writing a battery test
The B2G emulator allows you to arbitrarily set the battery level and charging state, by telnetting into the emulator’s console port and issuing certain commands. Marionette has an EmulatorBattery class which abstracts these operations, and allows you to interact with the emulator’s battery using a very simple API.
A simple example is given in the EmulatorBattery documentation on MDN. Save this example to a file named test_battery_example.py, and run this command:
python runtests.py --emulator --homedir /path/to/B2G/repo /path/to/test_battery_example.py
Marionette should launch an emulator and run the test; when it’s done you should see:
TEST-START test_battery_example.py test_level (test_battery_example.TestBatteryLevel) ... ok ---------------------------------------------------------------------- Ran 1 test in 0.391s OK SUMMARY ------- passed: 1 failed: 0 todo: 0
How it works
This test, like all Marionette Python tests, is written using Python’s unittest framework, which provides the assert methods used in the test. Other methods used by the test are provided by the Marionette and EmulatorBattery classes.
When the test executes this line:
self.marionette.emulator.battery.level = 0.25
the EmulatorBattery class telnets into the emulator and sets the battery’s level. We then read the level back (which invokes another telnet command) to verify that the emulator’s battery state was updated as expected. And finally, we execute a snippet of JavaScript inside gecko:
moz_level = self.marionette.execute_script("return navigator.mozBattery.level;")
and verify that it returns the same battery level as the emulator is reporting directly.
More tests with hardware interaction
In addition to battery interaction, the B2G emulator allows you to query and set the state of other properties normally set by hardware, like GPS location, network status, and various sensors. Tests for all these could be written in a similar way. It probably makes sense to make classes for these similar to EmulatorBattery which abstract the details of getting and setting the state of the underlying hardware. I would encourage WebAPI developers to add as many WebAPI tests as possible; if you would like us to add convenience classes, please ping us on IRC (jgriffin and mdas, on #ateam or #b2g) or file a bug under Testing:Marionette.
Multi-emulator tests
There are some WebAPIs which cannot be completely tested using a single device or emulator, like telephony and SMS. Marionette can help with these too, as Marionette can be used to manipulate two emulator instances which are capable of communicating with each other.
In any tests run with the --emulator switch, Marionette launches an emulator before running the tests, and this emulator is associated with an instance of the Marionette class available to the test as self.marionette. Tests can invoke a second emulator instance using self.get_new_emulator(), and these emulator instances can call and text each other using their port numbers as their phone numbers.
To illustrate how this works, Malini has written an example test in which one emulator is used to dial another, and the caller’s number is verified on the receiver. See this example at https://developer.mozilla.org/en/Marionette/Marionette_Python_Tests/Emulator_Integrated_Tests#Manage_Multiple_Emulators.
If you save this example to test_dial_example.py and run the command:
python runtests.py --emulator --homedir /path/to/B2G/repo /path/to/test_dial_example.py
you should see Marionette launch one emulator, and then after it starts execution of the test, you should see a second emulator instance launch. After the test is done, you should see a successful report, similar to the one shown for the battery test.
We currently have a few tests for mozTelephony, but many more could be added, and new tests should be added for SMS/MMS as well.
Adding new tests to the B2G continuous integration
When new test are ready to be added to the CI, they should be checked into gecko under their dom component, e.g., dom/telephony/test/marionette. They should be added to the manifest.ini file in the same directory, and then for new manifest.ini files, the path to the .ini file should be added to the master manifest at http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit-tests.ini. After this is done, it should be picked up by the B2G CI, after the gecko fork of B2G is updated, where it will be reported along with the other tests to autolog.
Caveats, provisos, and miscellanea
B2G builds go to sleep after 60 seconds of inactivity. In the emulator, this “sleep” will completely lock up Marionette if it occurs while a test is running. This is very inconvenient while testing. See bug 739476. Until some better mechanism of handling this is available, I usually edit gecko/b2g/apps/b2g.js to increase the value of the power.screen.timeout pref before building, to prevent the emulator from going to sleep.
The current test failures in autolog are being tracked as bug 751403 and bug 751406.
Network access in the emulator currently doesn’t seem to work (see https://github.com/andreasgal/B2G/issues/287). This prevents some parts of Gaia from working correctly but doesn’t interfere with the above style of WebAPI tests, none of which rely on Gaia or network access.
Building the emulator is very time-consuming, mostly due to the time required to sync all the various repos needed by B2G. We hope to be able to post emulator builds for download soon, after a few details are worked out.
More reading
Please contribute tests
There are many WebAPIs which are less tested than they could be. Please help us expand test coverage by contributing tests in areas similar to those described above. If you need help, contact :jgriffin or :mdas on IRC, or file a bug under Testing:Marionette.