Auto-tools/Projects/MozBase

From MozillaWiki
Jump to navigation Jump to search

Overview

The goal of the MozBase project is to make easy to use python modules for managing the common operations that we do in all our test harnesses. MozBase is the effort to consolidate harness software into high-quality atomic pieces that may be independently maintained and used. MozBase code needs to work with Buildbot/Talos, reftest, mochitest, mozmill, xpcshell, and future harnesses.

Repository: https://github.com/mozilla/mozbase

Bugs:

Please file bugs against https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Mozbase

The roadmap of getting mozbase on mozilla-central is detailed at https://wiki.mozilla.org/Auto-tools/Projects/MozBase/Roadmap

MozBase requires python 2.5, 2.6, or 2.7 however, Talos still requires compatibility with python 2.4 (bug 734466), so mozbase dependencies in http://hg.mozilla.org/build/talos/file/tip/setup.py *must* be kept compatible with python 2.4

Packages

MozBase is composed of several python packages. These packages work together to form the basis of a test harness.

  • Firefox is launched via mozrunner
    • which sets up a profile with preferences and extensions using mozprofile
    • and runs the application under test using mozprocess
  • mozInstall is used to install the test application
  • A test harness may direct Firefox to load web pages. These may be served using mozhttpd for testing
  • The machine environment is introspected by mozinfo
  • A test manifest may be read to determine the tests to be run. These manifests are processed by ManifestDestiny
  • For mozbile testing, the test runner communicates to the test agent using mozdevice

Process Management - mozprocess package

Cross-platform process management. See processhandler.py for the mozprocess API.

Goals:

  • ability to reliably terminate processes across platforms

Status:

Profile Management - mozprofile package

The mozprofile package is complete with the exception that it might need the ability to install plugins as well as addons. We may expand these interfaces to use a packaged xpcshell to insert state-specific items into the profile (i.e. fire up xpcshell and use JS + XPCOM to create a set of bookmarks, for instance).

You can find the code in the MozProfile package.

Status:

Profiles Creation in Existing Mozilla Python Code

Currently several test harnesses modify profiles. The collected knowledge should be upstreamed to the mozprofile package and existing code made to use this package.

1) Talos has profile directories with the prefs.js and other related files already in there. We copy those to the tmp dir and point firefox at it. In addition, we create a user.js file from prefs stored in the .config file. The last thing we copy is the extensions (such as pageloader). bug 694638

2) reftest creates a profile by writing user.js and putting files in the folder to setup the extension. It runs firefox with the -silent flag and this allows for full registration of the reftest handler (as we have a cli flag that needs to be parsed).

3) mochitest follows suite and creates a profile but has a HUGE pac config and a lot of permissions.sqlite insertions. While that is the extent of the complexity (prefs.js, user.js, permissions.sqlite, extensions), it is a lot of stuff that is all hacked into automation.py.in.

These and other test harnesses should be brought up to speed to use mozprofile and, correspondingly, mozprofile built-out to fill the needs of the harnesses

Platform Information - mozinfo package

mozinfo wraps python utilities that gather system information.

Status:

Test Manifests - ManifestDestiny package

We have one manifest parser that will be used across the test systems. The parser reads ".ini" files with each section header representing a test path and the section's key,value pairs are the test's metadata for consumption of the caller. The code is in a working state and used, it is found in the ManifestDestiny package. The project page is at https://wiki.mozilla.org/Auto-tools/Projects/ManifestDestiny

Status:

Currently reftest use a different syntax. These are planned to be unified in the future.

Android Device Access - mozdevice package

This provides an abstraction called DeviceManager useful for interacting with an Android device. There are two variants of this class, one which allows you to interact with an agent process on the device using a custom TCP/IP protocol (DeviceManagerSUT), another that allows you to interact with the device using Android's adb interface.

Status:

Unified Logging - mozlog package

This package will need to be implemented in both JavaScript and Python so that it is accessible from both sides of the test harnesses and we can get away from hand-formatted "print" and "dump" statements.

See also: https://developer.mozilla.org/en/Test_log_format

Status:

Test Instatiation - moztest package

Moztest is a package that allows you to store and process test results.

You can use the classes in the results submodule to store results and then the classes in the output subpackage to get useful representations of them (for example xUnit, Autolog).

You can store environment data (i.e. OS that was used) using the TestContext class.

Moztest supports two ways of storing test results: either creating them live, while running the tests, or creating them after the tests have been run.

Creating the results objects while running the tests

1. Instantiate a TestResult object: 

t = TestResult('example', test_class='doc', context=TestContext(product='Kuma'), result_expected='PASS')

The test's time_start property will be set the current time.

2. Finalize the object (assuming the test passed): 

t.finish('PASS')

The test's time_end property will be set to the current time as expected.

Or, if the test failed: 

t.finish('FAIL', output=['Traceback:', 'Line ..', ..], reason='AssertionError: True is not False')

After you call finish(), the test's result property will contain the standard string for the corresponding result it has. For example, UNEXPECTED-FAIL.

Also, you can use the duration property to see how long a test took.


You manage test result data by using a TestResultCollection. The output subpackage expects TestResultCollection objects as well. These behave similar to lists. In fact, they are lists.

The general pattern is this: 

collection = TestResultCollection(suite_name='Example test run')
for test in tests_to_run:
    tr = TestResult(...)
    # run test..
    tr.finish(...)
    collection.append(tr)
    collection.time_taken += test.duration # this was not automated because of the multiple way of adding stuff to a list


Creating the results objects after running the tests

One use case would be: you have some insert test harness here results and you want to generate a xUnit file with the data.

The pattern differs, depending on the type of the results.

If they are python unittest-based results (i.e. Marionette), you can use a convenience classmethod of TestResultCollection: 

# assuming results is an iterable (say, a list) of unittest result objects
collection = TestResultCollection.from_unittest_results(*results)
# you can optionally specify a context parameter as well

If the results you have are not based on python's unittest results - for example, XPCShellTest results are not, the general pattern is something like this (the way in which you get the relevant data from the test may vary): 

collection = TestResultCollection('example suite')
for result in results:  # these would be the results you already have
    duration = result.get('time', 0)  # or some other way of finding the duration

    # figure out the result of the test
    if 'skipped' in result:
        outcome = 'SKIPPED'
     elif 'todo' in result:
         outcome = 'KNOWN-FAIL'
     elif result['passed']:
         outcome = 'PASS'
     else:
         outcome = 'UNEXPECTED-FAIL'

     # find its output, and maybe even the reason
     output = None
     if 'failure' in result:
         output = result['failure']['text']

     # if you only know the duration (no start & end times), just pass in 0 as start time
     t = TestResult(name=result['name'], test_class='ExampleTestClass',
                    time_start=0, context=context)
     # pass in the result and Moztest will infer your expected and actual results
     t.finish(result=outcome, time_end=duration, output=output)

     collection.append(t)
     collection.time_taken += duration


Managing test data - TestResultCollection's methods

# get a list of unique test contexts used
contexts = collection.contexts

# get a list of tests that had errors
errors = collection.tests_with_result('ERROR')

# get a list of tests that took longer than one minute
long_tests = collection.filter(lambda t: t.duration > 360)

# get a new TestResultCollection with a subset of the tests
new_collection = collection.subset(lambda t: t.context == desired_context)

OS Environment Handling - MozEnv

This is currently not implemented. I envision something that wraps os.environ but provides good methods for adding and removing attributes from the environment. 

interface Environment {
  void __init__(dict env = None)                # Defaults to os.environ if None
  
  void add(string name, string value)
  
  void remove (string name, string value)

  # TODO: Might make this the native __str__ in python so we don't need a method
  # called out, but explicitly stating we should have a way to dump the environment
  # to a string (for logging or display or debugging)
  string to_string()
}

Status:

  • not yet implemented

Command Line Parsing - mozoptions

Implemented to varying degrees in every single test harness. This is simply a specialized subclass of the python optparse (or argparse, come python 2.7) object. We should use it to define common options across all the test harnesses, and allow each harness to add in a callback for verification of these options. Otherwise, the methods are the standard optparse methods.

Status: not implemented

File Handling - mozfile

Common file-related code

Status: proposed: bug 774916

Development Practices

If you're developing mozbase code, you're in luck! You've found the right place to read about our development practices. Please do :)

Getting Help with Mozbase

Mozbase is developed by the Automation and Tools team. It is under the governership of Will Lachance (:wlach) and Jeff Hammel (:jhammel). Please feel free to stop by #ateam on IRC with any questions and we will be happy to help you! Or if something is wrong, please file a bug and we'll look at it as soon as possible.

Installing Mozbase for Development

Initial Setup

(Some of these steps may not be necessary if you've already set up your computer for development.)

  • (windows-only) Launch the 'start-l10n.bat', to get a mozilla-build environment.
  • Install virtualenv, if it's not on your system already. If on Windows, be sure to do this from inside the shell launched via 'start-l10n.bat'.

Installing Mozbase

 virtualenv mozbase
  • Activate the virtualenv:
 cd mozbase; . bin/activate
  • Create a source directory:
 mkdir src
  • Clone mozbase in the source directory:
 cd src; git clone git://github.com/mozilla/mozbase.git
  • Install packages in the virtualenv
 cd mozbase; python setup_development.py

Working on Mozbase and Contributing Patches

Changes to mozbase require peer review of a properly filed bug. Here's a workflow that will help you make changes to mozbase in isolation.

  • If there isn't one already on file, create a bug corresponding to your issue in the mozbase bugzilla component.
  • Inside the mozbase checkout, create a local branch corresponding to your bug:
 git checkout -b bug-xxxxxxx
  • Develop your patch
  • When you're happy with your work, commit your changes, giving them a descriptive title from the bug summary:
 Bug 706981 - Check for DistributionNotFound error
  • Generate a patch for your bug
 git show --format="From: %an <%ae>%n%s%n%b" > bug-xxxx.patch
  • Attach the patch to the bug and ask the appropriate person for review. If further changes are required, you can amend changes to your patch by using 'git commit --amend'. When you're all done, gently ask someone with privileges to merge your changes into the mozilla mozbase repository.

This is a recommended way of working with mozbase. In general the important things are:

What NOT to do:

  • don't use github issues. We track our development with bugzilla, so if you file a github issue that just makes more work for us to file a corresponding bugzilla issue and have double the issues to close out
  • don't use pull requests. This again requires us to do more work duplicating the pull request and associated patch in bugzilla

Tests

Mozbase packages are accompanied by tests in order to ensure and illustrate proper functionality.

Continuous integration via autobot is at http://k0s.org:8010/

Running the tests

You will need make and gcc in order to run the tests. If you're running windows, this is available in MozillaBuild.

Ensure that all mozbase packages are installed as the tests require them.

Then run the test.py test runner from the root of the git repository: 

   python test.py

This will run all tests from test-manifest.ini which is parsed with ManifestDestiny. On success 'test.py' should print the number of tests run and 'OK': 

   ----------------------------------------------------------------------
   Ran 56 tests in 114.807s
   
   OK

Check-in policy

Review Policy

All changes should be reviewed before landing. The one exception is version bumps. See https://wiki.mozilla.org/Auto-tools/Projects/MozBase#Versioning for how these are done.

If you have admin permissions to the upstream MozBase repository, you can land the reviewed patch on your own. Otherwise you should ask someone else to get the patch landed.

Landing the patch

Please never use the Github auto-merge feature!

To land the patch on the target branch always make use of the git merge --squash command, which will land the patch as a single commit. After running this command you will have to issue a new commit message before being able to push the changes. Make sure that you add the right author if the patch hasn't been created by yourself. Also always use the right commit message format as shown below which includes the bug number and the list of reviewers.

Here an example how to land a patch from the feature branch onto master:

  • git checkout master
  • git checkout -b feature
  • git pull remote feature (or git am patch)
  • [TEST the patch by running existent tests]
  • git merge --squash feature
  • git commit -m "Bug XYZ - Add new feature to MozBase. r=reviewer" --author="foo <foo@bar" -a
  • git push upstream (whereby upstream is: git@github.com:mozilla/mozbase.git)

Versioning

The major version should be bumped when the API changes in a non-backwards-compatible way or for other large conceptual changes. Otherwise, the minor version may be bumped. The version doesn't have to be bumped if there is no new release planned for PyPI.

When you bump a version of any mozbase package, it should be released to pypi: 

 python setup.py egg_info -RDb "" sdist register upload

and the github repository should be tagged a la 'mozrunner-1.2'. If multiple package versions are bumped, then the same changeset should be tagged for each of the package versions bumped.

Tagging the version on github: 

git pull --tags mozilla master
git tag mozrunner-1.5.5
git push --tags mozilla master

Using versionbump.py

A script, versionbump.py, located at https://github.com/mozilla/mozbase/blob/master/versionbump.py has been written to perform the steps for MozBase Versioning

  • make sure you are on the master branch of the mozbase repository. If you aren't the script will bail out.
  • make sure you don't have any uncommitted changes. If you do, the script will bail out.
  • you should probably pull from git@github.com:mozilla/mozbase.git prior to running the script. If not, the script will attempt to do this for you, though it will not attempt to resolve conflicts
  • as per usual, running 
    versionbump.py --help
    will display usage information and help about CLI options
  • versionbump.py --info
    will display the package versions and their dependencies and exit

In order to bump versions, versionbump.py is used like: 

 versionbump.py mozrunner=5.8 mozprofile=0.5 -m 'bug 123456 - bump mozprofile to 0.5'

You should pass in all the packages that need bumping. If you specify a package which is pegged as an exact dependency of another package (e.g. mozrunner requires mozprofile == 0.5), you must also bump all packages which exactly depend on this package. As an example: 

 versionbump.py mozprofile=0.5

would not work as mozrunner requires mozprofile==0.4 (as of this writing) and all packages in the repository HEAD must be compatible with all other packages in the repository. In addition, packages pegged with '>=' also follow this rule unless --strict is passed.

  • running 
    versionbump.py --diff path/to/file
    will output the resultant diff to the file and revert the repository. The repository will not be tagged nor will anything be uploaded to pypi
  • -m or --message should be passed to the command line in order to finalize the commit, tag the repository, and upload to pypi. If you do not pass -m or --message, the changes will be made to your working copy but not committed. The repository will not be tagged nor will packages be uploaded to pypi
  • --dry-run will print out what versionbump.py will do but no files will be changed or commands actually called

It is encouraged to use --diff and/or --dry-run before actually doing the version bump to make sure that it will do what you expect it to do. In the case where a known error is encountered, versionbump.py will revert to the original (clean) repository state. In the case where an unexpected error is encountered, the repository state is not reverted (so that it is easier to debug what happened).

Limitations of versionbump.py:

  • it is expected that setup.py versions are specified as 
    PACKAGE_VERSION = '1.2.3'
    (or "1.2.3"). They currently all are.
  • versionbump.py can only handle simple dependencies. That is for install_requires, foo, foo == 1.2.3, bar >= 4.5.6 are fine, but foo == 1.1, == 1.2 is not. mozbase currently only has simple depdencies

Mirroring

A copy of mozbase is mirrored to mozilla-central for use by software there: http://mxr.mozilla.org/mozilla-central/source/testing/mozbase/ We mirror the repository into it using this procedure:

https://wiki.mozilla.org/Auto-tools/HowTo/MirrorRepo

Licensing

Documentation

  • documentation should be in a 'README.md' file in each package's base directory. The documentation is done in markdown syntax so that the it is automatically rendered in the package index presentation on github.
  • DocumentIt is used in conjunction with a manifest to update documentation on the Mozilla Developer Network (MDN). (TODO: figure out how to take updates from MDN in a semi-automatic way. Also, since switching to kuma, these documents cannot be mirrored with DocumentIt currently. Don't even try.)
  • this page is the MozBase project page that gives information for core MozBase Developers and about the project itself. Documentation may move to and from this page and the various README files

Directory Structure and Imports

The mozbase repository contains several packages. Each package should have a directory structure like: 

   ${PACKAGE}
   ${PACKAGE}/setup.py
   ${PACKAGE}/README.md
   ${PACKAGE}/${PACKAGE}
   ${PACKAGE}/${PACKAGE}/__init__.py

Even if the package consists of a single module, this structure should be used (vs. py_modules) and '__init__.py' set to 

   from ${MODULE} import *

(with a sensible '__all__' defined in the module).

This structure serves several purposes:

  • this allows packages to be installed with the 'setup.py' script
  • if packages cannot be installed (for deployment reasons), altering 'sys.path' (or '$PYTHONPATH') can be used to utilize
  • with single-module packages, you get a sensibly named file that is imported in '__init__.py' and can be mirrored easily and it conforms to how all of the other packages work


Downstream packages

Many mozbase packages depend on other mozbase packages. The packages and their versions at https://github.com/mozilla/mozbase should be kept compatible with one another.

Other packages also depend on mozbase: http://k0s.org/mozilla/mozbase/dependencies.html Releases and version changes should be tested against downstream packages to ensure compatibility

PyPI

Releases of mozbase packages are uploaded to the Python Package Index (pypi) so that they may be easily consumed by setuptools and other installers.

Mozbase packages:

Package owners:

  • ahal
  • ctalbert
  • davehunt
  • David.Burns
  • elvis314
  • jgriffin
  • k0s
  • markrcote
  • whimboo
  • wlach

Integration Notes

Notes on integration with existing test harnesses.

Mozbase and Mozharness

mozbase and mozharness overlap intent with respect to making testing software and tools reusable and easy to write and extend

mozharness docs: http://escapewindow.com/mozharness/

Production use of mozharness will use mozbase packages from an internal pypi: http://puppetagain.pub.build.mozilla.org/data/python/packages/ (from https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Python )

  • mozprocess (basic low level process management)
    • subclass popen
      • open
      • poll
      • kill
      • wait
      • stdout
      • pid
    • readWithTimeout
  • mozfile (os file i/o managemenet)
    • copyfile
    • removedir
    • makedir
  • mozlog (log file management)
    • write
    • init

unique to harnesses

  • mozenv
    • addvars
    • createenv
  • mozprofile
    • addExtension(s)
    • addPref(s)
    • writeProfile
    • copyProfile
    • renameProfile
    • deleteProfile
Retrieved from "https://dev.wikimo.nonprod.webservices.mozgcp.net/index.php?title=Auto-tools/Projects/MozBase&oldid=470054"