Firefox/Python 3 Migration

Why This is Important

In mozilla-central there are over 3500 Python files (excluding third party files), comprising roughly 230k lines of code. Most of it is Python 2.[1]

Python 2 will reach it's End Of Life on January 1st 2020. This page collects links and resources to assist the migration of the Firefox ecosystem off of Python 2 and onto Python 3.

How We Get There

As of today (May 2019) we require both the Python 2 and Python 3 interpreters to be installed to build and develop Firefox.

Next we:

1. Make Python 3 porting and development safe by adding a Python 3 option to the developer toolchain. (mach lint, coverage.py, parts of the test suite)
2. Make key support libraries run in both Python 2 and Python 3. (mach-core)
3. Build interpreter switching mechanisms into the toolchain so we can run sub-components in just 2 or just 3. (mach sub-commands, py_action)
4. Port Python 2-only components to Python 3 piecemeal.
5. Remove Python 2 compatibility. (When that might happen needs discussion. Possibly sometime in 2020.)

Getting Involved

Find the others

Most discussion happens in the #python channel on Slack (NDA'd Mozillians only).

What you'll need

You will need a working Firefox build environment.

Where to start:

Remove more of the "excluded" paths from the py2 and py3 linters

See this Etherpad for instructions.

Roughly:

• Pick a directory from the list, and put your name beside it so we know it's ported
• Edit py2.yml and/or py3.yml and remove that directory from the exclusion list
• Run `./mach lint -l py2 -l py3 <your directory name>` to get a list of errors
• To get a list of files that need fixing:

./mach lint -l py2 -l py3 <your directory name> --format unix > tofix.txt

• To install `futurize`:

pip install --user future

• To run through futurize

cat tofix.txt | cut -f1 -d: | sort -u | xargs futurize -1 -w

• Visually inspect the patch, especially around `except:` clauses. Watch out for "xxx_todo_changeme" in the patch. These are places where futurize couldn't port the code and you need to fix it manually.

When your patch is ready file:

1. Use moz-phab for submitting the patch - https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#submitting-patches
2. Use bug 1559975
3. Set one of callek, ahal, or catlee as the code reviewer

Run all python source tests with Python 3 and see what breaks

Make sure each of the tasks listed in https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/python.yml has Python 3 specified, and do a try push, enabling all the python source tests.

See what breaks, then fix it!

Port a Mach Command

The mach driver (toplevel `mach` file) contains a whitelist of commands that still run with Python 2. Pick the command that you'd like to convert and then:

1. Remove it from the whitelist
2. Run: ./mach <command>
3. See what breaks and fix it!
4. Repeat with a variety of different flags and arguments
5. Make sure any Python unittests running in CI have Python 3 enabled (see above)

How to Port the Code

Before starting

If you haven't ported modules before (or need a refresher), read the Porting Guide on python.org. It has a good outline of the porting process.

Porting modules in the Firefox source tree

• We've already standardized on Python 2.7 across the tree.
• Make sure you can test the code you're working on with python3!
  • If using `tox`, then adding `py35`, and `py36` to the list of environments should be sufficient.
  • If using |mach python-test|, make sure the relevant manifest doesn't skip-if Python 3 and run |mach python-test --python 3 <path>|.
• `futurize` Stage 1 fixes can be used to transform code into code that's compatible with both python2.7 and python3.
• The `six` module is available in-tree, feel free to use it as a dependency.
• Black can format Python 2 code and Python 3 code (but it requires Py3.6+ to run).

Porting examples

Prefer six when possible. Make use of six.PY2, six.PY3, six.moves, six.string_types, six.integer_types, etc. This make it easy to find the py2 code again so we can remove it.

import statements

Use the six.moves module:

Before:

import __builtin__

__builtin__.__import__ = myimporthook()

After:

from six.moves import builtins

builtins.__import__ = myimporthook()

if statements

Use six.PY2 and six.PY3:

import six

if six.PY2:
    # py2 stuff
else:
    # py3 stuff

If six isn't available or you are in the imports section of the module then this is fine too:

if sys.version_info < (3,):
   # py2 stuff
else:
   # py3 stuff

String data

Use six.string_types, six.text_type, six.binary_type, etc. Also see the six docs for "Binary and text data":

Before:

# using unicode
def write(self, buf):
    if isinstance(buf, unicode):
        buf = buf.encode('utf-8')
    BytesIO.write(self, buf)


# using types.StringTypes
if isinstance(value, types.StringTypes):
    ...

After:

import six

# fixed unicode
def write(self, buf):
    if isinstance(buf, six.text_type):
        buf = buf.encode('utf-8')
    BytesIO.write(self, buf)


# fixed types.StringTypes
if isinstance(value, six.string_types):
    ...

subprocess.check_output()

Python 2.7 code will often call subprocess.check_output() and then do string manipulation on the output, such as str.splitlines(). The code will fail under Python 3 because subprocess.check_output() now returns a binary data stream. However we can use the universal_newlines=True keyword argument to get back a text type object in both Python 2 and Python 3.

Before:

output = subprocess.check_output([callargs])
lines = output.splitlines()  # Broken in Python 3

After:

output = subprocess.check_output([callargs], universal_newlines=True)
lines = output.splitlines()

See the subprocess documentation for details.

Porting Resources

Porting Guide on python.org

Cheat Sheet: Writing Python 2-3 compatible code. Note: misses some advanced changes, like porting __eq__() to __hash__() and changes to the slice interface.

References

Bug 1496527 tracks the migration

https://ahal.ca/blog/2019/python-3-at-mozilla

Python 3 tree migration notes

mach Python 3 migration roadmap