Firefox/Python 3 Migration
Why This is Important
In mozilla-central there are over 3500 Python files (excluding third party files), comprising roughly 230k lines of code. Most of it is Python 2.[1]
Python 2 will reach it's End Of Life on January 1st 2020. This page collects links and resources to assist the migration of the Firefox ecosystem off of Python 2 and onto Python 3.
How We Get There
As of today (May 2019) we require both the Python 2 and Python 3 interpreters to be installed to build and develop Firefox.
Next we:
- Make Python 3 porting and development safe by adding a Python 3 option to the developer toolchain. (mach lint, coverage.py, parts of the test suite)
- Make key support libraries run in both Python 2 and Python 3. (mach-core)
- Build interpreter switching mechanisms into the toolchain so we can run sub-components in just 2 or just 3. (mach sub-commands, py_action)
- Port Python 2-only components to Python 3 piecemeal.
- Remove Python 2 compatibility. (When that might happen needs discussion. Possibly sometime in 2020.)
Getting Involved
Find the others
Most discussion happens in the #py3 channel on Slack (NDA'd Mozillians only). You can also join #python on irc.mozilla.org.
What you'll need
You will need a working Firefox build environment.
Where to start:
Remove more of the "excluded" paths from the py2 and py3 linters
See this Etherpad for instructions.
Roughly:
- Pick a directory from the list, and put your name beside it so we know it's ported
- Edit py2.yml and/or py3.yml and remove that directory from the exclusion list
- Run `./mach lint -l py2 -l py3 <your directory name>` to get a list of errors
- To get a list of files that need fixing:
./mach lint -l py2 -l py3 <your directory name> --format unix > tofix.txt
- To install `futurize`:
pip install --user future
- To run through futurize
cat tofix.txt | cut -f1 -d: | sort -u | xargs futurize -1 -w
- Visually inspect the patch, especially around `except:` clauses. Watch out for "xxx_todo_changeme" in the patch. These are places where futurize couldn't port the code and you need to fix it manually.
When your patch is ready file:
- Use moz-phab for submitting the patch - https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#submitting-patches
- Use bug 1559975
- Set one of callek, ahal, or catlee as the code reviewer
Enable Python 3 on more Python unittests
Python unittests are run with `./mach python-test` and are (usually) defined in `python.ini` files. To get a test running against Python 3:
- Find a test that is skipped with Python 3, you can use this query to find these tests.
- Edit the manifest and remove the Python 3 skip-if annotation.
- Run the test with `./mach python-test --python 3 path/to/test`.
- Fix any errors and repeat the previous step.
- Once it is passing locally, test it out on try. If there is no Python 3 task running this manifest already, you'll need to add one by adding `3` to the `python-version` key in python.yml.
- Push the task to try on all platforms it is configured for.
Port a Mach Command
The mach driver (toplevel `mach` file) contains a whitelist of commands that still run with Python 2. Pick the command that you'd like to convert and then:
- Remove it from the whitelist
- Run: ./mach <command>
- See what breaks and fix it!
- Repeat with a variety of different flags and arguments
- Make sure any Python unittests running in CI have Python 3 enabled (see above)
How to Port the Code
Before starting
If you haven't ported modules before (or need a refresher), read the Porting Guide on python.org. It has a good outline of the porting process.
Porting modules in the Firefox source tree
- We've already standardized on Python 2.7 across the tree.
- Make sure you can test the code you're working on with python3!
- If using `tox`, then adding `py35`, and `py36` to the list of environments should be sufficient.
- If using |mach python-test|, make sure the relevant manifest doesn't skip-if Python 3 and run |mach python-test --python 3 <path>|.
- `futurize` Stage 1 fixes can be used to transform code into code that's compatible with both python2.7 and python3.
- The `six` module is available in-tree, feel free to use it as a dependency.
- Black can format Python 2 code and Python 3 code (but it requires Py3.6+ to run).
Porting examples
Prefer six
when possible. Make use of six.PY2
, six.PY3
, six.moves
, six.string_types
, six.integer_types
, etc. This make it easy to find the py2 code again so we can remove it.
import
statements
Use the six.moves
module:
Before:
import __builtin__ __builtin__.__import__ = myimporthook()
After:
from six.moves import builtins builtins.__import__ = myimporthook()
if
statements
Use six.PY2
and six.PY3
:
import six if six.PY2: # py2 stuff else: # py3 stuff
If six isn't available or you are in the imports section of the module then this is fine too:
if sys.version_info < (3,): # py2 stuff else: # py3 stuff
String data
Use six.string_types
, six.text_type
, six.binary_type
, etc. Also see the six
docs for "Binary and text data":
Before:
# using unicode def write(self, buf): if isinstance(buf, unicode): buf = buf.encode('utf-8') BytesIO.write(self, buf) # using types.StringTypes if isinstance(value, types.StringTypes): ...
After:
import six # fixed unicode def write(self, buf): if isinstance(buf, six.text_type): buf = buf.encode('utf-8') BytesIO.write(self, buf) # fixed types.StringTypes if isinstance(value, six.string_types): ...
subprocess.check_output()
Python 2.7 code will often call subprocess.check_output()
and then do string manipulation on the output, such as str.splitlines()
. The code will fail under Python 3 because subprocess.check_output()
now returns a binary data stream. However we can use the universal_newlines=True
keyword argument to get back a text type object in both Python 2 and Python 3.
Before:
output = subprocess.check_output([callargs]) lines = output.splitlines() # Broken in Python 3
After:
output = subprocess.check_output([callargs], universal_newlines=True) lines = output.splitlines()
See the subprocess documentation for details.
Porting Resources
Cheat Sheet: Writing Python 2-3 compatible code. Note: misses some advanced changes, like porting __eq__() to __hash__() and changes to the slice interface.
References
Bug 1496527 tracks the migration