Firefox/Python 3 Migration: Difference between revisions
(→How to Port the Code: added porting examples with six) |
(→Porting examples: add an example of fixing subprocess.check_output() binary data errors under Py3) |
||
Line 159: | Line 159: | ||
... | ... | ||
</pre> | </pre> | ||
==== <code>subprocess.check_output()</code> ==== | |||
Python 2.7 code will often call <code>subprocess.check_output()</code> and then do string manipulation on the output, such as <code>str.splitlines()</code>. The code will fail under Python 3 because <code>subprocess.check_output()</code> now returns a binary data stream. However we can use the <code>universal_newlines=True</code> keyword argument to get back a text type object in both Python 2 and Python 3. | |||
Before: | |||
<pre> | |||
output = subprocess.check_output([callargs]) | |||
lines = output.splitlines() # Broken in Python 3 | |||
</pre> | |||
After: | |||
<pre> | |||
output = subprocess.check_output([callargs], universal_newlines=True) | |||
lines = output.splitlines() | |||
</pre> | |||
See the [https://docs.python.org/3.6/library/subprocess.html#frequently-used-arguments subprocess documentation] for details. | |||
=== Porting Resources === | === Porting Resources === |
Revision as of 14:00, 18 July 2019
Why This is Important
In mozilla-central there are over 3500 Python files (excluding third party files), comprising roughly 230k lines of code. Most of it is Python 2.[1]
Python 2 will reach it's End Of Life on January 1st 2020. This page collects links and resources to assist the migration of the Firefox ecosystem off of Python 2 and onto Python 3.
How We Get There
As of today (May 2019) we require both the Python 2 and Python 3 interpreters to be installed to build and develop Firefox.
Next we:
- Make Python 3 porting and development safe by adding a Python 3 option to the developer toolchain. (mach lint, coverage.py, parts of the test suite)
- Make key support libraries run in both Python 2 and Python 3. (mach-core)
- Build interpreter switching mechanisms into the toolchain so we can run sub-components in just 2 or just 3. (mach sub-commands, py_action)
- Port Python 2-only components to Python 3 piecemeal.
- Remove Python 2 compatibility. (When that might happen needs discussion. Possibly sometime in 2020.)
Getting Involved
Find the others
Most discussion happens in the #python channel on Slack (NDA'd Mozillians only).
What you'll need
You will need a working Firefox build environment.
Where to start:
Remove more of the "excluded" paths from the py2 and py3 linters
See this Etherpad for instructions.
Roughly:
- Pick a directory from the list, and put your name beside it so we know it's ported
- Edit py2.yml and/or py3.yml and remove that directory from the exclusion list
- Run `./mach lint -l py2 -l py3 <your directory name>` to get a list of errors
- To get a list of files that need fixing:
./mach lint -l py2 -l py3 <your directory name> --format unix > tofix.txt
- To install `futurize`:
pip install --user future
- To run through futurize
cat tofix.txt | cut -f1 -d: | sort -u | xargs futurize -1 -w
- Visually inspect the patch, especially around `except:` clauses. Watch out for "xxx_todo_changeme" in the patch. These are places where futurize couldn't port the code and you need to fix it manually.
When your patch is ready file:
- Use moz-phab for submitting the patch - https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#submitting-patches
- Use bug 1559975
- Set one of callek, ahal, or catlee as the code reviewer
Run all python source tests with Python 3 and see what breaks
Make sure each of the tasks listed in https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/python.yml has Python 3 specified, and do a try push, enabling all the python source tests.
See what breaks, then fix it!
Help port mach-core to Python 3 (bug 1473498)
- Edit python/mach/mach/test/python.ini to remove the skip-if
- Run: ./mach python-test --python 3 python/mach
- Fix the errors until all tests pass
See the #References for links to other ideas and tracking bugs.
How to Port the Code
Before starting
If you haven't ported modules before (or need a refresher), read the Porting Guide on python.org. It has a good outline of the porting process.
Porting modules in the Firefox source tree
- We've already standardized on Python 2.7 across the tree.
- Make sure you can test the code you're working on with python3!
- If using `tox`, then adding `py35`, and `py36` to the list of environments should be sufficient.
- If using |mach python-test|, make sure the relevant manifest doesn't skip-if Python 3 and run |mach python-test --python 3 <path>|.
- `futurize` Stage 1 fixes can be used to transform code into code that's compatible with both python2.7 and python3.
- The `six` module is available in-tree, feel free to use it as a dependency.
- Black can format Python 2 code and Python 3 code (but it requires Py3.6+ to run).
Porting examples
Prefer six
when possible. Make use of six.PY2
, six.PY3
, six.moves
, six.string_types
, six.integer_types
, etc. This make it easy to find the py2 code again so we can remove it.
import
statements
Use the six.moves
module:
Before:
import __builtin__ __builtin__.__import__ = myimporthook()
After:
from six.moves import builtins builtins.__import__ = myimporthook()
if
statements
Use six.PY2
and six.PY3
:
import six if six.PY2: # py2 stuff else: # py3 stuff
If six isn't available or you are in the imports section of the module then this is fine too:
if sys.version_info < (3,): # py2 stuff else: # py3 stuff
String data
Use six.string_types
, six.text_type
, six.binary_type
, etc. Also see the six
docs for "Binary and text data":
Before:
# using unicode def write(self, buf): if isinstance(buf, unicode): buf = buf.encode('utf-8') BytesIO.write(self, buf) # using types.StringTypes if isinstance(value, types.StringTypes): ...
After:
import six # fixed unicode def write(self, buf): if isinstance(buf, six.text_type): buf = buf.encode('utf-8') BytesIO.write(self, buf) # fixed types.StringTypes if isinstance(value, six.string_types): ...
subprocess.check_output()
Python 2.7 code will often call subprocess.check_output()
and then do string manipulation on the output, such as str.splitlines()
. The code will fail under Python 3 because subprocess.check_output()
now returns a binary data stream. However we can use the universal_newlines=True
keyword argument to get back a text type object in both Python 2 and Python 3.
Before:
output = subprocess.check_output([callargs]) lines = output.splitlines() # Broken in Python 3
After:
output = subprocess.check_output([callargs], universal_newlines=True) lines = output.splitlines()
See the subprocess documentation for details.
Porting Resources
Cheat Sheet: Writing Python 2-3 compatible code. Note: misses some advanced changes, like porting __eq__() to __hash__() and changes to the slice interface.
References
Bug 1496527 tracks the migration