Making Our Python Reusable
Motivation
Mozilla has a LOT of code for harnesses and tools, the majority
of it is duplicated code and
we
are going to do something about it!
- Mozilla has a wealth of
python tools
- We should make these tools modular, portable, flexible, and robust
- By curating a high quality ecosystem of tools, we make life better
for engineers and the community
Goals
- Make people aware of what tools we have and how they fit together
- Move towards ecosystem cultivation and tool reuse
- Solicit feedback on potential solutions
Python in Mozilla-Central
Our test harnesses
(xpcshell ,
reftest,
mochitest , etc.)
and build tools in
mozilla-central
have significant python pieces.
- this monolith makes it difficult to develop and share componentized tools
- python importing is ad hoc and not adaptable to other environments
- modifying sys.path
- pythonpath.py
- we avoid and rewrite quality third-party packages instead of
depending on them
Other Python at Mozilla
Many other harnesses
(mozmill,
talos ,
jetpack ,
firebug ,
speedtests,
...)
and tools
(buildbot ,
pulse ,
OrangeFactor , ...)
are an integral part of Mozilla's infrastructure
- All these do similar things, but do not reuse code
- These depend on more 3rd party resources than m-c does
- These are not required to be in m-c
Python in the Wild
- 3rd party packages: There are great modules out there.
Finding a way to access these and contribute to these
will foster a better environment for our tools.
- newer versions of python have features we can use (e.g. json
in 2.6); however, we have many contraints in our ecosystem wrt
upgrading python
- pypi: test slaves and the m-c workflow have constraints on network access
Proposed Changes in m-c
- make a virtualenv in
the to isolate the python environment
- rewrite the harness pieces
(automation.py ,
etc) using high quality modular components (e.g.
mozprofile ,
mozprocess )
- mirror the modular components to mozilla-central as they need
upgrading
- bundle virtualenv and 3rd party packages to the build prereqs and
the slaves
From Wonky to Reusable
- new projects can use shared tools from day 1, and be
integrated in with little work
- workflow on m-c will be the same: make mochitest, etc., just works
- instead of code being copy+pasted and disparately maintained,
maintainence will be centralized
- moving towards
mozharness +
mozbase
will simplify the buildbot infrastructure
Open Questions
There are known issues in python packaging with no singular
solution. What is the long term strategy? What should we do in the interim?
- What library to use to package? (e.g. dependencies?)
- How to find what other people are working on and have done already?
- How to automate packaging and deployment?
Upstreaming
We're a big consumer of python. We need to work on giving back to the
community. This helps us, and it helps python. If our needs aren't
known or addressed, they can't be fixed.
PACKAGE YOUR PACKAGES!
If it's generally useful, help others use it!