|
82
|
1 Internal Python Package Index for Build Infrastructure
|
|
|
2
|
|
|
3 In working on https://bugzilla.mozilla.org/show_bug.cgi?id=701506 ,
|
|
|
4 create python package webserver,
|
|
|
5 I have explored several different solutions. In short there are
|
|
|
6 many different python packages available for doing such a thing, none
|
|
|
7 of which I found completely appropriate to our needs.
|
|
|
8
|
|
|
9 My goals for this project are:
|
|
|
10
|
|
|
11 - get the build infrastructure able to depend on python packages (vs
|
|
|
12 having each package having to be either deployed to each slave or
|
|
|
13 having to bundle them a la
|
|
|
14 http://hg.mozilla.org/build/talos/file/8197dc094fe3/create_talos_zip.py
|
|
|
15 which does not scale)
|
|
|
16
|
|
|
17 - being able to declare python dependencies in the conventional way:
|
|
|
18 you declare ``install_requires`` as part of the ``setup`` function
|
|
|
19 and e.g. ``python setup.py develop`` should grab the dependencies
|
|
|
20 and versions it needs from a package index
|
|
|
21
|
|
|
22 - avoid hitting outside networks. Tests should not fail because
|
|
|
23 e.g. pypi.python.org is down. We should not depend on this or any
|
|
|
24 other networks being fast or up at all.
|
|
|
25
|
|
|
26 - being easy to upload new versions
|
|
|
27
|
|
|
28 - being easy to maintain
|
|
|
29
|
|
|
30 To this end, I wrote simpypi, as detailed in:
|
|
|
31 http://pypi.python.org/pypi/simpypi
|
|
|
32
|
|
|
33 Since it is likely that whatever is done for a first iteration will
|
|
|
34 not prove the solution we want to go with down the line, and instead
|
|
|
35 will be more of a talking point for developing a long-term solution, I
|
|
|
36 have decided to make the initial version of simpypi as simple as
|
|
|
37 possible. To this end, I made the following architecture choices:
|
|
|
38
|
|
|
39 - The simpypi GET view is just a file upload widget in a static HTML
|
|
|
40 form. We may want to add more to this page, such as a listing of
|
|
|
41 packages.
|
|
|
42
|
|
|
43 - simpypi currently does nothing to get packages from pypi. Whether
|
|
|
44 it should or not depends on our needs.
|
|
|
45
|
|
|
46 - there is no authentication. While the simple index may be served
|
|
|
47 within and outside of the build system without security impact, I
|
|
|
48 have been laboring under the assumption that file upload will be
|
|
|
49 protected by VPN. Other auth methods could also be considered
|
|
|
50
|
|
|
51 Other issues are described in the documentation:
|
|
|
52 http://k0s.org/mozilla/hg/simpypi/file/tip/README.txt
|
|
|
53
|
|
|
54 In crafting simpypi, I've realized that the simple index and the
|
|
|
55 upload mechanisms are actually uncoupled: the former serves a package index
|
|
|
56 for installation, the latter takes an upload an puts it in an
|
|
|
57 appropriate place in a directory. This uncoupling gives significant
|
|
|
58 flexibility with respect to deployment or development. For instance,
|
|
|
59 the simpypi piece can be swapped out as long as the simple index
|
|
|
60 directory server continues to work.
|
|
|
61
|
|
|
62 I initially (and still, to a lesser degree, continue) to investigate
|
|
|
63 https://github.com/SurveyMonkey/CheesePrism . CheesePrism is a
|
|
|
64 heavier solution (although, compared to my survey of existing python
|
|
|
65 package index solutions, it is not that heavy) that centers on taking
|
|
|
66 packages from pypi.python.org for population. As best I can tell,
|
|
|
67 this is not really what we want from a Mozilla package server: not
|
|
|
68 all of the packages we want or need are on pypi.python.org and the
|
|
|
69 workflow proscribed by such a solution is probably undesirable to us.
|
|
|
70 I initially hoped to add more options to CheesePrism, but bug fixes
|
|
|
71 and turnarounds have been slow.
|
|
|
72
|
|
|
73 You can see active simpypi at http://k0s.org:8080 for the upload page
|
|
|
74 and http://k0s.org:8080/index/ for the index page. I have uploaded
|
|
|
75 mozbase and a few other packages as a demonstration. If you want to
|
|
|
76 test, deploy a new virtualenv and run e.g.
|
|
|
77
|
|
|
78 easy_install -i http://k0s.org:8080/index/ mozrunner
|
|
|
79
|
|
|
80 Note that the packages come from k0s.org:8080 .
|
|
|
81
|
|
|
82 You can see an active CheesePrism instance at http://k0s.org:6543/
|
|
|
83
|
|
|
84 Note that this is a POC and is intended as a talking point more than a
|
|
|
85 final solution. A basic package index can be realized using tarballs
|
|
|
86 served with a static fileserver, but we need to have a machine to do
|
|
|
87 it on. We should also figure out our network needs: the package
|
|
|
88 index must be usable via the build infrastructure, but can also be
|
|
|
89 publicly available. The web UI for uploading packages, be it simpypi
|
|
|
90 or other, should be behind a VPN. The build infrastructure needs to
|
|
|
91 drastically begin to change to start installing dependencies from this
|
|
|
92 system vs. what we do now (which is largely work around the lack of a
|
|
|
93 package index).
|
|
|
94
|
|
|
95 We need to figure out what we want to do and drive this effort
|
|
|
96 forward. I can work on deployment issues if we come up with a system
|
|
|
97 that I am comfortable administrating and have a computer to put it
|
|
|
98 on, though I'm not necessarily ideal for the job. The web UI for
|
|
|
99 uploading packages should be worked through -- I give the simplest
|
|
|
100 possible model, though it can no doubt be improved. That said, the
|
|
|
101 web UI is not necessary for serving packages now, though a computer
|
|
|
102 and a static fileserver is.
|
