Mercurial > hg > simpypi

comparison simpypi.txt @ 82:f974c6d79804 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
lost and found
author Jeff Hammel <k0scist@gmail.com>
date Tue, 19 May 2015 21:02:59 -0700
parents
children
comparison
equal deleted inserted replaced
81:c61d0fe16509 82:f974c6d79804
1 Internal Python Package Index for Build Infrastructure
2
3 In working on https://bugzilla.mozilla.org/show_bug.cgi?id=701506 ,
4 create python package webserver,
5 I have explored several different solutions. In short there are
6 many different python packages available for doing such a thing, none
7 of which I found completely appropriate to our needs.
8
9 My goals for this project are:
10
11 - get the build infrastructure able to depend on python packages (vs
12 having each package having to be either deployed to each slave or
13 having to bundle them a la
14 http://hg.mozilla.org/build/talos/file/8197dc094fe3/create_talos_zip.py
15 which does not scale)
16
17 - being able to declare python dependencies in the conventional way:
18 you declare ``install_requires`` as part of the ``setup`` function
19 and e.g. ``python setup.py develop`` should grab the dependencies
20 and versions it needs from a package index
21
22 - avoid hitting outside networks. Tests should not fail because
23 e.g. pypi.python.org is down. We should not depend on this or any
24 other networks being fast or up at all.
25
26 - being easy to upload new versions
27
28 - being easy to maintain
29
30 To this end, I wrote simpypi, as detailed in:
31 http://pypi.python.org/pypi/simpypi
32
33 Since it is likely that whatever is done for a first iteration will
34 not prove the solution we want to go with down the line, and instead
35 will be more of a talking point for developing a long-term solution, I
36 have decided to make the initial version of simpypi as simple as
37 possible. To this end, I made the following architecture choices:
38
39 - The simpypi GET view is just a file upload widget in a static HTML
40 form. We may want to add more to this page, such as a listing of
41 packages.
42
43 - simpypi currently does nothing to get packages from pypi. Whether
44 it should or not depends on our needs.
45
46 - there is no authentication. While the simple index may be served
47 within and outside of the build system without security impact, I
48 have been laboring under the assumption that file upload will be
49 protected by VPN. Other auth methods could also be considered
50
51 Other issues are described in the documentation:
52 http://k0s.org/mozilla/hg/simpypi/file/tip/README.txt
53
54 In crafting simpypi, I've realized that the simple index and the
55 upload mechanisms are actually uncoupled: the former serves a package index
56 for installation, the latter takes an upload an puts it in an
57 appropriate place in a directory. This uncoupling gives significant
58 flexibility with respect to deployment or development. For instance,
59 the simpypi piece can be swapped out as long as the simple index
60 directory server continues to work.
61
62 I initially (and still, to a lesser degree, continue) to investigate
63 https://github.com/SurveyMonkey/CheesePrism . CheesePrism is a
64 heavier solution (although, compared to my survey of existing python
65 package index solutions, it is not that heavy) that centers on taking
66 packages from pypi.python.org for population. As best I can tell,
67 this is not really what we want from a Mozilla package server: not
68 all of the packages we want or need are on pypi.python.org and the
69 workflow proscribed by such a solution is probably undesirable to us.
70 I initially hoped to add more options to CheesePrism, but bug fixes
71 and turnarounds have been slow.
72
73 You can see active simpypi at http://k0s.org:8080 for the upload page
74 and http://k0s.org:8080/index/ for the index page. I have uploaded
75 mozbase and a few other packages as a demonstration. If you want to
76 test, deploy a new virtualenv and run e.g.
77
78 easy_install -i http://k0s.org:8080/index/ mozrunner
79
80 Note that the packages come from k0s.org:8080 .
81
82 You can see an active CheesePrism instance at http://k0s.org:6543/
83
84 Note that this is a POC and is intended as a talking point more than a
85 final solution. A basic package index can be realized using tarballs
86 served with a static fileserver, but we need to have a machine to do
87 it on. We should also figure out our network needs: the package
88 index must be usable via the build infrastructure, but can also be
89 publicly available. The web UI for uploading packages, be it simpypi
90 or other, should be behind a VPN. The build infrastructure needs to
91 drastically begin to change to start installing dependencies from this
92 system vs. what we do now (which is largely work around the lack of a
93 package index).
94
95 We need to figure out what we want to do and drive this effort
96 forward. I can work on deployment issues if we come up with a system
97 that I am comfortable administrating and have a computer to put it
98 on, though I'm not necessarily ideal for the job. The web UI for
99 uploading packages should be worked through -- I give the simplest
100 possible model, though it can no doubt be improved. That said, the
101 web UI is not necessary for serving packages now, though a computer
102 and a static fileserver is.