Mercurial > hg > config

annotate python/cheeseshop.txt @ 346:7eaaf99e3182

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
dammit
author Jeff Hammel <jhammel@mozilla.com>
date Mon, 24 Jun 2013 16:06:24 -0700
parents a599feff7c93
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
79
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
1 Making your own cheeseshop
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
2
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
3 It is a common misconception that the cheeseshop (pypi.python.org) is
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
4 *the* place to go for python packages. This is only true by
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
5 convention. I won't go into the history of
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
6 distutils/setuptools/distribute/pip packaging fiasco. Suffice it to
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
7 say that (I hope) things are slowly converging and that both pip and
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
8 easy_install, the two major ways of installing python software, both
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
9 support a -i option to specify the URL of the package index (which is,
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
10 by default http://pypi.python.org/simple).
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
11
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
12 In its base form, easy_install and pip just crawl links. You look at
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
13 the base URL (see above) /<package>/<package>-<version>-<extension>
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
14 and download this, unzip, run `python setup.py install` on it and
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
15 you're done. So if you want to make a cheeseshop, there are two
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
16 essential tasks:
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
17
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
18 1. Generating e.g. tarballs for a package and all of its dependencies
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
19 2. Putting these tarballs on the web with some appropriate parent
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
20 directory
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
21
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
22 Not rocket science...barely computer science, really.
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
23
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
24 For generating packages and their dependencies, I used pip. pip is
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
25 really great for this. I only used the command line interface, though
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
26 if I was smarter, I probably should have looked at the API and figured
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
27 out what pip is doing internally and I could have avoided a few
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
28 steps. Basically, using the --no-install option downloads the package
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
29 and its dependencies for you and lets you do what you want with it.
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
30
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
31 I made a program for this, see http://k0s.org/hg/stampit . It's a
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
32 python package, but it doesn't really do anything python-y. It was
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
33 just easier to write than a shell script for my purposes. Basically
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
34 it makes a virtualenv (probably overkill already), downloads the
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
35 packages and their dependencies into it, runs `python setup.py sdist`
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
36 on each package so that you have a source distribution, and prints out
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
37 the location of each tarball.
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
38
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
39 The source distribution is very important as we want packages that
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
40 will work independent of platform. These should. If they don't, we
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
41 can make them.
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
42
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
43 So problem #1 solved. Let's move on to problem #2: putting them
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
44 somewhere on the web.
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
45
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
46 Mozilla is so kind as to have given me a URL space on
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
47 people.mozilla.org. Since easy_install and pip are really dumb and
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
48 basically just crawl links, and since Apache is smart enough to
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
49 generate index pages for directories that don't have index.html files
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
50 in them, the hard part is already solved. I will note that
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
51 people.mozilla.org is not intended as a permanant place for these
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
52 tarballs, just an interim instance until we decide where we really
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
53 want to put them.
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
54
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
55 Since I like to write scripts, I wrote a script that will run stampit
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
56 and copy the resulting tarballs to a place appropriate to a
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
57 cheeseshop. You can see the code here:
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
58
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
59 http://k0s.org/mozilla/package-it.sh
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
60
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
61 The variables are pretty specialized to my setup, but of course that's fixable.
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
62
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
63 Does it really work?
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
64
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
65 Yes! You can try it for yourself. Try:
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
66
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
67 ``easy_install -i http://people.mozilla.org/~jhammel/packages/ mozmill``
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
68
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
69 Watch where the links come from. Surprise! They're all from
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
70 http://people.mozilla.org/~jhammel/packages/ !
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
71 I would *highly advise* doing this (and just about everything else in
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
72 python) in a virtualenv so that you don't pollute your global
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
73 site-packages.
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
74
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
75 Why am I doing this?
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
76
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
77 The Firefox buildslaves are supposed to fetch data only from mozilla
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
78 URLs for stability. So, if python packages need to be installed, they
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
79 need to be available internal to Mozilla. If a package didn't have
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
80 dependencies, then this is a no-brainer. But packages do have
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
81 dependencies. Mozmill depends jsbridge, simplejson, and mozrunner.
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
82 While this is a lot of work for just one package, if we want more
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
83 python stuff in our buildbot tests, we'll need to do more of this, and
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
84 I'd rather have a good solid methodology to do so. I also imagine
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
85 this growing as a place to put all of our python packages for internal
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
86 Mozilla needs.
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
87
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
88 I will note that I did this in a few hours from basically knowing the
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
89 problem space but never having actually done it. None of this is
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
90 supposed to be a clean and polished solution. But really, its not
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
91 bad. We did something similar but less functional at my last job, The
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
92 Open Planning Project, for similar reasons, so its not like I tackled
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
93 this blindly. This is not as fully functional as the cheeseshop. A maintainer
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
94 needs to run the package-it.sh script for each package (and its deps)
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
95 they want installed. There's no accounts or any of the other features
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
96 the cheeseshop has. But for a simple prototype and a way to move the
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
97 discussion forward, its actually not that bad of a solution. There
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
98 are more robust ways of really doing the cheeseshop, such as
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
99 http://github.com/ask/chishop , but for a package dumping ground, this
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
100 solution works and its really not even that hacky (in my opinion
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
101 anyway).
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
102
a599feff7c93 moving the cheeseshop document here; not the best place but it will work for now
Jeff Hammel <jhammel@mozilla.com>
parents:
diff changeset
103