changeset 396:f3472dd25d9e

already at http://k0s.org/portfolio/python/pypi.html
author Jeff Hammel <jhammel@mozilla.com>
date Mon, 29 Jul 2013 22:44:30 -0700
parents 28ca88607682
children 11ac996b9c49
files python/cheeseshop.txt
diffstat 1 files changed, 0 insertions(+), 103 deletions(-) [+]
line wrap: on
line diff
--- a/python/cheeseshop.txt	Mon Jul 29 22:15:23 2013 -0700
+++ /dev/null	Thu Jan 01 00:00:00 1970 +0000
@@ -1,103 +0,0 @@
-Making your own cheeseshop
-
-It is a common misconception that the cheeseshop (pypi.python.org) is
-*the* place to go for python packages.  This is only true by
-convention.  I won't go into the history of
-distutils/setuptools/distribute/pip packaging fiasco.  Suffice it to
-say that (I hope) things are slowly converging and that both pip and
-easy_install, the two major ways of installing python software, both
-support a -i option to specify the URL of the package index (which is,
-by default http://pypi.python.org/simple).
-
-In its base form, easy_install and pip just crawl links.  You look at
-the base URL (see above) /<package>/<package>-<version>-<extension>
-and download this, unzip, run `python setup.py install` on it and
-you're done.  So if you want to make a cheeseshop, there are two
-essential tasks:
-
- 1. Generating e.g. tarballs for a package and all of its dependencies
- 2. Putting these tarballs on the web with some appropriate parent
-    directory
-
-Not rocket science...barely computer science, really.
-
-For generating packages and their dependencies, I used pip.  pip is
-really great for this.  I only used the command line interface, though
-if I was smarter, I probably should have looked at the API and figured
-out what pip is doing internally and I could have avoided a few
-steps.  Basically, using the --no-install option downloads the package
-and its dependencies for you and lets you do what you want with it. 
-
-I made a program for this, see http://k0s.org/hg/stampit . It's a
-python package, but it doesn't really do anything python-y.  It was
-just easier to write than a shell script for my purposes.  Basically
-it makes a virtualenv (probably overkill already), downloads the
-packages and their dependencies into it, runs `python setup.py sdist`
-on each package so that you have a source distribution, and prints out
-the location of each tarball.
-
-The source distribution is very important as we want packages that
-will work independent of platform.  These should.  If they don't, we
-can make them.
-
-So problem #1 solved.  Let's move on to problem #2: putting them
-somewhere on the web.
-
-Mozilla is so kind as to have given me a URL space on
-people.mozilla.org. Since easy_install and pip are really dumb and
-basically just crawl links, and since Apache is smart enough to
-generate index pages for directories that don't have index.html files
-in them, the hard part is already solved.  I will note that
-people.mozilla.org is not intended as a permanant place for these
-tarballs, just an interim instance until we decide where we really
-want to put them.
-
-Since I like to write scripts, I wrote a script that will run stampit
-and copy the resulting tarballs to a place appropriate to a
-cheeseshop.  You can see the code here:
-
-http://k0s.org/mozilla/package-it.sh
-
-The variables are pretty specialized to my setup, but of course that's fixable.
-
-Does it really work?
-
-Yes!  You can try it for yourself. Try:
-
-``easy_install -i http://people.mozilla.org/~jhammel/packages/ mozmill``
-
-Watch where the links come from.  Surprise!  They're all from
-http://people.mozilla.org/~jhammel/packages/ !
-I would *highly advise* doing this (and just about everything else in
-python) in a virtualenv so that you don't pollute your global
-site-packages.
-
-Why am I doing this?
-
-The Firefox buildslaves are supposed to fetch data only from mozilla
-URLs for stability.  So, if python packages need to be installed, they
-need to be available internal to Mozilla.  If a package didn't have
-dependencies, then this is a no-brainer.  But packages do have
-dependencies. Mozmill depends jsbridge, simplejson, and mozrunner.
-While this is a lot of work for just one package, if we want more
-python stuff in our buildbot tests, we'll need to do more of this, and
-I'd rather have a good solid methodology to do so.  I also imagine
-this growing as a place to put all of our python packages for internal
-Mozilla needs.
-
-I will note that I did this in a few hours from basically knowing the
-problem space but never having actually done it.  None of this is
-supposed to be a clean and polished solution.  But really, its not
-bad.  We did something similar but less functional at my last job, The
-Open Planning Project, for similar reasons, so its not like I tackled
-this blindly. This is not as fully functional as the cheeseshop.  A maintainer
-needs to run the package-it.sh script for each package (and its deps)
-they want installed.  There's no accounts or any of the other features
-the cheeseshop has.  But for a simple prototype and a way to move the
-discussion forward, its actually not that bad of a solution.  There
-are more robust ways of really doing the cheeseshop, such as
-http://github.com/ask/chishop , but for a package dumping ground, this
-solution works and its really not even that hacky (in my opinion
-anyway).
-
-