Mercurial > hg > config
comparison python/cheeseshop.txt @ 79:a599feff7c93
moving the cheeseshop document here; not the best place but it will work for now
| author | Jeff Hammel <jhammel@mozilla.com> |
|---|---|
| date | Mon, 21 Jun 2010 13:16:24 -0700 |
| parents | |
| children |
comparison
equal
deleted
inserted
replaced
| 76:84ffa220796a | 79:a599feff7c93 |
|---|---|
| 1 Making your own cheeseshop | |
| 2 | |
| 3 It is a common misconception that the cheeseshop (pypi.python.org) is | |
| 4 *the* place to go for python packages. This is only true by | |
| 5 convention. I won't go into the history of | |
| 6 distutils/setuptools/distribute/pip packaging fiasco. Suffice it to | |
| 7 say that (I hope) things are slowly converging and that both pip and | |
| 8 easy_install, the two major ways of installing python software, both | |
| 9 support a -i option to specify the URL of the package index (which is, | |
| 10 by default http://pypi.python.org/simple). | |
| 11 | |
| 12 In its base form, easy_install and pip just crawl links. You look at | |
| 13 the base URL (see above) /<package>/<package>-<version>-<extension> | |
| 14 and download this, unzip, run `python setup.py install` on it and | |
| 15 you're done. So if you want to make a cheeseshop, there are two | |
| 16 essential tasks: | |
| 17 | |
| 18 1. Generating e.g. tarballs for a package and all of its dependencies | |
| 19 2. Putting these tarballs on the web with some appropriate parent | |
| 20 directory | |
| 21 | |
| 22 Not rocket science...barely computer science, really. | |
| 23 | |
| 24 For generating packages and their dependencies, I used pip. pip is | |
| 25 really great for this. I only used the command line interface, though | |
| 26 if I was smarter, I probably should have looked at the API and figured | |
| 27 out what pip is doing internally and I could have avoided a few | |
| 28 steps. Basically, using the --no-install option downloads the package | |
| 29 and its dependencies for you and lets you do what you want with it. | |
| 30 | |
| 31 I made a program for this, see http://k0s.org/hg/stampit . It's a | |
| 32 python package, but it doesn't really do anything python-y. It was | |
| 33 just easier to write than a shell script for my purposes. Basically | |
| 34 it makes a virtualenv (probably overkill already), downloads the | |
| 35 packages and their dependencies into it, runs `python setup.py sdist` | |
| 36 on each package so that you have a source distribution, and prints out | |
| 37 the location of each tarball. | |
| 38 | |
| 39 The source distribution is very important as we want packages that | |
| 40 will work independent of platform. These should. If they don't, we | |
| 41 can make them. | |
| 42 | |
| 43 So problem #1 solved. Let's move on to problem #2: putting them | |
| 44 somewhere on the web. | |
| 45 | |
| 46 Mozilla is so kind as to have given me a URL space on | |
| 47 people.mozilla.org. Since easy_install and pip are really dumb and | |
| 48 basically just crawl links, and since Apache is smart enough to | |
| 49 generate index pages for directories that don't have index.html files | |
| 50 in them, the hard part is already solved. I will note that | |
| 51 people.mozilla.org is not intended as a permanant place for these | |
| 52 tarballs, just an interim instance until we decide where we really | |
| 53 want to put them. | |
| 54 | |
| 55 Since I like to write scripts, I wrote a script that will run stampit | |
| 56 and copy the resulting tarballs to a place appropriate to a | |
| 57 cheeseshop. You can see the code here: | |
| 58 | |
| 59 http://k0s.org/mozilla/package-it.sh | |
| 60 | |
| 61 The variables are pretty specialized to my setup, but of course that's fixable. | |
| 62 | |
| 63 Does it really work? | |
| 64 | |
| 65 Yes! You can try it for yourself. Try: | |
| 66 | |
| 67 ``easy_install -i http://people.mozilla.org/~jhammel/packages/ mozmill`` | |
| 68 | |
| 69 Watch where the links come from. Surprise! They're all from | |
| 70 http://people.mozilla.org/~jhammel/packages/ ! | |
| 71 I would *highly advise* doing this (and just about everything else in | |
| 72 python) in a virtualenv so that you don't pollute your global | |
| 73 site-packages. | |
| 74 | |
| 75 Why am I doing this? | |
| 76 | |
| 77 The Firefox buildslaves are supposed to fetch data only from mozilla | |
| 78 URLs for stability. So, if python packages need to be installed, they | |
| 79 need to be available internal to Mozilla. If a package didn't have | |
| 80 dependencies, then this is a no-brainer. But packages do have | |
| 81 dependencies. Mozmill depends jsbridge, simplejson, and mozrunner. | |
| 82 While this is a lot of work for just one package, if we want more | |
| 83 python stuff in our buildbot tests, we'll need to do more of this, and | |
| 84 I'd rather have a good solid methodology to do so. I also imagine | |
| 85 this growing as a place to put all of our python packages for internal | |
| 86 Mozilla needs. | |
| 87 | |
| 88 I will note that I did this in a few hours from basically knowing the | |
| 89 problem space but never having actually done it. None of this is | |
| 90 supposed to be a clean and polished solution. But really, its not | |
| 91 bad. We did something similar but less functional at my last job, The | |
| 92 Open Planning Project, for similar reasons, so its not like I tackled | |
| 93 this blindly. This is not as fully functional as the cheeseshop. A maintainer | |
| 94 needs to run the package-it.sh script for each package (and its deps) | |
| 95 they want installed. There's no accounts or any of the other features | |
| 96 the cheeseshop has. But for a simple prototype and a way to move the | |
| 97 discussion forward, its actually not that bad of a solution. There | |
| 98 are more robust ways of really doing the cheeseshop, such as | |
| 99 http://github.com/ask/chishop , but for a package dumping ground, this | |
| 100 solution works and its really not even that hacky (in my opinion | |
| 101 anyway). | |
| 102 | |
| 103 |