# HG changeset patch # User Jeff Hammel # Date 1375163070 25200 # Node ID f3472dd25d9eb2a41e303afd5b4e9867eec30864 # Parent 28ca88607682e6c216ceead6fb29a47d76ad7236 already at http://k0s.org/portfolio/python/pypi.html diff -r 28ca88607682 -r f3472dd25d9e python/cheeseshop.txt --- a/python/cheeseshop.txt Mon Jul 29 22:15:23 2013 -0700 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,103 +0,0 @@ -Making your own cheeseshop - -It is a common misconception that the cheeseshop (pypi.python.org) is -*the* place to go for python packages. This is only true by -convention. I won't go into the history of -distutils/setuptools/distribute/pip packaging fiasco. Suffice it to -say that (I hope) things are slowly converging and that both pip and -easy_install, the two major ways of installing python software, both -support a -i option to specify the URL of the package index (which is, -by default http://pypi.python.org/simple). - -In its base form, easy_install and pip just crawl links. You look at -the base URL (see above) //-- -and download this, unzip, run `python setup.py install` on it and -you're done. So if you want to make a cheeseshop, there are two -essential tasks: - - 1. Generating e.g. tarballs for a package and all of its dependencies - 2. Putting these tarballs on the web with some appropriate parent - directory - -Not rocket science...barely computer science, really. - -For generating packages and their dependencies, I used pip. pip is -really great for this. I only used the command line interface, though -if I was smarter, I probably should have looked at the API and figured -out what pip is doing internally and I could have avoided a few -steps. Basically, using the --no-install option downloads the package -and its dependencies for you and lets you do what you want with it. - -I made a program for this, see http://k0s.org/hg/stampit . It's a -python package, but it doesn't really do anything python-y. It was -just easier to write than a shell script for my purposes. Basically -it makes a virtualenv (probably overkill already), downloads the -packages and their dependencies into it, runs `python setup.py sdist` -on each package so that you have a source distribution, and prints out -the location of each tarball. - -The source distribution is very important as we want packages that -will work independent of platform. These should. If they don't, we -can make them. - -So problem #1 solved. Let's move on to problem #2: putting them -somewhere on the web. - -Mozilla is so kind as to have given me a URL space on -people.mozilla.org. Since easy_install and pip are really dumb and -basically just crawl links, and since Apache is smart enough to -generate index pages for directories that don't have index.html files -in them, the hard part is already solved. I will note that -people.mozilla.org is not intended as a permanant place for these -tarballs, just an interim instance until we decide where we really -want to put them. - -Since I like to write scripts, I wrote a script that will run stampit -and copy the resulting tarballs to a place appropriate to a -cheeseshop. You can see the code here: - -http://k0s.org/mozilla/package-it.sh - -The variables are pretty specialized to my setup, but of course that's fixable. - -Does it really work? - -Yes! You can try it for yourself. Try: - -``easy_install -i http://people.mozilla.org/~jhammel/packages/ mozmill`` - -Watch where the links come from. Surprise! They're all from -http://people.mozilla.org/~jhammel/packages/ ! -I would *highly advise* doing this (and just about everything else in -python) in a virtualenv so that you don't pollute your global -site-packages. - -Why am I doing this? - -The Firefox buildslaves are supposed to fetch data only from mozilla -URLs for stability. So, if python packages need to be installed, they -need to be available internal to Mozilla. If a package didn't have -dependencies, then this is a no-brainer. But packages do have -dependencies. Mozmill depends jsbridge, simplejson, and mozrunner. -While this is a lot of work for just one package, if we want more -python stuff in our buildbot tests, we'll need to do more of this, and -I'd rather have a good solid methodology to do so. I also imagine -this growing as a place to put all of our python packages for internal -Mozilla needs. - -I will note that I did this in a few hours from basically knowing the -problem space but never having actually done it. None of this is -supposed to be a clean and polished solution. But really, its not -bad. We did something similar but less functional at my last job, The -Open Planning Project, for similar reasons, so its not like I tackled -this blindly. This is not as fully functional as the cheeseshop. A maintainer -needs to run the package-it.sh script for each package (and its deps) -they want installed. There's no accounts or any of the other features -the cheeseshop has. But for a simple prototype and a way to move the -discussion forward, its actually not that bad of a solution. There -are more robust ways of really doing the cheeseshop, such as -http://github.com/ask/chishop , but for a package dumping ground, this -solution works and its really not even that hacky (in my opinion -anyway). - -