Mercurial > hg > fetch
annotate README.txt @ 53:81fe0523a078
documentation
| author | Jeff Hammel <jhammel@mozilla.com> |
|---|---|
| date | Tue, 15 Nov 2011 16:17:55 -0800 |
| parents | 738d84b4de52 |
| children | 2dfdff7549b2 |
| rev | line source |
|---|---|
| 0 | 1 fetch |
| 20 | 2 ===== |
| 0 | 3 |
| 4 fetch stuff from the interwebs | |
| 5 | |
| 53 | 6 `fetch.py <http://k0s.org/mozilla/hg/fetch/raw-file/tip/fetch.py>`_ is |
| 7 a single-file python module bundled as a | |
| 8 `package <http://k0s.org/mozilla/hg/fetch/>`_ for easy installation | |
| 9 and python importing. The purpose of fetch is to mirror remote | |
| 10 resources (URLs) to a local filesystem in order to synchronize and | |
| 11 update dependencies that are desired to be mirrored in this way. | |
| 12 | |
| 20 | 13 |
| 0 | 14 Format |
| 15 ------ | |
| 16 | |
| 53 | 17 ``fetch`` fetches from a manifest of the format:: |
| 20 | 18 |
| 19 [URL] [Destination] [Type] | |
| 20 | |
| 21 A URL can contain a hash tag (e.g. http://example.com/foo#bar/fleem) | |
| 22 which is used to extract the subdirectories from a multi-directory | |
| 23 resource. | |
| 24 | |
| 53 | 25 The ``Type`` of the resource is used to dispatch to the included |
| 26 Fetchers that take care of fetching the object. | |
| 27 | |
| 28 Manifests are used so that a number of resources may be fetched from a | |
| 29 particular ``fetch`` run. | |
| 20 | 30 |
| 31 | |
| 32 Fetchers | |
| 33 -------- | |
| 34 | |
| 53 | 35 ``fetch`` includes several objects for fetching resources:: |
|
13
3fee8ecd1af8
restructure while we still just have one module
Jeff Hammel <jhammel@mozilla.com>
parents:
0
diff
changeset
|
36 |
| 20 | 37 file : fetch a single file |
| 38 tar : fetch and extract a tarball | |
| 39 hg : checkout a mercurial repository | |
| 40 git : checkout a git repository | |
| 41 | |
| 53 | 42 The ``file`` fetcher cannot have a hash tag subpath since it is a single |
| 20 | 43 resource. |
| 44 | |
| 53 | 45 Though ``fetch`` has a set of fetchers included, you can pass an |
| 46 arbitrary list into ``fetch.Fetch``'s constructor. | |
| 20 | 47 |
| 48 | |
| 49 Version Control | |
| 50 --------------- | |
| 51 | |
| 53 | 52 The ``hg`` and the ``git`` fetchers fetch from version control systems and |
| 53 have additional options. The only current option to the constructor | |
| 54 is ``export``, which is by default True. If ``export`` is True, then | |
| 55 the repository will be exported into a non-versioned structure. If a | |
| 56 subpath is specified with a ``#`` in the URL, the repository will also | |
| 57 be exported. | |
| 58 | |
| 59 | |
| 60 TODO | |
| 61 ---- | |
| 62 | |
| 63 * use `manifestparser <https://github.com/mozilla/mozbase/blob/master/manifestdestiny/manifestparser.py>`_ | |
| 64 ``.ini`` files versus another manifest | |
| 65 format: when I started work on ``fetch``, I thought a | |
| 66 domain-specific manifest would be a big win. But, now, maybe a | |
| 67 ``.ini`` type manifest looks about the same, and is something that | |
| 68 is already used. The switch internally wouldn't be that bad, but | |
| 69 if ``fetch.py`` is used as a single file, it cannot have | |
| 70 "traditional" python dependencies. Since ``manifestparser.py`` is | |
| 71 also a single file, and ``fetch`` is only usable with internet | |
| 72 access anyway, maybe the | |
| 73 `require <http://k0s.org/hg/config/file/tip/python/require.py>`_ | |
| 74 pattern could be used for this purpose | |
| 0 | 75 |
| 53 | 76 * clobber: generally, you will want the existing resource to be |
| 77 clobbered, avoiding renames regarding upstream dependencies | |
| 78 | |
| 79 * outputting only subpaths: often, you will not to fetch from the | |
| 80 whole manifest, only from certain subpaths of the manifest. You | |
| 81 should be able to output a subset of what is to be mirrored based | |
| 82 on destination subpaths. The CLI option ``--dest`` is intended for | |
| 83 this purpose but currently unused. | |
| 84 | |
| 85 * fetcher options: | |
| 86 | |
| 87 * note python 2.5+ specifics: | |
| 88 | |
| 89 Unsolved Problems | |
| 90 ----------------- | |
| 91 | |
| 92 A common story for ``fetch`` is mirroring files into a VCS repository | |
| 93 because the remote resources are needed as part of the repository and | |
| 94 there is no better way to retrieve and/or update them. However, what | |
| 95 do you do if these remote resources are altered? In an ideal | |
| 96 ecosystem, the fixes would be automatically triaged and triggered for | |
| 97 upstream inclusion, or the diffs from the upstream are kept in local | |
| 98 modifications (although vendor branches, etc, are more suitable for | |
| 99 the latter class of problems, and in general discouraged when a less | |
| 100 intrusive system of consuming upstream dependencies are available). | |
| 0 | 101 |
| 102 ---- | |
| 103 | |
| 104 Jeff Hammel | |
|
13
3fee8ecd1af8
restructure while we still just have one module
Jeff Hammel <jhammel@mozilla.com>
parents:
0
diff
changeset
|
105 http://k0s.org/mozilla/hg/fetch |