Mercurial > hg > fetch

comparison README.txt @ 53:81fe0523a078

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
documentation
author Jeff Hammel <jhammel@mozilla.com>
date Tue, 15 Nov 2011 16:17:55 -0800
parents 738d84b4de52
children 2dfdff7549b2
comparison
equal deleted inserted replaced
52:c9f2311dedea 53:81fe0523a078
1 fetch 1 fetch
2 ===== 2 =====
3 3
4 fetch stuff from the interwebs 4 fetch stuff from the interwebs
5 5
6 `fetch.py <http://k0s.org/mozilla/hg/fetch/raw-file/tip/fetch.py>`_ is
7 a single-file python module bundled as a
8 `package <http://k0s.org/mozilla/hg/fetch/>`_ for easy installation
9 and python importing. The purpose of fetch is to mirror remote
10 resources (URLs) to a local filesystem in order to synchronize and
11 update dependencies that are desired to be mirrored in this way.
12
6 13
7 Format 14 Format
8 ------ 15 ------
9 16
10 `fetch` fetches from a manifest of the format:: 17 ``fetch`` fetches from a manifest of the format::
11 18
12 [URL] [Destination] [Type] 19 [URL] [Destination] [Type]
13 20
14 A URL can contain a hash tag (e.g. http://example.com/foo#bar/fleem) 21 A URL can contain a hash tag (e.g. http://example.com/foo#bar/fleem)
15 which is used to extract the subdirectories from a multi-directory 22 which is used to extract the subdirectories from a multi-directory
16 resource. 23 resource.
17 24
18 The `Type` of the resource is used to dispatch to the included 25 The ``Type`` of the resource is used to dispatch to the included
19 Fetchers that take care of fetching the object. 26 Fetchers that take care of fetching the object.
27
28 Manifests are used so that a number of resources may be fetched from a
29 particular ``fetch`` run.
20 30
21 31
22 Fetchers 32 Fetchers
23 -------- 33 --------
24 34
25 `fetch` includes several objects for fetching resources:: 35 ``fetch`` includes several objects for fetching resources::
26 36
27 file : fetch a single file 37 file : fetch a single file
28 tar : fetch and extract a tarball 38 tar : fetch and extract a tarball
29 hg : checkout a mercurial repository 39 hg : checkout a mercurial repository
30 git : checkout a git repository 40 git : checkout a git repository
31 41
32 The `file` fetcher cannot have a hash tag subpath since it is a single 42 The ``file`` fetcher cannot have a hash tag subpath since it is a single
33 resource. 43 resource.
34 44
35 Though `fetch` has a set of fetchers included, you can pass an 45 Though ``fetch`` has a set of fetchers included, you can pass an
36 arbitrary list into `fetch.Fetch`'s constructor. 46 arbitrary list into ``fetch.Fetch``'s constructor.
37 47
38 48
39 Version Control 49 Version Control
40 --------------- 50 ---------------
41 51
42 The `hg` and the `git` fetchers fetch from version control systems and 52 The ``hg`` and the ``git`` fetchers fetch from version control systems and
43 have additional options. 53 have additional options. The only current option to the constructor
54 is ``export``, which is by default True. If ``export`` is True, then
55 the repository will be exported into a non-versioned structure. If a
56 subpath is specified with a ``#`` in the URL, the repository will also
57 be exported.
44 58
59
60 TODO
61 ----
62
63 * use `manifestparser <https://github.com/mozilla/mozbase/blob/master/manifestdestiny/manifestparser.py>`_
64 ``.ini`` files versus another manifest
65 format: when I started work on ``fetch``, I thought a
66 domain-specific manifest would be a big win. But, now, maybe a
67 ``.ini`` type manifest looks about the same, and is something that
68 is already used. The switch internally wouldn't be that bad, but
69 if ``fetch.py`` is used as a single file, it cannot have
70 "traditional" python dependencies. Since ``manifestparser.py`` is
71 also a single file, and ``fetch`` is only usable with internet
72 access anyway, maybe the
73 `require <http://k0s.org/hg/config/file/tip/python/require.py>`_
74 pattern could be used for this purpose
75
76 * clobber: generally, you will want the existing resource to be
77 clobbered, avoiding renames regarding upstream dependencies
78
79 * outputting only subpaths: often, you will not to fetch from the
80 whole manifest, only from certain subpaths of the manifest. You
81 should be able to output a subset of what is to be mirrored based
82 on destination subpaths. The CLI option ``--dest`` is intended for
83 this purpose but currently unused.
84
85 * fetcher options:
86
87 * note python 2.5+ specifics:
88
89 Unsolved Problems
90 -----------------
91
92 A common story for ``fetch`` is mirroring files into a VCS repository
93 because the remote resources are needed as part of the repository and
94 there is no better way to retrieve and/or update them. However, what
95 do you do if these remote resources are altered? In an ideal
96 ecosystem, the fixes would be automatically triaged and triggered for
97 upstream inclusion, or the diffs from the upstream are kept in local
98 modifications (although vendor branches, etc, are more suitable for
99 the latter class of problems, and in general discouraged when a less
100 intrusive system of consuming upstream dependencies are available).
45 101
46 ---- 102 ----
47 103
48 Jeff Hammel 104 Jeff Hammel
49 http://k0s.org/mozilla/hg/fetch 105 http://k0s.org/mozilla/hg/fetch