Mercurial > hg > silvermirror

comparison silvermirror-whitepaper.txt @ 0:abb358e2434c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
initial commit of silvermirror, from http://my-svn.assembla.com/svn/arbez/silvermirror
author k0s <k0scist@gmail.com>
date Mon, 07 Sep 2009 15:39:06 -0400
parents
children 9b139702a8f9
comparison
equal deleted inserted replaced
-1:000000000000 0:abb358e2434c
1 SilverMirror Whitepaper
2
3 It is necessary to maintain parallel directory structures of various
4 resources across an arbitrary number of computers. The traditional
5 approach is the central server model, where files live in one
6 canonical location and the network is used to give access to the data.
7 However, this model has deficiencies, chiefly among them that if the
8 server goes down or must be moved a considerable amount of effort must
9 be extended to set up a new central server.
10
11 Distributed version control, often of nominal use (in the case where a
12 canonical trunk exists) is ideally suited to provide mirroring of
13 desired resources across computers.
14
15 Implementation
16
17 A front end to a DVCS - most likely mercurial but potentially bzr -
18 will be written to keep resources in sync across an arbitrary number
19 of computers. The front end, called SilverMirror, may be used to push
20 or pull changes to resources. Optionally, a daemon will monitor
21 changes to resources and push or pull changes at desired intervals.
22
23 The use should be as natural as possible and require no interaction
24 for everday tasks. A resource consists of a directory and all
25 subdirectories and their contents. Once a resource is denoted as
26 versioned, any change to the resource's directory structure should be
27 mirrored across machines without user intervention. Files matching a
28 pattern may be ignored, either globally or on a per resource basis,
29 for the purpose of versioning.
30
31 Configuration
32
33 SilverMirror is configured via an INI file containing a section for
34 each resource and a section for application configuration.
35
36 The main section, denoted [::SilverMirror::], has the following options:
37
38 * directory: base directory for SilverMirror. The SilverMirror
39 configuration is stored in ${directory}/.silvermirror . If omitted,
40 the user's home directory is used.
41
42 * ignore: global patterns of files and directories to ignore. Paths
43 matching these patterns will not be versioned.
44
45 Each section has the following configuration options:
46
47 * directory: path of the resource. If a relative path is used, it is
48 joined with the directory setting from the main section. If this
49 setting is not specified, the section name is used as a relative path.
50
51 * ignore: paths not to version on a per resource basis. This is in
52 addition to the patterns specified by the ignore setting in the main
53 section.
54
55 * conflict: handler to resolve conflict.
56
57 * hosts: hosts to push/pull from
58
59 In order to ensure coherency among resources, all relevant
60 configuration options must be synced prior to push/pull transactions.
61
62 Default Configuration:
63
64 [::SilverMirror::]
65 conflict = ClobberRemote
66
67 Example of a more complex configuration:
68
69 [::SilverMirror::]
70 conflict.push = ClobberRemote
71 conflict.pull = ClobberLocal
72
73 Push
74
75 Push changes to remote resources. When resources are pushed, first
76 changes are pulled from each remote host in turn, conflicts between
77 local and remote changes are resolved (see Behavior on Conflicts),
78 then local modifications are pushed. This is done to keep the
79 resources in sync.
80
81 When new files are added to the resource they are automatically added
82 to the hg repository. When resource files are edited the changes are
83 pushed to the repository. When a conflict occurs between local
84 resources and remote resources, the conflict handler is used.
85
86 Pull
87
88 Get changes to the cloud filesystem resources. If no host is
89 specified, pull changes from all known + accessible hosts.
90
91 Namespaced Resources
92
93 It is possible to maintain versioning of a subdirectory within a
94 resource.
95
96 Example:
97
98 [docs]
99 directory = /path/to/docs
100
101 [docs:private]
102
103 This configuration snippet describes a resource, [docs:private],
104 namespaced within the [docs] resource. [docs:private] inherits
105 configuration and behavior from [docs] but may be dealt with
106 separately. For example, some computers in the cloud may not have
107 [docs:private] specified in their configuration and so will not get a
108 copy of it upon pulling. A common use case is specifying a
109 subdirectory to be omitted with the ignore option in the configuration
110 file, then, when this subdirectory needs to be shared between multiple
111 computers, removing it from the ignore values and including as a
112 namespaced resource.
113
114 In the above example, because the directory option was not specified
115 in the [docs:private] section, the path to [docs:private] is taken
116 from its namespace (private) and the directory of its parent resource.
117 So its base directory is /path/to/docs/private . If a relative path
118 was specified in the directory option of the [docs:private] section,
119 it would be joined with the base directory of [docs].
120
121 Behavior on Conflicts
122
123 Conflict handlers are set via setuptools entry points. Several
124 conflict handlers are provided with SilverMirror:
125
126 * ClobberLocal: replace local changes with changes from remote files
127
128 * ClobberRemote: replace remote file changes with changes from local
129 files
130
131 * Edit: invoke an editor (default: $EDITOR) to interactively resolve
132 the conflicts
133
134 The conflict handler may also be specified from the command line:
135
136 silvermirror push -d ClobberRemote
137
138 Command Line Usage
139
140 silvermirror [push|pull] [resource] [options]
141
142 In the simplest invocation, SilverMirror is used with no command line
143 arguments:
144
145 silvermirror
146
147 This pushes changes of the resource as determined by the current
148 working directory after pulling outstanding changes from all
149 applicable remote computers and invoking the conflict handler for
150 push. If the current working directory is not within a resource, all
151 resources will be pushed.
152
153 Finer control is obtained by specifying command line arguments:
154
155 [push|pull] : whether to use the push method (which includes pulling
156 for changes; see above) or the pull method. If not specified, the
157 resource is pushed.
158
159 [resource] : which resource to act upon. This can be the resource
160 name, as specified in the .ini file, or the path to the base directory
161 of the resource. Note that if a path is specified, it must be to the
162 base directory of the resource as SilverMirror has no notion of
163 disparate versioning within a resource. If the resource is not
164 specified, the resource that the current working directory is within
165 is used. If the current working directory is not in a resource path,
166 all resources are acted upon in turn. If the key word "--all" is used,
167 all resources will also be acted upon.
168
169 [options] : several command line switches are available to the
170 silvermirror program:
171
172 * -d <handler> : specify which conflict handler to use. <handler>
173 should be the name of the desired conflict handler. A list of all
174 conflict handlers is available with the "--conflict-handlers" option.
175
176 * -H <host>, --host=<host> : pull and/or push only to specified
177 hosts. If this option is used more than once, the hosts specified will
178 be acted upon.
179
180 * --conflict-handlers : list the name and description (if
181 specified) for all available conflict handlers.
182
183 * --help : displays help and usage information
184
185 Behavior Respecting Versioned Directories
186
187 SilverMirror does not desire to duplicate versioning on directories
188 already under version control (svn, bzr, hg). These resources are
189 automatically ignore. In a future implementation, these resources
190 would optionally be checked out or updated upon a pull.
191
192 Automatic Syncronization
193
194 SilverMirror includes a script that will automatically invoke
195 syncronizing the resources in a specified period of time. This daemon,
196 called silvermirrord, is invoked from the command line with options
197 parallel to the silvermirror program. One additional option, -s, tells
198 how many seconds between syncs. Upon invocation, this program puts
199 itself in the background and performs the desired sync every number of
200 seconds specified. It is important that the conflict handler specified
201 is noninteractive, otherwise the daemon will hang forever.
202
203 As an alternative, the silvermirror program may be invoked from a cron
204 job.
205
206 Future Work
207
208 SilverMirror implements a cloud filesystem which may be accessed
209 nearly transparently by an arbitrary number of computers. Several
210 improvements could extend SilverMirror to solve several deficiencies
211 of modern filesystem.
212
213 * tagging: in most filesystems, a file has a canonical location.
214 However, it may be desirable to have the file accesible via multiple
215 paths. In practice, this is achieved via symbolic links. However, this
216 requires manual maintaince of links vs the canonical location. Noting
217 that this problem is identical to tagging, a solution minimizing
218 manual intervention could be added to SilverMirror.
219
220 * update of web documents: modern computers deal heavily with
221 documents via URLs. It is noted that this includes files, the URL of a
222 file with path ${PATH} being file://${PATH} noted implicitly from
223 contexts. However, existing shells and operating systems have no
224 mechanism for indicating that a "file" is a web resource. Such
225 functionality could be added to SilverMirror so that up-to-date
226 versions of web resources could be maintained. This infrastructure
227 could also include notions for updating versioned resources (see
228 Behavior Respecting Versioned Directories) with parallel notation.