Continuous Integration: a good first step (?)

Several practical concerns have led me to wonder if the *first* thing
that should be setup is continuous integration.  Yes, even before
writing tests.  A project is defined from the early stages of
development -- much carries over from this nascent phase, from the
install procedure to the API.  Typically, a programmer expediently
deploys the technologies they find they need, write a shim script,
and develop from there.  Changes in their environment, from
non-versioned files to dependencies, are not recorded either in
documentation or in code.  Since the developer already has their
environment setup for their programming, they do not have immediate
need for these instructions.

When another developer needs to utilize the project, or the project
needs to be deployed, the setup instructions must be recreated,
typically at first by hand.  Then a set of instructions is typically
posted at a wiki page or issue tracker.  Since the first iterations
were by hand, this list is usually wrong.  The list gets corrected.
Tests are written.  At first it is sufficient to run these by hands,
but at some point the investment in infrastructure to run the tests on
all platforms pays off in terms of saving developer time. Then a CI
system is in order.  Despite common sense, the continuous integration
steps are often different from how developers install the software and
run the tests.  Clever developers may anticipate and mitigate these
losses of time and gains in frustration, but more work is done than is
needed for the outcome.

Take the developer time spend on setting up the environment as M.  If
N developers work (apart from the original developer) work on the
project before there is an installation mechanism, N*M work is lost. 
Time is also lost in reconciling differences between CI results and
hand-run results in builds and tests.

Contrast with an approach where continuous integration is introduced
from the start.  Instead of just modifying their environment, a
developer must record steps whereby they modify their environment.
Since this should be idempotent anyway, a developer may test their
results by invoking their recorded steps.  In addition to having a
recipe, you also provoke the advantage of thinking in terms of how you
are modifying your environment.  Is it over-complicated to set up? Is
it fragile? Are there sticking points?  You can guess that if there
are, there will be the same for other developers and potentially
users.  You are forced to generalize.  In addition, you work in a way
that highlights deployment efficiency.  Are your install scripts easy
to use over an existing instance?  Are you losing efficiency in your
install? If it takes a long time for you, it will take a long time for
others as well as the CI system.  Your deployment tools become a
first-class citizen instead of an afterthought.

A project will be started with a VCS repository.  A boilerplate
``INSTALL`` script checks out the repository and does whatever
boilerplate is appropriate to the project of that type.  Developing
the software, you write a script and a test for this.  Boilerplate
could include a script that exits 0 and a test that invokes the
script. Then you develop. You install dependencies.
There will not be a ``~/joeuser/handwritten.dat`` on the CI machine,
so you have to be a little more clever than that.  If you need a
resource, you have to set it up.  With every checkin (at least
initially), the CI system runs your install and tests your code. This
ensures your code and tests are generically (to the definition of the
CI setup) utilitarian. Of course, programmers can certainly bypass
these velvet ropes, but once you are in that situation you've already
lost. Programmers will be encouraged to write robust installs and
tests that illustrate what they're working on.

While there is an initial cost up front, the cost of initial developer
time lost and the cost of developing a CI system for existant software
is saved and may surmount the initial cost in short order.  In
addition, developers are encouraged to follow a tried and true pattern
that leads to robustness. The software resulting should be easy to get
started with and hacked on.  In addition, the CI system provides an
instant view into the status of developing software.  Where is this
project?  With a suitable display, the status and complexity of the
project as well as what is currently being worked on may be discerned.

In order to make this system better, tools should be developed in
order to avoid the up-front cost of infrastructure.  There should be a
way to create a new project.  This will inject a stub into VCS and the
CI system that will be cloned to the developer's area.  Check-ins
should trigger the install and test phase of the code.  Tools may be
developed to make the utilization of dependencies recipes.  In this
case, development is minimally disrupted, the developer left to focus
on the intent of their project versus the nuts and bolts.  Contrast
with the copy+paste approach whereby legacy is perpetuated instead of
condensed. Developers should be encouraged to treat functionality as
modular, and effort as intent, instead of a developer process
cultivated from the harvesting of corpses.

The setup of the continuous integration system, being the central hub
of developer activity, is an open area to explore expertise in
refining the development process.  Each organization and group will
have particular needs and requirements to manifest their projects.  So
the CI system is programmed with a number of recipes.  Recipes may
diverge as software organically diverges, but they serve as a starting
point of requirements.  The CI system may be set up as stringently or
as loosely as the current scoping requires.  For a small group
starting out, a very bare bones boilerplate suffices.  As boilerplates
grow and diverge, the system becomes more complex and specialized
(though contrast this with needing to make a system more
generalized). 

A typical recipe:

1. Install
2. Test
3. (Cleanup)

You can use *this system* to develop *this system*!

I think I will work on this software.

See also:

http://k0s.org/mozilla/hg/autobot/file/tip/autobot/projects/autobot/__init__.py