autobot
=======

buildbot for the A*Team


What is autobot?
----------------

autobot is a continuous integration solution for the Automation and
Tools Team.  We have a lot of software.  We're really talented, so
usually it doesn't break.  But we're not infalliable.  Our robot ally,
autobot, is there to test things for us.  Let's meet autobot!


Installing autobot
------------------

autobot may be installed using the install script::

 curl http://k0s.org/mozilla/hg/autobot/raw-file/tip/INSTALL.sh | bash

This will create a virtualenv and install autobot for development
($VIRTUAL_ENV/src/autobot).  You can also use the script as
instructions and install it in the same manner as any other python package.


Setting up a buildmaster and slave
----------------------------------

Once you have autobot installed and the virtualenv activated, you'll
want to create a buildmaster and a buildslave.

You can create a master-slave pair by running ``create-autobot`` after
activating the virtualenv. This is mostly useful for autobot
development. The scripts ``create-autobot-master`` and 
``create-autobot-slave`` are also available and are more useful for
production, as normally you will not want to keep the master and the
slave on the same machine except for development and testing.  The
scripts will prompt you for a factory or you can specify one or more
from the commandline using ``--factory`` or ``-f``. The factories are
from ``autobot.projects`` and its subdirectories.  You can list the
available factories with ``create-autobot --list-factories`` (or 
``create-autobot-master --list-factories``).

Running ``create-autobot --help`` will give the variables you can set
when it makes a new bot for you (``create-autobot-master`` and
``create-autobot-slave`` also have a ``--help``, the variables in
``create-autobot`` being a superset of these).

If you don't specify a variable, the default will be used to create
your particular bot.  You can change these in the resultant
configuration later.

 
Using autobot
-------------

The buildmaster and buildslave are started with 
``buildbot start master`` and ``buildslave stop slave`` from the bot's
directories. Respective ``stop`` commands also exist.  If you used the
``create-autobot`` command the generated bot will have a
``restart_buildbot.py`` script that will stop and start both the
master and slave and (if debug is set) remove the log as well.

The generated ``master.cfg`` file reads values from a ``master.ini``
file in the same directory.  The master.ini contains a number of
different sections:

* ``[:master:]`` contains the top level configuration for the master
  as well as default settings for slaves
* sections like ``[slave:slavename]`` indicate a slave of name
  ``slavename``
* all other sections are construction parameters for factories

The format of the ``.ini`` file is detailed in the subsequent
section.  You may change which .ini file is used by editing the
``master.cfg`` file, or, if for whatever reason you don't want to use
an .ini file as a driver you can construct the appropriate
configuration therein yourself.

An overview of the build status is detailed at the waterfall display,
by default at http://localhost:8010/waterfall . To force a build,
click on the desired builder and there will be a force build button
towards the bottom.

It is important to remember that continuous integration is a safety
net, not a first line of defense.  In other words, continue to check
your code *before* sending it to autobot.


.ini file format
----------------

As stated above, a ``master.ini`` file contains three different types
of sections:

* ``[:master:]`` contains the top level configuration for the master
  as well as default settings for slaves
* sections like ``[slave:slavename]`` indicate a slave of name
  ``slavename``
* other sections are construction parameters for factories

What goes in each of these?

``:master:`` The master section contains parameters appropriate to the
buildmaster:

* slaveport: which port to listen for the slaves on
* htmlport: which port to serve the waterfall display on
* pollInterval: how often to poll the repositories, in seconds [DEFAULT: 60]
* treeStable: how long to wait before the tree settles down before
  building [DEFAULT: 60]
  
The other defaults may be seen by running ``create-autobot-master --help``.

Other parameters given in the ``[:master:]`` section are used as the
baseline defaults for each slave.  They may be overridden in each
``slave:`` section

----

``slave:`` parameters for each slave:

* password: each slave *must* have a password.  Unless there's a
  reason to have a different password per slave, its probably better
  to put this in the ``:master:`` section

* factories: space-separated list of factories to run on that slave.
  If the special value ``*`` is used, all factories will be run.  You
  can view the factories available with 
  ``create-autobot --list-factories``. The factory name corresponds to
  the directory (or module) name in ``autobot.projects``.  If
  every slave is meant to run the same factories, you can put this in
  the ``:master:`` section

* os: the operating system of the slave. Should be one of ``linux``, 
  ``win``, or ``mac`` (though see TODO about future use of MozInfo
  making this obselescent). You probably *shouldn't* have a default
  key for this in the ``:master:`` section (though autobot won't try
  to stop you!).

----

factory sections:  All other sections are used to build the factories.
Each parameter in a factory section is used as a constructor
(``__init__``) keyword argument when they are created in the
``master.cfg``. So a factory section like:: 

 [foo]
 bar = fleem
 baz = another parameter

will invoke the foo factory (lets say it maps to ``MyFooFactory``) on
each slave like::

 MyFooFactory(**dict(bar='fleem', baz='another parameter'))

In addition, if a factory has a special key, ``platform``, the slave
will pass its platform information when instantiating a factory.
Currently, this is a dict with a single key, ``os``, but more may be
added as needed.  As noted in the TODO below, ideally this would be
deprecated entirely by MozInfo but such is the interim solution.


Projects
--------

What does autobot test?

* logparser      [WORKING]
* profilemanager [IN FLIGHT]
* mozmill        [IN FLIGHT]
* devicemanager  [IN FLIGHT]
* firebug        [TODO]
* jetpack        [TODO]

The projects are obtained from any factories in subdirectories of
``autobot.projects`` (use 
``python -c 'from autobot import projects; print projects.__file__'`` 
to see this location).



Adding a New Project
--------------------

Occassionally, you'll need to add a new project to test.  You can add
a ``__init__.py`` file to a directory under ``autobot.projects`` and,
if you have a factory therein, autobot will find it and add it to the
list of factories it knows about.

There is also a script, ``create-autobot-project``, that can create
this stub for you::

 Usage: create-autobot-project [options] project <output>

``project`` is the name of the project and ``output``, if specified,
is the path of the file to create.  If output is not specified, it
will find its way to a directory named for the project under
``autobot.projects``.


Is your autobot being feisty?
-----------------------------

Let me know! I'd like to make autobot a solution that works for all
stake-holders, and if you're reading this, that means you!

jhammel@mozilla.com


TODO
----

No software of any size is ever finished.  Here are a few things I
would like to add:

* singular checkout of repos on slaves: the slaves should have a
  singular master repo that is checked out once for each repo URL,
  branch pair.  It is then updated as the slaves need and (e.g.)
  cloned from there.  This should effectively minimize fetch time.

* use (as yet non-existent) MozInfo package on the slave: instead of
  specifying the operating system, etc on the master, MozInfo should
  determine the applicable information from introspection