workflow

What's the most important thing to figure out initially in a project? Workflow! Most drastic rewrites come from changing a software project's workflow midstream, or moving to configurable workflow from a static workflow. While I'm not in favor of top-down design , I do think its important to at least lay a conceptual foundation before diving in and writing code, and what workflow is to be used is a part of this foundation.

Like every other concept in software, once you grasp this, then you'll immediately move on to the meta-level : the workflow of software construction.

Why is workflow important?

Software grown to serve problems of the curation and management of information go through a life-cycle:

  1. The software serves static (maintainer-generated) content with a static workflow
  2. The software serves dynamic (user-generated) content with a static workflow
  3. Different workflow is applied according to content type
  4. Workflow is formalized into templates that may be applied by users as desired
  5. The workflow becomes entirely configurable by users

Each stage is a considerable evolution from the previous stage, bringing not only new possibilities of what one can do with information, but new ways of thinking about information.

Idea Workflow

The workflow of software-related ideas

Project Creation Workflow

Souce data:

Create:

See MakeItSo

Software Development Workflow

  1. pastescript template → project
  2. project → VCS
  3. project → Trac

The job of a software engineer is to transform a precise vision into a working piece of software. From the nature of the project, the corpus of knowledge. is explored for applicability of intent and form. Software tools are chosen and, if available, a pastescript template is selected that fits the project's needs.

I typically utilize a development process that involves several moving pieces at once, moving quickly between various tools and different roles. Programming, I try to sketch the outline of what I am trying to do first, often noting thoughts in comments in the code or more substantive thoughts in an Issue Tracker .

Lessons noted from project development may be put back into pastescript templates, or a new template may be created from the work (using the pastescripttemplatetemplate) if the project is archetypal of a general pattern. In this way, the software development process itself is iteratively improved, leading to both faster and more robust development.

Software Deployment Workflow

for python

This process can and should be automated.

Site Maintainence Worklow

I believe in the web as a distributed computing platform. Instead of using a variety of web services, I like to keep my information on my hard drive where I can easily organize, edit, and manipulate my documents.

To affect this end, I have developed and use a variety of tools:

This fairly simple set of apps is a powerful combination which bridges the gap often found between the traditional model of local files to the modern vision of web resources. No database is needed -- instead, this is the document driven model. Most of my editing is done with emacs. While I am as excited as anyone else about new and traditional multimedia, I am a lover of text and believe that it is among the most powerful forms of expressions and forms of interaction with computers and people.

Here is more on how this website works.

Website Deployment Workflow

Setting up a new server:

Patch Management Worfklow

Managing patch queues enables the association of a body of work (a patch) with a specific intent (an issue). I use the Mercurial Queue Extension as a patch management tool. With this workflow, commits are not made to the repository clone, but to a series of patches which are applied to the body of work.

Initializing a repository for a patch queue

  1. Clone master repository: The usual workflow for a body of programmers using DVCS is to have a canonical (by convention) master which is the active point of development. There may be branching and merging amongst several active points of development, but usually coders know on which shared branch their changes will likely end up.
  2. Initialize patch queue: You need to tell mercurial to setup a patch queue for you. 
hg clone [MASTER-REPO] [REPO]
cd [REPO]
hg qinit -c

Notice that there's nothing preventing you from initializing a patch queue on top of your versioned patch queue. This could be used to allow code review and extension of patches in a shared development system, though it is a pretty big abstraction layer (so know/decide what you're doing).

Serving the patch queue

The patch queue can itself be versioned and served like any filesystem resource. This allows patches to be shared with others who can then apply them and test them before pushing the work to the master repository. Generally, you will want to work with clones of the primary (served) patch queue so that you can share work between clones.

On the server: 

hg init [REPO]-patches

On the clone: Setup the server as the default push branch by editing .hg/patches/.hg/hgrc in your clone: 

[paths]
default = http://[SERVER]/path/to/[REPO]-patches
default-push = ssh://[SERVER]/path/to/[REPO]-patches
Then push the patch repository: 
cd .hg/patches
hg push

Making new patches

hg qnew bug-[BUG]

Committing patches

You will also want to push your changes to your served patch queue. 

hg qrefresh
hg qcommit -m [MESSAGE]
cd .hg/patches
hg push

Switching between patches

See what patches are applied: 

hg qseries -v

Switch to the patch you care about: 

hg qgoto [PATCH]

Splitting patches

It is sometimes desirable to split one patch into two.

Get to the root and make sure your patch is at the top: 

cd $(hg root)
hg push

View the files in the patch. Assuming you are using git diff: 

hg qdiff | grep '^+++ ' | sed 's/+++ b\///'

Add the paths you want to keep: 

hg qrefresh [paths] [to] [keep]

Make a new patch: 

hg qnew -f

See http://mercurial.selenic.com/wiki/MqTutorial#Split_a_patch_into_multiple_patches

Pulling from the tip

When you want the latest changes from the tip, as is the case when testing your patches prior to pushing, you need to unapply your patch queue before pulling and then reapply it following: 

hg qpop --all
hg pull [MASTER-REPO]
hg update
hg qpush --all
There should be no conflicts on hg update since after the qpop you will have no local changes. You may also want to apply patches one at a time to ensure they work correctly.

Pushing to tip

Only when you are ready to push to tip should you cement your patches to the tree mainline.

Don't forget to push your empty queue to your served patch repo to clear it out. But don't do this until your work is really upstream, as after that point you'll have to recover your queues from version history. 

cd .hg/patches
hg push
Ideally, you could keep your patches around and have separate repositories for patches applied to the mainline and patches under active development, but I am unsure how to easily support this workflow.

Reference

This patch queue workflow is a subset of that described in the Mercurial Queue extension documentation. Mozilla also features documentation on how to use Mercurial queues.