I thought of a new monkey

See http://techblog.netflix.com/2011/07/netflix-simian-army.html Given an HA system using salt and consul, I give you bleeding edge monkey. The monkey chooses a service and a node that services that service. It salt high-states the node and makes sure it doesn't fall down. You have to have reliable monitoring for this. How else would you know what goes wrong? You could add an extension i.e. that high states all the node of the service if the first one doesn't bleed too badly.