Talos Signal from Noise: Configurable Talos Data Filters
As part of Signal from Noise I introduced a patch that changes the way --ignoreFirst works and adds configurable data filters to Talos :
While this is a small change in terms of how the code currently works, it lays the groundwork for a window of possibilities in terms of Talos statistics. Currently, pageloader calculates the "median" (ignoring the high value), the mean, the max, and the min, and outputs these along with the raw run data. Pageloader is for loading pages and taking measurements, not really for doing statistics. So it would be nice to move this upstream: first to Talos, then to graphserver proper.
Being able to specify data filters with --filter from the command line and filter: in the .yml configuration file allows the test-runner to change the "interesting number" by which we measure performance metrics on the fly. While there are currently only a few filters available, it is easy to add more metrics as we need them.
In a parallel effort, the JetPerf software consumes Talos filters . This is a good example of the expansion of the Talos ecosystem: as a ciritical part of our performance testing infrastructure, building tests and frameworks on top of Talos. In general, the A Team is moving towards a testing ecosystem of reusable parts and sane APIs.
Data filters were added to talos as an interim measure to make the "interesting number" calculations more flexible. As we play with different types of statistics, we need the ability to change configuration without having to jump through too many hoops and this fulfills this immediate need.
However, in the longer term, Talos and pageloader shouldn't really be doing statistics at all. They are in the "statistics gathering" camp where graphserver is in the "statistics processing" business. It would also be nice if there was a piece of software that let you analyze Talos results locally, ideally using the same statistics processing package that graphserver uses. This is outlined in https://bugzilla.mozilla.org/show_bug.cgi?id=721902 .
