Hi, Pablo! First: please always cc: maria-developers@lists.launchpad.net, don't send emails to me only. Now: On Mar 11, Pablo Estrada wrote:
Hello! I am getting ready to submit my proposal. Is there any feedback for my application document? Anything that would be good to add, remove, rephrase, etc?
Go on and submit it. There will be a comment form under your proposal, if I will want you to change something, I will leave my comments there. Please mention what tools/languages you think you'll be using.
Also, if I could see the data available (the history of test results), I could write more in detail : )
Hm. Okay. Attached is the preprocessed table that I used two years ago. It lists test failures in buildbot in a given period of time. Columns are (as far as I remember): * time of the test run start * code branch that was tested * revision id that was tested - it's fake, our buildbot tables don't store revision id. I've constructed it from the set of changed files, revision number, etc. hopefully it's as unique as a real revision id * build number * platform where the tests were run * test type - basically, the set of tests to run and what protocol to use. one test may be run with different protocols * test name * test variant - or combination. certain tests may be run in many combinations Using tis file I tried to rearrange test execution order for every specific build number (using failure data from previous builds in the same table) to detect test failures as early as possible. I managed to have 90% of all test failures happening within first 10% of the tests. The goal of this project is to have something we can put back into buildbot, so that it could actually run only 10% of the tests, still having 90% probability of discovering a failure (or 20%, or 95%, whaveter). See also http://buildbot.askmonty.org/buildbot/ you can click on a link for any branch, for example, http://buildbot.askmonty.org/buildbot/grid?branch=10.0 then on any particular build, for example, http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297 and on a stdio log for a particular test run: http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297/step... on these pages you'll see what a platform (builder) is, build number, test types, combinations, etc. Regards, Sergei