Hello Elena and all,

I have pushed the fixed code. There are a lot of changes in it because I went through all the code making sure that it made sense. The commit is here, and although there are a lot of changes, the main line where failures are caught or missed is this.

The test result file edition information helps improve recall - if marginally
The time since last run information does not improve recall much at all - See [Weaknesses - 2]

A couple of concepts that I want to define before going on:

First failures. These are failures that happen because of new bugs. They don't occur close in time as part of a chain of failures. The occur as a consequence of a transaction that introduces a bug, but they might occur soon or long after this transaction (usually soon, rather than long). They might be correlated with the frequency of failure of a test (core or basic tests that fail often might be specially good at exposing bugs); but many of them are not (tests of a feature, that don't fail often, but rather, when that feature is modified).
Strict simulation mode. This is the mode where, if a test is not part of the running set, its failure is not considered.

Weaknesses:

It's very difficult to predict 'first failures'. With the current strategy, if it's been long since a test failed (or if it has never failed before), the relevancy of the test just goes down, and it never runs.
Specially in database, and parallel software, there are bugs that hide in the code for a long time until one test discovers them. Unfortunately, the analysis that I'm doing requires that the test runs exactly when the data indicates it will fail. If a test that would fail doesn't run in test run Z, even though it might run in test run Z+1, the failure is just considered as missed, as if the bug was 'not encountered' ever.

This affects the time since last run factor. This factor helps encounter 'hidden' bugs that can be exposed by tests that have not run, but the data available makes it difficult
This would also affect the correlation factor. If test A and B fail together often, and on test_run Z both of them would fail, but only A runs, the heightened relevancy of B on the next test_run would not make it catch anything (again, this is a limitation of the data, not of reality)

Humans are probably a lot better at predicting first failures than the current strategy.

Some ideas:

I need to be more strict with my testing, and reviewing my code : )
I need to improve prediction of 'first failures'. What would be a good way to improve this?

Correlation between files changed - Tests failed? Apparently Sergei tried this, but the results were not too good - But this is before running in strict simulation mode. With strict simulation mode, anything that could help spot first failures could be considered.

I am currently running tests to get the adjusted results. I will graph them, and send them out in a couple hours.

Regards

Pablo

On Fri, Jun 13, 2014 at 12:40 AM, Elena Stepanova <elenst@montyprogram.com> wrote:

Hi Pablo,

Thanks for the update.

On 12.06.2014 19:13, Pablo Estrada wrote:

Hello Sergei, Elena and all,
Today while working on the script, I found and fixed an issue:

There is some faulty code code in my script that is in charge of collecting
the statistics about whether a test failure was caught or not (here
<https://github.com/pabloem/Kokiri/blob/master/basic_simulator.py#L393>). I
looked into fixing it, and then I could see another *problem*: The *recall
numbers* that I had collected previously were *too high*.

The actual recall numbers, once we consider the test failures that are *not
caught*, are disappointingly lower. I won't show you results yet, since I

want to make sure that the code has been fixed, and I have accurate tests
first.

This is all for now. The strategy that I was using is a lot less effective
than it seemed initially. I will send out a more detailed report with
results, my opinion on the weak points of the strategy, and ideas,
including a roadmap to try to improve results.

Regards. All feedback is welcome.

Please push your fixed code that triggered the new results, even if you are not ready to share the results themselves yet. It will be easier to discuss then.

Regards,
Elena

Pablo

_______________________________________________
Mailing list: https://launchpad.net/~maria-developers
Post to : maria-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-developers
More help : https://help.launchpad.net/ListHelp