28 Apr
2014
28 Apr
'14
1:36 p.m.
Hi, Pablo!
On Apr 27, Pablo Estrada wrote:
> In any case, I already have a decent idea of how the data is arranged. I
> can see how it is difficult to trace test results back to code changes
> directly, but it might be possible.
>From historical data that you have it is strictly impossible.
One can for example:
+ do some changes
* bzr commit -> revno: 1234 (for example), revid: something@unique-1
* bzr push
+ buildbot builds and tests revno 1234
* bzr uncommit
+ do more changes
* bzr commit -> revno: 1234 (for example), revid: something@unique-2
* bzr push
+ buildbot builds and tests revno 1234
note, that two *different* revisions got the same revno! And the changes
from the first revision are completely and totally lost, there is no way
to retrieve from from anywhere.
Revision-id is the only unique identifier for a revision, unfortunately,
it's not logged in these tables. I believe we'll change buildbot so that
revid would be logged in the future. But so far it wasn't needed, and
this is one of the defects in the data.
In my testing data I've generated fake revision ids - using md5, revno,
set of changed files, timestamps, perhaps, I don't remember the details.
I've presumed that when this will be put in production, buildbot will
log real revids.
> I still haven't figured out how to recognize what code changes pertain
> to the same 'history', or 'line' (I think that you were not
> considering this in your experiments... and it might turn out to not
> be relevant at all, but I still want to consider it, to try to make a
> more complete model).
To a certain extent you can get this from revnos (but not from revids).
See any complex history graph in "bzr viz". For example this:
http://mhall119.com/2010/04/bzr-is-amazing/
(I've just googled it out with image search for "bzr viz")
Note, that changes in the same "line" have the same revno prefix, and
only the last component in the revno is being incremented. On that
screenshot, you can recognize these "lines" of changes:
black: 28 -> 29 -> 30 -> 31 -> 32
blue: 19.3.1 -> 19.3.2
green: 19.1.5 -> 19.1.6 -> 19.1.7 -> 19.1.8 -> 19.1.9 -> 19.1.10 -> 19.1.11
magenta: 19.4.1
red: 19.5.1
yellow: 19.6.1 -> 19.6.2
if you'd like you can install bzr, branch lp:maria and run "bzr viz" or
"qbzr".
> So I have just two questions that should be easy to answer by email:
> 1. In your graphs, exactly, what do you mean by Precision? What about
> Recall? And cutoff?
For many years I was very interested in Information Retrieval (I'm the
author of MySQL full-text search feature). These terms come from IR.
http://en.wikipedia.org/wiki/Precision_and_recall
Think of it this way:
* There's a revision we want to test in buildbot.
* There're NNN (more than 20000) tests to run in total (taking into
account all combinations, test variants, and builders).
* We only want to run N (much less than NNN) tests.
* We sort all NNN tests in some specific order, and take first N
tests. N is the "cutoff".
* Now, I check the history - what tests out of all NNN have failed and
succeeded.
* Ideally, first N tests should've failed, and all other NNN-N tests
should've succeeded.
* Precision corresponds to the number of succeeded tests in the first N,
where 100% means that all N tests have failed, no succeeded test was
selected.
* Recall corresponds to the number of failed tests outside of the first
N. An 100% there are no failed tests outside of the first N, no failed
test was missed.
In IR precision and recall should be always considered together. It's
trivial to optimize only one of these values. E.g. for N=0 you have a
perfect precision, you never select a succeeded test. And you have a
very bad recall - all failed tests were missed. Alternatively, for
N=NNN, you have a perfect recall, and bad precision.
In my results (see readme from v1) I've realized that I don't care about
precision as such, I am only interested in recall and the total
testing time. Although precision and the time are related, time is much
more meaningful here. So I optimize a {Time, Recall} pair, not
{Precision, Recall}.
> 2. I looked at your launchpad profile but I can't seem to find the repo
> with your tests (if you even pushed it to lp at all). Is it available?
Uhm. I've never published it. It was a pretty much write-only code in
very hairy perl. I don't know if it makes any sense to look at it at
all.
But ok, I've just pushed it, here you are:
bzr+ssh://bazaar.launchpad.net/~maria-captains/mariadb-tools/mtr-prob-tmp
Regards,
Sergei