Hi, Pablo! On Jun 02, Pablo Estrada wrote:
Hello everyone, Here's a small report on the news that I have so far:
1. I had a slow couple of weeks because of a quick holiday that I took. I will make up for that. 2. I added the metric that considers number of test_runs since a test_case ran for the last time. I graphed it, and it does not affect the results much at all. -I still think this is useful to uncover hidden bugs that might lurk in the code for a long time; but testing this condition is difficult with our data. I would like to keep this measure, specially since it doesn't seem to affect results negatively. Opinions?
What do you mean "added the metric"? You started to use it when selecting what tests to run and found that it doesn't affect results much? Well, it's your project, you can keep any measure you want. But please mark clearly (in comments or whatever) what factors affect results and what don't. It would be very useful to be able to see the simplest possible model that still delivers reasonably good results. Even if we'll decide to use something more complicated at the end.
3. I liked Sergei's idea about using the changes on the test files to calculate the relevancy index. If a test has been changed recently, its relevancy index should be high. This is also more realistic, and uses information that it's easy for us to figure out.
Same as above, basically. I'd prefer to use not the model that simply "looks realistic", but the one that makes best predictions. You can use whatever criteria you prefer, but if taking into account changed tests will not improve the results, I'd like it to be to clearly documented or visible in the code.
- I am trying to match the change_files table or the mysql-test directory with the test_failure table. I was confused about the name of tests and test suites, but I am making progress on it. Once I am able to match at least 90% of the test_names in test_failure with the filename in the change_files table, I will incorporate this data into the code and see how it works out. - *Question*: Looking at the change_files table, there are files that have been *ADDED* several times. Why would this be? Maybe when a new branch is created, all files are ADDED to it? Any ideas? : ) (If no one knows, I'll figure it out, but maybe you do know ; ))
In most cases it shows how a revision is travelling across different branches. It was commited somewhere (adding files there), then merged into another branch (adding files there), then into yet another, and so on. In some cases it could be committed, uncommitted, committed again, repeat until satisfied. Or simply rebased. I personally use this quite often to test my changes in buildbot (of course, it's only safe to use in a personal tree, the one which is not shared with others). Regards, Sergei