[Maria-developers] mysql-test-run (was Re: BuildBot)
Michael Widenius <michael.widenius@gmail.com> writes:
To go trough the things that are wrong with mysqltest:
In my opinion, the single biggest problem with mysqltest is that it is a seperate program from the test suite driver mysql-test-run.pl. It's even written in a different language (C vs. Perl), and test cases themselves are written in a third language (the mysqltest language) + a mixture of SQL and even some shell scripts. That's _5_ different languages in a single application :-( This is a disaster for the structure of mysql-test-run. Such a large part of the code (and complexity) is just trying to handle communication between the various parts, which is really hard to do well due to different languages. The first step in any major improvement (in my opinion) should be to re-implement mysqltest in Perl inside mysql-test-run.pl. But that would be a project of several months, not something I'm eager to start just now ...
I like the philosophy behind it: Write a test in pure SQL and then ensure that the generated result doesn't change. It's easy enough to allow any developers to generate a test case.
Yes. This is a very strong point for testing that correct results are produced by a query or similar. Keeping something like this is very important if thinking about extending or replacing the test framework. The weakness with mysqltest / mysql-test-run is not that it does this, it is that it does not provide additional/separate functinality for other kinds of tests that do not fit this simple framework well.
In particular, it's good that tests and results are different files as it makes it easier to understand what are wrong when you get a failure.
I am trying to understand what precisely you have in mind here... One big problem with other test suites I have seen is that if you want to debug a failure, you first have to spend a lot of time debugging and understanding the test case code just to understand what the problem actually is. Getting a diff of queries run and results produced like mysqltest does makes this step trivial (if the problem is wrong results from a query). Also the --record functionality for generating .result files can be really good to ease the production of test case and also make it more likely the developer will not be tempted to insufficiently check the results (everything is checked by default). It does have the big risk of not properly checking that the committed .result is in fact correct, something I have seen more than once. This does not really have to mean using separate files. I have in mind something like a Perl-based suite where I could do: query <<SQL, <<RESULT SELECT a, b FROM t1 JOIN t2 ON (a = b) ORDER BY a, b; SQL RECORDME RESULT and `mysql-test-run --record` would rewrite this as query <<SQL, <<RESULT SELECT a, b FROM t1 JOIN t2 ON (a = b) ORDER BY a, b; SQL a b 1 1 2 2 4 4 RESULT And in case of different result, it could output a diff of just the RESULT section of the failing query, along with the query. It could also give the correct line number in the .test file, avoiding the need to search the .test file for the query that produced the diff. I think that might make it even easier than separate result file. Something like that would of course allow putting arbitrary Perl code in-between running queries (with lots of useful modules available) where this is more appropriate. This would eliminate the need for the mysqltest language for new test cases, which is one less language again. And it would be quite easy to have a backwards-compatible emulation of the old .test and .result files (just a small wrapper that reads the .test and .result files and passes the contents to the query() function).
One benefit is that you can usually run a test by just piping it trough the normal MySQL client, which simplifies testing and debugging a lot!
Yes, this should be kept for simple stuff, very important. - Kristian.
Hi everyone, As a QA engineer, I had a few thoughts I would put into the discussion as well. Please feel free to take them or leave them (or to throw them a parade and give them the key to the city) ;-) On Mon, Apr 6, 2009 at 3:17 PM, Kristian Nielsen <knielsen@knielsen-hq.org>wrote:
Michael Widenius <michael.widenius@gmail.com> writes:
To go trough the things that are wrong with mysqltest:
In my opinion, the single biggest problem with mysqltest is that it is a seperate program from the test suite driver mysql-test-run.pl. It's even written in a different language (C vs. Perl), and test cases themselves are written in a third language (the mysqltest language) + a mixture of SQL and even some shell scripts. That's _5_ different languages in a single application :-(
This is a disaster for the structure of mysql-test-run. Such a large part of the code (and complexity) is just trying to handle communication between the various parts, which is really hard to do well due to different languages.
The first step in any major improvement (in my opinion) should be to re-implement mysqltest in Perl inside mysql-test-run.pl. But that would be a project of several months, not something I'm eager to start just now ...
I agree that more 'moving parts' = more overhead. I can't say that I agree with re-implementing in this fashion, but now would be a good time to see how we can make things leaner / more efficient / more to-the-point (ie we can focus on using the tool to test and not 'keeping the parts in harmony')
I like the philosophy behind it: Write a test in pure SQL and then ensure that the generated result doesn't change. It's easy enough to allow any developers to generate a test case.
Yes. This is a very strong point for testing that correct results are produced by a query or similar. Keeping something like this is very important if thinking about extending or replacing the test framework.
The weakness with mysqltest / mysql-test-run is not that it does this, it is that it does not provide additional/separate functinality for other kinds of tests that do not fit this simple framework well.
I very much like the straight-up SQL-> result paradigm *for certain tests*. This is GREAT for a basic, functional suite (does SELECT work. Are INSERT DEFAULT and INSERT () interchangeable <in certain scenarios>). However, getting into more 'scenario-based' testing, there are problems - coordination between 'users' in a test, tools for greybox-manipulation (tweaking the software mechanics beyond just INPUT/OUTPUT testing), etc. It would be great to keep the simplicity for certain scenarios, but to have extensible power for more complicated tests when needed. One issue I have seen with certain proposed replacements is that they allow complex testing, but the overhead of that power results in even the most simple tests becoming a monster (the overhead of defining a test).
In particular, it's good that tests and results are different files as
it makes it easier to understand what are wrong when you get a failure.
I am trying to understand what precisely you have in mind here...
One big problem with other test suites I have seen is that if you want to debug a failure, you first have to spend a lot of time debugging and understanding the test case code just to understand what the problem actually is. Getting a diff of queries run and results produced like mysqltest does makes this step trivial (if the problem is wrong results from a query).
Also the --record functionality for generating .result files can be really good to ease the production of test case and also make it more likely the developer will not be tempted to insufficiently check the results (everything is checked by default). It does have the big risk of not properly checking that the committed .result is in fact correct, something I have seen more than once.
This does not really have to mean using separate files. I have in mind something like a Perl-based suite where I could do:
query <<SQL, <<RESULT SELECT a, b FROM t1 JOIN t2 ON (a = b) ORDER BY a, b; SQL RECORDME RESULT
and `mysql-test-run --record` would rewrite this as
query <<SQL, <<RESULT SELECT a, b FROM t1 JOIN t2 ON (a = b) ORDER BY a, b; SQL a b 1 1 2 2 4 4 RESULT
And in case of different result, it could output a diff of just the RESULT section of the failing query, along with the query. It could also give the correct line number in the .test file, avoiding the need to search the .test file for the query that produced the diff. I think that might make it even easier than separate result file.
Something like that would of course allow putting arbitrary Perl code in-between running queries (with lots of useful modules available) where this is more appropriate. This would eliminate the need for the mysqltest language for new test cases, which is one less language again.
And it would be quite easy to have a backwards-compatible emulation of the old .test and .result files (just a small wrapper that reads the .test and .result files and passes the contents to the query() function).
I just gave a MySQL University session on some of these problems. One of the issues I have seen is that people don't expend the extra energy to make a test informative (through comments explaining the test, defining what a 'real' <not in setup / teardown, but in the behavior being tested> failure would look like / what 'next-steps' seem logical for diagnosing a failure, etc. http://forge.mysql.com/wiki/How_to_Create_a_Test_Case I do like the thought of defining query / result portions a LOT. One of the biggest time-sinks we have is sifting through more complicated tests and piecing together what is going on. I think a properly revised test-suite would help if it presented the ability to capture much more information: Test purpose Defining key portions of the test - maybe defining code blocks of a test as setup / teardown / and testcode. Each testcode portion could have things like "on_failure - If the test fails here, it is a problem with xyz." Each 'real' portion of the test could define this. Failures in setup / teardown are still bad, but they might point to a less-serious failure and could be a Build host issue (full filesystem, etc). Probably the biggest wishes for a testing tool I have are: Power Simplicity / Elegance (make it easy to do anything, and allow complex 'thoughts' to be spelled out in an easily understood way) Ease of comprehending a test (opening it up and understanding what is being tested, how it is being done, and why a method has been chosen) Speed of failure analysis. (seeing a failure and being able to quickly discern what failed and what kind of a problem it is pointing to). Some extra wishlist items: Ease of cataloging / cross-referencing tests (ie run all 'Innodb' related tests and not having this rely solely on suite organization / naming patterns) Execution speed (we are likely to expand the test suite, so ensuring a lean run-time is always nice) Anyway, I saw the topic and couldn't resist chiming in. Thanks, Patrick.
One benefit is that you can usually run a test by just piping it trough the normal MySQL client, which simplifies testing and debugging a lot!
Yes, this should be kept for simple stuff, very important.
- Kristian.
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Hi!
"Patrick" == Patrick Crews <gleebix@gmail.com> writes:
<cut>
This does not really have to mean using separate files. I have in mind something like a Perl-based suite where I could do:
query <<SQL, <<RESULT SELECT a, b FROM t1 JOIN t2 ON (a = b) ORDER BY a, b; SQL RECORDME RESULT
and `mysql-test-run --record` would rewrite this as
query <<SQL, <<RESULT SELECT a, b FROM t1 JOIN t2 ON (a = b) ORDER BY a, b; SQL a b 1 1 2 2 4 4 RESULT
And in case of different result, it could output a diff of just the RESULT section of the failing query, along with the query. It could also give the correct line number in the .test file, avoiding the need to search the .test file for the query that produced the diff. I think that might make it even easier than separate result file.
The reason I don't like the above, compared to have different files are: - We may want to run the test with different engines and different configure options which would produce different results. In this case, having different files, one for each combination that gives a different result, is much easier than having the result in the test case. - Most of the current test cases you can pipe through 'mysql' and then compare the result files; A very good way when you are trying to find out what's wrong. If everything is in the same file, things like this gets to be cumbersome. What I *would* like to change ASAP is that the test file and result file is in the same directory so that in 'bzr gcommit' and in 'emacs directory' the test and result files comes directly after eachother! This has been on the mysqltest TODO for a long time but for some reason never get done, even if it's a trivial change!
Something like that would of course allow putting arbitrary Perl code in-between running queries (with lots of useful modules available) where this is more appropriate. This would eliminate the need for the mysqltest language for new test cases, which is one less language again.
We can alread do this.
And it would be quite easy to have a backwards-compatible emulation of the old .test and .result files (just a small wrapper that reads the .test and .result files and passes the contents to the query() function).
Patrick> I just gave a MySQL University session on some of these problems. One of Patrick> the issues I have seen is that people don't expend the extra energy to make Patrick> a test informative (through comments explaining the test, defining what a Patrick> 'real' <not in setup / teardown, but in the behavior being tested> failure Patrick> would look like / what 'next-steps' seem logical for diagnosing a failure, Patrick> etc. Patrick> http://forge.mysql.com/wiki/How_to_Create_a_Test_Case Agree this is a common problem. <cut> Patrick> I think a properly revised test-suite would help if it presented the ability Patrick> to capture much more information: I don't see a big reason to do 'totally revised' (ie, rewritten) test-suite. There is however a great need to simplify the current architecture and have another test system for performance and 'heavy duty' testing. Patrick> Test purpose Patrick> Defining key portions of the test - maybe defining code blocks of a test as Patrick> setup / teardown / and testcode. Each testcode portion could have things Patrick> like "on_failure - If the test fails here, it is a problem with xyz." Each Patrick> 'real' portion of the test could define this. Failures in setup / teardown Patrick> are still bad, but they might point to a less-serious failure and could be a Patrick> Build host issue (full filesystem, etc). I see the point of this, but fear that simple tests would be made to complex. In practice we already have this parts in most tests, but it's not properly formalized or clear in all cases. Patrick> Probably the biggest wishes for a testing tool I have are: Patrick> Power Patrick> Simplicity / Elegance (make it easy to do anything, and allow complex Patrick> 'thoughts' to be spelled out in an easily understood way) The phrase 'If we would smart enough to understand our brains, our brains would not be smart enough to understand it'. In other words, it's hard to do a complex problem simple. Do you think we can come up with one tool to do this? Patrick> Ease of comprehending a test (opening it up and understanding what is being Patrick> tested, how it is being done, and why a method has been chosen) Most of the tests are done to: - Test that a feature works according to specification - Test that a bug is fixed - To get more code covered For most of the above, it would be hard to get people to specify why a method was choosen. How do you envision this part? (Not beeing negative, just trying to understand) Patrick> Speed of failure analysis. (seeing a failure and being able to quickly Patrick> discern what failed and what kind of a problem it is pointing to). Which is not an easy task in itself. Patrick> Some extra wishlist items: Patrick> Ease of cataloging / cross-referencing tests (ie run all 'Innodb' related Patrick> tests and not having this rely solely on suite organization / naming Patrick> patterns) Agree. Patrick> Execution speed (we are likely to expand the test suite, so ensuring a lean Patrick> run-time is always nice) That's one of the bigest challanges. Patrick> Anyway, I saw the topic and couldn't resist chiming in. You are welcome! It's a very important topic. Regards, Monty
Michael Widenius <monty@askmonty.org> writes:
What I *would* like to change ASAP is that the test file and result file is in the same directory so that in 'bzr gcommit' and in 'emacs directory' the test and result files comes directly after eachother! This has been on the mysqltest TODO for a long time but for some reason never get done, even if it's a trivial change!
If you make a (small) worklog task for this I should get to it eventually, as you say it shouldn't take too long. - Kristian.
participants (3)
-
Kristian Nielsen -
Michael Widenius -
Patrick Crews