[Maria-discuss] New testsuite option
Hello, I take care of MariaDB in Fedora & RHEL. We run the testsuite there. Every time, in every build, with --big-test. Time to time it happens that some of the tests fail. For example during a rebuild. For such cases, I'd use a way to tell MTR what is a unstable test treshold, which means "how many tests can fail to still count the testsuite run as a success". Most likely, I'd use it with value from 1-5. Any larger group of failing tests needs to get my attention. Anyway I still could spare myself of the unstable tests. -- (Asking before filling any JIRA feature request) 1) Is there already some way to do that? 2) Do you find such feature usefull? 3) Any ideas how to call such option? --max-test-fail is already used -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat
Hi, Michal! On Jul 17, Michal Schorm wrote:
Hello,
I take care of MariaDB in Fedora & RHEL.
We run the testsuite there. Every time, in every build, with --big-test. Time to time it happens that some of the tests fail. For example during a rebuild.
For such cases, I'd use a way to tell MTR what is a unstable test treshold, which means "how many tests can fail to still count the testsuite run as a success".
Most likely, I'd use it with value from 1-5. Any larger group of failing tests needs to get my attention. Anyway I still could spare myself of the unstable tests.
--
(Asking before filling any JIRA feature request)
1) Is there already some way to do that?
2) Do you find such feature usefull?
3) Any ideas how to call such option? --max-test-fail is already used
What if we just make mtr to do `exit 2` if it exists because of --max-test-fail ? Then you'll be able to do, say ./mtr --max-test-fail=5; test $? le 2 This is something that might (perhaps) have other use cases too. Regards, Sergei Chief Architect MariaDB and security@mariadb.org
What if we just make mtr to do `exit 2` if it exists because of --max-test-fail ?
I'm afraid that won't help me in any way. In such case, still, no matter how many tests fail (if >0), the testsuite will fail. With different exit code maybe, but it will fail. I'm looking for an option which would say "If only <N tests failed, return success anyway". That would spare me of rebuilds beacuse on or two unstable tests failed. -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat
Hi, Michal! On Aug 13, Michal Schorm wrote:
What if we just make mtr to do `exit 2` if it exists because of --max-test-fail ?
I'm afraid that won't help me in any way. In such case, still, no matter how many tests fail (if >0), the testsuite will fail. With different exit code maybe, but it will fail.
I've had an example in previous email, if mtr would do `exit 2` then you'll need to run (from the spec file, I presume) something like ./mtr --max-test-fail=5; test $? le 2
I'm looking for an option which would say "If only <N tests failed, return success anyway".
And this is exactly what it'll be. The command above returns success if only <5 tests failed. Would that work for you? Regards, Sergei Chief Architect MariaDB and security@mariadb.org
participants (2)
-
Michal Schorm -
Sergei Golubchik