Hi Elene, In another thread you mentioned that there's alternative solution used by Otto to maintain this list. I'm generally fine with either solution. This patch as such looks Ok. How do we qualify tests unstable? Failed since the beginning of the year, modified in last version, failed in Debian? Who shall maintain this list? What is the procedure of removing test case from unstable? I'm not completely happy that 10.0 is GA, TokuDB is "stable", but (almost) all TokuDB are disabled. This is rather a bad message to our users. Regards, Sergey On Mon, Aug 22, 2016 at 03:04:26PM +0300, Elena Stepanova wrote:
Hi Sergey (Vojtovich),
Please review the patch that we've discussed at length. Serg, FYI.
The length of the "unstable" list shouldn't scare anyone. At the end, this task should cause no change in our current procedure, the tests will still be run in buildbot and hopefully on developers' local machines (and every external user who wants to run them can do it easily). The change should mainly help those who run tests as a part of their build process.
The lists can be tuned later, now they have been created generously, to solve the most urgent issue in shortest time possible.
Please note that it's still not a 100% guarantee that nothing will fail sporadically, I've already got an absolutely new random failure after disabling all known ones, so it will still require some iterations.
Regards, Elena
-------- Forwarded Message -------- Subject: [Commits] 5ac5050: MDEV-10604 Disable unstable MTR tests by default Date: Mon, 22 Aug 2016 14:44:30 +0300 (EEST) From: elenst@montyprogram.com Reply-To: maria-developers@lists.launchpad.net To: commits@mariadb.org
revision-id: 5ac505064a75508213d4dbce388ff5ee62d1ad56 (mariadb-10.0.26-92-g5ac5050) parent(s): 05f61ba46046ca835071a73b9255e787dcce9255 author: Elena Stepanova committer: Elena Stepanova timestamp: 2016-08-22 14:38:27 +0300 message:
MDEV-10604 Disable unstable MTR tests by default
- add new syntax to disabled.def in MTR to distinguish "unstable" tests from disabled tests; - allow wildcards in test names in disabled.def; - add a new option enable-unstable and env option MTR_ENABLE_DISABLED; - mark "unstable" tests which have failed since the beginning of the year in buildbot on main 10.0 branch; - mark "unstable" tests which fail in Debian builds; - mark "unstable" tests which have been changed during the last two releases ...skip...
Regards, Sergey