- developers - lists.mariadb.org

Re: [Maria-developers] 8c34a242aa2: MDEV-20787: Script dgcov.pl does not work
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Anel! On Oct 21, Anel Husakovic wrote: > revision-id: 8c34a242aa2 (mariadb-10.2.31-478-g8c34a242aa2) > parent(s): 01ffccd6a4e > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-06 00:00:58 +0200 > message: > > MDEV-20787: Script dgcov.pl does not work > > - This patch is done after doing test on 00f9df29224ada9cdbb796c99f4 > where difference was found between versions of gcov. don't refer to commits that are not pushed and are not going to be pushed. Because if you push this one you'll have a commit forever in the history that refers to an non-existing commit. It isn't helpful. > - This patch doesn't take gcov versions in account since is extracting > filename of a `.gcov` file, which is generated with `gcov -i > <source-file>.gcda` during `dgcov.pl --generate`. > - There is a check if file exists. > Reason for that is because some generated `gcov` files are outliers from above > rule (that a file generated in intermediate format(gcov) has the same source name). > Example 1: `aestables.cpp.gcda` is not generating `aestables.cpp.gcov`, > but only the files used in #include `modes.hpp.gcov runtime.hpp.gcov` > Example 2: `my_sha256.cc.gcda` is generating `my_sha.ic.gcov` > - Make sure to test `MTR_BINDIR` which gets set during out-of-source > build and generate error when running `./mysql-test/mtr --gcov alias` What is a problem with out of source builds? If they don't work, perhaps it would be better to fix dgcov to work with out of source builds? > - During the testing: > 10.1 results are stored in > - `<basedir>//mysql-test-gcov.msg` and errors in > - `<basedir>//mysql-test-gcov.err` > 10.2 results should be stored in > `./mysql-test/var/last_changes.dgcov`, but only empty results are > obtained (tests `1st` and `alias`). I don't understand this ^^^ paragraph. And also because your patch is 10.2+ so who cares what it does on 10.1? > diff --git a/mysql-test/dgcov.pl b/mysql-test/dgcov.pl > index fbc5540e697..f2bbdad4d20 100755 > --- a/mysql-test/dgcov.pl > +++ b/mysql-test/dgcov.pl > @@ -161,8 +161,14 @@ sub gcov_one_file { > system($cmd)==0 or die "system($cmd): $? $!"; > } > > - # now, read the generated file > - open FH, '<', "$_.gcov" or die "open(<$_.gcov): $!"; > + (my $filename = $_)=~ s/\.[^.]+$//; # remove extension In your previous patch you either removed the extension or not based on gcov version. Why did you abandon this logic? > + my $gcov_file_path= $File::Find::dir."/$filename.gcov"; File::Find documentation seems to be saying that $File::Find::dir ends with a slash (/) so you don't need to prefix $filename.gcov with it. > + if (! -f $gcov_file_path) > + { > + return; > + } may be return unless -f $gcov_file_path it's perl after all :) > + open FH, '<', "$gcov_file_path" or die "open(<$gcov_file_path): $!"; you don't need double quotes around "$gcov_file_path" - they don't serve any purpose. > + > my $fname; > while (<FH>) { > chomp; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f83c1615d97: MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rucha! On Oct 21, Rucha Deodhar wrote: > revision-id: f83c1615d97 (mariadb-10.1.43-240-gf83c1615d97) > parent(s): 7e9ffc69ecd > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-08-05 00:33:03 +0530 > message: > > MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status > upon HANDLER READ > > Analysis: The error state is not stored while checking condition and key name. > Fix: Return true while checking condition and key name if error is reported > because geometry object can't be created from the data in the index value for > HANDLER READ. > > diff --git a/mysql-test/suite/handler/aria.test b/mysql-test/suite/handler/aria.test > index 1913d2b791c..50ef43ce5b2 100644 > --- a/mysql-test/suite/handler/aria.test > +++ b/mysql-test/suite/handler/aria.test > @@ -80,3 +80,23 @@ HANDLER t1 CLOSE; > DROP TABLE t1; > > --echo End of 5.1 tests > + > +--echo # > +--echo # 10.1 Test > +--echo # > +--echo # MDEV-20207: Assertion `! is_set()' failed in > +--echo # Diagnostics_area::set_eof_status upon HANDLER READ > +--echo # > + > +DROP TABLE IF EXISTS t1; there's no need to start the test from DROP TABLE IF EXISTS t1; historically people customarily did that. before mtr had after-test checks. but now all tests are supposed to clean up after itself, you do not need to worry about tests leaving tables around. > + > +CREATE TABLE t1 (a POINT, KEY(a)); > +HANDLER t1 OPEN h; > + > +--error ER_CANT_CREATE_GEOMETRY_OBJECT > +HANDLER h READ a = (0); I don't think it should be an error. SELECT * FROM t1 WHERE a=(0) -- this does not fail, it just doesn't find anything. I think HANDLER should do the same. > + > +HANDLER h CLOSE; > +DROP TABLE t1; note that all handler test files source handler.inc. that is, if you want a test to run for all engines, you do not need to add it to all individual test files, add it to handler.inc Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e3f45b2f9ea: MDEV-10267 Add ngram fulltext parser plugin
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rinat! On Oct 21, Rinat Ibragimov wrote: > > > + > > > + // Avoid creating partial n-grams by stopping at word boundaries. > > > + // Underscore is treated as a word character too, since identifiers in > > > + // programming languages often contain them. > > > + if (!is_alnum(char_type) && !is_underscore(cs, next, end)) { > > > + start= next + char_len; > > > + next= start; > > > + n_chars= 0; > > > + continue; > > > > I don't think it's a good idea. On the opposite, I would rather create > > ngrams over word boundaries, so that, say, I'd split "n-gram plugin" to > > > > "n-g", "-gr", "gra", "ram", "am ", "m p", " pl", "plu", "lug", "ugi", "gin". > > I see how this can increase index size: more different strings are > included. Implementation will become a tiny bit simpler. But can it > make searching better somehow? Yes, the idea is that it'll rank "One can use n-gram plugin in MariaDB fulltext search" higher than "Plugins are useful for extending functionality of a program" because some ngrams encode that "n-gram" and "plugin" words must be together as a phrase. > I can't decide. From my point of view, the current approach is fine. > Please pick a variant, and I'll try to implement that. No, I cannot guess which approach will produce more relevant searches. Implement something and then we test what works better > > > + } > > > + > > > + next += char_len; > > > + n_chars++; > > > + > > > + if (n_chars == ngram_token_size) { > > > + bool_info->position= static_cast<uint>(start - doc); > > > > nope, we don't have MYSQL_FTPARSER_FULL_BOOLEAN_INFO::position and this > > plugin is definitely not the reason to break the API and add it. > > There is a patch in the patchset that adds the "position" field. Without > that field, > it's possible to trigger an assertion in InnoDB's FTS code by executing: > > INSTALL PLUGIN simple_parser SONAME 'mypluglib'; > CREATE TABLE articles( > a TEXT DEFAULT NULL, > b TEXT DEFAULT NULL, > FULLTEXT (a, b) WITH PARSER simple_parser > ) ENGINE=InnoDB; > INSERT INTO articles VALUES ('111', '1234 1234 1234'); > DROP TABLE articles; > UNINSTALL PLUGIN simple_parser; > > Same assertion may be triggered by any fts plugin. Are you sure you > want that patch to be removed? I don't think InnoDB assert is a reason to extend the fulltext plugin API. What about this fix instead: diff --git a/storage/innobase/fts/fts0fts.cc b/storage/innobase/fts/fts0fts.cc --- a/storage/innobase/fts/fts0fts.cc +++ b/storage/innobase/fts/fts0fts.cc @@ -4669,7 +4669,7 @@ fts_tokenize_add_word_for_parser( ut_ad(boolean_info->position >= 0); position = boolean_info->position + fts_param->add_pos; */ - position = fts_param->add_pos; + position = fts_param->add_pos++; fts_add_token(result_doc, str, position); this fixed the crash for me. > > > + n_chars= ngram_token_size - 1; > > > + is_first= false; > > > + } > > > + } > > > + > > > + // If nothing was emitted yet, emit the remainder. > > > > what reminder, what is it for? > > It's "remainder", with "a" before "i". Remainder of the processed > string. That way strings that are too short for having even a single > n-gram, will still generate some index entries and will be > discoverable by exact queries. Perhaps the comment should say it? That if the string too short, it'll be emitted here? > > > + case MYSQL_FTPARSER_FULL_BOOLEAN_INFO: > > > + // Reusing InnoDB's boolean mode parser to handle all special characters. > > > + while (fts_get_word(cs, &start, end, &word, &bool_info)) { > > > > nope. InnoDB might not even be compiled it. Use param->mysql_parse() > > Copied fts_get_word() implementation into the plugin code. > > As far as I understand, param->mysql_parse() cannot be used here as it > splits a string into words like default fulltext search does. This > plugin however must generate n-grams. Of course, it can. Note that fts_get_word() doesn't generate n-grams either, it gets the whole word and the n-gram plugin later splits it into n-grams. Similarly param->mysql_parse() will extract words for you and you'll split them into n-grams. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Please assign PRs that modify debian/ contents for Otto to review
by Otto Kekäläinen 20 Oct '20

20 Oct '20

Hello! While working on putting MariaDB 10.5.5 into Debian I noticed that there were a couple of commits that landed more or less on the same day as the release was made and "broke" it just before release. Now there are surprises in a stable release I need to fix and the situation puts extra strain on me. It is good that many people participate in maintaining the contents of debian/ in the MariaDB Server, because I don't have time to do everything. I would however appreciate that when changes are made, I would be given a chance to review them. I will review them post-release anyway and it would save a lot of trouble, if we can fix stuff during the development cycle and not having to patch a post-release in order to be able to put it forward to Debian/Ubuntu. So, please make PRs of any debian/ changes you make, and feel free to add me as the reviewer (or one of the reviewers). This policy can also be automated using a .github/CODEOWNERS file, so I made a PR about it at https://github.com/MariaDB/server/pull/1650 Thanks! PS. I will deliver a MariaDB 10.5 and MariaDB in Debian status talk at https://debconf20.debconf.org/talks/98-whats-new-in-mariadb-server-105-and-… in case anybody is interested to listen in. -- Follow me at @ottokekalainen

2 3

Re: [Maria-developers] db1fbafbead: MDEV-23935: Fix warnings generated during compilation of plugin/auth_pam/testing/pam_mariadb_mtr.c on MacOS
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! ok to push On Oct 19, Dmitry Shulga wrote: > revision-id: db1fbafbead (mariadb-10.4.11-420-gdb1fbafbead) > parent(s): c2ac0ce1f02 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-11 21:51:45 +0700 > message: > > MDEV-23935: Fix warnings generated during compilation of plugin/auth_pam/testing/pam_mariadb_mtr.c on MacOS > > During build server 10.4 on MacOS warnings like the following one > are generated on compiling the file plugin/auth_pam/testing/pam_mariadb_mtr.c > > server-10.4/plugin/auth_pam/testing/pam_mariadb_mtr.c:25:22: error: > initializing 'char *' with an expression of type 'const char [23]' > discards qualifiers [-Werror,-Wincompatible-pointer-types-discards-qualifiers] > { PAM_TEXT_INFO, "Challenge input first." } > > The reason of the warnings is that the data member pam_message::msg is declared > as 'char *' on MacOS but initializer of the data member is 'const char *'. > > To eliminate warnings the compiler option > -Wno-incompatible-pointer-types-discards-qualifiers > has been added to the compiler flags used for compiling the file > pam_mariadb_mtr.c. > > --- > plugin/auth_pam/testing/CMakeLists.txt | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/plugin/auth_pam/testing/CMakeLists.txt b/plugin/auth_pam/testing/CMakeLists.txt > index 889cb524402..c8d2e3cbb97 100644 > --- a/plugin/auth_pam/testing/CMakeLists.txt > +++ b/plugin/auth_pam/testing/CMakeLists.txt > @@ -4,6 +4,12 @@ ADD_LIBRARY(pam_mariadb_mtr MODULE pam_mariadb_mtr.c) > SET_TARGET_PROPERTIES (pam_mariadb_mtr PROPERTIES PREFIX "") > TARGET_LINK_LIBRARIES(pam_mariadb_mtr pam) > > +IF(APPLE) > + SET_SOURCE_FILES_PROPERTIES( > + pam_mariadb_mtr.c > + PROPERTY COMPILE_FLAGS "-Wno-incompatible-pointer-types-discards-qualifiers") > +ENDIF() > + > SET(dest DESTINATION "${INSTALL_MYSQLTESTDIR}/suite/plugins/pam" COMPONENT Test) > INSTALL(TARGETS pam_mariadb_mtr ${dest}) > INSTALL(FILES mariadb_mtr.conf RENAME mariadb_mtr ${dest}) > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 727d97feb99: MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! plugin code should stay in the plugin directory. please move the check to plugin/auth_pam/CMakeLists.txt Also, what about an alternative fix: #ifdef HAVE_POSIX_GETGROUPLIST typedef gid_t my_gid_t; #else typedef int my_gid_t; #end would that help but with less ifdefs? On Oct 19, Dmitry Shulga wrote: > revision-id: 727d97feb99 (mariadb-10.2.31-489-g727d97feb99) > parent(s): 8984d779103 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-09 20:35:48 +0700 > message: > > MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS > > Compiler warnings listed below are generated during server build on MacOS: > > 88%] Building C object plugin/auth_pam/CMakeFiles/pam_user_map.dir/mapper/pam_user_map.c.o > /Users/shulga/projects/mariadb/server-10.2/plugin/auth_pam/mapper/pam_user_map.c:87:41: error: passing > 'gid_t *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types > with different sign [-Werror,-Wpointer-sign] > if (getgrouplist(user, user_group_id, loc_groups, &ng) < 0) > ^~~~~~~~~~ > /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/unistd.h:650:43: note: > passing argument to parameter here > int getgrouplist(const char *, int, int *, int *); > ^ > /Users/shulga/projects/mariadb/server-10.2/plugin/auth_pam/mapper/pam_user_map.c:95:46: error: passing > 'gid_t *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types > with different sign [-Werror,-Wpointer-sign] > (void) getgrouplist(user, user_group_id, loc_groups, &ng); > ^~~~~~~~~~ > /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/includ > e/unistd.h:650:43: note: > passing argument to parameter here > int getgrouplist(const char *, int, int *, int *); > ^ > 2 errors generated. > > In case MariaDB serer is build with -DCMAKE_BUILD_TYPE=Debug it results in > uild error. > > The reason of compiler warnings is that declaration of the Posix C API function > getgrouplist() on MacOS differs from declration of getgrouplist() proposed > by Posix. > > To suppress this compiler warning cmake configure was adapted to detect what > kind of getgrouplist() function is declared on the building platform and > set the macros HAVE_POSIX_GETGROUPLIST in case the building platform suppors > Posix compatible interface of the getgrouplist() function. Depending on > whether this macros is set the compatible type of arguments is used to pass > parameter values to the function. > > --- > config.h.cmake | 1 + > configure.cmake | 22 ++++++++++++++++++++++ > plugin/auth_pam/mapper/pam_user_map.c | 13 ++++++++++++- > 3 files changed, 35 insertions(+), 1 deletion(-) > > diff --git a/config.h.cmake b/config.h.cmake > index 0e19dd44694..07e4647f0bf 100644 > --- a/config.h.cmake > +++ b/config.h.cmake > @@ -188,6 +188,7 @@ > #cmakedefine HAVE_POSIX_FALLOCATE 1 > #cmakedefine HAVE_LINUX_FALLOC_H 1 > #cmakedefine HAVE_FALLOC_PUNCH_HOLE_AND_KEEP_SIZE 1 > +#cmakedefine HAVE_POSIX_GETGROUPLIST 1 > #cmakedefine HAVE_PREAD 1 > #cmakedefine HAVE_PAUSE_INSTRUCTION 1 > #cmakedefine HAVE_FAKE_PAUSE_INSTRUCTION 1 > diff --git a/configure.cmake b/configure.cmake > index 58e1111bfc7..c6db3ecb912 100644 > --- a/configure.cmake > +++ b/configure.cmake > @@ -1031,3 +1031,25 @@ IF(NOT MSVC) > HAVE_FALLOC_PUNCH_HOLE_AND_KEEP_SIZE > ) > ENDIF() > + > +# Check whether getgrouplist uses git_t for second and third arguments. > +SET(CMAKE_REQUIRED_FLAGS -Werror) > +CHECK_C_SOURCE_COMPILES( > +" > +#ifdef HAVE_GRP_H > +#include <grp.h> > +#endif > +#ifdef HAVE_UNISTD_H > +#include <unistd.h> > +#endif > +int main() { > + char *arg_1; > + gid_t arg_2, arg_3; > + int arg_4; > + (void)getgrouplist(arg_1,arg_2,&arg_3,arg_4); > + return 0; > +} > +" > +HAVE_POSIX_GETGROUPLIST > +) > +SET(CMAKE_REQUIRED_FLAGS) > diff --git a/plugin/auth_pam/mapper/pam_user_map.c b/plugin/auth_pam/mapper/pam_user_map.c > index 9d7ed53f8b1..b0d2affac5c 100644 > --- a/plugin/auth_pam/mapper/pam_user_map.c > +++ b/plugin/auth_pam/mapper/pam_user_map.c > @@ -31,6 +31,8 @@ These comments are written to the syslog as 'authpriv.debug' > and usually end up in /var/log/secure file. > */ > > +#include <my_config.h> > + > #include <stdlib.h> > #include <stdio.h> > #include <ctype.h> > @@ -72,8 +74,13 @@ static const char debug_keyword[]= "debug"; > > static int populate_user_groups(const char *user, gid_t **groups) > { > +#ifdef HAVE_POSIX_GETGROUPLIST > gid_t user_group_id; > gid_t *loc_groups= *groups; > +#else > + int user_group_id; > + int *loc_groups= (int*)*groups; > +#endif > int ng; > > { > @@ -88,12 +95,16 @@ static int populate_user_groups(const char *user, gid_t **groups) > { > /* The rare case when the user is present in more than */ > /* GROUP_BUFFER_SIZE groups. */ > +#ifdef HAVE_POSIX_GETGROUPLIST > loc_groups= (gid_t *) malloc(ng * sizeof (gid_t)); > +#else > + loc_groups= (int *) malloc(ng * sizeof (gid_t)); > +#endif > if (!loc_groups) > return 0; > > (void) getgrouplist(user, user_group_id, loc_groups, &ng); > - *groups= loc_groups; > + *groups= (gid_t*)loc_groups; > } > > return ng; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5e71a3e0df7: MDEV-23925: Fix warnings generated during compilation of mysys_ssl/openssl.c on MacOS
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! ok to push (again, comments below) On Oct 19, Dmitry Shulga wrote: > revision-id: 5e71a3e0df7 (mariadb-10.2.31-489-g5e71a3e0df7) > parent(s): 8984d779103 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-09 20:55:14 +0700 > message: > > MDEV-23925: Fix warnings generated during compilation of mysys_ssl/openssl.c on MacOS > > Compiler warnings listed below are generated during server build on MacOS: > In file included from server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > In file included from /usr/local/include/openssl/evp.h:16: > In file included from /usr/local/include/openssl/bio.h:20: > /usr/local/include/openssl/crypto.h:206:10: warning: 'CRYPTO_cleanup_all_ex_data' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:46:9: note: previous definition is here > ^ > In file included from /mariadb/server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > /usr/local/include/openssl/evp.h:501:10: warning: 'EVP_MD_CTX_init' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:33:9: note: previous definition is here > ^ > In file included from /mariadb/server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > /usr/local/include/openssl/evp.h:627:11: warning: 'EVP_CIPHER_CTX_init' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:35:9: note: previous definition is here > ^ > In file included from /mariadb/server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > /usr/local/include/openssl/evp.h:866:11: warning: 'EVP_cleanup' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:44:9: note: previous definition is here > ^ > 4 warnings generated. generally these long commit comments are not any more clear than if you'd only listed one compilation failure. > > In case MariaDB serer is build with -DCMAKE_BUILD_TYPE=Debug it results in > build error. > > The reason of compiler warnings is that header file <ssl_compat.h> > included before the openssl system header files. File ssl_compat.h > contains some macros with the name as SSL API functions declared > in the openssl system header files. It resulted in duplicate > symbols thats produces compiler warnings. > > To fix the issue the header file ssl_compat.h should be included > after a line where openssl system header is included. > > --- > mysys_ssl/openssl.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/mysys_ssl/openssl.c b/mysys_ssl/openssl.c > index f3dc8f4277c..a11db113b14 100644 > --- a/mysys_ssl/openssl.c > +++ b/mysys_ssl/openssl.c > @@ -15,7 +15,6 @@ > Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > > #include <my_global.h> > -#include <ssl_compat.h> > > /* > The check is only done for OpenSSL 1.1.x. > @@ -25,13 +24,14 @@ > */ > > #ifndef HAVE_OPENSSL11 > +#include <ssl_compat.h> > int check_openssl_compatibility() > { > return 0; > } > #else > #include <openssl/evp.h> > - > +#include <ssl_compat.h> > static uint testing, alloc_size, alloc_count; > > static void *coc_malloc(size_t size, const char *f __attribute__((unused)), > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b92597c220f: MDEV-23564: CMAKE failing due to deprecated Apple GSS method
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! ok to push. but see comments below On Oct 19, Dmitry Shulga wrote: > revision-id: b92597c220f (mariadb-10.2.31-479-gb92597c220f) > parent(s): 577c61e8be0 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-08 00:48:47 +0700 > message: > > MDEV-23564: CMAKE failing due to deprecated Apple GSS method > > Some GSS-API functions like gss_import_name(), gss_release_buffer() > used in plugin/auth_gssapi and libmariadb/plugins/auth are marked > as deprecated in MacOS starting from version 10.14+. It results in > extra warnings output on server building. > > To eliminate extra warnings the flag '-Wno-deprecated-declarations' > has been added to compiler invocation string for those source > files that invoke deprecated GSS-API functions. > > diff --git a/libmariadb b/libmariadb > index a746c3af449..fd0dc672309 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit a746c3af449a8754e78ad7971e59e79af7957cdb > +Subproject commit fd0dc6723095f066648a7c3b2a48523b3002e392 be careful when pushing this commit. you'd need something like 1. rebase your libmariadb fix on top of 3.1, push into 3.1 2. rebase your server fix on top of 10.2 2. update submodule reference in the server repo 3. amend your commit fix and push > diff --git a/plugin/auth_gssapi/CMakeLists.txt b/plugin/auth_gssapi/CMakeLists.txt > index bca4f5af3a1..7a3c32a00b1 100644 > --- a/plugin/auth_gssapi/CMakeLists.txt > +++ b/plugin/auth_gssapi/CMakeLists.txt > @@ -18,6 +18,13 @@ ELSE() > SET(GSSAPI_SERVER gssapi_server.cc) > SET(GSSAPI_ERRMSG gssapi_errmsg.cc) > > + IF(APPLE) > + SET_SOURCE_FILES_PROPERTIES( > + ${GSSAPI_CLIENT} ${GSSAPI_SERVER} ${GSSAPI_ERRMSG} > + PROPERTY COMPILE_FLAGS "-Wno-deprecated-declarations") > +# SET_SOURCE_FILES_PROPERTIES(${GSSAPI_SERVER} PROPERTY COMPILE_FLAGS "-Wno-deprecated-declarations") > +# SET_SOURCE_FILES_PROPERTIES(${GSSAPI_ERRMSG} PROPERTY COMPILE_FLAGS "-Wno-deprecated-declarations") you don't need these commented lines anymore, do you? Is it possible to use a compiler on APPLE that doesn't understand -Wno-deprecated-declarations? If not - your fix it ok. If it is - better use MY_CHECK_AND_SET_COMPILER_FLAG > + ENDIF() > SET(CMAKE_REQUIRED_INCLUDES ${GSSAPI_INCS}) > SET(CMAKE_REQUIRED_LIBRARIES ${GSSAPI_LIBS}) > INCLUDE(CheckCXXSymbolExists) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2d73406dad8: MDEV-16481: set global system_versioning_asof=sf() crashes in specific case
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Nikita! ok to push (better late than never :) On Oct 19, Nikita Malyavin wrote: > revision-id: 2d73406dad8 (mariadb-10.3.16-39-g2d73406dad8) > parent(s): 609f358a9bd > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Sergei Golubchik <serg(a)mariadb.com> > timestamp: 2020-10-19 17:37:04 +0200 > message: > > MDEV-16481: set global system_versioning_asof=sf() crashes in specific case > > * sys_vars.h: add `MYSQL_TIME` field to `set_var::save_result` > * sys_vars.ic: get rid of calling `var->value->get_date()` from `Sys_var_vers_asof::update()` > * versioning.sysvars: add test; remove double warning > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e3f45b2f9ea: MDEV-10267 Add ngram fulltext parser plugin
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Rinat! On Oct 19, Rinat Ibragimov wrote: > revision-id: e3f45b2f9ea (mariadb-10.2.31-368-ge3f45b2f9ea) > parent(s): 44c48cefdbe > author: Rinat Ibragimov <ri(a)tempesta-tech.com> > committer: Rinat Ibragimov <ri(a)tempesta-tech.com> > timestamp: 2020-08-25 01:59:06 +0300 > message: > > MDEV-10267 Add ngram fulltext parser plugin > > Ngram is a fulltext parser plugin that splits words into overlapping > segments of fixed lengths. For example, if 3-grams are in use, string > "abcdef" is split into "abc", "bcd", "cde", "def", thus allowing an > efficient substring search for CJK texts where splitting by words in not > feasible. Only word characters are going into n-grams. Spaces and > punctuation characters are treated as separators. > > diff --git a/plugin/fulltext/CMakeLists.txt b/plugin/fulltext/CMakeLists.txt > index 3c37666cb3e..9704733a9aa 100644 > --- a/plugin/fulltext/CMakeLists.txt > +++ b/plugin/fulltext/CMakeLists.txt > @@ -18,3 +18,7 @@ set(CMAKE_CXX_FLAGS_RELWITHDEBINFO "${CMAKE_CXX_FLAGS_RELWITHDEBINFO} -DNDEBUG") > > MYSQL_ADD_PLUGIN(ftexample plugin_example.c > MODULE_ONLY MODULE_OUTPUT_NAME "mypluglib" COMPONENT Test) > + > +# n-gram parser > +MYSQL_ADD_PLUGIN(ftngram ngram_parser/plugin_ngram.cc > + MODULE_ONLY MODULE_OUTPUT_NAME "fulltext_ngram") why to put the plugin under plugin/fulltext/ngram_parser/ ? it's only one file, just put it in plugin/fulltext/ > diff --git a/plugin/fulltext/mysql-test/ngram/disabled.def b/plugin/fulltext/mysql-test/ngram/disabled.def > new file mode 100644 > index 00000000000..0ca9dd33ef7 > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/disabled.def > @@ -0,0 +1 @@ > +ngram_gb18030 : GB18030 character set is not supported yet. I don't think there's any need to add tests for unsupported charsets. The plugin is trivial, no need to test it for every possible charset that we have and don't have. > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_basic.inc b/plugin/fulltext/mysql-test/ngram/ngram_basic.inc > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_basic.inc > @@ -0,0 +1,25 @@ > +--echo > +--eval CREATE TABLE t (lang TEXT, str TEXT, FULLTEXT(str) WITH PARSER ngram) ENGINE=$engine CHARSET $charset > + > +# Some character are not possible to represent in some encodings. Ignore them. > +--disable_warnings > +--disable_query_log > +# http://www.gerolf.org/doc/polyglot/polyglot.htm. > +INSERT IGNORE INTO t VALUES > + ('Chinese', '他们为什么不说中文？'), > + ('English', 'Why can''t they just speak English?'), > + ('Japanese', 'なぜ、みんな日本語を話してくれないのか？'), > + ('Korean', '세계의 모든 사람들이 한국어를 이해한다면 얼마나 좋을까?'); > +--enable_query_log > +--enable_warnings > + > +SELECT * FROM t ORDER BY lang; > + > +SELECT * FROM t WHERE MATCH(str) AGAINST('之一' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('数据' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('zzz' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('gli' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('ない' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('람들' IN BOOLEAN MODE) ORDER BY lang; > + > +DROP TABLE t; > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_basic.test b/plugin/fulltext/mysql-test/ngram/ngram_basic.test > new file mode 100644 > index 00000000000..eaa8829c3ee > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_basic.test > @@ -0,0 +1,25 @@ > +# Ngram fulltext parser basic functionality. > + > +SET NAMES utf8mb4; > +SELECT @@ngram_token_size; > + > +let $engine_index= 3; > +while ($engine_index) { > + if ($engine_index == 3) { let $engine= InnoDB; } > + if ($engine_index == 2) { let $engine= MyISAM; } > + if ($engine_index == 1) { let $engine= Aria; } > + > + let $charset_index= 6; > + while ($charset_index) { > + if ($charset_index == 6) { let $charset= utf8mb4; } > + if ($charset_index == 5) { let $charset= big5; } > + if ($charset_index == 4) { let $charset= gbk; } > + if ($charset_index == 3) { let $charset= gb2312; } > + if ($charset_index == 2) { let $charset= sjis; } > + if ($charset_index == 1) { let $charset= ujis; } > + > + source ngram_basic.inc; > + dec $charset_index; > + } eh. unroll the loop please, write let $engine= InnoDB; let $charset= utf8mb4; source ngram_basic.inc; let $charset= big5; source ngram_basic.inc; ... Or, better: CREATE TABLE t (lang TEXT, str TEXT, FULLTEXT(str) WITH PARSER ngram) ENGINE=InnoDB CHARSET utf8mb4; source ngram_basic.inc; CREATE TABLE t (lang TEXT, str TEXT, FULLTEXT(str) WITH PARSER ngram) ENGINE=InnoDB CHARSET big5; source ngram_basic.inc; ... > + dec $engine_index; > +} > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_missing_plugin.result b/plugin/fulltext/mysql-test/ngram/ngram_missing_plugin.result > new file mode 100644 > index 00000000000..aa17b1d6053 > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_missing_plugin.result > @@ -0,0 +1,23 @@ there are tests for the missing fulltest parser plugin. they don't depend on the individial plugin (that is, btw, *not installed*). No need to repeat them for every plugin, remove this test. > +SET NAMES utf8mb4; > +CREATE TABLE t (str TEXT, FULLTEXT(str) WITH PARSER ngram) > +ENGINE=InnoDB; > +INSERT INTO t VALUES ('Hello World'), ('abcdef ghijklm'); > +# Restart with ngram plugin disabled. > +SET NAMES utf8mb4; > +SELECT * FROM t WHERE MATCH(str) AGAINST('bcd') ORDER BY str; > +ERROR HY000: Plugin 'ngram' is not loaded > +SELECT * FROM t ORDER BY str; > +ERROR HY000: Plugin 'ngram' is not loaded > +INSERT INTO t VALUES ('something'); > +ERROR HY000: Plugin 'ngram' is not loaded > +# Restart with ngram plugin enabled again. > +SET NAMES utf8mb4; > +SELECT * FROM t WHERE MATCH(str) AGAINST('bcd') ORDER BY str; > +str > +abcdef ghijklm > +SELECT * FROM t ORDER BY str; > +str > +abcdef ghijklm > +Hello World > +INSERT INTO t VALUES ('something'); > +DROP TABLE t; > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_token_size.result b/plugin/fulltext/mysql-test/ngram/ngram_token_size.result > new file mode 100644 > index 00000000000..7043e147d5b > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_token_size.result > @@ -0,0 +1,109 @@ > +# Check 'ngram_token_size' variable. > +# > +# It's a global variable. > +SELECT @@global.ngram_token_size; > +@@global.ngram_token_size > +2 > +SHOW GLOBAL VARIABLES LIKE 'ngram_token_size'; > +Variable_name Value > +ngram_token_size 2 > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 2 > +# > +# Not a session variable. > +SELECT @@session.ngram_token_size; > +ERROR HY000: Variable 'ngram_token_size' is a GLOBAL variable > +SHOW SESSION VARIABLES LIKE 'ngram_token_size'; > +Variable_name Value > +ngram_token_size 2 > +SELECT * FROM information_schema.session_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 2 > +# > +# 0 is outside allowed range [1, 10], so it's changed to 1. > +SET GLOBAL ngram_token_size= 0; > +Warnings: > +Warning 1292 Truncated incorrect ngram_token_size value: '0' > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 1 > +# > +# 10 is an acceptable value. > +SET GLOBAL ngram_token_size= 10; > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 10 > +# > +# 1 is an acceptable value. > +SET GLOBAL ngram_token_size= 1; > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 1 > +# > +# 11 is outside allowed range [1, 10], so it's changed to 10. > +SET GLOBAL ngram_token_size= 11; > +Warnings: > +Warning 1292 Truncated incorrect ngram_token_size value: '11' > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 10 ^^^^ all tests above aren't needed, it's enough to do query_vertical select * from information_schema.system_variables where variable_name='ngram_token_size'; all the behavior above is implemented outside of the plugin, no need to test it. > +# > +# Try 3-grams. > +SET GLOBAL ngram_token_size= 3; > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB; > +INSERT INTO t VALUES > +('Lorem ipsum dolor sit amet, consectetur adipiscing elit.'), > +('Maecenas non pellentesque odio.'), > +('Sed fermentum egestas sapien vel consequat.'); > +# Only one row contains substring 'api'. > +SELECT str FROM t WHERE MATCH(str) AGAINST('api'); > +str > +Sed fermentum egestas sapien vel consequat. > +# 'lent' is parsed into ('len', 'ent') list. Two rows contain at least one of the tokens. > +SELECT str FROM t WHERE MATCH(str) AGAINST('lent*'); > +str > +Maecenas non pellentesque odio. > +Sed fermentum egestas sapien vel consequat. > +# 'zzz' is not present anywhere, so result is empty. > +SELECT str FROM t WHERE MATCH(str) AGAINST('zzzz'); > +str > +# 'piz' cannot be found anywhere, but 'api' is in one of the rows. > +SELECT str FROM t WHERE MATCH(str) AGAINST('apiz'); > +str > +Sed fermentum egestas sapien vel consequat. > +# > +# Try switching to 2-grams while keeping the index. > +SET GLOBAL ngram_token_size= 2; > +# 'api' is now parsed into ('ap', 'pi'). Since index only contains 3-grams, nothing is found. > +SELECT str FROM t WHERE MATCH(str) AGAINST('api'); > +str > +# > +# Add some data while ngram size is set to 2. > +INSERT INTO t VALUES('1234 56789'); > +# > +# Now index contains 3-grams from the rows added before and a number of > +# 2-grams from the just added row. > +# Querying for 'api' still returns nothing, but line with '678' is found. > +SELECT str from t WHERE MATCH(str) AGAINST('api'); > +str > +SELECT str FROM t WHERE MATCH(str) AGAINST('678'); > +str > +1234 56789 > +# > +# Switching back to 3-grams make intitially added rows searchable. > +SET GLOBAL ngram_token_size= 3; > +SELECT str FROM t WHERE MATCH(str) AGAINST('api'); > +str > +Sed fermentum egestas sapien vel consequat. > +# > +# But row with '678' in it cannot be found, as it was added with 2-grams. > +SELECT str FROM t WHERE MATCH(str) AGAINST('678'); > +str > +# > +# Try matching in boolean mode. > +SELECT str FROM t WHERE MATCH(str) AGAINST('+ent -api' IN BOOLEAN MODE); > +str > +Maecenas non pellentesque odio. > +# > +DROP TABLE t; > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_ucs2.result b/plugin/fulltext/mysql-test/ngram/ngram_ucs2.result > new file mode 100644 > index 00000000000..9d82743bd24 > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_ucs2.result > @@ -0,0 +1,8 @@ > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UCS2; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UTF16; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UTF16LE; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UTF32; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index same as above. This is fulltext index limitation, it does not depend on the parser plugin. The plugin doesn't need to test for it. > diff --git a/plugin/fulltext/mysql-test/ngram/suite.opt b/plugin/fulltext/mysql-test/ngram/suite.opt > new file mode 100644 > index 00000000000..ab37cea996e > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/suite.opt > @@ -0,0 +1,4 @@ > +--plugin-load-add=$FULLTEXT_NGRAM_SO > +--loose-innodb no need to enable innodb explicitly, `source have_innodb.inc` does that. > +--loose-enable-innodb_ft_index_table > +--loose-innodb_ft_index_cache why did you enable these two plugins if you don't use them in your tests? > diff --git a/plugin/fulltext/ngram_parser/plugin_ngram.cc b/plugin/fulltext/ngram_parser/plugin_ngram.cc > new file mode 100644 > index 00000000000..80498c3299b > --- /dev/null > +++ b/plugin/fulltext/ngram_parser/plugin_ngram.cc > @@ -0,0 +1,256 @@ > +/* > + Copyright (c) 2014, 2017, Oracle and/or its affiliates. All rights reserved. > + Copyright (c) 2020, MariaDB. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License, version 2.0, > + as published by the Free Software Foundation. > + > + This program is also distributed with certain software (including > + but not limited to OpenSSL) that is licensed under separate terms, > + as designated in a particular file or component or in included license > + documentation. The authors of MySQL hereby grant you an additional > + permission to link the program and your derivative works with the > + separately licensed software that they have included with MySQL. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License, version 2.0, for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > + > +#include <stddef.h> > +#include <limits.h> > +#include <my_attribute.h> > +#include <my_global.h> > +#include "../storage/innobase/include/fts0tokenize.h" No, the plugin should be self-contained and not include anything from other plugins. At least if a plugin is as trivial as that one. > + > +#define U_LOW_LINE 0x5f // Code point for underscore '_'. > +#define is_alnum(ct) ((ct) & (_MY_U | _MY_L | _MY_NMR)) > + > +static int ngram_token_size; > + > +static MYSQL_SYSVAR_INT(token_size, ngram_token_size, > + 0, > + "Ngram full text plugin parser token size in characters", > + NULL, NULL, 2, 1, 10, 0); > + > +static int my_ci_charlen(CHARSET_INFO *cs, const uchar *str, const uchar *end) > +{ > + return my_charlen(cs, reinterpret_cast<const char *>(str), > + reinterpret_cast<const char *>(end)); > +} > + > +static int my_ci_ctype(CHARSET_INFO *cs, int *char_type, const uchar *str, > + const uchar *end) > +{ > + return cs->cset->ctype(cs, char_type, str, end); > +} > + > +static bool is_underscore(CHARSET_INFO *cs, const uchar *str, const uchar *end) > +{ > + my_wc_t wc; > + cs->cset->mb_wc(cs, &wc, str, end); > + return wc == U_LOW_LINE; > +} > + > +// Splits a string into ngrams and emits them. > +static int split_into_ngrams(MYSQL_FTPARSER_PARAM *param, const uchar *doc, > + int len, MYSQL_FTPARSER_BOOLEAN_INFO *bool_info) > +{ > + const CHARSET_INFO *cs= param->cs; > + const uchar *start= doc; > + const uchar *end= doc + len; > + const uchar *next= start; > + int n_chars= 0; > + int ret= 0; > + bool is_first= true; > + > + while (next < end) { > + int char_type; > + int char_len= my_ci_ctype(cs, &char_type, next, end); > + > + // Broken data? > + if (next + char_len > end || char_len == 0) > + break; > + > + // Avoid creating partial n-grams by stopping at word boundaries. > + // Underscore is treated as a word character too, since identifiers in > + // programming languages often contain them. > + if (!is_alnum(char_type) && !is_underscore(cs, next, end)) { > + start= next + char_len; > + next= start; > + n_chars= 0; > + continue; I don't think it's a good idea. On the opposite, I would rather create ngrams over word boundaries, so that, say, I'd split "n-gram plugin" to "n-g", "-gr", "gra", "ram", "am ", "m p", " pl", "plu", "lug", "ugi", "gin". May be better would be to split it into "n-gr", "gra", "ram", "am p", "m pl", "plu", "lug", "ugi", "gin". (that is, always three letters in an n-gram, not three characters). This needs testing to find out what approach works better. > + } > + > + next += char_len; > + n_chars++; > + > + if (n_chars == ngram_token_size) { > + bool_info->position= static_cast<uint>(start - doc); nope, we don't have MYSQL_FTPARSER_FULL_BOOLEAN_INFO::position and this plugin is definitely not the reason to break the API and add it. > + ret= param->mysql_add_word(param, reinterpret_cast<const char *>(start), > + static_cast<int>(next - start), bool_info); > + if (ret != 0) > + return ret; > + > + start += my_ci_charlen(cs, start, end); why do you scan the string again here, you've already counted character by character above? > + n_chars= ngram_token_size - 1; > + is_first= false; > + } > + } > + > + // If nothing was emitted yet, emit the remainder. what reminder, what is it for? > + switch (param->mode) { > + case MYSQL_FTPARSER_FULL_BOOLEAN_INFO: > + case MYSQL_FTPARSER_WITH_STOPWORDS: > + if (n_chars > 0 && is_first) { > + assert(next > start); > + assert(n_chars < ngram_token_size); > + > + bool_info->position= static_cast<uint>(start - doc); > + ret= param->mysql_add_word(param, reinterpret_cast<const char *>(start), > + static_cast<int>(next - start), bool_info); > + } > + break; > + > + default: > + break; > + } > + > + return ret; > +} > + > +static int number_of_chars(const CHARSET_INFO *cs, const uchar *buf, int len) > +{ > + const uchar *ptr= buf; > + const uchar *end= buf + len; > + int size= 0; > + > + while (ptr < end) { > + ptr += my_ci_charlen(cs, ptr, end); > + size += 1; > + } > + > + return size; > +} > + > +// Convert term into phrase and handle wildcard. > +static int ngram_term_convert(MYSQL_FTPARSER_PARAM *param, const uchar *token, > + int len, MYSQL_FTPARSER_BOOLEAN_INFO *bool_info) > +{ > + MYSQL_FTPARSER_BOOLEAN_INFO token_info= > + { FT_TOKEN_WORD, 0, 0, 0, 0, 0, ' ', 0 }; > + const CHARSET_INFO *cs= param->cs; > + int ret= 0; > + > + assert(bool_info->type == FT_TOKEN_WORD); > + assert(bool_info->quot == NULL); > + > + /* Convert rules: > + * 1. if term with wildcard and term length is less than ngram_token_size, > + * we keep it as normal term search. > + * 2. otherwise, term is converted to phrase and wildcard is ignored. > + * e.g. 'abc' and 'abc*' are both equivalent to '"ab bc"'. > + */ > + > + if (bool_info->trunc && number_of_chars(cs, token, len) < ngram_token_size) { > + ret= param->mysql_add_word(param, reinterpret_cast<const char *>(token), > + len, bool_info); > + } else { > + bool_info->type= FT_TOKEN_LEFT_PAREN; > + bool_info->quot= reinterpret_cast<char*>(1); > + > + ret= param->mysql_add_word(param, NULL, 0, bool_info); > + if (ret != 0) > + return ret; > + > + ret= split_into_ngrams(param, token, len, &token_info); > + if (ret != 0) > + return ret; > + > + bool_info->type= FT_TOKEN_RIGHT_PAREN; > + ret= param->mysql_add_word(param, NULL, 0, bool_info); > + > + assert(bool_info->quot == NULL); > + bool_info->type= FT_TOKEN_WORD; > + } > + > + return ret; > +} > + > +static int ngram_parser_parse(MYSQL_FTPARSER_PARAM *param) > +{ > + MYSQL_FTPARSER_BOOLEAN_INFO bool_info= > + { FT_TOKEN_WORD, 0, 0, 0, 0, 0, ' ', 0 }; > + const CHARSET_INFO *cs= param->cs; > + uchar *start= reinterpret_cast<uchar *>(const_cast<char *>(param->doc)); > + uchar *end= start + param->length; > + FT_WORD word= {NULL, 0, 0}; > + int ret= 0; > + > + switch (param->mode) { > + case MYSQL_FTPARSER_SIMPLE_MODE: > + case MYSQL_FTPARSER_WITH_STOPWORDS: > + ret= split_into_ngrams(param, start, static_cast<int>(end - start), > + &bool_info); > + break; > + > + case MYSQL_FTPARSER_FULL_BOOLEAN_INFO: > + // Reusing InnoDB's boolean mode parser to handle all special characters. > + while (fts_get_word(cs, &start, end, &word, &bool_info)) { nope. InnoDB might not even be compiled it. Use param->mysql_parse() > + if (bool_info.type == FT_TOKEN_WORD) { > + if (bool_info.quot != NULL) { > + ret= split_into_ngrams(param, word.pos, word.len, &bool_info); > + } else { > + ret= ngram_term_convert(param, word.pos, word.len, &bool_info); > + } > + } else { > + ret= param->mysql_add_word(param, reinterpret_cast<char*>(word.pos), > + word.len, &bool_info); > + } > + > + if (ret != 0) > + return ret; > + } > + > + break; > + } > + > + return ret; > +} > + > +static struct st_mysql_ftparser ngram_parser_descriptor= > +{ > + MYSQL_FTPARSER_INTERFACE_VERSION, > + ngram_parser_parse, > + NULL, > + NULL, > +}; > + > +static struct st_mysql_sys_var *ngram_system_variables[]= > +{ > + MYSQL_SYSVAR(token_size), > + NULL > +}; > + > +maria_declare_plugin(ngram_parser) > +{ > + MYSQL_FTPARSER_PLUGIN, // Type. > + &ngram_parser_descriptor, // Descriptor. > + "ngram", // Name. > + "Oracle Corp", // Author. > + "Ngram Full-Text Parser", // Description. > + PLUGIN_LICENSE_GPL, // License. > + NULL, // Initialization function. > + NULL, // Deinitialization function. > + 0x0100, // Numeric version. > + NULL, // Status variables. > + ngram_system_variables, // System variables. > + "1.0", // String version representation. > + MariaDB_PLUGIN_MATURITY_EXPERIMENTAL, // Maturity. > +} > +maria_declare_plugin_end; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0