- developers - lists.mariadb.org

Re: [Maria-developers] e5fc78f84e3: MDEV-20220: Merge 5.7 P_S replication table 'replication_applier_status_by_worker
by sujatha 22 Mar '21

22 Mar '21

Hello Andrei, Good Morning. On 22/03/21 3:50 pm, Andrei Elkin wrote: > Sujatha, howdy. > > Just one note from me at this point, > >> ==> https://dev.mysql.com/doc/refman/5.7/en/replication-channels.html >> >> A multi-source replica can also be set up as a multi-threaded >> replica, by setting the slave_parallel_workers system variable to a >> value greater than 0. When you do this on a multi-source replica, each >> channel on the replica has the specified number of applier >> threads, plus a coordinator thread to manage them. You cannot >> configure the number of applier threads for individual channels. >> >> In case of MySQL, worker threads are dedicated to particular channel. >> >> In MariaDB "The pool of replication worker threads is shared among all >> multi-source master connections, and among all replication domains >> that can replicate in parallel using out-of-order". >> >> Because of this I didn't include CHANNEL_NAME. In MariaDB Slave_worker >> thread's don't have WORKER_ID they only have 'thread_id'. > Would it make sense to report > > LAST_CHANNEL (or LAST_SOURCE) similar to other 'LAST_' like > > |LAST_SEEN_TRANSACTION | Last GTID executed by worker | > > ? Yes, as 'LAST_SOURCE', we can display 'mi->connection_name'. In MariaDB documentation 'CONNECTION_NAME' is used to identify a source, so I used 'CONNECTION_NAME' instead of 'CHANNEL_NAME'(MySQL'). Thank you. S.Sujatha > >>>> Fix: >>>> === >>>> Iterate through rpl_parallel_thread_pool and display slave worker thread >>>> specific information as part of 'replication_applier_status_by_worker' table. >>>> >>>> --------------------------------------------------------------------------------- >>>> |Column Name: | Description: | >>>> |-------------------------------------------------------------------------------| >>>> | | | >>>> |THREAD_ID | Thread_Id as displayed in 'performance_schema.threads'| >>>> | | table for thread with name | >>>> | | 'thread/sql/rpl_parallel_thread' | >>>> | | | >>>> | | THREAD_ID will be NULL when worker threads are stopped| >>>> | | due to error/force stop | >>>> | | | >>>> |SERVICE_STATE | Thread is running or not | >>>> | | | >>>> |LAST_SEEN_TRANSACTION | Last GTID executed by worker | >>>> | | | >>>> |LAST_ERROR_NUMBER | Last Error that occurred on a particular worker | >>>> | | | >>>> |LAST_ERROR_MESSAGE | Last error specific message | >>>> | | | >>>> |LAST_ERROR_TIMESTAMP | Time stamp of last error | >>>> | | | >>>> |WORKER_IDLE_TIME | Total idle time in seconds that the worker thread has | >>>> | | spent waiting for work from SQL thread | >>>> | | | >>>> |LAST_TRANS_RETRY_COUNT | Total number of retries attempted by last transaction | >>>> --------------------------------------------------------------------------------- > Cheers, > > Andrei

1 0

Re: [Maria-developers] d86fe72c027: MDEV-22010: use executables MariaDB named in scripts
by Sergei Golubchik 21 Mar '21

21 Mar '21

Hi, Rucha! On Mar 21, Rucha Deodhar wrote: > revision-id: d86fe72c027 (mariadb-10.5.2-386-gd86fe72c027) > parent(s): de407e7cb4d > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-03-18 00:58:43 +0530 > message: > > MDEV-22010: use executables MariaDB named in scripts > > As a part of this MDEV following changes were made: > 1) Mariadb named executables used instead of mysql named executables in scripts > 2) renamed mysql-test-run and mysql-stress-test to mariadb-test-run and > mariadb-stress-test and created a symlink. > > diff --git a/.gitignore b/.gitignore > index d58dd2aa2b8..1a840d1a51d 100644 > --- a/.gitignore > +++ b/.gitignore > @@ -600,3 +600,7 @@ tests/mariadb-client-test > versioninfo_dll.rc > versioninfo_exe.rc > win/packaging/ca/symlinks.cc > +mysql-test/mysql-stress-test.pl > +mysql-test/mysql-test-run > +mysql-test/mysql-test-run.pl > +mysql-test/mtr Please, add these lines together with other mysql-test/ lines in the file. Then you'll see that mysql-test-run and mtr are already ignored. > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > index 5f183afe8fc..147bd7351bc 100644 > --- a/scripts/mysql_install_db.sh > +++ b/scripts/mysql_install_db.sh > @@ -338,11 +338,11 @@ then > cannot_find_file resolveip @resolveip_locations@ > exit 1 > fi > - mysqld=`find_in_dirs mysqld @mysqld_locations@` > + mysqld=`find_in_dirs mariadbd @mysqld_locations@` > if test -z "$mysqld" > then > - cannot_find_file mysqld @mysqld_locations@ > - exit 1 > + cannot_find_file mysqld @mysqld_locations@ you wanted to say "cannot_find_file mariadbd @mysqld_locations@" > + exit 1 > fi > langdir=`find_in_dirs --dir errmsg.sys @errmsg_locations@` > if test -z "$langdir" > @@ -579,7 +579,7 @@ else > echo > echo " shell> $mysqld --skip-grant-tables --general-log &" > echo > - echo "and use the command line tool $bindir/mysql" > + echo "and use the command line tool $bindir/mariadb or $bindir/mysql" I don't think you need any "or mysql" here. > echo "to connect to the mysql database and look at the grant tables:" > echo > echo " shell> $bindir/mysql -u root mysql" > @@ -613,7 +613,8 @@ then > echo "PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !" > echo "To do so, start the server, then issue the following commands:" > echo > - echo "'$bindir/mysqladmin' -u root password 'new-password'" > + echo "'$bindir/mariadb-admin' -u root password 'new-password' or" > + echo "'$bindir/mariadb-admin' -u root -h $hostname password 'new-password' or" and not here. > echo "'$bindir/mysqladmin' -u root -h $hostname password 'new-password'" > echo > echo "Alternatively you can run:" > diff --git a/scripts/mysql_secure_installation.sh b/scripts/mysql_secure_installation.sh > index b2a9edf4953..6098e0148b5 100644 > --- a/scripts/mysql_secure_installation.sh > +++ b/scripts/mysql_secure_installation.sh > @@ -159,15 +159,15 @@ then > cannot_find_file my_print_defaults $basedir/bin $basedir/extra > exit 1 > fi > - mysql_command=`find_in_basedir mysql bin` > + mysql_command=`find_in_basedir mariadb bin` > if test -z "$mysql_command" > then > - cannot_find_file mysql $basedir/bin > - exit 1 > + cannot_find_file mysql $basedir/bin should be cannot_find_file mariadb $basedir/bin" > + exit 1 > fi > else > print_defaults="@bindir@/my_print_defaults" > - mysql_command="@bindir@/mysql" > + mysql_command="@bindir@/mariadb" > fi > > if test ! -x "$print_defaults" > diff --git a/scripts/mysqld_multi.sh b/scripts/mysqld_multi.sh > index 0b33f61710e..d83663e03fc 100644 > --- a/scripts/mysqld_multi.sh > +++ b/scripts/mysqld_multi.sh > @@ -728,7 +728,7 @@ sub example > > [mysqld_multi] > mysqld = @bindir@/mysqld_safe mariadbd-safe ? (here and everywhere in this file) Also, you've missed mysqld_safe in FIND_PROC set in scripts/CMakeLists.txt > -mysqladmin = @bindir@/mysqladmin > +mysqladmin = @bindir@/mariadb-admin > user = multi_admin > password = my_password > > diff --git a/scripts/mysqld_safe.sh b/scripts/mysqld_safe.sh > index 126eb924814..7763fb9db91 100644 > --- a/scripts/mysqld_safe.sh > +++ b/scripts/mysqld_safe.sh > @@ -531,7 +531,11 @@ else > ledir='@libexecdir@' > fi > > -helper=`find_in_bin mysqld_safe_helper` > +helper=`find_in_bin mariadbd-safe-helper` > +if test -x helper > +then > + helper=`find_in_bin mysqld_safe_helper` > +fi no, remove the if, no need to fallback (and you've got the condition wrong anyway) > print_defaults=`find_in_bin my_print_defaults` > # Check if helper exists > command -v $helper --help >/dev/null 2>&1 > diff --git a/scripts/wsrep_sst_common.sh b/scripts/wsrep_sst_common.sh > index 5e134570881..b8621a76d6f 100644 > --- a/scripts/wsrep_sst_common.sh > +++ b/scripts/wsrep_sst_common.sh > @@ -273,16 +273,16 @@ SCRIPTS_DIR="$(cd $(dirname "$0"); pwd -P)" > EXTRA_DIR="$SCRIPTS_DIR/../extra" > CLIENT_DIR="$SCRIPTS_DIR/../client" > > -if [ -x "$CLIENT_DIR/mysql" ]; then > - MYSQL_CLIENT="$CLIENT_DIR/mysql" > +if [-x "$CLIENT_DIR/mariadb"]; then "[" is a name of an executable, always leave a space after it. there's no [-x executable. > + MYSQL_CLIENT="$CLIENT_DIR/mariadb" > else > - MYSQL_CLIENT=mysql > + MYSQL_CLIENT=mariadb > fi Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2858f9fb7e0: MDEV-22189: Change error messages inside code to have mariadb instead of
by Sergei Golubchik 21 Mar '21

21 Mar '21

Hi, Rucha! First, please check the buildbot and make sure all tests pass. You've changed MySQL->MariaDB in result files, but without corresponding changes in the code your result files now differ from the actual test output. One more comment below On Mar 21, Rucha Deodhar wrote: > revision-id: 2858f9fb7e0 (mariadb-10.5.2-274-g2858f9fb7e0) > parent(s): 41485551c58 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-03-18 13:58:34 +0530 > message: > > MDEV-22189: Change error messages inside code to have mariadb instead of > mysql > > Fix: Changed error messages, rerecorded results and changed other relevant > files. > > diff --git a/debian/additions/innotop/innotop b/debian/additions/innotop/innotop > index 19229a57a82..a045a479be9 100644 > --- a/debian/additions/innotop/innotop > +++ b/debian/additions/innotop/innotop > @@ -987,12 +987,12 @@ sub parse_tx_text { > # Parsing the line that begins 'MySQL thread id' is complicated. The only > # thing always in the line is the thread and query id. See function > # innobase_mysql_print_thd in InnoDB source file sql/ha_innodb.cc. > - my ( $thread_line ) = $txn =~ m/^(MySQL thread id .*)$/m; > + my ( $thread_line ) = $txn =~ m/^(MariaDB thread id .*)$/m; > my ( $mysql_thread_id, $query_id, $hostname, $ip, $user, $query_status ); > > if ( $thread_line ) { > # These parts can always be gotten. > - ( $mysql_thread_id, $query_id ) = $thread_line =~ m/^MySQL thread id $d, .*?query id $d/m; > + ( $mysql_thread_id, $query_id ) = $thread_line =~ m/^MariaDB thread id $d, .*?query id $d/m; > > # If it's a master/slave thread, "Has (read|sent) all" may be the thread's > # proc_info. In these cases, there won't be any host/ip/user info > @@ -4144,15 +4144,15 @@ my $innodb_parser = InnoDBParser->new; > > my $nonfatal_errs = join('|', > 'Access denied for user', > - 'Unknown MySQL server host', > + 'Unknown MariaDB server host', > 'Unknown database', > - 'Can\'t connect to local MySQL server through socket', > - 'Can\'t connect to MySQL server on', > + 'Can\'t connect to local MariaDB server through socket', > + 'Can\'t connect to MariaDB server on', > 'MySQL server has gone away', > 'Cannot call SHOW INNODB STATUS', > 'Access denied', > 'AutoCommit', > - 'Lost connection to MySQL server', > + 'Lost connection to MariaDB server', > 'Too many connections', > ); innotop is generally a separate tool that technically could work both with MariaDB and MySQL. So, better use (?:MariaDB|MySQL) in all regexp patterns above. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-13467 review.
by Alexey Botchkov 20 Mar '21

20 Mar '21

One thing i'd still like you to do differently. + if (g->get_class_info()->m_type_id == Geometry::wkb_multipoint) + { + const char point_size= 4 + WKB_HEADER_SIZE + POINT_DATA_SIZE+1; //1 for the type + char point_temp[point_size]; + memset(point_temp+4, Geometry::wkb_point, 1); + memcpy(point_temp+5, static_cast<const Gis_multi_point *>(g)->get_data_ptr()+5, 4); + memcpy(point_temp+4+WKB_HEADER_SIZE, g->get_data_ptr()+4+WKB_ HEADER_SIZE, + POINT_DATA_SIZE); + point_temp[point_size-1]= '\0'; + Geometry_buffer gbuff; + Geometry *gg= Geometry::construct(&gbuff, point_temp, point_size-1); > Gis_multi_point::geometry_n(uint32 num, String *result) stores the point in String *, > but I need Geometry* to use further, so still would need some recalculation. You could call the ::construct with the string->prr() instead of point_temp. Like str->q_append(0); //SRID g->geometry_n(str, 1); gg= Geometry::construct(&gbuff, str->ptr(), str->length()); But I propose an even simpler method. { Gis_pont p; p.set_data_ptr(g->get_data_ptr()+4+WKB_HEADER_SIZE, POINT_DATA_SIZE); calculate_haversine(&p, radius); } > Regarding radius, Daniel has suggestion: when r=NULL, result= NULL, but r=0, > how should that be treated? > Here is MySQL example (no error, but warning) There i see that MySQL returns the error when the radius is 0 or below. Right? I think it's ok for us to do same. > In a single function we are calculating radian for x,y, I think it is faster than to call function 2 times for x and y. I doubt so. Really would like to compare the speed here. Still i'm ok with that single function if you like it better. Best regards. HF On Tue, Mar 2, 2021 at 1:16 PM Anel Husakovic <anel(a)mariadb.org> wrote: > Hi Alexey, > thanks for your review, hope everything is ok now. > Here is the new patch 1f618e36577a2a322807 > <https://github.com/MariaDB/server/commit/1f618e36577a2a32280705c640d6f9a748…> according > to your review and comments below. > > On Sat, Jan 9, 2021 at 2:09 PM Alexey Botchkov <holyfoot(a)mariadb.com> > wrote: > >> Hi, Anel! >> >> Sorry for that long delay with this review. Had real difficult weeks of >> my life :( >> Now it seems to be over, so getting back to normal. >> Below is my opinions of the code i see in the branch >> bb-10.1-anel-MDEV-13467-gis-feature-st_sphere-v2 >> Hope it's the recent version. >> >> + Adapter for functions that takes two or three arguments. >> +*/ >> + >> +class Create_func_arg2_or_3 : public Create_func >> +{ >> Do you think we need the separate Create_fund_arg2_or_3 class? >> If you don't see any other class to inherit from it, >> i'd recommend to leave the Create_func_distance_sphere alone. >> No need to have two constructors in the Item_func_sphere_distance >> too. Just use the list argument as it's done for the >> Item_func_json_contains (in 10.2). >> > > Ack ^ droped the Create_func_arg2_or_3 and inherited from > Create_native_function which can use a variable number of parameters. > Thanks > > >> Anyway i belive we should get rid of duplicating code >> about the 'param1' and 'param2'. They can be popped/checked >> with no if (arg_count == 2) condition. >> >> + double distance= 0.0; >> + null_value= (args[0]->null_value || args[1]->null_value); >> + if (null_value) >> + { >> + // Returned item must be set to Null >> + DBUG_ASSERT(maybe_null); >> don't think it's needed. >> + return 0; >> Should be return 0.0 i belive. Or goto handle_errors. >> Thet goes to all but one 'return' statements in this function. >> >> + } >> >> > ... >> > if (!arg1 || !arg2) >> >> This last condition should never happen since we have not-null_value. >> So i'd remove that 'if' section completely. >> And anyway it should do the 'return' in that branch. >> >> + null_value= args[2]->null_value; >> + if (null_value) >> + { >> + return 0; >> + } >> >> seems nicer to me this way >> if (args[2]->null_value) >> { >> null_value= true; >> goto handle_errors; >> } >> >> > Ack. > Regarding radius, Daniel has suggestion > <https://github.com/MariaDB/server/commit/e42b69136a31cb596e940d1ceafffa5c11…>: > when r=NULL, result= NULL, but r=0, how should that be treated? > Here is MySQL > <https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=d467903dce23464df0ebf396220da886>example > (no error, but warning) > > > >> + if (!(g1= Geometry::construct(&buffer1, arg1->ptr(), arg1->length())) >> || >> + !(g2= Geometry::construct(&buffer2, arg2->ptr(), arg2->length()))) >> + { >> + goto handle_errors; >> >> we should launch the ER_GIS_INVALID_DATA here. >> >> > ER_GIS_INVALID_DATA is introduced in 10.2, I will change this (and test > case) when pushing to 10.2, but I would also push to 10.1 since there may > be users using it in 10.1. > Also when inspecting this error in 10.2 noted that lot tests have > commented this error, so will remove that comment in the same commit. > > mysql-test/suite/innodb_gis/t/gis.test > > mysql-test/suite/innodb_gis/t/1.test > > mysql-test/suite/innodb_gis/t/bug16236208.test > > mysql-test/suite/innodb_gis/t/precise.test > > mysql-test/suite/innodb_gis/t/rtree.test > > > + // Generate error message in case different geometry is used? >> Yes, i think the explainig message here is good to have. >> >> +double Item_func_sphere_distance::spherical_distance(Geometry *g1, >> + Geometry *g2, >> + const double >> sphere_radius) >> +double Item_func_sphere_distance::spherical_distance_points(Geometry *g1, >> + Geometry *g2, >> + const double >> r) >> Why these two are member static functions, not simply static? >> If you plan any 'external' use for them, please add comments about >> what exacly they do. >> >> > Changed function to be member function only. > > > >> +double Item_func_sphere_distance::spherical_distance(Geometry *g1, >> + Geometry *g2, >> + const double >> sphere_radius) >> +{ >> + double res= 0.0; >> + switch (g1->get_class_info()->m_type_id) >> + { >> + case Geometry::wkb_point: >> + res= spherical_distance_points(g1, g2, sphere_radius); >> + break; >> + >> + case Geometry::wkb_multipoint: >> + res= spherical_distance_points(g1, g2, sphere_radius); >> + break; >> >> Why do we need this function? >> We confirmed already that the type is either wkb_point or >> wkb_multipolygon, >> so just can call spherical_distance_points. No? >> >> > Indeed, left from earlier implementation (v1). > > >> >>>>> >> >> >> +class Item_func_sphere_distance: public Item_real_func >> +{ >> + double sphere_radius= 6370986.0; // Default radius equals Earth radius >> >> Some compilers don't like this syntax, and we didn't use it yet, so >> i'd recomment to move this to the constructor or even better to the >> ::val_real(). >> >> +double Gis_point::calculate_haversine(const Geometry *g, >> + const double sphere_radius) >> +{ >> + DBUG_ASSERT(sphere_radius > 0); >> + double x1r= 0.0; >> + double x2r= 0.0; >> + double y1r= 0.0; >> + double y2r= 0.0; >> That lot of unnecessary initializations seems excessive. >> >> Ack > > >> + if (g->get_class_info()->m_type_id == Geometry::wkb_multipoint) >> + { >> + const char point_size= 4 + WKB_HEADER_SIZE + POINT_DATA_SIZE+1; //1 >> for the type >> + char point_temp[point_size]; >> + memset(point_temp+4, Geometry::wkb_point, 1); >> + memcpy(point_temp+5, static_cast<const Gis_multi_point >> *>(g)->get_data_ptr()+5, 4); >> + memcpy(point_temp+4+WKB_HEADER_SIZE, >> g->get_data_ptr()+4+WKB_HEADER_SIZE, >> + POINT_DATA_SIZE); >> + point_temp[point_size-1]= '\0'; >> + Geometry_buffer gbuff; >> + Geometry *gg= Geometry::construct(&gbuff, point_temp, point_size-1); >> >> There is the Gis_multi_point::geometry_n() function, should be used here >> to get the point. >> >> Gis_multi_point::geometry_n(uint32 num, String *result) stores the point > in String *, but I need Geometry* to use further, so still would need some > recalculation. > > I didn’t find a multipoint API to convert String * to Geometry* so I did > my own calculation after debugging from row data. > > > I don't like that lot of static_cast-s. >> Isn't it nicer to have just the function of >> double calculate_harvesine_points(Geometry *p1, Geometry *p2, double >> radius) >> double calculate_harvesine_multipoint_point(Geometry *mp, Geometry *p, >> double radius) >> double calculate_harvesine_multipoint_multipoint(Geometry *mp1, Geometry >> *mp2, double radius) >> which would call each other? >> >> > I would prefer to have 2 static member functions and use static cast where > needed, if you don't mind. > The same could be done with 3 functions but with the code repeated. > > >> The ::get_xy_radian() method looks unneeded to me. >> It's enough to have the to_radian(double d) function. >> >> > In a single function we are calculating radian for x,y, I think it is > faster than to call function 2 times for x and y. > > >> BTW it's possible to do the multipoint/multipoint calculations faster >> than n^2. >> Not necessary to do it right now, just not to remember. >> >> > Regarding this there are some implementations that don't use Multipoints > <https://postgis.net/docs/manual-1.4/ST_Distance_Sphere.html> as both > arguments, MySQL > <https://dev.mysql.com/doc/refman/8.0/en/spatial-convenience-functions.html>also > use only Multipoints,Points combination, not MultiPoints,MultiPoints, so we > need to decide what to support. > However, there are algorithms to calculate max > <http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=CA2AE9B932194B2970…> > ,min <https://core.ac.uk/download/pdf/82507217.pdf> distance and > O(nlong) using convex hull <https://en.wikipedia.org/wiki/Convex_hull> > (which we have implemented <https://mariadb.com/kb/en/st_convexhull/>) > and using rotating calipers > <https://en.wikipedia.org/wiki/Rotating_calipers> from N points from > convex hull it could be O(n). > > Also there is suggested usage of iteration > <https://github.com/MariaDB/server/commit/e42b69136a31cb596e940d1ceafffa5c11…>from > Daniel. > > > <https://github.com/MariaDB/server/commit/e42b69136a31cb596e940d1ceafffa5c11…> > > > >> Best regards. >> HF >> >> > > -- > *Anel Husaković, * > *Software developer* > *MariaDB Foundation* >

1 0

Re: [Maria-developers] 0fc221a012d: MDEV-8334: Rename utf8 to utf8mb3
by Sergei Golubchik 19 Mar '21

19 Mar '21

Hi, Rucha! Thanks for combining all commits in one and a cleanup. See comments below. On Mar 18, Rucha Deodhar wrote: > revision-id: 0fc221a012d (mariadb-10.5.2-402-g0fc221a012d) > parent(s): a1542f8a573 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-03-16 23:31:12 +0530 > message: > > MDEV-8334: Rename utf8 to utf8mb3 > > This patch changes the main name of 3 byte character set from utf8 to > utf8mb3. New old_mode UTF8_IS_UTF8MB3 is added and set TRUE by default, > so that utf8 would mean utf8mb3. If not set, utf8 would mean utf8mb4. > diff --git a/client/mysqlcheck.c b/client/mysqlcheck.c > index fb3103a318d..a8990d8cb6b 100644 > --- a/client/mysqlcheck.c > +++ b/client/mysqlcheck.c > @@ -437,7 +437,7 @@ static int get_options(int *argc, char ***argv) > if (!default_charset) > { > if (opt_fix_db_names || opt_fix_table_names) > - default_charset= (char*) "utf8"; > + default_charset= (char*) "utf8mb3"; why not to keep it utf8? > else > default_charset= (char*) MYSQL_AUTODETECT_CHARSET_NAME; > } > diff --git a/client/mysqldump.c b/client/mysqldump.c > index 7c363973da2..900456b31b2 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -3235,7 +3235,7 @@ static uint get_table_structure(const char *table, const char *db, char *table_t > { > fprintf(sql_file, > "/*!40101 SET @saved_cs_client = @@character_set_client */;\n" > - "/*!40101 SET character_set_client = utf8 */;\n" > + "/*!40101 SET character_set_client = utf8mb3 */;\n" why not to keep it utf8? > "%s%s;\n" > "/*!40101 SET character_set_client = @saved_cs_client */;\n", > is_log_table ? "CREATE TABLE IF NOT EXISTS " : "", > diff --git a/extra/mariabackup/backup_mysql.cc b/extra/mariabackup/backup_mysql.cc > index 3083326a7e0..c62252257b9 100644 > --- a/extra/mariabackup/backup_mysql.cc > +++ b/extra/mariabackup/backup_mysql.cc > @@ -117,7 +117,7 @@ xb_mysql_connect() > mysql_options(connection, MYSQL_PLUGIN_DIR, xb_plugin_dir); > } > mysql_options(connection, MYSQL_OPT_PROTOCOL, &opt_protocol); > - mysql_options(connection,MYSQL_SET_CHARSET_NAME, "utf8"); > + mysql_options(connection,MYSQL_SET_CHARSET_NAME, "utf8mb3"); why not to keep it utf8? > > msg("Connecting to MySQL server host: %s, user: %s, password: %s, " > "port: %s, socket: %s", opt_host ? opt_host : "localhost", > @@ -1506,7 +1506,7 @@ write_xtrabackup_info(MYSQL *connection, const char * filename, bool history, > "incremental ENUM('Y', 'N') DEFAULT NULL," > "format ENUM('file', 'tar', 'xbstream') DEFAULT NULL," > "compressed ENUM('Y', 'N') DEFAULT NULL" > - ") CHARACTER SET utf8 ENGINE=innodb", false); > + ") CHARACTER SET utf8mb3 ENGINE=innodb", false); why not to keep it utf8? > > > #define ESCAPE_BOOL(expr) ((expr)?"'Y'":"'N'") > diff --git a/extra/mariabackup/xtrabackup.cc b/extra/mariabackup/xtrabackup.cc > index 62fdace654d..0f500c35ff6 100644 > --- a/extra/mariabackup/xtrabackup.cc > +++ b/extra/mariabackup/xtrabackup.cc > @@ -1716,7 +1716,7 @@ static int create_bootstrap_file() > if(!f) > return -1; > > - fputs("SET NAMES UTF8;\n",f); > + fputs("SET NAMES UTF8MB3;\n",f); why not to keep it utf8? > enumerate_ibd_files(append_export_table); > for (std::set<std::string>::iterator it = tables_for_export.begin(); > it != tables_for_export.end(); it++) > diff --git a/mysql-test/main/create-uca.test b/mysql-test/main/create-uca.test > index 0acb51f7286..f73f6114962 100644 > --- a/mysql-test/main/create-uca.test > +++ b/mysql-test/main/create-uca.test > @@ -1,5 +1,5 @@ > # Prerequisites > -let collation=utf8_unicode_ci; > +let collation=utf8mb3_unicode_ci; that's fine, as we generally don't want tests to depend on the old_mode. > --source include/have_collation.inc > > # Initial cleanup > diff --git a/mysql-test/main/ctype_ldml.result b/mysql-test/main/ctype_ldml.result > index 22b7a316111..7c284520733 100644 > --- a/mysql-test/main/ctype_ldml.result > +++ b/mysql-test/main/ctype_ldml.result > @@ -8,7 +8,6 @@ Variable_name Value > character_sets_dir MYSQL_TEST_DIR/std_data/ldml/ > show collation like 'utf8_phone_ci'; > Collation Charset Id Default Compiled Sortlen > -utf8_phone_ci utf8 352 8 I suppose the test should show there is a _phone_ci collation. As you've renamed it, you need to adjust the test to do show collation like 'utf8mb3_phone_ci'; > CREATE TABLE t1 ( > name VARCHAR(64), > phone VARCHAR(64) CHARACTER SET utf8 COLLATE utf8_phone_ci > @@ -37,7 +36,6 @@ Bar +7-912-800-80-01 > DROP TABLE t1; > show collation like 'utf8_test_ci'; > Collation Charset Id Default Compiled Sortlen > -utf8_test_ci utf8 353 8 same > create table t1 (c1 char(1) character set utf8 collate utf8_test_ci); > insert into t1 values ('a'); > select * from t1 where c1='b'; > @@ -526,7 +524,6 @@ DROP TABLE t1; > SET NAMES utf8 COLLATE utf8_phone_ci; > SHOW COLLATION LIKE 'utf8_phone_ci'; > Collation Charset Id Default Compiled Sortlen > -utf8_phone_ci utf8 352 8 and here > SET NAMES utf8; > SELECT hex(weight_string(_utf8mb4'a' collate utf8mb4_test_400_ci)); > hex(weight_string(_utf8mb4'a' collate utf8mb4_test_400_ci)) > diff --git a/mysql-test/main/show_check.result b/mysql-test/main/show_check.result > index d031c792922..1c85ef596c9 100644 > --- a/mysql-test/main/show_check.result > +++ b/mysql-test/main/show_check.result > @@ -874,12 +874,11 @@ set names utf8; > ---------------------------------------------------------------- > SHOW CHARACTER SET LIKE 'utf8'; > Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr > -def information_schema CHARACTER_SETS CHARACTER_SETS CHARACTER_SET_NAME Charset 253 96 4 N 1 0 33 > -def information_schema CHARACTER_SETS CHARACTER_SETS DESCRIPTION Description 253 180 13 N 1 0 33 > -def information_schema CHARACTER_SETS CHARACTER_SETS DEFAULT_COLLATE_NAME Default collation 253 96 15 N 1 0 33 > -def information_schema CHARACTER_SETS CHARACTER_SETS MAXLEN Maxlen 8 3 1 N 32769 0 63 > +def information_schema CHARACTER_SETS CHARACTER_SETS CHARACTER_SET_NAME Charset 253 96 0 N 1 0 33 > +def information_schema CHARACTER_SETS CHARACTER_SETS DESCRIPTION Description 253 180 0 N 1 0 33 > +def information_schema CHARACTER_SETS CHARACTER_SETS DEFAULT_COLLATE_NAME Default collation 253 96 0 N 1 0 33 > +def information_schema CHARACTER_SETS CHARACTER_SETS MAXLEN Maxlen 8 3 0 N 32769 0 63 > Charset Description Default collation Maxlen > -utf8 UTF-8 Unicode utf8_general_ci 3 again, I suspect this test should now do `SHOW CHARACTER SET LIKE 'utf8mb3'; > ---------------------------------------------------------------- > SHOW COLLATION LIKE 'latin1_bin'; > Catalog Database Table Table_alias Column Column_alias Type Length Max length Is_null Flags Decimals Charsetnr > diff --git a/mysql-test/suite/funcs_1/r/charset_collation.result b/mysql-test/suite/funcs_1/r/charset_collation.result > index 31bd30c5acf..6b52e80d6ba 100644 > --- a/mysql-test/suite/funcs_1/r/charset_collation.result > +++ b/mysql-test/suite/funcs_1/r/charset_collation.result > @@ -9,7 +9,6 @@ ORDER BY character_set_name; > CHARACTER_SET_NAME DEFAULT_COLLATE_NAME DESCRIPTION MAXLEN > binary binary Binary pseudo charset 1 > latin1 latin1_swedish_ci cp1252 West European 1 > -utf8 utf8_general_ci UTF-8 Unicode 3 and here too, changing the test is in order > > SELECT * > FROM information_schema.collations > diff --git a/mysql-test/suite/funcs_1/r/is_column_privileges.result b/mysql-test/suite/funcs_1/r/is_column_privileges.result > index b6be9118048..46b2d515041 100644 > --- a/mysql-test/suite/funcs_1/r/is_column_privileges.result > +++ b/mysql-test/suite/funcs_1/r/is_column_privileges.result > @@ -45,7 +45,7 @@ COLUMN_PRIVILEGES CREATE TEMPORARY TABLE `COLUMN_PRIVILEGES` ( > `COLUMN_NAME` varchar(64) NOT NULL DEFAULT '', > `PRIVILEGE_TYPE` varchar(64) NOT NULL DEFAULT '', > `IS_GRANTABLE` varchar(3) NOT NULL DEFAULT '' > -) ENGINE=MEMORY DEFAULT CHARSET=utf8 > +) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 Just a thought. Did you also fix all --embedded, all --ps, and all --big --big tests to pass? > SHOW COLUMNS FROM information_schema.COLUMN_PRIVILEGES; > Field Type Null Key Default Extra > GRANTEE varchar(190) NO > diff --git a/mysql-test/suite/innodb/r/innodb_ctype_ldml.result b/mysql-test/suite/innodb/r/innodb_ctype_ldml.result > index 502f57156c3..8a96c023134 100644 > --- a/mysql-test/suite/innodb/r/innodb_ctype_ldml.result > +++ b/mysql-test/suite/innodb/r/innodb_ctype_ldml.result > @@ -8,7 +8,6 @@ Variable_name Value > character_sets_dir MYSQL_TEST_DIR/std_data/ldml/ > show collation like 'utf8_phone_ci'; > Collation Charset Id Default Compiled Sortlen > -utf8_phone_ci utf8 352 8 deja vu. I rememeber I've commented on this very thing already :) please, update this test too. > CREATE TABLE t1 ( > name VARCHAR(64), > phone VARCHAR(64) CHARACTER SET utf8 COLLATE utf8_phone_ci > Binary files a/mysql-test/suite/sys_vars/r/character_set_results_basic.result and b/mysql-test/suite/sys_vars/r/character_set_results_basic.result differ > diff --git a/mysql-test/suite/sys_vars/r/old_mode_basic.result b/mysql-test/suite/sys_vars/r/old_mode_basic.result > index 39c8e554be2..a6b95f1c60c 100644 > --- a/mysql-test/suite/sys_vars/r/old_mode_basic.result > +++ b/mysql-test/suite/sys_vars/r/old_mode_basic.result > @@ -167,8 +167,100 @@ NO_PROGRESS_INFO > SET @@global.old_mode = @global_start_value; > SELECT @@global.old_mode; > @@global.old_mode > - > +UTF8_IS_UTF8MB3 > SET @@session.old_mode = @session_start_value; > SELECT @@session.old_mode; > @@session.old_mode > - > +UTF8_IS_UTF8MB3 > +# > +# Beginning of 10.6 test > +# > +# MDEV-8334: Rename utf8 to utf8mb3 > +# Ah! Great. > +# Save and display old values > +SET @save_old_mode = @@OLD_MODE; > +SET @save_character_set_server = @@character_set_server; > +SET @save_character_set_client = @@character_set_client; > +SET @save_character_set_results = @@character_set_results; > +SET @save_character_set_connection = @@character_set_connection; > +SET @save_character_set_filesystem = @@character_set_filesystem; > +SET @save_character_set_database = @@character_set_database; > +SET @save_collation_connection = @@collation_connection; > +SET @save_collation_server = @@collation_server; > +SET @save_collation_database = @@collation_database; > +SELECT @@OLD_MODE; > +@@OLD_MODE > +UTF8_IS_UTF8MB3 > +SELECT @@character_set_server,@@character_set_client,@@character_set_results, > +@@character_set_connection, @@character_set_filesystem, @@character_set_database, > +@@collation_connection, @@collation_server, @@collation_database; > +@@character_set_server @@character_set_client @@character_set_results @@character_set_connection @@character_set_filesystem @@character_set_database @@collation_connection @@collation_server @@collation_database > +latin1 latin1 latin1 latin1 binary latin1 latin1_swedish_ci latin1_swedish_ci latin1_swedish_ci > +# > +# UTF8MB3 alias for UTF8 > +# > +SET @@character_set_server = utf8; > +SET @@character_set_client = utf8; > +SET @@character_set_results = utf8; > +SET @@character_set_connection = utf8; > +SET @@character_set_filesystem = utf8; > +SET @@character_set_database = utf8; > +SET @@collation_connection = utf8_general_ci; > +SET @@collation_server = utf8_unicode_ci; > +SET @@collation_database = utf8_bin; > +SELECT @@character_set_server, @@character_set_client, @@character_set_results, > +@@character_set_connection, @@character_set_filesystem, @@character_set_database, > +@@collation_connection, @@collation_server, @@collation_database; > +@@character_set_server @@character_set_client @@character_set_results @@character_set_connection @@character_set_filesystem @@character_set_database @@collation_connection @@collation_server @@collation_database > +utf8mb3 utf8mb3 utf8mb3 utf8mb3 utf8mb3 utf8mb3 utf8mb3_general_ci utf8mb3_unicode_ci utf8mb3_bin > +CREATE DATABASE db1 CHARACTER SET = 'utf8' COLLATE = 'utf8_general_ci'; > +ALTER DATABASE db1 CHARACTER SET = 'utf8' COLLATE = 'utf8_unicode_ci'; > +CREATE TABLE tb1 (id1 INT) CHARACTER SET 'utf8' COLLATE 'utf8_bin'; > +SHOW CREATE TABLE tb1; > +Table Create Table > +tb1 CREATE TABLE `tb1` ( > + `id1` int(11) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_bin > +DROP TABLE tb1; > +DROP DATABASE db1; > +# > +# UTF8MB4 is alias for UTF8 > +# > +SET @@OLD_MODE=0; > +SET @@character_set_server = utf8; > +SET @@character_set_client = utf8; > +SET @@character_set_results = utf8; > +SET @@character_set_connection = utf8; > +SET @@character_set_filesystem = utf8; > +SET @@character_set_database = utf8; > +SET @@collation_connection = utf8_general_ci; > +SET @@collation_server = utf8_unicode_ci; > +SET @@collation_database = utf8_bin; > +SELECT @@character_set_server, @@character_set_client, @@character_set_results, > +@@character_set_connection, @@character_set_filesystem, @@character_set_database, > +@@collation_connection, @@collation_server, @@collation_database; > +@@character_set_server @@character_set_client @@character_set_results @@character_set_connection @@character_set_filesystem @@character_set_database @@collation_connection @@collation_server @@collation_database > +utf8mb4 utf8mb4 utf8mb4 utf8mb4 utf8mb4 utf8mb4 utf8mb4_general_ci utf8mb4_unicode_ci utf8mb4_bin > +CREATE DATABASE db1 CHARACTER SET = 'utf8' COLLATE = 'utf8_general_ci'; > +ALTER DATABASE db1 CHARACTER SET = 'utf8' COLLATE = 'utf8_unicode_ci'; > +CREATE TABLE tb1 (id1 INT) CHARACTER SET 'utf8' COLLATE 'utf8_bin'; > +SHOW CREATE TABLE tb1; > +Table Create Table > +tb1 CREATE TABLE `tb1` ( > + `id1` int(11) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin > +DROP TABLE tb1; > +DROP DATABASE db1; > +SET @@OLD_MODE = @save_old_mode; > +SET @@character_set_server = @save_character_set_server; > +SET @@character_set_client = @save_character_set_client; > +SET @@character_set_results = @save_character_set_results; > +SET @@character_set_connection = @save_character_set_connection; > +SET @@character_set_filesystem = @save_character_set_filesystem; > +SET @@character_set_database = @save_character_set_database; > +SET @@collation_connection = @save_collation_connection; > +SET @@collation_server = @save_collation_server; > +SET @@collation_database = @save_collation_database; > +# > +# End of 10.6 test > +# > diff --git a/mysys/charset.c b/mysys/charset.c > index 32cfeb56e2d..dbb2749d217 100644 > --- a/mysys/charset.c > +++ b/mysys/charset.c > @@ -763,22 +764,13 @@ get_charset_number_internal(const char *charset_name, uint cs_flags) > } > > > -static const char* > -get_charset_name_alias(const char *name) > -{ > - if (!my_strcasecmp(&my_charset_latin1, name, "utf8mb3")) > - return "utf8"; > - return NULL; > -} > - > - > uint get_charset_number(const char *charset_name, uint cs_flags) > { > uint id; > my_pthread_once(&charsets_initialized, init_available_charsets); > if ((id= get_charset_number_internal(charset_name, cs_flags))) > return id; > - if ((charset_name= get_charset_name_alias(charset_name))) > + if ((charset_name= !my_strcasecmp(&my_charset_latin1, charset_name, "utf8") ? "utf8mb3" : NULL)) Huh? Why do you not check MY_UTF8_IS_UTF8MB3 here? > return get_charset_number_internal(charset_name, cs_flags); > return 0; > } > @@ -820,7 +812,7 @@ static CHARSET_INFO *find_collation_data_inheritance_source(CHARSET_INFO *cs) > char name[MY_CS_NAME_SIZE + 1]; > memcpy(name, beg, end - beg); > name[end - beg]= '\0'; > - return inheritance_source_by_id(cs, get_collation_number(name)); > + return inheritance_source_by_id(cs, get_collation_number(name,MYF(0))); and not here? > } > return NULL; > } > @@ -961,7 +953,28 @@ my_collation_get_by_name(MY_CHARSET_LOADER *loader, > CHARSET_INFO *get_charset_by_name(const char *cs_name, myf flags) > { > MY_CHARSET_LOADER loader; > + my_bool utf8_is_utf8mb3= flags & MY_UTF8_IS_UTF8MB3 ? 1 : 0; > + char *copy_of_name= (char*)cs_name; > + char start[6], result[64]; > + char *temp_cs_name; > + > my_charset_loader_init_mysys(&loader); > + > + if (!strcasecmp("utf8",copy_of_name)) > + cs_name = (const char*)(utf8_is_utf8mb3 ? "utf8mb3" : "utf8mb4"); > + > + strncpy(start, cs_name, 5); > + temp_cs_name= (char *)(utf8_is_utf8mb3 ? "utf8mb3_":"utf8mb4_"); > + > + if (!strncasecmp("utf8_", start,5)) > + { > + copy_of_name+= 5; > + result[63]='\0'; > + strcpy(result, temp_cs_name); > + strcat(result, copy_of_name); > + result[strlen(copy_of_name)+strlen(temp_cs_name)]='\0'; > + cs_name= (const char *) result; > + } And why do you do all that ^^^ ? Old code didn't try to change utf8mb3 to utf8 here, because, I suppose, my_collation_get_by_name() below did all that. Why did you add alias resolution where none was? > return my_collation_get_by_name(&loader, cs_name, flags); > } > > @@ -1005,12 +1018,16 @@ get_charset_by_csname(const char *cs_name, uint cs_flags, myf flags) > { > MY_CHARSET_LOADER loader; > my_charset_loader_init_mysys(&loader); > + > + if (!strcasecmp("utf8",cs_name)) > + cs_name= (const char*)(flags & MY_UTF8_IS_UTF8MB3 ? "utf8mb3" : "utf8mb4"); same here > + > return my_charset_get_by_name(&loader, cs_name, cs_flags, flags); > } > > > /** > - Resolve character set by the character set name (utf8, latin1, ...). > + Resolve character set by the character set name (utf8mb3, latin1, ...). > > The function tries to resolve character set by the specified name. If > there is character set with the given name, it is assigned to the "cs" > @@ -1453,8 +1472,8 @@ static const MY_CSET_OS_NAME charsets[] = > > {"US-ASCII", "latin1", my_cs_approx}, > > - {"utf8", "utf8", my_cs_exact}, > - {"utf-8", "utf8", my_cs_exact}, > + {"utf8mb3", "utf8mb3", my_cs_exact}, eh, no. Try to understand what this array is for. > + {"utf-8", "utf8mb3", my_cs_exact}, > #endif > {NULL, NULL, 0} > }; > diff --git a/plugin/handler_socket/client/hslongrun.cpp b/plugin/handler_socket/client/hslongrun.cpp > index b7c02951340..7f88d48fff2 100644 > --- a/plugin/handler_socket/client/hslongrun.cpp > +++ b/plugin/handler_socket/client/hslongrun.cpp > @@ -897,7 +897,7 @@ hs_longrun_init_table(const config& conf, int num_prepare, > "v1 varchar(32) not null," > "v2 varchar(32) not null," > "v3 varchar(32) not null" > - ") character set utf8 collate utf8_bin engine = innodb"); > + ") character set utf8mb3 collate utf8_bin engine = innodb"); just keep it utf8 > for (int i = 0; i < num_prepare; ++i) { > const std::string i_str = to_stdstring(i); > const std::string v1 = "pv1_" + i_str; > diff --git a/plugin/handler_socket/client/hstest.pl b/plugin/handler_socket/client/hstest.pl > index 1363e153c44..5924d8a0ce5 100755 > --- a/plugin/handler_socket/client/hstest.pl > +++ b/plugin/handler_socket/client/hstest.pl > @@ -52,7 +52,7 @@ for my $action (@actions) { > "k $keytype primary key" . > ",v varchar(32) not null" . > $moreflds . > - ") character set utf8 collate utf8_bin " . > + ") character set utf8mb3 collate utf8_bin " . and here. forget about handlersocket > "engine = $engine"); > } elsif ($action eq "insert") { > print("INSERT $db.$table tablesize=$tablesize\n"); > diff --git a/plugin/win_auth_client/common.cc b/plugin/win_auth_client/common.cc > index 8b7319252ac..ddd34aec7da 100644 > --- a/plugin/win_auth_client/common.cc > +++ b/plugin/win_auth_client/common.cc > @@ -384,7 +384,7 @@ char* wchar_to_utf8(const wchar_t *string, size_t *len) > buf= (char*)malloc(buf_len + 1); > if (!buf) > { > - DBUG_PRINT("error",("Out of memory when converting string '%S' to utf8", > + DBUG_PRINT("error",("Out of memory when converting string '%S' to utf8mb3", Nope, see what this function is doing. > string)); > return NULL; > } > @@ -408,7 +408,7 @@ char* wchar_to_utf8(const wchar_t *string, size_t *len) > > #ifndef DBUG_OFF > Error_message_buf error_buf; > - DBUG_PRINT("error", ("Could not convert string '%S' to utf8" > + DBUG_PRINT("error", ("Could not convert string '%S' to utf8mb3" same > ", WideCharToMultiByte() failed with error %X (%s)", > string, GetLastError(), > get_last_error_message(error_buf))); > @@ -451,7 +451,7 @@ wchar_t* utf8_to_wchar(const char *string, size_t *len) > > if (!buf) > { > - DBUG_PRINT("error",("Out of memory when converting utf8 string '%s'" > + DBUG_PRINT("error",("Out of memory when converting utf8mb3 string '%s'" same > " to wide-char representation", string)); > return NULL; > } > diff --git a/scripts/fill_help_tables.sql b/scripts/fill_help_tables.sql > index d0efb750330..ad7c4fce9a4 100644 > --- a/scripts/fill_help_tables.sql > +++ b/scripts/fill_help_tables.sql > @@ -22,7 +22,7 @@ don't change help tables, please. they'll be regenerated from the documentation > > -- mysql -u root -p mysql < file_name > > -set names 'utf8'; > +set names 'utf8mb3'; > > set sql_log_bin = 0; > > diff --git a/scripts/mysql_system_tables.sql b/scripts/mysql_system_tables.sql > index e390f36a98b..7c8532577a1 100644 > --- a/scripts/mysql_system_tables.sql > +++ b/scripts/mysql_system_tables.sql > @@ -209,7 +209,7 @@ SET @create_transaction_registry="CREATE TABLE IF NOT EXISTS transaction_registr > UNIQUE KEY (commit_id), > INDEX (begin_timestamp), > INDEX (commit_timestamp, transaction_id) > -) ENGINE=INNODB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0"; > +) ENGINE=INNODB DEFAULT CHARSET=utf8mb3 COLLATE=utf8_bin STATS_PERSISTENT=0"; here and in all other places in this file and other .sql files: you need to change the collation too, not just the charset. > > SET @str=IF(@have_innodb <> 0, @create_innodb_table_stats, "SET @dummy = 0"); > PREPARE stmt FROM @str; > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 0bf21e02002..cc7568990b4 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -4039,7 +4039,10 @@ static int init_common_variables() > *next_character_set_name++= '\0'; > if (!(default_charset_info= > get_charset_by_csname(default_character_set_name, > - MY_CS_PRIMARY, MYF(MY_WME)))) > + MY_CS_PRIMARY, > + global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3 | MY_WME) : MYF(MY_WME)))) may be you'd better get this multi-line assignment out of if() ? e.g. myf utf8_flag= global_system_variables.old_behavior & OLD_MODE_UTF8_IS_UTF8MB3 : 0; default_charset_info= get_charset_by_csname(default_character_set_name, MY_CS_PRIMARY, MYF(utf8_flag | MY_WME)); if (!default_charset_info) ... or add a helper thd->utf8_alias() as I suggested below > { > if (next_character_set_name) > { > @@ -4056,7 +4059,10 @@ static int init_common_variables() > if (default_collation_name) > { > CHARSET_INFO *default_collation; > - default_collation= get_charset_by_name(default_collation_name, MYF(0)); > + default_collation= get_charset_by_name(default_collation_name, > + global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0)); and here you'll be able to use utf8_flag without the conditional operator > if (!default_collation) > { > #ifdef WITH_PERFSCHEMA_STORAGE_ENGINE > @@ -4097,7 +4103,10 @@ static int init_common_variables() > > if (!(character_set_filesystem= > get_charset_by_csname(character_set_filesystem_name, > - MY_CS_PRIMARY, MYF(MY_WME)))) > + MY_CS_PRIMARY, > + global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3 | MY_WME) : MYF(MY_WME)))) and here > return 1; > global_system_variables.character_set_filesystem= character_set_filesystem; > > @@ -7415,7 +7424,9 @@ static void usage(void) > DBUG_ENTER("usage"); > if (!(default_charset_info= get_charset_by_csname(default_character_set_name, > MY_CS_PRIMARY, > - MYF(MY_WME)))) > + global_system_variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3 | MY_WME) : MYF(MY_WME)))) split this one too? > exit(1); > if (!default_collation_name) > default_collation_name= (char*) default_charset_info->name; > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 50b746fe514..15aa9ca8199 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1007,6 +1008,36 @@ inline void update_global_memory_status(int64 size) > my_atomic_add64_explicit(ptr, size, MY_MEMORY_ORDER_RELAXED); > } > > +inline const char* get_alias_collation_or_charset_name(const char* name, > + bool utf8_is_utf8mb3) > +{ > + char *copy_of_name= (char*)name; > + char start[6], result[64]; > + char *temp_cs_name; > + > + if (!strchr(name,'_')) > + { > + if (!strcasecmp("utf8",name)) > + name = utf8_is_utf8mb3 ? "utf8mb3" : "utf8mb4"; > + return name; > + } > + else > + { > + strncpy(start, name, 5); > + temp_cs_name= (char *)(utf8_is_utf8mb3 ? "utf8mb3_":"utf8mb4_"); > + if (!strncasecmp("utf8_", start,5)) > + { > + copy_of_name+= 5; > + result[63]='\0'; > + strcpy(result, temp_cs_name); > + strcat(result, copy_of_name); > + result[strlen(copy_of_name)+strlen(temp_cs_name)]='\0'; > + strcpy((char*)name,result); > + } > + } > + return name; > +} Please, no. First, you cannot just copy `result` into `name`, because `result` is longer, you'll overwrite whatever was in memory after name's value. Second, you don't need to resolve aliases here, charset code already does it, don't duplicate that. Just pass MY_UTF8_IS_UTF8MB3 down to my_collation_get_by_name() below. > + > /** > Get collation by name, send error to client on failure. > @param name Collation name > diff --git a/sql/sql_db.cc b/sql/sql_db.cc > index 9bf16220535..f471d8edc66 100644 > --- a/sql/sql_db.cc > +++ b/sql/sql_db.cc > @@ -583,9 +583,14 @@ bool load_db_opt(THD *thd, const char *path, Schema_specification_st *create) > default-collation commands. > */ > if (!(create->default_table_charset= > - get_charset_by_csname(pos+1, MY_CS_PRIMARY, MYF(0))) && > + get_charset_by_csname(pos+1, MY_CS_PRIMARY, > + thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0))) && > !(create->default_table_charset= > - get_charset_by_name(pos+1, MYF(0)))) > + get_charset_by_name(pos+1, thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0)))) again, please move myf flags manipulations out of if() > { > sql_print_error("Error while loading database options: '%s':",path); > sql_print_error(ER_THD(thd, ER_UNKNOWN_CHARACTER_SET),pos+1); > @@ -595,7 +600,9 @@ bool load_db_opt(THD *thd, const char *path, Schema_specification_st *create) > else if (!strncmp(buf,"default-collation", (pos-buf))) > { > if (!(create->default_table_charset= get_charset_by_name(pos+1, > - MYF(0)))) > + thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0)))) and here > { > sql_print_error("Error while loading database options: '%s':",path); > sql_print_error(ER_THD(thd, ER_UNKNOWN_COLLATION),pos+1); > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 6871699dc5b..ad322eda097 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -2789,7 +2789,10 @@ int Lex_input_stream::scan_ident_middle(THD *thd, Lex_ident_cli_st *str, > body_utf8_append(m_cpp_text_start, m_cpp_tok_start + length); > ErrConvString csname(str->str + 1, str->length - 1, &my_charset_bin); > CHARSET_INFO *cs= get_charset_by_csname(csname.ptr(), > - MY_CS_PRIMARY, MYF(0)); > + MY_CS_PRIMARY, > + thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MYF(MY_UTF8_IS_UTF8MB3) : MYF(0)); may be it could be a helper in thd, like THD::utf8_alias() { return variables.old_behavior & OLD_MODE_UTF8_IS_UTF8MB3 ? MY_UTF8_IS_UTF8MB3 : 0; } > if (cs) > { > *introducer= cs; > diff --git a/storage/connect/mysql-test/connect/my.cnf b/storage/connect/mysql-test/connect/my.cnf > index 6310772d01f..83f0aa8ab30 100644 > --- a/storage/connect/mysql-test/connect/my.cnf > +++ b/storage/connect/mysql-test/connect/my.cnf > @@ -14,4 +14,4 @@ MASTER_MYSOCK= @mysqld.1.socket > SLAVE_MYPORT= @mysqld.2.port > SLAVE_MYSOCK= @mysqld.2.socket > > -PGCLIENTENCODING= UTF8 > +PGCLIENTENCODING= UTF8MB3 eh... really? Are you sure you've tested it and it worked? by the name of it I suspect it's a postgresql client encoding. > diff --git a/storage/connect/mysql-test/connect/t/odbc_postgresql.sql b/storage/connect/mysql-test/connect/t/odbc_postgresql.sql > index 1c302294393..3c78120a7a2 100644 > --- a/storage/connect/mysql-test/connect/t/odbc_postgresql.sql > +++ b/storage/connect/mysql-test/connect/t/odbc_postgresql.sql > @@ -4,13 +4,13 @@ > -- Run this script as a admin user: > -- psql -U postgres < odbc_postgresql.sql > > -SET NAMES 'UTF8'; > +SET NAMES 'UTF8MB3'; same, postgresql > > DROP DATABASE IF EXISTS mtr; > DROP USER IF EXISTS mtr; > > CREATE USER mtr WITH PASSWORD 'mtr'; > -CREATE DATABASE mtr OWNER=mtr ENCODING='UTF8'; > +CREATE DATABASE mtr OWNER=mtr ENCODING='UTF8MB3'; same > GRANT ALL ON DATABASE mtr TO mtr; > \c mtr > SET role mtr; > diff --git a/storage/innobase/fts/fts0opt.cc b/storage/innobase/fts/fts0opt.cc > index e5164fcc4fa..a26a05c2cf9 100644 > --- a/storage/innobase/fts/fts0opt.cc > +++ b/storage/innobase/fts/fts0opt.cc > @@ -330,7 +330,7 @@ fts_word_t* > fts_word_init( > /*==========*/ > fts_word_t* word, /*!< in: word to initialize */ > - byte* utf8, /*!< in: UTF-8 string */ > + byte* utf8mb3, /*!< in: UTF-8 string */ don't rename variables, please > ulint len) /*!< in: length of string in bytes */ > { > mem_heap_t* heap = mem_heap_create(sizeof(fts_node_t)); > diff --git a/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_french.result b/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_french.result > index 3f24de87035..3659aa5aee8 100644 > --- a/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_french.result > +++ b/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_french.result > @@ -7,5 +7,4 @@ FULLTEXT INDEX (content) > INSERT INTO diaries VALUES ("Je suis un garçon."); > SELECT * FROM diaries WHERE MATCH (content) AGAINST ("garcon"); > content > -Je suis un garçon. looks like a bug > DROP TABLE diaries; > diff --git a/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_japanese.result b/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_japanese.result > index 94ef2608b81..79dac1e63a7 100644 > --- a/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_japanese.result > +++ b/storage/mroonga/mysql-test/mroonga/storage/r/collation_utf8_unicode_ci_japanese.result > @@ -7,5 +7,4 @@ FULLTEXT INDEX (content) > INSERT INTO diaries VALUES ("ひらがなとカタカナを覚えました。"); > SELECT * FROM diaries WHERE MATCH (content) AGAINST ("かたかな"); > content > -ひらがなとカタカナを覚えました。 that too > DROP TABLE diaries; > diff --git a/storage/mroonga/vendor/groonga/CMakeLists.txt b/storage/mroonga/vendor/groonga/CMakeLists.txt > index d271d4c4eb9..fc134b81cde 100644 > --- a/storage/mroonga/vendor/groonga/CMakeLists.txt > +++ b/storage/mroonga/vendor/groonga/CMakeLists.txt > @@ -268,7 +268,7 @@ if(UNIX) > ac_check_funcs(pthread_condattr_setpshared) > endif() > > -option(GRN_WITH_NFKC "use NFKC based UTF8 normalization." ON) > +option(GRN_WITH_NFKC "use NFKC based UTF8MB3 normalization." ON) not here, please > > if(WIN32) > ac_check_headers(winsock2.h) > diff --git a/storage/mroonga/vendor/groonga/benchmark/bench-nfkc.c b/storage/mroonga/vendor/groonga/benchmark/bench-nfkc.c > index ebae95b273b..3997b933e87 100644 > --- a/storage/mroonga/vendor/groonga/benchmark/bench-nfkc.c > +++ b/storage/mroonga/vendor/groonga/benchmark/bench-nfkc.c > @@ -83,11 +83,11 @@ static void > bench_char_type(gpointer user_data) > { > uint64_t code_point; > - char utf8[7]; > + char utf8mb3[7]; don't rename variables > > for (code_point = 1; code_point < MAX_UNICODE; code_point++) { > - ucs2utf8(code_point, (unsigned char *)utf8); > - grn_nfkc50_char_type(utf8); > + ucs2utf8(code_point, (unsigned char *)utf8mb3); > + grn_nfkc50_char_type(utf8mb3); > } > } > > diff --git a/storage/mroonga/vendor/groonga/vendor/plugins/groonga-normalizer-mysql/tool/dump_difference_utf8.rb b/storage/mroonga/vendor/groonga/vendor/plugins/groonga-normalizer-mysql/tool/dump_difference_utf8.rb > index 4b6fde8c7b0..ce20a2c5b40 100644 > --- a/storage/mroonga/vendor/groonga/vendor/plugins/groonga-normalizer-mysql/tool/dump_difference_utf8.rb > +++ b/storage/mroonga/vendor/groonga/vendor/plugins/groonga-normalizer-mysql/tool/dump_difference_utf8.rb > @@ -36,8 +36,8 @@ parser.sorted_pages.each do |page, characters| > next if base == sort > n_differences += 1 > utf8s = [base, upper, lower, sort] > - formatted_code_points = utf8s.collect do |utf8| > - "%#07x" % Unicode.from_utf8(utf8) > + formatted_code_points = utf8s.collect do |utf8mb3| > + "%#07x" % Unicode.from_utf8(utf8mb3) just revert all changes under storage/mroonga/vendor/* > end > if sort.bytesize > base.bytesize > n_expanded_sort_characters += 1 > diff --git a/storage/sphinx/mysql-test/sphinx/my.cnf b/storage/sphinx/mysql-test/sphinx/my.cnf > index f60380b7171..22cc06914f4 100644 > --- a/storage/sphinx/mysql-test/sphinx/my.cnf > +++ b/storage/sphinx/mysql-test/sphinx/my.cnf > @@ -7,7 +7,7 @@ xmlpipe_command = cat @ENV.MTR_SUITE_DIR/testdata.xml > [index test1] > source = src1 > docinfo = extern > -charset_type = utf-8 > +charset_type = utf-8mb3 revert > path = @ENV.MYSQLTEST_VARDIR/searchd/test1 > > [indexer] > diff --git a/storage/spider/mysql-test/spider/bg/my.cnf b/storage/spider/mysql-test/spider/bg/my.cnf > index 246099c623e..39f5bd01c67 100644 > --- a/storage/spider/mysql-test/spider/bg/my.cnf > +++ b/storage/spider/mysql-test/spider/bg/my.cnf > @@ -75,15 +75,15 @@ MASTER_1_MYSOCK= @mysqld.1.1.socket > MASTER_1_ENGINE_TYPE= Spider > #MASTER_1_ENGINE_TYPE= MyISAM > MASTER_1_ENGINE= ENGINE=Spider > -MASTER_1_CHARSET= DEFAULT CHARSET=utf8 > +MASTER_1_CHARSET= DEFAULT CHARSET=utf8mb3 Make sure to update both charset and collation here and in all other .cnf files in the spider suite > MASTER_1_ENGINE2= ENGINE=MyISAM > -MASTER_1_CHARSET2= DEFAULT CHARSET=utf8 > -MASTER_1_CHARSET3= DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci > +MASTER_1_CHARSET2= DEFAULT CHARSET=utf8mb3 > +MASTER_1_CHARSET3= DEFAULT CHARSET=utf8mb3 COLLATE=utf8_unicode_ci > SLAVE1_1_MYPORT= @mysqld.4.1.port > SLAVE1_1_MYSOCK= @mysqld.4.1.socket > SLAVE1_1_ENGINE_TYPE= MyISAM > SLAVE1_1_ENGINE= ENGINE=MyISAM > -SLAVE1_1_CHARSET= DEFAULT CHARSET=utf8 > +SLAVE1_1_CHARSET= DEFAULT CHARSET=utf8mb3 > USE_CHILD_GROUP2= 1 > OUTPUT_CHILD_GROUP2= 0 > CHILD2_1_MYPORT= @mysqld.2.1.port > diff --git a/storage/spider/spd_init_query.h b/storage/spider/spd_init_query.h > index 19b04d50b82..f12cef377e2 100644 > --- a/storage/spider/spd_init_query.h > +++ b/storage/spider/spd_init_query.h > @@ -559,7 +559,7 @@ static LEX_STRING spider_init_queries[] = { > " table_name char(64) not null default ''," > " primary key (table_id)," > " unique uk1(db_name, table_name)" > - " ) engine=Aria transactional=1 default charset=utf8 collate=utf8_bin;" > + " ) engine=Aria transactional=1 default charset=utf8mb3 collate=utf8_bin;" always change both charset and collation (everywhere in this file) > " create table if not exists mysql.spider_rewrite_table_tables(" > " table_id bigint unsigned not null," > " partition_id bigint unsigned not null auto_increment," > diff --git a/tests/mysql_client_test.c b/tests/mysql_client_test.c > index 0043786d477..5d9213591fb 100644 > --- a/tests/mysql_client_test.c > +++ b/tests/mysql_client_test.c > @@ -19236,7 +19236,7 @@ static void test_bug12337762() > rc= mysql_query(mysql, "create table charset_tab("\ > "txt1 varchar(32) character set Latin1,"\ > "txt2 varchar(32) character set Latin1 collate latin1_bin,"\ > - "txt3 varchar(32) character set utf8 collate utf8_bin"\ > + "txt3 varchar(32) character set utf8mb3 collate utf8_bin"\ both charset and collation > ")"); > > DIE_UNLESS(rc == 0); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Old buildbot 10.5 branch failure review
by Otto Kekäläinen 19 Mar '21

19 Mar '21

Hello! I noticed there are quite a lot of upgrade test failures on the 10.5 branch, which is a stable release and should definitely be all green all the time. Here is quick review of those issues with suggested fixes. Maybe Daniel or Elena can do them? They are all easy ones and would make 7 currently red CI steps turn into green. ## Overview: http://buildbot.askmonty.org/buildbot/grid?branch=10.5&category=main&catego… ## kvm-deb-buster-amd6 minor-upgrade-all (http://buildbot.askmonty.org/buildbot/builders/kvm-deb-buster-amd64/builds/…) ++ diff -U1000 /home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64 /home/buildbot/ldd.new --- /home/buildbot/mariadb.org-tools-master/buildbot/baselines/ldd.10.5.buster.amd64 2021-03-16 21:03:04.000000000 -0400 +++ /home/buildbot/ldd.new 2021-03-17 17:23:42.066634818 -0400 - liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 Seems some dependency stopped including liblz4. This didn't affect MariaDB. This is a valid change and the fix is to "reset" the ldd.10.5.buster.amd64 file on buildbot.askmonty.org so that there would be no diff no more. ## kvm-deb-buster-amd64 minor-upgrade-all - liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 Same as above. ## kvm-deb-sid-x86 - minor-upgrade-all and ## kvm-deb-sid-amd64 - minor-upgrade-all: - libhogweed.so.5 => /lib/x86_64-linux-gnu/libhogweed.so.5 + libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6 - libnettle.so.7 => /lib/x86_64-linux-gnu/libnettle.so.7 + libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 Similar issue to buster minor-upgrade-all failure. This is a valid dependency change and the fix is to "reset" the ldd.10.5.sid.amd64 and ldd.10.5.sid.x86 filse on buildbot.askmonty.org so that there would be no diff no more. ## kvm-zyp-opensuse150-amd64 minor-upgrade-all: --- /tmp/ldd.baseline 2021-03-17 19:42:17.468328791 +0000 +++ /home/buildbot/ldd.new 2021-03-17 19:42:16.928328910 +0000 === /usr/sbin/mysqld === /usr/bin/mysql + libedit.so.0 => /usr/lib64/libedit.so.0 === /usr/bin/mysqlaccess === /usr/bin/mysqladmin === /usr/bin/mysqlbinlog === /usr/bin/mysqlcheck === /usr/bin/mysql_client_test === /usr/bin/mysql_client_test_embedded === /usr/bin/mysql_config === /usr/bin/mysql_convert_table_format === /usr/bin/mysqld_multi === /usr/bin/mysqld_safe === /usr/bin/mysqld_safe_helper === /usr/bin/mysqldump === /usr/bin/mysqldumpslow === /usr/bin/mysql_embedded + libedit.so.0 => /usr/lib64/libedit.so.0 Seems legit, library has correct path now. Fix by "resetting" the ldd.baseline. # kvm-zyp-sles150-amd64 minor-upgrade-all: --- /tmp/ldd.baseline 2021-03-17 20:59:56.412361298 +0000 +++ /home/buildbot/ldd.new 2021-03-17 20:59:56.024361282 +0000 === /usr/sbin/mysqld - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql === /usr/bin/mysqlaccess === /usr/bin/mysqladmin === /usr/bin/mysqlbinlog === /usr/bin/mysqlcheck === /usr/bin/mysql_client_test === /usr/bin/mysql_client_test_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql_config === /usr/bin/mysql_convert_table_format === /usr/bin/mysqld_multi === /usr/bin/mysqld_safe === /usr/bin/mysqld_safe_helper === /usr/bin/mysqldump === /usr/bin/mysqldumpslow === /usr/bin/mysql_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 === /usr/bin/mysql_find_rows === /usr/bin/mysql_fix_extensions === /usr/bin/mysqlhotcopy === /usr/bin/mysqlimport === /usr/bin/mysql_install_db === /usr/bin/mysql_ldb === /usr/bin/mysql_plugin === /usr/bin/mysql_secure_installation === /usr/bin/mysql_setpermission === /usr/bin/mysqlshow === /usr/bin/mysqlslap === /usr/bin/mysqltest - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 - libpcre2-posix.so.2 => /usr/lib64/libpcre2-posix.so.2 === /usr/bin/mysqltest_embedded - libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 - libpcre2-posix.so.2 => /usr/lib64/libpcre2-posix.so.2 Seems legit, library has correct path now. Fix by "resetting" the ldd.baseline. ## kvm-deb-sid-amd64 minor-upgrade-columnstore: The following packages have unmet dependencies: mariadb-plugin-columnstore : Depends: mariadb-server-10.5 (= 1:10.5.8+maria~sid) but 1:10.5.9+maria~sid is to be installed E: Unable to correct problems, you have held broken packages. The mariadb-plugin-columnstore binary packages don't exist for kvm-deb-sid-amd64, as they were skipped when making the 10.5.9 release. Either build them or disable the test until 10.5.10 is out and they are again built. By disabling the test there would be less "red" and less time wasted by developers checking the red ones on why their commits (potentially) fail on CI. Thanks!

4 3

[Maria-developers] 0038c24f2d3: MDEV-16146: MariaDB slave stops with following errors.
by sujatha 18 Mar '21

18 Mar '21

revision-id: 0038c24f2d388b9b0f3bff9219a53c8900153d02 (mariadb-10.5.4-539-g0038c24f2d3) parent(s): 190a8312f598fc4892331225104297f6288f23ac author: Sujatha committer: Sujatha timestamp: 2021-03-18 18:41:30 +0530 message: MDEV-16146: MariaDB slave stops with following errors. Problem: ======== 180511 11:07:58 [ERROR] Slave I/O: Unexpected master's heartbeat data: heartbeat is not compatible with local info;the event's data: log_file_name mysql-bin.000009 log_pos 1054262041, Error_code: 1623 Analysis: ========= In replication setup when master server doesn't have any events to send to slave server it sends an 'Heartbeat_log_event'. This event carries the current binary log filename and offset details. The offset values is stored within 4 bytes of event header. When the size of binary log is higher than UINT32_MAX the log_pos values will not fit in 4 bytes memory. It overflows and hence slave stops with an error. Fix: === Since we cannot extend the fixed header of Log_event class an additional header named 'extra_header' is introduced. 'extra_header' contains HB_FLAGS - 2 bytes HB_LONG_LOG_POS_OFFSET - 8 bytes This 'extra_header' is added only in a case where log_pos > UINT32_MAX and 'HB_LONG_LOG_POS_OFFSET_F' will be enabled. On slave while reading the 'Heartbeat_log_event' if 'HB_LONG_LOG_POS_OFFSET_F' is found to be set then value of 'log_pos' is extracted from 'extra_header'. --- .../suite/rpl/r/rpl_incompatible_heartbeat.result | 20 +++++++++ .../suite/rpl/t/rpl_incompatible_heartbeat.test | 47 ++++++++++++++++++++++ sql/log_event.h | 19 ++++++++- sql/log_event_server.cc | 8 +++- sql/slave.cc | 14 +++++++ sql/sql_repl.cc | 26 ++++++++++-- 6 files changed, 128 insertions(+), 6 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_incompatible_heartbeat.result b/mysql-test/suite/rpl/r/rpl_incompatible_heartbeat.result new file mode 100644 index 00000000000..fe2b0436f2d --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_incompatible_heartbeat.result @@ -0,0 +1,20 @@ +include/master-slave.inc +[connection master] +connection master; +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= '+d,simulate_pos_4G'; +connection slave; +include/stop_slave.inc +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= '+d,simulate_pos_4G'; +CHANGE MASTER TO MASTER_HEARTBEAT_PERIOD=0.001; +include/start_slave.inc +connection master; +SET @@GLOBAL.debug_dbug = @saved_dbug; +connection slave; +SET @@GLOBAL.debug_dbug = @saved_dbug; +connection master; +CREATE TABLE t (f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +DROP TABLE t; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_incompatible_heartbeat.test b/mysql-test/suite/rpl/t/rpl_incompatible_heartbeat.test new file mode 100644 index 00000000000..b1aa5435ff4 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_incompatible_heartbeat.test @@ -0,0 +1,47 @@ +# ==== Purpose ==== +# +# Test verifies that slave IO thread can process heartbeat events with log_pos +# values higher than UINT32_MAX. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Stop slave threads. Configure a small master_heartbeat_period. +# 1 - Using debug points, simulate a huge binlog offset higher than +# UINT32_MAX on master. +# 2 - Start the slave and observe that slave IO thread is able to process +# the offset received through heartbeat event. +# +# ==== References ==== +# +# MDEV-16146: MariaDB slave stops with incompatible heartbeat +# +--source include/have_debug.inc +--source include/have_innodb.inc +--source include/have_binlog_format_mixed.inc +# Test simulates binarylog offsets higher than UINT32_MAX +--source include/have_64bit.inc +--source include/master-slave.inc + +--connection master +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= '+d,simulate_pos_4G'; + +--connection slave +--source include/stop_slave.inc +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= '+d,simulate_pos_4G'; +CHANGE MASTER TO MASTER_HEARTBEAT_PERIOD=0.001; +--source include/start_slave.inc + +--connection master +sleep 1; +SET @@GLOBAL.debug_dbug = @saved_dbug; +--sync_slave_with_master +SET @@GLOBAL.debug_dbug = @saved_dbug; + +--connection master +CREATE TABLE t (f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +DROP TABLE t; +--source include/rpl_end.inc diff --git a/sql/log_event.h b/sql/log_event.h index 4e193232f4b..72d8766cea5 100644 --- a/sql/log_event.h +++ b/sql/log_event.h @@ -576,6 +576,22 @@ class String; #define MARIA_SLAVE_CAPABILITY_MINE MARIA_SLAVE_CAPABILITY_GTID +#define HB_EXTRA_HEADER_LEN 10 +/* + Extra header contains flags specific to Heartbeat_log_event and + respective data. The length needs to be updated when new memebers are + added to this header +*/ +#define HB_FLAGS_OFFSET 0 +/* + When the size of 'log_pos' within Heartbeat_log_event exceeds UINT_MAX32 it + cannot be stored in standard header as 'log_pos' is of 4 bytes size. Hence + extra_header is introduced. First 2 bytes represent flags. In case of long + 'log_pos' value 'HB_LONG_LOG_POS_OFFSET_F' bit within the flag will be set. + The log_pos is stored witin 'long_log_pos' variables. +*/ +#define HB_LONG_LOG_POS_OFFSET_F 0x1 +#define HB_LONG_LOG_POS_OFFSET 2 /** @enum Log_event_type @@ -5718,7 +5734,8 @@ bool copy_cache_to_file_wrapped(IO_CACHE *body, class Heartbeat_log_event: public Log_event { public: - Heartbeat_log_event(const char* buf, uint event_len, + uint16 hb_flags; + Heartbeat_log_event(const char* buf, ulong event_len, const Format_description_log_event* description_event); Log_event_type get_type_code() { return HEARTBEAT_LOG_EVENT; } bool is_valid() const diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc index 6dee9b9adf6..2701f4f2f9a 100644 --- a/sql/log_event_server.cc +++ b/sql/log_event_server.cc @@ -8493,14 +8493,18 @@ void Ignorable_log_event::pack_info(Protocol *protocol) #if defined(HAVE_REPLICATION) -Heartbeat_log_event::Heartbeat_log_event(const char* buf, uint event_len, +Heartbeat_log_event::Heartbeat_log_event(const char* buf, ulong event_len, const Format_description_log_event* description_event) :Log_event(buf, description_event) { + uint16 hb_flags; uint8 header_size= description_event->common_header_len; - ident_len = event_len - header_size; + ident_len = event_len - (header_size + HB_EXTRA_HEADER_LEN); set_if_smaller(ident_len,FN_REFLEN-1); log_ident= buf + header_size; + hb_flags= uint2korr(buf + header_size + ident_len); + if (hb_flags & HB_LONG_LOG_POS_OFFSET_F) + log_pos= uint8korr(buf + header_size + ident_len + HB_LONG_LOG_POS_OFFSET); } #endif diff --git a/sql/slave.cc b/sql/slave.cc index 1da030084ef..f365a1e7b5c 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -6493,6 +6493,12 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) TODO: handling `when' for SHOW SLAVE STATUS' snds behind */ + DBUG_EXECUTE_IF("simulate_pos_4G", + { + inc_pos= mi->master_log_pos; // temp_save + mi->master_log_pos= ((ulong)2394967295); + };); + if (memcmp(mi->master_log_name, hb.get_log_ident(), hb.get_ident_len()) || mi->master_log_pos > hb.log_pos) { /* missed events of heartbeat from the past */ @@ -6504,6 +6510,14 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) error_msg.append_ulonglong(hb.log_pos); goto err; } + DBUG_EXECUTE_IF("simulate_pos_4G", + { + if (hb.log_pos > UINT32_MAX) + { + mi->master_log_pos= inc_pos; // restore + DBUG_SET("-d, simulate_pos_4G"); + } + };); /* Heartbeat events doesn't count in the binlog size, so we don't have to diff --git a/sql/sql_repl.cc b/sql/sql_repl.cc index 010ea794254..d68ae05b60e 100644 --- a/sql/sql_repl.cc +++ b/sql/sql_repl.cc @@ -824,12 +824,14 @@ get_slave_until_gtid(THD *thd, String *out_str) */ static int send_heartbeat_event(binlog_send_info *info, NET* net, String* packet, - const struct event_coordinates *coord, + struct event_coordinates *coord, enum enum_binlog_checksum_alg checksum_alg_arg) { DBUG_ENTER("send_heartbeat_event"); ulong ev_offset; + char extra_buf[HB_EXTRA_HEADER_LEN]; + uint16 hb_flags= 0; if (reset_transmit_packet(info, info->flags, &ev_offset, &info->errmsg)) DBUG_RETURN(1); @@ -850,19 +852,30 @@ static int send_heartbeat_event(binlog_send_info *info, size_t event_len = ident_len + LOG_EVENT_HEADER_LEN + (do_checksum ? BINLOG_CHECKSUM_LEN : 0); int4store(header + SERVER_ID_OFFSET, global_system_variables.server_id); + if (coord->pos <= UINT32_MAX) + { + int4store(header + LOG_POS_OFFSET, coord->pos); // log_pos + } + else + { + hb_flags|= HB_LONG_LOG_POS_OFFSET_F; + int2store(extra_buf + HB_FLAGS_OFFSET, hb_flags); + int8store(extra_buf + HB_LONG_LOG_POS_OFFSET, coord->pos); + event_len+= HB_EXTRA_HEADER_LEN; + } int4store(header + EVENT_LEN_OFFSET, event_len); int2store(header + FLAGS_OFFSET, 0); - int4store(header + LOG_POS_OFFSET, coord->pos); // log_pos - packet->append(header, sizeof(header)); packet->append(p, ident_len); // log_file_name + packet->append(extra_buf, sizeof(extra_buf)); if (do_checksum) { char b[BINLOG_CHECKSUM_LEN]; ha_checksum crc= my_checksum(0, (uchar*) header, sizeof(header)); crc= my_checksum(crc, (uchar*) p, ident_len); + crc= my_checksum(crc, (uchar*) extra_buf, sizeof(extra_buf)); int4store(b, crc); packet->append(b, sizeof(b)); } @@ -2525,6 +2538,13 @@ static int wait_new_events(binlog_send_info *info, /* in */ } #endif mysql_bin_log.unlock_binlog_end_pos(); + + DBUG_EXECUTE_IF("simulate_pos_4G", + { + coord.pos= ((ulong)5394967295); + DBUG_SET("-d, simulate_pos_4G"); + };); + ret= send_heartbeat_event(info, info->net, info->packet, &coord, info->current_checksum_alg);

1 0

Re: [Maria-developers] 9acabefd276: MDEV-22010: use executables MariaDB named in scripts
by Sergei Golubchik 17 Mar '21

17 Mar '21

Hi, Rucha! On Mar 17, Rucha Deodhar wrote: > > mysql-test-run and mysql-stress-test was also supposed to be renamed as > part of this task. > I need to check again if debs and rpms package symlink. > > > > > > diff --git a/mysql-test/.mtr b/mysql-test/.mtr > > > new file mode 120000 > > > index 00000000000..68986c8cd7e > > > --- /dev/null > > > +++ b/mysql-test/.mtr > > > @@ -0,0 +1 @@ > > > +./mariadb-test-run > > > > What is that? What is .mtr file? > > I think after i built it got committed along with other files. please remove it from the commit > > > diff --git a/mysql-test/mysql-stress-test.pl b/mysql-test/ > > mysql-stress-test.pl > > > new file mode 120000 > > > index 00000000000..e704a5d70bf > > > --- /dev/null > > > +++ b/mysql-test/mysql-stress-test.pl > > > @@ -0,0 +1 @@ > > > +./mariadb-stress-test.pl > > > > looks like you've checked in files you shouldn't have. Like symlinks. > > As mysql-test-run and mysql-stress-test was supposed to be renamed too. So, > I created a symlink > so that ./mysql-test-run and ./mysql-stress-test would still work if > anybody uses it. symlinks - yes. But they are created by cmake during the build, they should not be added to a commit. please, remove. You might want to add them to the .gitignore file though. > > > \ No newline at end of file > > > diff --git a/scripts/msql2mysql.sh b/scripts/msql2mysql.sh > > > index 72a609fa6e7..11923fa15c7 100644 > > > --- a/scripts/msql2mysql.sh > > > +++ b/scripts/msql2mysql.sh > > > @@ -16,4 +16,4 @@ > > > # Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston > > > # MA 02110-1335 USA. > > > > > > -@bindir@/replace msqlConnect mysql_connect msqlListDBs mysql_list_dbs > > msqlNumRows mysql_num_rows msqlFetchRow mysql_fetch_row msqlFetchField > > mysql_fetch_field msqlFreeResult mysql_free_result msqlListFields > > mysql_list_fields msqlListTables mysql_list_tables msqlErrMsg > > 'mysql_error(mysql)' msqlStoreResult mysql_store_result msqlQuery > > mysql_query msqlField mysql_field msqlSelect mysql_select msqlSelectDB > > mysql_select_db msqlNumFields mysql_num_fields msqlClose mysql_close > > msqlDataSeek mysql_data_seek m_field MYSQL_FIELD m_result MYSQL_RES m_row > > MYSQL_ROW msql mysql mSQL mySQL MSQL MYSQL msqlCreateDB mysql_create_db > > msqlDropDB mysql_drop_db msqlFieldSeek mysql_field_seek -- $* > > > +@bindir@/replace msqlConnect mysql_connect msqlListDBs mysql_list_dbs > > msqlNumRows mysql_num_rows msqlFetchRow mysql_fetch_row msqlFetchField > > mysql_fetch_field msqlFreeResult mysql_free_result msqlListFields > > mysql_list_fields msqlListTables mysql_list_tables msqlErrMsg > > 'mysql_error(mysql)' msqlStoreResult mysql_store_result msqlQuery > > mysql_query msqlField mysql_field msqlSelect mysql_select msqlSelectDB > > mysql_select_db msqlNumFields mysql_num_fields msqlClose mysql_close > > msqlDataSeek mysql_data_seek m_field MYSQL_FIELD m_result MYSQL_RES m_row > > MYSQL_ROW msql mysql mysql mariadb mSQL mySQL MSQL MYSQL msqlCreateDB > > mysql_create_db msqlDropDB mysql_drop_db msqlFieldSeek mysql_field_seek -- > > $* > > > > Why? > > So that mysql in bin directory would get replaced with mariadb no, that's not it. See other renames, msqlListTables -> mysql_list_tables, msqlFetchRow -> mysql_fetch_row etc this (prehistoric) script was supposed to help migrating programs that work with mSQL (https://en.wikipedia.org/wiki/MSQL) to MySQL by renaming C API calls in the source code. In short, revert this change, it doesn't belong to this script. > > > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > > > index 5f183afe8fc..084af42d01c 100644 > > > --- a/scripts/mysql_install_db.sh > > > +++ b/scripts/mysql_install_db.sh > > > @@ -338,11 +338,15 @@ then > > > cannot_find_file resolveip @resolveip_locations@ > > > exit 1 > > > fi > > > - mysqld=`find_in_dirs mysqld @mysqld_locations@` > > > + mysqld=`find_in_dirs mariadbd @mysqld_locations@` > > > if test -z "$mysqld" > > > then > > > - cannot_find_file mysqld @mysqld_locations@ > > > - exit 1 > > > + mysqld=`find_in_dirs mysqld @mysqld_locations@` > > > + if test -z "$mysqld" > > > + then > > > + cannot_find_file mysqld @mysqld_locations@ > > > > Hmm. At least, it should be mariadbd in the cannot_find_file line. > > > > Why do you search for mysqld after mariadbd wasn't found? > > > I added it for additional check, just in case there is mysqld and not > mariadbd or if it could affect compatibility. I don't think it could. These scripts you've edited are installed together with MariaDB, there is no supported configuration where these scripts exist and mariadb* binaries don't. That is, I believe this extra check is not needed. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9acabefd276: MDEV-22010: use executables MariaDB named in scripts
by Sergei Golubchik 17 Mar '21

17 Mar '21

Hi, Rucha! On Mar 17, Rucha Deodhar wrote: > revision-id: 9acabefd276 (mariadb-10.5.2-386-g9acabefd276) > parent(s): de407e7cb4d > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-02-11 15:19:29 +0530 > message: > > MDEV-22010: use executables MariaDB named in scripts > > As a part of this MDEV following changes were made: > 1) Mariadb named executables used instead of mysql named executables in scripts > 2) renamed mysql-test-run and mysql-stress-test to mariadb-test-run and > mariadb-stress-test and created a symlink. > > diff --git a/debian/mariadb-test.links b/debian/mariadb-test.links > index 718a5355474..b3577667d01 100644 > --- a/debian/mariadb-test.links > +++ b/debian/mariadb-test.links > @@ -2,5 +2,7 @@ usr/bin/mariadb-client-test usr/bin/mysql_client_test > usr/bin/mariadb-client-test-embedded usr/bin/mysql_client_test_embedded > usr/bin/mariadb-test usr/bin/mysqltest > usr/bin/mariadb-test-embedded usr/bin/mysqltest_embedded > -usr/share/mysql/mysql-test/mysql-test-run.pl usr/share/mysql/mysql-test/mtr > -usr/share/mysql/mysql-test/mysql-test-run.pl usr/share/mysql/mysql-test/mysql-test-run > +usr/share/mysql/mysql-test/mariadb-test-run.pl usr/share/mysql/mysql-test/mysql-test-run.pl > +usr/share/mysql/mysql-test/mariadb-stress-test.pl usr/share/mysql/mysql-test/mysql-stress-test.pl > +usr/share/mysql/mysql-test/mariadb-test-run.pl usr/share/mysql/mysql-test/mysql-test-run > +usr/share/mysql/mysql-test/mariadb-test-run.pl usr/share/mysql/mysql-test/mtr Not sure that was part of the task, but ok. Did you check that both debs and rpms package all new symlinks correctly? > diff --git a/mysql-test/.mtr b/mysql-test/.mtr > new file mode 120000 > index 00000000000..68986c8cd7e > --- /dev/null > +++ b/mysql-test/.mtr > @@ -0,0 +1 @@ > +./mariadb-test-run What is that? What is .mtr file? > \ No newline at end of file > diff --git a/mysql-test/mysql-stress-test.pl b/mysql-test/mysql-stress-test.pl > new file mode 120000 > index 00000000000..e704a5d70bf > --- /dev/null > +++ b/mysql-test/mysql-stress-test.pl > @@ -0,0 +1 @@ > +./mariadb-stress-test.pl looks like you've checked in files you shouldn't have. Like symlinks. > \ No newline at end of file > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > new file mode 120000 > index 00000000000..56150a0ecad > --- /dev/null > +++ b/mysql-test/mysql-test-run.pl > @@ -0,0 +1 @@ > +./mariadb-test-run.pl another one > \ No newline at end of file > diff --git a/scripts/msql2mysql.sh b/scripts/msql2mysql.sh > index 72a609fa6e7..11923fa15c7 100644 > --- a/scripts/msql2mysql.sh > +++ b/scripts/msql2mysql.sh > @@ -16,4 +16,4 @@ > # Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston > # MA 02110-1335 USA. > > -@bindir@/replace msqlConnect mysql_connect msqlListDBs mysql_list_dbs msqlNumRows mysql_num_rows msqlFetchRow mysql_fetch_row msqlFetchField mysql_fetch_field msqlFreeResult mysql_free_result msqlListFields mysql_list_fields msqlListTables mysql_list_tables msqlErrMsg 'mysql_error(mysql)' msqlStoreResult mysql_store_result msqlQuery mysql_query msqlField mysql_field msqlSelect mysql_select msqlSelectDB mysql_select_db msqlNumFields mysql_num_fields msqlClose mysql_close msqlDataSeek mysql_data_seek m_field MYSQL_FIELD m_result MYSQL_RES m_row MYSQL_ROW msql mysql mSQL mySQL MSQL MYSQL msqlCreateDB mysql_create_db msqlDropDB mysql_drop_db msqlFieldSeek mysql_field_seek -- $* > +@bindir@/replace msqlConnect mysql_connect msqlListDBs mysql_list_dbs msqlNumRows mysql_num_rows msqlFetchRow mysql_fetch_row msqlFetchField mysql_fetch_field msqlFreeResult mysql_free_result msqlListFields mysql_list_fields msqlListTables mysql_list_tables msqlErrMsg 'mysql_error(mysql)' msqlStoreResult mysql_store_result msqlQuery mysql_query msqlField mysql_field msqlSelect mysql_select msqlSelectDB mysql_select_db msqlNumFields mysql_num_fields msqlClose mysql_close msqlDataSeek mysql_data_seek m_field MYSQL_FIELD m_result MYSQL_RES m_row MYSQL_ROW msql mysql mysql mariadb mSQL mySQL MSQL MYSQL msqlCreateDB mysql_create_db msqlDropDB mysql_drop_db msqlFieldSeek mysql_field_seek -- $* Why? > diff --git a/scripts/mysql_install_db.sh b/scripts/mysql_install_db.sh > index 5f183afe8fc..084af42d01c 100644 > --- a/scripts/mysql_install_db.sh > +++ b/scripts/mysql_install_db.sh > @@ -338,11 +338,15 @@ then > cannot_find_file resolveip @resolveip_locations@ > exit 1 > fi > - mysqld=`find_in_dirs mysqld @mysqld_locations@` > + mysqld=`find_in_dirs mariadbd @mysqld_locations@` > if test -z "$mysqld" > then > - cannot_find_file mysqld @mysqld_locations@ > - exit 1 > + mysqld=`find_in_dirs mysqld @mysqld_locations@` > + if test -z "$mysqld" > + then > + cannot_find_file mysqld @mysqld_locations@ Hmm. At least, it should be mariadbd in the cannot_find_file line. Why do you search for mysqld after mariadbd wasn't found? > + exit 1 > + fi > fi > langdir=`find_in_dirs --dir errmsg.sys @errmsg_locations@` > if test -z "$langdir" > @@ -579,7 +583,7 @@ else > echo > echo " shell> $mysqld --skip-grant-tables --general-log &" > echo > - echo "and use the command line tool $bindir/mysql" > + echo "and use the command line tool $bindir/mariadb or $bindir/mysql" again, why both? > echo "to connect to the mysql database and look at the grant tables:" > echo > echo " shell> $bindir/mysql -u root mysql" > @@ -613,7 +617,9 @@ then > echo "PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !" > echo "To do so, start the server, then issue the following commands:" > echo > + echo "'$bindir/mariadb-admin' -u root password 'new-password' or" > echo "'$bindir/mysqladmin' -u root password 'new-password'" > + echo "'$bindir/mariadb-admin' -u root -h $hostname password 'new-password' or" > echo "'$bindir/mysqladmin' -u root -h $hostname password 'new-password'" and here > echo > echo "Alternatively you can run:" > diff --git a/scripts/mysql_secure_installation.sh b/scripts/mysql_secure_installation.sh > index b2a9edf4953..12364f1be43 100644 > --- a/scripts/mysql_secure_installation.sh > +++ b/scripts/mysql_secure_installation.sh > @@ -159,15 +159,19 @@ then > cannot_find_file my_print_defaults $basedir/bin $basedir/extra > exit 1 > fi > - mysql_command=`find_in_basedir mysql bin` > + mysql_command=`find_in_basedir mariadb bin` > if test -z "$mysql_command" > then > - cannot_find_file mysql $basedir/bin > - exit 1 > + mysql_command=`find_in_basedir mysql bin` > + if test -z "$mysql_command" > + then > + cannot_find_file mysql $basedir/bin > + exit 1 > + fi same as above > fi > else > print_defaults="@bindir@/my_print_defaults" > - mysql_command="@bindir@/mysql" > + mysql_command="@bindir@/mariadb" > fi > > if test ! -x "$print_defaults" > diff --git a/scripts/mysqld_safe.sh b/scripts/mysqld_safe.sh > index 126eb924814..7763fb9db91 100644 > --- a/scripts/mysqld_safe.sh > +++ b/scripts/mysqld_safe.sh > @@ -531,7 +531,11 @@ else > ledir='@libexecdir@' > fi > > -helper=`find_in_bin mysqld_safe_helper` > +helper=`find_in_bin mariadbd-safe-helper` > +if test -x helper > +then > + helper=`find_in_bin mysqld_safe_helper` > +fi and here > print_defaults=`find_in_bin my_print_defaults` > # Check if helper exists > command -v $helper --help >/dev/null 2>&1 > diff --git a/scripts/wsrep_sst_common.sh b/scripts/wsrep_sst_common.sh > index 5e134570881..cf38bda8f70 100644 > --- a/scripts/wsrep_sst_common.sh > +++ b/scripts/wsrep_sst_common.sh > @@ -273,13 +273,21 @@ SCRIPTS_DIR="$(cd $(dirname "$0"); pwd -P)" > EXTRA_DIR="$SCRIPTS_DIR/../extra" > CLIENT_DIR="$SCRIPTS_DIR/../client" > > -if [ -x "$CLIENT_DIR/mysql" ]; then > +if [-x "$CLIENT_DIR/mariadb"]; then > + MYSQL_CLIENT="$CLIENT_DIR/mariadb" > +elif [-x mariadb] then; > + MYSQL_CLIENT=mariadb > +elif [ -x "$CLIENT_DIR/mysql" ]; then > MYSQL_CLIENT="$CLIENT_DIR/mysql" > else > MYSQL_CLIENT=mysql > fi > > -if [ -x "$CLIENT_DIR/mysqldump" ]; then > +if [ -x "$CLIENT_DIR/mariadb-dump" ]; then > + MYSQLDUMP="$CLIENT_DIR/mariadb-dump" > +elif [-x mariadb-dump]; then > + MYSQLDUMP=mariadb-dump > +elif [ -x "$CLIENT_DIR/mysqldump" ]; then > MYSQLDUMP="$CLIENT_DIR/mysqldump" > else > MYSQLDUMP=mysqldump and here Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 16 Mar '21

16 Mar '21

Hi, Andrei! Few comments/questions below. I still have troubles understanding the logic of your solution, so I did not ask any questions about those your replies - as I'd be just repeating the same thing over and over. After I'll start getting it finally, I'll go over your reply again. Meanwhile - minor comments/questions, not related to the main algorithm. On Mar 04, Andrei Elkin wrote: > >> To facilitate the failover on the slave side conditions to accept > >> own events (having been discarded by the above recovery) are > >> relaxed to let so for the semisync slave that connects to master in > >> gtid mode. gtid_strict_mode is further recommended to secure from > >> inadvertent re-applying out of order gtids in general. Non-gtid > >> mode connected semisync slave would require > >> --replicate-same-server-id (mind --log-slave-updates must be OFF > >> then). > > > > Sorry, I failed to understand this paragraph :( > > 'gtid_strict_mode is further recommended to secure > from inadvertent re-applying out of order gtids in general.' > > attempted to promote GTID connection with master, > which of course should not be of this patch's > business, but in a way it is 'cos the non-gtid's mode > (CHANGE MASTER TO ... master_use_gtid = NO) is covered by the patch > with a limitation of '--log-slave-updates must be OFF'. > > The GTID-ON connection is covered, and without any `--replicate-same-server-id'. This doesn't seem to clarify anything. I mean the whole paragraph, starting from "To facilitate" and to "then)." Could you try to rephrase it to make it clearer, please? > >> +--let $query1 = INSERT INTO t VALUES (20) > >> +--let $query2 = DELETE FROM t2 WHERE f = 666 /* no such record */ > >> +--source binlog_truncate_active_log.inc > >> + > >> +--echo # Case B. > >> +# The inverted sequence ends up to truncate only $query2 > >> +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-10 > >> +--let $query1 = DELETE FROM t2 WHERE f = 0 > > > > why not `= 666 /* no such record */` ? > > is `= 0` important here? > > Yes, there's a fine detail here. > > 'The inverted sequence ends up' to leave in binlog > > $query1 = DELETE FROM t2 WHERE f = 0 > > which is as ineffective as 666. The patch recognizes the ineffectiveness > through a special value of engines involved counter in Gtid event. > In principle the truncation could > start from it rather than the following $query2. > But that would mean more complicated coding. > So the semantics of the zero is to be good, as we're so used to :-)! Sorry, I didn't understand anything. What does the value in the WHERE clause has to do with engines and where the binlog is truncated? > > > >> +--let $query2 = INSERT INTO t VALUES (20) > >> +--source binlog_truncate_active_log.inc ... > >> +--connection master2 > >> +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > >> +if ($pre_q2) > > > > is $pre_q2 a flag or a query string? > > it's used as a flag, it's set as a query string. > > It's actually both with the idea execute it when defined. > Legit, right? legit, but it's not both. Were it both, you'd have if ($pre_q2) { eval $pre_q2; } but you have, if ($pre_q2) { CALL sp_blank_xa; } > >> +{ > >> + CALL sp_blank_xa; > >> +} ... > >> diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > >> new file mode 100644 > >> index 00000000000..fe153e5703c > >> --- /dev/null > >> +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > >> +# > >> +# ==== Implementation ==== > >> +# > >> +# Steps: > >> +# 0 - Create two tables in innodb and rocksdb engines, > >> +# > >> +# In loop for A,B,C cases (below) do 1-5: > > > > there's no loop now (which is good, don't add it please :) [x] ? > >> +# The 1st trx binlogs, rotate binlog and hold on before committing at engine > >> +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL master1_ready WAIT_FOR master1_go"; > > > > you don't wait for master1_ready anywhere. intentional? > > No, actually > > SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > > must be run by below master2 (sim to how master3 waits for master2). meaning [x] ? > >> +--source include/have_binlog_format_row.inc > >> +--let $rpl_topology=1->2 > >> +--source include/rpl_init.inc > > > > isn't this a normal master-slave.inc topology? > > Initially yes. > > I am guessing you mean why not to > --source it. > The test heavily uses 'server_`index`' format to automate > replication direction. as far as I remember, master-slave.inc also defines server_1 and server_2 connections. > >> diff --git a/sql/sql_class.h b/sql/sql_class.h > >> index 140394fefc1..87fa88d4c89 100644 > >> --- a/sql/sql_class.h > >> +++ b/sql/sql_class.h > >> @@ -4632,7 +4632,7 @@ class THD :public Statement, > >> LF_PINS *tdc_hash_pins; > >> LF_PINS *xid_hash_pins; > >> bool fix_xid_hash_pins(); > >> - > >> + bool is_1pc_ro_trans; > > > > I don't like how it looks, it's in THD (for every single connection > > and more) and it's only used in some exotic cases. It's set in two > > places around ha_commit_one_phase(), when it would be better to set > > it inside ha_commit_one_phase(), in one place only. > > > > But I cannot suggest how to get rid of it yet as I don't understand > > what it's for, I've asked a question about it below, in > > Gtid_log_event::Gtid_log_event > > The flag is to help sorting out ineffective (no engine data change) > STATEMENT binlog-format logged transaction. I thought it's practical > enough to face and important to handle. > Through an example, > let the table `t' be initially empty, and > let binlog contain > > trx1: INSERT INTO t SET a = 1; /* slave acked, trx got committed in engine */ > trx2: INSERT INTO t SET a = 2; /* not acked, trx remains prepared in engine */ > trx3: DELETE FROM t WHERE a = 0; /* ineffective, does not have Xid event */ > > prior a crash with their states as commented. > The truncate trx should be trx2 as the trx3's status > suggests it's harmless for involving into > truncation. > However we have yet to distinguish it from unsafe (to truncate) > event group ("transaction") that also ends with COMMIT query not Xid event. Hmm. I see. But thd isn't a universal dumpster for passing arguments to the function, don't use it like that. Please, pass this "read-only transaction with no xid" flag to the Gtid_log_event constructor as an argument. Or recalculate it inside the constructor, it's not such an expensive thing to do. > One rather drastic solution could be to not binlog DELETE altogether. > I did not look into this with much of interest earlier, but how about it? > Perhaps it'd more like a bug fixes than breaking legacy. No, I suspect it's too drastic > >> /* Members related to temporary tables. */ > >> public: > >> /* Opened table states. */ > >> diff --git a/sql/slave.cc b/sql/slave.cc > >> index 28e08e32346..9b3f73e5341 100644 > >> --- a/sql/slave.cc > >> +++ b/sql/slave.cc > >> @@ -6944,7 +6945,9 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) > >> } > >> else > >> if ((s_id == global_system_variables.server_id && > >> - !mi->rli.replicate_same_server_id) || > >> + (!mi->rli.replicate_same_server_id && > >> + !(semisync_recovery= (rpl_semi_sync_slave_enabled && > >> + mi->using_gtid != Master_info::USE_GTID_NO)))) || > > > > How can "semisync recovery" code path reach queue_event()? > > This is a failover now, after recovery. > The slave which is ex-master tries to adopt "own" (generated by it) events. 1. don't call it "semisync recovery" then 2. add a comment explaining this case (it's a rare and not obvious one) 3. why does it have to be specially recognized anyway? > > > >> event_that_should_be_ignored(buf) || > >> /* > >> the following conjunction deals with IGNORE_SERVER_IDS, if set > >> diff --git a/sql/log_event.h b/sql/log_event.h > >> index 8a342cb5cd3..1588ab85104 100644 > >> --- a/sql/log_event.h > >> +++ b/sql/log_event.h > >> @@ -482,6 +482,16 @@ class String; > >> */ > >> #define LOG_EVENT_IGNORABLE_F 0x80 > >> > >> +/** > >> + @def LOG_EVENT_ACCEPT_OWN_F > >> + > >> + Flag sets by the gtid-mode connected semisync slave for > >> + the same server_id ("own") events which the slave must not have > >> + in its state. Typically such events were never committed by > >> + their originator (this server) and discared at its crash recovery > > > > sorry, I failed to understand that > > This should've been explained above. > The slave which is ex-master truncated at recovery some of its ("own") transactions. > The flag serves to help recognize that at execution of the replication events. 1. yes, this is clearer, thanks. may be you should rewrite your comment putting this explanation in it? 2. why does it have to be specially recognized anyway? ... > > 3. also it's somewhat an unfounded assumption that only r/w engines > > will have the transaction prepared. But we can make it a fact, if we > > won't prepare r/o engines at all (in ha_prepare). > > [?] > > Sorry, I don't get it. > Do you mean that an r/o engine (branch) might need to be counted by > `ha_count_rw()`? But then it would be contributing with binlogged > events of its branch... I can't imagine that. I mean the following. Say, you have two engines participating in a transaction, like in INSERT innodb_table SELECT * FROM rocksdb_table; One is r/w, the other is r/o. It's a 2pc transaction, the server does prepare for everyone, then commit for everyone. I don't think there is any guarantee that prepare for an r/o engine will be a no-op. It is completely up to the engine what to do in this case, it's possible that the engine will optimize it to a no-op, but it is not a certainty. What I think we should do is not to call prepare() for r/o engines at all. Just skip straight to commit/rollback on the second phase. > >> diff --git a/sql/handler.cc b/sql/handler.cc > >> index 6792e80b8fe..247ab7267bf 100644 > >> --- a/sql/handler.cc > >> +++ b/sql/handler.cc > >> @@ -1245,6 +1245,24 @@ int ha_prepare(THD *thd) > >> DBUG_RETURN(error); > >> } > >> > >> +/* > >> + Returns counted number of > >> + read-write recoverable transaction participants. > >> +*/ > >> +uint ha_count_rw(THD *thd, bool all) > > > > the name doesn't match the implementation, please, rename > > [?] > > Please help here. > I could not find what specifically you mean. > It > returns rw_ha_count > > which I think as 'read-write recoverable transaction participants.' I read the name is "a number of rw engines in the ha_info", it does not mention "recoverable", and I from the name I woulnd't have guessed. May be ha_count_rw_2pc ? > >> +{ > >> + unsigned rw_ha_count= 0; > >> + THD_TRANS *trans=all ? &thd->transaction.all : &thd->transaction.stmt; > >> + > >> + for (Ha_trx_info * ha_info= trans->ha_list; ha_info; > >> + ha_info= ha_info->next()) > >> + { > >> + if (ha_info->is_trx_read_write() && ha_info->ht()->recover) > >> + ++rw_ha_count; > >> + } > >> + return rw_ha_count; > >> +} > >> + > >> /** > >> Check if we can skip the two-phase commit. > >> > >> @@ -1960,8 +1982,122 @@ struct xarecover_st > >> XID *list; > >> HASH *commit_list; > >> bool dry_run; > >> + MEM_ROOT *mem_root; > >> + bool error; > >> }; > >> > >> +/** > >> + Inserts a new hash member. > >> + > >> + returns a successfully created and inserted @c xid_recovery_member > >> + into hash @c hash_arg, > >> + or NULL. > >> +*/ > >> +static xid_recovery_member* > >> +xid_member_insert(HASH *hash_arg, my_xid xid_arg, MEM_ROOT *ptr_mem_root) > >> +{ > >> + xid_recovery_member *member= (xid_recovery_member*) > >> + alloc_root(ptr_mem_root, sizeof(xid_recovery_member)); > >> + if (!member) > >> + return NULL; > >> + > >> + member->xid= xid_arg; > >> + member->in_engine_prepare= 1; > >> + member->decided_to_commit= false; > >> + > >> + return my_hash_insert(hash_arg, (uchar*) member) ? NULL : member; > >> +} > >> + > >> +/* > >> + Inserts a new or updates an existing hash member to increment > >> + the member's prepare counter. > >> + > >> + returns false on success, > >> + true otherwise. > >> +*/ > >> +static bool xid_member_replace(HASH *hash_arg, my_xid xid_arg, > >> + MEM_ROOT *ptr_mem_root) > >> +{ > >> + xid_recovery_member* member; > >> + if ((member= (xid_recovery_member *) > >> + my_hash_search(hash_arg, (uchar *)& xid_arg, sizeof(xid_arg)))) > >> + member->in_engine_prepare++; > >> + else > >> + member= xid_member_insert(hash_arg, xid_arg, ptr_mem_root); > >> + > >> + return member == NULL; > >> +} > >> + > >> +/* > >> + Hash iterate function to complete with commit or rollback as decided > >> + (typically at binlog recovery processing) in member->in_engine_prepare. > >> +*/ > >> +static my_bool xarecover_do_commit_or_rollback(void *member_arg, > >> + void *hton_arg) > >> +{ > >> + xid_recovery_member *member= (xid_recovery_member*) member_arg; > >> + handlerton *hton= (handlerton*) hton_arg; > >> + xid_t x; > >> + my_bool rc; > >> + > >> + x.set(member->xid); > >> + rc= member->decided_to_commit ? hton->commit_by_xid(hton, &x) : > >> + hton->rollback_by_xid(hton, &x); > >> + > >> + DBUG_ASSERT(rc || member->in_engine_prepare > 0); > >> + > >> + if (!rc) > >> + { > > > > I don't think you can trust rc on that. > > if it's non-zero, it's an error all right. > > but it's a bit of a stretch to presume that > > nonexisting xid is always an error. > > > > also commit_by_xid returns int, not my_bool. > > [?] I mean, if an engine returns success when you ask it to commit or rollback an unknown xid - this is not a bug, right? Commit means "make all changes by this xid persistent", if there are no changes, there is nothing to do, so doing nothing and reporting success isn't exactly wrong. Same for rollback. > I'd rather to stick with the patch and require from any engine to > return \cite{xa spec} > > [XAER_NOTA] The specified XID is not known by the resource manager > > What'd you say? Okay. I'd say I was wrong, if the standard requires a resource manager to return an error for an unknown XID, then you surely can rely on that :) Sorry for confusion. > >> + > >> + return false; > >> +} > >> + > >> +static my_bool xarecover_do_count_in_prepare(void *member_arg, > >> + void *ptr_count) > >> +{ > >> + xid_recovery_member *member= (xid_recovery_member*) member_arg; > >> + if (member->in_engine_prepare) > >> + { > >> + *(uint*) ptr_count += member->in_engine_prepare; > > > > This is a rather weird semantics. > > it's kind of a number of transactions times number of engines. > > if one transaction wasn't committed in two engines and another > > transaction wasn't rolled back in one engine, the counter will be 3. > > how are you going to explain this to users? > > So `ptr_count` actually is for transaction branches, not transactions. > But the ultimate warning speaks erronously the latter > > + sql_print_warning("Could not complete %u number of transactions in " > "engines.", count_in_prepare); > > So I'd convert the counter to count the transactions: > > (*(uint*) ptr_count) ++ > > [?] yes, that'd be easier to understand, thanks > >> diff --git a/sql/log.cc b/sql/log.cc > >> index 8073f09ab88..a9808ed8823 100644 > >> --- a/sql/log.cc > >> +++ b/sql/log.cc ... > >> + char buf[21]; > >> + > >> + longlong10_to_str(ptr_gtid->seq_no, buf, 10); > >> + sql_print_information("Successfully truncated binlog file:%s " > >> + "to pos:%llu to remove transactions starting from " > >> + "GTID %u-%u-%s", file_name, pos, > >> + ptr_gtid->domain_id, ptr_gtid->server_id, buf); > >> + } > >> + if (!(error= init_io_cache(&cache, file, IO_SIZE, WRITE_CACHE, > >> + (my_off_t) pos, 0, MYF(MY_WME|MY_NABP)))) > >> + { > >> + /* > >> + Write Stop_log_event to ensure clean end point for the binary log being > >> + truncated. > >> + */ > > > > I'm not sure about it. The less you touch the corrupted binlog the better. > > simply truncating it would be enough, wouldn't it? > > Yes, it would. > But I somewhat preferred a tidy-up style. > > [?] > > If you feel strong to remove it will be done. I feel it'd be safer not to write anything into the corrupted binlog. > >> + Stop_log_event se; > >> + se.checksum_alg= cs_alg; > >> + if ((error= write_event(&se, NULL, &cache))) > >> + { > >> + sql_print_error("Failed to write stop event to " > >> + "binary log. Errno:%d", > >> + (cache.error == -1) ? my_errno : error); > >> + goto end; > >> + } > >> + clear_inuse_flag_when_closing(cache.file); > >> + if ((error= flush_io_cache(&cache)) || > >> + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > >> + { > >> + sql_print_error("Faild to write stop event to " > >> + "binary log. Errno:%d", > >> + (cache.error == -1) ? my_errno : error); > >> + } > >> + } > >> + else > >> + sql_print_error("Failed to initialize binary log " > >> + "cache for writing stop event. Errno:%d", > >> + (cache.error == -1) ? my_errno : error); > > > > 1. I don't see why you need to sync after every operation. > > Yep, there's one fsync before Stop event writing, redundant if Stop will > stay in the final patch. As far as I understand, if Stop is removed, you shouldn't need any syncs at all, just truncate and close. > >> + int next_binlog_or_round(int& round, > >> + const char *last_log_name, > >> + const char *binlog_checkpoint_name, > >> + LOG_INFO *linfo, MYSQL_BIN_LOG *log); > >> + bool is_safe() > > > > what do you mean "safe" ? > > Safe to truncate condition is explained elsewhere to mean there are no > non-transactional group of events within the truncated tail. > Earlier mentioned ineffective COMMIT-query ended "transactions" are safe. may be is_safe_to_truncate() then? ... > >> + if (!truncate_validated) > >> + { > >> + DBUG_ASSERT(round <= 2); > >> + /* > >> + Either truncate was not set or was reset, else > >> + it gets incremented, otherwise it may only set to an earlier > >> + offset only at the turn out of the 1st round. > > > > sorry, I cannot parse that :( > > The comments are for the assert in a branch that acts when the truncate position > (gtid) is not yet finalized. > Let me annotate under each assert's line. > > > > >> + */ > >> + DBUG_ASSERT(truncate_gtid.seq_no == 0 || > > 'truncate was not set or was reset' ^, that is truncate_gtid. > The 'was reset' is actually not present in the assert - that'd be > reflected with > truncate_reset_done == true > > > >> + last_gtid_coord >= binlog_truncate_coord || > > 'it gets incremented' ^ why does that mean "it gets incremented" ? > > >> + (round == 2 && truncate_set_in_1st)); > > And if it's neither of the above there must've been truncate candidate > set in the 1st round. ... > >> switch (typ) > >> { > >> + case FORMAT_DESCRIPTION_EVENT: > >> + if (round > 1) > >> + { > >> + if (fdle != fdle_arg) > >> + delete fdle; > >> + fdle= (Format_description_log_event*) ev; > > > > why is it needed? all binlog files should have the same > > Format_description_log_event anyway. > > Actually contrary to what was stated (the patch removed that claim) in > the parent revision of this commit, the FD:s of the > recovery binlog files may have different checksums. I.e > > set @@global.binlog_checksum = value > > can be issued to rotate binlog to a new hot file having different checksum value > while the binlog *CheckPoint* file is behind. So this is possible to > have at recovery: > > B_1(CP, checksum = NONE), B_2(hot, checksum = CRC32) > > which also manifests a normal use case bug I have not reported any bug (though wanted to). okay, checksum can differ. But how does it affect Format_description_log_event ? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 15 Mar '21

15 Mar '21

Hi, Andrei! On Mar 01, Andrei Elkin wrote: > > I've reviewed almost everything, see comments below. But not the > > Recovery_context methods. Please explain how it works and how all > > these truncate_validated, truncate_reset_done, truncate_set_in_1st, > > etc all work together. > > ...specifically to this point. Just in case I hope you did not miss to > read recovery_design.txt from the MDEV, which does not go into coding > details that you're effectively about in above. I've read it now. Still I don't understand why you need an extra round of binlog scanning (two rounds if only one file, three rounds if many). Also, this recovery_design.txt is not part of the code, so whoever will look at it later will be just as puzzled as I was. ... > [ G1, G2, ..., G_k, g_{k+1}, ... g_n ] > > here the uppercase `G' stands for committed trx, the smallcase `g' for > prepared,`_k' - sub-scripts in the recovery sequence. As the capital > letter first rules in the single-engine case the first occurrence of a > pattern `G_k,g_k+1' identifies the truncate index. The patch reflects > such fact with raising `truncate_validated' flag. > > But it's more complicated in the multiple binlog files / engines case. > The first `Gg' letter-case drop is not guaranteed to be the only drop > so `truncate_reset_done' and `truncate_set_in_1st' are introduced to > help with truncate index identification. Why is it not guaranteed? > When `truncate_validated' is set that indicates the truncate index is > determined and may not change in the current (1st of 2nd) nor future > rounds. `truncate_reset_done' says that an "inverse" `g_k,G_k+1' pair > is found so that any earlier truncation candidate gets reset (to > "zero"). If there will be later any candidate found in *this* (1st or > 2nd) round in the sequence its index will be obviously greater. > > `truncate_set_in_1st' function is to remember that the truncate > candidate was found in the 1st round (in the "hot" binlog file), but > if the candidate has not been validated `!truncate_validated' it may > be exacted in the 2nd round and then to an earlier transaction. So the > flag helps to handle exception from truncate candidate monotony rule: > e.g the hot binlog B2 contains `[g5,g6,...g_n]' and a ref to binlog > checkpoint file B1 that contains `[G1,g2,G3,g4]'. The first round > truncate candidate of g5 would be first exacted to `g2' before finally > ascertained to `g4' in the 2nd round. (Notice `g2 -> g4' preserves the > truncate index monotony). > > Notice that due to exacting like `g2 -> g4' in the 2nd round of the > above example `g2' got "to be up-cased" into `G2' for committing > (feasible with two trx:s on two different engine scenario - g2 with > Innodb only got prepared, G4 - on Rocksdb, and got committed prior the > crash). That's what the 3rd round is for. > > I hope this will be helpful. Unfortunately, not very much. It describes how you juggle with variables and scanning rounds. But not *what* you're trying to find. And it's kind of difficult to reverse engineer your "how" back into "what" :( Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-17399: JSON_TABLE: Incorrect code with table elimination
by Sergey Petrunia 15 Mar '21

15 Mar '21

Hi Alexey, > diff --git a/sql/opt_table_elimination.cc b/sql/opt_table_elimination.cc > index 3958797ec44..f2497d524ec 100644 > --- a/sql/opt_table_elimination.cc > +++ b/sql/opt_table_elimination.cc > @@ -637,6 +637,16 @@ void eliminate_tables(JOIN *join) > List_iterator<Item> it(join->fields_list); > while ((item= it++)) > used_tables |= item->used_tables(); > + > + { > + List_iterator<TABLE_LIST> it(*join->join_list); > + TABLE_LIST *tbl; > + while ((tbl= it++)) > + { > + if (tbl->table_function) > + used_tables|= tbl->table_function->used_tables(); > + } > + } > This only walks the tables that are "at the top level" of the join. if a JSON_TABLE(...) is inside a nested outer join, it will not be found. Please walk select_lex->leaf_tables instead. Please add the testcase (I provide one below). Please also add a comment, clarifying what is being done, something like: Table function JSON_TABLE() can have references to other tables. Do not eliminate the tables that JSON_TABLE() refers to. Note: the JSON_TABLE itself cannot be eliminated as it doesn't have unique keys. == Testcase == create table t20 (a int not null); create table t21 (a int not null primary key, js varchar(100)); insert into t20 select seq from seq_1_to_100; insert into t21 select a, '{"a":100}' from t20; create table t31(a int); create table t32(b int); insert into t31 values (1); insert into t32 values (1); explain select t20.a, jt1.ab from t20 left join t21 on t20.a=t21.a join (t31 left join (t32 join JSON_TABLE(t21.js,'$' COLUMNS (ab INT PATH '$.a')) AS jt1) on t31.a<3); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 3

Re: [Maria-developers] f1db957c5da: MDEV-21469: Implement crash-safe binary logging of the user XA
by Sergei Golubchik 14 Mar '21

14 Mar '21

Hi, Sujatha! Note, this is a review of a combined diff dc92235f21e + f1db957c5da On Mar 14, Sujatha wrote: > revision-id: f1db957c5da (mariadb-10.5.2-247-gf1db957c5da) > parent(s): dc92235f21e > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2020-12-21 16:10:46 +0200 > message: > > MDEV-21469: Implement crash-safe binary logging of the user XA > > Make XA PREPARE, XA COMMIT and XA ROLLBACK statements > crash-safe when --log-bin is specified. > > At execution of XA PREPARE, XA COMMIT and XA ROLLBACK their replication > events are made written into the binary log prior to execution > of the commands in storage engines. > In case the server crashes, after writing to binary log but before all > involved engines have processed the command, the following recovery > will execute the command's replication events to equalize the states > of involved engines with that of binlog. > That applies to all XA PREPARE *group* and XA COMMIT or ROLLBACK. > > On the implementation level the recovery time binary log parsing > is augmented to pay attention to > the user XA xids to identify the XA transactions' state:s in binary log, and > eventually match them against their states in engines, see > MYSQL_BIN_LOG::recover_explicit_xa_prepare(). > In discrepancy cases the outdated state in the engines is corrected with > resubmitting the transaction prepare group of events, or completion > ones. The multi-engine partly prepared XA PREPARE case > the XA is rolled back first. > The fact of multiple-engine involved is registered into > Gtid_log_event::flags2 as one bit. The boolean value is > sufficient and precise to deal with two engines in XA transaction. > With more than 2 recoverable engines the flag method is still correct though > may be pessimistic, as it treats all recoverable engines as XA > participants. So when the number of such Engines exceeds the number > of prepared engines of the XA that XA is treated > as partially completed, with all that ensued. > As an optimization no new bit is allocated in flags2, instead a > pre-existing ones (of MDEV-742) are reused, observing that > A. XA "COMPLETE" does not require multi-engine hint for its recovery and that > B. the MDEV-742 XA-completion bit is not anyhow used by > XA-PREPARE and its GTID log event. > > Notice the multi-engine flagging is superceded by MDEV-21117 extension > in Gtid log event so this part should be taken from there. > diff --git a/mysql-test/suite/binlog/t/binlog_xa_multi_binlog_crash_recovery.test b/mysql-test/suite/binlog/t/binlog_xa_multi_binlog_crash_recovery.test > new file mode 100644 > index 00000000000..d0852de2fcc > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_xa_multi_binlog_crash_recovery.test > @@ -0,0 +1,86 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA crash recovery works fine across multiple binary logs. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate an explicit XA transaction. Using debug simulation hold the "debug sync" != "debug sim" :) > +# execution of XA PREPARE statement after the XA PREPARE is written to > +# the binary log. With this the prepare will not be done in engine. > +# 1 - By executing FLUSH LOGS generate multiple binary logs. > +# 2 - Now make the server to disappear at this point. > +# 3 - Restart the server. During recovery the XA PREPARE from the binary > +# log will be read. It is cross checked with engine. Since it is not > +# present in engine it will be executed once again. > +# 4 - When server is up execute XA RECOVER to check that the XA is > +# prepared in engine as well. > +# 5 - XA COMMIT the transaction and check the validity of the data. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > +# > + > +--source include/have_innodb.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_log_bin.inc > + > +RESET MASTER; > + > +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > + > +connect(con1,localhost,root,,); > +XA START 'xa1'; > +INSERT INTO t1 SET a=1; > +SET DEBUG_SYNC= "simulate_hang_after_binlog_prepare SIGNAL con1_ready WAIT_FOR con1_go"; > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; > +XA END 'xa1'; > +--send XA PREPARE 'xa1'; > + > +connection default; > +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > +FLUSH LOGS; > +FLUSH LOGS; > +FLUSH LOGS; > + > +--source include/show_binary_logs.inc > +--let $binlog_file= master-bin.000004 > +--let $binlog_start= 4 > +--source include/show_binlog_events.inc > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait I don't see why you'd need to wait first, it'd be simpler to restart right away. > +EOF > + > +--error 0,2013 > +SET DEBUG_SYNC= "now SIGNAL con1_go"; > +--source include/wait_until_disconnected.inc > + > +--connection con1 > +--error 2013 > +--reap > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +XA RECOVER; > +XA COMMIT 'xa1'; > + > +SELECT * FROM t1; > + > +# Clean up. > +connection default; > +DROP TABLE t1; > + > diff --git a/mysql-test/suite/rpl/t/rpl_xa_commit_crash_safe.test b/mysql-test/suite/rpl/t/rpl_xa_commit_crash_safe.test > new file mode 100644 > index 00000000000..e972e3f09de > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_commit_crash_safe.test > @@ -0,0 +1,98 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA COMMIT statements are crash safe. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. > +# 'xa1' will be prepared and committed. > +# 1 - For 'xa2' let the XA COMMIT be done in binary log and crash the > +# server so that it is not committed in engine. > +# 2 - Restart the server. The recovery code should successfully recover > +# 'xa2'. The COMMIT should be executed during recovery. > +# 3 - Check the data in table. Both rows should be present in table. > +# 4 - Trying to commit 'xa2' should report unknown 'XA' error as COMMIT is > +# already complete during recovery. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > + > +--source include/have_innodb.inc > +--source include/master-slave.inc master-slave must be always included last > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); why do you need master2 connection? > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +XA COMMIT 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +XA START 'xa2'; > +INSERT INTO t VALUES (40); > +XA END 'xa2'; > +XA PREPARE 'xa2'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait > +EOF Same. Why do you wait? > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_commit_or_rollback"; > +--error 2013 # CR_SERVER_LOST > +XA COMMIT 'xa2'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > +XA RECOVER; > +--error 1397 # ER_XAER_NOTA > +XA COMMIT 'xa2'; > +SELECT * FROM t; > +--sync_slave_with_master > + > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_xa_event_apply_failure.test b/mysql-test/suite/rpl/t/rpl_xa_event_apply_failure.test > new file mode 100644 > index 00000000000..14cebbd9b13 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_event_apply_failure.test > @@ -0,0 +1,119 @@ > +# ==== Purpose ==== > +# > +# Test verifies that if for some reason an event cannot be applied during > +# recovery, appropriate error is reported. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. > +# 'xa1' will be prepared and committed. > +# 1 - For 'xa2' let the XA COMMIT be done in binary log and crash the > +# server so that it is not committed in engine. > +# 2 - Restart the server. Using debug simulation point make XA COMMIT 'xa2' > +# execution to fail. The server will resume anyway > +# to leave the error in the errlog (see "Recovery: Error.."). > +# 3 - Work around the simulated failure with Commit once again > +# from a connection that turns OFF binlogging. > +# Slave must catch up with the master. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > + > +--source include/have_innodb.inc > +--source include/master-slave.inc master-slave must be always included last > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); why do you need master2 connection? > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > +CALL mtr.add_suppression("Failed to execute binlog query event"); > +CALL mtr.add_suppression("Recovery: Error .Out of memory.."); > +CALL mtr.add_suppression("Crash recovery failed."); > +CALL mtr.add_suppression("Can.t init tc log"); > +CALL mtr.add_suppression("Aborting"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +XA COMMIT 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +XA START 'xa2'; > +INSERT INTO t VALUES (40); > +XA END 'xa2'; > +XA PREPARE 'xa2'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait ditto > +EOF > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_commit_or_rollback"; > +--error 2013 # CR_SERVER_LOST > +XA COMMIT 'xa2'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart: --debug-dbug=d,trans_xa_commit_fail > +EOF > + > +connection default; > +--source include/wait_until_disconnected.inc > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--echo *** must be no 'xa2' commit seen, as it's still prepared: > +SELECT * FROM t; > +XA RECOVER; > + > +# Commit it manually now to work around the extra binlog record > +# by turning binlogging OFF by the connection. > + > +SET GLOBAL DEBUG_DBUG=""; > +SET SQL_LOG_BIN=0; > +--error 0 why error 0? What will happen if you don't disable binlog? > +XA COMMIT 'xa2'; > +SET SQL_LOG_BIN=1; > + > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--source include/wait_until_connected_again.inc > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > + > +--sync_slave_with_master > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_xa_prepare_commit_prepare.test b/mysql-test/suite/rpl/t/rpl_xa_prepare_commit_prepare.test > new file mode 100644 > index 00000000000..7b987c7f29b > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_prepare_commit_prepare.test > @@ -0,0 +1,95 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA PREPARE transactions are crash safe. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. > +# 'xa1' will be prepared and committed. > +# 1 - For 'xa2' let the XA PREPARE be done in binary log and crash the > +# server so that it is not prepared in engine. > +# 2 - Restart the server. The recovery code should successfully recover > +# 'xa2'. > +# 3 - When server is up, execute XA RECOVER and verify that 'xa2' is > +# present. > +# 4 - Commit the XA transaction and verify its correctness. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > +--source include/have_innodb.inc > +--source include/master-slave.inc ditto > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); ditto > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +XA COMMIT 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +XA START 'xa2'; > +INSERT INTO t VALUES (40); > +XA END 'xa2'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait ditto > +EOF > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; > +--error 2013 # CR_SERVER_LOST > +XA PREPARE 'xa2'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > +XA RECOVER; > +XA COMMIT 'xa2'; > +SELECT * FROM t; > +--sync_slave_with_master > + > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_xa_prepare_crash_safe.test b/mysql-test/suite/rpl/t/rpl_xa_prepare_crash_safe.test > new file mode 100644 > index 00000000000..9d2c5cce528 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_prepare_crash_safe.test > @@ -0,0 +1,117 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA PREPARE transactions are crash safe. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 3 explicit XA transactions. 'xa1', 'xa2' and 'xa3'. > +# Using debug simulation hold the execution of second XA PREPARE > +# statement after the XA PREPARE is written to the binary log. > +# With this the prepare will not be done in engine. > +# 1 - For 'xa3' allow the PREPARE statement to be written to binary log and > +# simulate server crash. > +# 2 - Restart the server. The recovery code should successfully recover > +# 'xa2' and 'xa3'. > +# 3 - When server is up, execute XA RECOVER and verify that 'xa2' and 'xa3' > +# are present along with 'xa1'. > +# 4 - Commit all the XA transactions and verify their correctness. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > + > +--source include/have_innodb.inc > +--source include/master-slave.inc ditto > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); ok, in this test you actually use master2 :) > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > +CALL mtr.add_suppression("Found 2 prepared XA transactions"); > +CALL mtr.add_suppression("Found 3 prepared XA transactions"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +use test; > +xa start 'xa2'; > +insert into t values (30); > +xa end 'xa2'; > +SET DEBUG_SYNC="simulate_hang_after_binlog_prepare SIGNAL reached WAIT_FOR go"; > +send xa prepare 'xa2'; > + > +--connection master2 > +let $wait_condition= > + SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.PROCESSLIST > + WHERE STATE like "debug sync point: simulate_hang_after_binlog_prepare%"; > +--source include/wait_condition.inc eh? you can just 'WAIT_FOR signal', couldn't you? > + > +XA START 'xa3'; > +INSERT INTO t VALUES (40); > +XA END 'xa3'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait ditto > +EOF > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; > +--error 2013 # CR_SERVER_LOST > +XA PREPARE 'xa3'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--error 2013 > +--reap > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > +XA RECOVER; > +XA COMMIT 'xa1'; > +XA COMMIT 'xa2'; > +XA COMMIT 'xa3'; > +SELECT * FROM t; > +--sync_slave_with_master > + > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_xa_rollback_commit_crash_safe.test b/mysql-test/suite/rpl/t/rpl_xa_rollback_commit_crash_safe.test > new file mode 100644 > index 00000000000..1d19f96116e > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_xa_rollback_commit_crash_safe.test > @@ -0,0 +1,97 @@ > +# ==== Purpose ==== > +# > +# Test verifies that XA COMMIT statements are crash safe. XA ROLLBACK > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. > +# 'xa1' will be prepared and committed. > +# 1 - For 'xa2' let the XA ROLLBACK be done in binary log and crash the > +# server so that it is not committed in engine. not rolled back > +# 2 - Restart the server. The recovery code should successfully recover > +# 'xa2'. The ROLLBACK should be executed during recovery. > +# 3 - Check the data in table. Only one row should be present in table. > +# 4 - Trying to rollback 'xa2' should report unknown 'XA' error as rollback > +# is already complete during recovery. > +# > +# ==== References ==== > +# > +# MDEV-21469: Implement crash-safe logging of the user XA > + > +--source include/have_innodb.inc > +--source include/master-slave.inc ditto > +--source include/have_debug.inc > + > +connect (master2,localhost,root,,); unused again > +--connection master > +CALL mtr.add_suppression("Found 1 prepared XA transactions"); > + > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +XA START 'xa1'; > +INSERT INTO t VALUES (20); > +XA END 'xa1'; > +XA PREPARE 'xa1'; > +XA COMMIT 'xa1'; > +--sync_slave_with_master > +--source include/stop_slave.inc > + > +--connection master1 > +XA START 'xa2'; > +INSERT INTO t VALUES (40); > +XA END 'xa2'; > +XA PREPARE 'xa2'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait ditto > +EOF > + > +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_commit_or_rollback"; > +--error 2013 # CR_SERVER_LOST > +XA ROLLBACK 'xa2'; > +--source include/wait_until_disconnected.inc > + > +--connection master1 > +--source include/wait_until_disconnected.inc > + > +--connection master > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# rpl_end.inc needs to use the connection server_1 > +connection server_1; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection master > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--connection slave > +--source include/start_slave.inc > +--sync_with_master > + > +--connection master > +SELECT * FROM t; > +XA RECOVER; > +--error 1397 # ER_XAER_NOTA > +XA ROLLBACK 'xa2'; > +SELECT * FROM t; > +--sync_slave_with_master > + > +SELECT * FROM t; > + > +--connection master > +DROP TABLE t; > +--sync_slave_with_master > +--source include/rpl_end.inc > diff --git a/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.test b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.test > new file mode 100644 > index 00000000000..c7cefbda4bf > --- /dev/null > +++ b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.test > @@ -0,0 +1,46 @@ > +# MDEV-742, MDEV-21469 XA replication, and xa crash-safe. > +# Tests prove xa state is recovered to a prepared or completed state > +# upon post-crash recovery, incl a multi-engine case. > + > +--source include/have_rocksdb.inc > +--source include/have_innodb.inc > +--source include/have_debug.inc > +--source include/master-slave.inc ditto > +--source include/have_binlog_format_row.inc > + > +--connection slave > +--source include/stop_slave.inc > + > +--connection master > +CALL mtr.add_suppression("Found . prepared XA transactions"); > +CALL mtr.add_suppression("Failed to execute binlog query event"); > +CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=rocksdb; > +CREATE TABLE t2 (a INT PRIMARY KEY) ENGINE=innodb; > +INSERT INTO t1 SET a = 1; > +INSERT INTO t2 SET a = 1; > + > +--let $xa=xa1 > +--let $when=after_binlog > +--source rpl_rocksdb_xa_recover.inc > + > +--let $xa=xa2 > +--let $when=after_first_engine > +--let $finally_expected=$xa in the list > +--source rpl_rocksdb_xa_recover.inc > + > +--connection slave > +--source include/start_slave.inc > + > +--connection master > +--sync_slave_with_master > +let $diff_tables= master:t1, slave:t1; > +source include/diff_tables.inc; > +let $diff_tables= master:t2, slave:t2; > +source include/diff_tables.inc; > + > +--connection master > +SET SESSION DEBUG_DBUG=""; > +drop table t1,t2; > +--sync_slave_with_master > + > +--source include/rpl_end.inc > diff --git a/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.inc b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.inc > new file mode 100644 > index 00000000000..312b8b6a19e > --- /dev/null > +++ b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_rocksdb_xa_recover.inc > @@ -0,0 +1,67 @@ > +# callee of rpl_rocksdb_xa_recover.test > +# requires t1,t2 as defined in the caller. > + > +--let $at=_prepare > +--let $finally_expected=$xa in the list > +--let $init_val=`SELECT MAX(a) from t1 as t_1` why as t_1 ? > +--eval XA START '$xa' > +--eval INSERT INTO t1 SET a=$init_val + 1 > +--eval INSERT INTO t2 SET a=$init_val + 1 > +--eval XA END '$xa' > +--eval SET SESSION DEBUG_DBUG="d,simulate_crash_$when$at" > + > +--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +--error 2013 # CR_SERVER_LOST > +--eval XA PREPARE '$xa' > +--source include/wait_until_disconnected.inc > +--let $rpl_server_number = 1 > +--source include/rpl_reconnect.inc > +--echo "*** $finally_expected" > +XA RECOVER; > +--eval XA ROLLBACK '$xa' > +--eval SELECT MAX(a) - $init_val as zero FROM t1 > +--eval SELECT MAX(a) - $init_val as zero FROM t2 > + > + > +--let $at=_commit_or_rollback > +--let $finally_expected=empty list (rolled back) > +--eval XA START '$xa' > +--eval INSERT INTO t1 SET a=$init_val + 2 > +--eval INSERT INTO t2 SET a=$init_val + 2 > +--eval XA END '$xa' > +--eval XA PREPARE '$xa' > + > +--eval SET SESSION DEBUG_DBUG="d,simulate_crash_$when$at" > +--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +--error 2013 # CR_SERVER_LOST > +--eval XA ROLLBACK '$xa' > +--source include/wait_until_disconnected.inc > +--let $rpl_server_number = 1 > +--source include/rpl_reconnect.inc > + > +--echo "*** $finally_expected" > +XA RECOVER; > +--eval SELECT MAX(a) - $init_val as zero FROM t1 > +--eval SELECT MAX(a) - $init_val as zero FROM t2 > + > + > +--let $at=_commit_or_rollback > +--let $finally_expected=empty list (committed away) > +--eval XA START '$xa' > +--eval INSERT INTO t1 SET a=$init_val + 3 > +--eval INSERT INTO t2 SET a=$init_val + 3 > +--eval XA END '$xa' > +--eval XA PREPARE '$xa' > + > +--eval SET SESSION DEBUG_DBUG="d,simulate_crash_$when$at" > +--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +--error 2013 # CR_SERVER_LOST > +--eval XA COMMIT '$xa' > +--source include/wait_until_disconnected.inc > +--let $rpl_server_number = 1 > +--source include/rpl_reconnect.inc > + > +--echo "*** $finally_expected" > +XA RECOVER; > +--eval SELECT MAX(a) - $init_val as three FROM t1 > +--eval SELECT MAX(a) - $init_val as three FROM t2 > diff --git a/sql/log_event.h b/sql/log_event.h > index 639cbfbe7aa..e1880339e85 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -3614,6 +3616,12 @@ class Gtid_log_event: public Log_event > static const uchar FL_PREPARED_XA= 64; > /* FL_"COMMITTED or ROLLED-BACK"_XA is set for XA transaction. */ > static const uchar FL_COMPLETED_XA= 128; > + /* > + To mark the fact of multiple transactional engine participants > + in the prepared XA. The FL_COMPLETED_XA bit is reused by XA_PREPARE_LOG_EVENT, What do you mean by that? XA_prepare_log_event does not inherit from Gtid_log_event, it cannot use Gtid_log_event flags. And, please, explain in a comment why you can reuse a bit for two different flags. Like "the ambiguity between FL_COMPLETED_XA and FL_MULTI_ENGINE_XA is resolved by the flag FL_PREPARED_XA. if it's set, then 128 means FL_MULTI_ENGINE_XA, if it's not set, then 128 means FL_COMPLETED_XA" > + oth the XA completion events do not need such marking. other > + */ > + static const uchar FL_MULTI_ENGINE_XA= 128; > > #ifdef MYSQL_SERVER > Gtid_log_event(THD *thd_arg, uint64 seq_no, uint32 domain_id, bool standalone, > diff --git a/sql/handler.h b/sql/handler.h > index 0eff7bd930d..9acdd47bfc2 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1880,8 +1891,18 @@ class Ha_trx_info > m_ht= ht_arg; > m_flags= (int) TRX_READ_ONLY; /* Assume read-only at start. */ > > - m_next= trans->ha_list; > - trans->ha_list= this; > + if (likely(!past_head)) > + { > + m_next= trans->ha_list; > + trans->ha_list= this; > + } > + else > + { > + DBUG_ASSERT(trans->ha_list); > + > + m_next= trans->ha_list->m_next; > + trans->ha_list->m_next= this; > + } I don't like this fake generalization. There's no point for an arbitrary engine to be put at a specific place in the list. Only binlog is special. At least, remove this new argument and check for binlog_hton here. But really, think of this - binlog is so special, may be it'd be easier not to pretend it's a storage engine at all? No need for a fake binlog_hton, no need to register it, etc. It kind of made sense in 2005, but it doesn't necessarily make it now. > } > > /** Clear, prepare for reuse. */ > diff --git a/sql/handler.cc b/sql/handler.cc > index ffb89b30d92..b981294f0d6 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1338,6 +1345,9 @@ int ha_prepare(THD *thd) > handlerton *ht= ha_info->ht(); > if (ht->prepare) > { > + DBUG_EXECUTE_IF("simulate_crash_after_first_engine_prepare", > + if (!ha_info->next()) DBUG_SUICIDE();); looks like not simulate_crash_after_first_engine_prepare, but more like simulate_crash_before_last_engine_prepare. it's the same point in time if you have only two engines, but the code still implements "before last", not "after first" > + > if (unlikely(prepare_or_error(ht, thd, all))) > { > ha_rollback_trans(thd, all); > @@ -1368,22 +1378,22 @@ int ha_prepare(THD *thd) > } > > /* > - Like ha_check_and_coalesce_trx_read_only to return counted number of > - read-write transaction participants limited to two, but works in the 'all' > - context. > - Also returns the last found rw ha_info through the 2nd argument. > + Returns counted number of > + read-write recoverable transaction participants optionally limited to two. > + Also optionally returns the last found rw ha_info through the 2nd argument. > */ > -uint ha_count_rw_all(THD *thd, Ha_trx_info **ptr_ha_info) > +uint ha_count_rw_all(THD *thd, Ha_trx_info **ptr_ha_info, bool count_through) I don't understand why you need a new argument. ptr_ha_info==NULL means don't return a value there, this is a standard convention used everywhere. also, it's not an "count rw all" it's are there more rw-all htons than N where N can be 1 or 2. So: * either take N as an argument and return as soon as rw_ha_count > N * or don't overoptimize here and just return full count every time > { > unsigned rw_ha_count= 0; > > for (auto ha_info= thd->transaction.all.ha_list; ha_info; > ha_info= ha_info->next()) > { > - if (ha_info->is_trx_read_write()) > + if (ha_info->is_trx_read_write() && ha_info->ht()->recover) > { > - *ptr_ha_info= ha_info; > - if (++rw_ha_count > 1) > + if (ptr_ha_info) > + *ptr_ha_info= ha_info; > + if (++rw_ha_count > 1 && !count_through) > break; > } > } > @@ -1876,6 +1886,10 @@ commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, bool is_real_trans) > { > int err; > handlerton *ht= ha_info->ht(); > + > + DBUG_EXECUTE_IF("simulate_crash_after_first_engine_commit_or_rollback", > + if (!ha_info->next()) DBUG_SUICIDE();); 1. same as above 2. why not to use *different* names for these two crashes? in case, you know, you'll want to crash at a specific point in the code, not just somewhere? And in a case you'll actually want to crash just somewhere you can always use "+d,simulate_crash_after_first_engine_commit,simulate_crash_after_first_engine_rollback" (and it should be "before_last", of course, not "after_first") > + > if ((err= ht->commit(ht, thd, all))) > { > my_error(ER_ERROR_DURING_COMMIT, MYF(0), err); > @@ -1987,6 +2001,10 @@ int ha_rollback_trans(THD *thd, bool all) > { > int err; > handlerton *ht= ha_info->ht(); > + > + DBUG_EXECUTE_IF("simulate_crash_after_first_engine_commit_or_rollback", > + if (!ha_info->next()) DBUG_SUICIDE();); ditto > + > if ((err= ht->rollback(ht, thd, all))) > { // cannot happen > my_error(ER_ERROR_DURING_ROLLBACK, MYF(0), err); > @@ -2098,8 +2116,24 @@ int ha_commit_or_rollback_by_xid(XID *xid, bool commit) > xaop.xid= xid; > xaop.result= 1; > > + if (binlog_hton->recover) > + { > + /* > + When the binlogging service is enabled complete the transaction > + by it first. > + */ > + if (commit) > + binlog_hton->commit_by_xid(binlog_hton, xid); > + else > + binlog_hton->rollback_by_xid(binlog_hton, xid); > + } > plugin_foreach(NULL, commit ? xacommit_handlerton : xarollback_handlerton, > MYSQL_STORAGE_ENGINE_PLUGIN, &xaop); > + if (binlog_hton->recover) > + { > + THD *thd= current_thd; > + thd->reset_binlog_completed_by_xid(); > + } 1. this, precisely, shows that binlog does not need a hton anymoreo 2. how can binlog_hton->recover be NULL here? > > return xaop.result; > } > @@ -2222,6 +2258,7 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > > if (hton->recover) > { > + info->recover_htons++; don't forget to subtract 1 for binlog_hton->recover, otherwise it'll make you to rollback everything with a FL_MULTI_ENGINE_XA flag > while ((got= hton->recover(hton, info->list, info->len)) > 0 ) > { > sql_print_information("Found %d prepared transaction(s) in %s", > @@ -2263,7 +2300,21 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > _db_doprnt_("ignore xid %s", xid_to_str(buf, info->list[i])); > }); > xid_cache_insert(info->list + i); > + XID *foreign_xid= info->list + i; swap those two lines ^^^ then you can write xid_cache_insert(foreign_xid); > info->found_foreign_xids++; > + > + /* > + For each foreign xid prepraed in engine, check if it is present in > + xa_prepared_list of binlog. > + */ > + if (info->xa_prepared_list) for simplicity I'd rather always initialize xa_prepared_list and remove this if() > + { > + struct xa_recovery_member *member= NULL; > + if ((member= (xa_recovery_member *) > + my_hash_search(info->xa_prepared_list, foreign_xid->key(), > + foreign_xid->key_length()))) > + member->in_engine_prepare++; > + } > continue; > } > if (IF_WSREP(!(wsrep_emulate_bin_log && > @@ -2362,7 +2424,7 @@ int ha_recover(HASH *commit_list) > info.found_my_xids, opt_tc_log_file); > DBUG_RETURN(1); > } > - if (info.commit_list) > + if (info.commit_list && !info.found_foreign_xids) Why? after the finished crash recovery one can still have transactions in doubt. > sql_print_information("Crash recovery finished."); > DBUG_RETURN(0); > } > diff --git a/sql/log.cc b/sql/log.cc > index 731bb3e98f0..c69b8518cf4 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2019,28 +2027,49 @@ static int binlog_xa_recover_dummy(handlerton *hton __attribute__((unused)), > return 0; > } > > - > +/* > + The function invokes binlog_commit() and returns its result > + when it has not yet called it already. > + binlog_cache_mngr::completed_by_xid remembers the fact of > + the 1st of maximum two subsequent calls. > +*/ > static int binlog_commit_by_xid(handlerton *hton, XID *xid) > { > + int rc= 0; > THD *thd= current_thd; > - > - (void) thd->binlog_setup_trx_data(); > + binlog_cache_mngr *cache_mngr= thd->binlog_setup_trx_data(); > > DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_COMMIT); > > - return binlog_commit(hton, thd, TRUE); > -} > + if (!cache_mngr->completed_by_xid) On what code path can you have completed_by_xid == false here? > + { > + rc= binlog_commit(hton, thd, TRUE); > + cache_mngr->completed_by_xid= true; > + } > > + return rc; > +} > > +/* > + The function invokes binlog_rollback() and returns its results similarly > + to a commit branch. > +*/ > static int binlog_rollback_by_xid(handlerton *hton, XID *xid) > { > + int rc= 0; > THD *thd= current_thd; > - > - (void) thd->binlog_setup_trx_data(); > + binlog_cache_mngr *cache_mngr= thd->binlog_setup_trx_data(); > > DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_ROLLBACK || > (thd->transaction.xid_state.get_state_code() == XA_ROLLBACK_ONLY)); > - return binlog_rollback(hton, thd, TRUE); > + > + if (!cache_mngr->completed_by_xid) > + { > + rc= binlog_rollback(hton, thd, TRUE); > + cache_mngr->completed_by_xid= true; > + } > + > + return rc; > } > > > @@ -2195,6 +2224,12 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all) > if (!all) > cache_mngr->trx_cache.set_prev_position(MY_OFF_T_UNDEF); > > + DBUG_EXECUTE_IF("simulate_crash_after_binlog_commit_or_rollback", > + DBUG_SUICIDE();); > + DEBUG_SYNC(thd, "simulate_hang_after_binlog_prepare"); > + DBUG_EXECUTE_IF("simulate_crash_after_binlog_prepare", > + DBUG_SUICIDE();); wouldn't you say there's one DBUG_EXECUTE_IF too many? Just keep one, "simulate_crash_after_binlog_commit" > + > THD_STAGE_INFO(thd, org_stage); > DBUG_RETURN(error); > } > @@ -2298,6 +2333,9 @@ static int binlog_rollback(handlerton *hton, THD *thd, bool all) > cache_mngr->trx_cache.set_prev_position(MY_OFF_T_UNDEF); > thd->reset_binlog_for_next_statement(); > > + DBUG_EXECUTE_IF("simulate_crash_after_binlog_commit_or_rollback", same as above, use unique dbug keywords, like "simulate_crash_after_binlog_rollback" (because it's rollback here) > + DBUG_SUICIDE();); > + > DBUG_RETURN(error); > } > > @@ -5749,6 +5794,14 @@ binlog_cache_mngr *THD::binlog_setup_trx_data() > DBUG_RETURN(cache_mngr); > } > > +/* > + XA completion flag resetter for ha_commit_or_rollback_by_xid(). > +*/ > +void THD::reset_binlog_completed_by_xid() > +{ > + binlog_setup_trx_data()->reset_completed_by_xid(); > +} I don't think this makes any sense as a THD method > + > /* > Function to start a statement and optionally a transaction for the > binary log. > @@ -10333,14 +10386,108 @@ start_binlog_background_thread() > return 0; > } > > +#ifdef HAVE_REPLICATION > +/** > + Auxiliary function for TC_LOG::recover(). > + @returns a successfully created and inserted @c xa_recovery_member > + into hash @c hash_arg, > + or NULL. > +*/ > +static xa_recovery_member* > +xa_member_insert(HASH *hash_arg, xid_t *xid_arg, xa_binlog_state state_arg, > + MEM_ROOT *ptr_mem_root) > +{ > + xa_recovery_member *member= (xa_recovery_member*) > + alloc_root(ptr_mem_root, sizeof(xa_recovery_member)); > + if (!member) > + return NULL; > + > + member->xid.set(xid_arg); > + member->state= state_arg; > + member->in_engine_prepare= 0; > + return my_hash_insert(hash_arg, (uchar*) member) ? NULL : member; > +} > + > +/* Inserts or updates an existing hash member with a proper state */ > +static bool xa_member_replace(HASH *hash_arg, xid_t *xid_arg, bool is_prepare, > + MEM_ROOT *ptr_mem_root) > +{ > + if(is_prepare) > + { > + if (!(xa_member_insert(hash_arg, xid_arg, XA_PREPARE, ptr_mem_root))) > + return true; > + } > + else > + { > + /* > + Search if XID is already present in recovery_list. If found > + and the state is 'XA_PREPRAED' mark it as XA_COMPLETE. > + Effectively, there won't be XA-prepare event group replay. > + */ > + xa_recovery_member* member; > + if ((member= (xa_recovery_member *) > + my_hash_search(hash_arg, xid_arg->key(), xid_arg->key_length()))) > + { > + if (member->state == XA_PREPARE) > + member->state= XA_COMPLETE; > + } > + else // We found only XA COMMIT during recovery insert to list > + { > + if (!(member= xa_member_insert(hash_arg, > + xid_arg, XA_COMPLETE, ptr_mem_root))) > + return true; > + } > + } > + return false; > +} > +#endif > > +extern "C" uchar *xid_get_var_key(xid_t *entry, size_t *length, define the first argument as const char*, then you won't need to cast in my_hash_init, and you can be sure you pass the correct function with the correct number of arguments, even if my_hash_get_key changes in the future. and cast entry to xid_t* below explicitly. > + my_bool not_used __attribute__((unused))) > +{ > + *length= entry->key_length(); > + return (uchar*) entry->key(); > +} > + > +/** > + Performs recovery based on transaction coordinator log for 2pc. At the > + time of crash, if the binary log was in active state, then recovery for > + "implicit" 'xid's and explicit 'XA' transactions is initiated, > + otherwise merely the gtid binlog state is updated. > + For 'xid' and 'XA' based recovery the following steps are performed. > + > + Identify the active binlog checkpoint file. > + Scan the binary log from the beginning. > + From GTID_LIST and GTID_EVENTs reconstruct the gtid binlog state. > + Prepare a list of 'xid's for recovery. > + Prepare a list of explicit 'XA' transactions for recovery. > + Recover the 'xid' transactions. > + The explicit 'XA' transaction recovery is initiated once all the server > + components are initialized. Please check 'execute_xa_for_recovery()'. why explicit XA recovery is delayed? > + > + Called from @c MYSQL_BIN_LOG::do_binlog_recovery() > + > + @param linfo Store here the found log file name and position to > + the NEXT log file name in the index file. > + > + @param last_log_name Name of the last active binary log at the time of > + crash. > + > + @param first_log Pointer to IO_CACHE of active binary log > + @param fdle Format_description_log_event of active binary log > + @param do_xa Is 2pc recovery needed for 'xid's and explicit XA > + transactions. > + @return indicates success or failure of recovery. > + @retval 0 success > + @retval 1 failure > + > +*/ > int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > IO_CACHE *first_log, > Format_description_log_event *fdle, bool do_xa) > { > Log_event *ev= NULL; > HASH xids; > - MEM_ROOT mem_root; > char binlog_checkpoint_name[FN_REFLEN]; > bool binlog_checkpoint_found; > bool first_round; > @@ -10533,10 +10693,22 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > > if (do_xa) > { > - if (ha_recover(&xids)) > + if (ha_recover(&xids, &xa_recover_list, &xa_recover_htons)) > goto err2; > > - free_root(&mem_root, MYF(0)); > + DBUG_ASSERT(!xa_recover_list.records || > + (binlog_checkpoint_found && binlog_checkpoint_name[0] != 0)); why is that? > + > + if (!xa_recover_list.records) > + { > + free_root(&mem_root, MYF(0)); > + my_hash_free(&xa_recover_list); > + } > + else > + { > + xa_binlog_checkpoint_name= strmake_root(&mem_root, binlog_checkpoint_name, > + strlen(binlog_checkpoint_name)); > + } > my_hash_free(&xids); > } > return 0; > @@ -10561,6 +10734,219 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > return 1; > } > > +void MYSQL_BIN_LOG::execute_xa_for_recovery() > +{ > + if (xa_recover_list.records) > + (void) recover_explicit_xa_prepare(); > + free_root(&mem_root, MYF(0)); > + my_hash_free(&xa_recover_list); > +}; > + > +/** > + Performs recovery of user XA transactions. > + 'xa_recover_list' contains the list of XA transactions to be recovered. > + with possible replaying replication event from the binary log. > + > + @return indicates success or failure of recovery. > + @retval false success > + @retval true failure > + > +*/ > +bool MYSQL_BIN_LOG::recover_explicit_xa_prepare() why is it bool, if you only use it in execute_xa_for_recovery() and ignore the return value? > +{ > +#ifndef HAVE_REPLICATION > + /* Can't be supported without replication applier built in. */ > + return false; > +#else > + bool err= true; > + int error=0; > + Relay_log_info *rli= NULL; > + rpl_group_info *rgi; > + THD *thd= new THD(0); /* Needed by start_slave_threads */ > + thd->thread_stack= (char*) &thd; > + thd->store_globals(); > + thd->security_ctx->skip_grants(); > + IO_CACHE log; > + const char *errmsg; > + File file; > + bool enable_apply_event= false; > + Log_event *ev = 0; > + LOG_INFO linfo; > + int recover_xa_count= xa_recover_list.records; > + xa_recovery_member *member= NULL; > + > + if (!(rli= thd->rli_fake= new Relay_log_info(FALSE, "Recovery"))) > + { > + my_error(ER_OUTOFMEMORY, MYF(ME_FATAL), 1); > + goto err2; > + } > + rli->sql_driver_thd= thd; > + static LEX_CSTRING connection_name= { STRING_WITH_LEN("Recovery") }; > + rli->mi= new Master_info(&connection_name, false); > + if (!(rgi= thd->rgi_fake)) > + rgi= thd->rgi_fake= new rpl_group_info(rli); > + rgi->thd= thd; > + thd->system_thread_info.rpl_sql_info= > + new rpl_sql_thread_info(rli->mi->rpl_filter); > + > + if (rli && !rli->relay_log.description_event_for_exec) > + { > + rli->relay_log.description_event_for_exec= > + new Format_description_log_event(4); > + } > + if (find_log_pos(&linfo, xa_binlog_checkpoint_name, 1)) > + { > + sql_print_error("Binlog file '%s' not found in binlog index, needed " > + "for recovery. Aborting.", xa_binlog_checkpoint_name); > + goto err2; > + } > + > + tmp_disable_binlog(thd); > + thd->variables.pseudo_slave_mode= TRUE; > + for (;;) > + { > + if ((file= open_binlog(&log, linfo.log_file_name, &errmsg)) < 0) > + { > + sql_print_error("%s", errmsg); > + goto err1; > + } > + while (recover_xa_count > 0 && > + (ev= Log_event::read_log_event(&log, > + rli->relay_log.description_event_for_exec, > + opt_master_verify_checksum))) > + { > + if (!ev->is_valid()) > + { > + sql_print_error("Found invalid binlog query event %s" > + " at %s:%llu; error %d %s", ev->get_type_str(), > + linfo.log_file_name, > + (ev->log_pos - ev->data_written)); > + goto err1; > + } > + enum Log_event_type typ= ev->get_type_code(); > + ev->thd= thd; > + > + if (typ == FORMAT_DESCRIPTION_EVENT) > + enable_apply_event= true; > + > + if (typ == GTID_EVENT) > + { > + Gtid_log_event *gev= (Gtid_log_event *)ev; > + if (gev->flags2 & > + (Gtid_log_event::FL_PREPARED_XA | Gtid_log_event::FL_COMPLETED_XA)) > + { > + if ((member= > + (xa_recovery_member*) my_hash_search(&xa_recover_list, > + gev->xid.key(), > + gev->xid.key_length()))) > + { > + /* > + When XA PREPARE group of events (as flagged so) check > + its actual binlog state which may be COMPLETED. If the > + state is also PREPARED then analyze through > + in_engine_prepare whether the transaction needs replay. > + */ > + if (gev->flags2 & Gtid_log_event::FL_PREPARED_XA) > + { > + if (member->state == XA_PREPARE) > + { > + // XA prepared is not present in (some) engine then apply it > + if (member->in_engine_prepare == 0) > + enable_apply_event= true; > + else if (gev->flags2 & Gtid_log_event::FL_MULTI_ENGINE_XA && > + xa_recover_htons > member->in_engine_prepare) This needs a comment, something like "FL_MULTI_ENGINE_XA says the transaction must be prepared in more than one engine. We don't know in how many, so if it's not prepared in *all* engines we'll replay it just in case" I presume that (a bit silly) logic will do away when you will port this patch to use a counter instead of a flag. Please, don't push if before that. > + { > + enable_apply_event= true; > + // partially engine-prepared XA is first cleaned out prior replay > + thd->lex->sql_command= SQLCOM_XA_ROLLBACK; > + ha_commit_or_rollback_by_xid(&gev->xid, 0); > + } > + else > + --recover_xa_count; > + } > + } > + else if (gev->flags2 & Gtid_log_event::FL_COMPLETED_XA) > + { > + if (member->state == XA_COMPLETE && > + member->in_engine_prepare > 0) > + enable_apply_event= true; why? you cannot replay a fully prepared and partially committed transaction > + else > + --recover_xa_count; > + } > + } > + } > + } > + > + if (enable_apply_event) > + { > + if ((err= ev->apply_event(rgi))) > + { > + sql_print_error("Failed to execute binlog query event of type: %s," > + " at %s:%llu; error %d %s", ev->get_type_str(), > + linfo.log_file_name, > + (ev->log_pos - ev->data_written), > + thd->get_stmt_da()->sql_errno(), > + thd->get_stmt_da()->message()); > + delete ev; > + goto err1; > + } > + else if (typ == FORMAT_DESCRIPTION_EVENT) > + enable_apply_event=false; how can that happen? > + else if (thd->lex->sql_command == SQLCOM_XA_PREPARE || > + thd->lex->sql_command == SQLCOM_XA_COMMIT || > + thd->lex->sql_command == SQLCOM_XA_ROLLBACK) > + { > + --recover_xa_count; > + enable_apply_event=false; > + > + sql_print_information("Binlog event %s at %s:%llu" > + " successfully applied", > + typ == XA_PREPARE_LOG_EVENT ? > + static_cast<XA_prepare_log_event *>(ev)->get_query() : > + static_cast<Query_log_event *>(ev)->query, > + linfo.log_file_name, (ev->log_pos - ev->data_written)); > + } > + } > + if (typ != FORMAT_DESCRIPTION_EVENT) > + delete ev; > + } > + end_io_cache(&log); > + mysql_file_close(file, MYF(MY_WME)); > + file= -1; > + if (unlikely((error= find_next_log(&linfo, 1)))) > + { > + if (error != LOG_INFO_EOF) > + sql_print_error("find_log_pos() failed (error: %d)", error); > + else > + break; > + } > + } > +err1: > + reenable_binlog(thd); > + /* > + There should be no more XA transactions to recover upon successful > + completion. > + */ > + if (recover_xa_count > 0) > + goto err2; > + sql_print_information("Crash recovery finished."); > + err= false; > +err2: > + if (file >= 0) > + { > + end_io_cache(&log); > + mysql_file_close(file, MYF(MY_WME)); > + } > + thd->variables.pseudo_slave_mode= FALSE; > + delete rli->mi; > + delete thd->system_thread_info.rpl_sql_info; > + rgi->slave_close_thread_tables(thd); > + thd->reset_globals(); > + delete thd; > + > + return err; > +#endif /* !HAVE_REPLICATION */ > +} > > int > MYSQL_BIN_LOG::do_binlog_recovery(const char *opt_name, bool do_xa_recovery) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-17399: JSON_TABLE: final input
by Sergey Petrunia 13 Mar '21

13 Mar '21

Hi Alexey, Please find below final bits of input. After these are addressed the patch will probably be good to push (But I'll need a final pass before giving ok to push). == json_table_mysql* tests == * Please remove one of the json_table_mysql and json_table_mysql2 tests. * Please move the remaining test to be in the same suite as json_table.test. == Fix json_table.test == The test has this somewhere in the middle: set optimizer_switch='firstmatch=off'; Please restore the original setting. == ha_json_table::rnd_next() returns 0 on error == Consider this query: select * from json_table( '[ {"name": "X"}, {"name2": "Y"} ]', '$[*]' columns ( col1 varchar(100) path '$.name2' error on empty )) as T; The query produces an error. It happens like so: * the call to ha_json_table::rnd_next() returns 0. * The SQL layer finds out about the error by checking thd->is_error() which returns true. Please make ha_json_table::rnd_next() return an error. == Rename table_function.* == Looking at the contets of sql/table_function.{h,cc}, one can see that there's very little code there that is applicable to generic table function. Almost all code is specifi to JSON_TABLE. Because of that, please rename the files to e.g. json_table.{h,cc}. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 5

[Maria-developers] MDEV-17399: JSON_TABLE: Odd code in table.cc:create_view_field ?
by Sergey Petrunia 12 Mar '21

12 Mar '21

Hi Alexey, What does the code quoted below do? I don't recall seeing it on previous review iterations. In any case, * It needs a comment about why such special handling is needed. * It needs test coverage - I have reverted these changes and didn't see any test to fail? > diff --git a/sql/table.cc b/sql/table.cc > index 4f65dbd65f4..9c205fc4be6 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -6722,6 +6722,8 @@ Item *create_view_field(THD *thd, TABLE_LIST *view, Item **field_ref, > LEX_CSTRING *name) > { > bool save_wrapper= thd->lex->first_select_lex()->no_wrap_view_item; > + bool *wrapper_to_set= thd->lex->current_select ? > + &thd->lex->current_select->no_wrap_view_item : &save_wrapper; > Item *field= *field_ref; > DBUG_ENTER("create_view_field"); > > @@ -6737,17 +6739,17 @@ Item *create_view_field(THD *thd, TABLE_LIST *view, Item **field_ref, > } > > DBUG_ASSERT(field); > - thd->lex->current_select->no_wrap_view_item= TRUE; > + *wrapper_to_set= TRUE; > if (!field->is_fixed()) > { > if (field->fix_fields(thd, field_ref)) > { > - thd->lex->current_select->no_wrap_view_item= save_wrapper; > + *wrapper_to_set= save_wrapper; > DBUG_RETURN(0); > } > field= *field_ref; > } > - thd->lex->current_select->no_wrap_view_item= save_wrapper; > + *wrapper_to_set= save_wrapper; > if (save_wrapper) > { > DBUG_RETURN(field); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

[Maria-developers] GSOC21: MDEV-16375 & MDEV-23143
by Hollow Man 10 Mar '21

10 Mar '21

Hi MariaDB community!    Glad to be here! My github account is @HollowMan6. Though I'm new to MariaDB community, I'm interested in MDEV-16375 & MDEV-23143: Function to normalize a json value & missing a JSON_EQUALS function for this year's GSOC project. Here are my first thoughts on these issues:    I have checked part of the codebase and I think the two issues can be merged into one. First we can create a function named JSON_NORMALIZE to normalize the json, which automatically parses the inputed json document, recursively sorts the keys (for objects) / sorts the numbers (for arrays), removes the spaces, and then return the json document string.    Then we create a function named JSON_EQUALS, which can be used to compare 2 json documents for equality realized by first seperately normalize the two json documents using JSON_NORMALIZE, then the 2 can be compared exactly as binary strings.    I have taken some inspirations from the Item_func_json_keys and json_scan_start for parsing json documents, and I think it's possible to sort the keys using std::map in STL for objects.    That's all for my ideas so far. Please correct me if I made some mistakes, and I'm going to work on my ideas later. Cheers! Hollow Man

1 0

Re: [Maria-developers] MDEV-25075: Ignorable index makes the resulting CREATE TABLE invalid
by Sergey Petrunia 10 Mar '21

10 Mar '21

Hi Varun, > diff --git a/sql/lex.h b/sql/lex.h > index 542356c0e433..5a9ec2ec1b35 100644 > --- a/sql/lex.h > +++ b/sql/lex.h > @@ -289,6 +289,7 @@ static SYMBOL symbols[] = { > { "IDENTIFIED", SYM(IDENTIFIED_SYM)}, > { "IF", SYM(IF_SYM)}, > { "IGNORE", SYM(IGNORE_SYM)}, > + { "IGNORED", SYM(IGNORED_SYM)}, > { "IGNORE_DOMAIN_IDS", SYM(IGNORE_DOMAIN_IDS_SYM)}, > { "IGNORE_SERVER_IDS", SYM(IGNORE_SERVER_IDS_SYM)}, > { "IMMEDIATE", SYM(IMMEDIATE_SYM)}, Above this array one can see: >> ... >> NOTE!! >> If you add or delete symbols from this file, you must also update results for >> the perfschema.start_server_low_digest_sql_length test! >> */ >> >> static SYMBOL symbols[] = { and indeed the buildbot shows failures, e.g. http://buildbot.askmonty.org/buildbot/builders/kvm-rpm-centos74-amd64/build… CURRENT_TEST: perfschema.start_server_low_digest_sql_length --- /usr/share/mysql-test/suite/perfschema/r/start_server_low_digest_sql_length.result 2021-03-10 06:23:20.000000000 +0000 +++ /dev/shm/var/4/log/start_server_low_digest_sql_length.reject 2021-03-10 07:37:16.351213429 +0000 @@ -8,5 +8,5 @@ #################################### SELECT event_name, digest, digest_text, sql_text FROM events_statements_history_long; event_name digest digest_text sql_text -statement/sql/select beb5bd93b7e8c45bc5cb6060804988e8 SELECT ? + ? + SELECT ... -statement/sql/truncate faf6cefb662b443f05e97b5c5ab14a59 TRUNCATE TABLE truncat... +statement/sql/select ade774bdfbc132a71810ede8ef469660 SELECT ? + ? + SELECT ... +statement/sql/truncate 0f84807fb4a75d0f391f8a93e7c3c182 TRUNCATE TABLE truncat... Please fix this. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 1

Re: [Maria-developers] bf677c554ad: UTF8 and utf8
by Sergei Golubchik 08 Mar '21

08 Mar '21

Hi, Rucha! On Mar 08, Rucha Deodhar wrote: > revision-id: bf677c554ad (mariadb-10.5.2-402-gbf677c554ad) > parent(s): a1542f8a573 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2021-02-22 00:37:18 +0530 > message: > > UTF8 and utf8 Two main comments: * your global search-and-replace utf8 to utf8mb3 everywhere has replaced way too much. Please, revert it. * why did you have so many changes in tests with utf8mb4? You set the old-mode to be utf8_is_utf8mb3 by default. It should've applied to tests too, right? So why tests have switched to utf8mb4? Other comments below. > diff --git a/debian/additions/innotop/innotop b/debian/additions/innotop/innotop > index 19229a57a82..4f1e76ea497 100644 > --- a/debian/additions/innotop/innotop > +++ b/debian/additions/innotop/innotop > @@ -264,9 +264,9 @@ sub get_dbh { > MKDEBUG && _d("$dbh: $sql"); > $dbh->do($sql); > MKDEBUG && _d('Enabling charset for STDOUT'); > - if ( $charset eq 'utf8' ) { > - binmode(STDOUT, ':utf8') > - or die "Can't binmode(STDOUT, ':utf8'): $OS_ERROR"; > + if ( $charset eq 'utf8mb3' ) { > + binmode(STDOUT, ':utf8mb3') > + or die "Can't binmode(STDOUT, ':utf8mb3'): $OS_ERROR"; It seems you performed search-and-replace over the whole source tree. I don't think it was particularly meaningful. Here, for example, it was wrong, it's perl code and perl charser utf8, there's no utf8mb3 in perl. also all internal variables you've renamed, like below: > } > else { > binmode(STDOUT) or die "Can't binmode(STDOUT): $OS_ERROR"; > diff --git a/sql/field.h b/sql/field.h > index 4a4f7cee2a5..f672ae7e315 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -582,7 +582,7 @@ class Virtual_column_info: public Sql_alloc, > public: > /* Flag indicating that the field is physically stored in the database */ > bool stored_in_db; > - bool utf8; /* Already in utf8 */ > + bool utf8mb3; /* Already in utf8 */ I mean, here ^^^ > bool automatic_name; > Item *expr; > Lex_ident name; /* Name of constraint */ > diff --git a/client/mysql.cc b/client/mysql.cc > index f2938d4c824..b0ee239c0bc 100644 > --- a/client/mysql.cc > +++ b/client/mysql.cc > @@ -1918,6 +1918,10 @@ static int get_options(int argc, char **argv) > > if ((ho_error=handle_options(&argc, &argv, my_long_options, get_one_option))) > return(ho_error); > + if (!strcasecmp("utf8",default_charset)) > + { > + default_charset=(char*)"utf8mb3"; > + } why is that? (and the same everywhere else) > > *mysql_params->p_max_allowed_packet= opt_max_allowed_packet; > *mysql_params->p_net_buffer_length= opt_net_buffer_length; > diff --git a/client/mysqldump.c b/client/mysqldump.c > index 7c363973da2..ad69618fbd1 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -3160,7 +3160,7 @@ static uint get_table_structure(const char *table, const char *db, char *table_t > > fprintf(sql_file, > "SET @saved_cs_client = @@character_set_client;\n" > - "SET character_set_client = utf8;\n" > + "SET character_set_client = utf8mb3;\n" why is that? > "/*!50001 CREATE TABLE %s (\n", > result_table); > > diff --git a/include/m_ctype.h b/include/m_ctype.h > index 5fa8f28ff7a..f1a8166d9df 100644 > --- a/include/m_ctype.h > +++ b/include/m_ctype.h > @@ -223,6 +223,7 @@ extern MY_UNI_CTYPE my_uni_ctype[256]; > #define MY_CS_NON1TO1 0x40000 /* Has a complex mapping from characters > to weights, e.g. contractions, expansions, > ignorable characters */ > +#define MY_CS_UTF8_IS_UTF8MB3 0x80000 this isn't a MY_CS_xxx flag, it doesn't belong here, see all other MY_CS_xxx flags - they are *properties* of a charset. They are set as struct charset_info_st my_charset_tis620_thai_ci= { 18,0,0, /* number */ MY_CS_COMPILED|MY_CS_PRIMARY|MY_CS_STRNXFRM|MY_CS_NON1TO1, /* state */ charset_name_tis620, /* cs name */ "tis620_thai_ci", /* name */ that is, when defining a charset. See below > #define MY_CHARSET_UNDEFINED 0 > > /* Character repertoire flags */ > diff --git a/man/mysql.1 b/man/mysql.1 > index 03f23df3660..5ad953ba5eb 100644 > --- a/man/mysql.1 > +++ b/man/mysql.1 > @@ -401,7 +401,7 @@ Use > as the default character set for the client and connection\&. > .sp > A common issue that can occur when the operating system uses > -utf8 > +utf8mb3 this didn't need changing > or another multi\-byte character set is that output from the > \fBmysql\fR > client is formatted incorrectly, due to the fact that the MariaDB client uses the > diff --git a/mysys/charset.c b/mysys/charset.c > index 32cfeb56e2d..9d035406e8d 100644 > --- a/mysys/charset.c > +++ b/mysys/charset.c > @@ -361,7 +361,7 @@ static int add_collation(struct charset_info_st *cs) > newcs->state|= MY_CS_AVAILABLE | MY_CS_LOADED | MY_CS_NONASCII; > #endif > } > - else if (!strcmp(cs->csname, "utf8") || !strcmp(cs->csname, "utf8mb3")) > + else if (!strcmp(cs->csname, "utf8mb3") || !strcmp(cs->csname, "utf8mb3")) one more result of your mechanical search-and-replace, now you compare with "utf8mb3" twice. > { > #if defined (HAVE_CHARSET_utf8mb3) && defined(HAVE_UCA_COLLATIONS) > copy_uca_collation(newcs, newcs->state & MY_CS_NOPAD ? > @@ -767,7 +767,7 @@ static const char* > get_charset_name_alias(const char *name) > { > if (!my_strcasecmp(&my_charset_latin1, name, "utf8mb3")) > - return "utf8"; > + return "utf8mb3"; and again > return NULL; > } > > @@ -1004,13 +1004,19 @@ CHARSET_INFO * > get_charset_by_csname(const char *cs_name, uint cs_flags, myf flags) > { > MY_CHARSET_LOADER loader; > + > + if (!strcasecmp(cs_name, "utf8")) > + { > + cs_name = (const char*)(cs_flags & MY_CS_UTF8_IS_UTF8MB3 ? "utf8mb3" : "utf8mb4"); your flag should be in `myf flags` not in `uint cs_flags` (and defined in my_sys.h with other myf flags) > + } > + > my_charset_loader_init_mysys(&loader); > return my_charset_get_by_name(&loader, cs_name, cs_flags, flags); > } > > > /** > - Resolve character set by the character set name (utf8, latin1, ...). > + Resolve character set by the character set name (utf8mb3, latin1, ...). > > The function tries to resolve character set by the specified name. If > there is character set with the given name, it is assigned to the "cs" > @@ -1453,8 +1459,8 @@ static const MY_CSET_OS_NAME charsets[] = > > {"US-ASCII", "latin1", my_cs_approx}, > > - {"utf8", "utf8", my_cs_exact}, > - {"utf-8", "utf8", my_cs_exact}, > + {"utf8mb3", "utf8mb3", my_cs_exact}, > + {"utf-8mb3", "utf8mb3", my_cs_exact}, utf-8mb3 really? :) > #endif > {NULL, NULL, 0} > }; > diff --git a/plugin/win_auth_client/common.cc b/plugin/win_auth_client/common.cc > index 8b7319252ac..f46b0ec9696 100644 > --- a/plugin/win_auth_client/common.cc > +++ b/plugin/win_auth_client/common.cc > @@ -338,7 +338,7 @@ UPN::UPN(): m_buf(NULL) > m_buf= wchar_to_utf8(buf1, &m_len); > > if(!m_buf) > - ERROR_LOG(ERROR, ("Failed to convert UPN to utf8")); > + ERROR_LOG(ERROR, ("Failed to convert UPN to utf8mb3")); probably wrong too (everything in this file), see the code - it uses windows utf8 charset, not mariadb's utf8 > > // Note: possible error would be indicated by the fact that m_buf is NULL. > return; > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 0bf21e02002..fd131497f8c 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -4053,6 +4056,24 @@ static int init_common_variables() > break; > } > > + if (default_collation_name) > + { > + char *copy_of_name= (char*)default_collation_name; > + char start[6]; > + strncpy(start, default_collation_name, 5); > + char *fname= (char *)"utf8mb3_"; > + if (! strncasecmp("utf8_", start,5)) > + { > + copy_of_name+= 5; > + char result[64]; > + result[63]='\0'; > + strcpy(result, fname); > + strcat(result, copy_of_name); > + result[strlen(copy_of_name)+strlen(fname)]='\0'; > + default_collation_name= (char *) result; > + } > + } 1. wrong indentation 2. shouldn't it be in the get_charset_by_name ? 3. shouldn't it depend on the UTF8_IS_UTF8MB3 ? > + > if (default_collation_name) > { > CHARSET_INFO *default_collation; > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 50b746fe514..f0616b40361 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -5248,6 +5249,65 @@ class THD: public THD_count, /* this must be first */ > Item *sp_prepare_func_item(Item **it_addr, uint cols= 1); > bool sp_eval_expr(Field *result_field, Item **expr_item_ptr); > > + CHARSET_INFO *get_charset_by_csname(const char *cs_name, > + uint cs_flags, > + myf myf_flags) const > + { > + return ::get_charset_by_csname(cs_name, > + this->variables.old_behavior & OLD_MODE_UTF8_IS_UTF8MB3 ? > + cs_flags | MY_CS_UTF8_IS_UTF8MB3 : cs_flags, > + myf_flags); > + } > + > + inline const char* get_collation_or_charset_name(const char* name) shouldn't the name be get_collation_name_alias() ? all other method names you've created match corresponding charset-related function names > + { > + char *copy_of_name= (char*)name; > + char start[6]; > + strncpy(start, name, 5); > + char *fname= (char *)(this->variables.old_behavior & OLD_MODE_UTF8_IS_UTF8MB3?"utf8mb3_":"utf8mb4_"); > + if (!strncasecmp("utf8_", start,5)) > + { > + copy_of_name+= 5; > + char result[64]; > + result[63]='\0'; > + strcpy(result, fname); > + strcat(result, copy_of_name); > + result[strlen(copy_of_name)+strlen(fname)]='\0'; > + name= (const char *) result; > + } > + return name; here you return a pointer to the local variable also: indentation, too long line, end-of-line spaces > + } > + > + inline CHARSET_INFO * > + mysqld_collation_get_by_name(const char *name, > + CHARSET_INFO *name_cs= system_charset_info) > + { > + name=this->get_collation_or_charset_name(name); > + return ::mysqld_collation_get_by_name(name, name_cs); > +} > + > +CHARSET_INFO *get_charset_by_name(const char *cs_name, myf flags) > +{ > + > + cs_name=this->get_collation_or_charset_name(cs_name); > + return ::get_charset_by_name(cs_name,flags); here it'd be better to pass MY_UTF8_IS_UTF8MB3 in flags > +} > +my_bool resolve_charset(const char *cs_name, > + CHARSET_INFO *default_cs, > + CHARSET_INFO **cs) > +{ > + cs_name=this->get_collation_or_charset_name(cs_name); > + return ::resolve_charset(cs_name,default_cs,cs); perhaps it'd make sense to introduce flags argument here too? > +} > + > +my_bool resolve_collation(const char *cl_name, > + CHARSET_INFO *default_cl, > + CHARSET_INFO **cl) > +{ > + cl_name= this->get_collation_or_charset_name(cl_name); > + return ::resolve_collation(cl_name,default_cl,cl); > +} please make sure the indentation is correct, I won't comment on it anymore > + > }; > > > diff --git a/sql/sql_db.cc b/sql/sql_db.cc > index 9bf16220535..85e0e0dbd2f 100644 > --- a/sql/sql_db.cc > +++ b/sql/sql_db.cc > @@ -583,9 +583,13 @@ bool load_db_opt(THD *thd, const char *path, Schema_specification_st *create) > default-collation commands. > */ > if (!(create->default_table_charset= > - get_charset_by_csname(pos+1, MY_CS_PRIMARY, MYF(0))) && > + thd->get_charset_by_csname(pos+1, thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8MB3 ? > + MY_CS_UTF8_IS_UTF8MB3 | MY_CS_PRIMARY : > + MY_CS_PRIMARY, > + MYF(0))) && that's a bit redundant, don't you think? you use both thd-> method that replaces utf_ with a real charset name and *also* you set MY_CS_UTF8_IS_UTF8MB3 flag. there are other places like that, to keep the review shorter I won't comment on them below. > !(create->default_table_charset= > - get_charset_by_name(pos+1, MYF(0)))) > + thd->get_charset_by_name(pos+1, MYF(0)))) > { > sql_print_error("Error while loading database options: '%s':",path); > sql_print_error(ER_THD(thd, ER_UNKNOWN_CHARACTER_SET),pos+1); > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index a75066271d9..fd480bdced2 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -3705,7 +3709,7 @@ static Sys_var_set Sys_old_behavior( > "old_mode", > "Used to emulate old behavior from earlier MariaDB or MySQL versions", > SESSION_VAR(old_behavior), CMD_LINE(REQUIRED_ARG), > - old_mode_names, DEFAULT(0)); > + old_mode_names, DEFAULT(8)); DEFAULT(OLD_MODE_UTF8_IS_UTF8MB3) please > > #if defined(HAVE_OPENSSL) && !defined(EMBEDDED_LIBRARY) > #define SSL_OPT(X) CMD_LINE(REQUIRED_ARG,X) > diff --git a/storage/connect/ha_connect.cc b/storage/connect/ha_connect.cc > index 849cf7594eb..b6e8f3b3226 100644 > --- a/storage/connect/ha_connect.cc > +++ b/storage/connect/ha_connect.cc > @@ -6099,8 +6099,8 @@ static int connect_assisted_discovery(handlerton *, THD* thd, > case FLD_NAME: > if (ttp == TAB_PRX || > (ttp == TAB_CSV && topt->data_charset && > - (!stricmp(topt->data_charset, "UTF8") || > - !stricmp(topt->data_charset, "UTF-8")))) > + (!stricmp(topt->data_charset, "UTF8MB3") || > + !stricmp(topt->data_charset, "UTF-8MB3")))) UTF-8MB3 again > cnm= crp->Kdata->GetCharValue(i); > else > cnm= encode(g, crp->Kdata->GetCharValue(i)); > diff --git a/storage/mroonga/vendor/groonga/CMakeLists.txt b/storage/mroonga/vendor/groonga/CMakeLists.txt > index d271d4c4eb9..fa5d6f97ddd 100644 > --- a/storage/mroonga/vendor/groonga/CMakeLists.txt > +++ b/storage/mroonga/vendor/groonga/CMakeLists.txt > @@ -95,7 +95,7 @@ set(GRN_LOG_PATH > "${CMAKE_INSTALL_PREFIX}/var/log/${GRN_PROJECT_NAME}/${GRN_PROJECT_NAME}.log" > CACHE FILEPATH "log file path") > set(GRN_DEFAULT_ENCODING > - "utf8" > + "utf8mb3" utf8 is better here > CACHE STRING "Groonga's default encoding") > set(GRN_DEFAULT_MATCH_ESCALATION_THRESHOLD > 0 > diff --git a/storage/perfschema/unittest/pfs_connect_attr-t.cc b/storage/perfschema/unittest/pfs_connect_attr-t.cc > index b57ead3ec26..7e5b511d543 100644 > --- a/storage/perfschema/unittest/pfs_connect_attr-t.cc > +++ b/storage/perfschema/unittest/pfs_connect_attr-t.cc > @@ -343,7 +343,12 @@ int main(int, char **) > { > MY_INIT("pfs_connect_attr-t"); > > - cs_cp1251= get_charset_by_csname("cp1251", MY_CS_PRIMARY, MYF(0)); > + cs_cp1251= thd->get_charset_by_csname("cp1251", > + thd->variables.old_behavior & > + OLD_MODE_UTF8_IS_UTF8M3 ? > + MY_CS_PRIMARY | MY_CS_UTF8_IS_UTF8MB3 : well, here there is surely no reason to use MY_CS_UTF8_IS_UTF8MB3, because the charset is a hard-coded literal and it's not "utf8", is it? > + MY_CS_PRIMARY, > + MYF(0)); > if (!cs_cp1251) > diag("skipping the cp1251 tests : missing character set"); > plan(59 + (cs_cp1251 ? 10 : 0)); > diff --git a/strings/ctype.c b/strings/ctype.c > index 46951c3ae1f..9054cbfc3e6 100644 > --- a/strings/ctype.c > +++ b/strings/ctype.c > @@ -37,7 +37,7 @@ > */ > > const char charset_name_latin2[]= "latin2"; > -const char charset_name_utf8[]= "utf8"; > +const char charset_name_utf8[]= "utf8mb3"; better rename the variable to charset_name_utf8mb3 > const char charset_name_utf16[]= "utf16"; > const char charset_name_utf32[]= "utf32"; > const char charset_name_ucs2[]= "ucs2"; > diff --git a/tests/mysql_client_test.c b/tests/mysql_client_test.c > index 0043786d477..37da9e1f62e 100644 > --- a/tests/mysql_client_test.c > +++ b/tests/mysql_client_test.c > @@ -405,7 +405,7 @@ static void test_prepare_simple() > > /* update */ > strmov(query, "UPDATE test_prepare_simple SET id=? " > - "WHERE id=? AND CONVERT(name USING utf8)= ?"); > + "WHERE id=? AND CONVERT(name USING utf8mb3)= ?"); I suspect all tests should better use "utf8" > stmt= mysql_simple_prepare(mysql, query); > check_stmt(stmt); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-8306: Some pieces of input
by Sergey Petrunia 05 Mar '21

05 Mar '21

Hi Varun, I'm still looking at MDEV-8306 code. Sending the input I have so far. There's nothing important but I thought I'd share it now so you can address it while I'm continuing with the review. > propagate_equal_field_for_orderby Please use "order_by" as in all other functions, and also change "field" to "fields". The name becomes propagate_equal_fields_for_order_by. > void JOIN::propagate_equal_field_for_orderby() > { > if (!sort_nest_possible) > return; > ORDER *ord; > for (ord= order; ord; ord= ord->next) > { > if (optimizer_flag(thd, OPTIMIZER_SWITCH_ORDERBY_EQ_PROP) && cond_equal) > { Is there any reason not to check optimizer_switch flag value once at the start of the function? > void JOIN_TAB::find_keys_that_can_achieve_ordering() > { > if (!join->sort_nest_possible) > return; > > table->keys_with_ordering.clear_all(); > for (uint index= 0; index < table->s->keys; index++) > { > if (table->keys_in_use_for_query.is_set(index) && > test_if_order_by_key(join, join->order, table, index)) > table->keys_with_ordering.set_bit(index); > } > table->keys_with_ordering.intersect(table->keys_in_use_for_order_by); > } Is there any reason to call test_if_order_by_key() and then interset with keys_in_use_for_order_by? Why not just check that bitmap first, like it's done for keys_with_ordering? ... in struct JOIN_TAB: > void find_keys_that_can_achieve_ordering(); > bool check_if_index_satisfies_ordering(int index_used); are "achieving ordering" and "satisfying ordering" the same thing? If yes, it's better to use one term for this. > Item* Item_subselect::transform(THD *thd, Item_transformer transformer, > bool transform_subquery, uchar *arg) We've already discussed this, but I'll mention it here to keep track: this should ON expressions. I would also consider adding an assert that the subquery's joins do not have the query plans, yet. > class Field { > ... > void statistics_available_via_keys(); > void statistics_available_via_stat_tables(); These names need to be better. Function names typically start with a verb. This is especially true for functions that return nothing. > bool Item_func_between::predicate_selectivity_checker(void *arg) > bool Item_func_in::predicate_selectivity_checker(void *arg) These functions need a comment. > bool Item_equal::predicate_selectivity_checker(void *arg) > { > ... > while (it++) > { > Field *field= it.get_curr_field(); > field->is_range_statistics_available(); What is the above? Does the function have some side-effect? This needs to be fixed or at least commented. ... later in the same function: > it.rewind(); > Item *item= (it++); > SAME_FIELD *same_field_arg= (SAME_FIELD *) arg; > > if (same_field_arg->item == NULL) > { > item->is_resolved_by_same_column(arg); > return false; > } This looks very confusing, too. Why are we checking the first element in the Item_equal. Are we assuming it is the constant? But the execution reaches this point even when Item_equal doesn't contain a constant? > bool Item_equal::is_statistics_available() Maybe, rename this to is_range_statistics_available() to make it clear with what kind of statistics is it? > void add_nest_tables_to_trace(Mat_join_tab_nest_info* nest_info) rename this to trace_XXX() to be uniform with other tracing functions? e.g. rename to trace_tables_in_sort_nest BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] MDEV-17399: JSON_TABLE: Code cleanup
by Sergey Petrunia 04 Mar '21

04 Mar '21

Hi Alexey, Please check out this patch: http://lists.askmonty.org/pipermail/commits/2021-March/014492.html Any objections to it? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] MDEV-17399: JSON_TABLE: Crash with nested path
by Sergey Petrunia 01 Mar '21

01 Mar '21

Hi Alexey, I was looking at Json_table_nested_path::set_position(), wondering why does it have an assignment np->m_null= TRUE; but doesn't clear the NULL values and trying to come up with an example of this going wrong when I've hit this crash: select * from json_table( '[ {"name": "X", "colors":["blue"], "sizes": [1,2,3,4], "prices" : [10,20]}, {"name": "Y", "colors":["red"], "sizes": [10,11], "prices" : [100,200,300]} ]', '$[*]' columns ( seq0 for ordinality, name varchar(4) path '$.name', nested path '$.colors[*]' columns ( seq1 for ordinality, color text path '$' ), nested path '$.sizes[*]' columns ( seq2 for ordinality, size int path '$' ), nested path '$.prices[*]' columns ( seq3 for ordinality, price int path '$' ) ) ) as T order by seq0, name; Note this==NULL: (gdb) wher #0 0x00005555560edf72 in Json_table_nested_path::set_position (this=0x0, j_start=0x7ffeb0016e68 "[ \n {\"name\": \"X\", \n \"colors\":[\"blue\"], \"sizes\": [1,2,3,4], \"prices\" : [10,20]},\n {\"name\": \"Y\", \n \"colors\":[\"red\"], \"sizes\": [10,11], \"prices\" : [100,200,300]}\n]", j_end=0x7ffeb0016f12 "", pos=0x7ffeb0035e51 "\245\245\245\245\245\245\245\245\006") at /home/psergey/dev-git2/10.6-hf-review6/sql/table_function.cc:239 #1 0x00005555560ee12f in Json_table_nested_path::set_position (this=0x7ffeb0017060, j_start=0x7ffeb0016e68 "[ \n {\"name\": \"X\", \n \"colors\":[\"blue\"], \"sizes\": [1,2,3,4], \"prices\" : [10,20]},\n {\"name\": \"Y\", \n \"colors\":[\"red\"], \"sizes\": [10,11], \"prices\" : [100,200,300]}\n]", j_end=0x7ffeb0016f12 "", pos=0x7ffeb0035e48 "") at /home/psergey/dev-git2/10.6-hf-review6/sql/table_function.cc:262 #2 0x00005555560ee9f0 in ha_json_table::rnd_pos (this=0x7ffeb0014f00, buf=0x7ffeb0025570 "\377", pos=0x7ffeb0035e48 "") at /home/psergey/dev-git2/10.6-hf-review6/sql/table_function.cc:434 #3 0x00005555561ca6a4 in handler::ha_rnd_pos (this=0x7ffeb0014f00, buf=0x7ffeb0025570 "\377", pos=0x7ffeb0035e48 "") at /home/psergey/dev-git2/10.6-hf-review6/sql/handler.cc:3101 #4 0x00005555563852e3 in rr_from_pointers (info=0x7ffeb001f9e0) at /home/psergey/dev-git2/10.6-hf-review6/sql/records.cc:615 #5 0x0000555555da4a75 in READ_RECORD::read_record (this=0x7ffeb001f9e0) at /home/psergey/dev-git2/10.6-hf-review6/sql/records.h:81 #6 0x0000555555ee1876 in join_init_read_record (tab=0x7ffeb001f918) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:21644 #7 0x0000555555edf35a in sub_select (join=0x7ffeb001d948, join_tab=0x7ffeb001f918, end_of_records=false) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:20666 #8 0x0000555555ede8e6 in do_select (join=0x7ffeb001d948, procedure=0x0) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:20216 #9 0x0000555555eb24e7 in JOIN::exec_inner (this=0x7ffeb001d948) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:4484 #10 0x0000555555eb1613 in JOIN::exec (this=0x7ffeb001d948) at /home/psergey/dev-git2/10.6-hf-review6/sql/sql_select.cc:4264 Please fix. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 2

Re: [Maria-developers] ee538938345: MDEV-21117: refine the server binlog-based recovery for semisync
by Sergei Golubchik 26 Feb '21

26 Feb '21

Hi, Sujatha! The main comment - the logic is so complex, I wasn't able to understand it, unfortunately. I've reviewed almost everything, see comments below. But not the Recovery_context methods. Please explain how it works and how all these truncate_validated, truncate_reset_done, truncate_set_in_1st, etc all work together. On Feb 26, Sujatha wrote: > revision-id: ee538938345 (mariadb-10.3.26-68-gee538938345) > parent(s): 7d04ce6a2d4 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2021-02-08 17:58:03 +0200 > message: > > MDEV-21117: refine the server binlog-based recovery for semisync > this should be MDEV subject. But if you like this better you can rename the MDEV instead > > Problem: > ======= > When the semisync master is crashed and restarted as slave it could > recover transactions that former slaves may never have seen. > A known method existed to clear out all prepared transactions > with --tc-heuristic-recover=rollback does not care to adjust > binlog accordingly. > > Fix: > === > The binlog-based recovery is made to concern of the slave semisync role of > post-crash restarted server. > No changes in behaviour is done to the "normal" binloggging server > and the semisync master. > > When the restarted server is configured with > --rpl-semi-sync-slave-enabled=1 > the refined recovery attempts to roll back prepared transactions > and truncate binlog accordingly. > In case of a partically committed (that is committed at least > partially > > in one of the engine participants) such transaction gets committed. > It's guaranteed no (partially as well) committed transactions > exist beyond the truncate position. > In case there exists a non-transactional replication event > (being in a way a committed transaction) past the > computed truncate position the recovery ends with an error. > > To facilite the failover on the slave side facilitate > conditions to accept own events (having been discarded by the above recovery) > are relaxed to let so for the semisync slave that connects to master > in gtid mode. gtid_strict_mode is further recommended to secure > from inadvertent re-applying out of order gtids in general. > Non-gtid mode connected semisync slave would require > --replicate-same-server-id (mind --log-slave-updates must be OFF then). Sorry, I failed to understand this paragraph :( > > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.test b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > new file mode 100644 > index 00000000000..cf89525dcac > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.test > @@ -0,0 +1,76 @@ > +# ==== Purpose ==== > +# > +# Test verifies the truncation of single binary log file. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Create table t1 and insert/commit a row. > +# 1 - Insert an another row such that it gets written to binlog but commit > +# in engine fails as server crashed at this point. > +# 2 - Restart server with --rpl-semi-sync-slave-enabled=1 > +# 3 - Upon server start 'master-bin.000001' will be truncated to contain > +# only the first insert > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_aria.inc > +--source include/have_log_bin.inc > +--source include/have_debug.inc > +--source include/have_binlog_format_statement.inc have_binlog_format_statement obviously implies have_log_bin. it's redundant to specify it explicitly. and you don't need have_debug here > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > + > +# The following cases are tested: > +# A. 2pc transaction is followed by a blank "zero-engines" one > +# B. 2pc transaction follows the blank one > +# C. Similarly to A, with the XA blank transaction > + > +--connection default it's connection default by default (not surprisingly) :) > +RESET MASTER; > +CREATE TABLE t ( f INT ) ENGINE=INNODB; > +CREATE TABLE t2 ( f INT ) ENGINE=INNODB; > +CREATE TABLE tm ( f INT ) ENGINE=Aria; Why Aria? I mean, the default "standard" non-transactional engine is MyISAM. If you use Aria, it means you thought about it and decided to deviate intentionally from the default. What were your considerations? > + > +--echo # Case A. > +# Both are doomed into truncation. > +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-6 why not simply --let SEARCH_PATTERN = Successfully truncated...etc... > +--let $query1 = INSERT INTO t VALUES (20) > +--let $query2 = DELETE FROM t2 WHERE f = 666 /* no such record */ > +--source binlog_truncate_active_log.inc > + > +--echo # Case B. > +# The inverted sequence ends up to truncate only $query2 > +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-10 > +--let $query1 = DELETE FROM t2 WHERE f = 0 why not `= 666 /* no such record */` ? is `= 0` important here? > +--let $query2 = INSERT INTO t VALUES (20) > +--source binlog_truncate_active_log.inc > + > + > +delimiter |; > +CREATE PROCEDURE sp_blank_xa() > +BEGIN > + XA START 'blank'; > + DELETE FROM t2 WHERE f = 666 /* no such record */; > + XA END 'blank'; > + XA PREPARE 'blank'; > +END| > +delimiter ;| > + > + > +--echo # Case C. > +--let $this_search_pattern = Successfully truncated.*to remove transactions starting from GTID 0-1-13 > +--let $query1 = INSERT INTO t VALUES (20) > +--let $pre_q2 = CALL sp_blank_xa > +--let $query2 = XA COMMIT 'blank' > +--source binlog_truncate_active_log.inc > +DROP PROCEDURE sp_blank_xa; > + > +--echo # Cleanup > +DROP TABLE t,t2,tm; > + > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > new file mode 100644 > index 00000000000..b1ffaf18268 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_active_log.inc > @@ -0,0 +1,70 @@ you need source include/have_debug_sync.inc here, as this file uses DEBUG_SYNC > +connect(master1,localhost,root,,); > +connect(master2,localhost,root,,); > +connect(master3,localhost,root,,); just FYI (no need to change anything), you know that parentheses and extra commas are optional, right? that is, it could be written as connect master1,localhost,root; connect master2,localhost,root; connect master3,localhost,root; > + > +--connection default > + > +# First to commit few transactions > +INSERT INTO t VALUES (10); > +INSERT INTO tm VALUES (10); > + > +--connection master1 > +# Hold insert after write to binlog and before "run_commit_ordered" in engine > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master1_ready WAIT_FOR signal_never_arrives"; > +--send_eval $query1 > + > +--connection master2 > +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > +if ($pre_q2) is $pre_q2 a flag or a query string? it's used as a flag, it's set as a query string. > +{ > + CALL sp_blank_xa; > +} > +SET DEBUG_SYNC= "commit_before_get_LOCK_after_binlog_sync SIGNAL master2_ready"; > +# To binlog non-xid transactional group which will be truncated all right > +--send_eval $query2 > + > + > +--connection master3 > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +SELECT @@global.gtid_binlog_pos as 'Before the crash'; > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait > +EOF > + > +--source include/kill_mysqld.inc > +--source include/wait_until_disconnected.inc you know that kill_mysqld both writes 'wait' to the expect file and includes wait_until_disconnected ? > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart: --rpl-semi-sync-slave-enabled=1 > +EOF why "append_file" and not "write_file" ? and why wait, you could've written restart right away, couldn't you? > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# Check error log for a successful truncate message. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; isn't it always $MYSQLTEST_VARDIR/log/mysqld.1.err? other tests don't do it conditionally > +} > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_RANGE=-50000 if you're searching in the error log - and you are - it's best not to set the SEARCH_RANGE at all. Then it'll search from the last CURRENT_TEST: label. That is, it'll search in all error log entries generated by the current test and not in the error log that came from earlier tests. > +--let SEARCH_PATTERN=$this_search_pattern > +--replace_regex /FOUND [0-9]+/FOUND #/ can it be found multiple times? Why would binlog be truncated more than once? > +--source include/search_pattern_in_file.inc > + > +SELECT @@global.gtid_binlog_pos as 'After the crash'; > +--echo "One row should be present in table 't'" > +SELECT COUNT(*) as 'One' FROM t; Do you know which one? If yes, why not `SELECT * FROM t` ? > + > +# Local cleanup > +DELETE FROM t; > +--disconnect master1 > +--disconnect master2 > +--disconnect master3 > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > new file mode 100644 > index 00000000000..fe153e5703c > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.test > @@ -0,0 +1,78 @@ > +# ==== Purpose ==== > +# > +# Test verifies truncation of multiple binary logs. and "with multiple transactional storage engines" ? > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Create two tables in innodb and rocksdb engines, > +# > +# In loop for A,B,C cases (below) do 1-5: there's no loop now (which is good, don't add it please :) > +# 1 - execute FLUSH LOGS command to generate a new binary log. > +# Start a transaction inserting rows of sufficient sizes > +# so binary log gets rotated at commit > +# 2 - Using debug simulation make the server crash at a point where > +# the transaction is written to binary log *and* either of > +# A. neither of them commits > +# B. only one commits > +# C. both commit > +# 3 - print the # of binlog files before the transaction starts and after its > +# commit is submitted > +# 4 - Restart server with --tc-heuristic-recover=BINLOG_TRUNCATE outdated comment? > +# 5 - Restart normally to print post recovery status. > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > + > +--source include/have_innodb.inc > +--source include/have_rocksdb.inc > +--source include/have_log_bin.inc this is redundant > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +--let $old_max_binlog_size= `select @@global.max_binlog_size` > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +--let $MYSQLD_DATADIR= `SELECT @@datadir` > + > +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=rocksdb; > + > +--let $case = A: neither engine committed => rollback & binlog truncate > +# Hold off engine commits after write to binlog and its rotation. > +# The transaction is killed along with the server after that. > +--let $kill_server=1 > +--let $debug_sync_action = "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR signal_no_signal" > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 > +--let $test_outcome= 1 row should be present in both tables; binlog is truncated; number of binlogs at reconnect - 3 > + --source binlog_truncate_multi_engine.inc > +--echo Proof of the truncated binlog file is readable (two transactions must be seen): > +--let $MYSQLD_DATADIR = `select @@datadir` no need to set the $MYSQLD_DATADIR twice > +--exec $MYSQL_BINLOG --short-form --skip-annotate-row-events $MYSQLD_DATADIR/master-bin.000002 > + > +--let $case = B: one engine has committed its transaction branch > +# Hold off after one engine has committed. > +--let $kill_server=1 > +--let $debug_sync_action = "commit_after_run_commit_ordered SIGNAL con1_ready WAIT_FOR signal_no_signal" > +--let $restart_simulate_partial_commit = 1 this variable seems to be unused > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 --debug-dbug=d,binlog_truncate_partial_commit this seems to be a rather crude way of faking a partially committed transaction. better to crash after the first engine has committed, that'd be much more natural. > +--let $test_outcome= 2 rows should be present in both tables; no binlog truncation; one extra binlog file compare with A; number of binlogs at reconnect - 4 > + --source binlog_truncate_multi_engine.inc > + > +--let $case = C: both engines have committed its transaction branch you didn't do --let $debug_sync_action = "reset", so the old value is used. Intentional? > +# Hold off after both engines have committed. The server is shut down. > +--let $kill_server=0 > +--let $restart_parameters = --rpl-semi-sync-slave-enabled=1 > +--let $test_outcome= 2 rows should be present in both tables; no binlog truncation; the same # of binlog files as in B; number of binlogs at reconnect - 4 > + --source binlog_truncate_multi_engine.inc > + > + > + > +DROP TABLE t1, t2; > +--replace_result $old_max_binlog_size VALUE_AT_START > +--eval SET @@global.max_binlog_size = $old_max_binlog_size better use evalp instead of eval, then you won't need replace_result but really I think you don't need to save/restore max_binlog_size at all, because you restart the server anyway. > + > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc > new file mode 100644 > index 00000000000..c260a7987e2 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_engine.inc > @@ -0,0 +1,64 @@ > +# > +# Loop body of binlog_truncate_multi_engine.test > +# Parameters: > +# $debug_sync_action describes debug-sync actions > +# $kill_server 1 when to crash, 0 for regular restart > +# $restart_parameters the caller may simulate partial commit at recovery > +# $test_outcome summary of extected results > +# $MYSQLD_DATADIR > + > +--echo # > +--echo # > +--echo # Case $case > +--echo # > +RESET MASTER; > +FLUSH LOGS; > +SET GLOBAL max_binlog_size= 4096; > + > +connect(con1,localhost,root,,); > +#--connection con1 > +--echo List of binary logs before rotation > +--source include/show_binary_logs.inc > +INSERT INTO t1 VALUES (1, REPEAT("x", 1)); > +INSERT INTO t2 VALUES (1, REPEAT("x", 1)); > +BEGIN; > + INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); > + INSERT INTO t2 VALUES (2, REPEAT("x", 4100)); > + > +--eval SET DEBUG_SYNC= $debug_sync_action > +send COMMIT; > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > +--echo List of binary logs after rotation > +--source include/show_binary_logs.inc > + > +--echo # restart the server with $restart_parameters > +if ($kill_server) > +{ > + --echo # the server is crashed > + --source include/kill_mysqld.inc > + --source include/start_mysqld.inc > +} > +if (!$kill_server) > +{ > + --echo # the server is restarted > + --source include/restart_mysqld.inc it'd be simpler not to use $kill_server at all. And instead write let $shutdown_timeout=0; instead of $kill_server=1; and let $shutdown_timeout=; instead of $kill_server=0; and above you won't need two if's, you can just do source include/restart_mysqld.inc; > +} > + > +--connection default > +--echo # > +--echo # *** Summary: $test_outcome: > +--echo # > +SELECT COUNT(*) FROM t1; > +SELECT COUNT(*) FROM t2; > +SELECT @@GLOBAL.gtid_binlog_state; > +SELECT @@GLOBAL.gtid_binlog_pos; > +--echo List of binary logs at the end of the tests > +--source include/show_binary_logs.inc > +--echo # *** > +# cleanup > +DELETE FROM t1; > +DELETE FROM t2; > +--disconnect con1 > +--echo # > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test > new file mode 100644 > index 00000000000..231e90dbdc9 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test > @@ -0,0 +1,105 @@ > +# ==== Purpose ==== > +# > +# Test verifies truncation of multiple binary logs. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 1 - Set max_binlog_size= 4096, to help a series of inserts into a > +# transaction table 'ti' get binlog rotated so many time while the > +# transactions won't be committed, being stopped at > +# a prior to commit debug_sync point > +# 2 - kill and restart the server as semisync slave successfully to > +# end with an expected first binlog and the gtid state. > +# > +# ==== References ==== > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > +--source include/have_innodb.inc > +--source include/have_log_bin.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > + > +--let $old_max_binlog_size= `select @@global.max_binlog_size` > +SET @@global.max_binlog_size= 4096; > + > +RESET MASTER; > +FLUSH LOGS; > +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE tm (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=MyISAM; > + > +connect(master1,localhost,root,,); > +--echo "List of binary logs before rotation" > +--source include/show_binary_logs.inc > + > +# Some load to either non- and transactional egines > +# that should not affect the following recovery: > +INSERT INTO ti VALUES(1,"I am gonna survive"); > +INSERT INTO tm VALUES(1,"me too!"); > + > +# hold on near engine commit > +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL master1_ready WAIT_FOR con1_go"; > +--send_eval INSERT INTO ti VALUES (2, REPEAT("x", 4100)) > + > +connect(master2,localhost,root,,); > +# The 2nd trx for recovery, it does not rotate binlog > +SET DEBUG_SYNC= "now WAIT_FOR master1_ready"; > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master2_ready WAIT_FOR master2_go"; > +--send_eval INSERT INTO ti VALUES (3, "not gonna survive") > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +--echo "List of binary logs before crash" > +--source include/show_binary_logs.inc > +--echo # The gtid binlog state prior the crash will be truncated at the end of the test > +SELECT @@global.gtid_binlog_state; > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait > +EOF > + > +--source include/kill_mysqld.inc > +--source include/wait_until_disconnected.inc > + > +# > +# Server restart > +# > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart: --rpl-semi-sync-slave-enabled=1 > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# Check error log for a successful truncate message. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; > +} > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_RANGE=-50000 > +--let SEARCH_PATTERN=truncated binlog file:.*master.*000002 > +--replace_regex /FOUND [0-9]+/FOUND #/ > +--source include/search_pattern_in_file.inc > + > + > +--echo "One record should be present in table" > +SELECT count(*) FROM ti; > + > +--echo # The truncated gtid binlog state > +SELECT @@global.gtid_binlog_state; > +SELECT @@global.gtid_binlog_pos; > + > +--echo # Cleanup > +--replace_result $old_max_binlog_size VALUE_AT_START > +--eval SET @@global.max_binlog_size = $old_max_binlog_size > +DROP TABLE ti; > + many of my comments above apply to this test too > +--echo # End of the tests > diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test > new file mode 100644 > index 00000000000..6aba2935fde > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test > @@ -0,0 +1,142 @@ > +# ==== Purpose ==== > +# The test verifies attempt to recover by the semisync slave server whose > +# binlog is unsafe for truncation. > +# > +# ==== Implementation ==== > +# 2 binlog files are created with the 1st one destined to be the binlog > +# checkpoint file for recovery. > +# The final group of events is replication unsafe (myisam INSERT). > +# Therefore the semisync slave recovery may not. > +# > +# Steps: > +# 0 - Set max_binlog_size= 4096, to help an insert into a > +# transaction table 'ti' get binlog rotated while the > +# transaction won't be committed, being stopped at > +# a prior to commit debug_sync point > +# 1 - insert into a non-transactional 'tm' table completes with > +# binary logging as well > +# 2 - kill and attempt to restart the server as semisync slave that > +# must produce an expected unsafe-to-recover error > +# 3 - complete the test with a normal restart that successfully finds and > +# commits the transaction in doubt. > +# > +# ==== References ==== > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > + > +--source include/have_innodb.inc > +--source include/have_log_bin.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > + > +--let $old_max_binlog_size= `select @@global.max_binlog_size` > +SET @@global.max_binlog_size= 4096; > + > +call mtr.add_suppression("Table '.*tm' is marked as crashed and should be repaired"); > +call mtr.add_suppression("Got an error from unknown thread"); > +call mtr.add_suppression("Checking table: '.*tm'"); > +call mtr.add_suppression("Recovering table: '.*tm'"); > +call mtr.add_suppression("Cannot trim the binary log to file"); > +call mtr.add_suppression("Crash recovery failed"); > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +call mtr.add_suppression("Found 1 prepared transactions"); > +call mtr.add_suppression("mysqld: Table.*tm.*is marked as crashed"); > +call mtr.add_suppression("Checking table.*tm"); > + > +RESET MASTER; > +FLUSH LOGS; > +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +CREATE TABLE tm (f INT) ENGINE=MYISAM; > + > +--let $row_count = 5 > +--let $i = 3 > +--disable_query_log > +while ($i) > +{ > + --eval INSERT INTO ti VALUES ($i, REPEAT("x", 1)) > +--dec $i > +} > +--enable_query_log > +INSERT INTO tm VALUES(1); > + > +connect(master1,localhost,root,,); > +connect(master2,localhost,root,,); > +connect(master3,localhost,root,,); > + > +--connection master1 > + > +# The 1st trx binlogs, rotate binlog and hold on before committing at engine > +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL master1_ready WAIT_FOR master1_go"; you don't wait for master1_ready anywhere. intentional? > +--send_eval INSERT INTO ti VALUES ($row_count - 1, REPEAT("x", 4100)) if you plan to insert rows 1,2,3,4,5 then your $i=3 and $row_count=5 are not independent, you should use $i=$row_count-2. (probably $i=`select $row_count-2`) or don't pretent this test is tunable and use literals instead of variables. > + > +--connection master2 > + > +# The 2nd trx for recovery, it does not rotate binlog > +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL master2_ready WAIT_FOR master2_go"; > +--send_eval INSERT INTO ti VALUES ($row_count, REPEAT("x", 1)) > + > +--connection master3 > +SET DEBUG_SYNC= "now WAIT_FOR master2_ready"; > +SET DEBUG_SYNC= "commit_before_get_LOCK_after_binlog_sync SIGNAL master3_ready"; > +--send INSERT INTO tm VALUES (2) > + > +--connection default > +SET DEBUG_SYNC= "now WAIT_FOR master3_ready"; > +--echo # The gtid binlog state prior the crash must be restored at the end of the testSELECT @@global.gtid_binlog_state; eh? Was this SELECT supposed to be on a separate line? > + > +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +wait > +EOF > + > +SELECT @@global.gtid_binlog_state; > +--source include/kill_mysqld.inc > +--source include/wait_until_disconnected.inc > + > +# > +# Server restarts > +# > +--echo # Failed restart as the semisync slave > +--error 1 > +--exec $MYSQLD_LAST_CMD --rpl-semi-sync-slave-enabled=1 >> $MYSQLTEST_VARDIR/log/mysqld.1.err 2>&1 > + > +--echo # Normal restart > +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect > +restart: Okay. *this* is a reasonable use of wait/restart in the expect file. In other cases, I think, you can just restart, no need to wait, as you don't do anything while the server is down. still, here you don't need to write 'wait' to the expect file and don't need to wait_until_disconnected (kill_mysqld is doing both) > +EOF > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# Check error log for correct messages. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; > +} > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_RANGE=-50000 > +--let SEARCH_PATTERN=Cannot trim the binary log to file > +--replace_regex /FOUND [0-9]+/FOUND #/ > +--source include/search_pattern_in_file.inc > + > +--echo # Proof that the in-doubt transactions are recovered by the 2nd normal server restart > +--eval SELECT COUNT(*) = $row_count as 'True' FROM ti > +# myisam table may require repair (which is not tested here) > +--disable_warnings > +SELECT COUNT(*) <= 1 FROM tm; > +--enable_warnings > + > +--echo # The gtid binlog state prior the crash is restored now > +SELECT @@GLOBAL.gtid_binlog_state; > +SELECT @@GLOBAL.gtid_binlog_pos; > + > +--echo # Cleanup > +--replace_result $old_max_binlog_size VALUE_AT_START > +--eval SET @@global.max_binlog_size = $old_max_binlog_size see comments elsewhere about $old_max_binlog_size, -50000, $log_error_, etc > +DROP TABLE ti, tm; > +--echo End of test > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf > new file mode 100644 > index 00000000000..3518eb95b67 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.cnf > @@ -0,0 +1,13 @@ > +!include suite/rpl/rpl_1slave_base.cnf > +!include include/default_client.cnf > + > + > +[mysqld.1] > +log-slave-updates > +gtid-strict-mode=1 > +max_binlog_size= 4096 > + > +[mysqld.2] > +log-slave-updates > +gtid-strict-mode=1 > +max_binlog_size= 4096 why not an .opt file? why max_binlog_size here and not at run-time like in other tests? > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > new file mode 100644 > index 00000000000..972aaf2c8b4 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_fail_over.test > @@ -0,0 +1,144 @@ > +# ==== Purpose ==== > +# > +# Test verifies replication failover scenario. > +# > +# ==== Implementation ==== > +# > +# Steps: > +# 0 - Having two servers 1 and 2 enable semi-sync replication with > +# with the master wait 'after_sync'. > +# 1 - Insert a row. While inserting second row simulate > +# a server crash at once the transaction is written to binlog, flushed > +# and synced but the binlog position is not updated. > +# 2 - Post crash-recovery on the old master execute there CHANGE MASTER > +# TO command to connect to server id 2. > +# 3 - The old master new slave server 1 must connect to the new > +# master server 2. > +# 4 - repeat the above to crash the new master and restore in role the old one > +# > +# ==== References ==== > +# > +# MDEV-21117: recovery for --rpl-semi-sync-slave-enabled server > + > + > +--source include/have_innodb.inc > +--source include/have_log_bin.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > +--source include/have_binlog_format_row.inc > +--let $rpl_topology=1->2 > +--source include/rpl_init.inc isn't this a normal master-slave.inc topology? > + > +--connection server_2 > +--source include/stop_slave.inc > + > +--connection server_1 > +RESET MASTER; > + > +--connection server_2 > +RESET MASTER; > +set @@global.rpl_semi_sync_slave_enabled = 1; > +set @@global.gtid_slave_pos = ""; > +CHANGE MASTER TO master_use_gtid= slave_pos; > +--source include/start_slave.inc > + > + > +--connection server_1 > +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; why? > +set @@global.rpl_semi_sync_master_enabled = 1; > +set @@global.rpl_semi_sync_master_wait_point=AFTER_SYNC; > + > +call mtr.add_suppression("Can.t init tc log"); > +call mtr.add_suppression("Aborting"); > +call mtr.add_suppression("1 client is using or hasn.t closed the table properly"); > +call mtr.add_suppression("Table './mtr/test_suppressions' is marked as crashed and should be repaired"); > + > +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; > +INSERT INTO t1 VALUES (1, 'dummy1'); > + > +# > +# CRASH the original master, and FAILOVER to the new > +# > + > +# value 1 for server id 1 -> 2 failover > +--let $failover_to_slave=1 > +--let $query_to_crash= INSERT INTO t1 VALUES (2, REPEAT("x", 4100)) > +--let $log_search_pattern=truncated binlog file:.*master.*000001 > +--source rpl_semi_sync_crash.inc > + > +--connection server_2 > +--let $rows_so_far=3 > +--eval INSERT INTO t1 VALUES ($rows_so_far, 'dummy3') > +--save_master_pos > +--echo # The gtid state on current master must be equal to ... > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_1 > +--sync_with_master > +--eval SELECT COUNT(*) = $rows_so_far as 'true' FROM t1 > +--echo # ... the gtid states on the slave: > +SHOW VARIABLES LIKE 'gtid_slave_pos'; > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_2 > +# > +# CRASH the new master and FAILOVER back to the original > +# > + > +# value 0 for the reverse server id 2 -> 1 failover > +--let $failover_to_slave=0 > +--let $query_to_crash = INSERT INTO t1 VALUES (4, REPEAT("x", 4100)) > +--let $query2_to_crash= INSERT INTO t1 VALUES (5, REPEAT("x", 4100)) > +--let $log_search_pattern=truncated binlog file:.*slave.*000001 > +--source rpl_semi_sync_crash.inc > + > +--connection server_1 > +--let $rows_so_far=6 > +--eval INSERT INTO t1 VALUES ($rows_so_far, 'Done') > +--save_master_pos > +--echo # The gtid state on current master must be equal to ... > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > +--connection server_2 > +--sync_with_master > +--eval SELECT COUNT(*) = $rows_so_far as 'true' FROM t1 > +--echo # ... the gtid states on the slave: > +SHOW VARIABLES LIKE 'gtid_slave_pos'; > +SHOW VARIABLES LIKE 'gtid_binlog_pos'; > + > + > +--let $diff_tables=server_1:t1, server_2:t1 > +--source include/diff_tables.inc > + > +# > +--echo # Cleanup > +# > +--connection server_1 > +DROP TABLE t1; > +--save_master_pos > + > +--connection server_2 > +--sync_with_master > +--source include/stop_slave.inc > + > +--connection server_1 > +set @@global.rpl_semi_sync_master_enabled = 0; > +set @@global.rpl_semi_sync_slave_enabled = 0; > +set @@global.rpl_semi_sync_master_wait_point=default; > +RESET SLAVE; > +RESET MASTER; > + > +--connection server_2 > +set @@global.rpl_semi_sync_master_enabled = 0; > +set @@global.rpl_semi_sync_slave_enabled = 0; > +set @@global.rpl_semi_sync_master_wait_point=default; > + > +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 > +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1, master_user='root', master_use_gtid=no; evalp > +--source include/start_slave.inc > + > +connection default; > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_crash.inc b/mysql-test/suite/rpl/t/rpl_semi_sync_crash.inc > new file mode 100644 > index 00000000000..4289df9155f > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_crash.inc > @@ -0,0 +1,87 @@ > +if ($failover_to_slave) > +{ > + --let $server_to_crash=1 > + --let $server_to_promote=2 > + --let $new_master_port=$SERVER_MYPORT_2 > + --let $client_port=$SERVER_MYPORT_1 > + > + --connect (conn_client,127.0.0.1,root,,test,$SERVER_MYPORT_1,) > +} > +if (!$failover_to_slave) > +{ > + --let $server_to_crash=2 > + --let $server_to_promote=1 > + --let $new_master_port=$SERVER_MYPORT_1 > + --let $client_port=$SERVER_MYPORT_2 > + > + --connect (conn_client,127.0.0.1,root,,test,$SERVER_MYPORT_2,) > +} > + > + > +# Hold insert after write to binlog and before "run_commit_ordered" in engine > + > +SET DEBUG_SYNC= "commit_after_release_LOCK_after_binlog_sync SIGNAL con1_ready WAIT_FOR con1_go"; > +--send_eval $query_to_crash > + > +# complicate recovery with an extra binlog file > +if (!$failover_to_slave) > +{ > + --connect (conn_client_2,127.0.0.1,root,,test,$SERVER_MYPORT_2,) > + # use the same signal with $query_to_crash > + SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > + SET DEBUG_SYNC= "commit_after_release_LOCK_lock SIGNAL con1_ready WAIT_FOR con2_go"; both signal con1_ready? which one will you be waiting below then? > + --send_eval $query2_to_crash > +} > + > +--connection server_$server_to_crash > +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; > +--source include/kill_mysqld.inc > +--source include/wait_until_disconnected.inc > + > +--connection server_$server_to_promote > +--error 2003 > +--source include/stop_slave.inc > +SELECT @@GLOBAL.gtid_current_pos; > + > +--let $_expect_file_name=$MYSQLTEST_VARDIR/tmp/mysqld.$server_to_crash.expect wasn't $_expect_file_name already set above? > +--let $restart_parameters=--rpl-semi-sync-slave-enabled=1 > +--let $allow_rpl_inited=1 > +--source include/start_mysqld.inc > + > +--connection server_$server_to_crash > +--enable_reconnect > +--source include/wait_until_connected_again.inc > + > +# Check error log for correct messages. > +let $log_error_= `SELECT @@GLOBAL.log_error`; > +if(!$log_error_) > +{ > + # MySQL Server on windows is started with --console and thus > + # does not know the location of its .err log, use default location > + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.$server_to_crash.err; > +} > +--let SEARCH_FILE=$log_error_ > +--let SEARCH_RANGE=-50000 > +--let SEARCH_PATTERN=$log_search_pattern > +--source include/search_pattern_in_file.inc > + > +--disconnect conn_client > + > +# > +# FAIL OVER now to new master > +# > +--connection server_$server_to_promote > +set global rpl_semi_sync_master_enabled = 1; > +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; > + > +--connection server_$server_to_crash > +--let $master_port=$SERVER_MYPORT_2 > +if (`select $server_to_crash = 2`) > +{ > + --let $master_port=$SERVER_MYPORT_1 > +} > +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 $SERVER_MYPORT_2 SERVER_MYPORT_2 > +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$new_master_port, master_user='root', master_use_gtid=SLAVE_POS; > +set global rpl_semi_sync_slave_enabled = 1; > +set @@global.gtid_slave_pos=@@global.gtid_binlog_pos; > +--source include/start_slave.inc see comments about -50000, $log_error_, wait_until_disconnected, evalp, etc > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 140394fefc1..87fa88d4c89 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -4632,7 +4632,7 @@ class THD :public Statement, > LF_PINS *tdc_hash_pins; > LF_PINS *xid_hash_pins; > bool fix_xid_hash_pins(); > - > + bool is_1pc_ro_trans; I don't like how it looks, it's in THD (for every single connection and more) and it's only used in some exotic cases. It's set in two places around ha_commit_one_phase(), when it would be better to set it inside ha_commit_one_phase(), in one place only. But I cannot suggest how to get rid of it yet as I don't understand what it's for, I've asked a question about it below, in Gtid_log_event::Gtid_log_event > /* Members related to temporary tables. */ > public: > /* Opened table states. */ > diff --git a/sql/handler.h b/sql/handler.h > index 2a346e8d9d1..bd39f46bf1f 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -873,6 +873,14 @@ typedef struct xid_t XID; > /* The 'buf' has to have space for at least SQL_XIDSIZE bytes. */ > uint get_sql_xid(XID *xid, char *buf); > > +/* struct for heuristic binlog truncate recovery */ it's not "heuristic binlog truncate recovery" anymore, please don't call it that, it's confusing > +struct xid_recovery_member > +{ > + my_xid xid; > + uint in_engine_prepare; // number of engines that have xid prepared > + bool decided_to_commit; > +}; > + > /* for recover() handlerton call */ > #define MIN_XID_LIST_SIZE 128 > #define MAX_XID_LIST_SIZE (1024*128) > diff --git a/sql/slave.cc b/sql/slave.cc > index 28e08e32346..9b3f73e5341 100644 > --- a/sql/slave.cc > +++ b/sql/slave.cc > @@ -6944,7 +6945,9 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) > } > else > if ((s_id == global_system_variables.server_id && > - !mi->rli.replicate_same_server_id) || > + (!mi->rli.replicate_same_server_id && > + !(semisync_recovery= (rpl_semi_sync_slave_enabled && > + mi->using_gtid != Master_info::USE_GTID_NO)))) || How can "semisync recovery" code path reach queue_event()? > event_that_should_be_ignored(buf) || > /* > the following conjunction deals with IGNORE_SERVER_IDS, if set > diff --git a/sql/log_event.h b/sql/log_event.h > index 8a342cb5cd3..1588ab85104 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -482,6 +482,16 @@ class String; > */ > #define LOG_EVENT_IGNORABLE_F 0x80 > > +/** > + @def LOG_EVENT_ACCEPT_OWN_F > + > + Flag sets by the gtid-mode connected semisync slave for > + the same server_id ("own") events which the slave must not have > + in its state. Typically such events were never committed by > + their originator (this server) and discared at its crash recovery sorry, I failed to understand that > +*/ > +#define LOG_EVENT_ACCEPT_OWN_F 0x4000 > + > /** > @def LOG_EVENT_SKIP_REPLICATION_F > > diff --git a/sql/log_event.cc b/sql/log_event.cc > index e344fc8894f..a593db10d16 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -7918,6 +7924,7 @@ Gtid_log_event::Gtid_log_event(const char *buf, uint event_len, > domain_id= uint4korr(buf); > buf+= 4; > flags2= *buf; > + ++buf; I'd written flags2= *buf++; :) > if (flags2 & FL_GROUP_COMMIT_ID) > { > if (event_len < (uint)header_size + GTID_HEADER_LEN + 2) > @@ -7925,9 +7932,31 @@ Gtid_log_event::Gtid_log_event(const char *buf, uint event_len, > seq_no= 0; // So is_valid() returns false > return; > } > - ++buf; > commit_id= uint8korr(buf); > + buf+= 8; > + } > + /* the extra flags check and actions */ > + if (static_cast<uint>(buf - buf_0) < event_len) > + { > + flags_extra= *buf; > + ++buf; > + /* extra flags presence is identifed by non-zero byte value at this point */ "extra engines" > + if (flags_extra & FL_EXTRA_MULTI_ENGINE) > + { safety, check that buf+4-buf_0 <= event_len > + extra_engines= uint4korr(buf); > + buf += 4; four bytes for the number of engines participating in a transaction is very generous... and optimistic. I'd say one byte is more than enough. > + > + DBUG_ASSERT(extra_engines > 0); > + } > } > + /* > + the strict '<' part of the assert corresponds to extra zero-padded > + trailing bytes, > + */ > + DBUG_ASSERT(static_cast<uint>(buf - buf_0) <= event_len); > + /* and the last of them is tested. */ > + DBUG_ASSERT(static_cast<uint>(buf - buf_0) == event_len || > + buf_0[event_len - 1] == 0); > } > > > @@ -7958,6 +7990,22 @@ Gtid_log_event::Gtid_log_event(THD *thd_arg, uint64 seq_no_arg, > /* Preserve any DDL or WAITED flag in the slave's binlog. */ > if (thd_arg->rgi_slave) > flags2|= (thd_arg->rgi_slave->gtid_ev_flags2 & (FL_DDL|FL_WAITED)); > + /* count non-zero extra recoverable engines; total = extra + 1 */ > + if (is_transactional) > + { > + if (has_xid) > + { > + extra_engines= > + max<uint>(1, ha_count_rw(thd, thd_arg->in_multi_stmt_transaction_mode())) 1. a bit confusing to use both thd and thd_arg in the same line 2. max() is strange. it means the result is 1, when ha_count_rw() is 0. can ha_count_rw() here even be 0? 3. also it's somewhat an unfounded assumption that only r/w engines will have the transaction prepared. But we can make it a fact, if we won't prepare r/o engines at all (in ha_prepare). > + - 1; > + } > + else if (unlikely(thd_arg->is_1pc_ro_trans)) > + { > + extra_engines= UINT_MAX; // neither extra nor base engine why is that? > + } > + if (extra_engines > 0) > + flags_extra|= FL_EXTRA_MULTI_ENGINE; > + } > } > > > diff --git a/sql/handler.cc b/sql/handler.cc > index 6792e80b8fe..247ab7267bf 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1245,6 +1245,24 @@ int ha_prepare(THD *thd) > DBUG_RETURN(error); > } > > +/* > + Returns counted number of > + read-write recoverable transaction participants. > +*/ > +uint ha_count_rw(THD *thd, bool all) the name doesn't match the implementation, please, rename > +{ > + unsigned rw_ha_count= 0; > + THD_TRANS *trans=all ? &thd->transaction.all : &thd->transaction.stmt; > + > + for (Ha_trx_info * ha_info= trans->ha_list; ha_info; > + ha_info= ha_info->next()) > + { > + if (ha_info->is_trx_read_write() && ha_info->ht()->recover) > + ++rw_ha_count; > + } > + return rw_ha_count; > +} > + > /** > Check if we can skip the two-phase commit. > > @@ -1960,8 +1982,122 @@ struct xarecover_st > XID *list; > HASH *commit_list; > bool dry_run; > + MEM_ROOT *mem_root; > + bool error; > }; > > +/** > + Inserts a new hash member. > + > + returns a successfully created and inserted @c xid_recovery_member > + into hash @c hash_arg, > + or NULL. > +*/ > +static xid_recovery_member* > +xid_member_insert(HASH *hash_arg, my_xid xid_arg, MEM_ROOT *ptr_mem_root) > +{ > + xid_recovery_member *member= (xid_recovery_member*) > + alloc_root(ptr_mem_root, sizeof(xid_recovery_member)); > + if (!member) > + return NULL; > + > + member->xid= xid_arg; > + member->in_engine_prepare= 1; > + member->decided_to_commit= false; > + > + return my_hash_insert(hash_arg, (uchar*) member) ? NULL : member; > +} > + > +/* > + Inserts a new or updates an existing hash member to increment > + the member's prepare counter. > + > + returns false on success, > + true otherwise. > +*/ > +static bool xid_member_replace(HASH *hash_arg, my_xid xid_arg, > + MEM_ROOT *ptr_mem_root) > +{ > + xid_recovery_member* member; > + if ((member= (xid_recovery_member *) > + my_hash_search(hash_arg, (uchar *)& xid_arg, sizeof(xid_arg)))) > + member->in_engine_prepare++; > + else > + member= xid_member_insert(hash_arg, xid_arg, ptr_mem_root); > + > + return member == NULL; > +} > + > +/* > + Hash iterate function to complete with commit or rollback as decided > + (typically at binlog recovery processing) in member->in_engine_prepare. > +*/ > +static my_bool xarecover_do_commit_or_rollback(void *member_arg, > + void *hton_arg) > +{ > + xid_recovery_member *member= (xid_recovery_member*) member_arg; > + handlerton *hton= (handlerton*) hton_arg; > + xid_t x; > + my_bool rc; > + > + x.set(member->xid); > + rc= member->decided_to_commit ? hton->commit_by_xid(hton, &x) : > + hton->rollback_by_xid(hton, &x); > + > + DBUG_ASSERT(rc || member->in_engine_prepare > 0); > + > + if (!rc) > + { I don't think you can trust rc on that. if it's non-zero, it's an error all right. but it's a bit of a stretch to presume that nonexisting xid is always an error. also commit_by_xid returns int, not my_bool. > + member->in_engine_prepare--; > + if (global_system_variables.log_warnings > 2) > + sql_print_warning("%s transaction with xid %llu", > + member->decided_to_commit ? > + "Committed" : "Rolled back", (ulonglong) member->xid); > + } > + > + return false; > +} > + > +static my_bool xarecover_do_count_in_prepare(void *member_arg, > + void *ptr_count) > +{ > + xid_recovery_member *member= (xid_recovery_member*) member_arg; > + if (member->in_engine_prepare) > + { > + *(uint*) ptr_count += member->in_engine_prepare; This is a rather weird semantics. it's kind of a number of transactions times number of engines. if one transaction wasn't committed in two engines and another transaction wasn't rolled back in one engine, the counter will be 3. how are you going to explain this to users? > + if (global_system_variables.log_warnings > 2) > + sql_print_warning("Found prepared transaction with xid %llu", > + (ulonglong) member->xid); > + } > + > + return false; > +} > + > +static my_bool xarecover_binlog_truncate_handlerton(THD *unused, > + plugin_ref plugin, > + void *arg) > +{ > + handlerton *hton= plugin_hton(plugin); > + > + if (hton->state == SHOW_OPTION_YES && hton->recover) > + { > + my_hash_iterate((HASH*) arg, xarecover_do_commit_or_rollback, hton); Why is the function called xarecover_binlog_truncate_handlerton if it doesn't truncate anything? > + } > + > + return FALSE; > +} > + > +uint ha_recover_complete(HASH *commit_list) > +{ > + uint count= 0; > + > + plugin_foreach(NULL, xarecover_binlog_truncate_handlerton, > + MYSQL_STORAGE_ENGINE_PLUGIN, commit_list); > + my_hash_iterate(commit_list, xarecover_do_count_in_prepare, &count); > + > + return count; > +} > + > static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > void *arg) > { > @@ -1969,6 +2105,9 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > struct xarecover_st *info= (struct xarecover_st *) arg; > int got; > > + if (info->error) > + return TRUE; plugin_foreach() aborts as soon as the callback returns true. so, remove info->error, and return TRUE from xarecover_handlerton on error instead. > + > if (hton->state == SHOW_OPTION_YES && hton->recover) > { > while ((got= hton->recover(hton, info->list, info->len)) > 0 ) > @@ -1988,7 +2127,7 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > } > #endif /* WITH_WSREP */ > > - for (int i=0; i < got; i ++) > + for (int i=0; i < got && !info->error; i ++) eh? how can info->error ever be true above? you break out of the loop when setting info->error= true, > { > my_xid x= IF_WSREP(WSREP_ON && wsrep_is_wsrep_xid(&info->list[i]) ? > wsrep_xid_seqno(info->list[i]) : > @@ -2013,7 +2152,20 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, > info->found_my_xids++; > continue; > } > - // recovery mode > + > + /* > + Regular and semisync slave server recovery only collects > + xids to make decisions on them later by the caller. > + */ > + if (info->mem_root) > + { > + if (xid_member_replace(info->commit_list, x, info->mem_root)) > + { > + info->error= true; > + sql_print_error("Error in memory allocation at xarecover_handlerton"); > + break; > + } > + } else add into the else branch something like /* this branch is only for the legacy TC_LOG_MMAP */ compile_time_assert(sizeof(TC_LOG_MMAP) > 1 ); > if (IF_WSREP((wsrep_emulate_bin_log && > wsrep_is_wsrep_xid(info->list + i) && > x <= wsrep_limit), false) || > diff --git a/sql/log.cc b/sql/log.cc > index 8073f09ab88..a9808ed8823 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -7977,6 +7982,7 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) > #endif > } > > + DEBUG_SYNC(leader->thd, "commit_before_update_end_pos"); this isn't used in any tests > /* > update binlog_end_pos so it can be read by dump thread > Note: must be _after_ the RUN_HOOK(after_flush) or else > @@ -9609,6 +9618,180 @@ int TC_LOG::using_heuristic_recover() > /****** transaction coordinator log for 2pc - binlog() based solution ******/ > #define TC_LOG_BINLOG MYSQL_BIN_LOG > > +/** > + Truncates the current binlog to specified position. Removes the rest of binlogs > + which are present after this binlog file. > + > + @param truncate_file Holds the binlog name to be truncated > + @param truncate_pos Position within binlog from where it needs to > + truncated. > + > + @retval true ok > + @retval false error > + > +*/ > +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *file_name, > + my_off_t pos, > + rpl_gtid *ptr_gtid, > + enum_binlog_checksum_alg cs_alg) > +{ > + int error= 0; > +#ifdef HAVE_REPLICATION > + LOG_INFO log_info; > + THD *thd= current_thd; > + my_off_t index_file_offset= 0; > + File file= -1; > + IO_CACHE cache; > + MY_STAT s; > + > + if ((error= find_log_pos(&log_info, file_name, 1))) > + { > + sql_print_error("Failed to locate binary log file:%s." > + "Error:%d", file_name, error); > + goto end; > + } > + > + while (!(error= find_next_log(&log_info, 1))) > + { > + if (!index_file_offset) > + { > + index_file_offset= log_info.index_file_start_offset; > + if ((error= open_purge_index_file(TRUE))) > + { > + sql_print_error("Failed to open purge index " > + "file:%s. Error:%d", purge_index_file_name, error); > + goto end; > + } > + } > + if ((error= register_purge_index_entry(log_info.log_file_name))) > + { > + sql_print_error("Failed to copy %s to purge index" > + " file. Error:%d", log_info.log_file_name, error); > + goto end; > + } > + } > + > + if (error != LOG_INFO_EOF) > + { > + sql_print_error("Failed to find the next binlog to " > + "add to purge index register. Error:%d", error); > + goto end; > + } > + > + if (is_inited_purge_index_file()) > + { > + if (!index_file_offset) > + index_file_offset= log_info.index_file_start_offset; > + > + if ((error= sync_purge_index_file())) > + { > + sql_print_error("Failed to flush purge index " > + "file. Error:%d", error); > + goto end; > + } > + > + // Trim index file > + if ((error= > + mysql_file_chsize(index_file.file, index_file_offset, '\n', > + MYF(MY_WME))) || > + (error= > + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Failed to trim binlog index " > + "file:%s to offset:%llu. Error:%d", index_file_name, > + index_file_offset, error); > + goto end; > + } > + > + /* Reset data in old index cache */ > + if ((error= reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1))) > + { > + sql_print_error("Failed to reinit binlog index " > + "file. Error:%d", error); > + goto end; > + } > + > + /* Read each entry from purge_index_file and delete the file. */ > + if ((error= purge_index_entry(thd, NULL, TRUE))) > + { > + sql_print_error("Failed to process registered " > + "files that would be purged."); > + goto end; > + } > + } > + > + DBUG_ASSERT(pos); > + > + if ((file= mysql_file_open(key_file_binlog, file_name, > + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) > + { > + error= 1; > + sql_print_error("Failed to open binlog file:%s for " > + "truncation.", file_name); > + goto end; > + } > + my_stat(file_name, &s, MYF(0)); > + > + /* Change binlog file size to truncate_pos */ > + if ((error= > + mysql_file_chsize(file, pos, 0, MYF(MY_WME))) || > + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Failed to trim the " > + "binlog file:%s to size:%llu. Error:%d", > + file_name, pos, error); > + goto end; > + } > + else > + { > + char buf[21]; > + > + longlong10_to_str(ptr_gtid->seq_no, buf, 10); > + sql_print_information("Successfully truncated binlog file:%s " > + "to pos:%llu to remove transactions starting from " > + "GTID %u-%u-%s", file_name, pos, > + ptr_gtid->domain_id, ptr_gtid->server_id, buf); > + } > + if (!(error= init_io_cache(&cache, file, IO_SIZE, WRITE_CACHE, > + (my_off_t) pos, 0, MYF(MY_WME|MY_NABP)))) > + { > + /* > + Write Stop_log_event to ensure clean end point for the binary log being > + truncated. > + */ I'm not sure about it. The less you touch the corrupted binlog the better. simply truncating it would be enough, wouldn't it? > + Stop_log_event se; > + se.checksum_alg= cs_alg; > + if ((error= write_event(&se, NULL, &cache))) > + { > + sql_print_error("Failed to write stop event to " > + "binary log. Errno:%d", > + (cache.error == -1) ? my_errno : error); > + goto end; > + } > + clear_inuse_flag_when_closing(cache.file); > + if ((error= flush_io_cache(&cache)) || > + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) > + { > + sql_print_error("Faild to write stop event to " > + "binary log. Errno:%d", > + (cache.error == -1) ? my_errno : error); > + } > + } > + else > + sql_print_error("Failed to initialize binary log " > + "cache for writing stop event. Errno:%d", > + (cache.error == -1) ? my_errno : error); 1. I don't see why you need to sync after every operation. 2. you should clear inuse flag just before closing, otherwise flush_io_cache can overwrite it again if it'll happen to have it in the cache. > + > +end: > + if (file >= 0) > + { > + end_io_cache(&cache); > + mysql_file_close(file, MYF(MY_WME)); > + } > + error= error || close_purge_index_file(); > +#endif > + return error > 0; > +} > int TC_LOG_BINLOG::open(const char *opt_name) > { > int error= 1; > @@ -10040,26 +10223,515 @@ start_binlog_background_thread() > > return 0; > } > +#ifdef HAVE_REPLICATION > +class Recovery_context > +{ > +public: > + my_off_t prev_event_pos; > + rpl_gtid last_gtid; > + bool last_gtid_standalone; > + bool last_gtid_valid; > + bool last_gtid_no2pc; // true when the group does not end with Xid event > + uint last_gtid_engines; > + std::pair<uint, my_off_t> last_gtid_coord; may be a comment at the end // <binlog id, binlog offset> ? > + /* > + When true, it's semisync slave recovery mode > + rolls back transactions in doubt and wipes them off from binlog. > + The rest of declarations deal with this type of recovery. > + */ > + bool do_truncate; > + rpl_gtid binlog_unsafe_gtid, truncate_gtid; > + char binlog_truncate_file_name[FN_REFLEN] ; > + char binlog_unsafe_file_name[FN_REFLEN] ; > + /* > + When do_truncate is true, the truncate position may not be > + found in one round when recovered transactions are multi-engine > + or just on different engines. > + In the single recoverable engine case `truncate_reset_done` and > + therefore `truncate_validated` remain `false' when the last > + binlog is the binlog-checkpoint one. > + The meaning of `truncate_reset_done` is according to the following example: > + Let round = 1, Binlog contains the sequence of replication event groups: > + [g1, G2, g3] > + where `G` (in capital) stands for committed, `g` for prepared. > + g1 is first set as truncation candidate, then G2 reset it to indicate > + the actual truncation is behind (to the right of) it. > + `truncate_validated` is set to true when `binlog_truncate_pos` (as of g3) > + won't change. > + Observe last_gtid_valid is affected, so in the above example g1 > + would have to be discarded from the 1st round binlog state estimate which > + is handled by the 2nd round (see below). I failed to understand that either :( > + */ > + bool truncate_validated; // trued when the truncate position settled > + bool truncate_reset_done; // trued when the position is to reevaluate Nor that :( > + /* Flags the fact of truncate position estimation is done the 1st round */ > + bool truncate_set_in_1st; > + /* > + Monotonically indexes binlog files in the recovery list. > + When the list is "likely" singleton the value is UINT_MAX. > + Otherwise enumeration starts with zero for the first file, increments > + by one for any next file except for the last file in the list, which > + is also the initial binlog file for recovery, > + that is enumberated with UINT_MAX. > + */ > + uint id_binlog; > + enum_binlog_checksum_alg cs_alg; // for Stop_event with do_truncate rename to checksum_alg, please. "charset algorithm" looks very puzzling and "cs" almost everywhere means CHARSET_INFO. > + bool single_binlog; > + std::pair<uint, my_off_t> binlog_truncate_coord; > + std::pair<uint, my_off_t> binlog_unsafe_coord; > + > + Recovery_context(); > + bool decide_or_assess(xid_recovery_member *member, int round, > + Format_description_log_event *fdle, > + LOG_INFO *linfo, my_off_t pos); > + void process_gtid(int round, Gtid_log_event *gev, LOG_INFO *linfo); > + int next_binlog_or_round(int& round, > + const char *last_log_name, > + const char *binlog_checkpoint_name, > + LOG_INFO *linfo, MYSQL_BIN_LOG *log); > + bool is_safe() what do you mean "safe" ? > + { > + return !do_truncate ? true : > + (truncate_gtid.seq_no == 0 || // no truncate > + binlog_unsafe_coord < binlog_truncate_coord); // or unsafe is earlier > + } > + bool complete(MYSQL_BIN_LOG *log, HASH &xids); > + void update_binlog_unsafe_coord_if_needed(LOG_INFO *linfo); > + void reset_truncate_coord(my_off_t pos); > + void set_truncate_coord(LOG_INFO *linfo, int round, > + enum_binlog_checksum_alg fd_cs_alg); > +}; > comments about what every method does? > +bool Recovery_context::complete(MYSQL_BIN_LOG *log, HASH &xids) > +{ > + if (!do_truncate || is_safe()) > + { > + uint count_in_prepare= ha_recover_complete(&xids); > + if (count_in_prepare > 0 && global_system_variables.log_warnings > 2) > + { > + sql_print_warning("Could not complete %u number of transactions in " > + "engines.", count_in_prepare); > + return false; // there's later dry run ha_recover() to error out > + } > + } > + > + /* Truncation is not done when there's no transaction to roll back */ > + if (do_truncate && truncate_gtid.seq_no > 0) > + { > + if (is_safe()) > + { > + if (log->truncate_and_remove_binlogs(binlog_truncate_file_name, > + binlog_truncate_coord.second, > + &truncate_gtid, > + cs_alg)) > + { > + sql_print_error("Failed to trim the binary log to " > + "file:%s pos:%llu.", binlog_truncate_file_name, > + binlog_truncate_coord.second); > + return true; > + } > + } > + else > + { > + sql_print_error("Cannot trim the binary log to file:%s " > + "pos:%llu as unsafe statement " > + "is found at file:%s pos:%llu which is " > + "beyond the truncation position " > + "(the farest in binlog order only is reported); " "farest" doesn't look like a word > + "all transactions in doubt are left intact. ", > + binlog_truncate_file_name, binlog_truncate_coord.second, > + binlog_unsafe_file_name, binlog_unsafe_coord.second); > + return true; will this abort server startup? > + } > + } > + > + return false; > +} > + > +Recovery_context::Recovery_context() : > + prev_event_pos(0), > + last_gtid_standalone(false), last_gtid_valid(false), last_gtid_no2pc(false), > + last_gtid_engines(0), > + do_truncate(rpl_semi_sync_slave_enabled), > + truncate_validated(false), truncate_reset_done(false), > + truncate_set_in_1st(false), id_binlog(UINT_MAX), > + cs_alg(BINLOG_CHECKSUM_ALG_UNDEF), single_binlog(false) > +{ > + last_gtid_coord= std::pair<uint,my_off_t>(0,0); > + binlog_truncate_coord= std::pair<uint,my_off_t>(0,0); > + binlog_unsafe_coord= std::pair<uint,my_off_t>(0,0); > + binlog_truncate_file_name[0]= 0; > + binlog_unsafe_file_name [0]= 0; > + binlog_unsafe_gtid= truncate_gtid= rpl_gtid(); > +} > + > +/** > + Is called when a committed or to-be-committed transaction is detected. > + @c truncate_gtid is set to "nil" with its @c rpl_gtid::seq_no := 0. > + @c truncate_reset_done remembers the fact of that has been done at least > + once in the current round; > + @c binlog_truncate_coord is "suggested" to a next group start to indicate > + the actual settled value must be at most as the last suggested one. > +*/ > +void Recovery_context::reset_truncate_coord(my_off_t pos) > +{ > + DBUG_ASSERT(binlog_truncate_coord.second == 0 || > + last_gtid_coord >= binlog_truncate_coord || > + truncate_set_in_1st); > + > + binlog_truncate_coord= std::pair<uint,my_off_t>(id_binlog, pos); > + truncate_gtid= rpl_gtid(); > + truncate_reset_done= true; > +} > + > + > +/* > + Sets binlog_truncate_pos to the value of the current transaction's gtid. > + In multi-engine case that might be just an assessment to be exacted > + in the current round and confirmed in a next one. > +*/ > +void Recovery_context::set_truncate_coord(LOG_INFO *linfo, int round, > + enum_binlog_checksum_alg fd_cs_alg) > +{ > + binlog_truncate_coord= last_gtid_coord; > + strmake_buf(binlog_truncate_file_name, linfo->log_file_name); > + > + truncate_gtid= last_gtid; > + cs_alg= fd_cs_alg; > + truncate_set_in_1st= (round == 1); > +} > + > +bool Recovery_context::decide_or_assess(xid_recovery_member *member, int round, > + Format_description_log_event *fdle, > + LOG_INFO *linfo, my_off_t pos) > +{ > + if (member) > + { > + DBUG_EXECUTE_IF("binlog_truncate_partial_commit", > + if (last_gtid_engines == 2) > + { > + DBUG_ASSERT(member->in_engine_prepare > 0); > + member->in_engine_prepare= 1; > + }); > + /* > + xid in doubt are resolved as follows: > + in_engine_prepare is compared agaist binlogged info to > + yield the commit-or-rollback decision in the normal case. > + In the semisync-slave recovery the decision may be > + approximate to change in later rounds. > + */ > + if (member->in_engine_prepare > last_gtid_engines) > + { > + char buf[21]; > + longlong10_to_str(last_gtid.seq_no, buf, 10); > + sql_print_error("Error to recovery multi-engine transaction: " > + "the number of engines prepared %u exceeds the " > + "respective number %u in its GTID %u-%u-%s " > + "located at file:%s pos:%llu", > + member->in_engine_prepare, last_gtid_engines, > + last_gtid.domain_id, last_gtid.server_id, buf, > + linfo->log_file_name, last_gtid_coord.second); > + return true; > + } > + else if (member->in_engine_prepare < last_gtid_engines) > + { > + DBUG_EXECUTE_IF("binlog_truncate_partial_commit", > + member->in_engine_prepare= 2;); > + DBUG_ASSERT(member->in_engine_prepare > 0); > + /* > + This is an "unlikely" branch of two or more engines in transaction > + that is partially committed, so to complete. > + */ > + member->decided_to_commit= true; > + if (do_truncate) > + { > + /* > + A validated truncate pos may not change later. > + Any "unlikely" (two or more engines in transaction) reset > + to not-validated yet position ensues correcting early > + estimates in the following round(s). > + */ > + if (!truncate_validated) > + reset_truncate_coord(pos); > + } > + } > + else // member->in_engine_prepare == last_gtid_engines > + { > + if (!do_truncate) // "normal" recovery > + { > + member->decided_to_commit= true; > + } > + else > + { > + /* > + The first time or further estimate the truncate position > + until validation. Already set not validated yet postion > + may change only through previous reset > + unless this xid in doubt is the first in the 2nd round. > + */ > + if (!truncate_validated) > + { > + DBUG_ASSERT(round <= 2); > + /* > + Either truncate was not set or was reset, else > + it gets incremented, otherwise it may only set to an earlier > + offset only at the turn out of the 1st round. sorry, I cannot parse that :( > + */ > + DBUG_ASSERT(truncate_gtid.seq_no == 0 || > + last_gtid_coord >= binlog_truncate_coord || > + (round == 2 && truncate_set_in_1st)); > + > + last_gtid_valid= false; // may still (see gtid) flip later > + if (truncate_gtid.seq_no == 0 /* was reset or never set */ || > + (truncate_set_in_1st && round == 2 /* reevaluted at round turn */)) > + set_truncate_coord(linfo, round, fdle->checksum_alg); > + > + DBUG_ASSERT(member->decided_to_commit == false); // may flip later > + } > + else > + { > + DBUG_ASSERT(!truncate_reset_done); // the position was settled > + /* > + Correct earlier decisions of 1st and/or 2nd round to > + rollback and invalidate last_gtid in binlog state. > + */ > + if (!(member->decided_to_commit= > + last_gtid_coord < binlog_truncate_coord)) > + { > + last_gtid_valid= false; // settled > + if (truncate_gtid.seq_no == 0) > + truncate_gtid= last_gtid; > + } > + } > + } > + } > + } > + else if (do_truncate) // "0" < last_gtid_engines > + { > + /* > + Similar to the multi-engine partial commit of "then" branch above. > + The 2nd condition economizes away an extra (3rd) round in I don't think "economizes" is a word > + expected cases of first xid:s in the binlog checkpoint file > + are actually of committed transactions. So fully committed > + xid sequence is passed in this branch without any action. > + */ > + if (!truncate_validated) > + reset_truncate_coord(pos); > + } > + > + return false; > +} > + > +/* > + Is invoked when a standalone or non-2pc group is detected. > + Both are unsafe to truncate in the semisync-slave recovery so > + the maximum unsafe coordinate may be updated. > + In the non-2pc group case though, *exeptionally*, > + the no-engine group is considered safe, to be invalidated > + to not contribute to binlog state. > +*/ > +void Recovery_context::update_binlog_unsafe_coord_if_needed(LOG_INFO *linfo) > +{ > + if (!do_truncate) > + return; > + > + if (truncate_gtid.seq_no > 0 && // g1,U2, *not* G1,U2 > + last_gtid_coord > binlog_truncate_coord) > + { > + DBUG_ASSERT(binlog_truncate_coord.second > 0); > + /* > + Potentially unsafe when the truncate coordinate is not determined, > + just detected as unsafe when behind the latter. > + */ > + if (last_gtid_engines == 0) > + { > + last_gtid_valid= false; > + } > + else > + { > + binlog_unsafe_gtid= last_gtid; > + binlog_unsafe_coord= last_gtid_coord; > + strmake_buf(binlog_unsafe_file_name, linfo->log_file_name); > + } > + } > +} > + > +/* > + Assigns last_gtid and assesses the maximum (in the binlog offset term) > + unsafe gtid (group of events). > +*/ > +void Recovery_context::process_gtid(int round, Gtid_log_event *gev, > + LOG_INFO *linfo) > +{ > + last_gtid.domain_id= gev->domain_id; > + last_gtid.server_id= gev->server_id; > + last_gtid.seq_no= gev->seq_no; > + last_gtid_engines= gev->extra_engines != UINT_MAX ? > + gev->extra_engines + 1 : 0; > + last_gtid_coord= std::pair<uint,my_off_t>(id_binlog, prev_event_pos); > + if (round == 1 || do_truncate) > + { > + DBUG_ASSERT(!last_gtid_valid); > + > + last_gtid_no2pc= false; > + last_gtid_standalone= > + (gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false; > + if (do_truncate && last_gtid_standalone) > + update_binlog_unsafe_coord_if_needed(linfo); > + /* Update the binlog state with any 'valid' GTID logged after Gtid_list. */ > + last_gtid_valid= true; // may flip at Xid when falls to truncate > + } > +} > + > +/* > + At the end of processing of the current binlog compute next action. > + When round increments in the semisync-slave recovery > + truncate_validated, truncate_reset_done > + gets reset/set for the next round, > + as well as id_binlog gets reset either to zero or UINT_MAX > + when recovery deals with the only binlog file. > + > + @param[in,out] round the current round that may increment here > + @param last_log_name the recovery starting binlog file > + @param binlog_checkpoint_name > + binlog checkpoint file > + @param linfo binlog file list struct for next file > + @param log pointer to mysql_bin_log instance > + @return 0 when rounds continue, maybe the current one remains > + 1 when all rounds are done > + -1 on error > +*/ > +int Recovery_context::next_binlog_or_round(int& round, > + const char *last_log_name, > + const char *binlog_checkpoint_name, > + LOG_INFO *linfo, > + MYSQL_BIN_LOG *log) > +{ > + if (!strcmp(linfo->log_file_name, last_log_name)) > + { > + /* Either exit the loop now, or increment round */ > + DBUG_ASSERT(round <= 3); > + > + if ((round <= 2 && likely(!truncate_reset_done)) || round == 3) > + { > + // Exit now as the 1st round ends up with the binlog-checkpoint file > + // is the same as the initial binlog file, so already parsed; or > + // the 2nd round has not made own truncate_reset_done; or > + // at the end of the 3rd "truncate" round. > + DBUG_ASSERT(do_truncate || round == 1); > + // The 3rd round is done only when the 2nd trued truncate_reset_done > + // and consequently truncate_validated. > + // No instances of truncate_reset_done in first 2 rounds allows > + // for exiting now. our (unwritten) coding style says that multi-line comments should use /*...*/ > + DBUG_ASSERT(round < 3 || truncate_validated); > + > + return true; > + } > + else > + { > + DBUG_ASSERT(do_truncate); > + /* > + the last binlog file, having truncate_reset_done to indicate > + needed correction to member->decided_to_commit to reflect > + changed binlog_truncate_pos. > + */ > + truncate_reset_done= false; > + truncate_validated= true; > + rpl_global_gtid_binlog_state.reset_nolock(); > + > + if (round > 1 && > + log->find_log_pos(linfo, binlog_checkpoint_name, 1)) > + { > + sql_print_error("Binlog file '%s' not found in binlog index, needed " > + "for recovery. Aborting.", binlog_checkpoint_name); > + return -1; > + } > + id_binlog= (single_binlog= !strcmp(linfo->log_file_name, last_log_name)) ? > + UINT_MAX : 0; > + round++; > + } > + } > + else if (round == 1) > + { > + if (do_truncate) > + { > + truncate_validated= truncate_reset_done; > + rpl_global_gtid_binlog_state.reset_nolock(); > + truncate_reset_done= false; > + id_binlog= 0; > + > + DBUG_ASSERT(!single_binlog); > + } > + round++; > + } > + else > + { > + /* > + NOTE: reading other binlog's FD is necessary for finding out > + the checksum status of the respective binlog file. > + Round remains in this branch. > + */ > + if (log->find_next_log(linfo, 1)) > + { > + sql_print_error("Error reading binlog files during recovery. " > + "Aborting."); > + return -1; > + } > + if (!strcmp(linfo->log_file_name, last_log_name)) > + { > + if (do_truncate) > + { > + DBUG_ASSERT(!single_binlog); > + id_binlog= UINT_MAX; > + } > + else > + return 1; // regular recovery exit > + } > + else if (do_truncate) > + id_binlog++; > + > + DBUG_ASSERT(id_binlog <= UINT_MAX); // the assert is "practical" > + } > + > + DBUG_ASSERT(!single_binlog || > + !strcmp(linfo->log_file_name, binlog_checkpoint_name)); > + > + return 0; > +} > +#endif > > int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > IO_CACHE *first_log, > - Format_description_log_event *fdle, bool do_xa) > + Format_description_log_event *fdle_arg, bool do_xa) > { > Log_event *ev= NULL; > HASH xids; > MEM_ROOT mem_root; > char binlog_checkpoint_name[FN_REFLEN]; > bool binlog_checkpoint_found; > - bool first_round; > IO_CACHE log; > File file= -1; > const char *errmsg; > + Format_description_log_event *fdle= fdle_arg; > #ifdef HAVE_REPLICATION > - rpl_gtid last_gtid; > - bool last_gtid_standalone= false; > - bool last_gtid_valid= false; > + Recovery_context ctx; > #endif > + /* > + The for-loop variable is updated by the following rule set: > + Initially set to 1. > + After the initial binlog file is processed to identify > + the Binlog-checkpoint file it is incremented when the latter file > + is different from the initial one. Otherwise the only log has been > + fully parsed and the loop exits. > + The 2nd starts with the Binlog-checkpoint file and ends when the initial > + binlog file is reached. It is excluded from yet another processing > + in the "normal" non-semisync-slave configuration, and then the loop is > + done at this point. Otherwise in the semisync slave case it may be parsed > + over again. The 2nd round may turn to a third in "unlikely" condition of the > + semisync-slave is being recovered having multi- or different engine > + transactions in doubt. > + */ I don't understand that. Why semisync-slave case is special? > + int round; > > if (! fdle->is_valid() || > (do_xa && my_hash_init(&xids, &my_charset_bin, TC_LOG_PAGE_SIZE/3, 0, > @@ -10072,39 +10744,63 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > > fdle->flags&= ~LOG_EVENT_BINLOG_IN_USE_F; // abort on the first error > > + /* finds xids when root is not NULL */ > + if (do_xa && ha_recover(&xids, &mem_root)) > + goto err1; > + > /* > Scan the binlog for XIDs that need to be committed if still in the > prepared stage. > > Start with the latest binlog file, then continue with any other binlog > files if the last found binlog checkpoint indicates it is needed. > + > + In case the semisync slave recovery the 2nd round may include > + the initial file as well as cause a 3rd round when transactions > + with multiple engines are discovered. > + Additionally to find and decide on transactions in doubt that > + the semisync slave may need to roll back, the binlog can be truncated > + in the end to reflect the rolled back decisions. > */ > > binlog_checkpoint_found= false; > - first_round= true; > - for (;;) > + for (round= 1;;) > { > - while ((ev= Log_event::read_log_event(first_round ? first_log : &log, > + while ((ev= Log_event::read_log_event(round == 1 ? first_log : &log, > fdle, opt_master_verify_checksum)) > && ev->is_valid()) > { > enum Log_event_type typ= ev->get_type_code(); > switch (typ) > { > + case FORMAT_DESCRIPTION_EVENT: > + if (round > 1) > + { > + if (fdle != fdle_arg) > + delete fdle; > + fdle= (Format_description_log_event*) ev; why is it needed? all binlog files should have the same Format_description_log_event anyway. > + } > + break; > case XID_EVENT: > + if (do_xa) > { > - if (do_xa) > + xid_recovery_member *member= > + (xid_recovery_member*) > + my_hash_search(&xids, (uchar*) &static_cast<Xid_log_event*>(ev)->xid, > + sizeof(my_xid)); > +#ifndef HAVE_REPLICATION > { > - Xid_log_event *xev=(Xid_log_event *)ev; > - uchar *x= (uchar *) memdup_root(&mem_root, (uchar*) &xev->xid, > - sizeof(xev->xid)); > - if (!x || my_hash_insert(&xids, x)) > - goto err2; > + if (member) > + member->decided_to_commit= true; > } > - break; > +#else > + if (ctx.decide_or_assess(member, round, fdle, linfo, ev->log_pos)) > + goto err2; > +#endif > } > + break; > case BINLOG_CHECKPOINT_EVENT: > - if (first_round && do_xa) > + if (round == 1 && do_xa) > { > size_t dir_len; > Binlog_checkpoint_log_event *cev= (Binlog_checkpoint_log_event *)ev; > @@ -10124,9 +10820,18 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > } > } > break; > +#ifdef HAVE_REPLICATION > case GTID_LIST_EVENT: > - if (first_round) > + if (round == 1 || (ctx.do_truncate && > + (ctx.id_binlog == 0 || ctx.single_binlog))) > { > + /* > + Unlike the normal case, in do_truncate the initial state is > + in the first binlog file of the recovery list. > + */ don't understand that either :( > + DBUG_ASSERT(!ctx.do_truncate || !ctx.single_binlog || > + ctx.id_binlog == UINT_MAX); > + > Gtid_list_log_event *glev= (Gtid_list_log_event *)ev; > > /* Initialise the binlog state from the Gtid_list event. */ > @@ -10135,19 +10840,16 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > } > break; > > -#ifdef HAVE_REPLICATION > case GTID_EVENT: > - if (first_round) > + ctx.process_gtid(round, (Gtid_log_event *)ev, linfo); > + break; > + > + case QUERY_EVENT: > + if (((Query_log_event *)ev)->is_commit() || > + ((Query_log_event *)ev)->is_rollback()) > { > - Gtid_log_event *gev= (Gtid_log_event *)ev; > - > - /* Update the binlog state with any GTID logged after Gtid_list. */ > - last_gtid.domain_id= gev->domain_id; > - last_gtid.server_id= gev->server_id; > - last_gtid.seq_no= gev->seq_no; > - last_gtid_standalone= > - ((gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false); > - last_gtid_valid= true; > + ctx.last_gtid_no2pc= true; > + ctx.update_binlog_unsafe_coord_if_needed(linfo); what about DML on non-transactonal tables? it won't have COMMIT/ROLLBACK will it? > } > break; > #endif > @@ -10217,35 +10923,43 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, > file= -1; > } > > - if (!strcmp(linfo->log_file_name, last_log_name)) > - break; // No more files to do > +#ifdef HAVE_REPLICATION > + int rc= ctx.next_binlog_or_round(round, last_log_name, > + binlog_checkpoint_name, linfo, this); > + if (rc == -1) > + goto err2; > + else if (rc == 1) > + break; // all rounds done > +#else > + if (!strcmp(linfo->log_file_name, last_log_name)) > + break; // No more files to do indentation > +#endif > + > if ((file= open_binlog(&log, linfo->log_file_name, &errmsg)) < 0) > { > sql_print_error("%s", errmsg); > goto err2; > } > - /* > - We do not need to read the Format_description_log_event of other binlog > - files. It is not possible for a binlog checkpoint to span multiple > - binlog files written by different versions of the server. So we can use > - the first one read for reading from all binlog files. > - */ > - if (find_next_log(linfo, 1)) > - { > - sql_print_error("Error reading binlog files during recovery. Aborting."); > - goto err2; > - } > fdle->reset_crypto(); > - } > + } // end of for > > if (do_xa) > { > - if (ha_recover(&xids)) > - goto err2; > - > + if (binlog_checkpoint_found) > + { > +#ifndef HAVE_REPLICATION > + if (ha_recover_complete(&xids)) > +#else > + if (ctx.complete(this, xids)) > +#endif > + goto err2; > + } > free_root(&mem_root, MYF(0)); > my_hash_free(&xids); > } > + if (fdle != fdle_arg) > + delete fdle; > + > return 0; > > err2: Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Systems Integration and Raising of the Abstraction Level
by Alejandro Sánchez 25 Feb '21

25 Feb '21

We have highly evolved systems such as SQL, HTTP, HTML, file formats or high level programming languages such as Java or PHP that allow us to program many things with little code. Even so a lot of effort is invested in the integration of these systems. To try to reduce this problem libraries and frameworks that help in some ways are created but the integration is not complete. It is well known that most of the time when you try to create something to integrate several incompatible systems what you get in the end is to have another incompatible system :). Still I think the integration between the systems mentioned above is something very important that can mean a great step in the evolution of computing and worth a try. To explore how this integration can be I have created a framework that I have called NextTypes and that its main objective is the integration of data types. For me it is something very illogical that something as basic as a 16 bits integer receives a different name in each of the systems ("smallint", "short", "number") and can also be signed or unsigned. In any moment, due to a mistake from the programmer, the number of the programming language does not fit in the database column. Besides these names are little indicative of its characteristics, it would be clearer for example to use "int16". Whatever names are chosen the most important thing is to use in all systems the same names for types of the same characteristics. Also there is no standard system for defining composite data types of primitive types and other composite types. From an HTML form to a SQL table or from one application to another are required multiple transformations of the data. Lack of integration also lowers the level of abstraction, making it necessary to do lots of low level stuff for systems to fit. NextTypes at this moment is nothing more than another incompatible system with the others. It simply integrates them quite a bit and raises the level of abstraction. But what I would like is that the things that compose NextTypes were included in the systems it integrates. Finally I would like to list some examples of improvements in database managers, SQL, HTTP, HTML, browsers and programming languages that would help the integration and elevation of the level of abstraction. Some of these enhancements are already included in NextTypes and other frameworks. SQL --- - Custom metadata in tables and columns. - Date of creation and modification of the rows. - Date of creation and modification of the definition of the tables. - Use of table and column names in prepared statements. Example: select # from # where # = ?; - Use of arrays in prepared statements. Example: select # from article where id in (?); # = author,title ? = 10,24,45 - Standardization of ranges of valid values and resolution for date, time and datetime types in database managers an HTML time element. PostgreSQL ---------- - Facilitate access to the definition of full text search indexes with a function to parse "pg_index.indexprs" column. Other database managers ----------------------- - Allow transactional DDL, deferrable constraints and composite types. JDBC ---- - High level methods that allow queries with the execution of a single method. Example: Tuple [] tuples = query("select author,title from article where id in (?), ids); - Integration with java.time data types. HTTP - Servers -------------- - Processing of arrays of elements composed of several parameters. fields:0:type = string fields:0:name = title fields:0:parameters = 250 fields:0:not_null = true fields:1:type = string fields:1:name = author fields:1:parameters = 250 fields:1:not_null = true Another possibility is to generate in the browser arrays of JSON objects from the forms. "fields": [ { "type": "string", "name": "title", "parameters": 250, "not_null": true }, { "type": "string", "name": "author", "parameters": 250, "not_null": true } ] XML/HTML - BROWSER ------------------ - Input elements for different types of numbers with min and max values: 16 bits integer, 32 bits integer, 32 bits float and 64 bits float. - Input elements for images, audios and videos with preview. - Timezone input element. - Boolean input element with "true" and "false" values. - Null value in file inputs. - Clear button in file inputs like in date and time inputs. - Show size in file inputs. - Extension of the DOM API with high level and chainable methods. Example: paragraph.appendElement("a").setAttribute("href", "/article"); - Change of the "action" parameter of the forms to "target" to indicate the URL where to execute the action. The "action" parameter is moved to the different buttons on the form and allows executing a different action with each of the the buttons. Example: <form target="/article"> <button action="delete">Delete</button> <button action="export">Export</button> </form> - "select" elements that change a parameter of the current URL. Example: <select url-parameter="lang"/> <option>en</option> <option>es</option> URL = https://demo.nexttypes.com/?lang=en - "content-type" attribute in links and context menu in the browser to open links with external applications using WEBDAV, similar to a file manager. Example: <a href="" content- type="application/vnd.oasis.opendocument.text"> ------------------------------ | Open link with ... | | Open link with LibreOffice | - Background submission of forms without using XMLHttpRequest, display of result in dialog window or file download, and execution of a Javascript function for subsequent actions. Example: <form background show-progress callback=function()> - Dialog with progress indicator of form submission. Must show total size, transmitted data and average speed. Possibility to pause or cancel the submission. - Dynamic datalist with searchable JSON source. Over data source URL is added a "search" parameter with input value. Example: <input list="article-list" name="article" type="text" /> <datalist id="article-list" src="/article?lang=en&view=json&names" /> Example query URL: "/article?lang=en&view=json&names&search=Ne" - Same appearance and operation of inputs with dynamic datalist in all browsers. - Option tags with icons. Example: <option icon="/icons/save.svg">Save</option> - Tabs, Tree, etc widgets - Mechanism to close HTTPS sessions initiated with client certificate authentication. JAVA ---- - Subclasses of String or some system of variants of String that allows assigning a regular expression or class that limits its valid values to avoid code injection or values that will crash the system. Example: String:Type = "[a-z0-9 _] +"; Or String:Type = TypeChecks; String:Type type = "article_language"; -> correct String:Type type = "Article-Language"; -> error We can talk about the characteristics of each system in its mailing list or group. For the general topic of system integration I have created a discussion in the github project. https://github.com/alejsanc/nexttypes/discussions/6 This email has been sent to the following mailing lists and groups: pgsql-hackers(a)lists.postgresql.org pgsql-jdbc(a)lists.postgresql.org mozilla.dev.platform(a)googlegroups.com public-html(a)w3.org jdbc-spec-discuss(a)openjdk.java.net jdk-dev(a)openjdk.java.net maria-developers(a)lists.launchpad.net chromium-dev(a)chromium.org https://mysqlcommunity.slack.com/archives/C8R1336M7 Best regards. Alejandro Sánchez.

1 0

[Maria-developers] Re MDEV-7317: Ignored indexes
by Sergey Petrunia 24 Feb '21

24 Feb '21

Hi Varun, The code for the task is mostly fine. But I would like to question the user experience. == Ignoring an index causes InnoDB table rebuild == First (and foremost), making an index ignored (or not ignored) in an InnoDB table rebuilds the table: MariaDB [test]> set session alter_algorithm='instant'; alter table t1 alter index c ignore; Query OK, 0 rows affected (0.001 sec) ERROR 1845 (0A000): ALGORITHM=INSTANT is not supported for this operation. Try ALGORITHM=COPY This misses the point of the feature. If one needs to rebuild the table anyway, they can just drop (or recreate) the index. Curiously, it works with MyISAM. I think, the ideal solution would be to make ignoring/not ignoring the index an SQL layer change that doesn't concern the storage engine. If this is not possible, lets make InnoDB aware that it doesn't need to rebuild the table in this case. == Backward compatibility for SHOW CREATE TABLE == SHOW CREATE TABLE will show something like: KEY `a` (`a`) IGNORE, MySQL here is more backward-compatible and shows: KEY `a` (`a`) /*!80000 INVISIBLE */, Do we need to do this as well? I have no strong opinion here. == Use of Ignored index in hints == I can use Ignored index in FORCE/IGNORE index hints: MariaDB [test]> explain select * from t1 force index(a) where a<3 and b<3; +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 1000 | Using where | +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ This is allowed. I think, this misses the point of the feature. The point of this feature is to check "what will happen if one drops the index". For this particular query, the effect will be like this: MariaDB [test]> explain select * from t1 force index(no_such_index) where a<3 and b<3; ERROR 1176 (42000): Key 'no_such_index' doesn't exist in table 't1' and MySQL agrees with me here (below index a is present but is invisible) : mysql> explain select * from t1 force index(a) where a<3 and b<3; ERROR 1176 (42000): Key 'a' doesn't exist in table 't1' == optimizer_switch flag == Seeing "ignore_indexes=on" in optimizer_switch looks scary. I would say, we don't need an @@optimizer_switch flag. Maybe, Serg disagrees. If we have to have a switch flag, and its name should be short (my opinion), I would use the noun form: ignored_indexes=ON|OFF I'm not 100% happy with this, but ignoreD is much better here than ignore. (Before acting on this review point, lets check with others). BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

Re: [Maria-developers] MDEV-17399: JSON_TABLE, commit 85757ecbef44484270e385a8d52998c67f72d11a
by Sergey Petrunia 23 Feb '21

23 Feb '21

Hi Alexey, First reactions to the JSON_TABLE patch: == CI Build is failing == Please check http://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.6-mde… the JSON_TABLE tests are failing. == On testcases == I see now the patch has tests in 3 files: 1. mysql-test/main/json_table_mysql.test 2. mysql-test/main/json_table_mysql2.test 3. mysql-test/suite/json/t/json_table.test #1 and #2 are both fixed copies of MySQL's test. Please remove one of them (the one that has less coverage). Then, can we have both files in the same testsuite? == On Table_function_json_table::ready_for_lookup() == Now that we have end_lateral_table, do we need to use Table_function_json_table::ready_for_lookup? This looks like two solutions for [almost] the same problem? I see that this example from json_table_mysql.test still uses the code path with ready_for_lookup() : --error ER_BAD_FIELD_ERROR UPDATE t1, JSON_TABLE(t1.c1,'$[*]' COLUMNS (a INT PATH '$.a')) AS jt1 SET jt1.a=1; Here, the name resolution for "jt1.a" is done without adjustment for end_lateral_table. Do we need the ready_for_lookup() trick here? (If we do, are we fine with m_setup_done to be set to true but never reset back to false? How will this work with PS?) == On opt_sum.cc == The diff has this: diff --git a/sql/opt_sum.cc b/sql/opt_sum.cc index af2d9ddc2e7..d021c651196 100644 --- a/sql/opt_sum.cc +++ b/sql/opt_sum.cc @@ -299,6 +299,7 @@ int opt_sum_query(THD *thd, tl->schema_table) { maybe_exact_count&= MY_TEST(!tl->schema_table && +//mdev-17399 !tl->table_function && (tl->table->file->ha_table_flags() & HA_HAS_RECORDS)); is_exact_count= FALSE; Please remove this if it is no longer needed. == On table dependencies: table elimination == create table t20 (a int not null); create table t21 (a int not null primary key, js varchar(100)); insert into t20 values (1),(2); insert into t21 values (1, '{"a":100}'); explain select t20.a, jt1.ab from t20 left join t21 on t20.a=t21.a join JSON_TABLE(t21.js,'$' COLUMNS (ab INT PATH '$.a')) AS jt1; +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ | 1 | SIMPLE | t20 | ALL | NULL | NULL | NULL | NULL | 2 | | | 1 | SIMPLE | jt1 | ALL | NULL | NULL | NULL | NULL | 40 | Table function: json_table | +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ Here we can see an apparently invalid query plan: table t21 was eleminated even if JSON_TABLE uses it. == On table dependencies: regular subqueries == But it doesn't work for regular subqueries either (and this is unexpected): insert into t21 values (3, '{"a":300}'); explain select a, (select jt1.ab from JSON_TABLE(t21.js,'$' COLUMNS (ab INT PATH '$.a')) as jt1 ) from t21; +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ | 1 | PRIMARY | t21 | ALL | NULL | NULL | NULL | NULL | 2 | | | 2 | SUBQUERY | jt1 | ALL | NULL | NULL | NULL | NULL | 40 | Table function: json_table | +------+-------------+-------+------+---------------+------+---------+------+------+----------------------------+ Note the 'SUBQUERY', that is, the subquery is considered uncorrelated? This needs to be investigated. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 3

Re: [Maria-developers] MDEV-17399: Columns of JSON datatype?
by Sergey Petrunia 16 Feb '21

16 Feb '21

On Sun, Feb 14, 2021 at 05:14:10PM +0400, Alexey Botchkov wrote: > One question left about the json_table_mysql.text. > > --error ER_WRONG_OUTER_JOIN > SELECT * FROM t1 RIGHT JOIN > (SELECT o FROM t2, JSON_TABLE(j, '$[*]' COLUMNS (o FOR ORDINALITY)) AS jt) > AS t3 > ON (t3.o = t1.x); > > What is exactly wrong with the above join? It should work. The reason it doesn't work was described by me in the email with subject "MDEV-17399: JSON_TABLE's dependencies are not updated after derived merge" dated Feb, 8th. What happens here is: Thread 14 "mysqld" hit Breakpoint 1, Table_function_json_table::setup() (gdb) list 1091 } 1092 } 1093 > 1094 m_dep_tables= m_json->used_tables(); 1095 (gdb) p m_dep_tables $44 = 1 If you go one frame up, you will see that we are in select#2. tables are: (gdb) p join->select_lex->leaf_tables.elem(0)->alias.str $56 = 0x7ffeb0016550 "t2" (gdb) p join->select_lex->leaf_tables.elem(0)->table->map $58 = 1 (gdb) p join->select_lex->leaf_tables.elem(1)->alias.str $60 = 0x7ffeb0017d70 "jt" (gdb) p join->select_lex->leaf_tables.elem(1)->table->map $62 = 2 Ok, jt depends on t2. Then: Thread 14 "mysqld" hit Breakpoint 2, simplify_joins (gdb) p join->select_lex->leaf_tables.elements $65 = 3 (gdb) p join->select_lex->leaf_tables.elem(0)->alias.str $66 = 0x7ffeb0015870 "t1" (gdb) p join->select_lex->leaf_tables.elem(0)->table->map $67 = 1 (gdb) p join->select_lex->leaf_tables.elem(1)->alias.str $68 = 0x7ffeb0016550 "t2" (gdb) p join->select_lex->leaf_tables.elem(1)->table->map $69 = 2 (gdb) p join->select_lex->leaf_tables.elem(2)->alias.str $70 = 0x7ffeb0017d70 "jt" (gdb) p join->select_lex->leaf_tables.elem(2)->table->map $71 = 4 But the m_dep_tables still has the old value: (gdb) p join->select_lex->leaf_tables.elem(2)->table_function->m_dep_tables $72 = 1 BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

[Maria-developers] MDEV-8306: EXPLAIN JSON output is not JSON-ish
by Sergey Petrunia 15 Feb '21

15 Feb '21

Hi Varun, (This email is not the complete input, and not the most important part of the input. More to follow. I think it's better to split input into multiple smaller pieces, it's easier to track, and you get to see the first portions of the input sooner) EXPLAIN EXTENDED SELECT t3.b, t2.a, t1.b, t1.a FROM t1,t2,t3 WHERE t1.b=t3.b ORDER BY t1.b DESC, t2.a DESC LIMIT 3; +------+-------------+-------------+------+---------------+------+---------+------+------+----------+------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra | +------+-------------+-------------+------+---------------+------+---------+------+------+----------+------------------------------------+ | 1 | SIMPLE | t2 | ALL | NULL | NULL | NULL | NULL | 10 | 100.00 | | | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 20 | 100.00 | Using join buffer (flat, BNL join) | | 1 | SIMPLE | <sort-nest> | ALL | NULL | NULL | NULL | NULL | 3 | 100.00 | Using filesort | | 1 | SIMPLE | t3 | ALL | NULL | NULL | NULL | NULL | 100 | 5.00 | Using where | +------+-------------+-------------+------+---------------+------+---------+------+------+----------+------------------------------------+ So, here <sort-nest> includes t2 and t1, and then t3 is joined with it. However in EXPLAIN FORMAT=JSON output they all look like peers: { "query_block": { "select_id": 1, "table": { "table_name": "t2", "access_type": "ALL", "rows": 10, "filtered": 100 }, "block-nl-join": { "table": { "table_name": "t1", "access_type": "ALL", "rows": 20, "filtered": 100 }, "buffer_type": "flat", "buffer_size": "65", "join_type": "BNL" }, "read_sorted_file": { "filesort": { "sort_key": "`sort-nest`.b desc, `sort-nest`.a desc", "table": { "table_name": "<sort-nest>", "access_type": "ALL", "rows": 3, "filtered": 100 } } }, "table": { "table_name": "t3", "access_type": "ALL", "rows": 100, "filtered": 5, "attached_condition": "t3.b = `sort-nest`.b" } } } which is counter-intuitive. Also, `sort-nest` is quoted. I assume it is because of the '-' in it? Maybe we should change the name to avoid the quoting? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 3

Re: [Maria-developers] MDEV-17399 New patch for JSON_TABLE
by Sergey Petrunia 14 Feb '21

14 Feb '21

Hi Alexey, Input on the latest patch for MDEV-17399. It's good to see the patch taking shape, but have you tried running MySQL's tests for JSON_TABLE on it? I did, and it has caught several crashes, as well as error-no-error discrepancies and different query results. Some of these were expected, but some show that something needs to be fixed. Please find the first few cases below. It seems there is something odd going odd with the name resolution, both in MariaDB and MySQL - I don't have explanation for some of the errors. I intend to study the issue more and elaborate in a later email. Meanwhile, first obvious cases: === Crash in Name resolution == CREATE TABLE t1 (a INT, b INT); CREATE VIEW v2 AS SELECT * FROM t1 LIMIT 2; SELECT b FROM (SELECT * FROM v2) vq1, JSON_TABLE(CONCAT(vq1.b,'[{\"a\":\"3\"}]'), '$[*]' COLUMNS (id FOR ORDINALITY, jpath VARCHAR(100) PATH '$.a', JEXST INT EXISTS PATH '$.b') ) AS dt; === Crash in error reporting === select * from json_table( '[{"a":"3"},{"a":2},{"b":1},{"a":0}]', "!@#$!@#$" columns (id for ordinality, jpath varchar(100) path '$.a', jexst int exists path '$.b') ) as tt; === Item print misses quotes === CREATE VIEW v2 AS SELECT * FROM JSON_TABLE('{}', '$' COLUMNS ( x VARCHAR(10) PATH '$.a' DEFAULT '"isn''t here"' ON EMPTY) ) t; SHOW CREATE VIEW v2; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 't here"' ON EMPTY)) `t`' at line 1 === Error for outer reference == # JSON_TABLE referring outer scope CREATE TABLE t1(id int, jd JSON); INSERT INTO t1 values (1, '[1,3,5]'),(2,'[2,4,6]'); SELECT * FROM t1 WHERE id IN (SELECT * FROM JSON_TABLE(t1.jd, '$[*]' COLUMNS (id INT PATH '$')) AS jt); Produces: ERROR 1054 (42S22): Unknown column 't1.jd' in 'JSON_TABLE argument' Should not produce it (Note that for some other kinds of outer references it seemed to work) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 15

[Maria-developers] MDEV-17399: Name resolution and handling dependencies
by Sergey Petrunia 14 Feb '21

14 Feb '21

Hi Alexey, At the moment, name resolution of JSON_TABLE's first argument is done "like in the WHERE clause" - one can refer to any table that is defined in the WHERE clause. This allows one to write queries where JSON_TABLE tables have incorrect dependencies - circular - dependency that contradicts the dependency imposed by OUTER JOIN - dependency that contradicts STRAIGHT_JOIN (WRONG-DEPS) the patch checks for these cases but I've reported cases where it fails. I haven't been find any statement in the SQL Standard about this, but it makes a statement about similar constructs, table functions and LATERAL subqueries: https://jira.mariadb.org/browse/MDEV-17399?focusedCommentId=179327&page=com… Other databases seem to apply the same limitation to JSON_TABLE's argument. When they do it, this automatically fixes all the wrong-dependency issues - a query with (WRONG-DEPS) is rejected at the name resolution phase. I think we should follow this and modify name resolution process to work in the same way. The way MySQL did it is described here: https://dev.mysql.com/worklog/task/?id=8867, LLD, grep for end_lateral_table. I'm not fond of having implicit parameters (end_lateral_table) which change they way name resolution works, but I think it is an acceptable solution (and we already have other such parameters). (An alternative option would be to have items in JSON_TABLE's first argument to use their own Name_resolution_context object that would specify the correct first/last table they should look at. This seems to be much harder to do). What do you think? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

[Maria-developers] 4f9c5a8cf54: MDEV-24773: slave_compressed_protocol doesn't work properly with semi-sync replication
by sujatha 08 Feb '21

08 Feb '21

revision-id: 4f9c5a8cf545133c3d29c0de3f7e2e57564607f2 (mariadb-10.3.26-78-g4f9c5a8cf54) parent(s): 59eda73eff1a22ac0373d818bc802c05e82b5449 author: Sujatha committer: Sujatha timestamp: 2021-02-05 16:45:35 +0530 message: MDEV-24773: slave_compressed_protocol doesn't work properly with semi-sync replication Back port upstream fix commit 1800b015a1d487330f7b15f2020b887be348a66b Author: Venkatesh Duggirala <venkatesh.duggirala(a)oracle.com> Date: Fri Sep 8 20:29:22 2017 +0530 Bug#26027024 SLAVE_COMPRESSED_PROTOCOL DOESN'T WORK WITH SEMI-SYNC REPLICATION IN MYSQL-5.7 Analysis: In mysql-5.6, dump thread (the thread that is created on Master after Slave requested for a binlog dump) is also used to receive acknowledgements from the Slave and act on them accordingly. For performance reasons, a special thread called Ack Receiver thread is added in mysql-5.7 Semi synchronous replication plugin. This thread does not have special handling to receive acknowledgements if Slave has enabled compression in the protocol. Hence Master is unable to handle any slave if Slave_compressed_protocol is enabled on it. Fix: Enable compress flag on the communication channels if the Slave has Slave_compressed_protocol ON. --- .../rpl_semi_sync_slave_compressed_protocol.result | 19 ++++++++ ...l_semi_sync_slave_compressed_protocol-slave.opt | 1 + .../t/rpl_semi_sync_slave_compressed_protocol.test | 55 ++++++++++++++++++++++ sql/semisync_master_ack_receiver.cc | 5 ++ 4 files changed, 80 insertions(+) diff --git a/mysql-test/suite/rpl/r/rpl_semi_sync_slave_compressed_protocol.result b/mysql-test/suite/rpl/r/rpl_semi_sync_slave_compressed_protocol.result new file mode 100644 index 00000000000..86816652565 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_semi_sync_slave_compressed_protocol.result @@ -0,0 +1,19 @@ +include/master-slave.inc +[connection master] +SET @@GLOBAL.rpl_semi_sync_master_enabled = 1; +connection slave; +include/stop_slave.inc +SET @@GLOBAL.rpl_semi_sync_slave_enabled = 1; +include/start_slave.inc +connection master; +CREATE TABLE t1 (i INT); +DROP TABLE t1; +include/rpl_sync.inc +include/assert_grep.inc [Check that there is no 'Read semi-sync reply magic number error' in error log.] +connection master; +SET @@GLOBAL. rpl_semi_sync_master_enabled = 0; +connection slave; +include/stop_slave.inc +SET @@GLOBAL. rpl_semi_sync_slave_enabled = 0; +include/start_slave.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol-slave.opt b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol-slave.opt new file mode 100644 index 00000000000..a1b687d691e --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol-slave.opt @@ -0,0 +1 @@ +--slave_compressed_protocol diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol.test b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol.test new file mode 100644 index 00000000000..6343fb29ffe --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol.test @@ -0,0 +1,55 @@ +################################################################################ +# Bug#26027024 SLAVE_COMPRESSED_PROTOCOL DOESN'T WORK WITH SEMI-SYNC +# REPLICATION IN MYSQL-5.7 +# +# Steps to reproduce: +# 1) Set slave_compressed_protocol ON on Slave. +# 2) Do some sample work on Master +# 3) After the work is synced on Slave, check that there is no error +# (Read semi-sync reply magic number error) on Slave. +# 4) Cleanup +################################################################################ +# Test is independent of Binlog format. One of the three formats is enough +# for testing. Choosing 'Row' format. +--source include/have_binlog_format_row.inc +--source include/master-slave.inc + +--let $sav_enabled_master=`SELECT @@GLOBAL.rpl_semi_sync_master_enabled ` +SET @@GLOBAL.rpl_semi_sync_master_enabled = 1; + +--connection slave +source include/stop_slave.inc; +--let $sav_enabled_slave=`SELECT @@GLOBAL.rpl_semi_sync_slave_enabled ` +SET @@GLOBAL.rpl_semi_sync_slave_enabled = 1; +source include/start_slave.inc; + +--connection master +# Do some sample work on Master with slave_compressed_protocol ON. +# (slave_compressed_protocol is set to ON in -slave.opt file of this test.) +CREATE TABLE t1 (i INT); +DROP TABLE t1; + +# Make sure sync is done, so that next 'assert' step can be executed without +# any issues. +--source include/rpl_sync.inc + +# Without the fix, the test would have generated few +# errors in the error log. With the fix, test will +# pass without any errors in the error log. +--let $assert_text= Check that there is no 'Read semi-sync reply magic number error' in error log. +--let $assert_select=Read semi-sync reply magic number error +--let $assert_file= $MYSQLTEST_VARDIR/log/mysqld.1.err +--let $assert_count= 0 +--let $assert_only_after = CURRENT_TEST:rpl.rpl_semi_sync_slave_compressed_protocol.test +--source include/assert_grep.inc + +--connection master +--eval SET @@GLOBAL. rpl_semi_sync_master_enabled = $sav_enabled_master + +--connection slave +source include/stop_slave.inc; +--eval SET @@GLOBAL. rpl_semi_sync_slave_enabled = $sav_enabled_slave +source include/start_slave.inc; + +# Cleanup +--source include/rpl_end.inc diff --git a/sql/semisync_master_ack_receiver.cc b/sql/semisync_master_ack_receiver.cc index 81f494c9d34..cc3a56a56b2 100644 --- a/sql/semisync_master_ack_receiver.cc +++ b/sql/semisync_master_ack_receiver.cc @@ -268,6 +268,11 @@ void Ack_receiver::run() net_clear(&net, 0); net.vio= &slave->vio; + /* + Set compress flag. This is needed to support + Slave_compress_protocol flag enabled Slaves + */ + net.compress= slave->thd->net.compress; len= my_net_read(&net); if (likely(len != packet_error))

1 0

[Maria-developers] RFC: incompatible changes to mtr debugging flags
by Sergei Golubchik 27 Jan '21

27 Jan '21

Hi, all, Here's a feature I want to discuss, as it's rather incompatibly removes a bunch of mtr command-line options. It's a unified handling of all debuggers. I wanted to do it for quite a while, but the actual trigger was me trying to add support for --rr (https://rr-project.org/) in embedded. It was supported only for non-embedded server, and making it work for embedded needed way more copy-pasting than I was comfortable with. So, in my branch all debuggers are not haphazardly added using whatever syntax one thought of at the moment and copy-pasted all over. They've handled by a common module, all support the same set of options (using gdb as an example): --gdb, --boot-gdb, --client-gdb, --manual-gdb. Both --gdb and --client-gdb work for embedded. All four of them accept an optional argument that can be a semicolon separated list of gdb commands, like in ./mtr 1st --gdb='b mysql_parse;r' but can start from command-line options too: ./mtr 1st --boot-gdb='-nh -q;b mysql_parse;r' A "debugger" is anything that wraps mysqld or mysqltest execution, the current list of debuggers is: gdb, ddd, dbx, lldb, valgrind, strace, ktrace, rr, devenv, windbg, vc_express, vc, vsjitdebugger. There's also "valgdb" that does what '--valgrind --gdb' was doing before. This removes the following mtr options: --rr-args --rr-dir --manual-debug --debugger --strace-option --stracer --valgrind-all --valgrind-mysqltest --valgrind-mysqld --valgrind-options --valgrind-option --valgrind-path --callgrind, and --valgrind only enables it for the server, not for everything. If pushed, it'll definitely break backward compatibility. But I don't think mtr backward compatibility is all that important, it might only affect a handful of developers who use scripts to start mtr with specific hard-coded settings, those scripts could be easily adjusted. It's much more important to have same mtr features in all branches, so that when switching between branches, one wouldn't need to guess what command line options mtr supports now. That is, if pushed, this should go into 10.2. Opinions? Missing features? Push / not push? for the reference, it's in bb-10.2-serg branch at the moment. Regards, Sergei

5 7

Re: [Maria-developers] 3e01da668d1: MDEV-21785: sequences used as default by other table not dumped in right order by mysqldump
by Sergei Golubchik 21 Jan '21

21 Jan '21

Hi, Oleksandr! On Jan 21, Oleksandr Byelkin wrote: > revision-id: 3e01da668d1 (mariadb-10.3.26-64-g3e01da668d1) > parent(s): 59998d3480f > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2021-01-14 10:45:51 +0100 > message: > > MDEV-21785: sequences used as default by other table not dumped in right order by mysqldump > > Dump sequences first. > > This atch made to keep it small and > to keep number of queries to the server the same. > > Order of tables in a dump can not be changed > (except sequences first) because mysql_list_tables > uses SHOW TABLES and I used SHOW FULL TABLES. it looks like a rather overengineered implementation. I'd suggest to simplify it as: > diff --git a/client/mysqldump.c b/client/mysqldump.c > index a964f96437d..17837713c01 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -42,6 +42,11 @@ > /* on merge conflict, bump to a higher version again */ > #define DUMP_VERSION "10.19" > > +/** > + First mysql version supporting sequences. > +*/ > +#define FIRST_SEQUENCE_VERSION 100300 seems unnecessary, you implemented SHOW FULL TABLES in 5.0.2-alpha. Oct 2004. You can safely use SHOW FULL TABLES unconditionally. > + > #include <my_global.h> > #include <my_sys.h> > #include <my_user.h> > @@ -92,6 +97,13 @@ > /* Max length GTID position that we will output. */ > #define MAX_GTID_LENGTH 1024 > > + > +/* Dump sequence/tables control */ > +#define DUMP_TABLE_TABLE 1 > +#define DUMP_TABLE_SEQUENCE 2 > +#define DUMP_TABLE_ALL (DUMP_TABLE_TABLE | DUMP_TABLE_SEQUENCE) > + > + > static my_bool ignore_table_data(const uchar *hash_key, size_t len); > static void add_load_option(DYNAMIC_STRING *str, const char *option, > const char *option_value); > @@ -171,6 +183,7 @@ static DYNAMIC_STRING dynamic_where; > static MYSQL_RES *get_table_name_result= NULL; > static MEM_ROOT glob_root; > static MYSQL_RES *routine_res, *routine_list_res; > +static int get_table_name_result_short= 1; that'll be unnecessary > > > #include <sslopt-vars.h> > @@ -3853,13 +3866,16 @@ static char *alloc_query_str(size_t size) > ARGS > table - table name > db - db name > + type_ctrl - dump table or sequences or both > > RETURNS > void > */ > > > -static void dump_table(const char *table, const char *db, const uchar *hash_key, size_t len) > +static void dump_table(const char *table, const char *db, > + const uchar *hash_key, size_t len, > + const uint type_ctrl) > { > char ignore_flag; > char buf[200], table_buff[NAME_LEN+3]; > @@ -3881,9 +3897,11 @@ static void dump_table(const char *table, const char *db, const uchar *hash_key, > */ > if (check_if_ignore_table(table, table_type) & IGNORE_SEQUENCE_TABLE) > { > - get_sequence_structure(table, db); > + if (type_ctrl & DUMP_TABLE_SEQUENCE) > + get_sequence_structure(table, db); > + DBUG_VOID_RETURN; > + } else if (!(type_ctrl & DUMP_TABLE_TABLE)) > DBUG_VOID_RETURN; > - } here I'd remove the whole if(). Let's say the convention is that dump_table() should never be invoked for sequences, it's the caller's job to ensure it. > /* > Make sure you get the create table info before the following check for > --no-data flag below. Otherwise, the create table info won't be printed. > @@ -4368,18 +4386,49 @@ static void dump_table(const char *table, const char *db, const uchar *hash_key, > } /* dump_table */ > > > -static char *getTableName(int reset) > +static char *getTableName(int reset, uint table_control) > { > MYSQL_ROW row; > > if (!get_table_name_result) > { > - if (!(get_table_name_result= mysql_list_tables(mysql,NullS))) > - return(NULL); > + if (mysql_get_server_version(mysql) >= FIRST_SEQUENCE_VERSION) > + { > + const char *query= > + "SHOW FULL TABLES"; > + > + if (mysql_query_with_error_report(mysql, 0, query)) > + return (NULL); > + > + if (!(get_table_name_result= mysql_store_result(mysql))) > + return (NULL); > + > + get_table_name_result_short= 0; > + } > + else > + { > + if (!(get_table_name_result= mysql_list_tables(mysql,NullS))) > + return(NULL); > + get_table_name_result_short= 1; > + } this will be twice as small, if you won't try to support before-5.0 servers. > } > if ((row= mysql_fetch_row(get_table_name_result))) > - return((char*) row[0]); > - > + { > + int sequence; > + if (get_table_name_result_short || > + table_control == DUMP_TABLE_ALL) > + return((char*) row[0]); > + while (row && > + (((sequence= (strcmp(row[1], "SEQUENCE") == 0)) && > + (table_control & DUMP_TABLE_TABLE)) || > + ((!sequence) && > + (table_control & DUMP_TABLE_SEQUENCE)))) I'd rather just do static char *getTableName(int reset, uint want_sequence) and while (row && strcmp(row[1], "SEQUENCE") == want_sequence) row= mysql_fetch_row(get_table_name_result); > + { > + row= mysql_fetch_row(get_table_name_result); > + } > + if (row) > + return((char*) row[0]); > + } > if (reset) > mysql_data_seek(get_table_name_result,0); /* We want to read again */ > else > @@ -4785,17 +4834,17 @@ static int dump_all_stats() > /* EITS added in 10.0.1 */ > if (mysql_get_server_version(mysql) >= 100001) > { > - dump_table("column_stats", "mysql", NULL, 0); > - dump_table("index_stats", "mysql", NULL, 0); > - dump_table("table_stats", "mysql", NULL, 0); > + dump_table("column_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("index_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("table_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); this wouldn't be needed if the caller will only use dump_table for tables > } > /* Innodb may be disabled */ > if (!mysql_query(mysql, "show fields from innodb_index_stats")) > { > MYSQL_RES *tableres= mysql_store_result(mysql); > mysql_free_result(tableres); > - dump_table("innodb_index_stats", "mysql", NULL, 0); > - dump_table("innodb_table_stats", "mysql", NULL, 0); > + dump_table("innodb_index_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("innodb_table_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); > } > opt_no_create_info= prev_no_create_info; > return 0; > @@ -4817,11 +4866,11 @@ static int dump_all_timezones() > opt_prev_no_create_info= opt_no_create_info; > opt_no_create_info= 1; > fprintf(md_result_file,"\nUSE mysql;\n"); > - dump_table("time_zone", "mysql", NULL, 0); > - dump_table("time_zone_name", "mysql", NULL, 0); > - dump_table("time_zone_leap_second", "mysql", NULL, 0); > - dump_table("time_zone_transition", "mysql", NULL, 0); > - dump_table("time_zone_transition_type", "mysql", NULL, 0); > + dump_table("time_zone", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("time_zone_name", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("time_zone_leap_second", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("time_zone_transition", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("time_zone_transition_type", "mysql", NULL, 0, DUMP_TABLE_TABLE); > opt_no_create_info= opt_prev_no_create_info; > return 0; > } > @@ -5346,12 +5396,29 @@ static int dump_all_tables_in_db(char *database) > DBUG_RETURN(1); > } > } > - while ((table= getTableName(0))) > + > + if (mysql_get_server_version(mysql) >= FIRST_SEQUENCE_VERSION && > + !opt_no_create_info) > + { > + // First process sequences > + while ((table= getTableName(1, DUMP_TABLE_SEQUENCE))) > + { > + char *end= strmov(afterdot, table); > + if (include_table((uchar*) hash_key, end - hash_key)) > + { > + dump_table(table, database, (uchar*) hash_key, end - hash_key, > + DUMP_TABLE_SEQUENCE); > + } > + } > + table_ctrl= DUMP_TABLE_TABLE; // next pass > + } and instead of all the above, you could do just for (int table_ctrl= 1; table_ctrl >= 0; table_ctrl--) > + while ((table= getTableName(0, table_ctrl))) > { > char *end= strmov(afterdot, table); > if (include_table((uchar*) hash_key, end - hash_key)) > { > - dump_table(table, database, (uchar*) hash_key, end - hash_key); > + dump_table(table, database, (uchar*) hash_key, end - hash_key, > + table_ctrl); > my_free(order_by); > order_by= 0; > if (opt_dump_triggers && mysql_get_server_version(mysql) >= 50009) > @@ -5745,11 +5813,24 @@ static int dump_selected_tables(char *db, char **table_names, int tables) > DBUG_RETURN(1); > } > } > + > + if (mysql_get_server_version(mysql) >= FIRST_SEQUENCE_VERSION && > + !opt_no_create_info) > + { > + /* Dump Sequence first */ > + for (pos= dump_tables; pos < end; pos++) > + { > + DBUG_PRINT("info",("Dumping sequence(?) %s", *pos)); > + dump_table(*pos, db, NULL, 0, DUMP_TABLE_SEQUENCE); > + } > + } > + else > + table_ctrl|= DUMP_TABLE_SEQUENCE; // dump all on next pass and here, in dump_selected_tables(), you'd need to do check_if_ignore_table() == IGNORE_SEQUENCE_TABLE. > /* Dump each selected table */ > for (pos= dump_tables; pos < end; pos++) > { > DBUG_PRINT("info",("Dumping table %s", *pos)); > - dump_table(*pos, db, NULL, 0); > + dump_table(*pos, db, NULL, 0, table_ctrl); > if (opt_dump_triggers && > mysql_get_server_version(mysql) >= 50009) > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] eb75e8705d9: MDEV-8134: The relay-log is not flushed after the slave-relay-log.999999 showed
by sujatha 21 Jan '21

21 Jan '21

revision-id: eb75e8705d9a444e10057967eaebf947b1115ff8 (mariadb-10.2.31-695-geb75e8705d9) parent(s): 53acd1c1d88be82190c56af3e4cc11fb2770a169 author: Sujatha committer: Sujatha timestamp: 2021-01-21 13:00:02 +0530 message: MDEV-8134: The relay-log is not flushed after the slave-relay-log.999999 showed Problem: ======== Auto purge of relaylogs stops when relay-log-file is 'slave-relay-log.999999' and slave_parallel_threads is enabled. Analysis: ========= The problem is that in Relay_log_info::inc_group_relay_log_pos() function, when two log names are compared via strcmp() function, it gives correct result, when log name sequence numbers are of same digits(6 digits), But when the number goes to 7 digits, a 999999 compares greater than 1000000, which is wrong, hence the bug. Fix: ==== Extract the numeric extension part of the file name, convert it into unsigned long and compare. Thanks to David Zhao for the contribution. --- .../suite/rpl/r/rpl_relay_max_extension.result | 37 +++++++ .../suite/rpl/t/rpl_relay_max_extension.test | 109 +++++++++++++++++++++ sql/rpl_parallel.cc | 5 +- sql/rpl_rli.cc | 4 +- sql/sql_repl.cc | 17 ++++ sql/sql_repl.h | 1 + 6 files changed, 169 insertions(+), 4 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_relay_max_extension.result b/mysql-test/suite/rpl/r/rpl_relay_max_extension.result new file mode 100644 index 00000000000..4444398203e --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_relay_max_extension.result @@ -0,0 +1,37 @@ +include/rpl_init.inc [topology=1->2] +connection server_2; +include/stop_slave.inc +RESET SLAVE; +include/start_slave.inc +include/stop_slave.inc +# +# Stop slave server +# +# +# Simulate file number get close to 999997 +# by renaming relay logs and modifying index/info files +# +# Restart slave server +# +SET @save_slave_parallel_threads= @@GLOBAL.slave_parallel_threads; +SET @save_max_relay_log_size= @@GLOBAL.max_relay_log_size; +SET GLOBAL slave_parallel_threads=1; +SET GLOBAL max_relay_log_size=100 * 1024; +include/start_slave.inc +connection server_1; +create table t1 (i int, c varchar(1024)); +# +# Insert some data to generate enough amount of binary logs +# +connection server_2; +# +# Assert that 'slave-relay-bin.999999' is purged. +# +NOT FOUND /slave-relay-bin.999999/ in slave-relay-bin.index +include/stop_slave.inc +SET GLOBAL slave_parallel_threads= @save_slave_parallel_threads; +SET GLOBAL max_relay_log_size= @save_max_relay_log_size; +include/start_slave.inc +connection server_1; +DROP TABLE t1; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_relay_max_extension.test b/mysql-test/suite/rpl/t/rpl_relay_max_extension.test new file mode 100644 index 00000000000..e1e087f2e0e --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_relay_max_extension.test @@ -0,0 +1,109 @@ +# ==== Purpose ==== +# +# Test verifies that auto purging mechanism of relay logs works fine when the +# file extension grows beyond 999999. +# +# ==== Implementation ==== +# +# Steps: +# 0 - In master-slave setup clear all the relay logs on the slave server. +# 1 - Start the slave so that new relay logs starting from +# 'slave-relay-bin.000001' are created. +# 2 - Get the active relay-log file name by using SHOW SLAVE STATUS. +# Shutdown the slave server. +# 3 - Rename active relay log to '999997' in both 'relay-log.info' and +# 'slave-relay-bin.index' files. +# 4 - Restart the slave server by configuring 'slave_parallel_threads=1' +# and 'max_relay_log_size=100K'. +# 5 - Generate load on master such that few relay logs are generated on +# slave. The relay log sequence number will change to 7 digits. +# 6 - Sync slave with master to ensure that relay logs are applied on +# slave. They should have been automatically purged. +# 7 - Assert that there is no 'slave-relay-bin.999999' file in +# 'relay-log.info'. +# +# ==== References ==== +# +# MDEV-8134: The relay-log is not flushed after the slave-relay-log.999999 +# showed +# + +--source include/have_innodb.inc +--source include/have_binlog_format_row.inc +--let $rpl_topology=1->2 +--source include/rpl_init.inc + +--connection server_2 +--source include/stop_slave.inc +RESET SLAVE; +--source include/start_slave.inc +--source include/stop_slave.inc +--let $relay_log=query_get_value(SHOW SLAVE STATUS, Relay_Log_File, 1) + +--echo # +--echo # Stop slave server +--echo # + +--let $datadir = `select @@datadir` +--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.2.expect +--shutdown_server 10 +--source include/wait_until_disconnected.inc + +--exec sed -i "s/$relay_log/slave-relay-bin.999997/g" $datadir/relay-log.info +--exec sed -i "s/$relay_log/slave-relay-bin.999997/g" $datadir/slave-relay-bin.index + +--echo # +--echo # Simulate file number get close to 999997 +--echo # by renaming relay logs and modifying index/info files + +--move_file $datadir/$relay_log $datadir/slave-relay-bin.999997 + +--echo # +--echo # Restart slave server +--echo # + +--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.2.expect +--enable_reconnect +--source include/wait_until_connected_again.inc +SET @save_slave_parallel_threads= @@GLOBAL.slave_parallel_threads; +SET @save_max_relay_log_size= @@GLOBAL.max_relay_log_size; + +SET GLOBAL slave_parallel_threads=1; +SET GLOBAL max_relay_log_size=100 * 1024; +--source include/start_slave.inc + +--connection server_1 +create table t1 (i int, c varchar(1024)); +--echo # +--echo # Insert some data to generate enough amount of binary logs +--echo # +--let $count = 1000 +--disable_query_log +while ($count) +{ + eval insert into t1 values (1001 - $count, repeat('a',1000)); + dec $count; +} +--enable_query_log +--save_master_pos + +--connection server_2 +--sync_with_master + +--let $relay_log=query_get_value(SHOW SLAVE STATUS, Relay_Log_File, 1) + +--echo # +--echo # Assert that 'slave-relay-bin.999999' is purged. +--echo # +let SEARCH_FILE=$datadir/slave-relay-bin.index; +let SEARCH_PATTERN=slave-relay-bin.999999; +source include/search_pattern_in_file.inc; + +--source include/stop_slave.inc +SET GLOBAL slave_parallel_threads= @save_slave_parallel_threads; +SET GLOBAL max_relay_log_size= @save_max_relay_log_size; +--source include/start_slave.inc + +--connection server_1 +DROP TABLE t1; +--source include/rpl_end.inc diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc index 4cf87ba73b7..869640fd46f 100644 --- a/sql/rpl_parallel.cc +++ b/sql/rpl_parallel.cc @@ -4,6 +4,7 @@ #include "rpl_mi.h" #include "sql_parse.h" #include "debug_sync.h" +#include "sql_repl.h" /* Code for optional parallel execution of replicated events on the slave. @@ -82,7 +83,7 @@ handle_queued_pos_update(THD *thd, rpl_parallel_thread::queued_event *qev) return; mysql_mutex_lock(&rli->data_lock); - cmp= strcmp(rli->group_relay_log_name, qev->event_relay_log_name); + cmp= compare_log_name(rli->group_relay_log_name, qev->event_relay_log_name); if (cmp < 0) { rli->group_relay_log_pos= qev->future_event_relay_log_pos; @@ -91,7 +92,7 @@ handle_queued_pos_update(THD *thd, rpl_parallel_thread::queued_event *qev) rli->group_relay_log_pos < qev->future_event_relay_log_pos) rli->group_relay_log_pos= qev->future_event_relay_log_pos; - cmp= strcmp(rli->group_master_log_name, qev->future_event_master_log_name); + cmp= compare_log_name(rli->group_master_log_name, qev->future_event_master_log_name); if (cmp < 0) { strcpy(rli->group_master_log_name, qev->future_event_master_log_name); diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc index 40ab375571a..5273b33c728 100644 --- a/sql/rpl_rli.cc +++ b/sql/rpl_rli.cc @@ -989,7 +989,7 @@ void Relay_log_info::inc_group_relay_log_pos(ulonglong log_pos, if (rgi->is_parallel_exec) { /* In case of parallel replication, do not update the position backwards. */ - int cmp= strcmp(group_relay_log_name, rgi->event_relay_log_name); + int cmp= compare_log_name(group_relay_log_name, rgi->event_relay_log_name); if (cmp < 0) { group_relay_log_pos= rgi->future_event_relay_log_pos; @@ -1001,7 +1001,7 @@ void Relay_log_info::inc_group_relay_log_pos(ulonglong log_pos, In the parallel case we need to update the master_log_name here, rather than in Rotate_log_event::do_update_pos(). */ - cmp= strcmp(group_master_log_name, rgi->future_event_master_log_name); + cmp= compare_log_name(group_master_log_name, rgi->future_event_master_log_name); if (cmp <= 0) { if (cmp < 0) diff --git a/sql/sql_repl.cc b/sql/sql_repl.cc index 4af8ebc2dd8..59a3f686e45 100644 --- a/sql/sql_repl.cc +++ b/sql/sql_repl.cc @@ -4541,5 +4541,22 @@ rpl_gtid_pos_update(THD *thd, char *str, size_t len) return false; } +int compare_log_name(const char *log_1, const char *log_2) { + int res= 1; + const char *ext1_str= strrchr(log_1, '.'); + const char *ext2_str= strrchr(log_2, '.'); + char file_name_1[255], file_name_2[255]; + strmake(file_name_1, log_1, (ext1_str - log_1)); + strmake(file_name_2, log_2, (ext2_str - log_2)); + char *endptr = NULL; + res= strcmp(file_name_1, file_name_2); + if (!res) + { + ulong ext1= strtoul(++ext1_str, &endptr, 10); + ulong ext2= strtoul(++ext2_str, &endptr, 10); + res= (ext1 > ext2 ? 1 : ((ext1 == ext2) ? 0 : -1)); + } + return res; +} #endif /* HAVE_REPLICATION */ diff --git a/sql/sql_repl.h b/sql/sql_repl.h index 8ddfa9239f6..9129aaeed5e 100644 --- a/sql/sql_repl.h +++ b/sql/sql_repl.h @@ -56,6 +56,7 @@ bool show_binlogs(THD* thd); extern int init_master_info(Master_info* mi); void kill_zombie_dump_threads(uint32 slave_server_id); int check_binlog_magic(IO_CACHE* log, const char** errmsg); +int compare_log_name(const char *log_1, const char *log_2); struct LOAD_FILE_IO_CACHE : public IO_CACHE {

1 0

[Maria-developers] How to change max key length for MyISAM?
by Witold Filipczyk 15 Jan '21

15 Jan '21

Sorry, for private question, but I did not know where to ask. mariadb-10.3.27. I changed in include/myisam.h: #define MI_MAX_KEY_LENGTH 3072 /* Max length in bytes */ but, it did not help, still this ERROR 1071 (42000): Specified key was too long; max key length is 1000 bytes. What more to change to get key length 3072 for MyISAM tables? And what can be negative impact of such a change? Witold Filipczyk

5 8

[Maria-developers] MDEV-13467 review.
by Alexey Botchkov 09 Jan '21

09 Jan '21

Hi, Anel! Sorry for that long delay with this review. Had real difficult weeks of my life :( Now it seems to be over, so getting back to normal. Below is my opinions of the code i see in the branch bb-10.1-anel-MDEV-13467-gis-feature-st_sphere-v2 Hope it's the recent version. + Adapter for functions that takes two or three arguments. +*/ + +class Create_func_arg2_or_3 : public Create_func +{ Do you think we need the separate Create_fund_arg2_or_3 class? If you don't see any other class to inherit from it, i'd recommend to leave the Create_func_distance_sphere alone. No need to have two constructors in the Item_func_sphere_distance too. Just use the list argument as it's done for the Item_func_json_contains (in 10.2). Anyway i belive we should get rid of duplicating code about the 'param1' and 'param2'. They can be popped/checked with no if (arg_count == 2) condition. + double distance= 0.0; + null_value= (args[0]->null_value || args[1]->null_value); + if (null_value) + { + // Returned item must be set to Null + DBUG_ASSERT(maybe_null); don't think it's needed. + return 0; Should be return 0.0 i belive. Or goto handle_errors. Thet goes to all but one 'return' statements in this function. + } > ... > if (!arg1 || !arg2) This last condition should never happen since we have not-null_value. So i'd remove that 'if' section completely. And anyway it should do the 'return' in that branch. + null_value= args[2]->null_value; + if (null_value) + { + return 0; + } seems nicer to me this way if (args[2]->null_value) { null_value= true; goto handle_errors; } + if (!(g1= Geometry::construct(&buffer1, arg1->ptr(), arg1->length())) || + !(g2= Geometry::construct(&buffer2, arg2->ptr(), arg2->length()))) + { + goto handle_errors; we should launch the ER_GIS_INVALID_DATA here. + // Generate error message in case different geometry is used? Yes, i think the explainig message here is good to have. +double Item_func_sphere_distance::spherical_distance(Geometry *g1, + Geometry *g2, + const double sphere_radius) +double Item_func_sphere_distance::spherical_distance_points(Geometry *g1, + Geometry *g2, + const double r) Why these two are member static functions, not simply static? If you plan any 'external' use for them, please add comments about what exacly they do. +double Item_func_sphere_distance::spherical_distance(Geometry *g1, + Geometry *g2, + const double sphere_radius) +{ + double res= 0.0; + switch (g1->get_class_info()->m_type_id) + { + case Geometry::wkb_point: + res= spherical_distance_points(g1, g2, sphere_radius); + break; + + case Geometry::wkb_multipoint: + res= spherical_distance_points(g1, g2, sphere_radius); + break; Why do we need this function? We confirmed already that the type is either wkb_point or wkb_multipolygon, so just can call spherical_distance_points. No? >>>>> +class Item_func_sphere_distance: public Item_real_func +{ + double sphere_radius= 6370986.0; // Default radius equals Earth radius Some compilers don't like this syntax, and we didn't use it yet, so i'd recomment to move this to the constructor or even better to the ::val_real(). +double Gis_point::calculate_haversine(const Geometry *g, + const double sphere_radius) +{ + DBUG_ASSERT(sphere_radius > 0); + double x1r= 0.0; + double x2r= 0.0; + double y1r= 0.0; + double y2r= 0.0; That lot of unnecessary initializations seems excessive. + if (g->get_class_info()->m_type_id == Geometry::wkb_multipoint) + { + const char point_size= 4 + WKB_HEADER_SIZE + POINT_DATA_SIZE+1; //1 for the type + char point_temp[point_size]; + memset(point_temp+4, Geometry::wkb_point, 1); + memcpy(point_temp+5, static_cast<const Gis_multi_point *>(g)->get_data_ptr()+5, 4); + memcpy(point_temp+4+WKB_HEADER_SIZE, g->get_data_ptr()+4+WKB_HEADER_SIZE, + POINT_DATA_SIZE); + point_temp[point_size-1]= '\0'; + Geometry_buffer gbuff; + Geometry *gg= Geometry::construct(&gbuff, point_temp, point_size-1); There is the Gis_multi_point::geometry_n() function, should be used here to get the point. I don't like that lot of static_cast-s. Isn't it nicer to have just the function of double calculate_harvesine_points(Geometry *p1, Geometry *p2, double radius) double calculate_harvesine_multipoint_point(Geometry *mp, Geometry *p, double radius) double calculate_harvesine_multipoint_multipoint(Geometry *mp1, Geometry *mp2, double radius) which would call each other? The ::get_xy_radian() method looks unneeded to me. It's enough to have the to_radian(double d) function. BTW it's possible to do the multipoint/multipoint calculations faster than n^2. Not necessary to do it right now, just not to remember. Best regards. HF

1 0

[Maria-developers] 608b0ee52ef: MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11
by sujatha 04 Jan '21

04 Jan '21

revision-id: 608b0ee52ef3e854ce14a407e64e936adbbeba23 (mariadb-10.2.31-648-g608b0ee52ef) parent(s): 25db9ffa8bdab8a2f2af3c7f154343dd6c6d238f author: Sujatha committer: Sujatha timestamp: 2021-01-04 15:06:12 +0530 message: MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11 Problem: ======= Upon deleting or updating a row in a parent table (with primary key), if the child table has virtual column and an associated key with ON UPDATE CASCADE/ON DELETE CASCADE, it will result in slave crash. Analysis: ======== Tables which are related through foreign key require prelocking similar to triggers. i.e If a table has triggers/foreign keys we should add all tables and routines used by them to the prelocking set. This prelocking happens during 'open_and_lock_tables' call. Each table being opened is checked for foreign key references. If foreign key reference exists then the child table is opened and it is linked to the table_list. Upon any modification to parent table its corresponding child tables are retried from table_list and they are updated accordingly. This prelocking work fine on master. On slave prelocking works for following cases. - Statement/mixed based replication - In row based replication when trigger execution is enabled through 'slave_run_triggers_for_rbr=YES/LOGGING/ENFORCE' Otherwise it results in an assert/crash, as the parent table will not find the corresponding child table and it will be NULL. Dereferencing NULL pointer leads to slave server exit. Fix: === Introduce a new 'slave_fk_event_map' flag similar to 'trg_event_map'. This flag will ensure that when foreign key is enabled in row based replication all the parent and child tables are prelocked, so that parent is able to locate the child table. Note: This issue is specific to slave, hence only slave needs to be upgraded. --- mysql-test/suite/rpl/r/rpl_row_vcol_crash.result | 380 ++++++++++++++++++++ mysql-test/suite/rpl/t/rpl_row_vcol_crash.test | 425 +++++++++++++++++++++++ sql/log_event.cc | 42 +-- sql/sql_base.cc | 118 ++++--- sql/table.h | 6 +- 5 files changed, 906 insertions(+), 65 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result b/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result new file mode 100644 index 00000000000..f76d8935fa8 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result @@ -0,0 +1,380 @@ +include/master-slave.inc +[connection master] +# +# Test case 1: KEY on a virtual column with ON DELETE CASCADE +# +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1),(2),(3); +CREATE TABLE t2 (id INT NOT NULL PRIMARY KEY, +t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (90,1,NULL); +INSERT INTO t2 VALUES (91,2,default); +DELETE FROM t1 WHERE id=1; +connection slave; +# +# Verify data consistency on slave +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t2,t1; +connection slave; +# +# Test case 2: Verify "ON DELETE CASCADE" for parent->child->child scenario +# Parent table: users +# Child tables: matchmaking_groups, matchmaking_group_users +# Parent table: matchmaking_groups +# Child tables: matchmaking_group_users, matchmaking_group_maps +# +# Deleting a row from parent table should be reflected in +# child tables. +# matchmaking_groups->matchmaking_group_users->matchmaking_group_maps +# users->matchmaking_group_users->matchmaking_group_maps +# +connection master; +CREATE TABLE users (id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, +name VARCHAR(32) NOT NULL DEFAULT '' +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_groups ( +id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, +host_user_id INT UNSIGNED NOT NULL UNIQUE, +v_col INT AS (host_user_id+1) VIRTUAL, KEY (v_col), +CONSTRAINT FOREIGN KEY (host_user_id) REFERENCES users (id) +ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_group_users ( +matchmaking_group_id BIGINT UNSIGNED NOT NULL, +user_id INT UNSIGNED NOT NULL, +v_col1 int as (user_id+1) virtual, KEY (v_col1), +PRIMARY KEY (matchmaking_group_id,user_id), +UNIQUE KEY user_id (user_id), +CONSTRAINT FOREIGN KEY (matchmaking_group_id) +REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE, +CONSTRAINT FOREIGN KEY (user_id) +REFERENCES users (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_group_maps ( +matchmaking_group_id BIGINT UNSIGNED NOT NULL, +map_id TINYINT UNSIGNED NOT NULL, +v_col2 INT AS (map_id+1) VIRTUAL, KEY (v_col2), +PRIMARY KEY (matchmaking_group_id,map_id), +CONSTRAINT FOREIGN KEY (matchmaking_group_id) +REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +connection slave; +connection master; +INSERT INTO users VALUES (NULL,'foo'),(NULL,'bar'); +INSERT INTO matchmaking_groups VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_users VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_maps VALUES (10,55,default),(11,66,default); +DELETE FROM matchmaking_groups WHERE id = 10; +connection slave; +# +# No rows should be returned as ON DELETE CASCASE should have removed +# corresponding rows from child tables. There should not any mismatch +# of 'id' field between parent->child. +# +SELECT * FROM matchmaking_group_users WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +matchmaking_group_id user_id v_col1 +SELECT * FROM matchmaking_group_maps WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +matchmaking_group_id map_id v_col2 +# +# Rows with id=11 should be present +# +SELECT * FROM matchmaking_group_users; +matchmaking_group_id user_id v_col1 +11 2 3 +SELECT * FROM matchmaking_group_maps; +matchmaking_group_id map_id v_col2 +11 66 67 +connection master; +DELETE FROM users WHERE id = 2; +connection slave; +# +# No rows should be present in both the child tables +# +SELECT * FROM matchmaking_group_users; +matchmaking_group_id user_id v_col1 +SELECT * FROM matchmaking_group_maps; +matchmaking_group_id map_id v_col2 +connection master; +DROP TABLE matchmaking_group_maps, matchmaking_group_users, matchmaking_groups, users; +connection slave; +# +# Test case 3: KEY on a virtual column with ON UPDATE CASCADE +# +connection master; +CREATE TABLE t1 (a INT NOT NULL PRIMARY KEY, b INT NOT NULL) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 80); +CREATE TABLE t2 (a INT KEY, b INT, +v_col int as (b+1) virtual, KEY (v_col), +CONSTRAINT b FOREIGN KEY (b) REFERENCES t1(a) ON UPDATE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (51, 1, default); +connection slave; +connection master; +UPDATE t1 SET a = 50 WHERE a = 1; +# +# Master: Verify that ON UPDATE CASCADE works fine +# old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +# +SELECT * FROM t2 WHERE b=50; +a b v_col +51 50 51 +connection slave; +# +# Slave: Verify that ON UPDATE CASCADE works fine +# old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +# +SELECT * FROM t2 WHERE b=50; +a b v_col +51 50 51 +connection master; +DROP TABLE t2, t1; +connection slave; +# +# Test case 4: Define triggers on master, their results should be +# replicated as part of row events and they should be +# applied on slave with the default +# slave_run_triggers_for_rbr=NO +# +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t2 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +connection slave; +SHOW GLOBAL VARIABLES LIKE 'slave_run_triggers_for_rbr'; +Variable_name Value +slave_run_triggers_for_rbr NO +# +# As two rows are inserted in table 't1', two rows should get inserted +# into table 't2' as part of trigger. +# +include/assert.inc [Table t2 should have two rows.] +connection master; +DROP TABLE t1,t2; +connection slave; +# +# Test case 5: Define triggers + Foreign Keys on master, their results +# should be replicated as part of row events and master +# and slave should be in sync. +# +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t3 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# As two rows are inserted in table 't1', two rows should get inserted +# into table 't3' as part of trigger. +# +include/assert.inc [Table t3 should have two rows.] +# +# Verify ON DELETE CASCASE correctness +# +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +connection master; +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +include/diff_tables.inc [master:test.t3, slave:test.t3] +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case 6: Triggers are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= NO;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr NO +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), +KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 0 +# +include/assert.inc [Table t3 should have zero rows.] +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +# +# Verify t1, t2 are consistent on slave. +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case 7: Triggers are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= YES;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr YES +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), +KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 2 +# +include/assert.inc [Table t3 should have two rows.] +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +# +# Verify t1, t2 are consistent on slave. +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case 8: Triggers and Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= NO;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr NO +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 0 +# +include/assert.inc [Table t3 should have zero rows.] +connection master; +DELETE FROM t1 WHERE id=2; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have two rows +SELECT * FROM t2; +t1_id v_col +2 3 +3 4 +connection slave; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +t1_id v_col +3 4 +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case 9: Triggers are Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= YES;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr YES +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 2 +# +include/assert.inc [Table t3 should have two rows.] +connection master; +DELETE FROM t1 WHERE id=2; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have two rows +SELECT * FROM t2; +t1_id v_col +2 3 +3 4 +connection slave; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +t1_id v_col +3 4 +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +connection master; +DROP TABLE t3,t2,t1; +connection slave; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test b/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test new file mode 100644 index 00000000000..84ee14977f3 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test @@ -0,0 +1,425 @@ +# ==== Purpose ==== +# +# Test verifies that, slave doesn't report any assert on UPDATE or DELETE of +# row which tries to update the virtual columns with associated KEYs. +# +# Test scenarios are listed below. +# 1) KEY on a virtual column with ON DELETE CASCADE +# 2) Verify "ON DELETE CASCADE" for parent->child->child scenario +# 3) KEY on a virtual column with ON UPDATE CASCADE +# 4) Define triggers on master, their results should be replicated +# as part of row events and they should be applied on slave with +# the default slave_run_triggers_for_rbr=NO +# 5) Define triggers + Foreign Keys on master, their results should be +# replicated as part of row events and master and slave should be in sync. +# 6) Triggers are present only on slave and 'slave_run_triggers_for_rbr=NO' +# 7) Triggers are present only on slave and 'slave_run_triggers_for_rbr=YES' +# 8) Triggers and Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# 9) Triggers are Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +# ==== References ==== +# +# MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11 +# + +--source include/have_binlog_format_row.inc +--source include/have_innodb.inc +--source include/master-slave.inc + + +--echo # +--echo # Test case 1: KEY on a virtual column with ON DELETE CASCADE +--echo # +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1),(2),(3); + +CREATE TABLE t2 (id INT NOT NULL PRIMARY KEY, + t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; + +INSERT INTO t2 VALUES (90,1,NULL); +INSERT INTO t2 VALUES (91,2,default); + +# Following query results in an assert on slave +DELETE FROM t1 WHERE id=1; +--sync_slave_with_master + +--echo # +--echo # Verify data consistency on slave +--echo # +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc + +--connection master +DROP TABLE t2,t1; +--sync_slave_with_master + +--echo # +--echo # Test case 2: Verify "ON DELETE CASCADE" for parent->child->child scenario +--echo # Parent table: users +--echo # Child tables: matchmaking_groups, matchmaking_group_users +--echo # Parent table: matchmaking_groups +--echo # Child tables: matchmaking_group_users, matchmaking_group_maps +--echo # +--echo # Deleting a row from parent table should be reflected in +--echo # child tables. +--echo # matchmaking_groups->matchmaking_group_users->matchmaking_group_maps +--echo # users->matchmaking_group_users->matchmaking_group_maps +--echo # + +--connection master +CREATE TABLE users (id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, + name VARCHAR(32) NOT NULL DEFAULT '' +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_groups ( + id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, + host_user_id INT UNSIGNED NOT NULL UNIQUE, + v_col INT AS (host_user_id+1) VIRTUAL, KEY (v_col), + CONSTRAINT FOREIGN KEY (host_user_id) REFERENCES users (id) + ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_group_users ( + matchmaking_group_id BIGINT UNSIGNED NOT NULL, + user_id INT UNSIGNED NOT NULL, + v_col1 int as (user_id+1) virtual, KEY (v_col1), + PRIMARY KEY (matchmaking_group_id,user_id), + UNIQUE KEY user_id (user_id), + CONSTRAINT FOREIGN KEY (matchmaking_group_id) + REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE, + CONSTRAINT FOREIGN KEY (user_id) + REFERENCES users (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_group_maps ( + matchmaking_group_id BIGINT UNSIGNED NOT NULL, + map_id TINYINT UNSIGNED NOT NULL, + v_col2 INT AS (map_id+1) VIRTUAL, KEY (v_col2), + PRIMARY KEY (matchmaking_group_id,map_id), + CONSTRAINT FOREIGN KEY (matchmaking_group_id) + REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +--sync_slave_with_master + +--connection master +INSERT INTO users VALUES (NULL,'foo'),(NULL,'bar'); +INSERT INTO matchmaking_groups VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_users VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_maps VALUES (10,55,default),(11,66,default); + +DELETE FROM matchmaking_groups WHERE id = 10; +--sync_slave_with_master + +--echo # +--echo # No rows should be returned as ON DELETE CASCASE should have removed +--echo # corresponding rows from child tables. There should not any mismatch +--echo # of 'id' field between parent->child. +--echo # +SELECT * FROM matchmaking_group_users WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +SELECT * FROM matchmaking_group_maps WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); + +--echo # +--echo # Rows with id=11 should be present +--echo # +SELECT * FROM matchmaking_group_users; +SELECT * FROM matchmaking_group_maps; + +--connection master +DELETE FROM users WHERE id = 2; +--sync_slave_with_master + +--echo # +--echo # No rows should be present in both the child tables +--echo # +SELECT * FROM matchmaking_group_users; +SELECT * FROM matchmaking_group_maps; + +--connection master +DROP TABLE matchmaking_group_maps, matchmaking_group_users, matchmaking_groups, users; +--sync_slave_with_master + +--echo # +--echo # Test case 3: KEY on a virtual column with ON UPDATE CASCADE +--echo # + +--connection master +CREATE TABLE t1 (a INT NOT NULL PRIMARY KEY, b INT NOT NULL) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 80); + +CREATE TABLE t2 (a INT KEY, b INT, + v_col int as (b+1) virtual, KEY (v_col), + CONSTRAINT b FOREIGN KEY (b) REFERENCES t1(a) ON UPDATE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (51, 1, default); +--sync_slave_with_master + +--connection master +UPDATE t1 SET a = 50 WHERE a = 1; + +--echo # +--echo # Master: Verify that ON UPDATE CASCADE works fine +--echo # old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +--echo # +SELECT * FROM t2 WHERE b=50; +--sync_slave_with_master + +--echo # +--echo # Slave: Verify that ON UPDATE CASCADE works fine +--echo # old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +--echo # +SELECT * FROM t2 WHERE b=50; + +--connection master +DROP TABLE t2, t1; +--sync_slave_with_master + +--echo # +--echo # Test case 4: Define triggers on master, their results should be +--echo # replicated as part of row events and they should be +--echo # applied on slave with the default +--echo # slave_run_triggers_for_rbr=NO +--echo # + +# In row-based replication, the binary log contains row changes. It will have +# both the changes made by the statement itself, and the changes made by the +# triggers that were invoked by the statement. Slave server(s) do not need to +# run triggers for row changes they are applying. Hence verify that this +# property remains the same and data should be available as if trigger was +# executed. Please note by default slave_run_triggers_for_rbr=NO. + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t2 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +--sync_slave_with_master + +SHOW GLOBAL VARIABLES LIKE 'slave_run_triggers_for_rbr'; +--echo # +--echo # As two rows are inserted in table 't1', two rows should get inserted +--echo # into table 't2' as part of trigger. +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t2 +--let $assert_text= Table t2 should have two rows. +--source include/assert.inc + +--connection master +DROP TABLE t1,t2; +--sync_slave_with_master + +--echo # +--echo # Test case 5: Define triggers + Foreign Keys on master, their results +--echo # should be replicated as part of row events and master +--echo # and slave should be in sync. +--echo # +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t3 VALUES (1); + +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +--echo # +--echo # As two rows are inserted in table 't1', two rows should get inserted +--echo # into table 't3' as part of trigger. +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc + +--echo # +--echo # Verify ON DELETE CASCASE correctness +--echo # +--connection master +DELETE FROM t1 WHERE id=2; +--sync_slave_with_master + +--connection master +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc +--let $diff_tables= master:test.t3, slave:test.t3 +--source include/diff_tables.inc + +DROP TABLE t3,t2,t1; +--sync_slave_with_master + +# +# Test case: Triggers only on slave +# +--write_file $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc PROCEDURE + if ($slave_run_triggers_for_rbr == '') { + --die !!!ERROR IN TEST: you must set $slave_run_triggers_for_rbr + } + +--connection slave +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +--eval SET GLOBAL slave_run_triggers_for_rbr= $slave_run_triggers_for_rbr; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), + KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +--sync_slave_with_master + +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); + +--connection master +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +if ($slave_run_triggers_for_rbr == 'NO') { +--echo # +--echo # Count must be 0 +--echo # +--let $assert_cond= COUNT(*) = 0 FROM t3 +--let $assert_text= Table t3 should have zero rows. +--source include/assert.inc +} +if ($slave_run_triggers_for_rbr == 'YES') { +--echo # +--echo # Count must be 2 +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc +} + +--connection master +DELETE FROM t1 WHERE id=2; +--sync_slave_with_master +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; + +--echo # +--echo # Verify t1, t2 are consistent on slave. +--echo # +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc + +--connection master +DROP TABLE t3,t2,t1; +--sync_slave_with_master +#END OF +PROCEDURE + +--echo # +--echo # Test case 6: Triggers are present only on slave and +--echo # 'slave_run_triggers_for_rbr=NO' +--echo # +--let $slave_run_triggers_for_rbr=NO +--source $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc + +--echo # +--echo # Test case 7: Triggers are present only on slave and +--echo # 'slave_run_triggers_for_rbr=YES' +--echo # +--let $slave_run_triggers_for_rbr=YES +--source $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc +--remove_file $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc + +# +# Test case: Trigger and Foreign Key are present only on slave +# +--write_file $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc PROCEDURE + if ($slave_run_triggers_for_rbr == '') { + --die !!!ERROR IN TEST: you must set $slave_run_triggers_for_rbr + } + +--connection slave +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +--eval SET GLOBAL slave_run_triggers_for_rbr= $slave_run_triggers_for_rbr; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +--sync_slave_with_master + +# Have foreign key and trigger on slave. +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); + +--connection master +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +if ($slave_run_triggers_for_rbr == 'NO') { +--echo # +--echo # Count must be 0 +--echo # +--let $assert_cond= COUNT(*) = 0 FROM t3 +--let $assert_text= Table t3 should have zero rows. +--source include/assert.inc +} +if ($slave_run_triggers_for_rbr == 'YES') { +--echo # +--echo # Count must be 2 +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc +} + +--connection master +DELETE FROM t1 WHERE id=2; +--echo # t1: Should have one row +SELECT * FROM t1; +--echo # t2: Should have two rows +SELECT * FROM t2; +--sync_slave_with_master +--echo # t1: Should have one row +SELECT * FROM t1; +--echo # t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; + +--connection master +DROP TABLE t3,t2,t1; +--sync_slave_with_master +#END OF +PROCEDURE + +--echo # +--echo # Test case 8: Triggers and Foreign Keys are present only on slave and +--echo # 'slave_run_triggers_for_rbr=NO' +--echo # +--let $slave_run_triggers_for_rbr=NO +--source $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc + +--echo # +--echo # Test case 9: Triggers are Foreign Keys are present only on slave and +--echo # 'slave_run_triggers_for_rbr=YES' +--echo # +--let $slave_run_triggers_for_rbr=YES +--source $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc +--remove_file $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc + +--source include/rpl_end.inc diff --git a/sql/log_event.cc b/sql/log_event.cc index c649e1f64fa..c32f31db1f6 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -10718,7 +10718,7 @@ int Rows_log_event::do_add_row_data(uchar *row_data, size_t length) There was the same problem with MERGE MYISAM tables and so here we try to go the same way. */ -static void restore_empty_query_table_list(LEX *lex) +inline void restore_empty_query_table_list(LEX *lex) { if (lex->first_not_own_table()) (*lex->first_not_own_table()->prev_global)= NULL; @@ -10733,6 +10733,8 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) TABLE* table; DBUG_ENTER("Rows_log_event::do_apply_event(Relay_log_info*)"); int error= 0; + LEX *lex= thd->lex; + uint8 new_trg_event_map= get_trg_event_map(); /* If m_table_id == ~0ULL, then we have a dummy event that does not contain any data. In that case, we just remove all tables in the @@ -10823,27 +10825,29 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) DBUG_ASSERT(!debug_sync_set_action(thd, STRING_WITH_LEN(action))); };); - if (slave_run_triggers_for_rbr) - { - LEX *lex= thd->lex; - uint8 new_trg_event_map= get_trg_event_map(); - - /* - Trigger's procedures work with global table list. So we have to add - rgi->tables_to_lock content there to get trigger's in the list. + /* + Trigger's procedures work with global table list. So we have to add + rgi->tables_to_lock content there to get trigger's in the list. - Then restore_empty_query_table_list() restore the list as it was - */ - DBUG_ASSERT(lex->query_tables == NULL); - if ((lex->query_tables= rgi->tables_to_lock)) - rgi->tables_to_lock->prev_global= &lex->query_tables; + Then restore_empty_query_table_list() restore the list as it was + */ + DBUG_ASSERT(lex->query_tables == NULL); + if ((lex->query_tables= rgi->tables_to_lock)) + rgi->tables_to_lock->prev_global= &lex->query_tables; - for (TABLE_LIST *tables= rgi->tables_to_lock; tables; - tables= tables->next_global) + for (TABLE_LIST *tables= rgi->tables_to_lock; tables; + tables= tables->next_global) + { + if (slave_run_triggers_for_rbr) { tables->trg_event_map= new_trg_event_map; lex->query_tables_last= &tables->next_global; } + else if (!WSREP_ON) + { + tables->slave_fk_event_map= new_trg_event_map; + lex->query_tables_last= &tables->next_global; + } } if (open_and_lock_tables(thd, rgi->tables_to_lock, FALSE, 0)) { @@ -11193,8 +11197,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) } /* remove trigger's tables */ - if (slave_run_triggers_for_rbr) - restore_empty_query_table_list(thd->lex); + restore_empty_query_table_list(thd->lex); #if defined(WITH_WSREP) && defined(HAVE_QUERY_CACHE) if (WSREP(thd) && thd->wsrep_exec_mode == REPL_RECV) @@ -11212,8 +11215,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) DBUG_RETURN(error); err: - if (slave_run_triggers_for_rbr) - restore_empty_query_table_list(thd->lex); + restore_empty_query_table_list(thd->lex); rgi->slave_close_thread_tables(thd); DBUG_RETURN(error); } diff --git a/sql/sql_base.cc b/sql/sql_base.cc index 8e57ea437b6..b8d18abb50c 100644 --- a/sql/sql_base.cc +++ b/sql/sql_base.cc @@ -4341,6 +4341,70 @@ bool table_already_fk_prelocked(TABLE_LIST *tl, LEX_STRING *db, return false; } +/** + Extend the table_list to include foreign tables for prelocking. + + @param[in] thd Thread context. + @param[in] prelocking_ctx Prelocking context of the statement. + @param[in] table_list Table list element for table. + @param[in] sp Routine body. + @param[out] need_prelocking Set to TRUE if method detects that prelocking + required, not changed otherwise. + + @retval FALSE Success. + @retval TRUE Failure (OOM). +*/ +inline bool +prepare_fk_prelocking_list(THD *thd, Query_tables_list *prelocking_ctx, + TABLE_LIST *table_list, bool *need_prelocking, + uint8 op) +{ + List <FOREIGN_KEY_INFO> fk_list; + List_iterator<FOREIGN_KEY_INFO> fk_list_it(fk_list); + FOREIGN_KEY_INFO *fk; + Query_arena *arena, backup; + + arena= thd->activate_stmt_arena_if_needed(&backup); + + table_list->table->file->get_parent_foreign_key_list(thd, &fk_list); + if (thd->is_error()) + { + if (arena) + thd->restore_active_arena(arena, &backup); + return TRUE; + } + + *need_prelocking= TRUE; + + while ((fk= fk_list_it++)) + { + // FK_OPTION_RESTRICT and FK_OPTION_NO_ACTION only need read access + static bool can_write[]= { true, false, true, true, false, true }; + thr_lock_type lock_type; + + if ((op & (1 << TRG_EVENT_DELETE) && can_write[fk->delete_method]) + || (op & (1 << TRG_EVENT_UPDATE) && can_write[fk->update_method])) + lock_type= TL_WRITE_ALLOW_WRITE; + else + lock_type= TL_READ; + + if (table_already_fk_prelocked(prelocking_ctx->query_tables, + fk->foreign_db, fk->foreign_table, + lock_type)) + continue; + + TABLE_LIST *tl= (TABLE_LIST *) thd->alloc(sizeof(TABLE_LIST)); + tl->init_one_table_for_prelocking(fk->foreign_db->str, fk->foreign_db->length, + fk->foreign_table->str, fk->foreign_table->length, + NULL, lock_type, false, table_list->belong_to_view, + op, &prelocking_ctx->query_tables_last); + } + if (arena) + thd->restore_active_arena(arena, &backup); + + return FALSE; +} + /** Defines how prelocking algorithm for DML statements should handle table list @@ -4381,55 +4445,21 @@ handle_table(THD *thd, Query_tables_list *prelocking_ctx, add_tables_and_routines_for_triggers(thd, prelocking_ctx, table_list)) return TRUE; } - if (table_list->table->file->referenced_by_foreign_key()) { - List <FOREIGN_KEY_INFO> fk_list; - List_iterator<FOREIGN_KEY_INFO> fk_list_it(fk_list); - FOREIGN_KEY_INFO *fk; - Query_arena *arena, backup; - - arena= thd->activate_stmt_arena_if_needed(&backup); - - table_list->table->file->get_parent_foreign_key_list(thd, &fk_list); - if (thd->is_error()) - { - if (arena) - thd->restore_active_arena(arena, &backup); - return TRUE; - } - - *need_prelocking= TRUE; - - while ((fk= fk_list_it++)) - { - // FK_OPTION_RESTRICT and FK_OPTION_NO_ACTION only need read access - static bool can_write[]= { true, false, true, true, false, true }; - uint8 op= table_list->trg_event_map; - thr_lock_type lock_type; - - if ((op & (1 << TRG_EVENT_DELETE) && can_write[fk->delete_method]) - || (op & (1 << TRG_EVENT_UPDATE) && can_write[fk->update_method])) - lock_type= TL_WRITE_ALLOW_WRITE; - else - lock_type= TL_READ; - - if (table_already_fk_prelocked(prelocking_ctx->query_tables, - fk->foreign_db, fk->foreign_table, - lock_type)) - continue; - - TABLE_LIST *tl= (TABLE_LIST *) thd->alloc(sizeof(TABLE_LIST)); - tl->init_one_table_for_prelocking(fk->foreign_db->str, fk->foreign_db->length, - fk->foreign_table->str, fk->foreign_table->length, - NULL, lock_type, false, table_list->belong_to_view, - op, &prelocking_ctx->query_tables_last); - } - if (arena) - thd->restore_active_arena(arena, &backup); + return (prepare_fk_prelocking_list(thd, prelocking_ctx, table_list, + need_prelocking, + table_list->trg_event_map)); } } + else if (table_list->slave_fk_event_map && + table_list->table->file->referenced_by_foreign_key()) + { + return (prepare_fk_prelocking_list(thd, prelocking_ctx, + table_list, need_prelocking, + table_list->slave_fk_event_map)); + } return FALSE; } diff --git a/sql/table.h b/sql/table.h index 9a864f7ce9f..57706655d9b 100644 --- a/sql/table.h +++ b/sql/table.h @@ -2277,8 +2277,12 @@ struct TABLE_LIST Indicates what triggers we need to pre-load for this TABLE_LIST when opening an associated TABLE. This is filled after the parsed tree is created. + + slave_fk_event_map is filled on the slave side with bitmaps value + representing row-based event operation to help find and prelock + possible FK constrain-related child tables. */ - uint8 trg_event_map; + uint8 trg_event_map, slave_fk_event_map; /* TRUE <=> this table is a const one and was optimized away. */ bool optimized_away;

2 2

[Maria-developers] 3f48bc08b92: MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11
by sujatha 31 Dec '20

31 Dec '20

revision-id: 3f48bc08b92b2734c2af7408c246b96338675205 (mariadb-10.2.31-646-g3f48bc08b92) parent(s): 78292047a4747ccd9210ba36a185a1dbe825de89 author: Sujatha committer: Sujatha timestamp: 2020-12-31 15:43:53 +0530 message: MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11 Problem: ======= Upon deleting or updating a row in a parent table (with primary key), if the child table has virtual column and an associated key with ON UPDATE CASCADE/ON DELETE CASCADE, it will result in slave crash. Analysis: ======== Tables which are related through foreign key require prelocking similar to triggers. i.e If a table has triggers/foreign keys we should add all tables and routines used by them to the prelocking set. This prelocking happens during 'open_and_lock_tables' call. Each table being opened is checked for foreign key references. If foreign key reference exists then the child table is opened and it is linked to the table_list. Upon any modification to parent table its corresponding child tables are retried from table_list and they are updated accordingly. This prelocking work fine on master. On slave prelocking works for following cases. - Statement/mixed based replication - In row based replication when trigger execution is enabled through 'slave_run_triggers_for_rbr=YES/LOGGING/ENFORCE' Otherwise it results in an assert/crash, as the parent table will not find the corresponding child table and it will be NULL. Dereferencing NULL pointer leads to slave server exit. Fix: === Introduce a new 'slave_fk_event_map' flag similar to 'trg_event_map'. This flag will ensure that when foreign key is enabled in row based replication all the parent and child tables are prelocked, so that parent is able to locate the child table. --- mysql-test/suite/rpl/r/rpl_row_vcol_crash.result | 381 +++++++++++++++++++++ mysql-test/suite/rpl/t/rpl_row_vcol_crash.test | 410 +++++++++++++++++++++++ sql/log_event.cc | 44 ++- sql/sql_base.cc | 6 +- sql/table.h | 6 +- 5 files changed, 821 insertions(+), 26 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result b/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result new file mode 100644 index 00000000000..e2a5d9b65e5 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result @@ -0,0 +1,381 @@ +include/master-slave.inc +[connection master] +# +# Test case: KEY on a virtual column with ON DELETE CASCADE +# +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1),(2),(3); +CREATE TABLE t2 (id INT NOT NULL PRIMARY KEY, +t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (90,1,NULL); +INSERT INTO t2 VALUES (91,2,default); +DELETE FROM t1 WHERE id=1; +connection slave; +# +# Verify data consistency on slave +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t2,t1; +connection slave; +# +# Test case: Verify cascading on delete with multiple levels of child +# tables works fine. +# Parent table: users +# Child tables: matchmaking_groups, matchmaking_group_users +# Parent table: matchmaking_groups +# Child tables: matchmaking_group_users, matchmaking_group_maps +# +# Verify ON DELETE CASCASE works fine when a row is deleted +# from parent table users +# matchmaking_groups->matchmaking_group_users->matchmaking_group_maps +# users->matchmaking_group_users->matchmaking_group_maps +# +connection master; +CREATE TABLE users (id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, +name VARCHAR(32) NOT NULL DEFAULT '' +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_groups ( +id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, +host_user_id INT UNSIGNED NOT NULL UNIQUE, +v_col INT AS (host_user_id+1) VIRTUAL, KEY (v_col), +CONSTRAINT FOREIGN KEY (host_user_id) REFERENCES users (id) +ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_group_users ( +matchmaking_group_id BIGINT UNSIGNED NOT NULL, +user_id INT UNSIGNED NOT NULL, +v_col1 int as (user_id+1) virtual, KEY (v_col1), +PRIMARY KEY (matchmaking_group_id,user_id), +UNIQUE KEY user_id (user_id), +CONSTRAINT FOREIGN KEY (matchmaking_group_id) +REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE, +CONSTRAINT FOREIGN KEY (user_id) +REFERENCES users (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_group_maps ( +matchmaking_group_id BIGINT UNSIGNED NOT NULL, +map_id TINYINT UNSIGNED NOT NULL, +v_col2 INT AS (map_id+1) VIRTUAL, KEY (v_col2), +PRIMARY KEY (matchmaking_group_id,map_id), +CONSTRAINT FOREIGN KEY (matchmaking_group_id) +REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +connection slave; +connection master; +INSERT INTO users VALUES (NULL,'foo'),(NULL,'bar'); +INSERT INTO matchmaking_groups VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_users VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_maps VALUES (10,55,default),(11,66,default); +DELETE FROM matchmaking_groups WHERE id = 10; +connection slave; +# +# No rows should be returned as ON DELETE CASCASE should have removed +# corresponding rows from child tables. There should not any mismatch +# of 'id' field between parent->child. +# +SELECT * FROM matchmaking_group_users WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +matchmaking_group_id user_id v_col1 +SELECT * FROM matchmaking_group_maps WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +matchmaking_group_id map_id v_col2 +# +# Rows with id=11 should be present +# +SELECT * FROM matchmaking_group_users; +matchmaking_group_id user_id v_col1 +11 2 3 +SELECT * FROM matchmaking_group_maps; +matchmaking_group_id map_id v_col2 +11 66 67 +connection master; +DELETE FROM users WHERE id = 2; +connection slave; +# +# No rows should be present in both the child tables +# +SELECT * FROM matchmaking_group_users; +matchmaking_group_id user_id v_col1 +SELECT * FROM matchmaking_group_maps; +matchmaking_group_id map_id v_col2 +connection master; +DROP TABLE matchmaking_group_maps, matchmaking_group_users, matchmaking_groups, users; +connection slave; +# +# Test case: ON UPDATE CASCADE scenario +# +connection master; +CREATE TABLE t1 (a INT NOT NULL PRIMARY KEY, b INT NOT NULL) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 80); +CREATE TABLE t2 (a INT KEY, b INT, +v_col int as (b+1) virtual, KEY (v_col), +CONSTRAINT b FOREIGN KEY (b) REFERENCES t1(a) ON UPDATE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (51, 1, default); +connection slave; +connection master; +UPDATE t1 SET a = 50 WHERE a = 1; +# +# Master: Verify that ON UPDATE CASCADE works fine +# old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +# +SELECT * FROM t2 WHERE b=50; +a b v_col +51 50 51 +connection slave; +# +# Slave: Verify that ON UPDATE CASCADE works fine +# old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +# +SELECT * FROM t2 WHERE b=50; +a b v_col +51 50 51 +connection master; +DROP TABLE t2, t1; +connection slave; +# +# Test case: When triggers are defined on master they should be +# replicated as part of row events and they should be +# applied on slave with the default +# slave_run_tiggers_for_rbr=NO +# +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t2 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +connection slave; +SHOW GLOBAL VARIABLES LIKE 'slave_run_triggers_for_rbr'; +Variable_name Value +slave_run_triggers_for_rbr NO +# +# As two rows are inserted in table 't1', two rows should get inserted +# into table 't2' as part of trigger. +# +include/assert.inc [Table t2 should have two rows.] +connection master; +DROP TABLE t1,t2; +connection slave; +# +# Test case: On master create triggers and tables with foreign key +# relation. Upon replication to slave verify their results +# are fine. Master and Slave should be in sync. +# +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t3 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# As two rows are inserted in table 't1', two rows should get inserted +# into table 't3' as part of trigger. +# +include/assert.inc [Table t3 should have two rows.] +# +# Verify ON DELETE CASCASE correctness +# +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +connection master; +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +include/diff_tables.inc [master:test.t3, slave:test.t3] +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case: Triggers are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= NO;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr NO +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), +KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 0 +# +include/assert.inc [Table t3 should have zero rows.] +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +# +# Verify t1,t2 are consistent on slave. +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case: Triggers are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= YES;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr YES +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), +KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 2 +# +include/assert.inc [Table t3 should have two rows.] +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +# +# Verify t1,t2 are consistent on slave. +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case: Triggers and Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= NO;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr NO +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 0 +# +include/assert.inc [Table t3 should have zero rows.] +connection master; +DELETE FROM t1 WHERE id=2; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have two rows +SELECT * FROM t2; +t1_id v_col +2 3 +3 4 +connection slave; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +t1_id v_col +3 4 +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case: Triggers are Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= YES;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr YES +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 2 +# +include/assert.inc [Table t3 should have two rows.] +connection master; +DELETE FROM t1 WHERE id=2; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have two rows +SELECT * FROM t2; +t1_id v_col +2 3 +3 4 +connection slave; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +t1_id v_col +3 4 +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +connection master; +DROP TABLE t3,t2,t1; +connection slave; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test b/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test new file mode 100644 index 00000000000..2ed7c219aeb --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test @@ -0,0 +1,410 @@ +# ==== Purpose ==== +# +# Test verifies that, slave doesn't report any assert on UPDATE or DELETE of +# row which tries to update the virtual columns with associated KEYs. +# +# ==== References ==== +# +# MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11 +# + +--source include/have_binlog_format_row.inc +--source include/have_innodb.inc +--source include/master-slave.inc + + +--echo # +--echo # Test case: KEY on a virtual column with ON DELETE CASCADE +--echo # +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1),(2),(3); + +CREATE TABLE t2 (id INT NOT NULL PRIMARY KEY, + t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; + +INSERT INTO t2 VALUES (90,1,NULL); +INSERT INTO t2 VALUES (91,2,default); + +# Following query results in an assert on slave +DELETE FROM t1 WHERE id=1; +--sync_slave_with_master + +--echo # +--echo # Verify data consistency on slave +--echo # +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc + +--connection master +DROP TABLE t2,t1; +--sync_slave_with_master + +--echo # +--echo # Test case: Verify cascading on delete with multiple levels of child +--echo # tables works fine. +--echo # Parent table: users +--echo # Child tables: matchmaking_groups, matchmaking_group_users +--echo # Parent table: matchmaking_groups +--echo # Child tables: matchmaking_group_users, matchmaking_group_maps +--echo # +--echo # Verify ON DELETE CASCASE works fine when a row is deleted +--echo # from parent table users +--echo # matchmaking_groups->matchmaking_group_users->matchmaking_group_maps +--echo # users->matchmaking_group_users->matchmaking_group_maps +--echo # + +--connection master +CREATE TABLE users (id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, + name VARCHAR(32) NOT NULL DEFAULT '' +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_groups ( + id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, + host_user_id INT UNSIGNED NOT NULL UNIQUE, + v_col INT AS (host_user_id+1) VIRTUAL, KEY (v_col), + CONSTRAINT FOREIGN KEY (host_user_id) REFERENCES users (id) + ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_group_users ( + matchmaking_group_id BIGINT UNSIGNED NOT NULL, + user_id INT UNSIGNED NOT NULL, + v_col1 int as (user_id+1) virtual, KEY (v_col1), + PRIMARY KEY (matchmaking_group_id,user_id), + UNIQUE KEY user_id (user_id), + CONSTRAINT FOREIGN KEY (matchmaking_group_id) + REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE, + CONSTRAINT FOREIGN KEY (user_id) + REFERENCES users (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_group_maps ( + matchmaking_group_id BIGINT UNSIGNED NOT NULL, + map_id TINYINT UNSIGNED NOT NULL, + v_col2 INT AS (map_id+1) VIRTUAL, KEY (v_col2), + PRIMARY KEY (matchmaking_group_id,map_id), + CONSTRAINT FOREIGN KEY (matchmaking_group_id) + REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +--sync_slave_with_master + +--connection master +INSERT INTO users VALUES (NULL,'foo'),(NULL,'bar'); +INSERT INTO matchmaking_groups VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_users VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_maps VALUES (10,55,default),(11,66,default); + +DELETE FROM matchmaking_groups WHERE id = 10; +--sync_slave_with_master + +--echo # +--echo # No rows should be returned as ON DELETE CASCASE should have removed +--echo # corresponding rows from child tables. There should not any mismatch +--echo # of 'id' field between parent->child. +--echo # +SELECT * FROM matchmaking_group_users WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +SELECT * FROM matchmaking_group_maps WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); + +--echo # +--echo # Rows with id=11 should be present +--echo # +SELECT * FROM matchmaking_group_users; +SELECT * FROM matchmaking_group_maps; + +--connection master +DELETE FROM users WHERE id = 2; +--sync_slave_with_master + +--echo # +--echo # No rows should be present in both the child tables +--echo # +SELECT * FROM matchmaking_group_users; +SELECT * FROM matchmaking_group_maps; + +--connection master +DROP TABLE matchmaking_group_maps, matchmaking_group_users, matchmaking_groups, users; +--sync_slave_with_master + +--echo # +--echo # Test case: ON UPDATE CASCADE scenario +--echo # + +--connection master +CREATE TABLE t1 (a INT NOT NULL PRIMARY KEY, b INT NOT NULL) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 80); + +CREATE TABLE t2 (a INT KEY, b INT, + v_col int as (b+1) virtual, KEY (v_col), + CONSTRAINT b FOREIGN KEY (b) REFERENCES t1(a) ON UPDATE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (51, 1, default); +--sync_slave_with_master + +--connection master +UPDATE t1 SET a = 50 WHERE a = 1; + +--echo # +--echo # Master: Verify that ON UPDATE CASCADE works fine +--echo # old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +--echo # +SELECT * FROM t2 WHERE b=50; +--sync_slave_with_master + +--echo # +--echo # Slave: Verify that ON UPDATE CASCADE works fine +--echo # old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +--echo # +SELECT * FROM t2 WHERE b=50; + +--connection master +DROP TABLE t2, t1; +--sync_slave_with_master + +--echo # +--echo # Test case: When triggers are defined on master they should be +--echo # replicated as part of row events and they should be +--echo # applied on slave with the default +--echo # slave_run_tiggers_for_rbr=NO +--echo # + +# In row-based replication, the binary log contains row changes. It will have +# both the changes made by the statement itself, and the changes made by the +# triggers that were invoked by the statement. Slave server(s) do not need to +# run triggers for row changes they are applying. Hence verify that this +# property remains the same and data should be available as if trigger was +# executed. Please note by default slave_run_tiggers_for_rbr=NO. + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t2 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +--sync_slave_with_master + +SHOW GLOBAL VARIABLES LIKE 'slave_run_triggers_for_rbr'; +--echo # +--echo # As two rows are inserted in table 't1', two rows should get inserted +--echo # into table 't2' as part of trigger. +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t2 +--let $assert_text= Table t2 should have two rows. +--source include/assert.inc + +--connection master +DROP TABLE t1,t2; +--sync_slave_with_master + +--echo # +--echo # Test case: On master create triggers and tables with foreign key +--echo # relation. Upon replication to slave verify their results +--echo # are fine. Master and Slave should be in sync. +--echo # +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t3 VALUES (1); + +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +--echo # +--echo # As two rows are inserted in table 't1', two rows should get inserted +--echo # into table 't3' as part of trigger. +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc + +--echo # +--echo # Verify ON DELETE CASCASE correctness +--echo # +--connection master +DELETE FROM t1 WHERE id=2; +--sync_slave_with_master + +--connection master +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc +--let $diff_tables= master:test.t3, slave:test.t3 +--source include/diff_tables.inc + +DROP TABLE t3,t2,t1; +--sync_slave_with_master + +# +# Test case: Triggers only on slave +# +--write_file $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc PROCEDURE + if ($slave_run_triggers_for_rbr == '') { + --die !!!ERROR IN TEST: you must set $slave_run_triggers_for_rbr + } + +--connection slave +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +--eval SET GLOBAL slave_run_triggers_for_rbr= $slave_run_triggers_for_rbr; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), + KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +--sync_slave_with_master + +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); + +--connection master +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +if ($slave_run_triggers_for_rbr == 'NO') { +--echo # +--echo # Count must be 0 +--echo # +--let $assert_cond= COUNT(*) = 0 FROM t3 +--let $assert_text= Table t3 should have zero rows. +--source include/assert.inc +} +if ($slave_run_triggers_for_rbr == 'YES') { +--echo # +--echo # Count must be 2 +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc +} + +--connection master +DELETE FROM t1 WHERE id=2; +--sync_slave_with_master +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; + +--echo # +--echo # Verify t1,t2 are consistent on slave. +--echo # +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc + +--connection master +DROP TABLE t3,t2,t1; +--sync_slave_with_master +#END OF +PROCEDURE + +--echo # +--echo # Test case: Triggers are present only on slave and +--echo # 'slave_run_triggers_for_rbr=NO' +--echo # +--let $slave_run_triggers_for_rbr=NO +--source $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc + +--echo # +--echo # Test case: Triggers are present only on slave and +--echo # 'slave_run_triggers_for_rbr=YES' +--echo # +--let $slave_run_triggers_for_rbr=YES +--source $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc +--remove_file $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc + +# +# Test case: Trigger and Foreign Key are present only on slave +# +--write_file $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc PROCEDURE + if ($slave_run_triggers_for_rbr == '') { + --die !!!ERROR IN TEST: you must set $slave_run_triggers_for_rbr + } + +--connection slave +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +--eval SET GLOBAL slave_run_triggers_for_rbr= $slave_run_triggers_for_rbr; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +--sync_slave_with_master + +# Have foreign key and trigger on slave. +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); + +--connection master +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +if ($slave_run_triggers_for_rbr == 'NO') { +--echo # +--echo # Count must be 0 +--echo # +--let $assert_cond= COUNT(*) = 0 FROM t3 +--let $assert_text= Table t3 should have zero rows. +--source include/assert.inc +} +if ($slave_run_triggers_for_rbr == 'YES') { +--echo # +--echo # Count must be 2 +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc +} + +--connection master +DELETE FROM t1 WHERE id=2; +--echo # t1: Should have one row +SELECT * FROM t1; +--echo # t2: Should have two rows +SELECT * FROM t2; +--sync_slave_with_master +--echo # t1: Should have one row +SELECT * FROM t1; +--echo # t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; + +--connection master +DROP TABLE t3,t2,t1; +--sync_slave_with_master +#END OF +PROCEDURE + +--echo # +--echo # Test case: Triggers and Foreign Keys are present only on slave and +--echo # 'slave_run_triggers_for_rbr=NO' +--echo # +--let $slave_run_triggers_for_rbr=NO +--source $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc + +--echo # +--echo # Test case: Triggers are Foreign Keys are present only on slave and +--echo # 'slave_run_triggers_for_rbr=YES' +--echo # +--let $slave_run_triggers_for_rbr=YES +--source $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc +--remove_file $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc + +--source include/rpl_end.inc diff --git a/sql/log_event.cc b/sql/log_event.cc index c649e1f64fa..10aa0afbad8 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -10718,7 +10718,7 @@ int Rows_log_event::do_add_row_data(uchar *row_data, size_t length) There was the same problem with MERGE MYISAM tables and so here we try to go the same way. */ -static void restore_empty_query_table_list(LEX *lex) +inline void restore_empty_query_table_list(LEX *lex) { if (lex->first_not_own_table()) (*lex->first_not_own_table()->prev_global)= NULL; @@ -10733,6 +10733,8 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) TABLE* table; DBUG_ENTER("Rows_log_event::do_apply_event(Relay_log_info*)"); int error= 0; + LEX *lex= thd->lex; + uint8 new_trg_event_map= get_trg_event_map(); /* If m_table_id == ~0ULL, then we have a dummy event that does not contain any data. In that case, we just remove all tables in the @@ -10823,27 +10825,25 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) DBUG_ASSERT(!debug_sync_set_action(thd, STRING_WITH_LEN(action))); };); - if (slave_run_triggers_for_rbr) - { - LEX *lex= thd->lex; - uint8 new_trg_event_map= get_trg_event_map(); - - /* - Trigger's procedures work with global table list. So we have to add - rgi->tables_to_lock content there to get trigger's in the list. + /* + Trigger's procedures work with global table list. So we have to add + rgi->tables_to_lock content there to get trigger's in the list. - Then restore_empty_query_table_list() restore the list as it was - */ - DBUG_ASSERT(lex->query_tables == NULL); - if ((lex->query_tables= rgi->tables_to_lock)) - rgi->tables_to_lock->prev_global= &lex->query_tables; + Then restore_empty_query_table_list() restore the list as it was + */ + DBUG_ASSERT(lex->query_tables == NULL); + if ((lex->query_tables= rgi->tables_to_lock)) + rgi->tables_to_lock->prev_global= &lex->query_tables; - for (TABLE_LIST *tables= rgi->tables_to_lock; tables; - tables= tables->next_global) - { + for (TABLE_LIST *tables= rgi->tables_to_lock; tables; + tables= tables->next_global) + { + if (slave_run_triggers_for_rbr) tables->trg_event_map= new_trg_event_map; - lex->query_tables_last= &tables->next_global; - } + else + tables->slave_fk_event_map= new_trg_event_map; + + lex->query_tables_last= &tables->next_global; } if (open_and_lock_tables(thd, rgi->tables_to_lock, FALSE, 0)) { @@ -11193,8 +11193,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) } /* remove trigger's tables */ - if (slave_run_triggers_for_rbr) - restore_empty_query_table_list(thd->lex); + restore_empty_query_table_list(thd->lex); #if defined(WITH_WSREP) && defined(HAVE_QUERY_CACHE) if (WSREP(thd) && thd->wsrep_exec_mode == REPL_RECV) @@ -11212,8 +11211,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) DBUG_RETURN(error); err: - if (slave_run_triggers_for_rbr) - restore_empty_query_table_list(thd->lex); + restore_empty_query_table_list(thd->lex); rgi->slave_close_thread_tables(thd); DBUG_RETURN(error); } diff --git a/sql/sql_base.cc b/sql/sql_base.cc index 8e57ea437b6..a7cddbca16a 100644 --- a/sql/sql_base.cc +++ b/sql/sql_base.cc @@ -4371,7 +4371,8 @@ handle_table(THD *thd, Query_tables_list *prelocking_ctx, /* We rely on a caller to check that table is going to be changed. */ DBUG_ASSERT(table_list->lock_type >= TL_WRITE_ALLOW_WRITE); - if (table_list->trg_event_map) + if (table_list->trg_event_map || + (thd->rgi_slave && thd->is_current_stmt_binlog_format_row())) { if (table_list->table->triggers) { @@ -4405,7 +4406,8 @@ handle_table(THD *thd, Query_tables_list *prelocking_ctx, { // FK_OPTION_RESTRICT and FK_OPTION_NO_ACTION only need read access static bool can_write[]= { true, false, true, true, false, true }; - uint8 op= table_list->trg_event_map; + uint8 op= ((table_list->trg_event_map) ? table_list->trg_event_map : + table_list->slave_fk_event_map); thr_lock_type lock_type; if ((op & (1 << TRG_EVENT_DELETE) && can_write[fk->delete_method]) diff --git a/sql/table.h b/sql/table.h index 9a864f7ce9f..57706655d9b 100644 --- a/sql/table.h +++ b/sql/table.h @@ -2277,8 +2277,12 @@ struct TABLE_LIST Indicates what triggers we need to pre-load for this TABLE_LIST when opening an associated TABLE. This is filled after the parsed tree is created. + + slave_fk_event_map is filled on the slave side with bitmaps value + representing row-based event operation to help find and prelock + possible FK constrain-related child tables. */ - uint8 trg_event_map; + uint8 trg_event_map, slave_fk_event_map; /* TRUE <=> this table is a const one and was optimized away. */ bool optimized_away;

1 0

Re: [Maria-developers] [Commits] cdc305c8dd8: MDEV-19620: Changing join_buffer_size causes different results
by Sergey Petrunia 29 Dec '20

29 Dec '20

Hi Varun, Please find input below. On Mon, Dec 28, 2020 at 02:34:40PM +0530, varun wrote: > revision-id: cdc305c8dd89a726e09e5fe70ff890d06609cbfb (mariadb-10.3.21-309-gcdc305c8dd8) > parent(s): 043bd85a574a88856ab9c6d497e682ed06fe45e9 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-12-28 14:12:14 +0530 > message: > > MDEV-19620: Changing join_buffer_size causes different results > > The scenario here is that query refinement phase decides to use a hash join. > When the join buffers are allocated in the JOIN::init_join_caches, for a table > the size exceeds the value for join_buffer_space_limit (which is the limit of the > space available for all join buffers). When this happens then we disallow join > buffering for the table, this is done in the revise_cache_usage and set_join_cache_denial. > > In this issue the hash key is created on an index for which ref access is possible, so > when we disallow hash join then instead of switching to REF access we switch to a table > scan. This is a problem because the equijoin conditions for which a lookup can be made > are not attached to the table(or are not evaluated for the table). This leads to incorrect > results. > > The fix here would be to switch to using a lookup because it was picked by the join planner > to be more efficient than the table scan. > > --- > mysql-test/main/join_cache.result | 138 ++++++++++++++++++++++++++++++++++++++ > mysql-test/main/join_cache.test | 105 +++++++++++++++++++++++++++++ > sql/sql_select.cc | 69 +++++++++++++++---- > sql/sql_select.h | 6 ++ > 4 files changed, 304 insertions(+), 14 deletions(-) > > diff --git a/mysql-test/main/join_cache.result b/mysql-test/main/join_cache.result > index 3d1d91df997..e58503f422f 100644 > --- a/mysql-test/main/join_cache.result > +++ b/mysql-test/main/join_cache.result > @@ -6128,4 +6128,142 @@ EXPLAIN > } > } > drop table t1,t2,t3; > +# > +# MDEV-19620: Changing join_buffer_size causes different results > +# > +SET @save_join_cache_level= @@join_cache_level; > +SET @save_join_buffer_size= @@join_buffer_size; > +SET @save_join_buffer_space_limit= @@join_buffer_space_limit; > +SET join_cache_level = 3; > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT PRIMARY KEY, i3 VARCHAR(300), c3 VARCHAR(500)) ENGINE=MyISAM; > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','p'),(3,'1','q'); > +INSERT INTO t2 VALUES (4,'7','g'),(5,'4','p'),(6,'1','q'); > +INSERT INTO t2 VALUES (16,'7','g'),(17,'4','p'),(28,'1','q'); > +# > +# Hash join + table Scan on t2 > +# > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 ALL NULL NULL NULL NULL 9 Using where > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +i2 c2 pk3 i3 c3 > +1 v NULL NULL NULL > +7 s NULL NULL NULL > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 hash_ALL NULL #hash#$hj 503 test.t1.c2 9 Using where; Using join buffer (flat, BNLH join) > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +i2 c2 pk3 i3 c3 > +1 v NULL NULL NULL > +7 s NULL NULL NULL > +# > +# HASH join + ref access on t2 > +# > +ALTER TABLE t2 ADD KEY k1(c3); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 ref k1 k1 503 test.t1.c2 2 Using where > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +i2 c2 pk3 i3 c3 > +1 v NULL NULL NULL > +7 s NULL NULL NULL > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 hash_ALL k1 #hash#k1 503 test.t1.c2 9 Using where; Using join buffer (flat, BNLH join) > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +i2 c2 pk3 i3 c3 > +1 v NULL NULL NULL > +7 s NULL NULL NULL > +# > +# Hash join + index scan on t2 > +# > +ALTER TABLE t2 DROP KEY k1; > +ALTER TABLE t2 ADD KEY k1(i3,c3); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 index NULL k1 806 NULL 9 Using where; Using index > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +c3 i3 i2 c2 > +NULL NULL 1 v > +NULL NULL 7 s > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 hash_index NULL #hash#$hj:k1 503:806 test.t1.c2 9 Using where; Using index; Using join buffer (flat, BNLH join) > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +c3 i3 i2 c2 > +NULL NULL 1 v > +NULL NULL 7 s > +DROP TABLE t1,t2; > +# > +# Hash join + range scan on t2 > +# > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)); > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT PRIMARY KEY, i3 VARCHAR(300), c3 VARCHAR(500), INDEX(i3,c3)); > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','s'),(3,'1','q'); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 Using where > +1 SIMPLE t2 range i3 i3 303 NULL 2 Using index condition; Using where > +SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +i2 c2 pk3 i3 c3 > +7 s 2 4 s > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 Using where > +1 SIMPLE t2 hash_range i3 #hash#$hj:i3 503:303 test.t1.c2 2 Using where; Using join buffer (flat, BNLH join) > +SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +i2 c2 pk3 i3 c3 > +7 s 2 4 s > +DROP TABLE t1,t2; > +# > +# Hash join + eq ref access on t2 > +# > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)); > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT, i3 VARCHAR(300), c3 VARCHAR(500) PRIMARY KEY); > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','s'),(3,'1','q'); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 eq_ref PRIMARY PRIMARY 502 test.t1.c2 1 Using where > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +c3 i3 i2 c2 > +NULL NULL 1 v > +s 4 7 s > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 hash_ALL PRIMARY #hash#PRIMARY 502 test.t1.c2 3 Using where; Using join buffer (flat, BNLH join) > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +c3 i3 i2 c2 > +NULL NULL 1 v > +s 4 7 s > +DROP TABLE t1,t2; > +SET @save_join_cache_level= @@join_cache_level; > +SET @save_join_buffer_size= @@join_buffer_size; > +SET @save_join_buffer_space_limit= @@join_buffer_space_limit; > set @@optimizer_switch=@save_optimizer_switch; > diff --git a/mysql-test/main/join_cache.test b/mysql-test/main/join_cache.test > index 91339c2cb21..6670c62516b 100644 > --- a/mysql-test/main/join_cache.test > +++ b/mysql-test/main/join_cache.test > @@ -4054,5 +4054,110 @@ where > > drop table t1,t2,t3; > > +--echo # > +--echo # MDEV-19620: Changing join_buffer_size causes different results > +--echo # > + > +SET @save_join_cache_level= @@join_cache_level; > +SET @save_join_buffer_size= @@join_buffer_size; > +SET @save_join_buffer_space_limit= @@join_buffer_space_limit; > +SET join_cache_level = 3; > + > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT PRIMARY KEY, i3 VARCHAR(300), c3 VARCHAR(500)) ENGINE=MyISAM; > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','p'),(3,'1','q'); > +INSERT INTO t2 VALUES (4,'7','g'),(5,'4','p'),(6,'1','q'); > +INSERT INTO t2 VALUES (16,'7','g'),(17,'4','p'),(28,'1','q'); > + > +--echo # > +--echo # Hash join + table Scan on t2 > +--echo # > + > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > + > +--echo # > +--echo # HASH join + ref access on t2 > +--echo # > + > +ALTER TABLE t2 ADD KEY k1(c3); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > + > +--echo # > +--echo # Hash join + index scan on t2 > +--echo # > +ALTER TABLE t2 DROP KEY k1; > +ALTER TABLE t2 ADD KEY k1(i3,c3); > + > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > + > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +DROP TABLE t1,t2; > + > +--echo # > +--echo # Hash join + range scan on t2 > +--echo # > + > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)); > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT PRIMARY KEY, i3 VARCHAR(300), c3 VARCHAR(500), INDEX(i3,c3)); > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','s'),(3,'1','q'); > + > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > + > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > + > +DROP TABLE t1,t2; > + > +--echo # > +--echo # Hash join + eq ref access on t2 > +--echo # > + > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)); > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT, i3 VARCHAR(300), c3 VARCHAR(500) PRIMARY KEY); > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','s'),(3,'1','q'); > + > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > + > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +DROP TABLE t1,t2; > + > +SET @save_join_cache_level= @@join_cache_level; > +SET @save_join_buffer_size= @@join_buffer_size; > +SET @save_join_buffer_space_limit= @@join_buffer_space_limit; > + > # The following command must be the last one in the file > set @@optimizer_switch=@save_optimizer_switch; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 203422f0b43..8b7deb98dc6 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -111,6 +111,7 @@ static bool best_extension_by_limited_search(JOIN *join, > uint prune_level, > uint use_cond_selectivity); > static uint determine_search_depth(JOIN* join); > +static void pick_table_access_method(JOIN_TAB *tab); > C_MODE_START > static int join_tab_cmp(const void *dummy, const void* ptr1, const void* ptr2); > static int join_tab_cmp_straight(const void *dummy, const void* ptr1, const void* ptr2); > @@ -10081,6 +10082,7 @@ static bool create_ref_for_key(JOIN *join, JOIN_TAB *j, > j->ref.disable_cache= FALSE; > j->ref.null_ref_part= NO_REF_PART; > j->ref.const_ref_part_map= 0; > + j->ref.not_null_keyparts= 0; > keyuse=org_keyuse; > > store_key **ref_key= j->ref.key_copy; > @@ -10173,23 +10175,12 @@ static bool create_ref_for_key(JOIN *join, JOIN_TAB *j, > } > } /* not ftkey */ > *ref_key=0; // end_marker > + j->ref.not_null_keyparts= not_null_keyparts; > if (j->type == JT_FT) > DBUG_RETURN(0); > - ulong key_flags= j->table->actual_key_flags(keyinfo); > if (j->type == JT_CONST) > j->table->const_table= 1; > - else if (!((keyparts == keyinfo->user_defined_key_parts && > - ( > - (key_flags & (HA_NOSAME | HA_NULL_PART_KEY)) == HA_NOSAME || > - /* Unique key and all keyparts are NULL rejecting */ > - ((key_flags & HA_NOSAME) && keyparts == not_null_keyparts) > - )) || > - /* true only for extended keys */ > - (keyparts > keyinfo->user_defined_key_parts && > - MY_TEST(key_flags & HA_EXT_NOSAME) && > - keyparts == keyinfo->ext_key_parts) > - ) || > - null_ref_key) > + else if (!j->is_eq_ref_access()|| null_ref_key) > { > /* Must read with repeat */ > j->type= null_ref_key ? JT_REF_OR_NULL : JT_REF; > @@ -11582,11 +11573,25 @@ void set_join_cache_denial(JOIN_TAB *join_tab) > don't do join buffering for the first table in sjm nest. > */ > join_tab[-1].next_select= sub_select; > - if (join_tab->type == JT_REF && join_tab->is_ref_for_hash_join()) > + if ((join_tab->type == JT_REF || join_tab->type == JT_HASH) && > + join_tab->is_ref_for_hash_join()) > { > join_tab->type= JT_ALL; > join_tab->ref.key_parts= 0; Do we ever get here if join_tab->type == JT_REF? I think we don't. If this is so, please remove it. > } > + > + if (join_tab->type == JT_HASH && !join_tab->is_ref_for_hash_join()) > + { > + join_tab->type= join_tab->is_eq_ref_access() ? JT_EQ_REF : JT_REF; > + pick_table_access_method(join_tab); > + } Also please add a comment explaining that the join optimizer's choice was ref access and we fall back to that. > + > + if (join_tab->type == JT_HASH_NEXT) > + { > + join_tab->type = JT_NEXT; > + DBUG_ASSERT(join_tab->ref.key_parts == 0); > + } > + > join_tab->join->return_tab= join_tab; > } > } > @@ -27954,6 +27959,42 @@ void JOIN_TAB::partial_cleanup() > } > > > +/* > + @brief > + Check if the access method for the table is EQ_REF access or not > + > + @retval > + TRUE EQ_REF access > + FALSE Otherwise > +*/ > +bool JOIN_TAB::is_eq_ref_access() > +{ > + > + KEY *keyinfo; > + if (!is_hash_join_key_no(ref.key)) > + keyinfo= table->key_info + ref.key; > + else > + keyinfo= hj_key; This code now looks as if it was possible that hash join is used and also eq_ref is used. Is this really possible? I've tried catching an example of this with MTR and I wasn't able to do so. Maybe we should just return false if hash join is used and also a comment about this? > + > + uint keyparts= ref.key_parts; > + ulong key_flags= table->actual_key_flags(keyinfo); > + if ( (keyparts == keyinfo->user_defined_key_parts && > + ( > + (key_flags & (HA_NOSAME | HA_NULL_PART_KEY)) == HA_NOSAME || > + /* Unique key and all keyparts are NULL rejecting */ > + ((key_flags & HA_NOSAME) && keyparts == ref.not_null_keyparts) > + ) > + ) || > + /* true only for extended keys */ > + (keyparts > keyinfo->user_defined_key_parts && > + MY_TEST(key_flags & HA_EXT_NOSAME) && > + keyparts == keyinfo->ext_key_parts) > + ) > + return true; > + return false; > +} > + > + > /** > @} (end of group Query_Optimizer) > */ BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

Re: [Maria-developers] 33787ea47a3: MDEV-6899: Database Dump has Syntax Error When Importing
by Sergei Golubchik 23 Dec '20

23 Dec '20

Hi, Anel! On Dec 23, Anel Husakovic wrote: > revision-id: 33787ea47a3 (mariadb-10.2.31-588-g33787ea47a3) > parent(s): ceef26cf866 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-11-17 15:11:14 +0100 > message: > > MDEV-6899: Database Dump has Syntax Error When Importing > > --- > mysql-test/r/mysqldump.result | 32 ++++++++++++++++++++++++++++++++ > mysql-test/t/mysqldump.test | 26 ++++++++++++++++++++++++++ > sql/events.cc | 10 ++++++++-- > sql/sql_show.cc | 4 +++- > 4 files changed, 69 insertions(+), 3 deletions(-) > > diff --git a/sql/events.cc b/sql/events.cc > index abac2833833..2880113bc75 100644 > --- a/sql/events.cc > +++ b/sql/events.cc > @@ -680,6 +680,12 @@ send_show_create_event(THD *thd, Event_timed *et, Protocol *protocol) > if (et->get_create_event(thd, &show_str)) > DBUG_RETURN(TRUE); > > + LEX_STRING def_show_str= show_str.lex_string(); > + if (!strcmp(&def_show_str.str[def_show_str.length -1], ";")) > + { > + --def_show_str.length; > + def_show_str.str[def_show_str.length]='\0'; > + } No, this is clearly wrong. The bug is that semicolon even gets into the event definition, see other commands in the bug report. You should fix that, not strip it away in SHOW CREATE, which will only hide the bug. Why is the semicolon parsed as a part of the event anyway? > field_list.push_back(new (mem_root) > Item_empty_string(thd, "Event", NAME_CHAR_LEN), > mem_root); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ab7092a415f: MDEV-24179: Assertion `m_status == DA_ERROR || m_status == DA_OK ||
by Sergei Golubchik 21 Dec '20

21 Dec '20

Hi, Rucha! On Dec 21, Rucha Deodhar wrote: > revision-id: ab7092a415f (mariadb-10.2.31-584-gab7092a415f) > parent(s): 9b30212f15e > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-12-18 10:20:20 +0530 > message: > > MDEV-24179: Assertion `m_status == DA_ERROR || m_status == DA_OK || > m_status == DA_OK_BULK' failed in Diagnostics_area::message() > > Analysis: Assertion failure happens because we reach the maximum limit to > examine rows. > Fix: Return the error state. > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 5544c765775..73b7a84fae1 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -5046,6 +5046,12 @@ int get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond) > continue; > } > > + if (thd->killed == ABORT_QUERY) > + { > + error= 0; > + goto err; > + } > + you already have exactly the same if() few lines below. may be it shouldn't be duplicated, but moved down to the err: label? > DEBUG_SYNC(thd, "before_open_in_get_all_tables"); > if (fill_schema_table_by_open(thd, FALSE, > table, schema_table, Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 889f90365ef: MDEV-23536 : Race condition between KILL and transaction commit
by Sergei Golubchik 20 Dec '20

20 Dec '20

Hi, Jan! On Dec 19, Jan LindstrÃ¶m wrote: > revision-id: 889f90365ef (mariadb-10.2.31-623-g889f90365ef) > parent(s): 74223c33d1a > author: Jan Lindström <jan.lindstrom(a)mariadb.com> > committer: Jan Lindström <jan.lindstrom(a)mariadb.com> > timestamp: 2020-12-18 16:10:39 +0200 > message: > > MDEV-23536 : Race condition between KILL and transaction commit ... > If you look carefully into the above, you can conclude that > thd->free_connection() can be called concurrently with > KILL/thd->awake(). Which is the bug. And it is partially fixed in > THD::~THD(), that is destructor waits for KILL completion: > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 7eaafbd9044..46aeb4f2bb3 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1437,6 +1437,31 @@ void THD::cleanup(void) > void THD::free_connection() > { > DBUG_ASSERT(free_connection_done == 0); > +#ifdef WITH_WSREP > + /* > + wsrep_rgi should be protected with LOCK_thd_data mutex and > + mutexing ordering rules state that LOCK_thd_data must be taken > + before LOCK_thd_kill. > + */ > + if (wsrep_rgi && WSREP(this)) mysql_mutex_lock(&LOCK_thd_data); > +#endif > + /* > + Other threads may have a lock on LOCK_thd_kill to ensure that this > + THD is not deleted while they access it. The following mutex_lock > + ensures that no one else is using this THD and it's now safe to > + continue. > + */ > + mysql_mutex_lock(&LOCK_thd_kill); > + mysql_mutex_unlock(&LOCK_thd_kill); Note that THD destructor starts from `assert_not_linked()` - it's important, because if the THD is linked in the list of threads, then someone might start using it after you released LOCK_thd_kill. With the new logic you need to move assert_not_linked to free_connection(). otherwise ok to push > + > +#ifdef WITH_WSREP > + if (rpl_group_info *rgi= wsrep_rgi) > + { > + wsrep_rgi= 0; > + if (WSREP(this)) mysql_mutex_unlock(&LOCK_thd_data); > + delete rgi; > + } > +#endif > my_free(db); > db= NULL; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 4

Re: [Maria-developers] a36f27e200e: Fixes that enables my_new.cc (new wrapper using my_malloc)
by Sergei Golubchik 11 Dec '20

11 Dec '20

Hi, Michael! On Dec 11, Michael Widenius wrote: > revision-id: a36f27e200e (mariadb-10.5.2-275-ga36f27e200e) > parent(s): 9f2bbf84e8d > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-24 17:26:40 +0300 > message: > > Fixes that enables my_new.cc (new wrapper using my_malloc) > > This is not enabled by default as there are leaks in the > server that needs to be fixed first. One can compile > with -DREALLY_USE_MYSYS_NEW -DSF_REMEMBER_FRAMES=14 to find the > memory leaks from 'new' The change in my_new.cc is ok. The change in my_global.h and CMakeLists.txt is confusing and redundant. You unconditionaly define USE_MYSYS_NEW then conditionally undefine it again? Everything already worked just fine. By default cxx new is used, if one wants to use mysys version, it's done with -DHAVE_CXX_NEW=0 this works for me in 10.5 and fails to compile my_new.cc, as it should. Please, only keed your my_new.cc changes and say in the commit message "One can compile with -DHAVE_CXX_NEW=0 to find memory leaks from 'new'" As for -DSF_REMEMBER_FRAMES=14, perhaps it's better to do it automatically? Like #ifdef USE_MYSYS_NEW #define SF_REMEMBER_FRAMES 14 #else #define SF_REMEMBER_FRAMES 8 #endif or may be even just always make it 14. It should be easy to enable this feature, right? > diff --git a/CMakeLists.txt b/CMakeLists.txt > index 80fac4b2ccc..f97a6354f19 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -362,10 +362,8 @@ ENDIF() > # Run platform tests > INCLUDE(configure.cmake) > > -# force -DUSE_MYSYS_NEW unless already done by HAVE_CXX_NEW > -IF(NOT HAVE_CXX_NEW) > - ADD_DEFINITIONS(-DUSE_MYSYS_NEW) > -ENDIF() > +# Always use USE_MYSYS_NEW > +ADD_DEFINITIONS(-DUSE_MYSYS_NEW) > > # Find header files from the bundled libraries > # (wolfssl, readline, pcre2, etc) > diff --git a/include/my_global.h b/include/my_global.h > index 86ef5f882f6..8862a397f73 100644 > --- a/include/my_global.h > +++ b/include/my_global.h > @@ -527,6 +527,9 @@ typedef unsigned short ushort; > duplicate declaration of __cxa_pure_virtual, solved by declaring it a > weak symbol. > */ > +#ifndef REALLY_USE_MYSYS_NEW > +#undef USE_MYSYS_NEW > +#endif > #if defined(USE_MYSYS_NEW) && ! defined(DONT_DECLARE_CXA_PURE_VIRTUAL) > C_MODE_START > int __cxa_pure_virtual () __attribute__ ((weak)); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9f2bbf84e8d1: Fixed my_addr_resolve
by Sergei Golubchik 10 Dec '20

10 Dec '20

Hi, Michael! On Dec 03, Michael Widenius wrote: > revision-id: 9f2bbf84e8d1 (mariadb-10.5.2-274-g9f2bbf84e8d1) > parent(s): 173116c3a573 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-24 17:26:40 +0300 > message: > > Fixed my_addr_resolve > > At least on openSUSE Leap 15.1 one should not use the dli_fbase offset > provided by dladdr() for addresses in the main program. > Without this patch the stack traces from sf_report_leaked_memory() > are not usable. Sorry, I was not able to repeat it. I built on openSUSE Leap and the stack trace was perfectly usable without your patch. I tried both `kill -ABRT` and an artificially introduced memory leak. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 53a88b2b7032: Optimize usage of c_ptr(), c_ptr_quick() and String::alloc()
by Sergei Golubchik 04 Dec '20

04 Dec '20

Hi, Michael! On Dec 03, Michael Widenius wrote: > revision-id: 53a88b2b7032 (mariadb-10.5.2-272-g53a88b2b7032) > parent(s): 0aab153e05be > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-24 17:26:40 +0300 > message: > > Optimize usage of c_ptr(), c_ptr_quick() and String::alloc() > > The proble was that hen using String::alloc() to allocate a string, "hen" ? after a minute of staring I realized that you probably meant "when". (and "problem") > the String ensures that there is space for an extra NULL byte in the > buffer and if not, reallocates the string. This is a problem with the > String::set_int() that calls alloc(21), which forces > an extra malloc/free to happen. > > - We do not anymore re-allocate String if alloc() is called with the > Allocated_length. This reduces number of malloc() allocations, > especially one big re-allocation in Protocol::send_result_Set_metadata() > for almost every query that produced a result to the connnected client. > - Avoid extra mallocs when using LONGLONG_BUFFER_SIZE > This can now be done as alloc() doesn't increase buffers if new length is > not bigger than old one. > - c_ptr() is redesigned to be safer (but a bit longer) than before. > - Remove wrong usage of c_ptr_quick() > c_ptr_quick() where used in many cases to get the pointer to the used > buffer, even when it didn't need to be \0 terminated. In this case > ptr() is a better substitute. > Another problem with c_ptr_quick() is that it didn't guarantee that > the string would be \0 terminated. > - item_val_str(), an API function not used currently by the server, > now always returns a null terminated string (before it didn't always > do that). > - Ensure that all String allocations uses STRING_PSI_MEMORY_KEY. The old > mixed usage of performance keys caused assert's when String buffers > where shrunk. > - Binary_string::shrink() is simplifed > > diff --git a/sql/item.cc b/sql/item.cc > index 4a26cb6c140c..328fe96018a5 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -4574,7 +4574,7 @@ const String *Item_param::value_query_val_str(THD *thd, String *str) const > break; > } > DBUG_ASSERT(str->length() <= typelen); > - buf= str->c_ptr_quick(); > + buf= (char*) str->ptr(); > ptr= buf + str->length(); > *ptr++= '\''; > ptr+= (uint) my_TIME_to_str(&value.time, ptr, decimals); this is just wrong. One cannot simply write data at the String's end pointer. One must alloc() the space first. > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index 8ad14ca260c4..ce72a7af051d 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -126,7 +126,7 @@ partition_info *partition_info::get_clone(THD *thd) > /** > Mark named [sub]partition to be used/locked. > > - @param part_name Partition name to match. > + @param part_name Partition name to match. Must be \0 terminated! > @param length Partition name length. > > @return Success if partition found > @@ -172,9 +172,9 @@ bool partition_info::add_named_partition(const char *part_name, size_t length) > else > bitmap_set_bit(&read_partitions, part_def->part_id); > } > - DBUG_PRINT("info", ("Found partition %u is_subpart %d for name %s", > + DBUG_PRINT("info", ("Found partition %u is_subpart %d for name %.*s", > part_def->part_id, part_def->is_subpart, > - part_name)); > + length, part_name)); why did you specify a length if the name "must be \0 terminated!" ? (and I agree that it must be) > DBUG_RETURN(false); > } > > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 633d969b3dec..2b603380aa6c 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -404,12 +404,16 @@ static int item_value_type(struct st_mysql_value *value) > static const char *item_val_str(struct st_mysql_value *value, > char *buffer, int *length) > { > + size_t org_length= *length; > - String str(buffer, *length, system_charset_info), *res; > + String str(buffer, org_length, system_charset_info), *res; > if (!(res= ((st_item_value_holder*)value)->item->val_str(&str))) > return NULL; > *length= res->length(); > - if (res->c_ptr_quick() == buffer) > + if (res->ptr() == buffer && res->length() < org_length) > + { > + buffer[res->length()]= 0; > return buffer; > + } Okay. This looked like the *only* correct usage of c_ptr_quick(). Agree, better to remove c_ptr_quick(). > > /* > Lets be nice and create a temporary string since the > diff --git a/sql/sql_string.cc b/sql/sql_string.cc > index e2defba434dd..8ab156d0e556 100644 > --- a/sql/sql_string.cc > +++ b/sql/sql_string.cc > @@ -1251,21 +1259,16 @@ bool String::append_semi_hex(const char *s, uint len, CHARSET_INFO *cs) > // Shrink the buffer, but only if it is allocated on the heap. > void Binary_string::shrink(size_t arg_length) > { > - if (!is_alloced()) > - return; > - if (ALIGN_SIZE(arg_length + 1) < Alloced_length) > + if (is_alloced() && ALIGN_SIZE(arg_length + 1) < Alloced_length) > + { > + char *new_ptr; > + /* my_realloc() can't fail as new buffer is less than the original one */ > + if ((new_ptr= (char*) my_realloc(STRING_PSI_MEMORY_KEY, Ptr, arg_length, > + MYF(thread_specific ? > + MY_THREAD_SPECIFIC : 0)))) why do you need an if() if "my_realloc() can't fail" ? (and it really cannot, you're right) > { > - char* new_ptr; > - if (!(new_ptr = (char*)my_realloc(STRING_PSI_MEMORY_KEY, Ptr, arg_length, > - MYF(thread_specific ? MY_THREAD_SPECIFIC : 0)))) > - { > - Alloced_length = 0; > - real_alloc(arg_length); > - } > - else > - { > - Ptr = new_ptr; > - Alloced_length = (uint32)arg_length; > - } > + Ptr= new_ptr; > + Alloced_length= (uint32) arg_length; > } > + } > } > diff --git a/sql/sql_string.h b/sql/sql_string.h > index 2ef817ea0adc..01d5211fe850 100644 > --- a/sql/sql_string.h > +++ b/sql/sql_string.h > @@ -599,25 +599,34 @@ class Binary_string: public Static_binary_string > > inline char *c_ptr() > { > - DBUG_ASSERT(!alloced || !Ptr || !Alloced_length || > - (Alloced_length >= (str_length + 1))); > - > - if (!Ptr || Ptr[str_length]) // Should be safe > - (void) realloc(str_length); > + if (unlikely(!Ptr)) > + return (char*) ""; > + /* > + Here we assume that any buffer used to initalize String has > + an end \0 or have at least an accessable character at end. > + This is to handle the case of String("Hello",5) efficently. > + */ > + if (unlikely(!alloced && !Ptr[str_length])) No, this is wrong. Note the difference between String a("Hello", 5) and char hello[5]; String a(buf, 5); Your assumption should only apply to the first case, not to the second. In the first case alloced==Alloced_length==0, in the second case only alloced==0 and Alloced_length==5. So in the if() above you need to look at Alloced_length. > + return Ptr; > + if (str_length < Alloced_length) > + { > + Ptr[str_length]=0; > + return Ptr; > + } > + (void) realloc(str_length+1); /* This will add end \0 */ > return Ptr; > } > + /* Use c_ptr() instead. This will be deleted soon, kept for compatiblity */ > inline char *c_ptr_quick() > { > - if (Ptr && str_length < Alloced_length) > - Ptr[str_length]=0; > - return Ptr; > + return c_ptr_safe(); > } > inline char *c_ptr_safe() > { > if (Ptr && str_length < Alloced_length) > Ptr[str_length]=0; > else > - (void) realloc(str_length); > + (void) realloc(str_length + 1); what's the difference between c_ptr_safe and c_ptr, again? they're very similar, so they'd need a good comment explaining when to use what. > return Ptr; > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 994ea2af3973: Optimize size of lex structures
by Sergei Golubchik 04 Dec '20

04 Dec '20

Hi, Michael! On Dec 03, Michael Widenius wrote: > revision-id: 994ea2af3973 (mariadb-10.5.2-269-g994ea2af3973) > parent(s): 77500a1a1107 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-17 12:24:58 +0300 > message: > > Optimize size of lex structures > > LEX, st_select_lex, st_select_unit optimized for space: > - Use bit fields for bool variables > - Ensure that all bit fields are initialized (improves > performance for init functions as all bit fields can be > initalized with one memory access) > - Move members around in above structures to remove alignment > gaps I did not get any reply to my email From: Sergei Golubchik <serg(a)mariadb.org> To: Michael Widenius <michael.widenius(a)gmail.com> Cc: maria-developers(a)lists.launchpad.net Subject: Re: c3ecf0d6243: Optimize size of lex structures Date: Wed, 9 Sep 2020 15:57:23 +0200 and I don't see any related changes in this commit. Please, do at least one of the two. Regards, Sergei

1 0

Re: [Maria-developers] 77500a1a1107: Added typedef decimal_digits_t (uint16) for number of decimals
by Sergei Golubchik 04 Dec '20

04 Dec '20

Hi, Michael! I did not get any reply to my email From: Sergei Golubchik <serg(a)mariadb.org> To: Michael Widenius <michael.widenius(a)gmail.com> Cc: maria-developers(a)lists.launchpad.net Subject: Re: c79c6f78205: Added typedef decimal_digits_t (uint16) for number of decimals Date: Wed, 9 Sep 2020 13:34:57 +0200 and I don't see any related changes in this commit. Please, do at least one of the two. Regards, Sergei On Dec 03, Michael Widenius wrote: > revision-id: 77500a1a1107 (mariadb-10.5.2-268-g77500a1a1107) > parent(s): 32104b969513 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-17 12:24:58 +0300 > message: > > Added typedef decimal_digits_t (uint16) for number of decimals > > For fields and Item's uint8 should be good enough. After > discussions with Alexander Barkov we choose uint16 (for now) > as some format functions may accept +256 digits. > > The reason for this patch was to make the storage of decimal > digits simlar. Before this patch decimals was stored/used as > uint8, int and uint. > > Changed most decimal variables and functions to use the new > typedef. >

1 0

Re: [Maria-developers] 32104b969513: MDEV-23001 Precreate static Item_bool() to simplify code
by Sergei Golubchik 04 Dec '20

04 Dec '20

Hi, Michael! I did not get any reply to my email From: Sergei Golubchik <serg(a)mariadb.org> To: Michael Widenius <michael.widenius(a)gmail.com> Cc: maria-developers(a)lists.launchpad.net Subject: Re: bf8bd057344: MDEV-23001 Precreate static Item_bool() to simplify code Date: Tue, 8 Sep 2020 20:39:35 +0200 and I don't see any related changes in this commit. Please, do at least one of the two. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org On Dec 03, Michael Widenius wrote: > revision-id: 32104b969513 (mariadb-10.5.2-267-g32104b969513) > parent(s): 32a29afea777 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-17 12:24:45 +0300 > message: > > MDEV-23001 Precreate static Item_bool() to simplify code > > The following changes where done: > - Create global Item: Item_false and Item_true > - Replace all creation if 'FALSE' and 'TRUE' top level items used for > WHERE/HAVING/ON clauses to use Item_false and Item_true. > > The benefit are: > - Less and faster code > - No test needed if we where able to create the new item. > - Fixed possible errors if 'new' would have failed for the Item_bool's >

1 0

[Maria-developers] MDEV-21287 dev community opinion wanted
by Pierre-Jean Clement 02 Dec '20

02 Dec '20

Hi all, I created the Jira ticket MDEV-21287 10 months ago and didn't get much feedback. I tried to take a shot at it myself, but found out that the resolution MDEV-20403 introduced this behavior by purposefully evaluating 'update functions' before the BEFORE UPDATE trigger. I couldn't really understand why this was necessary, and I'd like to get feedback from the dev community to try and understand what is really supposed to happen on update. I kinda hate getting my trigger called with a new value that's not going to be persisted anyway, what do you think about that? Regards, Pierre-Jean

2 3

Re: [Maria-developers] MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison
by Sergey Petrunia 26 Nov '20

26 Nov '20

Hi Varun, Please find some cosmetic input below. Ok to push after addresed. > diff --git a/mysql-test/main/subselect4.test b/mysql-test/main/subselect4.test > index 6f5eb1f2985f..e9da028618f6 100644 > --- a/mysql-test/main/subselect4.test > +++ b/mysql-test/main/subselect4.test > @@ -2238,3 +2238,38 @@ SET join_cache_level= @save_join_cache_level; > DROP TABLE t1,t2,t3,t4; > > --echo # End of 10.2 tests > + > +--echo # > +--echo # MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison > +--echo # > + > +CREATE TABLE t1(a VARCHAR(50) collate utf8_general_ci, b INT); > +INSERT INTO t1 VALUES ('abc',1), ('def', 2), ('ghi', 3), ('jkl', 4), ('mno', 5); > + > +CREATE TABLE t2(a VARCHAR(50) collate utf8mb4_general_ci, b INT); > +INSERT INTO t2 VALUES ('abc',1), ('def', 2), ('ghi', 3), ('jkl', 4), ('mno', 5); > + > +set in_predicate_conversion_threshold=2; > + ... > +EXPLAIN > +SELECT * FROM t2 WHERE (t2.a,t2.b) IN (('abx',1),('def',2), ('abc', 3)); > + > +set names default; > +DROP TABLE t1,t2; > + > +--echo # End of 10.3 tests Please restore the value of in_predicate_conversion_threshold. > diff --git a/sql/item_subselect.h b/sql/item_subselect.h > index 1cea7291c9ef..d1b2f86088d9 100644 > --- a/sql/item_subselect.h > +++ b/sql/item_subselect.h > @@ -573,6 +573,13 @@ class Item_in_subselect :public Item_exists_subselect > TRUE<=>registered in the list of semijoins in outer select > */ > bool is_registered_semijoin; > + > + Item_func_not_all *upper_item; // point on NOT/NOP before ALL/SOME subquery Is this necessary to touch the above member? I think it's better leave it as is. > + > + /* > + SET to TRUE if IN subququery is converted from an IN rredicate > + */ > + bool converted_from_in_predicate; ... > diff --git a/sql/sql_type.h b/sql/sql_type.h > index 907225b7c505..a1403165fe38 100644 > --- a/sql/sql_type.h > +++ b/sql/sql_type.h > @@ -1291,9 +1291,21 @@ class Type_handler > Item *target_expr, Item *target_value, > Item_bool_func2 *source, > Item *source_expr, Item *source_const) const= 0; > + > + /* > + @brief > + Check if an IN subquery allows materialization or not > + @param > + inner expression on the inner side of the IN subquery > + outer expression on the inner side of the IN subquery Typo. Should it should be 'outer' ? > + is_in_predicate SET to true if IN subquery was converted from an > + IN predicate or we are checking if materialization > + strategy can be used for an IN predicate > + */ > virtual bool > subquery_type_allows_materialization(const Item *inner, > - const Item *outer) const= 0; > + const Item *outer, > + bool is_in_predicate) const= 0; > /** > Make a simple constant replacement item for a constant "src", > so the new item can futher be used for comparison with "cmp", e.g.: BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Question about JSONPath and '**' wildcard
by Sergey Petrunia 20 Nov '20

20 Nov '20

Hi Alexey, I've found this discrepancy in JSONPath evaluation: set @json_doc3 = ' { "root": { "child1" : { "child2" : { "child1" : { "x":124 } } } } } '; select json_extract(@json_doc3, '$**.child1**.x'); MariaDB [test]> select json_extract(@json_doc3, '$**.child1**.x'); +--------------------------------------------+ | json_extract(@json_doc3, '$**.child1**.x') | +--------------------------------------------+ | NULL | +--------------------------------------------+ 1 row in set (0.001 sec) MySQL-8> select json_extract(@json_doc3, '$**.child1**.x'); +--------------------------------------------+ | json_extract(@json_doc3, '$**.child1**.x') | +--------------------------------------------+ | [124] | +--------------------------------------------+ 1 row in set (0.00 sec) Which one is right? My opinion is that MariaDB's answer is definitely incorrect. As for MySQL's answer, it depends on how the result of JSONPath expression is defined. If it is "a set of nodes in the JSON document which match the pattern", MySQL's result is correct. Coming from XPath world, I was expecting a semantics in form find the set of nodes matching the search step#1 ( $**, then apply step #2 ( .child1) then apply step #3 ( any children of those) ... in which case "124" would be in the result twice as it is reachable via two possible paths. Any thoughts about this? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 2

Re: [Maria-developers] 255e5f269d80: MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade to 10.5, mysql_upgrade should take of that
by Sergei Golubchik 13 Nov '20

13 Nov '20

Hi, Sujatha! Looks good, thanks! One detail to fix - I think you've mistakenly committed an old test from the previous version of the patch. And two minor suggestions, you can change or not, as you like. otherwise ok to push On Nov 13, Sujatha wrote: > revision-id: 255e5f269d80 (mariadb-10.5.4-319-g255e5f269d80) > parent(s): 97569d3c37a6 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Sujatha <sujatha.sivakumar(a)mariadb.com> > timestamp: 2020-11-13 19:08:02 +0530 > message: > > MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade to 10.5, mysql_upgrade should take of that > > Add a new privilege "SLAVE MONITOR" which will grant user the permission > to execute "SHOW SLAVE STATUS" and "SHOW RELAYLOG EVENTS" commands. > > SHOW SLAVE STATUS requires either SLAVE MONITOR/SUPER > SHOW RELAYLOG EVENTS requires SLAVE MONITOR privilege. > > diff --git a/mysql-test/main/grant_slave_monitor.test b/mysql-test/main/grant_slave_monitor.test > new file mode 100644 > index 000000000000..a4f21085e537 > --- /dev/null > +++ b/mysql-test/main/grant_slave_monitor.test > @@ -0,0 +1,101 @@ > +# ==== Purpose ==== > +# > +# SLAVE MONITOR privilege is required to execute following commands. > +# SHOW SLAVE STATUS > +# SHOW RELAYLOG EVENTS > +# > +# ==== Implementation ==== > +# > +# Step1: GRANT ALL privileges for a new user 'user1' and then REVOKE > +# SLAVE MONITOR and SUPER privileges. > +# Step2: Execute SHOW SLAVE STAUTS/SHOW RELAYLOG EVENTS commands and expect > +# ER_SPECIFIC_ACCESS_DENIED_ERROR. This also verifies that REPLICATION > +# SLAVE ADMIN privilege is not required for these two commands. > +# Step3: GRANT SLAVE MONITOR privilege and observe that both commands are > +# allowd to execute. > +# Step4: GRANT SUPER privilege and observe that only SHOW SLAVE STATUS command > +# is allowed. > +# > +# ==== References ==== > +# > +# MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade > +# to 10.5, mysql_upgrade should take of that > +# MDEV-23918: admin privlege required to view contents of relay logs in 10.5 > +# > + > +--source include/not_embedded.inc > + > +CREATE USER user1@localhost IDENTIFIED BY ''; > +GRANT ALL PRIVILEGES ON *.* TO user1@localhost; > +REVOKE SLAVE MONITOR, SUPER ON *.* FROM user1@localhost; > +FLUSH PRIVILEGES; > + > +--connect(con1,localhost,user1,,) > +--connection con1 > +SHOW GRANTS; > + > +--echo # > +--echo # Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW SLAVE STATUS > +--echo # Expected error: Access denied; you need (at least one of) the SUPER, SLAVE > +--echo # MONITOR privilege(s) for this operation > +--echo # > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW SLAVE STATUS; > + > +--echo # > +--echo # Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW RELAYLOG EVENTS > +--echo # Expected error: Access denied; you need (at least one of) the REPLICA MONITOR > +--echo # privilege(s) for this operation > +--echo # > +--disable_ps_protocol > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW RELAYLOG EVENTS; > +--enable_ps_protocol > +--disconnect con1 > + > +--echo # > +--echo # SHOW SLAVE STATUS and SHOW RELAYLOG EVENTS are allowed with SLAVE MONITOR privilege > +--echo # > + > +--connection default > +GRANT SLAVE MONITOR ON *.* TO user1@localhost; > +FLUSH PRIVILEGES; > + > +--connect(con1,localhost,user1,,) > +--connection con1 > +SHOW GRANTS; > + > +--disable_result_log > +SHOW SLAVE STATUS; > +#--disable_ps_protocol > +SHOW RELAYLOG EVENTS; > +#--enable_ps_protocol > +--enable_result_log > +--disconnect con1 > + > +--connection default > +DROP USER user1@localhost; > + > +--echo # > +--echo # SHOW SLAVE STATUS command is allowed with SUPER privilege > +--echo # > +CREATE USER user1@localhost IDENTIFIED BY ''; > +GRANT SUPER ON *.* TO user1@localhost; > + > +--connect(con1,localhost,user1,,) > +--disable_result_log > +SHOW SLAVE STATUS; > +--enable_result_log > + > +--echo # > +--echo # SHOW RELAYLOG EVENTS is not allowed with SUPER privilege, it requires SLAVE MONITOR > +--echo # > + > +--disable_ps_protocol > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW RELAYLOG EVENTS; > +--enable_ps_protocol > +--disconnect con1 > + > +--connection default > +DROP USER user1@localhost; > diff --git a/mysql-test/main/rpl_replica_monitor_priv.test b/mysql-test/main/rpl_replica_monitor_priv.test > --- /dev/null > +++ b/mysql-test/main/rpl_replica_monitor_priv.test > @@ -0,0 +1,90 @@ Did you check in this file by mistake? It's a replication test (source include/master-slave.inc) but not in the rpl suite. It tests the feature that does not need master-slave.inc. And it's duplicated by grant_slave_monitor.test > diff --git a/sql/privilege.h b/sql/privilege.h > index 37cdf4da01ad..d0e6e532b4b5 100644 > --- a/sql/privilege.h > +++ b/sql/privilege.h > @@ -93,13 +94,18 @@ enum privilege_t: unsigned long long > */ > }; > > +constexpr static inline privilege_t ALL_KNOWN_BITS(privilege_t x) > +{ > + return (privilege_t)((x << 1) - 1); Generally it's safer to write x | (x - 1) it won't overflow if x is already the highest possible bit. In privileges we're quite far from it at the moment, so it's not a big deal, just a style issue and setting a good example to others. > +} > > // Version markers > constexpr privilege_t LAST_100304_ACL= DELETE_HISTORY_ACL; > constexpr privilege_t LAST_100502_ACL= BINLOG_REPLAY_ACL; > +constexpr privilege_t LAST_100509_ACL= SLAVE_MONITOR_ACL; > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index ef96e8f24843..b5b1472f004f 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -1015,7 +1015,7 @@ class User_table_tabular: public User_table > access|= LOCK_TABLES_ACL | CREATE_TMP_ACL | SHOW_DB_ACL; > if (access & FILE_ACL) > access|= BINLOG_MONITOR_ACL | REPL_SLAVE_ACL | BINLOG_ADMIN_ACL | > - BINLOG_REPLAY_ACL; > + BINLOG_REPLAY_ACL | SLAVE_MONITOR_ACL; this does not hurt, but it's rather redundant, you set SLAVE_MONITOR_ACL below anyway. > if (access & PROCESS_ACL) > access|= SUPER_ACL | EXECUTE_ACL; > } > @@ -1057,6 +1057,9 @@ class User_table_tabular: public User_table > if (access & REPL_SLAVE_ACL) > access|= REPL_MASTER_ADMIN_ACL; > > + if (access & REPL_SLAVE_ACL) > + access|= SLAVE_MONITOR_ACL; > + > return access & GLOBAL_ACLS; > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] d66d5457589d: MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade to 10.5, mysql_upgrade should take of that
by Sergei Golubchik 12 Nov '20

12 Nov '20

Hi, Sujatha! On Nov 12, Sujatha wrote: > revision-id: d66d5457589d (mariadb-10.5.4-315-gd66d5457589d) > parent(s): 10b2d5726fa2 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Sujatha <sujatha.sivakumar(a)mariadb.com> > timestamp: 2020-11-11 18:41:44 +0530 > message: > > MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade to 10.5, mysql_upgrade should take of that > > Add a new privilege "REPLICA MONITOR" which will grant user the permission > to execute "SHOW SLAVE STATUS" and "SHOW RELAYLOG EVENTS" commands. > > SHOW SLAVE STATUS requires either REPLICA MONITOR/SUPER > SHOW RELAYLOG EVENTS requires REPLICA MONITOR privilege. > > diff --git a/mysql-test/main/grant.result b/mysql-test/main/grant.result > index 01daf0271867..a41b38aaa0ba 100644 > --- a/mysql-test/main/grant.result > +++ b/mysql-test/main/grant.result > @@ -1966,10 +1967,11 @@ GRANT REPLICATION SLAVE ON *.* TO mysqltest_u1@localhost; > GRANT SHOW DATABASES ON *.* TO mysqltest_u1@localhost; > GRANT SHUTDOWN ON *.* TO mysqltest_u1@localhost; > GRANT USAGE ON *.* TO mysqltest_u1@localhost; > +GRANT REPLICA MONITOR ON *.* TO mysqltest_u1@localhost; I don't think we should have a mix of REPLICA and SLAVE terms. Call it SLAVE MONITOR and when we flip the switch on grants, we'll do it for all grants at the same time. > SHOW GRANTS FOR mysqltest_u1@localhost; > Grants for mysqltest_u1@localhost > -GRANT RELOAD, SHUTDOWN, PROCESS, FILE, SHOW DATABASES, REPLICATION SLAVE, BINLOG MONITOR, CREATE USER ON *.* TO `mysqltest_u1`@`localhost` > +GRANT RELOAD, SHUTDOWN, PROCESS, FILE, SHOW DATABASES, REPLICATION SLAVE, BINLOG MONITOR, CREATE USER, REPLICA MONITOR ON *.* TO `mysqltest_u1`@`localhost` > GRANT CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE ROUTINE, ALTER ROUTINE, EVENT ON `mysqltest_db1`.* TO `mysqltest_u1`@`localhost` > connect con1,localhost,mysqltest_u1,,mysqltest_db1; > connection con1; > diff --git a/mysql-test/main/grant.test b/mysql-test/main/grant.test > index 38baf6738255..63be18ecb292 100644 > --- a/mysql-test/main/grant.test > +++ b/mysql-test/main/grant.test > @@ -1833,6 +1833,7 @@ GRANT REPLICATION SLAVE ON *.* TO mysqltest_u1@localhost; > GRANT SHOW DATABASES ON *.* TO mysqltest_u1@localhost; > GRANT SHUTDOWN ON *.* TO mysqltest_u1@localhost; > GRANT USAGE ON *.* TO mysqltest_u1@localhost; > +GRANT REPLICA MONITOR ON *.* TO mysqltest_u1@localhost; But the parser should, of course, accept both, as everywhere. > diff --git a/mysql-test/suite/rpl/t/rpl_replica_monitor_priv.test b/mysql-test/suite/rpl/t/rpl_replica_monitor_priv.test > new file mode 100644 > index 000000000000..f7b2af2e0c51 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_replica_monitor_priv.test > @@ -0,0 +1,90 @@ > +# ==== Purpose ==== > +# > +# REPLICA MONITOR privilege is required to execute following commands. > +# SHOW SLAVE STATUS > +# SHOW RELAYLOG EVENTS > +# > +# ==== Implementation ==== > +# > +# Step1: GRANT ALL privileges for a new user 'user1' and then REVOKE REPLICA > +# MONITOR and SUPER privileges. > +# Step2: Execute SHOW SLAVE STAUTS/SHOW RELAYLOG commands and expect > +# ER_SPECIFIC_ACCESS_DENIED_ERROR. This also verifies that REPLICATION > +# SLAVE ADMIN privilege is not required for these two commands. > +# Step3: GRANT REPLICA MONITOR privilege and observe that both commands are > +# allowd to execute. > +# Step4: GRANT SUPER privilege and observe that only SHOW SLAVE STATUS command > +# is allowed. why do you need a full master-slave replication setup for that? > +# ==== References ==== > +# > +# MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade > +# to 10.5, mysql_upgrade should take of that > +# MDEV-23918: admin privlege required to view contents of relay logs in 10.5 > +# > + > +--source include/not_embedded.inc > +--source include/have_binlog_format_mixed.inc > +--source include/master-slave.inc > + > +--connection master > +CREATE USER user1@localhost IDENTIFIED BY ''; > +GRANT ALL PRIVILEGES ON *.* TO user1@localhost; > +REVOKE REPLICA MONITOR, SUPER ON *.* FROM user1@localhost; > +FLUSH PRIVILEGES; > +--sync_slave_with_master > +--connect(con1,127.0.0.1,user1,,test,$SLAVE_MYPORT) > +--connection con1 > +SELECT CURRENT_USER(); > +SHOW GRANTS; > +--echo "Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW SLAVE STATUS" > +--echo "Expected error:Access denied; you need (at least one of) the SUPER, REPLICA MONITOR privilege(s) for this operation" better use not quotes, but #, like elsewhere. > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW SLAVE STATUS; > +--echo "Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW RELAYLOG EVENTS" > +--echo "Expected error: Access denied; you need (at least one of) the REPLICA MONITOR privilege(s) for this operation" > +--disable_ps_protocol why? > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW RELAYLOG EVENTS; > +--enable_ps_protocol > +--disconnect con1 > + > diff --git a/sql/privilege.h b/sql/privilege.h > index 37cdf4da01ad..e7686a206b84 100644 > --- a/sql/privilege.h > +++ b/sql/privilege.h > @@ -115,8 +117,12 @@ constexpr privilege_t ALL_KNOWN_ACL_100304 = > constexpr privilege_t ALL_KNOWN_ACL_100502= > (privilege_t) ((LAST_100502_ACL << 1) - 1); > > +// A combination of all bits defined in 10.5.8 > +constexpr privilege_t ALL_KNOWN_ACL_100508= > + (privilege_t) ((LAST_100508_ACL << 1) - 1); > + > // A combination of all bits defined as of the current version > -constexpr privilege_t ALL_KNOWN_ACL= ALL_KNOWN_ACL_100502; > +constexpr privilege_t ALL_KNOWN_ACL= ALL_KNOWN_ACL_100508; may be (privilege_t) ((LAST_CURRENT_ACL << 1) - 1) and then we won't need to change it anymore? Also, I think this deserves a function, like static inline privilege_t ALL_ACLS_TO(privilege_t x) { return (x << 1) - 1; } or, rather, a bit safer return x | (x - 1) > // Unary operators > @@ -505,9 +511,11 @@ constexpr privilege_t PRIV_STMT_STOP_SLAVE= REPL_SLAVE_ADMIN_ACL | SUPER_ACL; > // Was SUPER_ACL prior to 10.5.2 > constexpr privilege_t PRIV_STMT_CHANGE_MASTER= REPL_SLAVE_ADMIN_ACL | SUPER_ACL; > // Was (SUPER_ACL | REPL_CLIENT_ACL) prior to 10.5.2 > -constexpr privilege_t PRIV_STMT_SHOW_SLAVE_STATUS= REPL_SLAVE_ADMIN_ACL | SUPER_ACL; > +// Was REPL_SLAVE_ADMIN_ACL from 10.5.2 to 10.5.7 // Was SUPER_ACL | REPL_SLAVE_ADMIN_ACL from 10.5.2 to 10.5.7 otherwise it's unclear why you have SUPER_ACL below and not just REPLICA_MONITOR_ACL > +constexpr privilege_t PRIV_STMT_SHOW_SLAVE_STATUS= REPLICA_MONITOR_ACL | SUPER_ACL; > // Was REPL_SLAVE_ACL prior to 10.5.2 > -constexpr privilege_t PRIV_STMT_SHOW_RELAYLOG_EVENTS= REPL_SLAVE_ADMIN_ACL; > +// Was REPL_SLAVE_ADMIN_ACL from 10.5.2 to 10.5.7 > +constexpr privilege_t PRIV_STMT_SHOW_RELAYLOG_EVENTS= REPLICA_MONITOR_ACL; > > /* > Privileges related to binlog replying. > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index ef96e8f24843..8a3551c14f53 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -1057,6 +1057,9 @@ class User_table_tabular: public User_table > if (access & REPL_SLAVE_ACL) > access|= REPL_MASTER_ADMIN_ACL; > > + if (access & FILE_ACL) > + access|= REPLICA_MONITOR_ACL; Why is that? > + > return access & GLOBAL_ACLS; > } > > @@ -1541,19 +1548,22 @@ class User_table_json: public User_table > */ > if (access & SUPER_ACL) > { > - if (access & REPL_SLAVE_ACL) > - { > - /* > - The user could do both before the upgrade: > - - set global variables (because of SUPER_ACL) > - - execute "SHOW SLAVE HOSTS" (because of REPL_SLAVE_ACL) > - Grant all new privileges that were splitted from SUPER (in 10.5.2), > - and REPLICATION MASTER ADMIN, so it still can do "SHOW SLAVE HOSTS". > - */ > - access|= REPL_MASTER_ADMIN_ACL; > - } > access|= GLOBAL_SUPER_ADDED_SINCE_USER_TABLE_ACLS; > } > + if (access & REPL_SLAVE_ACL) > + { > + /* > + The user could do both before the upgrade: > + - set global variables (because of SUPER_ACL) this looked fine where it was before, but not anymore after you moved it out of `if (access & SUPER_ACL)` > + - execute "SHOW SLAVE HOSTS" (because of REPL_SLAVE_ACL) > + Grant all new privileges that were splitted from SUPER (in 10.5.2), > + and REPLICATION MASTER ADMIN, so it still can do "SHOW SLAVE HOSTS". > + - execute "SHOW RELAYLOG EVENTS" (because of REPL_SLAVE_ACL) > + */ > + access|= (REPL_MASTER_ADMIN_ACL | REPLICA_MONITOR_ACL); and REPL_MASTER_ADMIN_ACL allows to modify global variables was was earlier possible only with SUPER_ACL. So, indeed, old code was correct, that if() should be inside SUPER_ACL, one should only get REPL_MASTER_ADMIN_ACL if one had both SUPER_ACL and REPL_SLAVE_ACL. > + } > + if (access & BINLOG_MONITOR_ACL) > + access|= REPLICA_MONITOR_ACL; > } > > if (orig_access & ~mask) > @@ -8974,7 +8984,7 @@ static const char *command_array[]= > "CREATE USER", "EVENT", "TRIGGER", "CREATE TABLESPACE", "DELETE HISTORY", > "SET USER", "FEDERATED ADMIN", "CONNECTION ADMIN", "READ_ONLY ADMIN", > "REPLICATION SLAVE ADMIN", "REPLICATION MASTER ADMIN", "BINLOG ADMIN", > - "BINLOG REPLAY" > + "BINLOG REPLAY", "REPLICA MONITOR" SLAVE MONITOR > }; > > static uint command_lengths[]= Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] e3f45b2f9ea: MDEV-10267 Add ngram fulltext parser plugin
by Sergei Golubchik 11 Nov '20

11 Nov '20

Hi, Rinat! On Nov 02, Rinat Ibragimov wrote: > > > I can't decide. From my point of view, the current approach is > > > fine. Please pick a variant, and I'll try to implement that. > > > > No, I cannot guess which approach will produce more relevant > > searches. Implement something and then we test what works better > > Variable-length n-grams approach is too innovative, and hard to reason > about. I've never heard about such an approach, and it doesn't look > good to me. So I'll stick with a simple slicer. If you mean that variant where it splits "n-grams approach" to "n-gr", "gra", "ram", "ams", "ms a", "s ap", "app", ... then it's just "n letters in every chunk" very easy to explain. But ok, let's start simple and benchmark. > > Of course, it can. Note that fts_get_word() doesn't generate n-grams > > either, it gets the whole word and the n-gram plugin later splits it > > into n-grams. Similarly param->mysql_parse() will extract words for > > you and you'll split them into n-grams. > > Changed to use param->mysql_parse(). > > Turns out that in Aria, MyISAM, and InnoDB, param->mysql_parser() does > call back param->mysql_add_word(). Is it part of the plugin API? Yes, it is. E.g. slide 12 from my old presentation: http://conferences.oreillynet.com/presentations/mysql06/golubchik_sergei.pdf shows that there are three points where a plugin can add functionality. * It can extract the text and then call param->mysql_parser(), this allows to parse, say, gzip-ed texts or EXIF comments in images. * It can replace param->mysql_parser(), to use different rules for spliting the text into words. This is what the n-gram plugin normally does * It can replace param->mysql_add_word() to post-process every word after the built-in parser did the splitting. For example, stemming or soundex plugin can do that. > Comments in include/mysql/plugin_ftparser.h do not mention that at > all. That's why I initially thought that param->mysql_parse() will > parse the string like the default parser do, without any ways to > interact with the process. I've edited the comment to mention this possibility. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 0cd5377e9ff: MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table
by Sergei Golubchik 02 Nov '20

02 Nov '20

Hi, Aleksey! On Oct 20, Aleksey Midenkov wrote: > revision-id: 0cd5377e9ff (mariadb-10.2.31-357-g0cd5377e9ff) > parent(s): dc716da4571 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-08-17 00:52:35 +0300 > message: > > MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table > > update_virtual_fields() may reallocate Field_blob::value buffer due to > different reasons (f.ex. different types, see > Field_str::memcpy_field_possible()). But the buffer address may be > already in m_ordered_rec_buffer which is then used in ordered index > scan queue. > > The patch updates blob virtual fields when the ordered index scan > queue is sorted as well as the top record is restored from the queue. > > New VCOL_UPDATE_BLOBS mode is introduced to update only blobs. Swap is > not needed as it was already done if it was needed. > > Related to ea1b2504 No, I think we need to go deeper here. The bug is not only not caused by system versioning, it is also not caused by virtual columns. The problem is that partitioning copies record[0] out and then back to save and restore values. This does not work for blobs, because record[0] may store the pointer to the memory that Field_blob owns and can free or reallocate as needed. Normally it's not the problem because typically after row reads the engine owns the memory, not Field_blob. When a virtual column is set to a calculated value - then Field_blob owns the memory and we have this bug. But there are engines that use field->store() to set values - in this case Field_blob will own the memory too. Here's a test case with FederatedX: ============================================================= source include/have_partition.inc; source include/have_sequence.inc; source have_federatedx.inc; source include/federated.inc; connection slave; use federated; create table t1_1 (x int, b text, key(x)); create table t1_2 (x int, b text, key(x)); connection master; eval create table t1 (x int, b text, key(x)) engine=federated partition by range columns (x) ( partition p1 values less than (40) connection='mysql://root@127.0.0.1:$SLAVE_MYPORT/federated/t1_1', partition pn values less than (maxvalue) connection='mysql://root@127.0.0.1:$SLAVE_MYPORT/federated/t1_2' ); insert t1 select seq, seq from seq_1_to_35; insert t1 select seq, repeat(seq, 100) from seq_50_to_90; flush tables; select x, b from t1 where x > 30 and x < 60 order by x; select x, b from t1 where x > 30 and x < 60 order by b; drop table t1; connection slave; drop table t1_1, t1_2; source include/federated_cleanup.inc; ============================================================= first select doesn't trigger ASAN error, but it produces incorrect results (I'm showing it here, so that you could create a non-ASAN test that could be run on all builders). the second select gets the same ASAN error as your test below. > diff --git a/mysql-test/suite/vcol/r/partition.result b/mysql-test/suite/vcol/r/partition.result > index bd1353fa145..74c3c3394dc 100644 > --- a/mysql-test/suite/vcol/r/partition.result > +++ b/mysql-test/suite/vcol/r/partition.result > @@ -28,3 +28,26 @@ set statement sql_mode= '' for update t1 set i= 1, v= 2; > Warnings: > Warning 1906 The value specified for generated column 'v' in table 't1' has been ignored > drop table t1; > +# > +# MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table > +# > +# Cover queue_fix() in ha_partition::handle_ordered_index_scan() > +create table t1 ( > +x int auto_increment primary key, > +b text, v mediumtext as (b) virtual, > +index (v(10)) > +) partition by range columns (x) ( > +partition p1 values less than (3), > +partition p2 values less than (6), > +partition pn values less than (maxvalue)); > +insert into t1 (b) values ('q'), ('a'), ('b'); > +update t1 set b= 'bar' where v > 'a'; > +drop table t1; this test above doesn't cause ASAN failures for me without your fix. the test below fails all right. > +# Cover return_top_record() in ha_partition::handle_ordered_index_scan() > +create table t1 (x int primary key, b tinytext, v text as (b) virtual) > +partition by range columns (x) ( > +partition p1 values less than (4), > +partition pn values less than (maxvalue)); > +insert into t1 (x, b) values (1, ''), (2, ''), (3, 'a'), (4, 'b'); > +update t1 set b= 'bar' where x > 0 order by v limit 2; > +drop table t1; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 7807d9b6939: MDEV-22974: mysql_native_password make "invalid" valid
by Sergei Golubchik 30 Oct '20

30 Oct '20

Hi, Daniel! On Oct 22, Daniel Black wrote: > revision-id: 7807d9b6939 (mariadb-10.5.4-62-g7807d9b6939) > parent(s): 054f10365c4 > author: Daniel Black <daniel(a)mariadb.org> > committer: Daniel Black <daniel(a)mariadb.org> > timestamp: 2020-07-16 17:06:36 +1000 > message: > > MDEV-22974: mysql_native_password make "invalid" valid > > Per b9f3f06857ac, mysql_system_tables_data.sql creates > a mysql_native_password with a salted hash of "invalid" so that `set password` > will detect a native password can be pplied:. "applied" > > SHOW CREATE USER; dilignently uses this value in its output > generating the SQL: > > MariaDB [(none)]> show create user; > > +---------------------------------------------------------------------------------------------------+ > | CREATE USER for dan@localhost | > +---------------------------------------------------------------------------------------------------+ > | CREATE USER `dan`@`localhost` IDENTIFIED VIA mysql_native_password USING 'invalid' OR unix_socket | > +---------------------------------------------------------------------------------------------------+ > > Attempting to execute this before this patch resutls in: > > MariaDB [(none)]> CREATE USER `dan2`@`localhost` IDENTIFIED VIA mysql_native_password USING 'invalid' OR unix_socket; > ERROR 1372 (HY000): Password hash should be a 41-digit hexadecimal number > > As such deep the the implementation of mysql_native_password we make > "invalid" valid (pun intended) such that the above create user will > succeed. > > native_password_get_salt is only called in the context of > set_user_salt, so all setting of native passwords to hashed content of > 'invalid', quite literally create an invalid password. > > So other forms of "invalid" are valid SQL in creating invalid passwords: > > MariaDB [(none)]> set password = 'invalid'; > Query OK, 0 rows affected (0.001 sec) > > MariaDB [(none)]> alter user dan@localhost IDENTIFIED BY PASSWORD 'invalid'; > Query OK, 0 rows affected (0.000 sec) > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index d94016b7815..3cd7a67ae1a 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -14539,6 +14539,12 @@ static int native_password_get_salt(const char *hash, size_t hash_length, > > if (hash_length != SCRAMBLED_PASSWORD_CHAR_LENGTH) > { > + if (hash_length == 7 && strcmp(hash, "invalid") == 0) > + { > + memcpy(out, "invalid", 7); > + *out_length= 7; > + return 0; > + } okay. After you said ASAN, I think I can see why this could be problematic. You can, of course, pad it to SCRAMBLED_PASSWORD_CHAR_LENGTH, but then you'll create a *valid* scramble that would correspond to some actual password. One option would be to allow "invalid" literal in set_user_auth(), before even any plugin checks. But I'm unsure of the implications. > my_error(ER_PASSWD_LENGTH, MYF(0), SCRAMBLED_PASSWORD_CHAR_LENGTH); > return 1; > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] f112c4a07475: MDEV-20787: Script dgcov.pl does not work
by Sergei Golubchik 30 Oct '20

30 Oct '20

Hi, Anel! On Oct 30, Anel Husakovic wrote: > revision-id: f112c4a07475 (mariadb-10.2.31-541-gf112c4a07475) > parent(s): 784473b98662 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-26 14:25:41 +0100 > message: > > MDEV-20787: Script dgcov.pl does not work > > - Patch is changing CMakeList with `--coverage` flag as an alias for > `-fprofile-arcs -ftest-coverage -lgcov` in addition. > > - About the usage of `dgcov`: > When the server is compiled with `-DENABLE_GCOV=ON`, from object files are generated > `.gcno` and `.gcda` files. > `./mtr --gcov is_check_constraint` is invoking the following script calls: > - `./dgcov.pl --purge`, `./mtr is_check_constraint` and > - `./dgcov.pl --generate>/var/last_changes.dgcov`. > The `purge` flag is clearing `.gcda` files (and others extensions), > while running the test (which follows after the purge) new `.gcda` files are generated. > With dgcov `generate` flag, `gcov -i` (`intermediate format`) is called > on generated `<source-files-name>.gcda` files (`dbug.c.gcda` e.g.). > The patch is tested on `gcov 6.3` and `gcov 7.4` versions > and can be seen that resulting `.gcov` file for `6.3` creates > `<full path>.gcov` (`dbug.c.gcda.gcov` e.g.) file, > where `gcov 7.4+` is still creating `source-file-names.gcov`(`dbug.c.gcov`) files > as `gcov` in general is doing. > Results are stored in `./mysql-test/var/last_changes.dgcov`. > Note: for the newer versions of `gcov` intermediate format is > deprecated `-i` and should be used `--json-format`. does it mean that eventually we'll have to rewrite dgcov to support json format? > > - This patch doesn't take gcov versions in account since is extracting > filename of a `.gcov` file, which is generated with `gcov -i > <source-file>.gcda` during `dgcov.pl --generate`. > > - There is a check if file exists. > Reason for that is because some generated `gcov` files are outliers from above > rule (that a file generated in intermediate format(gcov) has the same source name). > Example 1: `aestables.cpp.gcda` is not generating `aestables.cpp.gcov`, > but only the files used in #include `modes.hpp.gcov runtime.hpp.gcov` > Example 2: `my_sha256.cc.gcda` is generating `my_sha.ic.gcov` > > - Gcov with out-of-source is not working, this patch will solve that > (probably?) > Make sure to test `MTR_BINDIR` which gets set during out-of-source > build and generate an error when running `./mysql-test/mtr --gcov alias`. > > diff --git a/mysql-test/dgcov.pl b/mysql-test/dgcov.pl > index fbc5540e6973..f2bbdad4d20e 100755 > --- a/mysql-test/dgcov.pl > +++ b/mysql-test/dgcov.pl > @@ -161,8 +161,14 @@ sub gcov_one_file { > system($cmd)==0 or die "system($cmd): $? $!"; > } I don't see any changes since the last review. Is it the correct commit? > - # now, read the generated file > - open FH, '<', "$_.gcov" or die "open(<$_.gcov): $!"; > + (my $filename = $_)=~ s/\.[^.]+$//; # remove extension I think this contradicts your explanation in the commit comment. You wrote that the file name can be either dbug.c.gcda.gcov or dbug.c.gcov. Here $_ is dbug.c.gcda, you unconditionally remove .gcda and then below you open $filename.gcov, that is dbug.c.gcov Which means this won't work with an older gcov. Is this your intention? I think it's fine not to support old gcov. But this should be a conscious decision, e.g. you can write in the commit comment "Now dgcov.pl only supports gcov 7.4+" - if that's indeed what you want. Also, if you do it this way, then there's no need to use s///, it's simpler to do, like sub gcov_one_file { return unless /\.gcda$/; + my $filename= $`; unless ($opt_skip_gcov) { or, may be even sub gcov_one_file { return unless /\.gcda$/; + my $gcov_file_path= "$File::Find::dir$`.gcov"; unless ($opt_skip_gcov) { > + my $gcov_file_path= $File::Find::dir."/$filename.gcov"; still an extra slash > + if (! -f $gcov_file_path) > + { > + return; > + } still not return unless -f $gcov_file_path; > + open FH, '<', "$gcov_file_path" or die "open(<$gcov_file_path): $!"; still extra quotes > + > my $fname; > while (<FH>) { > chomp; > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index 900ef736a455..2189529d82a7 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -1791,7 +1791,7 @@ sub command_line_setup { > # -------------------------------------------------------------------------- > # Gcov flag > # -------------------------------------------------------------------------- > - if ( ($opt_gcov or $opt_gprof) and ! $source_dist ) > + if ( ($opt_gcov or $opt_gprof) and (! $source_dist or -d $ENV{MTR_BINDIR})) > { > mtr_error("Coverage test needs the source - please use source dist"); > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-24015: SQL Error (1038): Out of sort memory when enough memory for the sort buffer is provided
by Sergey Petrunia 27 Oct '20

27 Oct '20

Hi Varun, I have only cosmetic comments, please find them below. Ok to push after they are addressed. > The issue here was with the re-execution of a subquey, after few executions > the subquery execution returned error that not enough space was provided inside the sort > buffer. > This is happening because with the introduction of packed sort keys, during the phase > where sort length is calculated the parameters of the sort keys are not reset and > we keep up adding the sort length to the previous calculated sort length for the > earlier executions of the subquery. This happens because the sort keys structure is > allocated only once and so if their is re-execution of a subquery involving sorting > then we should be resetting the parameters of the sort keys. The comment is hard to read. How about being more specific and using shorter sentences: The error was: filesort was done in a subquery which was executed multiple times. During each execution, sortlength() computed total sort key length in Sort_keys::sort_length, without resetting it first. Eventually Sort_keys::sort_length got larger than @@sort_buffer_size, which caused filesort() to be aborted with error. Fixed by making sortlength() to compute lengths only during the first invocation. Subsequent invocations return pre-computed values. > @@ -2518,7 +2526,7 @@ void Sort_param::try_to_pack_sortkeys() > return; > > const uint sz= Sort_keys::size_of_length_field; > - uint sort_len= sort_keys->get_sort_length(); > + uint sort_len= sort_keys->get_sort_length_with_original_values(); > > /* > Heuristic introduced, skip packing sort keys if saving less than 128 bytes > diff --git a/sql/sql_sort.h b/sql/sql_sort.h > index 40f0c5ede5f..6951d4d190a 100644 > --- a/sql/sql_sort.h > +++ b/sql/sql_sort.h > @@ -258,7 +258,9 @@ class Sort_keys :public Sql_alloc, > Sort_keys_array(arr, count), > m_using_packed_sortkeys(false), > size_of_packable_fields(0), > - sort_length(0) > + sort_length_with_original_values(0), > + sort_length_with_memcmp_values(0), > + first_execution(true) This violates the coding style. Please ident the member initialization (see e.g. item.h for examples) > { > DBUG_ASSERT(!is_null()); > }a ... > @@ -307,9 +319,12 @@ class Sort_keys :public Sql_alloc, > > void increment_original_sort_length(uint len) > { > - sort_length+= len; > + sort_length_with_original_values+= len; > } > > + bool is_first_execution() { return first_execution; } > + void set_first_execution(bool val) { first_execution= val; } A comment just about the naming: I would not drag "execution" into here. "Sort_keys" are not really executed. How about changing the functions to be bool is_parameters_computed(); bool set_parameters_computed(); and the variable to be "bool parameters_computed" ? > + > static const uint size_of_length_field= 4; > > private: BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 5150dfd6ab3: MDEV-17891 Assertion failures in select_insert::abort_result_set and mysql_load upon attempt to replace into a full table
by Sergei Golubchik 25 Oct '20

25 Oct '20

Hi, Nikita! On Apr 27, Nikita Malyavin wrote: > revision-id: 5150dfd6ab3 (mariadb-10.3.12-230-g5150dfd6ab3) > parent(s): cf78b8c699d > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2019-06-12 22:11:45 +1000 > message: > > MDEV-17891 Assertion failures in select_insert::abort_result_set and mysql_load upon attempt to replace into a full table > > * set modified_non_trans_table in one missed place This commit needs way more text than that. This is just too misterious now. What asserts fails? What has the test case to do with the fix? Why modified_non_trans_table even matters without replication? Why it's specific to versioning and partitioning? Where trans_safe becomes false? > diff --git a/mysql-test/suite/versioning/r/partition.result b/mysql-test/suite/versioning/r/partition.result > index 3fcb59bdb40..99a297417d5 100644 > --- a/mysql-test/suite/versioning/r/partition.result > +++ b/mysql-test/suite/versioning/r/partition.result > @@ -541,6 +541,29 @@ t1 CREATE TABLE `t1` ( > PARTITION BY SYSTEM_TIME INTERVAL 7 SECOND > (PARTITION `ver_p1` HISTORY ENGINE = DEFAULT_ENGINE, > PARTITION `ver_pn` CURRENT ENGINE = DEFAULT_ENGINE) > +set timestamp= default; > +# MDEV-17891 Assertion failures in select_insert::abort_result_set and > +# mysql_load upon attempt to replace into a full table > +set @@max_heap_table_size= 1024*1024; > +create or replace table t1 ( > +pk integer auto_increment, > +primary key (pk), > +f varchar(45000) > +) with system versioning engine=memory > +partition by system_time interval 1 year (partition p1 history, > +partition pn current); > +# fill the table until full > +insert into t1 () values (),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(); > +insert into t1 (f) select f from t1; > +ERROR HY000: The table 't1' is full > +# leave space for exactly one record in current partition > +delete from t1 where pk = 1; > +# copy all data into history partition > +replace into t1 select * from t1; > +replace into t1 select * from t1; > +ERROR HY000: The table 't1' is full > +drop table t1; > +set @@max_heap_table_size= 1048576; > # Test cleanup > drop database test; > create database test; > diff --git a/mysql-test/suite/versioning/t/partition.test b/mysql-test/suite/versioning/t/partition.test > index d5c83b4d3bb..23768836efc 100644 > --- a/mysql-test/suite/versioning/t/partition.test > +++ b/mysql-test/suite/versioning/t/partition.test > @@ -474,6 +474,7 @@ set timestamp=1523466002.799571; > insert into t1 values (11),(12); > set timestamp=1523466004.169435; > delete from t1 where pk in (11, 12); > +set timestamp= default; > > --echo # > --echo # MDEV-18136 Server crashes in Item_func_dyncol_create::prepare_arguments > @@ -489,6 +490,34 @@ partition by system_time interval column_get(column_create(7,7), 7 as int) secon > --replace_result $default_engine DEFAULT_ENGINE > show create table t1; > > +--echo # MDEV-17891 Assertion failures in select_insert::abort_result_set and > +--echo # mysql_load upon attempt to replace into a full table > + > +--let $max_heap_table_size_orig= `select @@max_heap_table_size;` > +set @@max_heap_table_size= 1024*1024; > +create or replace table t1 ( > + pk integer auto_increment, > + primary key (pk), > + f varchar(45000) > +) with system versioning engine=memory > + partition by system_time interval 1 year (partition p1 history, > + partition pn current); > + > +--echo # fill the table until full > +insert into t1 () values (),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(); > +--error ER_RECORD_FILE_FULL > +insert into t1 (f) select f from t1; > +--echo # leave space for exactly one record in current partition > +delete from t1 where pk = 1; > +--echo # copy all data into history partition > +replace into t1 select * from t1; > +--error ER_RECORD_FILE_FULL > +replace into t1 select * from t1; > + > +# cleanup > +drop table t1; > +eval set @@max_heap_table_size= $max_heap_table_size_orig; > + > --echo # Test cleanup > drop database test; > create database test; > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index 1f3a70721fc..44502ea6704 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -1953,6 +1953,8 @@ int write_record(THD *thd, TABLE *table,COPY_INFO *info) > if (likely(!error)) > { > info->deleted++; > + if (!table->file->has_transactions()) > + thd->transaction.stmt.modified_non_trans_table= TRUE; > if (table->versioned(VERS_TIMESTAMP)) > { > store_record(table, record[2]); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 05968211699: MDEV-16026: Global system_versioning_asof must not be used if client sessions can have non-default time zone
by Sergei Golubchik 25 Oct '20

25 Oct '20

Hi, Nikita! Okay. This reverts my "ok to push" on MDEV-16481 :) If you want to store the value in unix timestamp, you don't need MYSQL_TIME in save_result. You should calculate the value as a unix timestamp in ::check and store that in save_result. Now you still have timezone conversion in ::update and that can fail. Everything that can fail should be done in ::check, that's the contract. May be also I reapplied your patches to 10.3 incorrectly when I was reviewing. Perhaps you could rebase them yourself - to be sure I'm looking at the correct patch? On Oct 22, Nikita Malyavin wrote: > revision-id: 05968211699 (mariadb-10.4.11-291-g05968211699) > parent(s): efb1023b6f4 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Sergei Golubchik <serg(a)mariadb.com> > timestamp: 2020-10-22 17:07:03 +0200 > message: > > MDEV-16026: Global system_versioning_asof must not be used if client sessions can have non-default time zone > > * store `system_versioning_asof` in unix time; > * both session and global vars are processed in session timezone; > * setting `default` does not copy global varibale abymore. Instead, it sets system_time to SYSTEM_TIME_UNSPECIFIED, which means that no 'AS OF' time is applied and `now()` can be assumed > As a regression, we cannot assign values below 1970 anymore > > --- > mysql-test/suite/versioning/r/sysvars.result | 67 +++++++++++++++++++++++ > mysql-test/suite/versioning/t/sysvars.test | 80 ++++++++++++++++++++++++---- > sql/mysqld.h | 3 +- > sql/sql_select.cc | 6 ++- > sql/sys_vars.ic | 27 ++++++---- > 5 files changed, 162 insertions(+), 21 deletions(-) > > diff --git a/sql/mysqld.h b/sql/mysqld.h > index bd45ff7b798..884e5a06066 100644 > --- a/sql/mysqld.h > +++ b/sql/mysqld.h > @@ -194,7 +194,8 @@ enum vers_system_time_t > struct vers_asof_timestamp_t > { > ulong type; > - MYSQL_TIME ltime; > + my_time_t unix_time; > + ulong second_part; > }; > > enum vers_alter_history_enum > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index d0bb0c816ec..4d62a9829eb 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -722,8 +722,12 @@ bool vers_select_conds_t::init_from_sysvar(THD *thd) > if (type != SYSTEM_TIME_UNSPECIFIED && type != SYSTEM_TIME_ALL) > { > DBUG_ASSERT(type == SYSTEM_TIME_AS_OF); > + MYSQL_TIME ltime; > + thd->variables.time_zone->gmt_sec_to_TIME(&ltime, in.unix_time); > + ltime.second_part = in.second_part; > + > start.item= new (thd->mem_root) > - Item_datetime_literal(thd, &in.ltime, TIME_SECOND_PART_DIGITS); > + Item_datetime_literal(thd, &ltime, TIME_SECOND_PART_DIGITS); > if (!start.item) > return true; > } > diff --git a/sql/sys_vars.ic b/sql/sys_vars.ic > index f33f469b160..417fa7842c2 100644 > --- a/sql/sys_vars.ic > +++ b/sql/sys_vars.ic > @@ -2646,9 +2646,11 @@ class Sys_var_vers_asof: public Sys_var_enum > } > > private: > - bool update(set_var *var, vers_asof_timestamp_t &out) > + bool update(set_var *var, vers_asof_timestamp_t &out, system_variables& pool) > { > bool res= false; > + uint error; > + MYSQL_TIME ltime; > out.type= static_cast<enum_var_type>(var->save_result.ulonglong_value); > if (out.type == SYSTEM_TIME_AS_OF) > { > @@ -2658,11 +2660,14 @@ class Sys_var_vers_asof: public Sys_var_enum > Datetime::Options opt(TIME_CONV_NONE | > TIME_NO_ZERO_IN_DATE | > TIME_NO_ZERO_DATE, thd); > - res= var->value->get_date(thd, &out.ltime, opt); > + res= var->value->get_date(thd, &ltime, opt); > + out.unix_time= pool.time_zone->TIME_to_gmt_sec(&ltime, &error); > + out.second_part= ltime.second_part; > + res|= (error != 0); > } > else // set DEFAULT from global var > { > - out= global_var(vers_asof_timestamp_t); > + out.type= SYSTEM_TIME_UNSPECIFIED; > res= false; > } > } > @@ -2672,11 +2677,11 @@ class Sys_var_vers_asof: public Sys_var_enum > public: > virtual bool global_update(THD *thd, set_var *var) > { > - return update(var, global_var(vers_asof_timestamp_t)); > + return update(var, global_var(vers_asof_timestamp_t), thd->variables); > } > virtual bool session_update(THD *thd, set_var *var) > { > - return update(var, session_var(thd, vers_asof_timestamp_t)); > + return update(var, session_var(thd, vers_asof_timestamp_t), thd->variables); > } > > private: > @@ -2690,10 +2695,14 @@ class Sys_var_vers_asof: public Sys_var_enum > case SYSTEM_TIME_AS_OF: > { > uchar *buf= (uchar*) thd->alloc(MAX_DATE_STRING_REP_LENGTH); > - if (buf &&!my_datetime_to_str(&val.ltime, (char*) buf, 6)) > + MYSQL_TIME ltime; > + > + thd->variables.time_zone->gmt_sec_to_TIME(&ltime, val.unix_time); > + ltime.second_part= val.second_part; > + > + if (buf && !my_datetime_to_str(&ltime, (char*) buf, 6)) > { > - // TODO: figure out variable name > - my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), "system_versioning_asof_timestamp", "NULL (wrong datetime)"); > + my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), name.str, "NULL (wrong datetime)"); > return (uchar*) thd->strdup("Error: wrong datetime"); > } > return buf; > @@ -2701,7 +2710,7 @@ class Sys_var_vers_asof: public Sys_var_enum > default: > break; > } > - my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), "system_versioning_asof_timestamp", "NULL (wrong range type)"); > + my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), name.str, "NULL (wrong range type)"); > return (uchar*) thd->strdup("Error: wrong range type"); > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] caec6b4b3d7: MDEV-17573 Assertion in federatedx on multi-update
by Sergei Golubchik 23 Oct '20

23 Oct '20

Hi, Aleksey! Looks reasonable to me. MYSQL_RES is stored in the handler. Of course, MYSQL_ROWS *current, which is a pointer into MYSQL_RES, has to be in the handler, it cannot be shared. Ok to push, thanks! On Oct 23, Aleksey Midenkov wrote: > revision-id: caec6b4b3d7 (mariadb-10.3.21-250-gcaec6b4b3d7) > parent(s): a00c668ddf2 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-10-21 13:38:37 +0300 > message: > > MDEV-17573 Assertion in federatedx on multi-update > > Cause: shared federatedx_io cannot store table-specific data. > > Fix: move current row reference `federatedx_io_mysql::current` to > ha_federatedx. > > FederatedX connection (represented by federatedx_io) is stored into > federatedx_txn::txn_list of per-server connections (see > federatedx_txn::acquire()). federatedx_txn object is stored into THD > (see ha_federatedx::external_lock()). When multiple handlers acquire > FederatedX connection they get single federatedx_io instance. Multiple > handlers do their operation via federatedx_io_mysql::mark_position() > and federatedx_io_mysql::fetch_row() in arbitrarty manner. They access > the same federatedx_io_mysql instance and same MYSQL_ROWS *current > pointer, so one handler disrupts the work of the other. > > Related to "MDEV-14551 Can't find record in table on multi-table update > with ORDER BY". Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2e0bac4688a: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 23 Oct '20

23 Oct '20

Hi, Anel! On Oct 23, Anel Husakovic wrote: > revision-id: 2e0bac4688a (mariadb-10.2.31-517-g2e0bac4688a) > parent(s): 985ede92034 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-23 12:38:11 +0200 > message: > > MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > > diff --git a/mysql-test/r/grant5.result b/mysql-test/r/grant5.result > index 338bbd36e14..a69070e8399 100644 > --- a/mysql-test/r/grant5.result > +++ b/mysql-test/r/grant5.result ... > +SHOW GRANTS; > +Grants for root@localhost > +GRANT test_role TO 'root'@'localhost' WITH ADMIN OPTION > +GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION > +GRANT PROXY ON ''@'%' TO 'root'@'localhost' WITH GRANT OPTION > +GRANT USAGE ON *.* TO 'test_role' > +SET DEFAULT ROLE test_role FOR 'root'@'localhost' ... Yes, that's what I mean. I don't have a strong opinion on that, it seems that I very slightly prefer to have SET DEFAULT ROLE after GRANT PROXY, that is to have all statements for root, and then statements for roles. But having SET DEFAULT ROLE at the end makes sense too - all GRANTs first, then SET DEFAULT ROLE. Just make it the way you want. But either way it's good to have a test for it, a test that shows we've considered this issue and intentionally implemented that particular order. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6d881a271a9: MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING
by Sergei Golubchik 23 Oct '20

23 Oct '20

Hi, Rucha! On Oct 23, Rucha Deodhar wrote: > revision-id: 6d881a271a9 (mariadb-10.5.4-108-g6d881a271a9) > parent(s): bbd70fcc43c > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-10-23 12:57:02 +0530 > message: > > MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 45ce4be3eb5..4ce7b034029 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -7959,7 +7959,7 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > bool any_privileges, uint *hidden_bit_fields) > { > Field_iterator_table_ref field_iterator; > - bool found; > + bool found, is_insert_or_replace_returning= false; > char name_buff[SAFE_NAME_LEN+1]; > DBUG_ENTER("insert_fields"); > DBUG_PRINT("arena", ("stmt arena: %p",thd->stmt_arena)); > @@ -7978,13 +7978,31 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > > found= FALSE; > > + if ((thd->lex->sql_command == SQLCOM_INSERT_SELECT || > + thd->lex->sql_command == SQLCOM_REPLACE_SELECT || > + thd->lex->sql_command == SQLCOM_INSERT || > + thd->lex->sql_command == SQLCOM_REPLACE) && > + thd->lex->has_returning()) > + is_insert_or_replace_returning= true; > + > /* > If table names are qualified, then loop over all tables used in the query, > else treat natural joins as leaves and do not iterate over their underlying > tables. > + Also, When we have INSERT/REPLACE...RETURNING and have qualified asterisk, > + table_name is not NULL and context->table_list is either NULL or has > + incorrect reference because context->table_list has tables from the > + FROM clause. > + context->table_list has incorrect reference (has table from the FROM clause > + instead of table we are inserting into) for > + INSERT/REPLACE...SELECT...RETURNING because we have a FROM clause from the > + SELECT statement. For INSERT/REPLACE...RETURNING it is NULL because > + there is no FROM clause. > */ > - for (TABLE_LIST *tables= (table_name ? context->table_list : > - context->first_name_resolution_table); > + for (TABLE_LIST *tables= (table_name ? (is_insert_or_replace_returning ? > + context->first_name_resolution_table : > + context->table_list) : > + (context->first_name_resolution_table)); ok, I've compiled it to run few queries, now I see that this is wrong. you need to understand the old code and the old comment before modifying them. one way to do it is to change the condition from table_name ? context->table_list : context->first_name_resolution_table to simply context->first_name_resolution_table and see what breaks. You'll see that main.type_varchar will break with At line 319: query 'SELECT t1.* FROM t1 LEFT JOIN t2 USING (c1)' failed: 1051: Unknown table 'test.t1' indeed, t1 is only present in context->table_list, but not in context->first_name_resolution_table. So after your fix the following statement will no longer work: INSERT INTO t2 SELECT t1.* FROM t1 JOIN t1 AS t USING (id1) WHERE id1=1 RETURNING *; it's because you set is_insert_or_replace_returning both for the SELECT and for the RETURNING clauses. > tables; > tables= (table_name ? tables->next_local : > tables->next_name_resolution_table) also this looks suspicious. Normally one starts from table_list and follows the next_local pointer or one starts from first_name_resolution_table and follows the next_name_resolution_table pointer. Your change creates a strange mix of both, it looks like it would be wrong. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 581857bd999: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 23 Oct '20

23 Oct '20

Hi, Anel! Looks good, thanks! A couple of small comments below. On Oct 23, Anel Husakovic wrote: > revision-id: 581857bd999 (mariadb-10.1.43-314-g581857bd999) let's fix it in 10.2 and up, we're doing the very last 10.1 release, so better be extra-super-conservative about it. > parent(s): 43ec9370b32 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-23 00:20:53 +0200 > message: > > MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > > diff --git a/mysql-test/t/grant5.test b/mysql-test/t/grant5.test > index 74a69952124..bf97a930097 100644 > --- a/mysql-test/t/grant5.test > +++ b/mysql-test/t/grant5.test > @@ -52,6 +52,23 @@ disconnect u1; > drop user u1@localhost; > drop database mysqltest1; > > +# > +# MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > +# > +CREATE ROLE test_role; > +CREATE USER test_user; > +GRANT test_role TO test_user; > +SET DEFAULT ROLE test_role FOR test_user; > +SHOW GRANTS FOR test_user; > +SET DEFAULT ROLE NONE for test_user; > +SHOW GRANTS FOR test_user; > +SET DEFAULT ROLE test_role; oops, sorry, my mistake. Please, add `SET ROLE test_role` before SHOW GRANTS. The goal is to see that SET DEFAULT ROLE is shown *after* all grants to root@localhost, but before any grants to test_role. > +SHOW GRANTS; > +SET DEFAULT ROLE NONE; > +SHOW GRANTS; > +DROP USER test_user; > +DROP ROLE test_role; > + > # > # End of 10.1 tests > # > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index cf0b1d87bd7..6c852976061 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -8001,15 +8007,44 @@ static ROLE_GRANT_PAIR *find_role_grant_pair(const LEX_STRING *u, > my_hash_search(&acl_roles_mappings, (uchar*)pair_key.ptr(), key_length); > } > > +static bool show_default_role(THD *thd, const char *hostname, > + ACL_USER *acl_entry, char *buff, size_t buffsize) > +{ > + Protocol *protocol= thd->protocol; > + LEX_STRING def_rolename= acl_entry->default_rolename; > + > + if (def_rolename.length) > + { > + String def_str(buff, buffsize, system_charset_info); > + def_str.length(0); > + def_str.append(STRING_WITH_LEN("SET DEFAULT ROLE ")); > + def_str.append(&def_rolename); > + def_str.append(" FOR '"); > + def_str.append(&acl_entry->user); > + DBUG_ASSERT(!(acl_entry->flags & IS_ROLE)); > + def_str.append(STRING_WITH_LEN("'@'")); > + def_str.append(acl_entry->host.hostname, acl_entry->hostname_length, > + system_charset_info); > + def_str.append('\''); > + protocol->prepare_for_resend(); > + protocol->store(def_str.ptr(),def_str.length(),def_str.charset()); > + if (protocol->write()) > + { > + return TRUE; > + } > + } > + return FALSE; looks good. The hostname argument seems to be unused now, doesn't it? > +} > + > -static bool show_role_grants(THD *thd, const char *username, > - const char *hostname, ACL_USER_BASE *acl_entry, > +static bool show_role_grants(THD *thd, const char *hostname, > + ACL_USER_BASE *acl_entry, > char *buff, size_t buffsize) > { > uint counter; > Protocol *protocol= thd->protocol; > LEX_STRING host= {const_cast<char*>(hostname), strlen(hostname)}; > > - String grant(buff,sizeof(buff),system_charset_info); > + String grant(buff, buffsize, system_charset_info); ouch, indeed. thanks! > for (counter= 0; counter < acl_entry->role_grants.elements; counter++) > { > grant.length(0); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 01612f594b3: MDEV-23769: MTR can abort before it prints the test result summary
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Anel! On Oct 22, Anel Husakovic wrote: > Hi Sergei, > thanks for the review. > > Having warning/error is a necessary condition. > Sufficient condition is to call at least 2 tests. > Incorrect behaviour can be seen if: > - Both tests are correct but there is error generated (incorrect option > passed): > So this `./mysql-test/mtr --mysqld=--lock-wait-timeout=-xx 1st` will > generate and write statistic at the end, > but this `./mysql-test/mtr --mysqld=--lock-wait-timeout=-xx 1st 1st > --force` will not generate the statistic. > Note: warning is generated (--mysqld=--lock-wait-timeout=-1) statistic > will be generated Okay, thanks. This would be useful to show in the commit comment, that one can use ./mtr --mysqld=--lock-wait-timeout=-xx 1st 1st --force --parallel 1 to repeat the error. But then I don't understand why this happens. Why one failure is not enough? Why --parallel 2 destroys the effect? Apparently, it's check_warnings_post_shutdown() that causes that, not exit(1). One can change to exit(0) or remove exit(1) completely - the bug will still happen. Why check_warnings_post_shutdown() writing to the master server causes this bug? It looks rather fishy that in your fix it's the worker process that prints the statistics, not the master. Btw, note that check_warnings_post_shutdown() doesn't have any return statement, it very implicitly return "number of bytes sent" - it would be easier to read if it'd return something explicitly. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 99c14476251: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Anel! On Oct 22, Anel Husakovic wrote: > revision-id: 99c14476251 (mariadb-10.1.43-314-g99c14476251) > parent(s): 43ec9370b32 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-22 16:49:28 +0200 > message: > > MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > > diff --git a/mysql-test/t/grant5.test b/mysql-test/t/grant5.test > index 74a69952124..8756648c3e1 100644 > --- a/mysql-test/t/grant5.test > +++ b/mysql-test/t/grant5.test > @@ -53,5 +53,19 @@ drop user u1@localhost; > drop database mysqltest1; > > # > -# End of 10.1 tests > +# MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > +# > +CREATE ROLE test_role; > +CREATE USER test_user; > +GRANT test_role TO test_user; > +SET DEFAULT ROLE test_role FOR test_user; > +SHOW GRANTS FOR test_user; > +SET DEFAULT ROLE NONE for test_user; > +SHOW GRANTS FOR test_user; > +SHOW GRANTS; The point is to run `SHOW GRANTS` for a user with a default role. E.g. SET DEFAULT ROLE test_role; SHOW GRANTS; SET DEFAULT ROLE NONE; > +DROP USER test_user; > +DROP ROLE test_role; > + > # > +# End of 10.1 tests > +# > \ No newline at end of file new line at the end of file? > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index cf0b1d87bd7..f42f7bdcfaf 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -8001,6 +8007,36 @@ static ROLE_GRANT_PAIR *find_role_grant_pair(const LEX_STRING *u, > my_hash_search(&acl_roles_mappings, (uchar*)pair_key.ptr(), key_length); > } > > +static bool show_default_role(THD *thd, const char *hostname, > + ACL_USER *acl_entry, char *buff) > +{ > + Protocol *protocol= thd->protocol; > + LEX_STRING host= {const_cast<char*>(hostname), strlen(hostname)}; > + > + LEX_STRING def_rolename= acl_entry->default_rolename; > + if (def_rolename.length) > + { > + String def_str(buff,sizeof(buff),system_charset_info); eh? sizeof(buff) is sizeof(char*) that is usually 8 on 64-bit, 4 on 32-bit architecture. You want the size of the buffer here, not the size of the pointer to the buffer. > + def_str.length(0); > + def_str.append(STRING_WITH_LEN("SET DEFAULT ROLE ")); > + def_str.append(&def_rolename); > + def_str.append(" FOR '"); > + def_str.append(acl_entry->user.str, acl_entry->user.length, > + system_charset_info); again, just as above, you can use def_str.append(&acl_entry->user); > + DBUG_ASSERT(!(acl_entry->flags & IS_ROLE)); > + def_str.append(STRING_WITH_LEN("'@'")); > + def_str.append(&host); why not to use def_str.append(acl_entry->host.hostname, acl_entry->hostname_length, system_charset_info); > + def_str.append('\''); > + protocol->prepare_for_resend(); > + protocol->store(def_str.ptr(),def_str.length(),def_str.charset()); > + if (protocol->write()) > + { > + return TRUE; > + } > + } > + return FALSE; > +} Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6729bf5c031: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Anel! On Oct 22, Anel Husakovic wrote: > Hi Sergei, > thank you for your review. > How is easier for you (or to ask a general question: how does the review > process go?) regarding the commits? > Is it better to add a new commit as a review commit and if the patch gets > accepted to squash after, or to rebase commits and send only a single > commit again for the review? One single commit is better, thanks. At least for small patches. > Addressed in the new patch: > https://github.com/MariaDB/server/commit/99c144762514d4dea3b6ec395de96aaf10… Thanks! I'll reply to the commit directly. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 83614913d9a: MDEV-16243 [Warning] "Versioned table ...: partition ... is full, add more HISTORY partitions" floods the error log
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Aleksey! On Oct 22, Aleksey Midenkov wrote: > revision-id: 83614913d9a (mariadb-10.4.11-291-g83614913d9a) > parent(s): efb1023b6f4 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Sergei Golubchik <serg(a)mariadb.com> > timestamp: 2020-10-22 16:44:13 +0200 > message: > > MDEV-16243 [Warning] "Versioned table ...: partition ... is full, add more HISTORY partitions" floods the error log > > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index 8989a918c0c..f287cfddcd2 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -879,9 +879,12 @@ void partition_info::vers_set_hist_part(THD *thd) > } > return; > warn: > - my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > - table->s->db.str, table->s->table_name.str, > - vers_info->hist_part->partition_name); > + if (vers_info->hist_part->warn_vers_part_full) > + { > + my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > + table->s->db.str, table->s->table_name.str, > + vers_info->hist_part->partition_name); > + } would it make sense to put warn_vers_part_full into vers_info, instead of storing it per-partition? > } > > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2830b9c290e: MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Dmitry! On Oct 22, Dmitry Shulga wrote: > revision-id: 2830b9c290e (mariadb-10.2.31-489-g2830b9c290e) > parent(s): 8984d779103 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-22 00:42:10 +0700 > message: > > MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS > > Compiler warnings like one listed below are generated during server build on MacOS: > > > --- > plugin/auth_pam/CMakeLists.txt | 26 ++++++++++++++++++++++++++ > plugin/auth_pam/config.h.cmake | 1 + > plugin/auth_pam/mapper/pam_user_map.c | 30 +++++++++++++++++++----------- > 3 files changed, 46 insertions(+), 11 deletions(-) Nice! Could you also - in a second follow-up commit, not to confuse with the pure bug fix - do a small cleanup and move all other checks from plugin/auth_pam/CMakeLists.txt into your new config.h.cmake ? Currently they're added to the command line, like IF(HAVE_PAM_EXT_H) ADD_DEFINITIONS(-DHAVE_PAM_EXT_H) ENDIF() but as you've introduced config.h.cmake, there's no need to put them all on the command line anymore. You'll need to include config_auth_pam.h into more source files. And, I suppose. add config_auth_pam.h to .gitignore too. ok to push these two commits when done. Thanks! Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 65af1d07771: MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Rucha! On Oct 22, Rucha Deodhar wrote: > revision-id: 65af1d07771 (mariadb-10.5.4-108-g65af1d07771) > parent(s): bbd70fcc43c > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-08-06 17:23:49 +0530 > message: > > MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING > > Anaylsis: When we have INSERT/REPLACE...RETURNING and have qualified asterisk, > table_name is not NULL and context->table_list is either NULL or has > incorrect reference because context->table_list has tables from the FROM clause. > context->table_list has incorrect reference (has table from the FROM clause > instead of table we are inserting into) for INSERT/REPLACE...SELECT...RETURNING > because we have a FROM clause from the SELECT statement. > For INSERT/REPLACE...RETURNING it is NULL because there is no FROM clause. > Fix: If table_name is not null, check if we have INSERT/REPLACE...RETURNING. > If so, the reference should be the table we are inserting into and not > the table in the FROM clause. > > diff --git a/mysql-test/main/insert_returning.result b/mysql-test/main/insert_returning.result > index e664e02bedc..e1bab6f0329 100644 > --- a/mysql-test/main/insert_returning.result > +++ b/mysql-test/main/insert_returning.result > @@ -89,6 +89,9 @@ total val1 id1 && id1 id1 UPPER(val1) f(id1) good tests! > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 45ce4be3eb5..4ce7b034029 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -7959,7 +7959,7 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > bool any_privileges, uint *hidden_bit_fields) > { > Field_iterator_table_ref field_iterator; > - bool found; > + bool found, is_insert_or_replace_returning= false; > char name_buff[SAFE_NAME_LEN+1]; > DBUG_ENTER("insert_fields"); > DBUG_PRINT("arena", ("stmt arena: %p",thd->stmt_arena)); > @@ -7978,13 +7978,31 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > > found= FALSE; > > + if ((thd->lex->sql_command == SQLCOM_INSERT_SELECT || > + thd->lex->sql_command == SQLCOM_REPLACE_SELECT || > + thd->lex->sql_command == SQLCOM_INSERT || > + thd->lex->sql_command == SQLCOM_REPLACE) && > + thd->lex->has_returning()) > + is_insert_or_replace_returning= true; What about INSERT t1 SELECT t2.* FROM t2 RETURNING t1.*; can your is_insert_or_replace_returning distinguish between the first and the second asterisk? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 01612f594b3: MDEV-23769: MTR can abort before it prints the test result summary
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Anel! On Oct 22, Anel Husakovic wrote: > revision-id: 01612f594b3 (mariadb-10.1.43-300-g01612f594b3) > parent(s): 82301aea4f2 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-03 02:34:03 +0200 > message: > > MDEV-23769: MTR can abort before it prints the test result summary > > - Patch is solving generating report on warning > - Patch is doing cosmetic fix of `current_test` log file which holds the old log value of test `CURRENT TEST:..` in `mark_log()` in case of `unknown option` and as such > the logic which is using it's content doesn't output valid log content and doesn't generate valid `$test->{'comment'}` message. more is not always better. please remove everything in the commit comment below this line - 400 lines that don't explain anything beyond what your *one line* of the comment already did. And now you have to press PageDown many times to get to my main comment about the fix :) > Here are the test cases observed. Note that only in 6. scenario there is `print $child "HELLO\n";` problem in printing (to delete?). > > 1. Valid test case, but warning in server log > ``` > $ ./mtr MDEV-23769 --force > Logging: ./mtr MDEV-23769 --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:13:00 > line > 2020-10-03 2:13:00 140309996242496 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > -------------------------------------------------------------------------- > The servers were restarted 0 times > Spent 0.000 of 4 seconds executing testcases > mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/warnings' for details. > > Completed: Failed 1/1 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > mysql-test-run: *** ERROR: there were failing test cases > ``` > > 2. Valid test cases (same), but warning in server log (Note reported single test case) > ``` > $ ./mtr MDEV-23769 MDEV-23769 --force > Logging: ./mtr MDEV-23769 MDEV-23769 --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:15:05 > line > 2020-10-03 2:15:04 139868136609344 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:15:05 > line > 2020-10-03 2:15:05 140513782085184 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.000 of 5 seconds executing testcases > mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/warnings' for details. > > Completed: Failed 2/2 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > mysql-test-run: *** ERROR: there were failing test cases > ``` > > 3. Valid test cases (different test cases), but warning in server log (Note one passed one failed) > > ``` > $ ./mtr MDEV-23769 1st --force > Logging: ./mtr MDEV-23769 1st --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:17:09 > line > 2020-10-03 2:17:08 140658805515840 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > main.1st [ pass ] 1 > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.001 of 4 seconds executing testcases > mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/warnings' for details. > > Completed: Failed 1/2 tests, 50.00% were successful. > > Failing test(s): main.MDEV-23769 > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > mysql-test-run: *** ERROR: there were failing test cases > > ``` > > 4. Now for example 3. create artificial failure with warnings in server log (note, both test failed) > > ``` > > $ ./mtr MDEV-23769 1st --force > Logging: ./mtr MDEV-23769 1st --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:17:52 > line > 2020-10-03 2:17:51 140260493149760 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > main.1st [ fail ] > Test ended at 2020-10-03 02:17:52 > > CURRENT_TEST: main.1st > mysqltest: At line 5: End of line junk detected: "show tables in mysql > " > > The result from queries just before the failure was: > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.1st/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.1st/' > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.000 of 4 seconds executing testcases > mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/warnings' for details. > > Completed: Failed 2/2 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 main.1st > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > mysql-test-run: *** ERROR: there were failing test cases > > ``` > > 5. Generate unknown option, for the same example as 3. (note: both failed) > ``` > > $ ./mtr MDEV-23769 1st --mysqld=--xx --force > Logging: ./mtr MDEV-23769 1st --mysqld=--xx --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 52531, winpid: 52531, exit: 512] died after mysql-test-run waited 0.3 seconds for /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/run/mysqld.1.pid to be created. > main.MDEV-23769 [ fail ] > Test ended at 2020-10-03 02:19:01 > > CURRENT_TEST: main.MDEV-23769 > > Failed to start mysqld.1 > mysqltest failed but provided no output > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > ***Warnings generated in error logs during shutdown after running tests: main.MDEV-23769 > > 2020-10-03 2:19:01 140238937720384 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > 2020-10-03 2:19:01 140238937720384 [ERROR] /home/anel/mariadb/10.1-mtr-in-source/sql/mysqld: unknown option '--xx' > 2020-10-03 2:19:01 140238937720384 [ERROR] Aborting > > worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 52537, winpid: 52537, exit: 512] died after mysql-test-run waited 0.3 seconds for /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/run/mysqld.1.pid to be created. > main.1st [ fail ] > Test ended at 2020-10-03 02:19:02 > > CURRENT_TEST: main.1st > > Failed to start mysqld.1 > mysqltest failed but provided no output > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.1st/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.1st/' > ***Warnings generated in error logs during shutdown after running tests: main.1st > > 2020-10-03 2:19:02 140048201700928 [ERROR] /home/anel/mariadb/10.1-mtr-in-source/sql/mysqld: unknown option '--xx' > 2020-10-03 2:19:02 140048201700928 [ERROR] Aborting > > Can't use an undefined value as a symbol reference at ./mtr line 657. > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.000 of 5 seconds executing testcases > > Completed: Failed 2/2 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 main.1st > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > Errors/warnings were found in logfiles during server shutdown after running the > following sequence(s) of tests: > main.MDEV-23769 > main.1st > mysql-test-run: *** ERROR: there were failing test cases > > ``` > > 6. Again the same option, but with 2 same tests -> not perl error: `Can't use an undefined value as a symbol reference at ./mtr line 657. > ` ` print $child "HELLO\n";` (solution delete the line). > > ``` > $ ./mtr MDEV-23769 MDEV-23769 --mysqld=--xx --force > Logging: ./mtr MDEV-23769 MDEV-23769 --mysqld=--xx --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 52595, winpid: 52595, exit: 512] died after mysql-test-run waited 0.3 seconds for /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/run/mysqld.1.pid to be created. > main.MDEV-23769 [ fail ] > Test ended at 2020-10-03 02:20:12 > > CURRENT_TEST: main.MDEV-23769 > > Failed to start mysqld.1 > mysqltest failed but provided no output > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > ***Warnings generated in error logs during shutdown after running tests: main.MDEV-23769 > > 2020-10-03 2:20:12 139815724283456 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > 2020-10-03 2:20:12 139815724283456 [ERROR] /home/anel/mariadb/10.1-mtr-in-source/sql/mysqld: unknown option '--xx' > 2020-10-03 2:20:12 139815724283456 [ERROR] Aborting > > worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 52601, winpid: 52601, exit: 512] died after mysql-test-run waited 0.3 seconds for /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/run/mysqld.1.pid to be created. > main.MDEV-23769 [ fail ] > Test ended at 2020-10-03 02:20:12 > > CURRENT_TEST: main.MDEV-23769 > > Failed to start mysqld.1 > mysqltest failed but provided no output > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > ***Warnings generated in error logs during shutdown after running tests: main.MDEV-23769 > > 2020-10-03 2:20:12 140650797745728 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > 2020-10-03 2:20:12 140650797745728 [ERROR] /home/anel/mariadb/10.1-mtr-in-source/sql/mysqld: unknown option '--xx' > 2020-10-03 2:20:12 140650797745728 [ERROR] Aborting > > Can't use an undefined value as a symbol reference at ./mtr line 657. > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.000 of 5 seconds executing testcases > > Completed: Failed 2/2 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > Errors/warnings were found in logfiles during server shutdown after running the > following sequence(s) of tests: > main.MDEV-23769 > main.MDEV-23769 > mysql-test-run: *** ERROR: there were failing test cases > > ``` > > diff --git a/mysql-test/disabled.def b/mysql-test/disabled.def > index ee30c4f4d3c..fdb62a7e5f5 100644 > --- a/mysql-test/disabled.def > +++ b/mysql-test/disabled.def > @@ -20,3 +20,4 @@ ssl_crl_clrpath : broken upstream > file_contents : MDEV-6526 these files are not installed anymore > max_statement_time : cannot possibly work, depends on timing > partition_open_files_limit : open_files_limit check broken by MDEV-18360 > +MDEV-23769 : Intentionally disabled, for validating purposes. I don't know if it makes any sense. A dead test that nobody will ever run. May be it's better not to add it at all? In the past I've added tests like that too, the "mtr" and "mtr2" suites. I regret it now :) Tests only help if they're run. > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index 52920530132..8d761fd70cf 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -1005,6 +1005,13 @@ sub run_worker ($) { > stop_servers(reverse all_servers()); > if(check_warnings_post_shutdown($server)) { > # Warnings appeared in log file(s) during final server shutdown. > + # Make sure first to run statistics > + my $tests= collect_test_cases($opt_reorder, $opt_suites, \@opt_cases, \@opt_skip_test_list); > + my ($prefix, $fail, $completed, $extra_warnings)= > + run_test_server($server, $tests, $opt_parallel); > + mtr_print_line(); > + print_total_times($opt_parallel) if $opt_report_times; > + mtr_report_stats($prefix, $fail, $completed, $extra_warnings); I don't understand the fix. You're saying that if a warning happens on shutdown then mtr won't print the result summary. This doesn't seem to be the case, I've tried few times to repeat it - modified mtr to always pretend there's a warning on shutdown, and modified mysqld to always issue a warning on shutdown - and in all cases I saw the result summary all right. > exit(1); > } > } > @@ -3971,6 +3978,7 @@ sub run_testcase ($$) { > if (start_servers($tinfo)) > { > report_failure_and_restart($tinfo); > + unlink $path_current_testlog; > return 1; > } > } > diff --git a/mysql-test/t/MDEV-23769.opt b/mysql-test/t/MDEV-23769.opt > new file mode 100644 > index 00000000000..fd8dbcab53f > --- /dev/null > +++ b/mysql-test/t/MDEV-23769.opt > @@ -0,0 +1 @@ > +--lock-wait-timeout=-1 > diff --git a/mysql-test/t/MDEV-23769.test b/mysql-test/t/MDEV-23769.test > new file mode 100644 > index 00000000000..9938c7cf644 > --- /dev/null > +++ b/mysql-test/t/MDEV-23769.test > @@ -0,0 +1,4 @@ > +--echo # > +--echo # MDEV-23769: MTR can abort before it prints the test result summary > +--echo # > +show databases; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 8c745467560: MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rucha! what happens in this bug? Why does the server need to open_table() after my_ok()? On Oct 21, Rucha Deodhar wrote: > revision-id: 8c745467560 (mariadb-10.5.4-226-g8c745467560) > parent(s): 308f8350c7b > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-10-20 11:53:22 +0530 > message: > > MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in > Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK) > > Analysis: Assertion failure happens because we don't ignore kill query > even if 'OK' is already sent. > Fix: Check if we already sent 'OK'. If sent, ignore kill query. > > diff --git a/mysql-test/main/alter_table.result b/mysql-test/main/alter_table.result > index 197e52b35ca..204bfa5249c 100644 > --- a/mysql-test/main/alter_table.result > +++ b/mysql-test/main/alter_table.result > @@ -3322,5 +3322,23 @@ Note 1176 Key 'x' doesn't exist in table 't1' > unlock tables; > drop table t1; > # > +# MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in > +# Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK) > +# > +set @max_session_mem_used_save= @@max_session_mem_used; > +CREATE TABLE t1 (a INT); > +SELECT * FROM t1; > +a > +ALTER TABLE x MODIFY xx INT; > +ERROR 42S02: Table 'test.x' doesn't exist > +SET SESSION max_session_mem_used= 8192; > +LOCK TABLE t1 WRITE; > +ALTER TABLE t1 CHANGE COLUMN IF EXISTS b c INT; > +Warnings: > +Note 1054 Unknown column 'b' in 't1' > +set SESSION max_session_mem_used = @max_session_mem_used_save; > +UNLOCK TABLES; > +DROP TABLE t1; > +# > # End of 10.5 tests > # > diff --git a/mysql-test/main/alter_table.test b/mysql-test/main/alter_table.test > index 6fe61a3222a..d09c4378339 100644 > --- a/mysql-test/main/alter_table.test > +++ b/mysql-test/main/alter_table.test > @@ -2529,6 +2529,27 @@ alter table t1 rename key if exists x to xx; > unlock tables; > drop table t1; > > +--echo # > +--echo # MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in > +--echo # Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK) > +--echo # > + > +set @max_session_mem_used_save= @@max_session_mem_used; > + > +CREATE TABLE t1 (a INT); > +SELECT * FROM t1; > + > +--error ER_NO_SUCH_TABLE > +ALTER TABLE x MODIFY xx INT; > + > +SET SESSION max_session_mem_used= 8192; > +LOCK TABLE t1 WRITE; > +ALTER TABLE t1 CHANGE COLUMN IF EXISTS b c INT; > + > +set SESSION max_session_mem_used = @max_session_mem_used_save; > +UNLOCK TABLES; > +DROP TABLE t1; > + > --echo # > --echo # End of 10.5 tests > --echo # > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index e6d0ab13e75..e8d39917ba6 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -1682,7 +1682,9 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > > if (!(flags & MYSQL_OPEN_IGNORE_KILLED) && thd->killed) > { > - thd->send_kill_message(); > + /* If we already sent 'ok', we can ignore any kill query statements */ > + if (! thd->get_stmt_da()->is_set()) > + thd->send_kill_message(); > DBUG_RETURN(TRUE); > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1060cc4235a: MDEV-23500: Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())'
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rucha! On Oct 21, Rucha Deodhar wrote: > revision-id: 1060cc4235a (mariadb-10.2.31-438-g1060cc4235a) > parent(s): 80075ba011f > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-09-22 17:12:57 +0530 > message: > > MDEV-23500: Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())' > failed in Diagnostics_area::set_ok_status upon multi-table update > > Analysis: When NULL field is copied to NOT-NULL so there is error but > this error is not reported so we get the failure. > Fix: Return the error state. > > diff --git a/mysql-test/r/update.result b/mysql-test/r/update.result > index 9e19abc4e9c..c1d37fc36e7 100644 > --- a/mysql-test/r/update.result > +++ b/mysql-test/r/update.result > @@ -731,4 +731,14 @@ INSERT INTO t2 VALUES > UPDATE t1,t2 SET t1.i1 = -39 WHERE t2.d1 <> t1.i1 AND t2.d1 = t1.d2; > ERROR 22007: Incorrect datetime value: '19' for column `test`.`t1`.`i1` at row 1 > DROP TABLE t1,t2; > +# > +# MDEV-23500: Assertion `!is_set() || (m_status == DA_OK_BULK && > +# is_bulk_op())' failed in Diagnostics_area::set_ok_status upon > +# multi-table update > +# > +CREATE TABLE t (f INT NOT NULL); > +INSERT INTO t VALUES (0),(0); > +UPDATE t AS t1 LEFT JOIN t AS t2 ON t1.f <=> t2.f SET t2.f = NULL; > +ERROR 01000: Data truncated for column 'f' at row 4 with a single-table update I get MariaDB [test]> update t set f=NULL; ERROR 1048 (23000): Column 'f' cannot be null should multi-table update produce the same error? > +DROP TABLE t; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9c2570e6c42: MDEV-22422: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rucha! On Oct 21, Rucha Deodhar wrote: > revision-id: 9c2570e6c42 (mariadb-10.1.43-279-g9c2570e6c42) > parent(s): 5768f57d243 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-09-22 17:18:37 +0530 > message: > > MDEV-22422: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status > > Analysis: The error is not returned when the statement can't be used. > And so we still go on to search for keywords. > Fix: Return the error state. > > diff --git a/mysql-test/t/errors.test b/mysql-test/t/errors.test > index 55461002fd4..e2b10559299 100644 > --- a/mysql-test/t/errors.test > +++ b/mysql-test/t/errors.test > @@ -213,3 +213,22 @@ select * from seq_1_to_1000; > --enable_result_log > # We may not be able to execute any more queries with this connection > # because of too little memory# > + > +--echo # > +--echo # 10.1 Test > +--echo # > +--echo # MDEV-22422: Assertion `! is_set()' failed in > +--echo # Diagnostics_area::set_eof_status > +--echo # > + > +SET @saved_max_session_mem_used = @@max_session_mem_used; > + > +SET max_session_mem_used = 50000; > +--error ER_OPTION_PREVENTS_STATEMENT > +help 'it is going to crash'; > + > +SET @@max_session_mem_used=@saved_max_session_mem_used; > + > +--echo # > +--echo # End of 10.1 Test > +--echo # > diff --git a/sql/sql_help.cc b/sql/sql_help.cc > index cd9da566e51..654a3915b29 100644 > --- a/sql/sql_help.cc > +++ b/sql/sql_help.cc > @@ -736,6 +736,9 @@ static bool mysqld_help_internal(THD *thd, const char *mask) > &name, &description, &example); > delete select; > > + if (thd->is_error()) > + goto error; does the error happen during the first count_topics= search_topics() ? There are lots of places below that look kind of fragile too. May be they're safe. Or may be by increasing max_session_mem_used somewhat you can get mysqld_help_internal to crash below again and again? Could you look into that please? > if (count_topics == 0) > { > int UNINIT_VAR(key_id); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6729bf5c031: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Anel! On Oct 21, Anel Husakovic wrote: > revision-id: 6729bf5c031 (mariadb-10.1.43-112-g6729bf5c031) > parent(s): ad4b70562bb > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-04-22 20:48:55 +0200 > message: > > MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > > diff --git a/mysql-test/t/grant5.test b/mysql-test/t/grant5.test > index 649bba7d1ca..5eefd9b9185 100644 > --- a/mysql-test/t/grant5.test > +++ b/mysql-test/t/grant5.test > @@ -33,3 +33,16 @@ REVOKE EXECUTE ON PROCEDURE sp FROM u; > --error ER_TABLE_NOT_LOCKED > REVOKE PROCESS ON *.* FROM u; > DROP TABLE t1; > + > +# > +# MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > +# > +CREATE ROLE test_role; > +CREATE USER test_user; > +GRANT test_role TO test_user; > +SET DEFAULT ROLE test_role FOR test_user; > +SHOW GRANTS FOR test_user; > +SET DEFAULT ROLE NONE for test_user; > +SHOW GRANTS FOR test_user; add a test for SHOW GRANTS without FOR username > +DROP USER test_user; > +DROP ROLE test_role; > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index b6a6f806e50..90dd552ec43 100644 > --- a/sql/sql_acl.cc > @@ -7891,6 +7893,10 @@ bool mysql_show_grants(THD *thd, LEX_USER *lex_user) > if (show_role_grants(thd, username, hostname, acl_user, buff, sizeof(buff))) > goto end; > > + /* Show default role to acl_user */ > + if (show_default_role(thd, username, hostname, acl_user, buff, sizeof(buff))) > + goto end; Please, print SET DEFAULT ROLE after all grants for this user. > + > /* Add first global access grants */ > if (show_global_privileges(thd, acl_user, FALSE, buff, sizeof(buff))) > goto end; > @@ -7963,6 +7969,42 @@ static ROLE_GRANT_PAIR *find_role_grant_pair(const LEX_STRING *u, > my_hash_search(&acl_roles_mappings, (uchar*)pair_key.ptr(), key_length); > } > > +static bool show_default_role(THD *thd, const char *username, > + const char *hostname, ACL_USER_BASE *acl_entry, why did you cast acl_entry (which you know is ACL_USER) to the base class ACL_USER_BASE? why did you pass the username if you don't use it? > + char *buff, size_t buffsize) > +{ > + Protocol *protocol= thd->protocol; > + LEX_STRING host= {const_cast<char*>(hostname), strlen(hostname)}; This doesn't make much sense. You only create this LEX_STRING to use it for String::append(LEX_STRING *). But you can directly use String::append(const char*) and you won't need any LEX_STRING here. Furthermore, you can print acl_entry->host.hostname just as you print acl_entry->user, so you don't need the above hostname argument at all. > + String def_role(buff,sizeof(buff),system_charset_info); > + def_role.length(0); you don't use def_role anywhere. > + LEX_STRING def_rolename= ((ACL_USER *)acl_entry)->default_rolename; > + if (def_rolename.length) > + { > + String def_str(buff,sizeof(buff),system_charset_info); > + def_str.length(0); > + def_str.append("SET DEFAULT ROLE "); prefer to use STRING_WITH_LEN() for string literals. E.g. here it should be def_str.append(STRING_WITH_LEN("SET DEFAULT ROLE ")); > + def_str.append(def_rolename.str, def_rolename.length, > + system_charset_info); here you can do def_str.append(&def_rolename); > + def_str.append(" FOR '"); > + def_str.append(acl_entry->user.str, acl_entry->user.length, > + system_charset_info); > + if (!(acl_entry->flags & IS_ROLE)) better put an assert here not an if > + { > + def_str.append(STRING_WITH_LEN("'@'")); > + def_str.append(&host); > + } > + def_str.append('\''); > + protocol->prepare_for_resend(); > + protocol->store(def_str.ptr(),def_str.length(),def_str.charset()); > + if (protocol->write()) > + { > + return TRUE; > + } > + } > + return FALSE; > +} > + > static bool show_role_grants(THD *thd, const char *username, > const char *hostname, ACL_USER_BASE *acl_entry, > char *buff, size_t buffsize) > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 8c34a242aa2: MDEV-20787: Script dgcov.pl does not work
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Anel! On Oct 21, Anel Husakovic wrote: > revision-id: 8c34a242aa2 (mariadb-10.2.31-478-g8c34a242aa2) > parent(s): 01ffccd6a4e > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-06 00:00:58 +0200 > message: > > MDEV-20787: Script dgcov.pl does not work > > - This patch is done after doing test on 00f9df29224ada9cdbb796c99f4 > where difference was found between versions of gcov. don't refer to commits that are not pushed and are not going to be pushed. Because if you push this one you'll have a commit forever in the history that refers to an non-existing commit. It isn't helpful. > - This patch doesn't take gcov versions in account since is extracting > filename of a `.gcov` file, which is generated with `gcov -i > <source-file>.gcda` during `dgcov.pl --generate`. > - There is a check if file exists. > Reason for that is because some generated `gcov` files are outliers from above > rule (that a file generated in intermediate format(gcov) has the same source name). > Example 1: `aestables.cpp.gcda` is not generating `aestables.cpp.gcov`, > but only the files used in #include `modes.hpp.gcov runtime.hpp.gcov` > Example 2: `my_sha256.cc.gcda` is generating `my_sha.ic.gcov` > - Make sure to test `MTR_BINDIR` which gets set during out-of-source > build and generate error when running `./mysql-test/mtr --gcov alias` What is a problem with out of source builds? If they don't work, perhaps it would be better to fix dgcov to work with out of source builds? > - During the testing: > 10.1 results are stored in > - `<basedir>//mysql-test-gcov.msg` and errors in > - `<basedir>//mysql-test-gcov.err` > 10.2 results should be stored in > `./mysql-test/var/last_changes.dgcov`, but only empty results are > obtained (tests `1st` and `alias`). I don't understand this ^^^ paragraph. And also because your patch is 10.2+ so who cares what it does on 10.1? > diff --git a/mysql-test/dgcov.pl b/mysql-test/dgcov.pl > index fbc5540e697..f2bbdad4d20 100755 > --- a/mysql-test/dgcov.pl > +++ b/mysql-test/dgcov.pl > @@ -161,8 +161,14 @@ sub gcov_one_file { > system($cmd)==0 or die "system($cmd): $? $!"; > } > > - # now, read the generated file > - open FH, '<', "$_.gcov" or die "open(<$_.gcov): $!"; > + (my $filename = $_)=~ s/\.[^.]+$//; # remove extension In your previous patch you either removed the extension or not based on gcov version. Why did you abandon this logic? > + my $gcov_file_path= $File::Find::dir."/$filename.gcov"; File::Find documentation seems to be saying that $File::Find::dir ends with a slash (/) so you don't need to prefix $filename.gcov with it. > + if (! -f $gcov_file_path) > + { > + return; > + } may be return unless -f $gcov_file_path it's perl after all :) > + open FH, '<', "$gcov_file_path" or die "open(<$gcov_file_path): $!"; you don't need double quotes around "$gcov_file_path" - they don't serve any purpose. > + > my $fname; > while (<FH>) { > chomp; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f83c1615d97: MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rucha! On Oct 21, Rucha Deodhar wrote: > revision-id: f83c1615d97 (mariadb-10.1.43-240-gf83c1615d97) > parent(s): 7e9ffc69ecd > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-08-05 00:33:03 +0530 > message: > > MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status > upon HANDLER READ > > Analysis: The error state is not stored while checking condition and key name. > Fix: Return true while checking condition and key name if error is reported > because geometry object can't be created from the data in the index value for > HANDLER READ. > > diff --git a/mysql-test/suite/handler/aria.test b/mysql-test/suite/handler/aria.test > index 1913d2b791c..50ef43ce5b2 100644 > --- a/mysql-test/suite/handler/aria.test > +++ b/mysql-test/suite/handler/aria.test > @@ -80,3 +80,23 @@ HANDLER t1 CLOSE; > DROP TABLE t1; > > --echo End of 5.1 tests > + > +--echo # > +--echo # 10.1 Test > +--echo # > +--echo # MDEV-20207: Assertion `! is_set()' failed in > +--echo # Diagnostics_area::set_eof_status upon HANDLER READ > +--echo # > + > +DROP TABLE IF EXISTS t1; there's no need to start the test from DROP TABLE IF EXISTS t1; historically people customarily did that. before mtr had after-test checks. but now all tests are supposed to clean up after itself, you do not need to worry about tests leaving tables around. > + > +CREATE TABLE t1 (a POINT, KEY(a)); > +HANDLER t1 OPEN h; > + > +--error ER_CANT_CREATE_GEOMETRY_OBJECT > +HANDLER h READ a = (0); I don't think it should be an error. SELECT * FROM t1 WHERE a=(0) -- this does not fail, it just doesn't find anything. I think HANDLER should do the same. > + > +HANDLER h CLOSE; > +DROP TABLE t1; note that all handler test files source handler.inc. that is, if you want a test to run for all engines, you do not need to add it to all individual test files, add it to handler.inc Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e3f45b2f9ea: MDEV-10267 Add ngram fulltext parser plugin
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rinat! On Oct 21, Rinat Ibragimov wrote: > > > + > > > + // Avoid creating partial n-grams by stopping at word boundaries. > > > + // Underscore is treated as a word character too, since identifiers in > > > + // programming languages often contain them. > > > + if (!is_alnum(char_type) && !is_underscore(cs, next, end)) { > > > + start= next + char_len; > > > + next= start; > > > + n_chars= 0; > > > + continue; > > > > I don't think it's a good idea. On the opposite, I would rather create > > ngrams over word boundaries, so that, say, I'd split "n-gram plugin" to > > > > "n-g", "-gr", "gra", "ram", "am ", "m p", " pl", "plu", "lug", "ugi", "gin". > > I see how this can increase index size: more different strings are > included. Implementation will become a tiny bit simpler. But can it > make searching better somehow? Yes, the idea is that it'll rank "One can use n-gram plugin in MariaDB fulltext search" higher than "Plugins are useful for extending functionality of a program" because some ngrams encode that "n-gram" and "plugin" words must be together as a phrase. > I can't decide. From my point of view, the current approach is fine. > Please pick a variant, and I'll try to implement that. No, I cannot guess which approach will produce more relevant searches. Implement something and then we test what works better > > > + } > > > + > > > + next += char_len; > > > + n_chars++; > > > + > > > + if (n_chars == ngram_token_size) { > > > + bool_info->position= static_cast<uint>(start - doc); > > > > nope, we don't have MYSQL_FTPARSER_FULL_BOOLEAN_INFO::position and this > > plugin is definitely not the reason to break the API and add it. > > There is a patch in the patchset that adds the "position" field. Without > that field, > it's possible to trigger an assertion in InnoDB's FTS code by executing: > > INSTALL PLUGIN simple_parser SONAME 'mypluglib'; > CREATE TABLE articles( > a TEXT DEFAULT NULL, > b TEXT DEFAULT NULL, > FULLTEXT (a, b) WITH PARSER simple_parser > ) ENGINE=InnoDB; > INSERT INTO articles VALUES ('111', '1234 1234 1234'); > DROP TABLE articles; > UNINSTALL PLUGIN simple_parser; > > Same assertion may be triggered by any fts plugin. Are you sure you > want that patch to be removed? I don't think InnoDB assert is a reason to extend the fulltext plugin API. What about this fix instead: diff --git a/storage/innobase/fts/fts0fts.cc b/storage/innobase/fts/fts0fts.cc --- a/storage/innobase/fts/fts0fts.cc +++ b/storage/innobase/fts/fts0fts.cc @@ -4669,7 +4669,7 @@ fts_tokenize_add_word_for_parser( ut_ad(boolean_info->position >= 0); position = boolean_info->position + fts_param->add_pos; */ - position = fts_param->add_pos; + position = fts_param->add_pos++; fts_add_token(result_doc, str, position); this fixed the crash for me. > > > + n_chars= ngram_token_size - 1; > > > + is_first= false; > > > + } > > > + } > > > + > > > + // If nothing was emitted yet, emit the remainder. > > > > what reminder, what is it for? > > It's "remainder", with "a" before "i". Remainder of the processed > string. That way strings that are too short for having even a single > n-gram, will still generate some index entries and will be > discoverable by exact queries. Perhaps the comment should say it? That if the string too short, it'll be emitted here? > > > + case MYSQL_FTPARSER_FULL_BOOLEAN_INFO: > > > + // Reusing InnoDB's boolean mode parser to handle all special characters. > > > + while (fts_get_word(cs, &start, end, &word, &bool_info)) { > > > > nope. InnoDB might not even be compiled it. Use param->mysql_parse() > > Copied fts_get_word() implementation into the plugin code. > > As far as I understand, param->mysql_parse() cannot be used here as it > splits a string into words like default fulltext search does. This > plugin however must generate n-grams. Of course, it can. Note that fts_get_word() doesn't generate n-grams either, it gets the whole word and the n-gram plugin later splits it into n-grams. Similarly param->mysql_parse() will extract words for you and you'll split them into n-grams. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Please assign PRs that modify debian/ contents for Otto to review
by Otto Kekäläinen 20 Oct '20

20 Oct '20

Hello! While working on putting MariaDB 10.5.5 into Debian I noticed that there were a couple of commits that landed more or less on the same day as the release was made and "broke" it just before release. Now there are surprises in a stable release I need to fix and the situation puts extra strain on me. It is good that many people participate in maintaining the contents of debian/ in the MariaDB Server, because I don't have time to do everything. I would however appreciate that when changes are made, I would be given a chance to review them. I will review them post-release anyway and it would save a lot of trouble, if we can fix stuff during the development cycle and not having to patch a post-release in order to be able to put it forward to Debian/Ubuntu. So, please make PRs of any debian/ changes you make, and feel free to add me as the reviewer (or one of the reviewers). This policy can also be automated using a .github/CODEOWNERS file, so I made a PR about it at https://github.com/MariaDB/server/pull/1650 Thanks! PS. I will deliver a MariaDB 10.5 and MariaDB in Debian status talk at https://debconf20.debconf.org/talks/98-whats-new-in-mariadb-server-105-and-… in case anybody is interested to listen in. -- Follow me at @ottokekalainen

2 3

Re: [Maria-developers] db1fbafbead: MDEV-23935: Fix warnings generated during compilation of plugin/auth_pam/testing/pam_mariadb_mtr.c on MacOS
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! ok to push On Oct 19, Dmitry Shulga wrote: > revision-id: db1fbafbead (mariadb-10.4.11-420-gdb1fbafbead) > parent(s): c2ac0ce1f02 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-11 21:51:45 +0700 > message: > > MDEV-23935: Fix warnings generated during compilation of plugin/auth_pam/testing/pam_mariadb_mtr.c on MacOS > > During build server 10.4 on MacOS warnings like the following one > are generated on compiling the file plugin/auth_pam/testing/pam_mariadb_mtr.c > > server-10.4/plugin/auth_pam/testing/pam_mariadb_mtr.c:25:22: error: > initializing 'char *' with an expression of type 'const char [23]' > discards qualifiers [-Werror,-Wincompatible-pointer-types-discards-qualifiers] > { PAM_TEXT_INFO, "Challenge input first." } > > The reason of the warnings is that the data member pam_message::msg is declared > as 'char *' on MacOS but initializer of the data member is 'const char *'. > > To eliminate warnings the compiler option > -Wno-incompatible-pointer-types-discards-qualifiers > has been added to the compiler flags used for compiling the file > pam_mariadb_mtr.c. > > --- > plugin/auth_pam/testing/CMakeLists.txt | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/plugin/auth_pam/testing/CMakeLists.txt b/plugin/auth_pam/testing/CMakeLists.txt > index 889cb524402..c8d2e3cbb97 100644 > --- a/plugin/auth_pam/testing/CMakeLists.txt > +++ b/plugin/auth_pam/testing/CMakeLists.txt > @@ -4,6 +4,12 @@ ADD_LIBRARY(pam_mariadb_mtr MODULE pam_mariadb_mtr.c) > SET_TARGET_PROPERTIES (pam_mariadb_mtr PROPERTIES PREFIX "") > TARGET_LINK_LIBRARIES(pam_mariadb_mtr pam) > > +IF(APPLE) > + SET_SOURCE_FILES_PROPERTIES( > + pam_mariadb_mtr.c > + PROPERTY COMPILE_FLAGS "-Wno-incompatible-pointer-types-discards-qualifiers") > +ENDIF() > + > SET(dest DESTINATION "${INSTALL_MYSQLTESTDIR}/suite/plugins/pam" COMPONENT Test) > INSTALL(TARGETS pam_mariadb_mtr ${dest}) > INSTALL(FILES mariadb_mtr.conf RENAME mariadb_mtr ${dest}) > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 727d97feb99: MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! plugin code should stay in the plugin directory. please move the check to plugin/auth_pam/CMakeLists.txt Also, what about an alternative fix: #ifdef HAVE_POSIX_GETGROUPLIST typedef gid_t my_gid_t; #else typedef int my_gid_t; #end would that help but with less ifdefs? On Oct 19, Dmitry Shulga wrote: > revision-id: 727d97feb99 (mariadb-10.2.31-489-g727d97feb99) > parent(s): 8984d779103 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-09 20:35:48 +0700 > message: > > MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS > > Compiler warnings listed below are generated during server build on MacOS: > > 88%] Building C object plugin/auth_pam/CMakeFiles/pam_user_map.dir/mapper/pam_user_map.c.o > /Users/shulga/projects/mariadb/server-10.2/plugin/auth_pam/mapper/pam_user_map.c:87:41: error: passing > 'gid_t *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types > with different sign [-Werror,-Wpointer-sign] > if (getgrouplist(user, user_group_id, loc_groups, &ng) < 0) > ^~~~~~~~~~ > /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/unistd.h:650:43: note: > passing argument to parameter here > int getgrouplist(const char *, int, int *, int *); > ^ > /Users/shulga/projects/mariadb/server-10.2/plugin/auth_pam/mapper/pam_user_map.c:95:46: error: passing > 'gid_t *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types > with different sign [-Werror,-Wpointer-sign] > (void) getgrouplist(user, user_group_id, loc_groups, &ng); > ^~~~~~~~~~ > /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/includ > e/unistd.h:650:43: note: > passing argument to parameter here > int getgrouplist(const char *, int, int *, int *); > ^ > 2 errors generated. > > In case MariaDB serer is build with -DCMAKE_BUILD_TYPE=Debug it results in > uild error. > > The reason of compiler warnings is that declaration of the Posix C API function > getgrouplist() on MacOS differs from declration of getgrouplist() proposed > by Posix. > > To suppress this compiler warning cmake configure was adapted to detect what > kind of getgrouplist() function is declared on the building platform and > set the macros HAVE_POSIX_GETGROUPLIST in case the building platform suppors > Posix compatible interface of the getgrouplist() function. Depending on > whether this macros is set the compatible type of arguments is used to pass > parameter values to the function. > > --- > config.h.cmake | 1 + > configure.cmake | 22 ++++++++++++++++++++++ > plugin/auth_pam/mapper/pam_user_map.c | 13 ++++++++++++- > 3 files changed, 35 insertions(+), 1 deletion(-) > > diff --git a/config.h.cmake b/config.h.cmake > index 0e19dd44694..07e4647f0bf 100644 > --- a/config.h.cmake > +++ b/config.h.cmake > @@ -188,6 +188,7 @@ > #cmakedefine HAVE_POSIX_FALLOCATE 1 > #cmakedefine HAVE_LINUX_FALLOC_H 1 > #cmakedefine HAVE_FALLOC_PUNCH_HOLE_AND_KEEP_SIZE 1 > +#cmakedefine HAVE_POSIX_GETGROUPLIST 1 > #cmakedefine HAVE_PREAD 1 > #cmakedefine HAVE_PAUSE_INSTRUCTION 1 > #cmakedefine HAVE_FAKE_PAUSE_INSTRUCTION 1 > diff --git a/configure.cmake b/configure.cmake > index 58e1111bfc7..c6db3ecb912 100644 > --- a/configure.cmake > +++ b/configure.cmake > @@ -1031,3 +1031,25 @@ IF(NOT MSVC) > HAVE_FALLOC_PUNCH_HOLE_AND_KEEP_SIZE > ) > ENDIF() > + > +# Check whether getgrouplist uses git_t for second and third arguments. > +SET(CMAKE_REQUIRED_FLAGS -Werror) > +CHECK_C_SOURCE_COMPILES( > +" > +#ifdef HAVE_GRP_H > +#include <grp.h> > +#endif > +#ifdef HAVE_UNISTD_H > +#include <unistd.h> > +#endif > +int main() { > + char *arg_1; > + gid_t arg_2, arg_3; > + int arg_4; > + (void)getgrouplist(arg_1,arg_2,&arg_3,arg_4); > + return 0; > +} > +" > +HAVE_POSIX_GETGROUPLIST > +) > +SET(CMAKE_REQUIRED_FLAGS) > diff --git a/plugin/auth_pam/mapper/pam_user_map.c b/plugin/auth_pam/mapper/pam_user_map.c > index 9d7ed53f8b1..b0d2affac5c 100644 > --- a/plugin/auth_pam/mapper/pam_user_map.c > +++ b/plugin/auth_pam/mapper/pam_user_map.c > @@ -31,6 +31,8 @@ These comments are written to the syslog as 'authpriv.debug' > and usually end up in /var/log/secure file. > */ > > +#include <my_config.h> > + > #include <stdlib.h> > #include <stdio.h> > #include <ctype.h> > @@ -72,8 +74,13 @@ static const char debug_keyword[]= "debug"; > > static int populate_user_groups(const char *user, gid_t **groups) > { > +#ifdef HAVE_POSIX_GETGROUPLIST > gid_t user_group_id; > gid_t *loc_groups= *groups; > +#else > + int user_group_id; > + int *loc_groups= (int*)*groups; > +#endif > int ng; > > { > @@ -88,12 +95,16 @@ static int populate_user_groups(const char *user, gid_t **groups) > { > /* The rare case when the user is present in more than */ > /* GROUP_BUFFER_SIZE groups. */ > +#ifdef HAVE_POSIX_GETGROUPLIST > loc_groups= (gid_t *) malloc(ng * sizeof (gid_t)); > +#else > + loc_groups= (int *) malloc(ng * sizeof (gid_t)); > +#endif > if (!loc_groups) > return 0; > > (void) getgrouplist(user, user_group_id, loc_groups, &ng); > - *groups= loc_groups; > + *groups= (gid_t*)loc_groups; > } > > return ng; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5e71a3e0df7: MDEV-23925: Fix warnings generated during compilation of mysys_ssl/openssl.c on MacOS
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! ok to push (again, comments below) On Oct 19, Dmitry Shulga wrote: > revision-id: 5e71a3e0df7 (mariadb-10.2.31-489-g5e71a3e0df7) > parent(s): 8984d779103 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-09 20:55:14 +0700 > message: > > MDEV-23925: Fix warnings generated during compilation of mysys_ssl/openssl.c on MacOS > > Compiler warnings listed below are generated during server build on MacOS: > In file included from server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > In file included from /usr/local/include/openssl/evp.h:16: > In file included from /usr/local/include/openssl/bio.h:20: > /usr/local/include/openssl/crypto.h:206:10: warning: 'CRYPTO_cleanup_all_ex_data' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:46:9: note: previous definition is here > ^ > In file included from /mariadb/server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > /usr/local/include/openssl/evp.h:501:10: warning: 'EVP_MD_CTX_init' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:33:9: note: previous definition is here > ^ > In file included from /mariadb/server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > /usr/local/include/openssl/evp.h:627:11: warning: 'EVP_CIPHER_CTX_init' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:35:9: note: previous definition is here > ^ > In file included from /mariadb/server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > /usr/local/include/openssl/evp.h:866:11: warning: 'EVP_cleanup' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:44:9: note: previous definition is here > ^ > 4 warnings generated. generally these long commit comments are not any more clear than if you'd only listed one compilation failure. > > In case MariaDB serer is build with -DCMAKE_BUILD_TYPE=Debug it results in > build error. > > The reason of compiler warnings is that header file <ssl_compat.h> > included before the openssl system header files. File ssl_compat.h > contains some macros with the name as SSL API functions declared > in the openssl system header files. It resulted in duplicate > symbols thats produces compiler warnings. > > To fix the issue the header file ssl_compat.h should be included > after a line where openssl system header is included. > > --- > mysys_ssl/openssl.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/mysys_ssl/openssl.c b/mysys_ssl/openssl.c > index f3dc8f4277c..a11db113b14 100644 > --- a/mysys_ssl/openssl.c > +++ b/mysys_ssl/openssl.c > @@ -15,7 +15,6 @@ > Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > > #include <my_global.h> > -#include <ssl_compat.h> > > /* > The check is only done for OpenSSL 1.1.x. > @@ -25,13 +24,14 @@ > */ > > #ifndef HAVE_OPENSSL11 > +#include <ssl_compat.h> > int check_openssl_compatibility() > { > return 0; > } > #else > #include <openssl/evp.h> > - > +#include <ssl_compat.h> > static uint testing, alloc_size, alloc_count; > > static void *coc_malloc(size_t size, const char *f __attribute__((unused)), > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b92597c220f: MDEV-23564: CMAKE failing due to deprecated Apple GSS method
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! ok to push. but see comments below On Oct 19, Dmitry Shulga wrote: > revision-id: b92597c220f (mariadb-10.2.31-479-gb92597c220f) > parent(s): 577c61e8be0 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-08 00:48:47 +0700 > message: > > MDEV-23564: CMAKE failing due to deprecated Apple GSS method > > Some GSS-API functions like gss_import_name(), gss_release_buffer() > used in plugin/auth_gssapi and libmariadb/plugins/auth are marked > as deprecated in MacOS starting from version 10.14+. It results in > extra warnings output on server building. > > To eliminate extra warnings the flag '-Wno-deprecated-declarations' > has been added to compiler invocation string for those source > files that invoke deprecated GSS-API functions. > > diff --git a/libmariadb b/libmariadb > index a746c3af449..fd0dc672309 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit a746c3af449a8754e78ad7971e59e79af7957cdb > +Subproject commit fd0dc6723095f066648a7c3b2a48523b3002e392 be careful when pushing this commit. you'd need something like 1. rebase your libmariadb fix on top of 3.1, push into 3.1 2. rebase your server fix on top of 10.2 2. update submodule reference in the server repo 3. amend your commit fix and push > diff --git a/plugin/auth_gssapi/CMakeLists.txt b/plugin/auth_gssapi/CMakeLists.txt > index bca4f5af3a1..7a3c32a00b1 100644 > --- a/plugin/auth_gssapi/CMakeLists.txt > +++ b/plugin/auth_gssapi/CMakeLists.txt > @@ -18,6 +18,13 @@ ELSE() > SET(GSSAPI_SERVER gssapi_server.cc) > SET(GSSAPI_ERRMSG gssapi_errmsg.cc) > > + IF(APPLE) > + SET_SOURCE_FILES_PROPERTIES( > + ${GSSAPI_CLIENT} ${GSSAPI_SERVER} ${GSSAPI_ERRMSG} > + PROPERTY COMPILE_FLAGS "-Wno-deprecated-declarations") > +# SET_SOURCE_FILES_PROPERTIES(${GSSAPI_SERVER} PROPERTY COMPILE_FLAGS "-Wno-deprecated-declarations") > +# SET_SOURCE_FILES_PROPERTIES(${GSSAPI_ERRMSG} PROPERTY COMPILE_FLAGS "-Wno-deprecated-declarations") you don't need these commented lines anymore, do you? Is it possible to use a compiler on APPLE that doesn't understand -Wno-deprecated-declarations? If not - your fix it ok. If it is - better use MY_CHECK_AND_SET_COMPILER_FLAG > + ENDIF() > SET(CMAKE_REQUIRED_INCLUDES ${GSSAPI_INCS}) > SET(CMAKE_REQUIRED_LIBRARIES ${GSSAPI_LIBS}) > INCLUDE(CheckCXXSymbolExists) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2d73406dad8: MDEV-16481: set global system_versioning_asof=sf() crashes in specific case
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Nikita! ok to push (better late than never :) On Oct 19, Nikita Malyavin wrote: > revision-id: 2d73406dad8 (mariadb-10.3.16-39-g2d73406dad8) > parent(s): 609f358a9bd > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Sergei Golubchik <serg(a)mariadb.com> > timestamp: 2020-10-19 17:37:04 +0200 > message: > > MDEV-16481: set global system_versioning_asof=sf() crashes in specific case > > * sys_vars.h: add `MYSQL_TIME` field to `set_var::save_result` > * sys_vars.ic: get rid of calling `var->value->get_date()` from `Sys_var_vers_asof::update()` > * versioning.sysvars: add test; remove double warning > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e3f45b2f9ea: MDEV-10267 Add ngram fulltext parser plugin
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Rinat! On Oct 19, Rinat Ibragimov wrote: > revision-id: e3f45b2f9ea (mariadb-10.2.31-368-ge3f45b2f9ea) > parent(s): 44c48cefdbe > author: Rinat Ibragimov <ri(a)tempesta-tech.com> > committer: Rinat Ibragimov <ri(a)tempesta-tech.com> > timestamp: 2020-08-25 01:59:06 +0300 > message: > > MDEV-10267 Add ngram fulltext parser plugin > > Ngram is a fulltext parser plugin that splits words into overlapping > segments of fixed lengths. For example, if 3-grams are in use, string > "abcdef" is split into "abc", "bcd", "cde", "def", thus allowing an > efficient substring search for CJK texts where splitting by words in not > feasible. Only word characters are going into n-grams. Spaces and > punctuation characters are treated as separators. > > diff --git a/plugin/fulltext/CMakeLists.txt b/plugin/fulltext/CMakeLists.txt > index 3c37666cb3e..9704733a9aa 100644 > --- a/plugin/fulltext/CMakeLists.txt > +++ b/plugin/fulltext/CMakeLists.txt > @@ -18,3 +18,7 @@ set(CMAKE_CXX_FLAGS_RELWITHDEBINFO "${CMAKE_CXX_FLAGS_RELWITHDEBINFO} -DNDEBUG") > > MYSQL_ADD_PLUGIN(ftexample plugin_example.c > MODULE_ONLY MODULE_OUTPUT_NAME "mypluglib" COMPONENT Test) > + > +# n-gram parser > +MYSQL_ADD_PLUGIN(ftngram ngram_parser/plugin_ngram.cc > + MODULE_ONLY MODULE_OUTPUT_NAME "fulltext_ngram") why to put the plugin under plugin/fulltext/ngram_parser/ ? it's only one file, just put it in plugin/fulltext/ > diff --git a/plugin/fulltext/mysql-test/ngram/disabled.def b/plugin/fulltext/mysql-test/ngram/disabled.def > new file mode 100644 > index 00000000000..0ca9dd33ef7 > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/disabled.def > @@ -0,0 +1 @@ > +ngram_gb18030 : GB18030 character set is not supported yet. I don't think there's any need to add tests for unsupported charsets. The plugin is trivial, no need to test it for every possible charset that we have and don't have. > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_basic.inc b/plugin/fulltext/mysql-test/ngram/ngram_basic.inc > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_basic.inc > @@ -0,0 +1,25 @@ > +--echo > +--eval CREATE TABLE t (lang TEXT, str TEXT, FULLTEXT(str) WITH PARSER ngram) ENGINE=$engine CHARSET $charset > + > +# Some character are not possible to represent in some encodings. Ignore them. > +--disable_warnings > +--disable_query_log > +# http://www.gerolf.org/doc/polyglot/polyglot.htm. > +INSERT IGNORE INTO t VALUES > + ('Chinese', '他们为什么不说中文？'), > + ('English', 'Why can''t they just speak English?'), > + ('Japanese', 'なぜ、みんな日本語を話してくれないのか？'), > + ('Korean', '세계의 모든 사람들이 한국어를 이해한다면 얼마나 좋을까?'); > +--enable_query_log > +--enable_warnings > + > +SELECT * FROM t ORDER BY lang; > + > +SELECT * FROM t WHERE MATCH(str) AGAINST('之一' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('数据' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('zzz' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('gli' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('ない' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('람들' IN BOOLEAN MODE) ORDER BY lang; > + > +DROP TABLE t; > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_basic.test b/plugin/fulltext/mysql-test/ngram/ngram_basic.test > new file mode 100644 > index 00000000000..eaa8829c3ee > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_basic.test > @@ -0,0 +1,25 @@ > +# Ngram fulltext parser basic functionality. > + > +SET NAMES utf8mb4; > +SELECT @@ngram_token_size; > + > +let $engine_index= 3; > +while ($engine_index) { > + if ($engine_index == 3) { let $engine= InnoDB; } > + if ($engine_index == 2) { let $engine= MyISAM; } > + if ($engine_index == 1) { let $engine= Aria; } > + > + let $charset_index= 6; > + while ($charset_index) { > + if ($charset_index == 6) { let $charset= utf8mb4; } > + if ($charset_index == 5) { let $charset= big5; } > + if ($charset_index == 4) { let $charset= gbk; } > + if ($charset_index == 3) { let $charset= gb2312; } > + if ($charset_index == 2) { let $charset= sjis; } > + if ($charset_index == 1) { let $charset= ujis; } > + > + source ngram_basic.inc; > + dec $charset_index; > + } eh. unroll the loop please, write let $engine= InnoDB; let $charset= utf8mb4; source ngram_basic.inc; let $charset= big5; source ngram_basic.inc; ... Or, better: CREATE TABLE t (lang TEXT, str TEXT, FULLTEXT(str) WITH PARSER ngram) ENGINE=InnoDB CHARSET utf8mb4; source ngram_basic.inc; CREATE TABLE t (lang TEXT, str TEXT, FULLTEXT(str) WITH PARSER ngram) ENGINE=InnoDB CHARSET big5; source ngram_basic.inc; ... > + dec $engine_index; > +} > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_missing_plugin.result b/plugin/fulltext/mysql-test/ngram/ngram_missing_plugin.result > new file mode 100644 > index 00000000000..aa17b1d6053 > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_missing_plugin.result > @@ -0,0 +1,23 @@ there are tests for the missing fulltest parser plugin. they don't depend on the individial plugin (that is, btw, *not installed*). No need to repeat them for every plugin, remove this test. > +SET NAMES utf8mb4; > +CREATE TABLE t (str TEXT, FULLTEXT(str) WITH PARSER ngram) > +ENGINE=InnoDB; > +INSERT INTO t VALUES ('Hello World'), ('abcdef ghijklm'); > +# Restart with ngram plugin disabled. > +SET NAMES utf8mb4; > +SELECT * FROM t WHERE MATCH(str) AGAINST('bcd') ORDER BY str; > +ERROR HY000: Plugin 'ngram' is not loaded > +SELECT * FROM t ORDER BY str; > +ERROR HY000: Plugin 'ngram' is not loaded > +INSERT INTO t VALUES ('something'); > +ERROR HY000: Plugin 'ngram' is not loaded > +# Restart with ngram plugin enabled again. > +SET NAMES utf8mb4; > +SELECT * FROM t WHERE MATCH(str) AGAINST('bcd') ORDER BY str; > +str > +abcdef ghijklm > +SELECT * FROM t ORDER BY str; > +str > +abcdef ghijklm > +Hello World > +INSERT INTO t VALUES ('something'); > +DROP TABLE t; > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_token_size.result b/plugin/fulltext/mysql-test/ngram/ngram_token_size.result > new file mode 100644 > index 00000000000..7043e147d5b > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_token_size.result > @@ -0,0 +1,109 @@ > +# Check 'ngram_token_size' variable. > +# > +# It's a global variable. > +SELECT @@global.ngram_token_size; > +@@global.ngram_token_size > +2 > +SHOW GLOBAL VARIABLES LIKE 'ngram_token_size'; > +Variable_name Value > +ngram_token_size 2 > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 2 > +# > +# Not a session variable. > +SELECT @@session.ngram_token_size; > +ERROR HY000: Variable 'ngram_token_size' is a GLOBAL variable > +SHOW SESSION VARIABLES LIKE 'ngram_token_size'; > +Variable_name Value > +ngram_token_size 2 > +SELECT * FROM information_schema.session_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 2 > +# > +# 0 is outside allowed range [1, 10], so it's changed to 1. > +SET GLOBAL ngram_token_size= 0; > +Warnings: > +Warning 1292 Truncated incorrect ngram_token_size value: '0' > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 1 > +# > +# 10 is an acceptable value. > +SET GLOBAL ngram_token_size= 10; > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 10 > +# > +# 1 is an acceptable value. > +SET GLOBAL ngram_token_size= 1; > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 1 > +# > +# 11 is outside allowed range [1, 10], so it's changed to 10. > +SET GLOBAL ngram_token_size= 11; > +Warnings: > +Warning 1292 Truncated incorrect ngram_token_size value: '11' > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 10 ^^^^ all tests above aren't needed, it's enough to do query_vertical select * from information_schema.system_variables where variable_name='ngram_token_size'; all the behavior above is implemented outside of the plugin, no need to test it. > +# > +# Try 3-grams. > +SET GLOBAL ngram_token_size= 3; > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB; > +INSERT INTO t VALUES > +('Lorem ipsum dolor sit amet, consectetur adipiscing elit.'), > +('Maecenas non pellentesque odio.'), > +('Sed fermentum egestas sapien vel consequat.'); > +# Only one row contains substring 'api'. > +SELECT str FROM t WHERE MATCH(str) AGAINST('api'); > +str > +Sed fermentum egestas sapien vel consequat. > +# 'lent' is parsed into ('len', 'ent') list. Two rows contain at least one of the tokens. > +SELECT str FROM t WHERE MATCH(str) AGAINST('lent*'); > +str > +Maecenas non pellentesque odio. > +Sed fermentum egestas sapien vel consequat. > +# 'zzz' is not present anywhere, so result is empty. > +SELECT str FROM t WHERE MATCH(str) AGAINST('zzzz'); > +str > +# 'piz' cannot be found anywhere, but 'api' is in one of the rows. > +SELECT str FROM t WHERE MATCH(str) AGAINST('apiz'); > +str > +Sed fermentum egestas sapien vel consequat. > +# > +# Try switching to 2-grams while keeping the index. > +SET GLOBAL ngram_token_size= 2; > +# 'api' is now parsed into ('ap', 'pi'). Since index only contains 3-grams, nothing is found. > +SELECT str FROM t WHERE MATCH(str) AGAINST('api'); > +str > +# > +# Add some data while ngram size is set to 2. > +INSERT INTO t VALUES('1234 56789'); > +# > +# Now index contains 3-grams from the rows added before and a number of > +# 2-grams from the just added row. > +# Querying for 'api' still returns nothing, but line with '678' is found. > +SELECT str from t WHERE MATCH(str) AGAINST('api'); > +str > +SELECT str FROM t WHERE MATCH(str) AGAINST('678'); > +str > +1234 56789 > +# > +# Switching back to 3-grams make intitially added rows searchable. > +SET GLOBAL ngram_token_size= 3; > +SELECT str FROM t WHERE MATCH(str) AGAINST('api'); > +str > +Sed fermentum egestas sapien vel consequat. > +# > +# But row with '678' in it cannot be found, as it was added with 2-grams. > +SELECT str FROM t WHERE MATCH(str) AGAINST('678'); > +str > +# > +# Try matching in boolean mode. > +SELECT str FROM t WHERE MATCH(str) AGAINST('+ent -api' IN BOOLEAN MODE); > +str > +Maecenas non pellentesque odio. > +# > +DROP TABLE t; > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_ucs2.result b/plugin/fulltext/mysql-test/ngram/ngram_ucs2.result > new file mode 100644 > index 00000000000..9d82743bd24 > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_ucs2.result > @@ -0,0 +1,8 @@ > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UCS2; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UTF16; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UTF16LE; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UTF32; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index same as above. This is fulltext index limitation, it does not depend on the parser plugin. The plugin doesn't need to test for it. > diff --git a/plugin/fulltext/mysql-test/ngram/suite.opt b/plugin/fulltext/mysql-test/ngram/suite.opt > new file mode 100644 > index 00000000000..ab37cea996e > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/suite.opt > @@ -0,0 +1,4 @@ > +--plugin-load-add=$FULLTEXT_NGRAM_SO > +--loose-innodb no need to enable innodb explicitly, `source have_innodb.inc` does that. > +--loose-enable-innodb_ft_index_table > +--loose-innodb_ft_index_cache why did you enable these two plugins if you don't use them in your tests? > diff --git a/plugin/fulltext/ngram_parser/plugin_ngram.cc b/plugin/fulltext/ngram_parser/plugin_ngram.cc > new file mode 100644 > index 00000000000..80498c3299b > --- /dev/null > +++ b/plugin/fulltext/ngram_parser/plugin_ngram.cc > @@ -0,0 +1,256 @@ > +/* > + Copyright (c) 2014, 2017, Oracle and/or its affiliates. All rights reserved. > + Copyright (c) 2020, MariaDB. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License, version 2.0, > + as published by the Free Software Foundation. > + > + This program is also distributed with certain software (including > + but not limited to OpenSSL) that is licensed under separate terms, > + as designated in a particular file or component or in included license > + documentation. The authors of MySQL hereby grant you an additional > + permission to link the program and your derivative works with the > + separately licensed software that they have included with MySQL. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License, version 2.0, for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > + > +#include <stddef.h> > +#include <limits.h> > +#include <my_attribute.h> > +#include <my_global.h> > +#include "../storage/innobase/include/fts0tokenize.h" No, the plugin should be self-contained and not include anything from other plugins. At least if a plugin is as trivial as that one. > + > +#define U_LOW_LINE 0x5f // Code point for underscore '_'. > +#define is_alnum(ct) ((ct) & (_MY_U | _MY_L | _MY_NMR)) > + > +static int ngram_token_size; > + > +static MYSQL_SYSVAR_INT(token_size, ngram_token_size, > + 0, > + "Ngram full text plugin parser token size in characters", > + NULL, NULL, 2, 1, 10, 0); > + > +static int my_ci_charlen(CHARSET_INFO *cs, const uchar *str, const uchar *end) > +{ > + return my_charlen(cs, reinterpret_cast<const char *>(str), > + reinterpret_cast<const char *>(end)); > +} > + > +static int my_ci_ctype(CHARSET_INFO *cs, int *char_type, const uchar *str, > + const uchar *end) > +{ > + return cs->cset->ctype(cs, char_type, str, end); > +} > + > +static bool is_underscore(CHARSET_INFO *cs, const uchar *str, const uchar *end) > +{ > + my_wc_t wc; > + cs->cset->mb_wc(cs, &wc, str, end); > + return wc == U_LOW_LINE; > +} > + > +// Splits a string into ngrams and emits them. > +static int split_into_ngrams(MYSQL_FTPARSER_PARAM *param, const uchar *doc, > + int len, MYSQL_FTPARSER_BOOLEAN_INFO *bool_info) > +{ > + const CHARSET_INFO *cs= param->cs; > + const uchar *start= doc; > + const uchar *end= doc + len; > + const uchar *next= start; > + int n_chars= 0; > + int ret= 0; > + bool is_first= true; > + > + while (next < end) { > + int char_type; > + int char_len= my_ci_ctype(cs, &char_type, next, end); > + > + // Broken data? > + if (next + char_len > end || char_len == 0) > + break; > + > + // Avoid creating partial n-grams by stopping at word boundaries. > + // Underscore is treated as a word character too, since identifiers in > + // programming languages often contain them. > + if (!is_alnum(char_type) && !is_underscore(cs, next, end)) { > + start= next + char_len; > + next= start; > + n_chars= 0; > + continue; I don't think it's a good idea. On the opposite, I would rather create ngrams over word boundaries, so that, say, I'd split "n-gram plugin" to "n-g", "-gr", "gra", "ram", "am ", "m p", " pl", "plu", "lug", "ugi", "gin". May be better would be to split it into "n-gr", "gra", "ram", "am p", "m pl", "plu", "lug", "ugi", "gin". (that is, always three letters in an n-gram, not three characters). This needs testing to find out what approach works better. > + } > + > + next += char_len; > + n_chars++; > + > + if (n_chars == ngram_token_size) { > + bool_info->position= static_cast<uint>(start - doc); nope, we don't have MYSQL_FTPARSER_FULL_BOOLEAN_INFO::position and this plugin is definitely not the reason to break the API and add it. > + ret= param->mysql_add_word(param, reinterpret_cast<const char *>(start), > + static_cast<int>(next - start), bool_info); > + if (ret != 0) > + return ret; > + > + start += my_ci_charlen(cs, start, end); why do you scan the string again here, you've already counted character by character above? > + n_chars= ngram_token_size - 1; > + is_first= false; > + } > + } > + > + // If nothing was emitted yet, emit the remainder. what reminder, what is it for? > + switch (param->mode) { > + case MYSQL_FTPARSER_FULL_BOOLEAN_INFO: > + case MYSQL_FTPARSER_WITH_STOPWORDS: > + if (n_chars > 0 && is_first) { > + assert(next > start); > + assert(n_chars < ngram_token_size); > + > + bool_info->position= static_cast<uint>(start - doc); > + ret= param->mysql_add_word(param, reinterpret_cast<const char *>(start), > + static_cast<int>(next - start), bool_info); > + } > + break; > + > + default: > + break; > + } > + > + return ret; > +} > + > +static int number_of_chars(const CHARSET_INFO *cs, const uchar *buf, int len) > +{ > + const uchar *ptr= buf; > + const uchar *end= buf + len; > + int size= 0; > + > + while (ptr < end) { > + ptr += my_ci_charlen(cs, ptr, end); > + size += 1; > + } > + > + return size; > +} > + > +// Convert term into phrase and handle wildcard. > +static int ngram_term_convert(MYSQL_FTPARSER_PARAM *param, const uchar *token, > + int len, MYSQL_FTPARSER_BOOLEAN_INFO *bool_info) > +{ > + MYSQL_FTPARSER_BOOLEAN_INFO token_info= > + { FT_TOKEN_WORD, 0, 0, 0, 0, 0, ' ', 0 }; > + const CHARSET_INFO *cs= param->cs; > + int ret= 0; > + > + assert(bool_info->type == FT_TOKEN_WORD); > + assert(bool_info->quot == NULL); > + > + /* Convert rules: > + * 1. if term with wildcard and term length is less than ngram_token_size, > + * we keep it as normal term search. > + * 2. otherwise, term is converted to phrase and wildcard is ignored. > + * e.g. 'abc' and 'abc*' are both equivalent to '"ab bc"'. > + */ > + > + if (bool_info->trunc && number_of_chars(cs, token, len) < ngram_token_size) { > + ret= param->mysql_add_word(param, reinterpret_cast<const char *>(token), > + len, bool_info); > + } else { > + bool_info->type= FT_TOKEN_LEFT_PAREN; > + bool_info->quot= reinterpret_cast<char*>(1); > + > + ret= param->mysql_add_word(param, NULL, 0, bool_info); > + if (ret != 0) > + return ret; > + > + ret= split_into_ngrams(param, token, len, &token_info); > + if (ret != 0) > + return ret; > + > + bool_info->type= FT_TOKEN_RIGHT_PAREN; > + ret= param->mysql_add_word(param, NULL, 0, bool_info); > + > + assert(bool_info->quot == NULL); > + bool_info->type= FT_TOKEN_WORD; > + } > + > + return ret; > +} > + > +static int ngram_parser_parse(MYSQL_FTPARSER_PARAM *param) > +{ > + MYSQL_FTPARSER_BOOLEAN_INFO bool_info= > + { FT_TOKEN_WORD, 0, 0, 0, 0, 0, ' ', 0 }; > + const CHARSET_INFO *cs= param->cs; > + uchar *start= reinterpret_cast<uchar *>(const_cast<char *>(param->doc)); > + uchar *end= start + param->length; > + FT_WORD word= {NULL, 0, 0}; > + int ret= 0; > + > + switch (param->mode) { > + case MYSQL_FTPARSER_SIMPLE_MODE: > + case MYSQL_FTPARSER_WITH_STOPWORDS: > + ret= split_into_ngrams(param, start, static_cast<int>(end - start), > + &bool_info); > + break; > + > + case MYSQL_FTPARSER_FULL_BOOLEAN_INFO: > + // Reusing InnoDB's boolean mode parser to handle all special characters. > + while (fts_get_word(cs, &start, end, &word, &bool_info)) { nope. InnoDB might not even be compiled it. Use param->mysql_parse() > + if (bool_info.type == FT_TOKEN_WORD) { > + if (bool_info.quot != NULL) { > + ret= split_into_ngrams(param, word.pos, word.len, &bool_info); > + } else { > + ret= ngram_term_convert(param, word.pos, word.len, &bool_info); > + } > + } else { > + ret= param->mysql_add_word(param, reinterpret_cast<char*>(word.pos), > + word.len, &bool_info); > + } > + > + if (ret != 0) > + return ret; > + } > + > + break; > + } > + > + return ret; > +} > + > +static struct st_mysql_ftparser ngram_parser_descriptor= > +{ > + MYSQL_FTPARSER_INTERFACE_VERSION, > + ngram_parser_parse, > + NULL, > + NULL, > +}; > + > +static struct st_mysql_sys_var *ngram_system_variables[]= > +{ > + MYSQL_SYSVAR(token_size), > + NULL > +}; > + > +maria_declare_plugin(ngram_parser) > +{ > + MYSQL_FTPARSER_PLUGIN, // Type. > + &ngram_parser_descriptor, // Descriptor. > + "ngram", // Name. > + "Oracle Corp", // Author. > + "Ngram Full-Text Parser", // Description. > + PLUGIN_LICENSE_GPL, // License. > + NULL, // Initialization function. > + NULL, // Deinitialization function. > + 0x0100, // Numeric version. > + NULL, // Status variables. > + ngram_system_variables, // System variables. > + "1.0", // String version representation. > + MariaDB_PLUGIN_MATURITY_EXPERIMENTAL, // Maturity. > +} > +maria_declare_plugin_end; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 12d5c4f2813: Fixed problems with S3 after "DROP TABLE FORCE" changes
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Monty! On Oct 16, Michael Widenius wrote: > revision-id: 12d5c4f2813 (mariadb-10.5.4-190-g12d5c4f2813) > parent(s): afa5f735f30 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-13 17:48:15 +0300 > message: > > Fixed problems with S3 after "DROP TABLE FORCE" changes > > MDEV-23691 S3 storage engine: delayed slave can drop the table please put the MDEV on the first line of the commit comment > > This commit also fixes all failing replication S3 tests. > > A slave is delayed if it is trying to execute replicated queries on a > table that is already converted to S3 by the master. What do you mean, a slave is delayed? > > Fixes for replication events on S3 tables for delayed slaves: > - INSERT and INSERT ... SELECT and CREATE TABLE are ignored but written > to the binary log. UPDATE & DELETE will be fixed in a future commit. > > Other things: > - On slaves with --s3-slave-ignore-updates set, allow S3 tables to be > opened in read-write mode. This was done to be able to > ignore-but-replicate queries like insert. Without this change any > open of an S3 table failed with 'Table is read only' which is too > early to be able to replicate the original query. > - Errors are now printed if handler::extra() call fails in > wait_while_tables_are_used() > - Error message for row changes are changed from HA_ERR_WRONG_COMMAND > to HA_ERR_TABLE_READONLY > - Disable some maria_extra() calls for S3 tables. This could cause > S3 tables to fail in some cases. > - Added missing thr_lock_delete() to ma_open() in case of failure > > diff --git a/sql/log.cc b/sql/log.cc > index 2a887a68606..9c656387e90 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2151,6 +2151,12 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all) > > DBUG_RETURN(0); > } > + /* > + This is true if we are doing an alter table that is replicated as > + CREATE TABLE ... SELECT > + */ > + if (thd->variables.option_bits & OPTION_BIN_COMMIT_OFF) > + DBUG_RETURN(0); why binlog_commit is even called here? > > DBUG_PRINT("debug", > ("all: %d, in_transaction: %s, all.modified_non_trans_table: %s, stmt.modified_non_trans_table: %s", > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index f4cbda5490c..7c103756566 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -7476,6 +7476,34 @@ int THD::binlog_query(THD::enum_binlog_query_type qtype, char const *query_arg, > DBUG_RETURN(0); > } > > + > +/** > + Binlog current query as a statement if binary log is open > + > + @retval false ok > + @retval true error > +*/ > + > + > +bool THD::binlog_current_query() > +{ > + if (!mysql_bin_log.is_open()) > + return 0; > + > + /* > + The following is needed if the query was "ignored" by a shared distributed > + engine, in which case decide_logging_format() will set the binlog filter > + */ > + reset_binlog_local_stmt_filter(); > + clear_binlog_local_stmt_filter(); > + return binlog_query(THD::STMT_QUERY_TYPE, query(), query_length(), > + /* is_trans */ FALSE, > + /* direct */ FALSE, > + /* suppress_use */ FALSE, > + /* Error */ 0) > 0; > +} you created a generally looking function with a rather special behavior. Please rename it to make sure it's not a general purpose helper. For example, call it binlog_current_query_unfiltered() or force_binlog_current_query(). And the function comment should say Force-binlog current query as a statement ignoring the binlog filter setting > + > + > void > THD::wait_for_wakeup_ready() > { > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index 4ad4c478937..5f59e064a35 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -724,6 +724,7 @@ bool mysql_insert(THD *thd, TABLE_LIST *table_list, > Item *unused_conds= 0; > DBUG_ENTER("mysql_insert"); > > + bzero((char*) &info,sizeof(info)); why did you move it up? > create_explain_query(thd->lex, thd->mem_root); > /* > Upgrade lock type if the requested lock is incompatible with > @@ -764,16 +765,27 @@ bool mysql_insert(THD *thd, TABLE_LIST *table_list, > DBUG_RETURN(TRUE); > value_count= values->elements; > > - if (mysql_prepare_insert(thd, table_list, table, fields, values, > - update_fields, update_values, duplic, > - &unused_conds, FALSE)) > + if ((res= mysql_prepare_insert(thd, table_list, table, fields, values, > + update_fields, update_values, duplic, > + &unused_conds, FALSE))) > + { > + retval= thd->is_error(); > + if (res < 0) > + { > + /* > + Insert should be ignored but we have to log the query in statement > + format in the binary log > + */ > + res= thd->binlog_current_query(); abort label doesn't look at `res` it returns `retval`. So I don't think you need `retval= thd->is_error()` at all, and instead you need if (res < 0) retval= thd->binlog_current_query(); > + } > goto abort; > + } > + /* mysql_prepare_insert sets table_list->table if it was not set */ > + table= table_list->table; > > /* Prepares LEX::returing_list if it is not empty */ > if (returning) > result->prepare(returning->item_list, NULL); > - /* mysql_prepare_insert sets table_list->table if it was not set */ > - table= table_list->table; why did you move this up? > > context= &thd->lex->first_select_lex()->context; > /* > @@ -1575,21 +1587,29 @@ bool mysql_prepare_insert(THD *thd, TABLE_LIST *table_list, > DBUG_ASSERT (!select_insert || !values); > > if (mysql_handle_derived(thd->lex, DT_INIT)) > DBUG_RETURN(1); > if (table_list->handle_derived(thd->lex, DT_MERGE_FOR_INSERT)) > DBUG_RETURN(1); > if (thd->lex->handle_list_of_derived(table_list, DT_PREPARE)) > DBUG_RETURN(1); > > if (duplic == DUP_UPDATE) > { > /* it should be allocated before Item::fix_fields() */ > if (table_list->set_insert_values(thd->mem_root)) > DBUG_RETURN(1); > + } > + > + if (!table) > + table= table_list->table; As far as I can see, table in mysql_prepare_insert is *always* either table_list->table or NULL. At least in 10.5. So I'd rather remove the table argument completely and let mysql_prepare_insert always use table_list->table. > + > + if (table->file->check_if_updates_are_ignored("INSERT")) > + { > + DBUG_RETURN(-1); > } > > if (mysql_prepare_insert_check_table(thd, table_list, fields, select_insert)) > DBUG_RETURN(1); > > /* Prepare the fields in the statement. */ > if (values) > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 15d190c3139..2843496f783 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -5206,7 +5215,19 @@ int create_table_impl(THD *thd, const LEX_CSTRING &orig_db, > goto err; > } > else if (options.if_not_exists()) > + { > + /* > + We never come here as part of normal create table as table existance > + is checked in open_and_lock_tables(). We may come here as part of > + ALTER TABLE > + */ How so? There is no ALTER IF NOT EXISTS. And this is the old code, so it seems to be for CREATE TABLE IF NOT EXISTS. > + > + /* Log CREATE IF NOT EXISTS on slave for distributed engines */ > + if (thd->slave_thread && (exists_hton && exists_hton->flags & > + HTON_IGNORE_UPDATES)) > + thd->log_current_statement= 1; > goto warn; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison
by Sergey Petrunia 16 Oct '20

16 Oct '20

Hi Varun, > Commit 3e29c5a1620deef2fa9b25a0d42b4c07fe59e96f > > MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison > > Allow materialization strategy when the collation of the columns of the > left side of the IN subquery are not same as the one on the right side > of the IN subquery but are compatible with each other. > This allows conversion from utf8mb3 to utf8mb4 as the former is a subset of the later. Please fix the comment to say what the patch does: When collations on the inner and outer sides are the same, and the character set of the inner side is a proper subset of the charset on the outer. > > diff --git a/mysql-test/main/subselect4.result b/mysql-test/main/subselect4.result > index 020a70642913..f3ae2398d45e 100644 > --- a/mysql-test/main/subselect4.result > +++ b/mysql-test/main/subselect4.result > @@ -2718,3 +2718,172 @@ Warning 1931 Query execution was interrupted. The query examined at least 3020 r > SET join_cache_level= @save_join_cache_level; > DROP TABLE t1,t2,t3,t4; > # End of 10.2 tests > +# > +# MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison > +# > +CREATE TABLE t1 ( col1 varchar (32) collate utf8_general_ci); > +CREATE TABLE t2 ( col1 varchar (32) collate utf8mb4_general_ci, key(col1)); > +INSERT INTO t1 select seq from seq_1_to_100; > +INSERT INTO t2 values (_utf8mb4 X'F09F9883'), (1),('?'); ... > EXPLAIN FORMAT=JSON SELECT HEX(col1), col1 IN (SELECT col1 FROM t1) FROM t2; Please insert the '?' into the table t1, not t2. The idea of the check is to check for the following: - We get smiley symbol from table t2. - We make the lookup key from it. This results in an error and the '?' as output. - Then, a lookup is made in t1. And we are trying to provoke a wrong result by having the '?' in t1. OK to push into the custom-build tree when the above is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 0d05c9f8699: Add mytop to Client component for rpm package
by Sergei Golubchik 13 Oct '20

13 Oct '20

Hi, Anel! About the diff to 1.9, I tend to agree that our mytop is a superset. I did have some comments when looking at the diff, couldn't help it :) Here they are: > diff --git a/scripts/mytop.sh b/../mytop-1.9.1-4 > index 321b79537fc..056b7a9a806 100644 > --- b/../mytop-1.9.1-4 > +++ a/scripts/mytop.sh > @@ -1,13 +1,13 @@ > -#!/usr/bin/perl -w > +#!/usr/bin/env perl I thought we're getting rid of "env perl" ? > # > -# $Id: mytop,v 1.91 2012/01/18 16:49:12 mgrennan Exp $ > +# $Id: mytop,v 1.99-maria6 2019/10/22 14:53:51 jweisbuch Exp $ > > =pod > > =head1 NAME > > -mytop - display MySQL server performance info like `top' > +mytop - display MariaDB server performance info like `top' may be "MariaDB/MySQL" here and below? > > =cut > > @@ -242,11 +253,15 @@ my $dsn; > > ## Socket takes precedence. > > -$dsn ="DBI:mysql:database=$config{db};mysql_read_default_group=mytop;"; > +if (eval {DBI->install_driver("MariaDB")}) { > + $dsn = "DBI:MariaDB:database=$config{db};mariadb_read_default_group=mytop;"; > +} else { > + $dsn = "DBI:mysql:database=$config{db};mysql_read_default_group=mytop;"; > +} > > if ($config{socket} and -S $config{socket}) > { > - $dsn .= "mysql_socket=$config{socket}"; > + $dsn .= "mariadb_socket=$config{socket}"; if you support both drivers, you need to use the correct option here > } > else > { > @@ -1027,10 +918,15 @@ sub GetData() > } > } > > - open L, "</proc/loadavg"; > - my $l = <L>; > - close L; > - chomp $l; > + my $l; > + if (-e "/proc/loadavg") > + { > + ## To avoid warnings if the OS is not Linux > + open (my $fh, "<", "/proc/loadavg"); > + ## Only the first 3 values are interresting rr? > + $l = join(" ", (split /\s+/, <$fh>)[0..2]); > + close $fh; > + } > > $last_time = $now_time; > > @@ -1960,11 +1694,15 @@ sub PrintHelp() > S - change slow query hightlighting > t - switch to thread view (default) > u - show only a specific user > + V - show variables > : - enter a command (not yet implemented) > ! - Skip an error that has stopped replications (at your own risk) > + L - show full queries (do not strip to terminal width) > + w - adjust the User and DB columns width > + a - toggle the progress column > > -${GREEN}http://jeremy.zawodny.com/mysql/mytop/${RESET} > -${GREEN}http://www.mysqlfanboy.com/mytop-3/${RESET} > +Base version from ${GREEN}http://www.mysqlfanboy.com/mytop-3${RESET} if it's mytop-3, perhaps it should have the version 3.x and not 1.99-maria6 ? > +This version comes as part of the ${GREEN}MariaDB${RESET} distribution. > ]; > > print $help; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 2

Re: [Maria-developers] d19f380d62c: MDEV-21445 Strange/inconsistent behavior of IN condition when mixing numbers and strings
by Sergei Golubchik 12 Oct '20

12 Oct '20

Hi, Alexander! On Oct 12, Alexander Barkov wrote: > revision-id: d19f380d62c (mariadb-10.3.21-213-gd19f380d62c) > parent(s): 90c8d773ed0 > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2020-08-18 13:19:21 +0400 > message: > > MDEV-21445 Strange/inconsistent behavior of IN condition when mixing numbers and strings > > Comparison is now performed as DECIMAL for pairs (int,string) and (string,int). Are there tests with strings like '1e100', outside of bigint range ? > Before the patch the comparison was performed in DOUBLE format, > so precision loss happened on huge values, e.g.: > > '97716021308405775' was converted to > 97716021308405770 (notice, the last digit was lost) > > diff --git a/sql/field.h b/sql/field.h > index be3a648617b..c9f5a34a3ae 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -311,7 +311,7 @@ class Value_source > const char *end) > { > Converter_str2my_decimal_with_warn(NULL, Warn_filter_all(), > - E_DEC_FATAL_ERROR & ~E_DEC_BAD_NUM, > + 0, why did you remove warnings (here and elsewhere)? would be good to have it in the commit comment too, not just in email. > cs, cptr, end - cptr, decimal_value); > return decimal_value; > } > diff --git a/sql/sql_type.cc b/sql/sql_type.cc > index 9d3a47adfa5..7a847c90b8f 100644 > --- a/sql/sql_type.cc > +++ b/sql/sql_type.cc > @@ -764,6 +764,11 @@ Type_handler_hybrid_field_type::aggregate_for_comparison(const Type_handler *h) > m_type_handler= &type_handler_datetime; > } > } > + else if ((a == INT_RESULT && b == STRING_RESULT) || > + (b == INT_RESULT && a == INT_RESULT)) looks like a typo, the second line should be b == INT_RESULT && a == STRING_RESULT please add some tests to cover it. > + { > + m_type_handler= &type_handler_newdecimal; > + } > else if ((a == INT_RESULT || a == DECIMAL_RESULT) && > (b == INT_RESULT || b == DECIMAL_RESULT)) > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2792940391a: MDEV-23327 Can't uninstall UDF if the implementation library file doesn't exist
by Sergei Golubchik 11 Oct '20

11 Oct '20

Hi, Oleksandr! Mostly ok, one problem below: On Oct 10, Oleksandr Byelkin wrote: > revision-id: 2792940391a (mariadb-10.2.31-343-g2792940391a) > parent(s): 2107e3bb9c0 > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-07-29 21:54:24 +0200 > message: > > MDEV-23327 Can't uninstall UDF if the implementation library file doesn't exist > > Made cleanup of DROP (udf) FUNCTION procedure and also check of mysql.func (not only loaded udf). > > @@ -606,43 +648,54 @@ int mysql_create_function(THD *thd,udf_func *udf) > } > > > -int mysql_drop_function(THD *thd,const LEX_STRING *udf_name) > +enum drop_udf_result mysql_drop_function(THD *thd, const LEX_STRING *udf_name) > { > TABLE *table; > - TABLE_LIST tables; > udf_func *udf; > DBUG_ENTER("mysql_drop_function"); > > + if (!(table= open_udf_func_table(thd))) > + DBUG_RETURN(UDF_DEL_RESULT_ERROR); > + > + // Pre-check without protection > + if (!find_udf_everywhere(thd, udf_name->str, udf_name->length, table)) you cannot check the udf_hash without a lock > + { > + close_mysql_tables(thd); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

[Maria-developers] Unsubscribe
by TINKU BANSAL 11 Oct '20

11 Oct '20

Hi there, Please Unsubscribe me to this mailing list. Thanks

2 1

Re: [Maria-developers] 62c35fe93e1: MDEV-15703 Crash in EXECUTE IMMEDIATE 'CREATE OR REPLACE TABLE t1 (a INT DEFAULT ?)' USING DEFAULT
by Sergei Golubchik 09 Oct '20

09 Oct '20

Hi, Oleksandr! On Oct 09, Oleksandr Byelkin wrote: > revision-id: 62c35fe93e1 (mariadb-10.2.31-487-g62c35fe93e1) > parent(s): f584d567dbb > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-10-07 13:43:39 +0200 > message: > > MDEV-15703 Crash in EXECUTE IMMEDIATE 'CREATE OR REPLACE TABLE t1 (a INT DEFAULT ?)' USING DEFAULT > > part 2: > > - check that expressions are evaluable for > making empty row and assigning PS variable > > - correctly handling writing to a temporary tabe during multi-update > by setting associated field > > - Item::raise_error_not_evaluable() bakported from 10.4 and made vitrual > > - Item::is_evaluable_expression() bakported from 10.4 > > - Item::check_is_evaluable_expression_or_error() bakported from 10.4 I still don't quite understand all details of your patch. But here're more questions and comments below: > diff --git a/sql/field.cc b/sql/field.cc > index bdaaecc2026..8ce412a207d 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -11183,6 +11205,16 @@ bool Field::save_in_field_default_value(bool view_error_processing) > { > THD *thd= table->in_use; > > + if ( // table is not opened properly > + table == NULL || table->pos_in_table_list == NULL || > + table->pos_in_table_list->select_lex == NULL || > + // we are in subquery/derived/CTE > + !table->pos_in_table_list->top_table()->select_lex->is_top_level_select()) > + { > + DBUG_ASSERT(0); // shoud not happened > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return false; > + } I'd still think it'd be better to do a series of asserts. a complex condition to check for something that cannot ever happen? > if (flags & NO_DEFAULT_VALUE_FLAG && > real_type() != MYSQL_TYPE_ENUM) > { > @@ -11221,13 +11253,30 @@ bool Field::save_in_field_default_value(bool view_error_processing) > > bool Field::save_in_field_ignore_value(bool view_error_processing) > { > - enum_sql_command com= table->in_use->lex->sql_command; > - // All insert-like commands > - if (com == SQLCOM_INSERT || com == SQLCOM_REPLACE || > - com == SQLCOM_INSERT_SELECT || com == SQLCOM_REPLACE_SELECT || > - com == SQLCOM_LOAD) > - return save_in_field_default_value(view_error_processing); > - return 0; // ignore > + if ( // table is not opened properly > + table == NULL || table->pos_in_table_list == NULL || > + table->pos_in_table_list->select_lex == NULL || > + // we are in subquery/derived/CTE > + !table->pos_in_table_list->top_table()->select_lex->is_top_level_select()) > + { > + DBUG_ASSERT(0); // shoud not happened > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return false; > + } ditto > + switch (find_ignore_reaction(table->in_use)) > + { > + case IGNORE_MEANS_DEFAULT: > + return save_in_field_default_value(view_error_processing); > + case IGNORE_MEANS_FIELD_VALUE: > + return 0; // ignore > + default: > + ; // fall through to error > + } > + > + // unexpected command > + DBUG_ASSERT(0); // shoud not happened > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return false; > } > > > diff --git a/sql/item.cc b/sql/item.cc > index 994d45a9dc3..401ee5d6be2 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -84,6 +85,17 @@ void item_init(void) > } > > > +void Item::raise_error_not_evaluable() > +{ > + String tmp; > + this->print(&tmp, QT_ORDINARY); > + // TODO-10.5: add an error message to errmsg-utf8.txt > + my_printf_error(ER_UNKNOWN_ERROR, > + "'%s' is not allowed in this context", MYF(0), tmp.ptr()); > + DBUG_ASSERT(0); // cannot be happen before 10.5 > +} > + > + better remove it, this dead code won't simplify merges. In 10.3 it'll be added with a merge normally. having check_is_evaluable_expression_or_error() makes sense though, it is used and copying it from 10.3 will help to merge indeed or, may be, just backport the patch from 10.3 to 10.2? > void Item::push_note_converted_to_negative_complement(THD *thd) > { > push_warning(thd, Sql_condition::WARN_LEVEL_NOTE, ER_UNKNOWN_ERROR, > @@ -8968,18 +9084,26 @@ bool Item_default_value::fix_fields(THD *thd, Item **items) > def_field->move_field_offset((my_ptrdiff_t) > (def_field->table->s->default_values - > def_field->table->record[0])); > - set_field(def_field); > - return FALSE; > + return def_field; > +} > > -error: > - context->process_error(thd); > - return TRUE; > +bool Item_default_value::associate_with_target_field(THD *thd, > + Item_field *field) > +{ > + associated= true; > + arg= field; iirc, `arg` used to tell the difference between DEFAULT and DEFAULT(x). how are you doing it now? > + return tie_field(thd); > } > > void Item_default_value::cleanup() > { > delete cached_field; // Free cached blob data > cached_field= 0; > + if (associated) > + { > + arg= NULL; > + associated= false; > + } > Item_field::cleanup(); > } > > @@ -9102,8 +9226,54 @@ void Item_ignore_value::print(String *str, enum_query_type query_type) > str->append(STRING_WITH_LEN("ignore")); > } > > + > +bool Item_ignore_value::associate_with_target_field(THD *thd, > + Item_field *field) > +{ > + bitmap_set_bit(field->field->table->read_set, field->field->field_index); > + return Item_default_value::associate_with_target_field(thd, field); > +} > + > + > int Item_ignore_value::save_in_field(Field *field_arg, bool no_conversions) > { > + if (arg) how is IGNORE with arg!=0 differs from IGNORE with arg==0? when happens what? > + { > + switch (find_ignore_reaction(field->table->in_use)) > + { > + case IGNORE_MEANS_DEFAULT: > + DBUG_ASSERT(0); // impossible now, but fully working code if needed > + /* > + save default value > + Note: (((Field*)(arg()->real_item()) is Item_field* (checked in > + tie_field(). > + */ > + if ((((Item_field*)(arg->real_item()))->field->flags & > + NO_DEFAULT_VALUE_FLAG)) > + { > + my_error(ER_NO_DEFAULT_FOR_FIELD, MYF(0), > + ((Item_field*)(arg->real_item()))->field->field_name); > + return true; > + } > + calculate(); > + return Item_field::save_in_field(field_arg, no_conversions); > + case IGNORE_MEANS_FIELD_VALUE: > + // checked on tie_field > + DBUG_ASSERT(arg->real_item()->type() == FIELD_ITEM); > + /* > + save original field (Item::save_in_field is not applicable because > + result_field for the referenced field set to this temporary table). > + */ > + arg->save_val(field_arg); > + return false; > + default: > + ; // fall through to error > + } > + DBUG_ASSERT(0); //impossible > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return true; > + } > + > return field_arg->save_in_field_ignore_value(context->error_processor == > &view_error_processor); > } > diff --git a/sql/sp_head.cc b/sql/sp_head.cc > index 6a650183fb8..41a27a3aaaf 100644 > --- a/sql/sp_head.cc > +++ b/sql/sp_head.cc > @@ -418,6 +418,9 @@ sp_eval_expr(THD *thd, Field *result_field, Item **expr_item_ptr) > if (!(expr_item= sp_prepare_func_item(thd, expr_item_ptr))) > goto error; > > + if (expr_item->check_is_evaluable_expression_or_error()) > + goto error; for CALL(?) ? > + > /* > Set THD flags to emit warnings/errors in case of overflow/type errors > during saving the item into the field. > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index cc77b58cb3e..b52d6f96a84 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -8345,6 +8346,9 @@ fill_record(THD *thd, TABLE *table, Field **ptr, List<Item> &values, > field->field_name, table->s->table_name.str); > } > > + if(evaluable && value->check_is_evaluable_expression_or_error()) > + goto err; Is it a good idea to do this check constantly for every user and for every row in selects, insert/update, etc? Could it be done inside IGNORE/DEFAULT? For example, by checking if field->table->tmp_table == INTERNAL_TMP_TABLE. > + > if (use_value) > value->save_val(field); > else > diff --git a/sql/unireg.cc b/sql/unireg.cc > index 083960523c1..91f9073c7ef 100644 > --- a/sql/unireg.cc > +++ b/sql/unireg.cc > @@ -1002,6 +1002,12 @@ static bool make_empty_rec(THD *thd, uchar *buff, uint table_options, > > int res= !expr->fixed && // may be already fixed if ALTER TABLE > expr->fix_fields(thd, &expr); > + if (!res && expr->check_is_evaluable_expression_or_error()) is this for CREATE TABLE case? > + { > + error= 1; > + delete regfield; //To avoid memory leak > + goto err; > + } > if (!res) > res= expr->save_in_field(regfield, 1); > if (!res && (field->flags & BLOB_FLAG)) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 56f2b3d12b6: MDEV-23264 Unique blobs allow duplicate values upon UPDATE
by Sergei Golubchik 08 Oct '20

08 Oct '20

Hi, Sachin! I suspect is_null() could be wrong here. Please, add a test case where a table has a BEFORE UPDATE trigger and a long UNIQUE over a NOT NULL field. And see how cmp_binary_offset() is used in compare_record(), for example. On Oct 08, Sachin Setiya wrote: > revision-id: 56f2b3d12b6 (mariadb-10.4.11-385-g56f2b3d12b6) > parent(s): af834c218a5 > author: Sachin Setiya <sachin.setiya(a)mariadb.com> > committer: Sachin Setiya <sachin.setiya(a)mariadb.com> > timestamp: 2020-09-14 20:08:18 +0100 > message: > > MDEV-23264 Unique blobs allow duplicate values upon UPDATE > > Problem:- > We are able to insert duplicate value in table because cmp_binary_offset > is not able to differentiate between NULL and empty string. So > check_duplicate_long_entry_key is never called and we don't check for > duplicate. > Solution > Added a if condition with is_null() on field which can differentiate > between NULL and empty string. > > diff --git a/sql/handler.cc b/sql/handler.cc > index 40dea349272..71e6a904f23 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -6699,8 +6699,13 @@ static int check_duplicate_long_entries_update(TABLE *table, handler *h, uchar * > for (uint j= 0; j < key_parts; j++, keypart++) > { > field= keypart->field; > - /* Compare fields if they are different then check for duplicates*/ > - if(field->cmp_binary_offset(reclength)) > + /* > + Compare fields if they are different then check for duplicates > + cmp_binary_offset cannot differentiate between null and empty string > + So also check for that too > + */ > + if((field->is_null(0) != field->is_null(reclength)) || > + field->cmp_binary_offset(reclength)) > { > if((error= check_duplicate_long_entry_key(table, table->update_handler, > new_rec, i))) > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Review of MDEV-4851
by Vicențiu Ciorbaru 07 Oct '20

07 Oct '20

Hi Daniel! The patch looks good, some comments about the test cases. About the commit message, we have --write-binlog not just --binlog. Also, check for typos in the last commit message and also, I would squash the two test cases commits into just one. > diff --git a/mysql-test/suite/rpl/t/rpl_mysql_upgrade.test b/mysql-test/suite/rpl/t/rpl_mysql_upgrade.test > index 1931e2eab2a..b836e889429 100644 > --- a/mysql-test/suite/rpl/t/rpl_mysql_upgrade.test > +++ b/mysql-test/suite/rpl/t/rpl_mysql_upgrade.test > @@ -14,8 +14,8 @@ call mtr.add_suppression("table or database name 'mysqltest-1'"); > > connection master; > --disable_warnings > -DROP DATABASE IF EXISTS `#mysql50#mysqltest-1`; > -CREATE DATABASE `#mysql50#mysqltest-1`; > +DROP DATABASE IF EXISTS `mysqltest-1`; > +CREATE DATABASE `mysqltest-1`; Since you are doing cleanup here anyway, let's get rid of the DROP statements a test case should not leave any dirty context behind and if we drop things just to be sure, we are masking potential problems. > --enable_warnings > sync_slave_with_master; > > @@ -34,13 +34,20 @@ if ($before_position == $after_position) > echo Master position is not changed; > } > > -#Some log events of the mysql_upgrade's will cause errors on slave. > +# Some log events of the mysql_upgrade previously caused errors on slave, > +# however with MDEV-4851 this should be ok, so we test it: > connection slave; > -STOP SLAVE SQL_THREAD; > -source include/wait_for_slave_sql_to_stop.inc; > +SET @old_general_log_state = @@global.general_log; > +SET @old_slow_log_state = @@global.slow_query_log; > +SET @old_log_output = @@global.log_output; > +SET GLOBAL general_log = 'ON'; > +SET GLOBAL slow_query_log = 'ON'; > +SET GLOBAL log_output = 'FILE'; > > connection master; > #With '--force' option, mysql_upgrade always executes all sql statements for upgrading. > +ALTER TABLE mysql.slow_log DROP COLUMN thread_id, DROP COLUMN rows_affected; > +DROP DATABASE IF EXISTS `mysqltest-1`; Why IF EXISTS? It *should* be there already. Can you add some syncs between master and slave and then show from the slave connection that the log table table is missing those columns and also that they get applied *after* mysql_upgrade is run? > --exec $MYSQL_UPGRADE --skip-verbose --write-binlog --force --user=root > $MYSQLTEST_VARDIR/log/mysql_upgrade.log 2>&1 > > let $datadir= `select @@datadir`; > @@ -55,8 +62,14 @@ if ($before_position != $after_position) > echo Master position has been changed; > } > > -DROP DATABASE `mysqltest-1`; Notice that the original code didn't use "IF EXISTS". > +sync_slave_with_master; > connection slave; > -DROP DATABASE `#mysql50#mysqltest-1`; > ---let $rpl_only_running_threads= 1 > +SET GLOBAL general_log = 'OFF'; > +SET GLOBAL slow_query_log = 'OFF'; > +truncate mysql.slow_log; > +truncate mysql.general_log; Why do you need to clear the tables? I tested without clearing and it still needs to pass. > +SET GLOBAL general_log = @old_general_log_state; > +SET GLOBAL slow_query_log = @old_slow_log_state; > +SET GLOBAL log_output = @old_log_output; > + > --source include/rpl_end.inc > diff --git a/mysql-test/t/log_tables.test b/mysql-test/t/log_tables.test > index f822ec8d758..0196e62f12f 100644 > --- a/mysql-test/t/log_tables.test > +++ b/mysql-test/t/log_tables.test

1 0

Re: [Maria-developers] Fwd: [Commits] 244f5bd: MDEV-23811: With large number of indexes optimizer chooses an inefficient plan
by Sergey Petrunia 06 Oct '20

06 Oct '20

Hello Igor, On Mon, Oct 05, 2020 at 10:13:17PM -0700, Igor Babaev wrote: > > I've accepted all your changes for my original fix. > This is a new patch for 10.2. > ... > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index e933d2a..9f1fb9a 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -8899,11 +8902,10 @@ tree_or(RANGE_OPT_PARAM *param,SEL_TREE > *tree1,SEL_TREE *tree2) > { > SEL_ARG *key1= tree1->keys[key_no]; > SEL_ARG *key2= tree2->keys[key_no]; > - if (!must_be_ored) > - { > + if (!must_be_ored || no_merges1) > key1->incr_refs(); > + if (!must_be_ored || no_merges2) > key2->incr_refs(); > - } > if ((result->keys[key_no]= key_or(param, key1, key2))) > result->keys_map.set_bit(key_no); > } What is the purpose of this change? As far as I understand, the bug is fixed without it, too? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Test failure in main.flush_logs_not_windows on x32
by Otto Kekäläinen 06 Oct '20

06 Oct '20

Hello! After uploading MariaDB 10.5.5 to Debian I came across that the test main.mysqld--help is failing on arch x32: ****************** main.flush_logs_not_windows w2 [ fail ] Test ended at 2020-09-28 19:25:55 CURRENT_TEST: main.flush_logs_not_windows mysqltest: At line 10: query 'flush logs' succeeded - should have failed with errno 1004... ****************** Full log: https://people.debian.org/~jrtc27/mariadb-10.5_10.5.5-1+jrtc27.1_x32-2020-0… Any tips on what to do about this one?

1 0

[Maria-developers] Test failure in main.mysqld--help on x32
by Otto Kekäläinen 05 Oct '20

05 Oct '20

Hello! After uploading MariaDB 10.5.5 to Debian I came across that the test main.mysqld--help is failing on arch x32: ****************** main.mysqld--help 'unix' w1 [ fail ] Test ended at 2020-09-28 19:24:57 CURRENT_TEST: main.mysqld--help --- /<<PKGBUILDDIR>>/mysql-test/main/mysqld--help.result 2020-08-07 12:57:35.000000000 +0000 +++ /<<PKGBUILDDIR>>/mysql-test/main/mysqld--help.reject 2020-09-28 19:24:57.628353544 +0000 @@ -1797,10 +1797,10 @@ sync-relay-log-info 10000 sysdate-is-now FALSE system-versioning-alter-history ERROR -table-cache 421 +table-cache 2000 table-definition-cache 400 -table-open-cache 421 -table-open-cache-instances 1 +table-open-cache 2000 +table-open-cache-instances 8 tc-heuristic-recover OFF tcp-keepalive-interval 0 tcp-keepalive-probes 0 main.flush_logs_not_windows w2 [ fail ] Test ended at 2020-09-28 19:25:55 CURRENT_TEST: main.flush_logs_not_windows mysqltest: At line 10: query 'flush logs' succeeded - should have failed with errno 1004... ****************** Any tips on what to do about this? On Zulip I was tipped about https://github.com/MariaDB/server/commit/deb365581b96527c5c96b3f5ed1235e105… but it does not apply on 10.5.

2 2

[Maria-developers] Test failure in main.index_intersect on hppa
by Otto Kekäläinen 04 Oct '20

04 Oct '20

Hello! I already asked this on Zulip but did not get anywhere, so re-posting here on the mailing list. The basic test suite run directly after the build fails with the following test on hppa on Debian: https://buildd.debian.org/status/fetch.php?pkg=mariadb-10.5&arch=hppa&ver=1… ****************** main.index_intersect w3 [ fail ] Test ended at 2020-09-27 16:41:38 CURRENT_TEST: main.index_intersect --- /<<PKGBUILDDIR>>/mysql-test/main/index_intersect.result 2020-08-07 12:57:35.000000000 +0000 +++ /<<PKGBUILDDIR>>/mysql-test/main/index_intersect.reject 2020-09-27 16:41:38.035115473 +0000 @@ -715,7 +715,7 @@ SELECT * FROM City WHERE Name BETWEEN 'G' AND 'J' AND Population > 500000 AND Country LIKE 'C%'; id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE City range Population,Country,Name Name 35 NULL # Using index condition; Using where +1 SIMPLE City index_merge Population,Country,Name Name,Country 35,3 NULL # Using sort_intersect(Name,Country); Using where EXPLAIN SELECT * FROM City WHERE ID BETWEEN 1 AND 500 AND Population > 700000 AND Country LIKE 'C%'; mysqltest: Result l ****************** Any suggestions what this 'range' -> 'index_merge' word change in the result might step from?

1 0

Re: [Maria-developers] 19bdb03e97d: MDEV-21211 LOCK_plugin optimization.
by Sergei Golubchik 02 Oct '20

02 Oct '20

Hi, Alexey! Thanks, this is great! Comments are below, there aren't many. The main concern - you only do RCU when installingplugins, but not when uninstalling. I'm not sure how it's supposed to work. On Oct 02, Alexey Botchkov wrote: > revision-id: 19bdb03e97d (mariadb-10.4.11-337-g19bdb03e97d) > parent(s): 0041dacc1b8 > author: Alexey Botchkov <holyfoot(a)mariadb.com> > committer: Alexey Botchkov <holyfoot(a)mariadb.com> > timestamp: 2020-08-11 00:01:53 +0400 > message: > > MDEV-21211 LOCK_plugin optimization. > > Use Apc_target calls to 'notify' threads about changes in plugins. > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index bf33148811e..1f38c55b922 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -1348,10 +1348,10 @@ bool do_command(THD *thd) > my_net_set_read_timeout(net, thd->variables.net_read_timeout); > > DBUG_ASSERT(packet_length); > - DBUG_ASSERT(!thd->apc_target.is_enabled()); > + thd->apc_target.enable(); > return_value= dispatch_command(command, thd, packet+1, > (uint) (packet_length-1), FALSE, FALSE); > - DBUG_ASSERT(!thd->apc_target.is_enabled()); > + thd->apc_target.disable(); is this because you don't want to post apc requests to threads that wait for the client? > > out: > thd->lex->restore_set_statement_var(); > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 981420f03ae..84438e3ff70 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -217,17 +217,28 @@ static struct > #include "sql_plugin_services.ic" > > /* > - A mutex LOCK_plugin must be acquired before accessing the > - following variables/structures. > - We are always manipulating ref count, so a rwlock here is unneccessary. > + A mutex LOCK_plugin must be acquired when we change the > + new_plugins_state content and when we swap the global_plugins_state > + and the new_plugins_state. > */ > mysql_mutex_t LOCK_plugin; > +struct st_plugins_state > +{ > + DYNAMIC_ARRAY m_array; > + HASH m_hash[MYSQL_MAX_PLUGIN_TYPE_NUM]; > +#ifndef DBUG_OOF DBUG_OFF > + uint m_ref_counter; > +#endif > +}; > + > static DYNAMIC_ARRAY plugin_dl_array; > -static DYNAMIC_ARRAY plugin_array; > -static HASH plugin_hash[MYSQL_MAX_PLUGIN_TYPE_NUM]; > static MEM_ROOT plugin_mem_root; > static bool reap_needed= false; > volatile int global_plugin_version= 1; > +static st_plugins_state plugins_states[2]; > +st_plugins_state *global_plugins_state= plugins_states; it seems this can be static too, you don't use it outside of sql_plugin.cc (and making it global_plugins_state= &plugins_states[0], new_plugins_state= &plugins_state[1]; would be a bit more explicit) > +static st_plugins_state *new_plugins_state= plugins_states + 1; > + > > static bool initialized= 0; > ulong dlopen_count; > @@ -871,7 +882,8 @@ static void plugin_dl_del(struct st_plugin_dl *plugin_dl) > } > > > -static struct st_plugin_int *plugin_find_internal(const LEX_CSTRING *name, > +static struct st_plugin_int *plugin_find_internal(st_plugins_state *st, shouldn't `st` be const here too? > + const LEX_CSTRING *name, > int type) > { > uint i; > @@ -879,21 +891,19 @@ static struct st_plugin_int *plugin_find_internal(const LEX_CSTRING *name, > if (! initialized) > DBUG_RETURN(0); > > - mysql_mutex_assert_owner(&LOCK_plugin); may be you'd want here something like if (st == new_plugins_state) mysql_mutex_assert_owner(&LOCK_plugin); > - > if (type == MYSQL_ANY_PLUGIN) > { > for (i= 0; i < MYSQL_MAX_PLUGIN_TYPE_NUM; i++) > { > struct st_plugin_int *plugin= (st_plugin_int *) > - my_hash_search(&plugin_hash[i], (const uchar *)name->str, name->length); > + my_hash_search(&st->m_hash[i], (const uchar *)name->str, name->length); > if (plugin) > DBUG_RETURN(plugin); > } > } > else > DBUG_RETURN((st_plugin_int *) > - my_hash_search(&plugin_hash[type], (const uchar *)name->str, > + my_hash_search(&st->m_hash[type], (const uchar *)name->str, > name->length)); > DBUG_RETURN(0); > } > @@ -1210,13 +1209,6 @@ static void plugin_variables_deinit(struct st_plugin_int *plugin) > > static void plugin_deinitialize(struct st_plugin_int *plugin, bool ref_check) > { > - /* > - we don't want to hold the LOCK_plugin mutex as it may cause > - deinitialization to deadlock if plugins have worker threads > - with plugin locks > - */ > - mysql_mutex_assert_not_owner(&LOCK_plugin); does it mean you *do* keep LOCK_plugin here now? > - > if (plugin->plugin->status_vars) > { > /* > @@ -1280,24 +1273,25 @@ static void plugin_del(struct st_plugin_int *plugin) > DBUG_VOID_RETURN; > } > > + > static void reap_plugins(void) > { > uint count; > struct st_plugin_int *plugin, **reap, **list; > > - mysql_mutex_assert_owner(&LOCK_plugin); > - > if (!reap_needed) > return; > > + mysql_mutex_lock(&LOCK_plugin); > reap_needed= false; > - count= plugin_array.elements; > + count= global_plugins_state->m_array.elements; > reap= (struct st_plugin_int **)my_alloca(sizeof(plugin)*(count+1)); > *(reap++)= NULL; > > for (uint i=0; i < MYSQL_MAX_PLUGIN_TYPE_NUM; i++) > { > - HASH *hash= plugin_hash + plugin_type_initialization_order[i]; > + HASH *hash= global_plugins_state->m_hash + > + plugin_type_initialization_order[i]; do you need LOCK_plugin when changing plugin states? your comment at the LOCK_plugin declaration doesn't mention it. > for (uint j= 0; j < hash->records; j++) > { > plugin= (struct st_plugin_int *) my_hash_element(hash, j); > @@ -1542,6 +1529,38 @@ static void init_plugin_psi_keys(void) > } > #endif /* HAVE_PSI_INTERFACE */ > > + > +static bool copy_plugins_state(st_plugins_state *dest, st_plugins_state *src) > +{ > + uint i; > + mysql_mutex_assert_owner ? for documentation purposes. > + reset_dynamic(&dest->m_array); > + for (i= 0; i < MYSQL_MAX_PLUGIN_TYPE_NUM; i++) > + my_hash_reset(&dest->m_hash[i]); > + > + for (i= 0; i < src->m_array.elements; i++) > + { > + struct st_plugin_int *ptr; > + ptr= *dynamic_element(&src->m_array, i, struct st_plugin_int **); > + if (insert_dynamic(&dest->m_array, (uchar *) &ptr) || > + (ptr->state != PLUGIN_IS_FREED && > + my_hash_insert(&dest->m_hash[ptr->plugin->type], > + (uchar*) ptr))) > + goto err; > + } > + > + return false; > +err: > + return true; > +} > + > + > +static void set_new_global_plugins_state() > +{ mysql_mutex_assert_owner ? for documentation purposes. > + swap_variables(st_plugins_state *, global_plugins_state, new_plugins_state); > +} > + > + > /* > The logic is that we first load and initialize all compiled in plugins. > From there we load up the dynamic types (assuming we have not been told to > @@ -1815,7 +1834,9 @@ static void plugin_load(MEM_ROOT *tmp_root) > tables.init_one_table(&MYSQL_SCHEMA_NAME, &MYSQL_PLUGIN_NAME, 0, TL_READ); > tables.open_strategy= TABLE_LIST::OPEN_NORMAL; > > + set_new_global_plugins_state(); > result= open_and_lock_tables(new_thd, &tables, FALSE, MYSQL_LOCK_IGNORE_TIMEOUT); > + set_new_global_plugins_state(); why here, why twice? > > table= tables.table; > if (result) > @@ -1851,52 +1872,31 @@ static void plugin_load(MEM_ROOT *tmp_root) > if (!name.length || !dl.length) > continue; > > - /* > - Pre-acquire audit plugins for events that may potentially occur > - during [UN]INSTALL PLUGIN. > - > - When audit event is triggered, audit subsystem acquires interested > - plugins by walking through plugin list. Evidently plugin list > - iterator protects plugin list by acquiring LOCK_plugin, see > - plugin_foreach_with_mask(). > - > - On the other hand plugin_load is acquiring LOCK_plugin > - rather for a long time. > - > - When audit event is triggered during plugin_load plugin > - list iterator acquires the same lock (within the same thread) > - second time. > - > - This hack should be removed when LOCK_plugin is fixed so it > - protects only what it supposed to protect. > - > - See also mysql_install_plugin(), mysql_uninstall_plugin() and > - initialize_audit_plugin() > - */ > - if (mysql_audit_general_enabled()) > - mysql_audit_acquire_plugins(new_thd, event_class_mask); > - \o/ > /* > there're no other threads running yet, so we don't need a mutex. > but plugin_add() before is designed to work in multi-threaded > environment, and it uses mysql_mutex_assert_owner(), so we lock > the mutex here to satisfy the assert > */ > - mysql_mutex_lock(&LOCK_plugin); > plugin_add(tmp_root, false, &name, &dl, MYF(ME_ERROR_LOG)); > free_root(tmp_root, MYF(MY_MARK_BLOCKS_FREE)); > - mysql_mutex_unlock(&LOCK_plugin); > } > if (unlikely(error > 0)) > sql_print_error(ER_THD(new_thd, ER_GET_ERRNO), my_errno, > table->file->table_type()); > + set_new_global_plugins_state(); > end_read_record(&read_record_info); > table->mark_table_for_reopen(); > close_mysql_tables(new_thd); > -end: > +err_ret: > new_thd->db= null_clex_str; // Avoid free on thd->db > delete new_thd; > + set_new_global_plugins_state(); > DBUG_VOID_RETURN; > + > +end: > + set_new_global_plugins_state(); > + goto err_ret; eh, really? you swap twice again? > } > > > @@ -2058,13 +2055,8 @@ void plugin_shutdown(void) > plugin_deinitialize(plugins[i], false); > } > > - /* > - It's perfectly safe not to lock LOCK_plugin, as there're no > - concurrent threads anymore. But some functions called from here > - use mysql_mutex_assert_owner(), so we lock the mutex to satisfy it > - */ > - mysql_mutex_lock(&LOCK_plugin); > > + mysql_mutex_lock(&LOCK_plugin); the comment was helpful, I think > /* > We defer checking ref_counts until after all plugins are deinitialized > as some may have worker threads holding on to plugin references. > @@ -2176,6 +2173,44 @@ static bool finalize_install(THD *thd, TABLE *table, const LEX_CSTRING *name, > return 0; > } > > + > +/* > + Request object for ending the new_plugins_state change. > +*/ > + > +class Ping_all_threads_request : public Apc_target::Apc_call > +{ > +public: > + /* Overloaded virtual functions. */ > + void call_in_target_thread(); > +}; > + > + > +void Ping_all_threads_request::call_in_target_thread() > +{ > + /* Do nothing. We just make sure the thread got to this point. */ > +} > + > + > +static my_bool fix_plugins_state_callback(THD *thd, THD *caller_thd) > +{ > + bool timed_out; > + int timeout_sec= 30; > + > + Ping_all_threads_request ping_req; > + > + mysql_mutex_lock(&thd->LOCK_thd_kill); > + if (thd->get_command() == COM_SLEEP) here you also skip threads waiting on the client? > + { > + mysql_mutex_unlock(&thd->LOCK_thd_kill); > + return FALSE; > + } > + thd->apc_target.make_apc_call(caller_thd, &ping_req, > + timeout_sec, &timed_out); > + return FALSE; > +} > + > + > bool mysql_install_plugin(THD *thd, const LEX_CSTRING *name, > const LEX_CSTRING *dl_arg) > { > @@ -2252,12 +2272,19 @@ bool mysql_install_plugin(THD *thd, const LEX_CSTRING *name, > > if (unlikely(error != INSTALL_GOOD)) > { > + mysql_mutex_unlock(&LOCK_plugin); > reap_needed= true; > reap_plugins(); > + goto err; > } > -err: > global_plugin_version++; > + set_new_global_plugins_state(); > + thd->apc_target.disable(); > mysql_mutex_unlock(&LOCK_plugin); > + server_threads.iterate(fix_plugins_state_callback, thd); > + thd->apc_target.enable(); a comment be good here. to explain why you disable apc and why disable/unlock/iterate must happen in this particlar order. > + > +err: > if (argv) > free_defaults(argv); > DBUG_RETURN(error == INSTALL_FAIL_NOT_OK); > @@ -2414,10 +2416,12 @@ bool mysql_uninstall_plugin(THD *thd, const LEX_CSTRING *name, > error|= !MyFlags; > } > } > - reap_plugins(); > - > global_plugin_version++; > + thd->apc_target.disable(); > mysql_mutex_unlock(&LOCK_plugin); > + server_threads.iterate(fix_plugins_state_callback, thd); > + reap_plugins(); > + thd->apc_target.enable(); I don't quite understand this. You don't do copy-update here. I presume because plugins are only marked PLUGIN_IS_FREED but not actually removed from the array or hash? but you modify plugin state under a mutex, and read it without a mutex. how is that supposed to work? > DBUG_RETURN(error); > #ifdef WITH_WSREP > wsrep_error_label: Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6f211b1f01b: MDEV-23518 Syntax error in ond SP results in misleading message on SHOW CREATE PROCEDURE
by Sergei Golubchik 02 Oct '20

02 Oct '20

Hi, Oleksandr! On Oct 01, Oleksandr Byelkin wrote: > revision-id: 6f211b1f01b (mariadb-10.4.11-369-g6f211b1f01b) > parent(s): 571764c04fc > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-08-28 16:05:38 +0200 > message: > > MDEV-23518 Syntax error in ond SP results in misleading message on SHOW CREATE PROCEDURE > > Add info to the error message how to get details about error which happened. > > diff --git a/mysql-test/main/errors.test b/mysql-test/main/errors.test > index f3aedc8e491..a4b95d48e5d 100644 > --- a/mysql-test/main/errors.test > +++ b/mysql-test/main/errors.test > @@ -246,3 +246,39 @@ SELECT UPDATEXML(-73 * -2465717823867977728,@@global.auto_increment_increment,nu > SELECT UPDATEXML(-73 * -2465717823867977728,@@global.long_query_time,null); > > -- echo # End of 10.3 tests > + > +--echo # > +--echo # MDEV-23518: Syntax error in ond SP results in misleading > +--echo # message on SHOW CREATE PROCEDURE > +--echo # > + > +DELIMITER $$; > + > +CREATE PROCEDURE P1 () > +BEGIN NOT ATOMIC > + IF (SELECT 2) THEN > + SELECT 4; > + END IF ; > +END; > +$$ > + > +DELIMITER ;$$ you could've had just CREATE PROCEDURE P1 () SELECT 1; one line, no need to change a delimiter. You overwrite the body below anyway :) > + > +select name,db,body from mysql.proc where name = "P1"; > +update mysql.proc set body_utf8="BEGIN NOT ATOMIC > +IF (SELECT 2) OR foo = 3 THEN > +SELECT 4; > +END IF ; > +END", body="BEGIN NOT ATOMIC > +IF (SELECT 2) OR foo = 3 THEN > +SELECT 4; > +END IF ; > +END"where name = "P1"; > + > +--error ER_SP_PROC_TABLE_CORRUPT > +show create procedure P1; > +show warnings; > + > +drop procedure P1; > + > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index ddc52702e42..582c80e2ad8 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -5714,8 +5714,7 @@ ER_SP_RECURSION_LIMIT > eng "Recursive limit %d (as set by the max_sp_recursion_depth variable) was exceeded for routine %.192s" > ger "Rekursionsgrenze %d (durch Variable max_sp_recursion_depth gegeben) wurde für Routine %.192s überschritten" > ER_SP_PROC_TABLE_CORRUPT > - eng "Failed to load routine %-.192s. The table mysql.proc is missing, corrupt, or contains bad data (internal code %d)" > - ger "Routine %-.192s konnte nicht geladen werden. Die Tabelle mysql.proc fehlt, ist beschädigt, oder enthält fehlerhaften Daten (interner Code: %d)" > + eng "Failed to load routine %-.192s. The table mysql.proc is missing, corrupt, etc. (internal code %d). For more details, run SHOW WARNINGS" Okay, so you just extended the error message. Perhaps "The table mysql.proc is missing, corrupt, etc." is now redundant and shoud go away too? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] New patch on the JSON_TABLE.
by Sergey Petrunia 27 Sep '20

27 Sep '20

Hi Alexey, On Mon, Sep 14, 2020 at 03:38:38PM +0400, Alexey Botchkov wrote: > Hi, Sergei!. > > so the branch name is 'bb-mdef17399-hf'. > It has slightly changed since last patch so please pay attention. > > There i present the patch that resolves issues that you mentioned > in your last comments. > One important is not fixed though. > That is: > CREATE TABLE t1(id INT, f1 JSON); > INSERT INTO t1 VALUES (1, '{\"1\": 1}'), (2, '{\"1\": 2}'), (3, '{\"1\": > 3}'), > (4, '{\"1\": 4}'), (5, '{\"1\": 5}'), (6, '{\"1\": 6}'); > > SELECT * FROM t1 WHERE id IN > (SELECT id FROM t1 as tt2, > JSON_TABLE(f1, "$" COLUMNS (jf FOR ORDINALITY)) AS tbl); > > That SELECT crashes in the optimizer, as the greedy_search() can't find any > satisfying plan. > I got rid of the crash with this line: > @@ -9520,7 +9528,7 @@ best_extension_by_limited_search(JOIN *join, > table_map real_table_bit= s->table->map; > if ((remaining_tables & real_table_bit) && > (allowed_tables & real_table_bit) && > - !(remaining_tables & s->dependent) && > +// !(remaining_tables & s->dependent) && > > But i don't mean it's an acceptable fix. > > It seems to me the problem is in the optimizer, so i'd like to ask your > opinion. > Maybe you just know the answer at once. After some debugging I see that the following happens: 1. Execution reaches Table_function_json_table::setup It computes m_dep_tables= m_json->used_tables(); this is =1 (In the subquery, table tt2 has map=1) 2. Then, subquery is converted into a semi-join. That is, it is merged into its parent subquery. There, we have: table t1: map=1 table tt2: map=2 table JSON_TABLE(...) AS tbl: map=3 3. The subquery was uncorrelated, so SJ-Materialization is a possible option and the join optimizer attempts to construct a join order for the subquery tables. However it can't succeed, as the table "JSON_TABLE(..) AS tbl" is set to be dependent on table with map=1, which is not put into the join order because it is not a part of subquery. As a result, we get assertion failure when the optimizer fails to produce any join orders. The probem here is on step #2. When the subquery's tables are moved to the parent query, they get new values of TABLE::map, and also all their attributes that contain table maps need to be updated. This done here in convert_subq_to_sj() function: ``` /* n. Walk through child's tables and adjust table->map */ List_iterator_fast<TABLE_LIST> si(subq_lex->leaf_tables); while ((tl= si++)) { tl->set_tablenr(table_no); if (tl->is_jtbm()) { tl->jtbm_table_no= table_no; Item *dummy= tl->jtbm_subselect; tl->jtbm_subselect->fix_after_pullout(parent_lex, &dummy, true); ``` In that code one can see: if the subquery has table that has 'is_jtbm()==true', then ... some processing is done to re-compute its attributes after the table has been pulled up into the parent select. The same should be done for TABLE_LIST elements that are JSON_TABLE(..). If you'll introduce a new function, fix_after_pullout() is a good name. It is already used for Item-derived classes. > If not, i'll dig into it. > And you're welcome to observe the new patch. > > Best regards. > HF -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] MDEV-21829: Unique class: discussion followup part #2
by Sergey Petrunia 25 Sep '20

25 Sep '20

Hello Igor, Trying to capture the input from the last call. Currently class Unique is using RB-Tree for its in-memory storage. RB-Tree is implemented by struct TREE and tree_XXX() functions (further I'll refer to them both as just "TREE"). It is possible to get Unique to use some other data structure for its in-memory storage. The most obvious candidate is a hash table. (There are currently no plans to do this, though). So, your request was that - An interface is developed for in-memory storage (let's refer to it as Unique::MemStorage) - An implementation of this interface is developed that uses TREE. (We can't change the TREE itself as it is used in multple places). Is this really the intent? Please clarify. My reaction is that the above described change does not seem to have any relevance to MDEV-21829. MDEV-21829 is about supporting variable-length keys, not about changing, or preparing to change Unique::MemStorage. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

Re: [Maria-developers] 2564334b4c1: Backported setting of transcation.on=1 in THD::reset_for_reuse()
by Sergei Golubchik 25 Sep '20

25 Sep '20

Hi, Michael! > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 7327f270c33..8f6356b15c7 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1640,6 +1640,7 @@ void THD::reset_for_reuse() > abort_on_warning= 0; > free_connection_done= 0; > m_command= COM_CONNECT; > + transaction.on= 1; > #if defined(ENABLED_PROFILING) > profiling.reset(); > #endif This looks risky, to change transaction.on at some random point in time. I understand that it's not a random point and it should be actually safe here. But perhaps you can add an assert to document that it is safe? Like, transaction.on should be already 1 or there should be no active transaction. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] fc07306120f: MDEV-23586 Mariabackup: GTID saved for replication in 10.4.14 is wrong
by Sergei Golubchik 25 Sep '20

25 Sep '20

Hi, Monty! Just a couple of minor comments Note, the below is from the combined diff of this commit and your earlier MDEV-21953 commit, so you can see some lines here that are not part of the fc07306120f. > diff --git a/sql/handler.cc b/sql/handler.cc > index cc7eedd18f2..402db310ed6 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -133,6 +133,23 @@ static plugin_ref ha_default_tmp_plugin(THD *thd) > return ha_default_plugin(thd); > } > > +#if defined(WITH_ARIA_STORAGE_ENGINE) && MYSQL_VERSION_ID < 100500 > +void ha_maria_implicit_commit(THD *thd, bool new_trn) > +{ > + if (ha_maria::has_active_transaction(thd)) > + { > + int error; > + MDL_request mdl_request; > + mdl_request.init(MDL_key::BACKUP, "", "", MDL_BACKUP_COMMIT, MDL_EXPLICIT); > + error= thd->mdl_context.acquire_lock(&mdl_request, > + thd->variables.lock_wait_timeout); > + ha_maria::implicit_commit(thd, new_trn); > + if (!error) > + thd->mdl_context.release_lock(mdl_request.ticket); This looks rather fishy. You commit in aria unconditionally? whether you got the lock or not? > + } > +} > +#endif > + > > /** @brief > Return the default storage engine handlerton for thread > @@ -1753,13 +1777,14 @@ int ha_commit_one_phase(THD *thd, bool all) > if ((res= thd->wait_for_prior_commit())) > DBUG_RETURN(res); > } > - res= commit_one_phase_2(thd, all, trans, is_real_trans); > + res= commit_one_phase_2(thd, all, trans, is_real_trans, rw_trans); > DBUG_RETURN(res); > } > > > static int > -commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, bool is_real_trans) > +commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, bool is_real_trans, > + bool rw_trans) rw_trans seems to be unused here (also in ha_commit_one_phase()) > { > int error= 0; > uint count= 0; > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 40e606425c5..71b447fb920 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1383,7 +1384,11 @@ void THD::update_all_stats() > void THD::init_for_queries() > { > set_time(); > - ha_enable_transaction(this,TRUE); > + /* > + We don't need to call ha_enable_transaction() as we can't have > + any active transactions that has to be commited > + */ > + transaction.on= TRUE; I just commented on the similar code in another commit of yours. don't just say "can't have any active transactions", add an assert instead, please. > > reset_root_defaults(mem_root, variables.query_alloc_block_size, > variables.query_prealloc_size); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] New patch on the JSON_TABLE.
by Sergey Petrunia 23 Sep '20

23 Sep '20

One more thing: On Mon, Sep 14, 2020 at 03:38:38PM +0400, Alexey Botchkov wrote: > CREATE TABLE t1(id INT, f1 JSON); > INSERT INTO t1 VALUES (1, '{\"1\": 1}'), (2, '{\"1\": 2}'), (3, '{\"1\": > 3}'), > (4, '{\"1\": 4}'), (5, '{\"1\": 5}'), (6, '{\"1\": 6}'); > > SELECT * FROM t1 WHERE id IN > (SELECT id FROM t1 as tt2, > JSON_TABLE(f1, "$" COLUMNS (jf FOR ORDINALITY)) AS tbl); I've noticed that Optimizer Trace is not saved for this query. I get MariaDB [test]> select * from information_schema.optimizer_trace\G *************************** 1. row *************************** QUERY: TRACE: MISSING_BYTES_BEYOND_MAX_MEM_SIZE: 0 INSUFFICIENT_PRIVILEGES: 1 1 row in set (0.001 sec) INSUFFICIENT_PRIVILEGES: 1! This is only supposed to happen when the query uses VIEWs which definition cannot be viewed by the user executing the query. Could you investigate and fix this please? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] On bb-10.6-monty work
by Nikita Malyavin 16 Sep '20

16 Sep '20

Hello, Monty! I've been asked to review your recent work regarding memory footprint optimizations. I went through all the work done, and mostly liked the changes done. So here are only two commits i'd like to highlight > commit bf8bd0573440dc39490eb795b5455735dd63132b > Author: Monty <monty(a)mariadb.org> > Date: Tue Jul 14 18:36:05 2020 +0300 > > MDEV-23001 Precreate static Item_bool() to simplify code > > The following changes where done: > - Create global Item: Item_false and Item_true > - Replace all creation if 'FALSE' and 'TRUE' top level items used for > WHERE/HAVING/ON clauses to use Item_false and Item_true. > > The benefit are: > - Less and faster code > - No test needed if we where able to create the new item. > - Fixed possible errors if 'new' would have failed for the Item_bool's > > diff --git a/sql/item.h b/sql/item.h > index c4fbf8f9c0a..4cdee637415 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -4236,6 +4239,23 @@ class Item_bool :public Item_int > }; > > > +class Item_bool_static :public Item_bool > +{ > +public: > + Item_bool_static(const char *str_arg, longlong i): > + Item_bool(NULL, str_arg, i) {}; > + > + /* Create dummy functions for things that may change the static item */ > + void fix_length_and_charset(uint32 max_char_length_arg, CHARSET_INFO *cs) > + {} > + void fix_char_length(size_t max_char_length_arg) > + {} > + void set_join_tab_idx(uint join_tab_idx_arg) > + { DBUG_ASSERT(0); } > +}; > + > +extern const Item_bool_static Item_false, Item_true; > + > class Item_uint :public Item_int > { > public: Do we really need an additional class for static Item_bool's? I know that we have constructors hidden for Item, so maybe just unhide it in Item_bool, and that's it. Besides, I see fix_char_length is not virtual at all, so reimplementing it has no efect. And why can't we do all that in Item_bool itself? Note that adding new class increases a virtual method table, and it's huge already, for Item hierarchy. We could also add some other frequently used Items, like 0, 1, '', NULL, to a static memory. > commit 6ca3ab6c2c18cdbf614984cb9d5312d58e718372 > Author: Monty <monty(a)mariadb.org> > Date: Wed Sep 2 03:13:32 2020 +0300 > > Split item->flags into base_flags and with_flags > > This was done to simplify copying of with_* flags > > Other things: > - Changed Flags to C++ enums, which enables gdb to print > out bit values for the flags. This also enables compiler > errors if one tries to manipulate a non existing bit in > a variable. > - Added set_maybe_null() as a shortcut as setting the > MAYBE_NULL flags was used in a LOT of places. > - Renamed PARAM flag to SP_VAR to ensure it's not confused with persistent > statement parameters. > > diff --git a/sql/item.h b/sql/item.h > index 09618bbb52c..9a794d96982 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -717,28 +717,94 @@ class Item_const > > #define STOP_PTR ((void *) 1) > > -/* Bits used in Item.flags */ > -/* If item may be null */ > -#define ITEM_FLAG_MAYBE_NULL_SHIFT 0 > -#define ITEM_FLAG_MAYBE_NULL (1<<ITEM_FLAG_MAYBE_NULL_SHIFT) > -/* If used in GROUP BY list of a query with ROLLUP */ > -#define ITEM_FLAG_IN_ROLLUP (1<<1) > -/* If Item contains an SP parameter */ > -#define ITEM_FLAG_WITH_PARAM (1<<2) > -/* If item contains a window func */ > -#define ITEM_FLAG_WITH_WINDOW_FUNC (1<<3) > -/* True if any item except Item_sum contains a field. Set during parsing. */ > -#define ITEM_FLAG_WITH_FIELD (1<<4) > -/* If item was fixed with fix_fields */ > -#define ITEM_FLAG_FIXED (1<<5) > -/* Indicates that name of this Item autogenerated or set by user */ > -#define ITEM_FLAG_IS_AUTOGENERATED_NAME (1 << 6) > -/* Indicates that this item is in CYCLE clause of WITH */ > -#define ITEM_FLAG_IS_IN_WITH_CYCLE (1<<7) > -/* True if item contains a sum func */ > -#define ITEM_FLAG_WITH_SUM_FUNC (1<< 8) > -/* True if item containts a sub query */ > -#define ITEM_FLAG_WITH_SUBQUERY (1<< 9) > + > +/* Base flags (including IN) for an item */ > + > + > +typedef uint8 item_flags_t; > + > +enum class item_base_t : item_flags_t > +{ > + NONE= 0, > +#define ITEM_FLAGS_MAYBE_NULL_SHIFT 0 // Must match MAYBE_NULL > + MAYBE_NULL= (1<<0), // May be NULL. > + IN_ROLLUP= (1<<1), // Appears in GROUP BY list > + // of a query with ROLLUP. > + FIXED= (1<<2), // Was fixed with fix_fields(). > + IS_AUTOGENERATED_NAME= (1<<3), // The name if this Item was > + // generated or set by the user. > + IS_IN_WITH_CYCLE= (1<<4) // This item is in CYCLE clause > + // of WITH. > +}; > + > + > +/* Flags that tells us what kind of items the item contains */ > + > +enum class item_with_t : item_flags_t > +{ > + NONE= 0, > + SP_VAR= (1<<0), // If Item contains a stored procedure variable > + WINDOW_FUNC= (1<<1), // If item contains a window func > + FIELD= (1<<2), // If any item except Item_sum contains a field. > + SUM_FUNC= (1<<3), // If item contains a sum func > + SUBQUERY= (1<<4) // If item containts a sub query > +}; > + > + > +/* Make operations in item_base_t and item_with_t work like 'int' */ > +static inline item_base_t operator&(const item_base_t a, const item_base_t b) > +{ > + return (item_base_t) (((item_flags_t) a) & ((item_flags_t) b)); > +} > + > +static inline item_base_t & operator&=(item_base_t &a, item_base_t b) > +{ > + a= (item_base_t) (((item_flags_t) a) & (item_flags_t) b); > + return a; > +} > + > +static inline item_base_t operator|(const item_base_t a, const item_base_t b) > +{ > + return (item_base_t) (((item_flags_t) a) | ((item_flags_t) b)); > +} > + > +static inline item_base_t & operator|=(item_base_t &a, item_base_t b) > +{ > + a= (item_base_t) (((item_flags_t) a) | (item_flags_t) b); > + return a; > +} > + > +static inline item_base_t operator~(const item_base_t a) > +{ > + return (item_base_t) ~(item_flags_t) a; > +} > + > +static inline item_with_t operator&(const item_with_t a, const item_with_t b) > +{ > + return (item_with_t) (((item_flags_t) a) & ((item_flags_t) b)); > +} > + > +static inline item_with_t & operator&=(item_with_t &a, item_with_t b) > +{ > + a= (item_with_t) (((item_flags_t) a) & (item_flags_t) b); > + return a; > +} > + > +static inline item_with_t operator|(const item_with_t a, const item_with_t b) > +{ > + return (item_with_t) (((item_flags_t) a) | ((item_flags_t) b)); > +} > + > +static inline item_with_t & operator|=(item_with_t &a, item_with_t b) > +{ > + a= (item_with_t) (((item_flags_t) a) | (item_flags_t) b); > + return a; > +} > + > +static inline item_with_t operator~(const item_with_t a) > +{ > + return (item_with_t) ~(item_flags_t) a; > +} > > > class Item :public Value_source, > @@ -932,8 +998,8 @@ class Item :public Value_source, > const char *orig_name; > > /* All common bool variables for an Item is stored here */ > - typedef uint16 item_flags_t; > - item_flags_t flags; > + item_base_t base_flags; > + item_with_t with_flags; > > /* Marker is used in some functions to temporary mark an item */ > int16 marker; Nice experimentation! I wanted to suggest using constexprs inside Item, but this is even better, because you cant freely add a bit that not in the enum. Though I think it is too bloaty to specify all the operators each time, so I'd prefer to see duplications metaprogrammed out somehow. There is no sane way in C++ (even in modern one) to specify traits, which extend the class functionality (see Golang's traits), so I'd suggest to use good old preprocessor way, like following: #define T item_base_t #include "setup_bitwise_operators.inc" #define T item_with_t #include "setup_bitwise_operators.inc" and then in setup_bitwise_operators.inc: static inline T operator&(const T a, const T b) { return (T) (((item_flags_t) a) & ((item_flags_t) b)); } And so on. Another variant I've just googled is to make a template operator for all types, but restrict its applicatory type set with std::enable_if<T>: https://www.justsoftwaresolutions.co.uk/cplusplus/using-enum-classes-as-bit… So we would only need to add a structure: template<> struct enable_bitmask_operators<item_base_t>{ static constexpr bool enable=true; }; To work. Thereby, the first approach is: + Good old way + Works fast - not a C++ way, though we work in a C++-only file The second approach is: + modern + C++ way - may compile slow - may produce uncomfortable errors when compiler tries to deduce a function/operator to apply Up to you whic way to use, but I think it should be done, since we have many other flag sets, which are likely to be prettified with the same style. -- Yours truly, Nikita Malyavin

2 3

[Maria-developers] MDEV-21829:Followup: On Unique object and variable-size keys
by Sergey Petrunia 15 Sep '20

15 Sep '20

Hi Varun, I was looking at the code for MDEV-21829, trying to understand whether there are any issues with how Unique object was extended to handle variable-size keys. == TREE object (the RB tree) == TREE (the structure and functions that operate on it) support variable-sized keys, however the keys "are expected to know" their size. tree_insert() has "uint key_size" argument. This way, tree_insert() knows how many bytes to allocate and copy when putting the key into the tree. The rest of the TREE code doesn't seem to care about the size of the keys. When TREE code needs to compare a couple of keys, it will call TREE::compare callack. The callback only receives key pointers as arguments, that is, it is expected to infer key size from the contents of the key. The same happens when when walking the tree. Only "uchar *key" is passed to tree_walk_action callback. The walk function has to figure out they key size on its own. == Unique == Unique class cannot do the same what TREE functions function do. It needs to be aware of the sizes of the values it is storing: - It will need to pass value size to tree_insert() - It will need to know value size to write it into tmp file. - It will need to read it back when merging. How does it know it? Unique::merge() and Unique::walk() with merge_walk() functions do make certain assumptions about the data format being used. == The review points == Is this bad? I think tight coupling between the Unique object and the specifics of the data format it is storing is bad. What is worse is that currently this coupling is implicit, it is not immediately apparent to somebody just looking at the code. Possible ways out: 1. Explicitly declare in Unique class that variable-sized records must use certain data format. 2. Isolate the Unique from the specifics of the data format used. 2A. Let Unique get the size of the key as argument of unique_add() and then keep it together with the key. This creates some RAM/storage overhead. 2B. Pass a read_size() callback function which would get the size from the data? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] 099596296ce: Avoid mallocs when using LONGLONG_BUFFER_SIZE
by Sergei Golubchik 13 Sep '20

13 Sep '20

Hi, Michael! This is a good catch! But note that LONGLONG_BUFFER_SIZE already reserves +1 for '\0', it's defined as /* Max length needed for a buffer to hold a longlong or ulonglong + end \0 */ #define LONGLONG_BUFFER_SIZE 21 The problem is that String::set_int doesn't use LONGLONG_BUFFER_SIZE, it uses 20*cs->mbmaxlen+1 This +1 is supposed to reserve one byte for '\0'. Which is incorrect, because alloc() does it too, so a better fix would be to remove +1 from String::set_int(). On Sep 08, Michael Widenius wrote: > revision-id: 099596296ce (mariadb-10.5.2-272-g099596296ce) > parent(s): 2de0c74f94c > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-03 15:17:56 +0300 > message: > > Avoid mallocs when using LONGLONG_BUFFER_SIZE > > When using String::alloc() to allocate a string, the String ensures that > there is space for an extra NULL byte in the buffer and if not, reallocates > the string. This is a problem with the String::set_int() that calls > alloc(LONGLONG_BUFFER_SIZE), which forces an extra malloc/free to > happen. > > Fixed by using LONGLONG_BUFFER_SIZE+1 for StringBuffer's. > > diff --git a/sql/field.cc b/sql/field.cc > index ab7254e483d..2be4da8d4a8 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -1085,7 +1085,7 @@ Field_longstr::make_packed_sort_key_part(uchar *buff, > uchar* > Field_longstr::pack_sort_string(uchar *to, const SORT_FIELD_ATTR *sort_field) > { > - StringBuffer<LONGLONG_BUFFER_SIZE> buf; > + StringBuffer<LONGLONG_BUFFER_SIZE+1> buf; > val_str(&buf, &buf); > return to + sort_field->pack_sort_string(to, &buf, field_charset()); > } > diff --git a/sql/item.cc b/sql/item.cc > index 4a26cb6c140..0d947749e01 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -3612,7 +3612,7 @@ String *Item_int::val_str(String *str) > > void Item_int::print(String *str, enum_query_type query_type) > { > - StringBuffer<LONGLONG_BUFFER_SIZE> buf; > + StringBuffer<LONGLONG_BUFFER_SIZE+1> buf; > // my_charset_bin is good enough for numbers > buf.set_int(value, unsigned_flag, &my_charset_bin); > str->append(buf); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 23814b75241: Report memory leaks from mariadbd if -T or --debug is used
by Sergei Golubchik 11 Sep '20

11 Sep '20

Hi, Michael! ok to push, of course On Sep 08, Michael Widenius wrote: > revision-id: 23814b75241 (mariadb-10.5.2-273-g23814b75241) > parent(s): 099596296ce > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-03 15:17:56 +0300 > message: > > Report memory leaks from mariadbd if -T or --debug is used > > Before memory leaks was only reported if server stopped normally. > This made it harder to find out where the leaks happened when > debugging test cases. > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 331684ff728..7468f436a03 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -1939,7 +1939,7 @@ static void mysqld_exit(int exit_code) > { > fprintf(stderr, "Warning: Memory not freed: %lld\n", > (longlong) global_status_var.global_memory_used); > - if (exit_code == 0) > + if (exit_code == 0 || opt_endinfo) > SAFEMALLOC_REPORT_MEMORY(0); > } > DBUG_LEAVE; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2de0c74f94c: Optimize Sql_alloc
by Sergei Golubchik 11 Sep '20

11 Sep '20

Hi, Michael! ok to push On Sep 08, Michael Widenius wrote: > revision-id: 2de0c74f94c (mariadb-10.5.2-271-g2de0c74f94c) > parent(s): 21eb8969ce9 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-03 15:17:56 +0300 > message: > > Optimize Sql_alloc > > - Remove 'dummy_for_valgrind' overrun marker as this doesn't help much. > The element also distorts the sizes of objects a bit, which makes it > harder to calculate gain in object sizes when doing size optimizations. > - Avoid one extra call indirection when using thd_get_current_thd(), which > is used by Sql_alloc. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c3ecf0d6243: Optimize size of lex structures
by Sergei Golubchik 09 Sep '20

09 Sep '20

Hi, Michael! On Sep 08, Michael Widenius wrote: > revision-id: c3ecf0d6243 (mariadb-10.5.2-269-gc3ecf0d6243) > parent(s): c79c6f78205 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-02 20:58:33 +0300 > message: > > Optimize size of lex structures > > LEX, st_select_lex, st_select_unit optimized for space: > - Use bit fields for bool variables > - Ensure that all bit fields are initialized (improves > performance for init functions as all bit fields can be > initalized with one memory access) > - Move members around in above structures to remove alignment > gaps > > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index f516219c01a..1f6e50d8b60 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -725,12 +725,15 @@ class st_select_lex_node { > st_select_lex_node *next, **prev, /* neighbor list */ > *master, *slave, /* vertical links */ > *link_next, **link_prev; /* list of whole SELECT_LEX */ > + enum sub_select_type linkage; > > void init_query_common(); > -public: > > +public: > ulonglong options; > - > + uint8 uncacheable; > + bool distinct:1; > + bool no_table_names_allowed:1; /* used for global order by */ I thought you didn't like this approach and preferred flags & DISTINCT instead. Either you forgot to change LEX to flags or you forgot to squash your commits. > /* > result of this query can't be cached, bit field, can be : Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c79c6f78205: Added typedef decimal_digits_t (uint16) for number of decimals
by Sergei Golubchik 09 Sep '20

09 Sep '20

Hi, Michael! On Sep 08, Michael Widenius wrote: > revision-id: c79c6f78205 (mariadb-10.5.2-268-gc79c6f78205) > parent(s): bf8bd057344 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-02 20:58:33 +0300 > message: > > Added typedef decimal_digits_t (uint16) for number of decimals > > For fields and Item's uint8 should be good enough. After > discussions with Alexander Barkov we choose uint16 (for now) > as some format functions may accept +256 digits. > > The reason for this patch was to make the storage of decimal > digits simlar. Before this patch decimals was stored/used as > uint8, int and uint. > > Changed most decimal variables and functions to use the new > typedef. > > diff --git a/include/decimal.h b/include/decimal.h > index cab18f99348..f713409ede1 100644 > --- a/include/decimal.h > +++ b/include/decimal.h > @@ -53,10 +53,11 @@ int decimal2double(const decimal_t *from, double *to); > int double2decimal(double from, decimal_t *to); > int decimal_actual_fraction(const decimal_t *from); > int decimal2bin(const decimal_t *from, uchar *to, int precision, int scale); > -int bin2decimal(const uchar *from, decimal_t *to, int precision, int scale); > +int bin2decimal(const uchar *from, decimal_t *to, int precision, > + decimal_digits_t scale); I don't understand. Is it the type for precision (max number of digits in a decimal number) or for scale (mux number of digits after the dot)? If the former - why did you not change the type of the precision argument? If the latter - why is it called not decimal_scale_t? I personally think the former, using decimal_digits_t for both precision and scale, should be fine. > -int decimal_size(int precision, int scale); > -int decimal_bin_size(int precision, int scale); > +int decimal_size(int precision, decimal_digits_t scale); > +int decimal_bin_size(int precision, decimal_digits_t scale); > int decimal_result_size(decimal_t *from1, decimal_t *from2, char op, > int param); > > diff --git a/include/m_ctype.h b/include/m_ctype.h > index 59ac7814aee..34821f76474 100644 > --- a/include/m_ctype.h > +++ b/include/m_ctype.h > @@ -79,6 +79,7 @@ typedef const struct my_collation_handler_st MY_COLLATION_HANDLER; > typedef const struct unicase_info_st MY_UNICASE_INFO; > typedef const struct uni_ctype_st MY_UNI_CTYPE; > typedef const struct my_uni_idx_st MY_UNI_IDX; > +typedef uint16 decimal_digits_t; A bit strange to see it here and not in decimal.h above. > > typedef struct unicase_info_char_st > { > diff --git a/plugin/type_inet/sql_type_inet.h b/plugin/type_inet/sql_type_inet.h > index 8c42431ccaa..df74b91172a 100644 > --- a/plugin/type_inet/sql_type_inet.h > +++ b/plugin/type_inet/sql_type_inet.h > @@ -402,19 +402,19 @@ class Type_handler_inet6: public Type_handler > bool can_return_time() const override { return false; } > bool convert_to_binary_using_val_native() const override { return true; } > > - uint Item_time_precision(THD *thd, Item *item) const override > + decimal_digits_t Item_time_precision(THD *thd, Item *item) const override here you return the precision using the decimal_digits_t type. So it seems that it should apply to both precision and scale. > { > return 0; > } > - uint Item_datetime_precision(THD *thd, Item *item) const override > +decimal_digits_t Item_datetime_precision(THD *thd, Item *item) const override > { > return 0; > } > - uint Item_decimal_scale(const Item *item) const override > + decimal_digits_t Item_decimal_scale(const Item *item) const override > { > return 0; > } > - uint Item_decimal_precision(const Item *item) const override > + decimal_digits_t Item_decimal_precision(const Item *item) const override > { > /* > This will be needed if we ever allow cast from INET6 to DECIMAL. > diff --git a/sql/field.cc b/sql/field.cc > index f50ddec1c80..a5d00dd6fd5 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -3242,10 +3242,11 @@ Field *Field_decimal::make_new_field(MEM_ROOT *root, TABLE *new_table, > ** Field_new_decimal > ****************************************************************************/ > > -static uint get_decimal_precision(uint len, uint8 dec, bool unsigned_val) > +static decimal_digits_t get_decimal_precision(uint len, decimal_digits_t dec, > + bool unsigned_val) > { > uint precision= my_decimal_length_to_precision(len, dec, unsigned_val); shouldn't my_decimal_length_to_precision() return decimal_digits_t result? > - return MY_MIN(precision, DECIMAL_MAX_PRECISION); > + return (decimal_digits_t) MY_MIN(precision, DECIMAL_MAX_PRECISION); > } > > Field_new_decimal::Field_new_decimal(uchar *ptr_arg, > diff --git a/sql/item.h b/sql/item.h > index 4cdee637415..948bb85f253 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1644,14 +1644,14 @@ class Item: public Value_source, > return type_handler()->Item_decimal_precision(this); > } > /* Returns the number of integer part digits only */ > - inline int decimal_int_part() const > - { return my_decimal_int_part(decimal_precision(), decimals); } > + inline decimal_digits_t decimal_int_part() const > + { return (decimal_digits_t) my_decimal_int_part(decimal_precision(), decimals); } again, shouldn't my_decimal_int_part() return decimal_digits_t? > /* > Returns the number of fractional digits only. > NOT_FIXED_DEC is replaced to the maximum possible number > of fractional digits, taking into account the data type. > */ > - uint decimal_scale() const > + decimal_digits_t decimal_scale() const > { > return type_handler()->Item_decimal_scale(this); > } > @@ -4856,7 +4856,7 @@ class Item_temporal_literal :public Item_literal > Item_literal(thd) > { > collation= DTCollation_numeric(); > - decimals= dec_arg; > + decimals= (decimal_digits_t) dec_arg; shouldnt dec_arg be decimal_digits_t? > cached_time= *ltime; > } > > diff --git a/sql/item_func.cc b/sql/item_func.cc > index 8a75d2c9946..805647d2a04 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -2713,7 +2713,7 @@ my_decimal *Item_func_round::decimal_op(my_decimal *decimal_value) > dec= INT_MIN; > > if (!(null_value= (value.is_null() || args[1]->null_value || > - value.round_to(decimal_value, (uint) dec, > + value.round_to(decimal_value, (int) dec, You don't need to cast uint16 to int explicitly > truncate ? TRUNCATE : HALF_UP) > 1))) > return decimal_value; > return 0; > diff --git a/sql/item_func.h b/sql/item_func.h > index 5b4acdce1c6..d5aa1477ff3 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -1369,10 +1369,10 @@ class Item_decimal_typecast :public Item_func > { > my_decimal decimal_value; > public: > - Item_decimal_typecast(THD *thd, Item *a, uint len, uint dec) > + Item_decimal_typecast(THD *thd, Item *a, uint len, decimal_digits_t dec) > :Item_func(thd, a) > { > - decimals= (uint8) dec; > + decimals= (decimal_digits_t) dec; you don't need a cast here > collation= DTCollation_numeric(); > fix_char_length(my_decimal_precision_to_length_no_truncation(len, dec, > unsigned_flag)); > diff --git a/sql/my_decimal.cc b/sql/my_decimal.cc > index edf3e82de40..ac86ff71b64 100644 > --- a/sql/my_decimal.cc > +++ b/sql/my_decimal.cc > @@ -198,7 +198,8 @@ str_set_decimal(uint mask, const my_decimal *val, > E_DEC_OVERFLOW > */ > > -int my_decimal::to_binary(uchar *bin, int prec, int scale, uint mask) const > +int my_decimal::to_binary(uchar *bin, int prec, decimal_digits_t scale, again, shouldn't prec be decimal_digits_t too? > + uint mask) const > { > int err1= E_DEC_OK, err2; > my_decimal rounded; > @@ -329,7 +330,7 @@ my_decimal *date2my_decimal(const MYSQL_TIME *ltime, my_decimal *dec) > } > > > -void my_decimal_trim(ulonglong *precision, uint *scale) > +void my_decimal_trim(ulonglong *precision, decimal_digits_t *scale) and here? > { > if (!(*precision) && !(*scale)) > { > diff --git a/sql/my_decimal.h b/sql/my_decimal.h > index 2db08bf01e3..9f7232f94b4 100644 > --- a/sql/my_decimal.h > +++ b/sql/my_decimal.h > @@ -50,7 +50,7 @@ typedef struct st_mysql_time MYSQL_TIME; > #define DECIMAL_MAX_FIELD_SIZE DECIMAL_MAX_PRECISION > > > -inline uint my_decimal_size(uint precision, uint scale) > +inline uint my_decimal_size(uint precision, decimal_digits_t scale) and here? (and in many places below, that I'll stop commenting on) > { > /* > Always allocate more space to allow library to put decimal point > diff --git a/sql/sql_type.cc b/sql/sql_type.cc > index 9e0f9b013c0..6569ea1a415 100644 > --- a/sql/sql_type.cc > +++ b/sql/sql_type.cc > @@ -1197,11 +1198,12 @@ Type_numeric_attributes::aggregate_numeric_attributes_decimal(Item **item, > uint nitems, > bool unsigned_arg) > { > - int max_int_part= find_max_decimal_int_part(item, nitems); > + decimal_digits_t max_int_part= find_max_decimal_int_part(item, nitems); > decimals= find_max_decimals(item, nitems); > - int precision= MY_MIN(max_int_part + decimals, DECIMAL_MAX_PRECISION); > + decimal_digits_t precision= (decimal_digits_t) why do you need a cast here? > + MY_MIN(max_int_part + decimals, DECIMAL_MAX_PRECISION); > max_length= my_decimal_precision_to_length_no_truncation(precision, > - (uint8) decimals, > + decimals, > unsigned_flag); > } > > @@ -6910,20 +6912,20 @@ bool Type_handler_string_result:: > > /***************************************************************************/ > > -uint Type_handler::Item_time_precision(THD *thd, Item *item) const > +decimal_digits_t Type_handler::Item_time_precision(THD *thd, Item *item) const > { > return MY_MIN(item->decimals, TIME_SECOND_PART_DIGITS); note, no cast here (good) > } > > > -uint Type_handler::Item_datetime_precision(THD *thd, Item *item) const > +decimal_digits_t Type_handler::Item_datetime_precision(THD *thd, Item *item) const > { > return MY_MIN(item->decimals, TIME_SECOND_PART_DIGITS); and not here > } > > > -uint Type_handler_string_result::Item_temporal_precision(THD *thd, Item *item, > - bool is_time) const > +decimal_digits_t Type_handler_string_result:: > +Item_temporal_precision(THD *thd, Item *item, bool is_time) const > { > StringBuffer<64> buf; > String *tmp; > @@ -6939,34 +6941,34 @@ uint Type_handler_string_result::Item_temporal_precision(THD *thd, Item *item, > Datetime(thd, &status, tmp->ptr(), tmp->length(), tmp->charset(), > Datetime::Options(TIME_FUZZY_DATES, TIME_FRAC_TRUNCATE)). > is_valid_datetime())) > - return MY_MIN(status.precision, TIME_SECOND_PART_DIGITS); > - return MY_MIN(item->decimals, TIME_SECOND_PART_DIGITS); > + return (decimal_digits_t) MY_MIN(status.precision, TIME_SECOND_PART_DIGITS); > + return (decimal_digits_t) MY_MIN(item->decimals, TIME_SECOND_PART_DIGITS); and here again. why? (and below too, I'll stop commenting on casts now) > } > > diff --git a/sql/sql_type.h b/sql/sql_type.h > index a6d85b5bb47..7b1a4f730db 100644 > --- a/sql/sql_type.h > +++ b/sql/sql_type.h > @@ -2955,9 +2955,10 @@ class Type_numeric_attributes > class Type_temporal_attributes: public Type_numeric_attributes > { > public: > - Type_temporal_attributes(uint int_part_length, uint dec, bool unsigned_arg) > + Type_temporal_attributes(uint32 int_part_length, decimal_digits_t dec, bool unsigned_arg) > :Type_numeric_attributes(int_part_length + (dec ? 1 : 0), > - MY_MIN(dec, TIME_SECOND_PART_DIGITS), > + MY_MIN(dec, > + (decimal_digits_t) TIME_SECOND_PART_DIGITS), this should not be necessary > unsigned_arg) > { > max_length+= decimals; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] bf8bd057344: MDEV-23001 Precreate static Item_bool() to simplify code
by Sergei Golubchik 08 Sep '20

08 Sep '20

Hi, Michael! On Sep 08, Michael Widenius wrote: > revision-id: bf8bd057344 (mariadb-10.5.2-267-gbf8bd057344) > parent(s): 32a29afea77 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-02 20:58:32 +0300 > message: > > MDEV-23001 Precreate static Item_bool() to simplify code > > The following changes where done: > - Create global Item: Item_false and Item_true > - Replace all creation if 'FALSE' and 'TRUE' top level items used for > WHERE/HAVING/ON clauses to use Item_false and Item_true. > > The benefit are: > - Less and faster code > - No test needed if we where able to create the new item. > - Fixed possible errors if 'new' would have failed for the Item_bool's > > diff --git a/sql/item.cc b/sql/item.cc > index dbf20f31d7a..b24103f2330 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -55,7 +55,8 @@ const char *item_empty_name=""; > const char *item_used_name= "\0"; > > static int save_field_in_field(Field *, bool *, Field *, bool); > - > +const Item_bool_static Item_false("FALSE", 0); > +const Item_bool_static Item_true("TRUE", 1); The maain question, that Bar asked in his email - we have tons of modifiable Item members. How do you know none of them will modified in your Item_bool_static objects? > /** > Compare two Items for List<Item>::add_unique() > @@ -412,7 +413,6 @@ Item::Item(THD *thd): > is_expensive_cache(-1), rsize(0), name(null_clex_str), orig_name(0), > common_flags(IS_AUTO_GENERATED_NAME) > { > - DBUG_ASSERT(thd); > marker= 0; > maybe_null= null_value= with_window_func= with_field= false; > in_rollup= 0; > @@ -421,6 +421,9 @@ Item::Item(THD *thd): > /* Initially this item is not attached to any JOIN_TAB. */ > join_tab_idx= MAX_TABLES; > > + /* thd is NULL in case of static items like Item_true */ > + if (!thd) > + return; I don't like that. thd can be NULL only twice, for statically initialized Item_true and Item_false. For that you removed a useful assert and added an if() that is completely unnecessary for billions of items created runtime. Better to have special contructor just for these two very special items. Or to create a THD temporarily when constructing them. > /* Put item in free list so that we can free all items at end */ > next= thd->free_list; > thd->free_list= this; > @@ -3614,6 +3618,19 @@ void Item_int::print(String *str, enum_query_type query_type) > } > > > +/* > + This function is needed to ensure that Item_bool_static doesn't change > + the value of the member str_value. > +*/ > + > +void Item_bool::print(String *str, enum_query_type query_type) > +{ > + // my_charset_bin is good enough for numbers > + String tmp(value ? (char*) "1" : (char*) "0" , 1, &my_charset_bin); > + str->append(tmp); > +} This is not needed anymore. Item_bool inherits from Item_int, and Item_int::print is void Item_int::print(String *str, enum_query_type query_type) { StringBuffer<LONGLONG_BUFFER_SIZE> buf; // my_charset_bin is good enough for numbers buf.set_int(value, unsigned_flag, &my_charset_bin); str->append(buf); } You removed str_value from Item_int::print in July. > + > + > Item *Item_bool::neg_transformer(THD *thd) > { > value= !value; > diff --git a/sql/item.h b/sql/item.h > index c4fbf8f9c0a..4cdee637415 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1969,7 +1969,7 @@ class Item: public Value_source, > virtual bool limit_index_condition_pushdown_processor(void *arg) { return 0; } > virtual bool exists2in_processor(void *arg) { return 0; } > virtual bool find_selective_predicates_list_processor(void *arg) { return 0; } > - bool cleanup_is_expensive_cache_processor(void *arg) > + virtual bool cleanup_is_expensive_cache_processor(void *arg) You've added a new virtual method? You tried so hard to remove virtual methods that it's a bit strange to see a new virtual method added specifically for these two static items. Why does it matter what cleanup_is_expensive_cache_processor will do with them? You overwrite is_expensive() anyway. > { > is_expensive_cache= (int8)(-1); > return 0; > @@ -3225,6 +3225,8 @@ class Item_literal: public Item_basic_constant > bool check_partition_func_processor(void *int_arg) { return false;} > bool const_item() const { return true; } > bool basic_const_item() const { return true; } > + bool is_expensive() { return false; } > + bool cleanup_is_expensive_cache_processor(void *arg) { return 0; } > }; > > diff --git a/sql/opt_index_cond_pushdown.cc b/sql/opt_index_cond_pushdown.cc > index 360ae028f36..d66f2f2e14e 100644 > --- a/sql/opt_index_cond_pushdown.cc > +++ b/sql/opt_index_cond_pushdown.cc > @@ -185,8 +185,8 @@ bool uses_index_fields_only(Item *item, TABLE *tbl, uint keyno, > static Item *make_cond_for_index(THD *thd, Item *cond, TABLE *table, uint keyno, > bool other_tbls_ok) > { > - if (!cond) > - return NULL; > + if (!cond || cond->basic_const_item()) > + return cond; The effect of this change is that you don't set cond->marker= ICP_COND_USES_INDEX_ONLY for basic_const_item's anymore. This looked suspicious at first. But then I noticed (and Sergey Petrunia confirmed) that ICP_COND_USES_INDEX_ONLY is not used at all. Since 5.3. So here I'd suggest to revert this your change and completely remove any ICP_COND_USES_INDEX_ONLY manipulations together with the ICP_COND_USES_INDEX_ONLY definition. > if (cond->type() == Item::COND_ITEM) > { > uint n_marked= 0; > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 144b86e8fc9..1140f32c9c9 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2313,6 +2313,7 @@ bool dispatch_command(enum enum_server_command command, THD *thd, > thd->update_server_status(); > thd->protocol->end_statement(); > query_cache_end_of_result(thd); > + /* Check that the query didn't change const items */ what does that mean? Is it a TODO comment? Or it documents what happens below? > } > if (drop_more_results) > thd->server_status&= ~SERVER_MORE_RESULTS_EXISTS; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 22bf2eb37b5..fdd609a96ce 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -22560,6 +22560,8 @@ make_cond_for_table_from_pred(THD *thd, Item *root_cond, Item *cond, > return new_cond; > } > } > + else if (cond->basic_const_item()) > + return cond; this is rather fragile, isn't it? You skip item->set_join_tab_idx(join_tab_idx_arg), quite understandably. And currently, as far as I can see, item->get_join_tab_idx() is always used under if (!item->const_item()), so what you did is safe. But can it change in the future? Or is join_tab_idx something that conceptually doesn't make sense for const items? > > if (is_top_and_level && used_table == rand_table_bit && > (cond->used_tables() & ~OUTER_REF_TABLE_BIT) != rand_table_bit) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] 5d9386892b5: MDEV-4633: multi_source.simple test fails sporadically
by sujatha 08 Sep '20

08 Sep '20

revision-id: 5d9386892b58477c4e28de3e40d930b44f5b42d4 (mariadb-10.1.43-274-g5d9386892b5) parent(s): 3ee2422624ffb3d7ffefff8db7ef9398816299bc author: Sujatha committer: Sujatha timestamp: 2020-09-08 13:10:36 +0530 message: MDEV-4633: multi_source.simple test fails sporadically Analysis: ======== Writes to 'rli->log_space_total' needs to be synchronized, otherwise both SQL_THREAD and IO_THREAD can try to modify the variable simultaneously resulting in incorrect rli->log_space_total. In the current test scenario SQL_THREAD is trying to decrement 'rli->log_space_total' in 'purge_first_log' and IO_THREAD is trying to increment the 'rli->log_space_total' in 'queue_event' simultaneously. Hence test occasionally fails with result mismatch. Fix: === Convert 'rli->log_space_total' variable to atomic type. --- sql/log.cc | 4 +++- sql/log.h | 4 +++- sql/rpl_rli.cc | 14 ++++++++------ sql/slave.cc | 19 +++++++++++++++---- 4 files changed, 29 insertions(+), 12 deletions(-) diff --git a/sql/log.cc b/sql/log.cc index 8049b94bab1..90d5c46a15f 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -4400,7 +4400,9 @@ int MYSQL_BIN_LOG::purge_first_log(Relay_log_info* rli, bool included) 0, 0, &log_space_reclaimed); mysql_mutex_lock(&rli->log_space_lock); - rli->log_space_total-= log_space_reclaimed; + my_atomic_add64_explicit((volatile int64*)(&rli->log_space_total), + (-(ulonglong)log_space_reclaimed), + MY_MEMORY_ORDER_RELAXED); mysql_cond_broadcast(&rli->log_space_cond); mysql_mutex_unlock(&rli->log_space_lock); diff --git a/sql/log.h b/sql/log.h index 277e5c6f69c..9e3807e9939 100644 --- a/sql/log.h +++ b/sql/log.h @@ -712,7 +712,9 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG char buf1[22],buf2[22]; #endif DBUG_ENTER("harvest_bytes_written"); - (*counter)+=bytes_written; + + my_atomic_add64_explicit((volatile int64*)(counter), bytes_written, + MY_MEMORY_ORDER_RELAXED); DBUG_PRINT("info",("counter: %s bytes_written: %s", llstr(*counter,buf1), llstr(bytes_written,buf2))); bytes_written=0; diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc index c196a65809a..8815af4ea4a 100644 --- a/sql/rpl_rli.cc +++ b/sql/rpl_rli.cc @@ -140,8 +140,8 @@ int init_relay_log_info(Relay_log_info* rli, rli->slave_skip_counter=0; rli->abort_pos_wait=0; rli->log_space_limit= relay_log_space_limit; - rli->log_space_total= 0; - + my_atomic_store64_explicit((volatile int64*)(&rli->log_space_total), 0, + MY_MEMORY_ORDER_RELAXED); char pattern[FN_REFLEN]; (void) my_realpath(pattern, slave_load_tmpdir, 0); if (fn_format(pattern, PREFIX_SQL_LOAD, pattern, "", @@ -467,7 +467,8 @@ static inline int add_relay_log(Relay_log_info* rli,LOG_INFO* linfo) linfo->log_file_name); DBUG_RETURN(1); } - rli->log_space_total += s.st_size; + my_atomic_add64_explicit((volatile int64*)(&rli->log_space_total), + s.st_size, MY_MEMORY_ORDER_RELAXED); DBUG_PRINT("info",("log_space_total: %llu", rli->log_space_total)); DBUG_RETURN(0); } @@ -477,7 +478,8 @@ static int count_relay_log_space(Relay_log_info* rli) { LOG_INFO linfo; DBUG_ENTER("count_relay_log_space"); - rli->log_space_total= 0; + my_atomic_store64_explicit((volatile int64*)(&rli->log_space_total), 0, + MY_MEMORY_ORDER_RELAXED); if (rli->relay_log.find_log_pos(&linfo, NullS, 1)) { sql_print_error("Could not find first log while counting relay log space"); @@ -1202,8 +1204,8 @@ int purge_relay_logs(Relay_log_info* rli, THD *thd, bool just_reset, strmake_buf(rli->group_relay_log_name, rli->relay_log.get_log_fname()); strmake_buf(rli->event_relay_log_name, rli->relay_log.get_log_fname()); rli->group_relay_log_pos= rli->event_relay_log_pos= BIN_LOG_HEADER_SIZE; - rli->log_space_total= 0; - + my_atomic_store64_explicit((volatile int64*)(&rli->log_space_total), 0, + MY_MEMORY_ORDER_RELAXED); if (count_relay_log_space(rli)) { *errmsg= "Error counting relay log space"; diff --git a/sql/slave.cc b/sql/slave.cc index 06f2b0d955a..9f4b0f53fb4 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -2371,7 +2371,10 @@ static bool wait_for_relay_log_space(Relay_log_info* rli) &rli->log_space_lock, &stage_waiting_for_relay_log_space, &old_stage); - while (rli->log_space_limit < rli->log_space_total && + while (rli->log_space_limit < + (ulonglong)my_atomic_load64_explicit((volatile int64*) + (&rli->log_space_total), + MY_MEMORY_ORDER_RELAXED) && !(slave_killed=io_slave_killed(mi)) && !rli->ignore_log_space_limit) mysql_cond_wait(&rli->log_space_cond, &rli->log_space_lock); @@ -2912,7 +2915,10 @@ static bool send_show_master_info_data(THD *thd, Master_info *mi, bool full, protocol->store(mi->rli.last_error().message, &my_charset_bin); protocol->store((uint32) mi->rli.slave_skip_counter); protocol->store((ulonglong) mi->rli.group_master_log_pos); - protocol->store((ulonglong) mi->rli.log_space_total); + protocol->store((ulonglong) + my_atomic_load64_explicit((volatile int64*) + (&mi->rli.log_space_total), + MY_MEMORY_ORDER_RELAXED)); protocol->store( mi->rli.until_condition==Relay_log_info::UNTIL_NONE ? "None": @@ -4428,7 +4434,9 @@ Stopping slave I/O thread due to out-of-memory error from master"); #endif if (rli->log_space_limit && rli->log_space_limit < - rli->log_space_total && + (ulonglong) my_atomic_load64_explicit((volatile int64*) + (&rli->log_space_total), + MY_MEMORY_ORDER_RELAXED) && !rli->ignore_log_space_limit) if (wait_for_relay_log_space(rli)) { @@ -7031,7 +7039,10 @@ static Log_event* next_event(rpl_group_info *rgi, ulonglong *event_size) is are able to rotate and purge sometime soon. */ if (rli->log_space_limit && - rli->log_space_limit < rli->log_space_total) + rli->log_space_limit < + (ulonglong) my_atomic_load64_explicit((volatile int64*) + (&rli->log_space_total), + MY_MEMORY_ORDER_RELAXED)) { /* force rotation if not in an unfinished group */ rli->sql_force_rotate_relay= !rli->is_in_group();

1 0

[Maria-developers] 41d807dfc8c: MENT-731:Setting of SHUTDOWN_WAIT_FOR_SLAVES should require special privilege, not generic SUPER
by sujatha 31 Aug '20

31 Aug '20

revision-id: 41d807dfc8c5f640ccbbac5838cf572b2b1712c1 (mariadb-10.4.4-1804-g41d807dfc8c) parent(s): 38cbb0915eb4a0c43f1a83afa37c7de5e481a718 author: Sujatha committer: Sujatha timestamp: 2020-08-31 22:30:38 +0530 message: MENT-731:Setting of SHUTDOWN_WAIT_FOR_SLAVES should require special privilege, not generic SUPER Fix: === 'shutdown_wait_for_slaves' variable can be set either by having SUPER or SHUTDOWN privilege. --- mysql-test/suite/binlog/r/binlog_grant.result | 34 +++++++++++++++++++++++ mysql-test/suite/binlog/t/binlog_grant.test | 40 +++++++++++++++++++++++++++ sql/privilege.h | 3 ++ sql/sys_vars.cc | 5 +++- 4 files changed, 81 insertions(+), 1 deletion(-) diff --git a/mysql-test/suite/binlog/r/binlog_grant.result b/mysql-test/suite/binlog/r/binlog_grant.result index af4c02411d5..aee0730c0ab 100644 --- a/mysql-test/suite/binlog/r/binlog_grant.result +++ b/mysql-test/suite/binlog/r/binlog_grant.result @@ -124,6 +124,40 @@ disconnect user1; connection default; DROP USER user1@localhost; # +# MENT-731: Setting of SHUTDOWN_WAIT_FOR_SLAVES should require special +# privilege, not generic SUPER +# +# Test that 'shutdown_wait_for_slaves' variable cannot be set without +# SHUTDOWN or SUPER +CREATE USER user1@localhost; +GRANT ALL PRIVILEGES ON *.* TO user1@localhost; +REVOKE SHUTDOWN, SUPER ON *.* FROM user1@localhost; +connect user1,localhost,user1,,; +connection user1; +SET GLOBAL SHUTDOWN_WAIT_FOR_SLAVES=ON; +ERROR 42000: Access denied; you need (at least one of) the SHUTDOWN, SUPER privilege(s) for this operation +disconnect user1; +connection default; +DROP USER user1@localhost; +# Test that 'shutdown_wait_for_slaves' variable can be set with SHUTDOWN +CREATE USER user1@localhost; +GRANT SHUTDOWN ON *.* TO user1@localhost; +connect user1,localhost,user1,,; +connection user1; +SET GLOBAL SHUTDOWN_WAIT_FOR_SLAVES=ON; +disconnect user1; +connection default; +DROP USER user1@localhost; +# Test that 'shutdown_wait_for_slaves' variable can be set with SUPER +CREATE USER user1@localhost; +GRANT SUPER ON *.* TO user1@localhost; +connect user1,localhost,user1,,; +connection user1; +SET GLOBAL SHUTDOWN_WAIT_FOR_SLAVES=OFF; +disconnect user1; +connection default; +DROP USER user1@localhost; +# # MDEV-21975 Add BINLOG REPLAY privilege and bind new privileges to # gtid_seq_no, preudo_thread_id, server_id, gtid_domain_id # diff --git a/mysql-test/suite/binlog/t/binlog_grant.test b/mysql-test/suite/binlog/t/binlog_grant.test index ec8d2aeccd2..ca0e40299a9 100644 --- a/mysql-test/suite/binlog/t/binlog_grant.test +++ b/mysql-test/suite/binlog/t/binlog_grant.test @@ -172,6 +172,46 @@ SHOW BINLOG EVENTS; connection default; DROP USER user1@localhost; +--echo # +--echo # MENT-731: Setting of SHUTDOWN_WAIT_FOR_SLAVES should require special +--echo # privilege, not generic SUPER +--echo # + +--echo # Test that 'shutdown_wait_for_slaves' variable cannot be set without +--echo # SHUTDOWN or SUPER +CREATE USER user1@localhost; +GRANT ALL PRIVILEGES ON *.* TO user1@localhost; +REVOKE SHUTDOWN, SUPER ON *.* FROM user1@localhost; +--connect(user1,localhost,user1,,) +--connection user1 +--error ER_SPECIFIC_ACCESS_DENIED_ERROR +SET GLOBAL SHUTDOWN_WAIT_FOR_SLAVES=ON; +--disconnect user1 +--connection default +DROP USER user1@localhost; + + +--echo # Test that 'shutdown_wait_for_slaves' variable can be set with SHUTDOWN +CREATE USER user1@localhost; +GRANT SHUTDOWN ON *.* TO user1@localhost; +--connect(user1,localhost,user1,,) +--connection user1 +SET GLOBAL SHUTDOWN_WAIT_FOR_SLAVES=ON; +--disconnect user1 +connection default; +DROP USER user1@localhost; + + +--echo # Test that 'shutdown_wait_for_slaves' variable can be set with SUPER +CREATE USER user1@localhost; +GRANT SUPER ON *.* TO user1@localhost; +--connect(user1,localhost,user1,,) +--connection user1 +SET GLOBAL SHUTDOWN_WAIT_FOR_SLAVES=OFF; +--disconnect user1 +connection default; +DROP USER user1@localhost; + --echo # --echo # MDEV-21975 Add BINLOG REPLAY privilege and bind new privileges to --echo # gtid_seq_no, preudo_thread_id, server_id, gtid_domain_id diff --git a/sql/privilege.h b/sql/privilege.h index 37cdf4da01a..f4bfd835013 100644 --- a/sql/privilege.h +++ b/sql/privilege.h @@ -433,6 +433,9 @@ constexpr privilege_t PRIV_SET_SYSTEM_GLOBAL_VAR_SLOW_LAUNCH_TIME= constexpr privilege_t PRIV_SET_SYSTEM_GLOBAL_VAR_THREAD_POOL= CONNECTION_ADMIN_ACL | SUPER_ACL; +// Was SUPER_ACL prior to 10.5.5 +constexpr privilege_t PRIV_SET_SYSTEM_VAR_SHUTDOWN_WAIT_FOR_SLAVES= + SHUTDOWN_ACL | SUPER_ACL; /* Binary log related privileges that are checked regardless diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc index 1577457a5d7..396c0570215 100644 --- a/sql/sys_vars.cc +++ b/sql/sys_vars.cc @@ -4573,7 +4573,10 @@ static Sys_var_ulong Sys_profiling_history_size( VALID_RANGE(0, 100), DEFAULT(15), BLOCK_SIZE(1)); #endif -static Sys_var_mybool Sys_shutdown_wait_for_slaves( +static Sys_var_on_access_global<Sys_var_mybool, + PRIV_SET_SYSTEM_VAR_SHUTDOWN_WAIT_FOR_SLAVES> + +Sys_shutdown_wait_for_slaves( "shutdown_wait_for_slaves", "when ON, SHUTDOWN command runs with implicit WAIT FOR ALL SLAVES option.", GLOBAL_VAR(opt_shutdown_wait_for_slaves), CMD_LINE(OPT_ARG),

1 0

[Maria-developers] e4fa0c805ad: MDEV-4633: multi_source.simple test fails sporadically
by sujatha 28 Aug '20

28 Aug '20

revision-id: e4fa0c805adf06c637cc3867427ca529b40e19ce (mariadb-10.1.43-267-ge4fa0c805ad) parent(s): de49e603770463e64cf4a79a7414bac93c02254f author: Sujatha committer: Sujatha timestamp: 2020-08-28 12:03:09 +0530 message: MDEV-4633: multi_source.simple test fails sporadically Analysis: ======== Writes to 'rli->log_space_total' needs to be synchronized. All modifications to 'rli->log_space_total' should be performed by holding 'rli->log_space_lock'. But IO thread doesn't acquire this mutex while writing events to relay log. Hence there is a race condition between SQL THREAD and IO THREAD, when both try to modify the same variable. SQL_THREAD is trying to decrement 'rli->log_space_total' in 'purge_first_log' and IO_THREAD is trying to increment the 'rli->log_space_total' in 'queue_event' simultaneously . Hence test occasionally fails with result mismatch. Fix: === Acquire 'rli->log_space_lock' while modifying 'rli->log_space_total'. --- sql/log.cc | 15 +++++++++++++++ sql/log.h | 13 +------------ sql/rpl_rli.cc | 4 ++++ sql/slave.cc | 28 ++++++++++++++-------------- 4 files changed, 34 insertions(+), 26 deletions(-) diff --git a/sql/log.cc b/sql/log.cc index 8049b94bab1..28a3d58d9e4 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -10265,6 +10265,21 @@ MYSQL_BIN_LOG::do_binlog_recovery(const char *opt_name, bool do_xa_recovery) return error; } +void MYSQL_BIN_LOG::harvest_bytes_written(Relay_log_info *rli) +{ +#ifndef DBUG_OFF + char buf1[22],buf2[22]; +#endif + DBUG_ENTER("harvest_bytes_written"); + mysql_mutex_lock(&rli->log_space_lock); + rli->log_space_total+=bytes_written; + DBUG_PRINT("info",("relay_log_space: %s bytes_written: %s", + llstr(rli->log_space_total,buf1), + llstr(bytes_written,buf2))); + bytes_written=0; + mysql_mutex_unlock(&rli->log_space_lock); + DBUG_VOID_RETURN; +} #ifdef INNODB_COMPATIBILITY_HOOKS /** diff --git a/sql/log.h b/sql/log.h index 277e5c6f69c..3b136c47bcf 100644 --- a/sql/log.h +++ b/sql/log.h @@ -706,18 +706,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG { bytes_written = 0; } - void harvest_bytes_written(ulonglong* counter) - { -#ifndef DBUG_OFF - char buf1[22],buf2[22]; -#endif - DBUG_ENTER("harvest_bytes_written"); - (*counter)+=bytes_written; - DBUG_PRINT("info",("counter: %s bytes_written: %s", llstr(*counter,buf1), - llstr(bytes_written,buf2))); - bytes_written=0; - DBUG_VOID_RETURN; - } + void harvest_bytes_written(Relay_log_info *rli); void set_max_size(ulong max_size_arg); void signal_update(); void wait_for_sufficient_commits(); diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc index c196a65809a..18d7db234fe 100644 --- a/sql/rpl_rli.cc +++ b/sql/rpl_rli.cc @@ -467,7 +467,9 @@ static inline int add_relay_log(Relay_log_info* rli,LOG_INFO* linfo) linfo->log_file_name); DBUG_RETURN(1); } + mysql_mutex_lock(&rli->log_space_lock); rli->log_space_total += s.st_size; + mysql_mutex_unlock(&rli->log_space_lock); DBUG_PRINT("info",("log_space_total: %llu", rli->log_space_total)); DBUG_RETURN(0); } @@ -477,7 +479,9 @@ static int count_relay_log_space(Relay_log_info* rli) { LOG_INFO linfo; DBUG_ENTER("count_relay_log_space"); + mysql_mutex_lock(&rli->log_space_lock); rli->log_space_total= 0; + mysql_mutex_unlock(&rli->log_space_lock); if (rli->relay_log.find_log_pos(&linfo, NullS, 1)) { sql_print_error("Could not find first log while counting relay log space"); diff --git a/sql/slave.cc b/sql/slave.cc index 06f2b0d955a..3d9e23441bc 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -2508,7 +2508,7 @@ static void write_ignored_events_info_to_relay_log(THD *thd, Master_info *mi) } if (likely (rev || glev)) { - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); if (flush_master_info(mi, TRUE, TRUE)) sql_print_error("Failed to flush master info file"); } @@ -5270,7 +5270,7 @@ static int process_io_create_file(Master_info* mi, Create_file_log_event* cev) "error writing Exec_load event to relay log"); goto err; } - mi->rli.relay_log.harvest_bytes_written(&mi->rli.log_space_total); + mi->rli.relay_log.harvest_bytes_written(&mi->rli); break; } if (unlikely(cev_not_written)) @@ -5285,7 +5285,7 @@ static int process_io_create_file(Master_info* mi, Create_file_log_event* cev) goto err; } cev_not_written=0; - mi->rli.relay_log.harvest_bytes_written(&mi->rli.log_space_total); + mi->rli.relay_log.harvest_bytes_written(&mi->rli); } else { @@ -5299,7 +5299,7 @@ static int process_io_create_file(Master_info* mi, Create_file_log_event* cev) "error writing Append_block event to relay log"); goto err; } - mi->rli.relay_log.harvest_bytes_written(&mi->rli.log_space_total) ; + mi->rli.relay_log.harvest_bytes_written(&mi->rli) ; } } } @@ -5489,7 +5489,7 @@ static int queue_binlog_ver_1_event(Master_info *mi, const char *buf, mysql_mutex_unlock(&mi->data_lock); DBUG_RETURN(1); } - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); } delete ev; mi->master_log_pos+= inc_pos; @@ -5547,7 +5547,7 @@ static int queue_binlog_ver_3_event(Master_info *mi, const char *buf, mysql_mutex_unlock(&mi->data_lock); DBUG_RETURN(1); } - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); delete ev; mi->master_log_pos+= inc_pos; err: @@ -5784,9 +5784,7 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) mysql_mutex_lock(log_lock); if (likely(!rli->relay_log.write_event(&fdle) && !rli->relay_log.flush_and_sync(NULL))) - { - rli->relay_log.harvest_bytes_written(&rli->log_space_total); - } + rli->relay_log.harvest_bytes_written(rli); else { error= ER_SLAVE_RELAY_LOG_WRITE_FAILURE; @@ -6221,7 +6219,7 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) (rli->relay_log.description_event_for_queue)) error= ER_SLAVE_RELAY_LOG_WRITE_FAILURE; else - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); } else if (mi->gtid_reconnect_event_skip_count == 0) { @@ -6234,7 +6232,7 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) if (rli->relay_log.append_no_lock(&fake_rev)) error= ER_SLAVE_RELAY_LOG_WRITE_FAILURE; else - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); } } else @@ -6302,7 +6300,7 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) { mi->master_log_pos+= inc_pos; DBUG_PRINT("info", ("master_log_pos: %lu", (ulong) mi->master_log_pos)); - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); } else { @@ -6382,7 +6380,7 @@ void end_relay_log_info(Relay_log_info* rli) log_lock= rli->relay_log.get_log_lock(); mysql_mutex_lock(log_lock); rli->relay_log.close(LOG_CLOSE_INDEX | LOG_CLOSE_STOP_EVENT); - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); mysql_mutex_unlock(log_lock); /* Delete the slave's temporary tables from memory. @@ -7318,7 +7316,9 @@ int rotate_relay_log(Master_info* mi) Note that it needs to be protected by mi->data_lock. */ mysql_mutex_assert_owner(&mi->data_lock); - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + + rli->relay_log.harvest_bytes_written(rli); + end: DBUG_RETURN(error); }

1 0

[Maria-developers] 2f859962b03: MDEV-23534: SIGSEGV in sf_malloc_usable_size/my_free on SET GLOBAL REPLICATE_DO_TABLE
by sujatha 27 Aug '20

27 Aug '20

revision-id: 2f859962b032cc75176e05df8d704eec413cdb17 (mariadb-10.1.43-266-g2f859962b03) parent(s): 62d1e3bf67a12eb6f48ac615bda119e2ed65cf16 author: Sujatha committer: Sujatha timestamp: 2020-08-26 16:25:28 +0530 message: MDEV-23534: SIGSEGV in sf_malloc_usable_size/my_free on SET GLOBAL REPLICATE_DO_TABLE Backporting fixes for: MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA) MDEV-22059: MSAN report at replicate_ignore_table_grant --- .../suite/rpl/r/rpl_filter_tables_dynamic.result | 2 + .../rpl/r/rpl_filter_wild_tables_dynamic.result | 2 + .../suite/rpl/t/rpl_filter_tables_dynamic.test | 2 + .../rpl/t/rpl_filter_wild_tables_dynamic.test | 2 + sql/rpl_filter.cc | 52 ++++++++++++++++------ 5 files changed, 46 insertions(+), 14 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_filter_tables_dynamic.result b/mysql-test/suite/rpl/r/rpl_filter_tables_dynamic.result index 5a746c88458..9709e24fbde 100644 --- a/mysql-test/suite/rpl/r/rpl_filter_tables_dynamic.result +++ b/mysql-test/suite/rpl/r/rpl_filter_tables_dynamic.result @@ -5,6 +5,8 @@ ERROR HY000: This operation cannot be performed as you have a running slave ''; SET @@GLOBAL.replicate_ignore_table="test.t4,test.t5,test.t6"; ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first include/stop_slave.inc +SET @@GLOBAL.replicate_do_table=""; +SET @@GLOBAL.replicate_ignore_table=""; SET @@GLOBAL.replicate_do_table="test.t1,test.t2,test.t3"; SET @@GLOBAL.replicate_ignore_table="test.t4,test.t5,test.t6"; include/start_slave.inc diff --git a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result index 19d8e513e6f..338f4b3bbcf 100644 --- a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result +++ b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result @@ -5,6 +5,8 @@ ERROR HY000: This operation cannot be performed as you have a running slave ''; SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first include/stop_slave.inc +SET @@GLOBAL.replicate_wild_do_table=""; +SET @@GLOBAL.replicate_wild_ignore_table=""; SET @@GLOBAL.replicate_wild_do_table="test.a%"; SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; include/start_slave.inc diff --git a/mysql-test/suite/rpl/t/rpl_filter_tables_dynamic.test b/mysql-test/suite/rpl/t/rpl_filter_tables_dynamic.test index 97ecc167356..ebededc36b3 100644 --- a/mysql-test/suite/rpl/t/rpl_filter_tables_dynamic.test +++ b/mysql-test/suite/rpl/t/rpl_filter_tables_dynamic.test @@ -51,6 +51,8 @@ SET @@GLOBAL.replicate_ignore_table="test.t4,test.t5,test.t6"; connection slave; source include/stop_slave.inc; +SET @@GLOBAL.replicate_do_table=""; +SET @@GLOBAL.replicate_ignore_table=""; SET @@GLOBAL.replicate_do_table="test.t1,test.t2,test.t3"; SET @@GLOBAL.replicate_ignore_table="test.t4,test.t5,test.t6"; source include/start_slave.inc; diff --git a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test index c822c81f270..09db91aa4d3 100644 --- a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test +++ b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test @@ -13,6 +13,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; connection slave; source include/stop_slave.inc; +SET @@GLOBAL.replicate_wild_do_table=""; +SET @@GLOBAL.replicate_wild_ignore_table=""; SET @@GLOBAL.replicate_wild_do_table="test.a%"; SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; source include/start_slave.inc; diff --git a/sql/rpl_filter.cc b/sql/rpl_filter.cc index 366902c1f26..0d5d9ffeea8 100644 --- a/sql/rpl_filter.cc +++ b/sql/rpl_filter.cc @@ -349,14 +349,20 @@ Rpl_filter::set_do_table(const char* table_spec) int status; if (do_table_inited) - my_hash_reset(&do_table); + { + my_hash_free(&do_table); + do_table_inited= 0; + } status= parse_filter_rule(table_spec, &Rpl_filter::add_do_table); - if (!do_table.records) + if (do_table_inited && status) { - my_hash_free(&do_table); - do_table_inited= 0; + if (!do_table.records) + { + my_hash_free(&do_table); + do_table_inited= 0; + } } return status; @@ -369,14 +375,20 @@ Rpl_filter::set_ignore_table(const char* table_spec) int status; if (ignore_table_inited) - my_hash_reset(&ignore_table); + { + my_hash_free(&ignore_table); + ignore_table_inited= 0; + } status= parse_filter_rule(table_spec, &Rpl_filter::add_ignore_table); - if (!ignore_table.records) + if (ignore_table_inited && status) { - my_hash_free(&ignore_table); - ignore_table_inited= 0; + if (!ignore_table.records) + { + my_hash_free(&ignore_table); + ignore_table_inited= 0; + } } return status; @@ -411,14 +423,20 @@ Rpl_filter::set_wild_do_table(const char* table_spec) int status; if (wild_do_table_inited) + { free_string_array(&wild_do_table); + wild_do_table_inited= 0; + } status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); - if (!wild_do_table.elements) + if (wild_do_table_inited && status) { - delete_dynamic(&wild_do_table); - wild_do_table_inited= 0; + if (!wild_do_table.elements) + { + delete_dynamic(&wild_do_table); + wild_do_table_inited= 0; + } } return status; @@ -431,14 +449,20 @@ Rpl_filter::set_wild_ignore_table(const char* table_spec) int status; if (wild_ignore_table_inited) + { free_string_array(&wild_ignore_table); + wild_ignore_table_inited= 0; + } status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_ignore_table); - if (!wild_ignore_table.elements) + if (wild_ignore_table_inited && status) { - delete_dynamic(&wild_ignore_table); - wild_ignore_table_inited= 0; + if (!wild_ignore_table.elements) + { + delete_dynamic(&wild_ignore_table); + wild_ignore_table_inited= 0; + } } return status;

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 24 Aug '20

24 Aug '20

Hi, This week - Implemented the BZip2, LZ4 and LZ4HC services along with tests, as per original project schedule. This means that all services have been implemented (🎉), although some of them need review and fine-tuning, as documented in the table below. - Here is the current state of all services: | \ | BZip2 | LZ4 | LZ4HC | LZMA | LZO | Snappy | ZStandard | | Connect | * | | | | | | | | InnoDB | Y | Y | | Y | Y | Y | | | Mroonga | | Y | | | | | Y | | RocksDB | * | ? | ? | | | Y | ? | Y → Service is confirmed working, and tests are written. * → Service is working, but needs tweaking or additional tests. ? → Service is implemented, but it might not be worth the additional complexity. - Wrote tests for Connect and Mroonga. Next week - Work on the final report for GSoC. - Made a list of LZ4 and ZStandard versions on all supported platforms. RocksDB uses different sets of functions depending on the compile-time version of the library available, so this will allow us to make a more informed choice of the minimum version number. - Finish writing the HOWTO file. Thanks, Kartik

1 0

[Maria-developers] Building from source failing on Mac OS X complaining of librarymariadb
by Rodrigo Souza 21 Aug '20

21 Aug '20

Hello all, I’m a newbie, trying to familiarize myself with the source code, however, once I followed the steps outlined here https://mariadb.org/get-involved/getting-started-for-developers/get-code-bu… <https://mariadb.org/get-involved/getting-started-for-developers/get-code-bu…> to build the server on Mac OS X, currently the MAKE fails on me complaining about a deprecated GSSAPI.h as follows: /mariadb/server/libmariadb/plugins/auth/gssapi_client.c:72:12: error: 'gss_import_name' is deprecated: use GSS.framework [-Werror,-Wdeprecated-declarations] major= gss_import_name(&minor, &principal_name_buf, GSS_C_NT_USER_NAME, &service_name); ^ /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/gssapi/gssapi.h:591:19: note: 'gss_import_name' has been explicitly marked deprecated here gss_name_t *) GSSKRB_APPLE_DEPRECATED("use GSS.framework") __API_DEPRECATED("Use GSS.framework", macos(10.0, 10.8)) __API_UNAVAILABLE(macCatalyst); /* output_name */ ^ /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/gssapi/gssapi.h:57:51: note: expanded from macro 'GSSKRB_APPLE_DEPRECATED' #define GSSKRB_APPLE_DEPRECATED(x) __attribute__((deprecated(x))) My Mac OS X is Catalina 10.15.6 and XCode Tools 11.6. Have you guys ever had this kind of issue trying to build on Mac OS? Thank you, Rodrigo Souza

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 17 Aug '20

17 Aug '20

Hi, This week - Worked out, fixed and documented some bugs in the Snappy service. My initial conclusion needed some fine-tuning, it might be possible to make RocksDB work with our modifications, if we are able to switch the default compression engine on startup. I also fixed the crash on startup for the Snappy service. - Implemented a solution to conditionally include the custom headers only for the relevant storage engines. This will also allow us to skip over a storage engine (for example RocksDB) if it does not work well with our modifications. # in cmake/plugin.cmake IF( plugin STREQUAL "CONNECT" OR plugin STREQUAL "INNOBASE" OR plugin STREQUAL "MROONGA" OR plugin STREQUAL "ROCKSDB" ) INCLUDE_DIRECTORIES(${CMAKE_SOURCE_DIR}/include/compression) ENDIF() - Reverted the implementation for the Snappy and LZO service. - Rebased the branches onto the latest 10.5 branch. Next week - Implement the BZip2 service. - Work on the ZStd and Snappy service to try and make it work with RocksDB. - Continue writing the HOWTO file. Thanks, Kartik

1 0

[Maria-developers] Progress Report GSoC 2020 (10-16 August)
by Anton Mikhaylenko 17 Aug '20

17 Aug '20

Good day, everybody. In this post, I want to describe what I did this week, August 10-16. This week I've running benchmarks for threadpool and compare results. Regards, Anton Mikhailenko

1 0

[Maria-developers] 5dc2ebae474: MDEV-22330: mysqlbinlog stops with an error Don't know how to handle column type: 255 meta: 4 (0004)
by sujatha 17 Aug '20

17 Aug '20

revision-id: 5dc2ebae47481afe5b0769f0e0463d64b77143c0 (mariadb-10.1.43-260-g5dc2ebae474) parent(s): 101ce10d0ddda1e38806b8d2c491298482e7ab78 author: Sujatha committer: Sujatha timestamp: 2020-08-13 09:58:25 +0530 message: MDEV-22330: mysqlbinlog stops with an error Don't know how to handle column type: 255 meta: 4 (0004) Analysis: ======== "mysqlbinlog -v" option will reconstruct row events and display them as commented SQL statements. If this option is given twice, the output includes comments to indicate column data types and some metadata. `log_event_print_value` is the function responsible for printing values and their types. This function doesn't handle GEOMETRY type. Hence the above error gets printed. Fix: === Add support for GEOMETRY datatype. --- .../suite/binlog/r/binlog_mysqlbinlog_row.result | 56 +++++++++++++++++++++- .../suite/binlog/t/binlog_mysqlbinlog_row.test | 14 +++++- sql/log_event.cc | 17 +++++-- 3 files changed, 80 insertions(+), 7 deletions(-) diff --git a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result index 9a3f4751165..0b05c4f449f 100644 --- a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result +++ b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result @@ -339,7 +339,11 @@ a 123.47 999.99 DROP TABLE t1dec102; -flush logs; +CREATE TABLE t1 (a GEOMETRY DEFAULT NULL); +INSERT INTO t1 VALUES (NULL); +INSERT INTO t1 VALUES (POINT(10,10)); +DROP TABLE t1; +FLUSH LOGS; /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; @@ -4623,6 +4627,56 @@ SET TIMESTAMP=1000000000/*!*/; DROP TABLE `t1dec102` /* generated by server */ /*!*/; # at # +#010909 4:46:40 server id 1 end_log_pos # GTID 0-1-321 ddl +/*!100001 SET @@session.gtid_seq_no=321*//*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # Query thread_id=# exec_time=# error_code=0 +SET TIMESTAMP=1000000000/*!*/; +CREATE TABLE t1 (a GEOMETRY DEFAULT NULL) +/*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # GTID 0-1-322 +/*!100001 SET @@session.gtid_seq_no=322*//*!*/; +BEGIN +/*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +# at # +#010909 4:46:40 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +### INSERT INTO `test`.`t1` +### SET +### @1=NULL /* GEOMETRY meta=4 nullable=1 is_null=1 */ +# at # +#010909 4:46:40 server id 1 end_log_pos # Query thread_id=# exec_time=# error_code=0 +SET TIMESTAMP=1000000000/*!*/; +COMMIT +/*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # GTID 0-1-323 +/*!100001 SET @@session.gtid_seq_no=323*//*!*/; +BEGIN +/*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +# at # +#010909 4:46:40 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +### INSERT INTO `test`.`t1` +### SET +### @1='\x00\x00\x00\x00\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00$@\x00\x00\x00\x00\x00\x00$@' /* GEOMETRY meta=4 nullable=1 is_null=0 */ +# at # +#010909 4:46:40 server id 1 end_log_pos # Query thread_id=# exec_time=# error_code=0 +SET TIMESTAMP=1000000000/*!*/; +COMMIT +/*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # GTID 0-1-324 ddl +/*!100001 SET @@session.gtid_seq_no=324*//*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # Query thread_id=# exec_time=# error_code=0 +SET TIMESTAMP=1000000000/*!*/; +DROP TABLE `t1` /* generated by server */ +/*!*/; +# at # #010909 4:46:40 server id 1 end_log_pos # Rotate to master-bin.000002 pos: 4 DELIMITER ; # End of log file diff --git a/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row.test b/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row.test index 0c94d968338..93c26cc57a2 100644 --- a/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row.test +++ b/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row.test @@ -450,7 +450,19 @@ INSERT INTO t1dec102 VALUES (999.99); SELECT * FROM t1dec102 ORDER BY a; DROP TABLE t1dec102; -flush logs; + +# +# MDEV-22330: mysqlbinlog stops with an error Don't know how to handle column +# type: 255 meta: 4 (0004) +# Check support for GEOMETRY type with verbose mode. +# +CREATE TABLE t1 (a GEOMETRY DEFAULT NULL); + +INSERT INTO t1 VALUES (NULL); +INSERT INTO t1 VALUES (POINT(10,10)); +DROP TABLE t1; + +FLUSH LOGS; --replace_result $MYSQLTEST_VARDIR MYSQLTEST_VARDIR --replace_regex /SQL_LOAD_MB-[0-9]-[0-9]/SQL_LOAD_MB-#-#/ /exec_time=[0-9]*/exec_time=#/ /end_log_pos [0-9]*/end_log_pos #/ /# at [0-9]*/# at #/ /thread_id=[0-9]*/thread_id=#/ /table id [0-9]*/table id #/ /mapped to number [0-9]*/mapped to number #/ /server v [^ ]*/server v #.##.##/ /((a)[0-9]*=[0-9]*[.][0-9]{1,3})[0-9e+-]*[^ ]*(.*(FLOAT|DOUBLE).*[*].)/\1...\2/ diff --git a/sql/log_event.cc b/sql/log_event.cc index a1a442df43f..e8121ba35ff 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -2575,13 +2575,20 @@ log_event_print_value(IO_CACHE *file, const uchar *ptr, "Not enough metadata to display the value. "); break; + case MYSQL_TYPE_GEOMETRY: + strmake(typestr, "GEOMETRY", typestr_length); + if (!ptr) + goto return_null; + + length= uint4korr(ptr); + my_b_write_quoted(file, ptr + meta, length); + return length + meta; + default: { - char tmp[5]; - my_snprintf(tmp, sizeof(tmp), "%04x", meta); - my_b_printf(file, - "!! Don't know how to handle column type=%d meta=%d (%s)", - type, meta, tmp); + fprintf(stderr, + "\nError: Don't know how to handle column type: %d meta: %d (%04x)\n", + type, meta, meta); } break; }

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 10 Aug '20

10 Aug '20

Hi, This week - Had a meeting with Robert about further actions regarding GSoC and discussed expectations for the end of the month. - Started extending the tests to include all combinations of engine + library. let %TEST_MATRIX = { "connect" => ("bzip2", ), "innodb" => ("bzip2", "lz4", "lzma", "lzo", "snappy" ), "mroonga" => ( "lz4", "zstd"), "rocksdb" => ("bzip2", "lz4", "lz4hc", "snappy", "zstd") }; - Started writing a HOWTO about adding a new compression library in sql/compression/HOWTO. - Finished the first draft of the ZStd service. Since ZStd and RocksDB are both written by Facebook, RocksDB makes extensive use of the ZStd functions. RocksDB also compiles with different sets of functions depending on the compile time version of ZStd, so we need to decide on a minimum version number and enforce it while loading the library. Next week - Finish the implementation for ZStd. - Continue writing the HOWTO file. - Update the old service implementations. - Continue replicating the implementation for other libraries. Thanks, Kartik

1 0

[Maria-developers] Progress Report GSoC 2020 (3-9 August)
by Anton Mikhaylenko 10 Aug '20

10 Aug '20

Good day, everybody. In this post, I want to describe what I did this week, August 3-9. This week, I wrote a function, that gets current CPU Utilization for Windows and Linux systems. Also, I merged commits with squash into one. Regards, Anton Mikhailenko

1 0

[Maria-developers] 0664e1e766d: MDEV-22338: Assertion `!current_stmt_is_commit || !rgi->tables_to_lock' failed in Query_log_event::do_apply_event
by sujatha 10 Aug '20

10 Aug '20

revision-id: 0664e1e766d303187b381ace1c53547d1821e35c (mariadb-10.1.43-252-g0664e1e766d) parent(s): 85bd5314c56c150d756066806d4a6cb5b682383f author: Sujatha committer: Sujatha timestamp: 2020-08-10 12:29:48 +0530 message: MDEV-22338: Assertion `!current_stmt_is_commit || !rgi->tables_to_lock' failed in Query_log_event::do_apply_event Analysis: ========= With row based replication, master can generate event groups like shown below. BEGIN TABLE_MAP COMMIT i.e A binlog event group, without any rows event in it. Such groups are possible when a query operating on transactional table, doesn't modify any data in the table. For example a transaction is trying to modify both trans and non-trans tables and a trigger exists on non-transtable which modifies a trans table. CREATE TABLE t1 (id int) engine=innodb; CREATE TABLE t2 (id int) engine=myisam; CREATE TRIGGER t2 BEFORE INSERT ON t2 FOR EACH ROW DELETE FROM t1 LIMIT 1; INSERT INTO t2 VALUES (1); Above insert will write a record in 't2' table and its corresponding binlog entry is generated. First table map events are added to appropriate caches. t2's Table_Map is written into statement_cache. t1's Table_Map is written into transaction_cache. 'Write_rows' event is added to t2's statement_cache. Since there are no rows to remove from table 't1' there will not be any 'Delete_rows' event in the transaction cache. There won't be any 'Xid' event as well. During flush operation 'COMMIT' will be added to the transaction cache to complete the group. "t2's statement_cache" "t1's transaction_cache" BEGIN BEGIN TABLE_MAP TABLE_MAP WRITE_ROWS COMMIT COMMIT On the slave side it has strict rules, that a 'Table_Map' event should be followed by a rows event. This rule gets violated for 't1' transaction and results in an assert. Fix: ==== Replace the assert with if-else block. In a case where TABLE_MAP event is not followed by rows event invoke 'rows_event_stmt_cleanup' to initiate appropriate cleanup. The fix is based on upstream patch: commit da01f3846b62a2e41111256f9d6f0a2bb2de23c4 Author: Andrei Elkin <aelkin(a)mysql.com> --- .../suite/rpl/r/rpl_no_rows_event_assert.result | 26 ++++++++++++ .../suite/rpl/t/rpl_no_rows_event_assert.test | 46 ++++++++++++++++++++++ sql/log_event.cc | 22 ++++++++++- 3 files changed, 92 insertions(+), 2 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_no_rows_event_assert.result b/mysql-test/suite/rpl/r/rpl_no_rows_event_assert.result new file mode 100644 index 00000000000..7661bb62305 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_no_rows_event_assert.result @@ -0,0 +1,26 @@ +include/master-slave.inc +[connection master] +######################################################################### +# Test case1: Transaction cache has 'Table_Map' event followed by # +# 'COMMIT'. # +######################################################################### +CREATE TABLE t1 (id int) ENGINE=INNODB; +CREATE TABLE t2 (id int) ENGINE=MYISAM; +CREATE TRIGGER t2 BEFORE INSERT ON t2 FOR EACH ROW DELETE FROM t1 LIMIT 1; +INSERT INTO t2 VALUES (1); +DROP TABLE t1,t2; +######################################################################### +# Test case2: Transaction cache has 'Query', 'Table_Map' followed by # +# 'COMMIT'. # +######################################################################### +CREATE TABLE t1(f INT) ENGINE=INNODB; +CREATE TABLE t2(f INT) ENGINE=MYISAM; +CREATE TABLE t3(id INT AUTO_INCREMENT, i INT, PRIMARY KEY (id)) ENGINE=INNODB; +CREATE TABLE t4(f INT) ENGINE=MYISAM; +CREATE TRIGGER trig1 BEFORE INSERT ON t2 FOR EACH ROW INSERT INTO t3(i) SELECT * FROM t4 LIMIT 0; +BEGIN; +DELETE FROM t1 WHERE f=0; +INSERT INTO t2 VALUES (2); +COMMIT; +DROP TABLE t1,t2,t3,t4; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_no_rows_event_assert.test b/mysql-test/suite/rpl/t/rpl_no_rows_event_assert.test new file mode 100644 index 00000000000..c5cc7634615 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_no_rows_event_assert.test @@ -0,0 +1,46 @@ +--source include/have_innodb.inc +--source include/have_binlog_format_mixed_or_row.inc +--source include/master-slave.inc +# ==== References ==== +# +# MDEV-22338: Assertion `!current_stmt_is_commit || !rgi->tables_to_lock' +# failed in Query_log_event::do_apply_event +# + +--echo ######################################################################### +--echo # Test case1: Transaction cache has 'Table_Map' event followed by # +--echo # 'COMMIT'. # +--echo ######################################################################### +--connection master +CREATE TABLE t1 (id int) ENGINE=INNODB; +CREATE TABLE t2 (id int) ENGINE=MYISAM; +CREATE TRIGGER t2 BEFORE INSERT ON t2 FOR EACH ROW DELETE FROM t1 LIMIT 1; +INSERT INTO t2 VALUES (1); +--sync_slave_with_master + +# Cleanup +--connection master +DROP TABLE t1,t2; + +--echo ######################################################################### +--echo # Test case2: Transaction cache has 'Query', 'Table_Map' followed by # +--echo # 'COMMIT'. # +--echo ######################################################################### + +CREATE TABLE t1(f INT) ENGINE=INNODB; +CREATE TABLE t2(f INT) ENGINE=MYISAM; +CREATE TABLE t3(id INT AUTO_INCREMENT, i INT, PRIMARY KEY (id)) ENGINE=INNODB; +CREATE TABLE t4(f INT) ENGINE=MYISAM; +CREATE TRIGGER trig1 BEFORE INSERT ON t2 FOR EACH ROW INSERT INTO t3(i) SELECT * FROM t4 LIMIT 0; + +BEGIN; +DELETE FROM t1 WHERE f=0; +INSERT INTO t2 VALUES (2); +COMMIT; +--sync_slave_with_master + +# Cleanup +--connection master +DROP TABLE t1,t2,t3,t4; + +--source include/rpl_end.inc diff --git a/sql/log_event.cc b/sql/log_event.cc index a1a442df43f..c37512aa6c1 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -4329,8 +4329,26 @@ int Query_log_event::do_apply_event(rpl_group_info *rgi, thd->clear_error(1); current_stmt_is_commit= is_commit(); - DBUG_ASSERT(!current_stmt_is_commit || !rgi->tables_to_lock); - rgi->slave_close_thread_tables(thd); + if (strcmp("COMMIT", query) == 0 && rgi->tables_to_lock != NULL) + { + /* + Cleaning-up the last statement context: + Found a row based event group which didn't modfiy any rows. i.e. + BEGIN + TABLE_MAP + COMMIT + */ + int error; + if ((error = rows_event_stmt_cleanup(rgi, thd))) + { + rli->report(ERROR_LEVEL, error, "Error in cleaning up after an event " + "preceding the commit; the group log file/position: %s %llu", + rli->group_master_log_name, + (ulong) rli->group_master_log_pos); + } + } + else + rgi->slave_close_thread_tables(thd); /* Note: We do not need to execute reset_one_shot_variables() if this

1 0

Re: [Maria-developers] [Commits] c776cad5f8e: MDEV-22702: Assertion `!field->is_null()' failed in my_decimal::my_decimal
by Sergey Petrunia 06 Aug '20

06 Aug '20

Hi Varun, First: I can still get the assertion failure if I run this: set @@sql_mode='only_full_group_by'; select min(1 mod a1), bit_or(a2) over () from t1; As far as I understand, the query is not valid for only_full_group_by mode. (Window functions as such are ok, but here window function is using a2, which cannot be accessed in the post-grouping context). More input below. On Wed, Jul 08, 2020 at 02:58:31PM +0530, Varun wrote: > revision-id: c776cad5f8ecf2675510deeb55724d7255a52503 (mariadb-10.4.11-267-gc776cad5f8e) > parent(s): cc0dca366357651ddb549e31a12b1ecd39c7380e > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-07-08 14:58:17 +0530 > message: > > MDEV-22702: Assertion `!field->is_null()' failed in my_decimal::my_decimal > > With implicit grouping with window functions, we need to make sure that all the > fields inside the window functions are nullable as any non-aggregated field can > produce a NULL value. ... > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index 0983cea44d0..16a0b3b08a6 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -1510,6 +1510,8 @@ class st_select_lex: public st_select_lex_node > } > > bool have_window_funcs() const { return (window_funcs.elements !=0); } > + uint32 get_number_of_window_funcs() const > + { return (uint32)window_funcs.elements; } Why use uint32 when n_sum_items is uint, and elements is also uint? I suggest just keep using uint. > ORDER *find_common_window_func_partition_fields(THD *thd); > > bool cond_pushdown_is_allowed() const > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 0ca5ab23288..05463efe9a5 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -1201,6 +1201,18 @@ JOIN::prepare(TABLE_LIST *tables_init, > break; > } > } > + /* > + If the query has a window function with an aggregate function, > + then also we have a mix of elements with and without grouping. > + Window function can be in the ORDER BY clause too so the check > + is made separately. Wording could be better. I don't understand what did you mean to say about ORDER BY. can you elaborate? > + Window function is inherited from Item_sum so each window function is > + also registered as a sum item, so need to check that we have an > + explicit aggregate function also in the query. what's "explicit aggregate" ? A regular aggregate function (i.e. not a window function)? > + */ > + if (select_lex->have_window_funcs() && > + select_lex->get_number_of_window_funcs() < select_lex->n_sum_items) > + mixed_implicit_grouping= true; > } > > table_count= select_lex->leaf_tables.elements; BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Optimizer call followup: for the parser, JSON_TABLE is not a "table function"
by Sergei Petrunia 05 Aug '20

05 Aug '20

Hello, A followup to the optimizer call discussion: Here is the production from the SQL Standard describing JSON_TABLE: section 7.6 <table reference>: <table primary> ::= <table or query name> [ <query system time period specification> ] [ <correlation or recognition> ] | <derived table> <correlation or recognition> | <lateral derived table> <correlation or recognition> | <collection derived table> <correlation or recognition> | <table function derived table> <correlation or recognition> | <only spec> [ <correlation or recognition> ] | <data change delta table> [ <correlation or recognition> ] | <JSON table> <correlation or recognition> | <JSON table primitive> <correlation name> | <parenthesized joined table> <table function derived table> ::= TABLE <left paren> <collection value expression> <right parent> Section 7.11 <JSON table>: <JSON table> ::= JSON_TABLE <left paren> ....... <right parent> That is, JSON_TABLE and TABLE (...) are two different productions that have nothing in common. So, as far as syntax is concerned, JSON_TABLE is not a "table function". BR -- Sergei P.

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 03 Aug '20

03 Aug '20

Hi, This week - Finished writing common testing foundation that allows other services to be tested easily. For example: --let $test_library = lzma --source test_library.inc --let $test_library = lzo --source test_library.inc ... - Finished the first draft of the Snappy service. Snappy has a C++ api that is used by RocksDB, so we have two options: - Use the mangled C++ function names directly, and load them normally with dlopen. - Re-implement the C++ function from Snappy’s C exports. The mangled function names may vary across platforms, so we are going with the second option for now. - Implemented global status variables for the services. This lays the foundation for integration with the storage engines. - Finished a first draft of the auto logic for the --use-compression= switch. Next week - Need to review my refactoring and decide on next steps. Details of refactor: - Removed the service struct. - Removed the struct from sql/sql_plugin_services.ic. - Define an anonymous struct in sql/compression/<library>.cc. - Initialize the struct with dummy function pointers. - If the library is successfully loaded, replace them with pointers to the real functions. - Define wrapper functions that forward the call to the function pointers in the struct. - Continue replicating the implementation for other libraries. - Decide what to do when dynamic libraries have the versions suffixed to the .so file. For example, liblzma.so.5.2.4 is included with Ubuntu 20, but liblzma.so is not. The liblzma.so -> liblzma.so.5.2.4 symlink is only added after installing the liblzma-dev package. - Sync up with Robert to discuss further actions. Thanks, Kartik

1 0

[Maria-developers] Review for: MDEV-17399 Add support for JSON_TABLE, part #9
by Sergey Petrunia 03 Aug '20

03 Aug '20

Hi Alexey, More input: == Circular dependencies are allowed == This query will cause an assert in the optimizer, for obvious reasons: select * from json_table(JS3.size, '$' columns (size INT PATH '$.size')) as JS1, json_table(JS1.size, '$' columns (size INT PATH '$.size')) as JS2, json_table(JS1.size, '$' columns (size INT PATH '$.size')) as JS3 where 1; == Character set inference is wrong == create table t20 (json varchar(100) character set utf8); insert into t20 values ('{"value":"АБВ"}'); create table tj20 as select T.value from t20, json_table(t20.json, '$' columns (value varchar(32) PATH '$.value')) T; mysql> show create table tj20\G *************************** 1. row *************************** Table: tj20 Create Table: CREATE TABLE `tj20` ( `value` varchar(32) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1 The source document uses UTF-8 but the values from it are in latin1? == Comments and code cleanup == There are a lot of comments missing, some duplicate code, and code using out-of-date ways of doing things. Please find attached a patch with obvious fixes for most of it. == Explain == Let's follow MySQL here: let the tabular EXPLAIN show "Table function: json_table" in the Extra column, let EXPLAIN FORMAT=JSON show "table_function": "json_table" in the "table" element. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Obtaining usernames in MariaDB UDFs
by Denis Scherbakov 02 Aug '20

02 Aug '20

Hello guys, I am building a set of MariaDB UDFs that require to know a name of a user who called them in each and every call. Users should not be able to pretend as someone else, hence usernames as an argument cannot be used. I compile UDFs against MariaDB headers and I was able to obtain the current_thd, but the security_ctx pointer inside is NULL. Could somebody give me a hint what I am doing wrong? Thank you for your answers! Best regards, Denis Scherbakov

2 2

Re: [Maria-developers] 7311586ca25: MDEV-15703 Crash in EXECUTE IMMEDIATE 'CREATE OR REPLACE TABLE t1 (a INT DEFAULT ?)' USING DEFAULT
by Sergei Golubchik 02 Aug '20

02 Aug '20

Hi, Oleksandr! On Jul 29, Oleksandr Byelkin wrote: > revision-id: 7311586ca25 (mariadb-10.2.31-317-g7311586ca25) > parent(s): a1e52e7f32c > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-07-17 13:47:26 +0200 > message: > > MDEV-15703 Crash in EXECUTE IMMEDIATE 'CREATE OR REPLACE TABLE t1 (a INT DEFAULT ?)' USING DEFAULT > > Part 1: make better asserts. > > diff --git a/sql/field.cc b/sql/field.cc > index 65bd9d22857..cf0cc655888 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -11138,6 +11138,16 @@ bool Field::save_in_field_default_value(bool view_error_processing) > { > THD *thd= table->in_use; > > + if ( // table is not opened properly > + table == NULL || table->pos_in_table_list == NULL || > + table->pos_in_table_list->select_lex == NULL || > + // we are in subquery/derived/CTE > + !table->pos_in_table_list->top_table()->select_lex->is_top_level_select()) > + { > + DBUG_ASSERT(0); // shoud not happened > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return false; > + } that's kind of questionable. if this condition can never be true, why do you have my_error() there? Just do asserts, like DBUG_ASSERT(table); DBUG_ASSERT(table->pos_in_table_list); DBUG_ASSERT(table->pos_in_table_list->select_lex); DBUG_ASSERT(table->pos_in_table_list->top_table()->select_lex->is_top_level_select()); > if (flags & NO_DEFAULT_VALUE_FLAG && > real_type() != MYSQL_TYPE_ENUM) > { > @@ -11177,12 +11187,29 @@ bool Field::save_in_field_default_value(bool view_error_processing) > bool Field::save_in_field_ignore_value(bool view_error_processing) > { > enum_sql_command com= table->in_use->lex->sql_command; > + > + if ( // table is not opened properly > + table == NULL || table->pos_in_table_list == NULL || > + table->pos_in_table_list->select_lex == NULL || > + // we are in subquery/derived/CTE > + !table->pos_in_table_list->top_table()->select_lex->is_top_level_select()) > + { > + DBUG_ASSERT(0); // shoud not happened > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return false; > + } ditto > // All insert-like commands > if (com == SQLCOM_INSERT || com == SQLCOM_REPLACE || > com == SQLCOM_INSERT_SELECT || com == SQLCOM_REPLACE_SELECT || > com == SQLCOM_LOAD) > return save_in_field_default_value(view_error_processing); > - return 0; // ignore > + if (com == SQLCOM_UPDATE || com == SQLCOM_UPDATE_MULTI) > + return 0; // ignore > + > + // unexpected command > + DBUG_ASSERT(0); // shoud not happened ditto but why it should not happen? What if one does SELECT DEFAULT(f) FROM t? > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return false; > } > > > diff --git a/sql/item.cc b/sql/item.cc > index 9451d4203ca..382eb3ac47a 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -3930,13 +3930,19 @@ int Item_param::save_in_field(Field *field, bool no_conversions) > case NULL_VALUE: > return set_field_to_null_with_conversions(field, no_conversions); > case DEFAULT_VALUE: > - return field->save_in_field_default_value(field->table->pos_in_table_list-> > + return field->save_in_field_default_value((field->table && > + field->table->pos_in_table_list)? in what case can field->table be NULL here? > + field->table->pos_in_table_list-> > top_table() != > - field->table->pos_in_table_list); > + field->table->pos_in_table_list: > + FALSE); > case IGNORE_VALUE: > - return field->save_in_field_ignore_value(field->table->pos_in_table_list-> > + return field->save_in_field_ignore_value((field->table && > + field->table->pos_in_table_list)? > + field->table->pos_in_table_list-> > top_table() != > - field->table->pos_in_table_list); > + field->table->pos_in_table_list: > + FALSE); > case NO_VALUE: > DBUG_ASSERT(0); // Should not be possible > return true; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] f854ac4d9e0: MDEV-21916: COM_STMT_BULK_EXECUTE with RETURNING insert wrong values
by Sergei Golubchik 02 Aug '20

02 Aug '20

Hi, Oleksandr! On Jul 29, Oleksandr Byelkin wrote: > revision-id: f854ac4d9e0 (mariadb-10.5.4-20-gf854ac4d9e0) > parent(s): 6cee9b1953b > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-07-03 13:39:49 +0200 > message: > > MDEV-21916: COM_STMT_BULK_EXECUTE with RETURNING insert wrong values > > To allocate new net buffer to avoid changing bufer we are reading. please add a more descriptive comment. I'm still not sure I understood correctly what the bug was. > diff --git a/sql/net_serv.cc b/sql/net_serv.cc > --- a/sql/net_serv.cc > +++ b/sql/net_serv.cc > @@ -178,6 +178,14 @@ my_bool my_net_init(NET *net, Vio *vio, void *thd, uint my_flags) > DBUG_RETURN(0); > } > > + > +inline void net_reset_new_packet(NET *net) static > +{ > + net->buff_end= net->buff + net->max_packet; > + net->write_pos= net->read_pos= net->buff; > +} > + > + > my_bool net_allocate_new_packet(NET *net, void *thd, uint my_flags) > { > DBUG_ENTER("net_allocate_new_packet"); > @@ -186,11 +194,26 @@ my_bool net_allocate_new_packet(NET *net, void *thd, uint my_flags) > NET_HEADER_SIZE + COMP_HEADER_SIZE + 1, > MYF(MY_WME | my_flags)))) > DBUG_RETURN(1); > - net->buff_end=net->buff+net->max_packet; > - net->write_pos=net->read_pos = net->buff; > + net_reset_new_packet(net); > DBUG_RETURN(0); > } > > +unsigned char *net_try_allocate_new_packet(NET *net, void *thd, uint my_flags) > +{ > + unsigned char * old_buff= net->buff; > + if (net_allocate_new_packet(net, thd, my_flags)) > + { > + /* > + We failed to allocate a new buffer => return the old buffer and > + return an error > + */ > + net->buff= old_buff; > + net_reset_new_packet(net); > + old_buff= NULL; wouldn't it be better to make net_allocate_new_packet not to change net->buff on failure? because currently you have exactly the same problem in sql_parse.cc (in 10.4) and your patch doesn't fix it. simply if (!(buff= (uchar*) my_malloc(...))) DBUG_RETURN(1); net->buff= buff; net->buff_end= net->buff + net->max_packet; net->write_pos= net->read_pos= net->buff; DBUG_RETURN(0); > + } > + return old_buff; > +} > + > > void net_end(NET *net) > { > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -685,7 +685,7 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > !table->prepare_triggers_for_delete_stmt_or_event()) > will_batch= !table->file->start_bulk_delete(); > > - if (returning) > + if (returning && !thd->get_stmt_da()->is_set()) how are result sets for bulk delete returning sent? > { > if (result->send_result_set_metadata(returning->item_list, > Protocol::SEND_NUM_ROWS | Protocol::SEND_EOF)) > diff --git a/sql/sql_error.cc b/sql/sql_error.cc > --- a/sql/sql_error.cc > +++ b/sql/sql_error.cc > @@ -380,16 +380,33 @@ Diagnostics_area::set_eof_status(THD *thd) > if (unlikely(is_error() || is_disabled())) > return; > > - /* > - If inside a stored procedure, do not return the total > - number of warnings, since they are not available to the client > - anyway. > - */ > - m_statement_warn_count= (thd->spcont ? > - 0 : > - current_statement_warn_count()); > + if (m_status == DA_EOF_BULK) > + { > + /* > + If inside a stored procedure, do not return the total > + number of warnings, since they are not available to the client > + anyway. > + */ > + if (!thd->spcont) > + m_statement_warn_count+= current_statement_warn_count(); > + } > + else > + { > + /* > + If inside a stored procedure, do not return the total > + number of warnings, since they are not available to the client > + anyway. > + */ did you have to repeat the same comment twice? > + if (thd->spcont) > + { > + m_statement_warn_count= 0; > + m_affected_rows= 0; affected rows too? old code didn't do that > + } > + else > + m_statement_warn_count= current_statement_warn_count(); > + m_status= (is_bulk_op() ? DA_EOF_BULK : DA_EOF); > + } > > - m_status= DA_EOF; > DBUG_VOID_RETURN; > } > > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -4364,24 +4368,37 @@ Prepared_statement::execute_bulk_loop(String *expanded_query, > if (state == Query_arena::STMT_ERROR) > { > my_message(last_errno, last_error, MYF(0)); > - thd->set_bulk_execution(0); > - return TRUE; > + goto err; > } > /* Check for non zero parameter count*/ > if (param_count == 0) > { > DBUG_PRINT("error", ("Statement with no parameters for bulk execution.")); > my_error(ER_UNSUPPORTED_PS, MYF(0)); > - thd->set_bulk_execution(0); > - return TRUE; > + goto err; > } > > if (!(sql_command_flags[lex->sql_command] & CF_SP_BULK_SAFE)) > { > DBUG_PRINT("error", ("Command is not supported in bulk execution.")); > my_error(ER_UNSUPPORTED_PS, MYF(0)); > - thd->set_bulk_execution(0); > - return TRUE; > + goto err; > + } > + /* > + Here second buffer for not optimized commands, > + optimized commands do it inside thier internal loop. > + */ > + if (!(sql_command_flags[lex->sql_command] & CF_SP_BULK_OPTIMIZED) && > + this->lex->has_returning()) > + { > + // Above check can be true for SELECT in future > + DBUG_ASSERT(lex->sql_command != SQLCOM_SELECT); > + if ((readbuff= > + net_try_allocate_new_packet(&thd->net, thd, > + MYF(MY_THREAD_SPECIFIC))) == NULL) sorry, I'm confused. You allocate a buffer here, in Prepared_statement::execute_bulk_loop, and then again in mysql_insert() ? > + { > + goto err; > + } > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] ca7e6f8d572: MDEV-19632 Replication aborts with ER_SLAVE_CONVERSION_FAILED upon CREATE ... SELECT in ORACLE mode
by Sergei Golubchik 31 Jul '20

31 Jul '20

Hi, Alexander! On Jul 28, Alexander Barkov wrote: > revision-id: ca7e6f8d572 (mariadb-10.3.21-178-gca7e6f8d572) > parent(s): f3f23b5c4bd > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2020-07-08 12:37:17 +0400 > message: > > MDEV-19632 Replication aborts with ER_SLAVE_CONVERSION_FAILED upon CREATE ... SELECT in ORACLE mode > > diff --git a/mysql-test/main/lowercase_fs_off.result b/mysql-test/main/lowercase_fs_off.result > index f2a8ec14641..3f0b08a78c4 100644 > --- a/mysql-test/main/lowercase_fs_off.result > +++ b/mysql-test/main/lowercase_fs_off.result > @@ -158,3 +158,13 @@ show triggers like '%T1%'; > Trigger Event Table Statement Timing Created sql_mode Definer character_set_client collation_connection Database Collation > drop table t1; > set GLOBAL sql_mode=default; > +# > +# MDEV-19632 Replication aborts with ER_SLAVE_CONVERSION_FAILED upon CREATE ... SELECT in ORACLE mode > +# > +# Compatibility schema names respect the filesystem case sensitivity I agree, it should. But this is not consistent with information_schema. So I'd suggest a comment in eq_name method, explaining why compatibility schema names respect the filesystem case sensitivity. (because, e.g., we eventually turn them into real schemas, on disk). > +CREATE TABLE t1 (a MARIADB_SCHEMA.date); > +ERROR HY000: Unknown data type: 'MARIADB_SCHEMA.date' > +CREATE TABLE t1 (a Mariadb_schema.date); > +ERROR HY000: Unknown data type: 'Mariadb_schema.date' > +CREATE TABLE t1 (a mariadb_schema.date); > +DROP TABLE t1; > diff --git a/mysql-test/suite/compat/oracle/t/type_date.test b/mysql-test/suite/compat/oracle/t/type_date.test > index 61f7aa53944..34adc5889b4 100644 > --- a/mysql-test/suite/compat/oracle/t/type_date.test > +++ b/mysql-test/suite/compat/oracle/t/type_date.test > @@ -2,3 +2,101 @@ SET sql_mode=ORACLE; ... > +--echo # > +--echo # Qualified syntax is not supported yet in SP > +--echo # an MDEV for that? > diff --git a/sql/field.cc b/sql/field.cc > index 8accfb35b0b..4ed38f2c06f 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -10407,6 +10407,19 @@ void Column_definition::set_attributes(const Lex_field_type_st &type, > set_handler(type.type_handler()); > charset= cs; > > +#if MYSQL_VERSION_ID > 100500 > +#error When merging to 10.5, please move the code below to > +#error Type_handler_timestamp_common::Column_definition_set_attributes() good > +#else > + /* > + Unlike other types TIMESTAMP fields are NOT NULL by default. > + Unless --explicit-defaults-for-timestamp is given. > + */ > + if (!opt_explicit_defaults_for_timestamp && > + type.type_handler()->field_type() == MYSQL_TYPE_TIMESTAMP) > + flags|= NOT_NULL_FLAG; > +#endif > + > if (type.length()) > { > int err; > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 0b423355170..3fb07601f99 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -2080,6 +2081,13 @@ int Lex_input_stream::scan_ident_start(THD *thd, Lex_ident_cli_st *str) > > uint length= yyLength(); > yyUnget(); // ptr points now after last token char > + > + if (thd->lex->parsing_options.lookup_keywords_after_qualifier) > + { > + if (int tokval= find_keyword(str, length, 0)) > + return tokval; > + } why is that? > + > str->set_ident(m_tok_start, length, is_8bit); > m_cpp_text_start= m_cpp_tok_start; > m_cpp_text_end= m_cpp_text_start + length; > @@ -8284,3 +8292,32 @@ bool LEX::tvc_finalize_derived() > current_select->linkage= DERIVED_TABLE_TYPE; > return tvc_finalize(); > } > + > + > +bool LEX::map_data_type(const Lex_ident_sys_st &schema_name, > + Lex_field_type_st *type) const > +{ > + const Schema *schema= schema_name.str ? > + Schema::find_by_name(schema_name) : > + Schema::find_implied(thd); > + if (!schema) > + { > + char buf[128]; > + const Name type_name= type->type_handler()->name(); > + my_snprintf(buf, sizeof(buf), "%.*s.%.*s", > + (int) schema_name.length, schema_name.str, > + (int) type_name.length(), type_name.ptr()); > +#if MYSQL_VERSION_ID > 100500 > +#error Please remove the old code > + my_error(ER_UNKNOWN_DATA_TYPE, MYF(0), buf); > +#else > + my_printf_error(ER_UNKNOWN_ERROR, "Unknown data type: '%-.64s'", > + MYF(0), buf); > +#endif > + return true; > + } > + const Type_handler *mapped= schema->map_data_type(thd, type->type_handler()); > + if (mapped != type->type_handler()) > + type->set_handler(mapped); why do you need an if() here? type->set_handler(mapped) is not that expensive. > + return false; > +} > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 3dbc7724928..fcc2d44fdd6 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -2224,6 +2224,13 @@ int show_create_table(THD *thd, TABLE_LIST *table_list, String *packet, > append_identifier(thd, packet, &field->field_name); > packet->append(' '); > > + const Type_handler *th= field->type_handler(); > + const Schema *implied_schema= Schema::find_implied(thd); > + if (th != implied_schema->map_data_type(thd, th)) > + { > + packet->append(th->schema()->name(), system_charset_info); sorry, I don't see it. How can th->schema() return not mariadb_schema? > + packet->append(STRING_WITH_LEN("."), system_charset_info); > + } > type.set(tmp, sizeof(tmp), system_charset_info); > field->sql_type(type); > packet->append(type.ptr(), type.length(), system_charset_info); > diff --git a/storage/test_sql_discovery/mysql-test/sql_discovery/simple.result b/storage/test_sql_discovery/mysql-test/sql_discovery/simple.result > index 23b7804638f..1feea5e47ee 100644 > --- a/storage/test_sql_discovery/mysql-test/sql_discovery/simple.result > +++ b/storage/test_sql_discovery/mysql-test/sql_discovery/simple.result > @@ -82,7 +82,7 @@ select * from t1; > ERROR HY000: Engine TEST_SQL_DISCOVERY failed to discover table `test`.`t1` with 'create table t1 (a uint)' > show warnings; > Level Code Message > -Error 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'uint)' at line 1 > +Error 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ')' at line 1 this is not ideal :( can you try to keep the old error message? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

[Maria-developers] Review for: MDEV-17399 Add support for JSON_TABLE, part #7
by Sergey Petrunia 30 Jul '20

30 Jul '20

Hi Alexey, On Thu, Jul 16, 2020 at 04:34:46PM +0400, Alexey Botchkov wrote: > > In the new patch i think i addressed most of your comments. > https://github.com/MariaDB/server/commit/d3a311a8e9508ef94dddb6b7f4d366337a… > ... > > The above doesn't do proper quoting, which is wrong. Testcase: > > > > select * > > from > > json_table('[{"co\\\\lor": "blue", "price": 50}]', > > '$[*]' columns( color varchar(100) path '$.co\\\\lor') > > ) as T; > > +-------+ > > | color | > > +-------+ > > | blue | > > +-------+ > > > > create view v2 as > > select * > > from > > json_table('[{"co\\\\lor": "blue", "price": 50}]', > > '$[*]' columns( color varchar(100) path '$.co\\\\lor') > > ) as T; > > select * from v2; Quoting is still an issue apparently: create view v1 as select * from json_table('[{"color": "blue", "price": 50}]', '$[*]' columns(color text path '$.nonexistent', seq for ordinality ) ) as `ALIAS NOT QUOTED`; select * from v1; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'NOT QUOTED' at line 1 create view v2 as select * from json_table('[{"color": "blue", "price": 50}, {"color": "red", "price": 100}]', '$[*]' columns(color text path '$.QUOTES " HERE "', seq for ordinality ) ) as T; select * from v2; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'HERE "", `seq` FOR ORDINALITY)) T' at line 1 Could you please go through the printout functions and make sure all printed literals are properly escaped/quoted? Printout code for other functions can be used as an example. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

Re: [Maria-developers] 2b527a4e858: MDEV-17729: Assertion `! is_set() || m_can_overwrite_status' failed in
by Sergei Golubchik 30 Jul '20

30 Jul '20

Hi, Rucha! On Jul 30, Rucha Deodhar wrote: > > > > > > - if ((level == Sql_condition::WARN_LEVEL_WARN) && > > > - really_abort_on_warning()) > > > + if ((killed == NOT_KILLED) && (level == Sql_condition::WARN_LEVEL_WARN) && > > > + really_abort_on_warning()) > > > > I don't understand this. I presume the assert that fails is in > > da->set_error_status(), called below as > > > > Yes, the assertion fails in set_error_status(). > > > if (!da->is_error()) > > { > > set_row_count_func(-1); > > da->set_error_status(sql_errno, msg, sqlstate, cond); > > } > > > > but if(!da->is_error()) should prevent set_error_status() from being > > called when it would need to overwrite the status. So how comes it is > > still called and triggers the assert? > > The status does not change to DA_ERROR yet, it changes in > set_error_status(), so !thd->is_error() becomes true and does not prevent > set_error_status() from being called. The reason we get assertion failure > in set_error_status() is because !is_set() is false (because status is > DA_OK) and m_can_overwrite_status is false too. I see. I also was confused by how can THD::raise_condition() be called after the status is already DA_OK? An error should not happen after DA_OK. But it turns out it was not just any error, it was push_warning() with the message "Query execution was interrupted. The query examined at least 4 rows, which exceeds LIMIT ROWS EXAMINED (2). The query result may be incomplete". This warning should not be converted to an error, obviously. It is converted, because thd->abort_on_warning is true. It's set to true in select_insert::prepare(). And the warning is pushed in handle_select(). Perhaps this warning should never have thd->abort_on_warning ? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2b527a4e858: MDEV-17729: Assertion `! is_set() || m_can_overwrite_status' failed in
by Sergei Golubchik 30 Jul '20

30 Jul '20

Hi, Rucha! On Jul 30, Rucha Deodhar wrote: > revision-id: 2b527a4e858 (mariadb-10.1.43-218-g2b527a4e858) > parent(s): 57ec42bc321 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-07-27 09:50:00 +0530 > message: > > MDEV-17729: Assertion `! is_set() || m_can_overwrite_status' failed in > Diagnostics_area::set_error_status > > Analysis: When strict mode is enabled, all warnings are converted to errors > even when there is no bad data and the query is already killed. > Fix: If the query is already killed because examined rows exceeded the limit, > the warning is not because of bad data and should not be converted to error. > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 6897a26bda1..3591695a22d 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1156,13 +1156,17 @@ Sql_condition* THD::raise_condition(uint sql_errno, > if (sqlstate == NULL) > sqlstate= mysql_errno_to_sqlstate(sql_errno); > > - if ((level == Sql_condition::WARN_LEVEL_WARN) && > - really_abort_on_warning()) > + if ((killed == NOT_KILLED) && (level == Sql_condition::WARN_LEVEL_WARN) && > + really_abort_on_warning()) I don't understand this. I presume the assert that fails is in da->set_error_status(), called below as if (!da->is_error()) { set_row_count_func(-1); da->set_error_status(sql_errno, msg, sqlstate, cond); } but if(!da->is_error()) should prevent set_error_status() from being called when it would need to overwrite the status. So how comes it is still called and triggers the assert? > { > - /* > - FIXME: > - push_warning and strict SQL_MODE case. > - */ > level= Sql_condition::WARN_LEVEL_ERROR; > set_killed(KILL_BAD_DATA); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] bc73b455ba2: MDEV-15703 Crash in EXECUTE IMMEDIATE 'CREATE OR REPLACE TABLE t1 (a INT DEFAULT ?)' USING DEFAULT
by Sergei Golubchik 29 Jul '20

29 Jul '20

Hi, Oleksandr! On Jul 29, Oleksandr Byelkin wrote: > revision-id: bc73b455ba2 (mariadb-10.2.31-318-gbc73b455ba2) > parent(s): 7311586ca25 > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-07-17 13:47:26 +0200 > message: > > MDEV-15703 Crash in EXECUTE IMMEDIATE 'CREATE OR REPLACE TABLE t1 (a INT DEFAULT ?)' USING DEFAULT > > part 2: ... > diff --git a/sql/field.h b/sql/field.h > index d5e2cb25788..2755f5f687c 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -51,6 +51,16 @@ enum enum_check_fields > CHECK_FIELD_ERROR_FOR_NULL > }; > > + > +enum ignore_value_reaction > +{ > + IGNORE_MEANS_ERROR, > + IGNORE_MEANS_DEFAULT, > + IGNORE_MEANS_FIELD_VALUE I thought you'll have IGNORE_MEANS_IGNORE :) > +}; > + > +ignore_value_reaction find_ignore_reaction(THD *thd); > + > /* > Common declarations for Field and Item > */ > diff --git a/sql/item.h b/sql/item.h > index 7338c8be47b..dabf237712b 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1404,6 +1424,16 @@ class Item: public Value_source, > LOWEST_PRECEDENCE); > } > virtual void print(String *str, enum_query_type query_type); > + > + class Print: public String > + { > + public: > + Print(Item *item, enum_query_type type) > + { > + item->print(this, type); > + } no, I don't think this should be part of the bugfix. > + }; > + > void print_item_w_name(String *str, enum_query_type query_type); > void print_value(String *str); > > @@ -2077,6 +2107,15 @@ class Item: public Value_source, > { > marker &= ~EXTRACTION_MASK; > } > + > + virtual bool associate_with_target_field(THD *thd __attribute__((unused)), > + Item_field *field > + __attribute__((unused))) > + { > + DBUG_ASSERT(fixed); > + DBUG_ASSERT(fixed); you want to be double-sure? :) > + return false; > + } > }; > > > diff --git a/sql/item.cc b/sql/item.cc > index 382eb3ac47a..d11017f3ca6 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -84,6 +85,15 @@ void item_init(void) > } > > > +void Item::raise_error_not_evaluable() > +{ > + Item::Print tmp(this, QT_ORDINARY); > + // TODO-10.5: add an error message to errmsg-utf8.txt > + my_printf_error(ER_UNKNOWN_ERROR, > + "'%s' is not allowed in this context", MYF(0), tmp.ptr()); I don't see any tests for this error > +} > + > + > void Item::push_note_converted_to_negative_complement(THD *thd) > { > push_warning(thd, Sql_condition::WARN_LEVEL_NOTE, ER_UNKNOWN_ERROR, > @@ -8945,6 +9022,14 @@ bool Item_default_value::fix_fields(THD *thd, Item **items) > return FALSE; > } > > + return tie_field(thd); > +} > + > + comment? what does tie_field do? > +bool Item_default_value::tie_field(THD *thd) > +{ > + Item *real_arg; > + Item_field *field_arg; > /* > DEFAULT() do not need table field so should not ask handler to bring > field value (mark column for read) > @@ -9131,8 +9241,54 @@ void Item_ignore_value::print(String *str, enum_query_type query_type) > str->append(STRING_WITH_LEN("ignore")); > } > > + > +bool Item_ignore_value::associate_with_target_field(THD *thd, > + Item_field *field) > +{ > + bitmap_set_bit(field->field->table->read_set, field->field->field_index); may be, remove it from the write_set instead? > + return Item_default_value::associate_with_target_field(thd, field); > +} > + > + > int Item_ignore_value::save_in_field(Field *field_arg, bool no_conversions) > { > + if (arg) > + { > + switch (find_ignore_reaction(field->table->in_use)) > + { > + case IGNORE_MEANS_DEFAULT: > + DBUG_ASSERT(0); // impossible now, but fully working code if needed Impossible? One cannot use IGNORE in INSERT? Not even in EXECUTE IMMEDIATE "INSERT ... " USING IGNORE? > + /* > + save default value > + Note: (((Field*)(arg()->real_item()) is Item_field* (checked in > + tie_field(). > + */ > + if ((((Item_field*)(arg->real_item()))->field->flags & > + NO_DEFAULT_VALUE_FLAG)) > + { > + my_error(ER_NO_DEFAULT_FOR_FIELD, MYF(0), > + ((Item_field*)(arg->real_item()))->field->field_name); > + return true; > + } > + calculate(); > + return Item_field::save_in_field(field_arg, no_conversions); > + case IGNORE_MEANS_FIELD_VALUE: > + // checked on tie_field > + DBUG_ASSERT(arg->real_item()->type() == FIELD_ITEM); > + /* > + save original field (Item::save_in_field is not applicable because > + result_field for the referenced field set to this temporary table). > + */ > + arg->save_val(field_arg); > + return false; > + default: > + ; // fall through to error > + } > + DBUG_ASSERT(0); //impossible > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return true; > + } > + > return field_arg->save_in_field_ignore_value(context->error_processor == > &view_error_processor); > } > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 436f753557e..4dbda30708a 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -8340,6 +8341,9 @@ fill_record(THD *thd, TABLE *table, Field **ptr, List<Item> &values, > field->field_name, table->s->table_name.str); > } > > + if(evaluable && value->check_is_evaluable_expression_or_error()) > + goto err; this looks like the wrong way of doing it. whether an item is "evaluable" or not is metadata. It should be checked once before the statement execution > + > if (use_value) > value->save_val(field); > else > diff --git a/sql/sql_update.cc b/sql/sql_update.cc > index 65a828147ae..26ee427e3e7 100644 > --- a/sql/sql_update.cc > +++ b/sql/sql_update.cc > @@ -1796,6 +1796,9 @@ int multi_update::prepare(List<Item> &not_used_values, > while ((item= (Item_field *) field_it++)) > { > Item *value= value_it++; > + if (value->associate_with_target_field(thd, item)) > + DBUG_RETURN(1); what happens for single table updates? > + > uint offset= item->field->table->pos_in_table_list->shared; > fields_for_table[offset]->push_back(item, thd->mem_root); > values_for_table[offset]->push_back(value, thd->mem_root); > diff --git a/sql/unireg.cc b/sql/unireg.cc > index 083960523c1..a3bfc678e07 100644 > --- a/sql/unireg.cc > +++ b/sql/unireg.cc > @@ -1000,8 +1000,14 @@ static bool make_empty_rec(THD *thd, uchar *buff, uint table_options, > { > Item *expr= field->default_value->expr; > > - int res= !expr->fixed && // may be already fixed if ALTER TABLE > - expr->fix_fields(thd, &expr); > + int res= MY_TEST(!expr->fixed && // may be already fixed if ALTER TABLE > + expr->fix_fields(thd, &expr)); eh? what's the point of this change? > + if (!res && expr->check_is_evaluable_expression_or_error()) > + { > + error= 1; > + delete regfield; //To avoid memory leak > + goto err; > + } > if (!res) > res= expr->save_in_field(regfield, 1); > if (!res && (field->flags & BLOB_FLAG)) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Progress Report GSoC 2020 (20-26 Jule)
by Anton Mikhaylenko 28 Jul '20

28 Jul '20

Good day, everybody. In this post, I want to describe what I did this week, Jule 20-26. This week, I was refactoring. I added a flag for logging the number of threads and corrected the calculation of completed tasks. Also added comments to the code. Regards, Anton MIkhailenko

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 27 Jul '20

27 Jul '20

Hi, This week - Resolved the abi_check issue with different compilers. Also added checks and updated documentation so that it does not happen again. - Started writing some tests for the lzma service. - Finished the first draft of the lzo service. - Worked on some refactors. - Discussed the idea of adding status variables to allow the storage engines to know which libraries are loaded. This will also allow the user to view the libraries that are loaded while the server is running. Next week - Finish writing tests for the lzma service. - Finalize the implementation of the service. - Start replicating the implementation for other libraries. - Discuss if we should change all to auto for the --use-compression= switch. That way, the server can throw an error if it is started with --use-compression=lzma but not if it is started normally. This will also allow for a configuration like --use-compression=lzma,auto so that the server will try to load all libraries but will not start unless lzma is loaded. Thanks, Kartik

1 0

[Maria-developers] 9b999e79a35: MDEV-23108: Point in time recovery of binary log fails when sql_mode=ORACLE
by sujatha 24 Jul '20

24 Jul '20

revision-id: 9b999e79a358de2c86ae44fe83cf6c62cff1d2ef (mariadb-10.3.21-180-g9b999e79a35) parent(s): 73aa31fbfd793bdb597bb19c52118ab4e637f4bc author: Sujatha committer: Sujatha timestamp: 2020-07-17 12:23:32 +0530 message: MDEV-23108: Point in time recovery of binary log fails when sql_mode=ORACLE Problem: ======== During point in time recovery of binary log syntax error is reported for BEGIN statement and recovery fails. Analysis: ========= In MariaDB 10.3 and later, setting the sql_mode system variable to Oracle allows the server to understand a subset of Oracle's PL/SQL language. When sql_mode=ORACLE is set, it switches the parser from the MariaDB parser to Oracle compatible parser. With this change 'BEGIN' is not considered as 'START TRANSACTION'. Hence the syntax error is reported. Fix: === Add a new option to 'mysqlbinlog' tool named 'sql_mode_oracle'. When 'sql_mode_oracle' option is specified 'BEGIN' statements will be replaced with 'START TRANSACTION' in the mysqlbinlog output. --- client/mysqlbinlog.cc | 7 +- .../compat/oracle/r/binlog_ptr_mysqlbinlog.result | 100 +++++++++++++++++ .../oracle/t/binlog_ptr_mysqlbinlog-master.opt | 1 + .../compat/oracle/t/binlog_ptr_mysqlbinlog.test | 120 +++++++++++++++++++++ sql/log_event.cc | 15 +-- sql/log_event.h | 1 + 6 files changed, 237 insertions(+), 7 deletions(-) diff --git a/client/mysqlbinlog.cc b/client/mysqlbinlog.cc index a732a6ab8c3..ad995629272 100644 --- a/client/mysqlbinlog.cc +++ b/client/mysqlbinlog.cc @@ -145,6 +145,7 @@ static MYSQL* mysql = NULL; static const char* dirname_for_local_load= 0; static bool opt_skip_annotate_row_events= 0; +static my_bool opt_sql_mode_oracle= 0; static my_bool opt_flashback; #ifdef WHEN_FLASHBACK_REVIEW_READY static my_bool opt_flashback_review; @@ -1045,7 +1046,8 @@ Exit_status process_event(PRINT_EVENT_INFO *print_event_info, Log_event *ev, DBUG_ENTER("process_event"); Exit_status retval= OK_CONTINUE; IO_CACHE *const head= &print_event_info->head_cache; - + /* Propogate sql_mode_oracle settings to event */ + ev->is_sql_mode_oracle= opt_sql_mode_oracle; /* Bypass flashback settings to event */ ev->is_flashback= opt_flashback; #ifdef WHEN_FLASHBACK_REVIEW_READY @@ -1704,6 +1706,9 @@ static struct my_option my_options[] = "use --base64-output=never", &short_form, &short_form, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, + {"sql_mode_oracle", 0, "Converts BEGIN statement to 'ORALCE' " + "sql_mode compatible START TRANSACTION.", &opt_sql_mode_oracle, + &opt_sql_mode_oracle, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, {"socket", 'S', "The socket file to use for connection.", &sock, &sock, 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, diff --git a/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result b/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result new file mode 100644 index 00000000000..3080a4b69a0 --- /dev/null +++ b/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result @@ -0,0 +1,100 @@ +SET @@SQL_MODE = 'ORACLE'; +########################################################################## +# Test verifies Gtid_log_event/Xid_log_event specific print # +########################################################################## +CREATE TABLE tm (f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +CREATE OR REPLACE PROCEDURE simpleproc (param1 OUT INT) AS +BEGIN +SELECT COUNT(*) INTO param1 FROM t; +END; +/ +CREATE FUNCTION f1 RETURN INT +AS +BEGIN +RETURN 10; +END; +/ +FLUSH LOGS; +########################################################################## +# Delete data from master so that it can be restored from binlog # +########################################################################## +DROP FUNCTION f1; +DROP PROCEDURE simpleproc; +DROP TABLE tm; +DROP TABLE t; +########################################################################## +# Post recovery using mysqlbinlog --sql-mode-oracle # +########################################################################## +SHOW TABLES; +Tables_in_test +t +tm +SELECT * FROM tm; +f +10 +SELECT * FROM t; +f +10 +SELECT f1(); +f1() +10 +CALL simpleproc(@a); +SELECT @a; +@a +1 +"***** Clean Up *****" +DROP TABLE t,tm; +DROP PROCEDURE simpleproc; +DROP FUNCTION f1; +RESET MASTER; +########################################################################## +# Test verifies Gtid_log_event/Xid_log_event/Qery_log_event # +# specific print along with flashback option # +########################################################################## +CREATE TABLE tm(f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); +INSERT INTO tm VALUES (20); +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +INSERT INTO t VALUES (20); +########################################################################## +# Initial data # +########################################################################## +SELECT * FROM tm; +f +10 +20 +SELECT * FROM t; +f +10 +20 +FLUSH LOGS; +DELETE FROM tm WHERE f=20; +DELETE FROM t WHERE f=20; +FLUSH LOGS; +########################################################################## +# Data after deletion # +########################################################################## +SELECT * FROM tm; +f +10 +SELECT * FROM t; +f +10 +FOUND 2 /START TRANSACTION/ in test_sql_mode.sql +########################################################################## +# Data after recovery using flashback, sql-mode-oracle # +########################################################################## +SELECT * FROM tm; +f +10 +20 +SELECT * FROM t; +f +10 +20 +"***** Clean Up *****" +DROP TABLE t,tm; diff --git a/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog-master.opt b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog-master.opt new file mode 100644 index 00000000000..8f0cc182f51 --- /dev/null +++ b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog-master.opt @@ -0,0 +1 @@ +--flashback diff --git a/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test new file mode 100644 index 00000000000..07a1b5c4fdb --- /dev/null +++ b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test @@ -0,0 +1,120 @@ +# ==== Purpose ==== +# +# Test verifies that a binary log which was generated by setting +# 'sql_mod=ORACLE' can be succefully used for point in time recovery with +# following options. +# 1) mysqlbinlog --sql-mode-oracle +# 2) mysqlbinlog --flashback --sql-mode-oracle +# +# BEGIN statement is printed in three places +# 1) "Gtid_log_event::print" +# 2) "Xid_log_event::print" if flashback is enabled +# 3) "Query_log_event::print" if flashback is enabled and engine is +# non-transacional. +# +# Test verifies all these cases. +# +# ==== References ==== +# +# MDEV-23108: Point in time recovery of binary log fails when sql_mode=ORACLE +# +--source include/have_log_bin.inc +--source include/have_innodb.inc + +let $MYSQLD_DATADIR= `select @@datadir`; +SET @@SQL_MODE = 'ORACLE'; + +--echo ########################################################################## +--echo # Test verifies Gtid_log_event/Xid_log_event specific print # +--echo ########################################################################## +CREATE TABLE tm (f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); + +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); + +DELIMITER /; +CREATE OR REPLACE PROCEDURE simpleproc (param1 OUT INT) AS + BEGIN + SELECT COUNT(*) INTO param1 FROM t; + END; +/ +CREATE FUNCTION f1 RETURN INT +AS +BEGIN + RETURN 10; +END; +/ +DELIMITER ;/ + +FLUSH LOGS; +--echo ########################################################################## +--echo # Delete data from master so that it can be restored from binlog # +--echo ########################################################################## +DROP FUNCTION f1; +DROP PROCEDURE simpleproc; +DROP TABLE tm; +DROP TABLE t; + +--exec $MYSQL_BINLOG --sql-mode-oracle $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/test.sql +--exec $MYSQL < $MYSQLTEST_VARDIR/tmp/test.sql + +--echo ########################################################################## +--echo # Post recovery using mysqlbinlog --sql-mode-oracle # +--echo ########################################################################## +SHOW TABLES; +SELECT * FROM tm; +SELECT * FROM t; +--horizontal_results +SELECT f1(); +CALL simpleproc(@a); +SELECT @a; + +--echo "***** Clean Up *****" +DROP TABLE t,tm; +DROP PROCEDURE simpleproc; +DROP FUNCTION f1; +--remove_file $MYSQLTEST_VARDIR/tmp/test.sql +RESET MASTER; + +--echo ########################################################################## +--echo # Test verifies Gtid_log_event/Xid_log_event/Qery_log_event # +--echo # specific print along with flashback option # +--echo ########################################################################## +CREATE TABLE tm(f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); +INSERT INTO tm VALUES (20); +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +INSERT INTO t VALUES (20); +--echo ########################################################################## +--echo # Initial data # +--echo ########################################################################## +SELECT * FROM tm; +SELECT * FROM t; +FLUSH LOGS; +DELETE FROM tm WHERE f=20; +DELETE FROM t WHERE f=20; +FLUSH LOGS; + +--echo ########################################################################## +--echo # Data after deletion # +--echo ########################################################################## +SELECT * FROM tm; +SELECT * FROM t; +--exec $MYSQL_BINLOG --flashback --sql-mode-oracle $MYSQLD_DATADIR/master-bin.000002 > $MYSQLTEST_VARDIR/tmp/test_sql_mode.sql + +--let SEARCH_FILE=$MYSQLTEST_VARDIR/tmp/test_sql_mode.sql +--let SEARCH_PATTERN=START TRANSACTION +--source include/search_pattern_in_file.inc +--exec $MYSQL < $MYSQLTEST_VARDIR/tmp/test_sql_mode.sql + +--echo ########################################################################## +--echo # Data after recovery using flashback, sql-mode-oracle # +--echo ########################################################################## +SELECT * FROM tm; +SELECT * FROM t; + +--echo "***** Clean Up *****" +DROP TABLE t,tm; +--remove_file $MYSQLTEST_VARDIR/tmp/test_sql_mode.sql diff --git a/sql/log_event.cc b/sql/log_event.cc index 57a39085e86..e06e6a1b1d6 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -5358,8 +5358,9 @@ bool Query_log_event::print(FILE* file, PRINT_EVENT_INFO* print_event_info) } else if (strcmp("COMMIT", query) == 0) { - if (my_b_write(&cache, (uchar*) "BEGIN", 5) || - my_b_printf(&cache, "\n%s\n", print_event_info->delimiter)) + if (my_b_printf(&cache, is_sql_mode_oracle ? + "START TRANSACTION\n%s\n" : "BEGIN\n%s\n" , + print_event_info->delimiter)) goto err; } } @@ -8254,9 +8255,10 @@ Gtid_log_event::print(FILE *file, PRINT_EVENT_INFO *print_event_info) goto err; } if (!(flags2 & FL_STANDALONE)) - if (my_b_printf(&cache, is_flashback ? "COMMIT\n%s\n" : "BEGIN\n%s\n", print_event_info->delimiter)) + if (my_b_printf(&cache, is_flashback ? "COMMIT\n%s\n" : + (is_sql_mode_oracle ? "START TRANSACTION\n%s\n" : + "BEGIN\n%s\n"), print_event_info->delimiter)) goto err; - return cache.flush_data(); err: return 1; @@ -8937,8 +8939,9 @@ bool Xid_log_event::print(FILE* file, PRINT_EVENT_INFO* print_event_info) my_b_printf(&cache, "\tXid = %s\n", buf)) goto err; } - if (my_b_printf(&cache, is_flashback ? "BEGIN%s\n" : "COMMIT%s\n", - print_event_info->delimiter)) + if (my_b_printf(&cache, is_flashback ? (is_sql_mode_oracle ? + "START TRANSACTION\n%s\n" : "BEGIN\n%s\n") : + "COMMIT\n%s\n", print_event_info->delimiter)) goto err; return cache.flush_data(); diff --git a/sql/log_event.h b/sql/log_event.h index 73809953d70..66e29579fbc 100644 --- a/sql/log_event.h +++ b/sql/log_event.h @@ -1270,6 +1270,7 @@ class Log_event /* The following code used for Flashback */ #ifdef MYSQL_CLIENT + my_bool is_sql_mode_oracle; my_bool is_flashback; my_bool need_flashback_review; String output_buf; // Storing the event output

3 7

Re: [Maria-developers] 0dd0bbe54e1: MDEV-14836: Assertion `m_status == DA_ERROR' failed in
by Sergei Golubchik 22 Jul '20

22 Jul '20

Hi, Rucha! On Jul 22, Rucha Deodhar wrote: > revision-id: 0dd0bbe54e1 (mariadb-10.2.31-307-g0dd0bbe54e1) > parent(s): 0994af43e58 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-07-16 22:29:06 +0530 > message: > > MDEV-14836: Assertion `m_status == DA_ERROR' failed in > Diagnostics_area::sql_errno upon query from I_S with LIMIT ROWS EXAMINED > > Query is aborted because number of examined rows exceeds LIMIT EXAMINED ROWS. > Even though this is not a real error, open_normal_and_derived_tables() > returns true. As a result, thd->is_errno() is called and thus the assertion > failure. > > diff --git a/mysql-test/r/information_schema.result b/mysql-test/r/information_schema.result > index 2d404aff6bc..7644ff4e7a2 100644 > --- a/mysql-test/r/information_schema.result > +++ b/mysql-test/r/information_schema.result > @@ -2184,3 +2184,16 @@ SCHEMA_NAME > # > # End of 10.1 tests > # > +# > +# Start of 10.2 Test > +# > +# MDEV-14836: Assertion `m_status == DA_ERROR' failed in > +# Diagnostics_area::sql_errno upon query from I_S with LIMIT ROWS EXAMINED > +# > +SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` LIMIT ROWS EXAMINED 10; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE DATETIME_PRECISION CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT IS_GENERATED GENERATION_EXPRESSION > +Warnings: > +Warning 1931 Query execution was interrupted. The query examined at least 666 rows, which exceeds LIMIT ROWS EXAMINED (10). The query result may be incomplete I don't understand why there are no rows in the output here? It has examined quite a lot of rows, I would expect a truncated result, but not a completely empty one. Could you check that out, please? > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 436f753557e..8f1c0ebab29 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -1590,6 +1590,12 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > if (!(flags & MYSQL_OPEN_IGNORE_KILLED) && thd->killed) > { > thd->send_kill_message(); > + /* > + Query was only aborted, but this is not an actual error. > + So return false. > + */ > + if (thd->killed == ABORT_QUERY) > + DBUG_RETURN(FALSE); > DBUG_RETURN(TRUE); This looks somewhat risky. open_table() returns TRUE on error and FALSE on success. You here make it to return FALSE even if the table wasn't opened. I am not sure the caller will always be able to handle that. On the other hand, normally all tables are opened first, and then the server starts reading the data. It's only in I_S the server opens tables one by one. So get_all_tables() could be the only possibility for ABORT_QUERY to happen before open_table(). In that case it is safer to add a check for thd->killed into get_all_tables(). Could you try that approach instead? Also, note, don't compare thd->killed with ABORT_QUERY, use thd->killed_errno() method instead. > } > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 18b7e92bca5..9f834ee118b 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -5622,6 +5622,16 @@ static int get_schema_column_record(THD *thd, TABLE_LIST *tables, > } > DBUG_RETURN(res); > } > + /* > + Query was aborted. This could be because of number of > + examined rows exceeded LIMIT ROWS EXAMINED. But don't > + issue warning here. It is done later, in handle_select(). > + */ > + if(thd->killed == ABORT_QUERY) > + { > + res= 0; > + DBUG_RETURN(res); > + } > > show_table= tables->table; > count= 0; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] 27b046d2e95: MDEV-14203: rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist failed in buildbot with a warning
by sujatha 21 Jul '20

21 Jul '20

revision-id: 27b046d2e95add5ba87e231279d2a7774db5b0c7 (mariadb-10.2.31-317-g27b046d2e95) parent(s): a1e52e7f32cee7c204078db4c4beb88250d7e1f6 author: Sujatha committer: Sujatha timestamp: 2020-07-21 11:16:40 +0530 message: MDEV-14203: rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist failed in buildbot with a warning Problem: ======= rpl.rpl_slave_load_tmpdir_not_exist 'stmt' w3 [ fail ] Found warnings/errors in server log file! Test ended at 2017-09-27 20:34:55 [Warning] Master is configured to log replication events with checksum, but will not send such events to slaves that cannot process them ^ Found warnings in /mnt/buildbot/build/mariadb-10.2.10/mysql-test/var/3/log/mysqld.1.err ok Analysis: ======== When slave tries to connect to master 'get_master_version_and_clock' function is invoked to perform elaborated slave-master handshake. During this process slave server queries master server, to know if it is checksum aware and at the same time master is notified about its CRC-awareness. The master's side instant value of @@global.binlog_checksum is stored in the dump thread's uservar area as well as cached locally to become known in consensus by master and slave. Post hand-shake slave requests master for binlog dump. It sends 'COM_BINLOG_DUMP'. This command is sent to master by 'cli_advanced_command' call. If there is some temporary network failure during this request_dump call, 'end_server' is invoked to close the current connection between master and slave. Upon connection close the dump thread on the master gets terminated and it clears the 'uservar' data it got through master-slave handshake. The 'COM_BINLOG_DUMP' command is sent once again without master-slave handshake. Since the checksum data is not available with new dump thread a warning gets reported. Fix: === Upon network write error donot attempt reconnect, proceed to master-slave handshake. This ensures that master is aware of slave's capability to use checksums. --- .../rpl/r/rpl_dump_request_retry_warning.result | 22 ++++++++ .../rpl/t/rpl_dump_request_retry_warning.test | 60 ++++++++++++++++++++++ sql-common/client.c | 2 + sql/net_serv.cc | 13 +++++ sql/slave.cc | 3 +- 5 files changed, 99 insertions(+), 1 deletion(-) diff --git a/mysql-test/suite/rpl/r/rpl_dump_request_retry_warning.result b/mysql-test/suite/rpl/r/rpl_dump_request_retry_warning.result new file mode 100644 index 00000000000..3a80d5b5f31 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_dump_request_retry_warning.result @@ -0,0 +1,22 @@ +include/master-slave.inc +[connection master] +CREATE TABLE t1(a int); +INSERT INTO t1 VALUES(1); +connection slave; +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= 'd,simulate_error_on_packet_write'; +START SLAVE; +SET DEBUG_SYNC= 'now WAIT_FOR parked'; +SET @@GLOBAL.debug_dbug = @saved_dbug; +SET DEBUG_SYNC= 'now SIGNAL continue'; +SET DEBUG_SYNC= 'RESET'; +include/wait_for_slave_io_to_start.inc +include/wait_for_slave_sql_to_start.inc +connection master; +include/sync_slave_sql_with_master.inc +SELECT * FROM t1; +a +1 +connection master; +DROP TABLE t1; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_dump_request_retry_warning.test b/mysql-test/suite/rpl/t/rpl_dump_request_retry_warning.test new file mode 100644 index 00000000000..d9fb8258765 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_dump_request_retry_warning.test @@ -0,0 +1,60 @@ +# ==== Purpose ==== +# +# Test verifies that, due to a temporary network error, if request dump +# command specific packet write operation fails then the write error gets +# handled appropriately. Further retry will be initiated with appropriate +# slave registration on master. This will ensure that master has all the +# details of slave and no warnings are reported on the master side. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Skip the slave start. +# 1 - Enable debug simulation which will simulate packet write error during +# dump request command execution. +# 2 - Start the slave. Observe that slave is able to reconnect post +# temporary network write error. +# +# ==== References ==== +# +# MDEV-14203: rpl.rpl_extra_col_master_myisam, +# rpl.rpl_slave_load_tmpdir_not_exist failed in buildbot with a +# warning +# +# MDEV-13258: rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master +# failed in buildbot +# + +--source include/have_debug.inc +--source include/have_debug_sync.inc +--let $rpl_skip_start_slave=1 +--source include/master-slave.inc + +# Do an insert on master +CREATE TABLE t1(a int); +INSERT INTO t1 VALUES(1); + +# Add a debug point and start the slave so that dump request fails. +connection slave; +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= 'd,simulate_error_on_packet_write'; + +START SLAVE; +SET DEBUG_SYNC= 'now WAIT_FOR parked'; +SET @@GLOBAL.debug_dbug = @saved_dbug; +SET DEBUG_SYNC= 'now SIGNAL continue'; +SET DEBUG_SYNC= 'RESET'; + +--source include/wait_for_slave_io_to_start.inc +--source include/wait_for_slave_sql_to_start.inc + +# Sync the slave and verify that slave has caught up with the master. +connection master; +--source include/sync_slave_sql_with_master.inc +SELECT * FROM t1; + +# Cleanup +connection master; +DROP TABLE t1; + +--source include/rpl_end.inc diff --git a/sql-common/client.c b/sql-common/client.c index 227759f4163..b66eee2a508 100644 --- a/sql-common/client.c +++ b/sql-common/client.c @@ -721,6 +721,8 @@ cli_advanced_command(MYSQL *mysql, enum enum_server_command command, set_mysql_error(mysql, CR_NET_PACKET_TOO_LARGE, unknown_sqlstate); goto end; } + if (net->last_errno == ER_NET_ERROR_ON_WRITE && command == COM_BINLOG_DUMP) + goto end; end_server(mysql); if (mysql_reconnect(mysql) || stmt_skip) goto end; diff --git a/sql/net_serv.cc b/sql/net_serv.cc index f39eaaadf46..e625b307343 100644 --- a/sql/net_serv.cc +++ b/sql/net_serv.cc @@ -45,6 +45,7 @@ #include <violite.h> #include <signal.h> #include "probes_mysql.h" +#include <debug_sync.h> #ifdef EMBEDDED_LIBRARY #undef MYSQL_SERVER @@ -486,6 +487,18 @@ net_write_command(NET *net,uchar command, DBUG_ENTER("net_write_command"); DBUG_PRINT("enter",("length: %lu", (ulong) len)); + DBUG_EXECUTE_IF("simulate_error_on_packet_write", + { + if (command == COM_BINLOG_DUMP) + { + net->last_errno = ER_NET_ERROR_ON_WRITE; + socket_errno = 2; + DBUG_ASSERT(!debug_sync_set_action( + (THD *)net->thd, + STRING_WITH_LEN("now SIGNAL parked WAIT_FOR continue"))); + DBUG_RETURN(true); + } + };); MYSQL_NET_WRITE_START(length); buff[4]=command; /* For first packet */ diff --git a/sql/slave.cc b/sql/slave.cc index 1bf83aa9652..87eacfcfd0a 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -3297,7 +3297,8 @@ static int request_dump(THD *thd, MYSQL* mysql, Master_info* mi, in the future, we should do a better error analysis, but for now we just fill up the error log :-) */ - if (mysql_errno(mysql) == ER_NET_READ_INTERRUPTED) + if (mysql_errno(mysql) == ER_NET_READ_INTERRUPTED || + mysql_errno(mysql) == ER_NET_ERROR_ON_WRITE) *suppress_warnings= TRUE; // Suppress reconnect warning else sql_print_error("Error on COM_BINLOG_DUMP: %d %s, will retry in %d secs",

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 20 Jul '20

20 Jul '20

Hi, This week - Changed the build system to use our headers instead of the libraries’ headers. - I found that running make abi_update on gcc and compiling on clang (or vice versa) fails the abi_check. - During cmake documentation review, I discovered that the existing cmake configuration uses a lot of depricated commands. Even for this project, it might be easier and more stable to rewrite the relavent portions of the cmake files instead of trying to work around the depricated commands. - Removed the test cases and printf statements from the code. Next week - Resolve the abi_check issue with different compilers. - Write tests for the lzma service. - Start work on the next service. Thanks, Kartik

1 0

[Maria-developers] Progress Report GSoC 2020 (13-19 Jule)
by Anton Mikhaylenko 20 Jul '20

20 Jul '20

Good day, everybody. In this post, I want to describe what I did this week, Jule 13-19. I kept working on the benchmarking for a treadpool. I ran several configurations, which included the version without the algorithm and the version with the algorithm, also changed the time parameters, a number of clients, types of requests. Regards, Anton MIkhailenko

1 0

[Maria-developers] f963fa52ed0: MDEV-23108: Point in time recovery of binary log fails when sql_mode=ORACLE
by sujatha 17 Jul '20

17 Jul '20

revision-id: f963fa52ed07469881e75349e73efede8240fcea (mariadb-10.3.21-180-gf963fa52ed0) parent(s): 73aa31fbfd793bdb597bb19c52118ab4e637f4bc author: Sujatha committer: Sujatha timestamp: 2020-07-17 19:53:43 +0530 message: MDEV-23108: Point in time recovery of binary log fails when sql_mode=ORACLE Problem: ======== During point in time recovery of binary log syntax error is reported for BEGIN statement and recovery fails. Analysis: ========= In MariaDB 10.3 and later, setting the sql_mode system variable to Oracle allows the server to understand a subset of Oracle's PL/SQL language. When sql_mode=ORACLE is set, it switches the parser from the MariaDB parser to Oracle compatible parser. With this change 'BEGIN' is not considered as 'START TRANSACTION'. Hence the syntax error is reported. Fix: === 'mysqlbinlog' tool will convert all 'BEGIN' statements to 'START TRANSACTION' as 'START TRANSACTION' works fine in all sql_modes. --- .../include/binlog_parallel_replication_marks.test | 2 +- mysql-test/main/mysqlbinlog-innodb.result | 8 +- mysql-test/main/mysqlbinlog.result | 74 ++--- mysql-test/main/mysqlbinlog_row_compressed.result | 16 +- mysql-test/main/mysqlbinlog_row_minimal.result | 18 +- mysql-test/main/mysqlbinlog_stmt_compressed.result | 16 +- mysql-test/main/user_var-binlog.result | 4 +- .../suite/binlog/r/binlog_mysqlbinlog2.result | 206 ++++++------ .../suite/binlog/r/binlog_mysqlbinlog_row.result | 358 ++++++++++----------- .../binlog/r/binlog_mysqlbinlog_row_innodb.result | 40 +-- .../binlog/r/binlog_mysqlbinlog_row_myisam.result | 40 +-- .../binlog/r/binlog_mysqlbinlog_row_trans.result | 16 +- .../r/binlog_parallel_replication_marks_row.result | 18 +- ...inlog_parallel_replication_marks_stm_mix.result | 18 +- .../suite/binlog/r/binlog_row_annotate.result | 72 ++--- .../suite/binlog/r/binlog_row_ctype_ucs.result | 14 +- .../binlog/r/binlog_row_mysqlbinlog_options.result | 28 +- .../suite/binlog/r/binlog_stm_ctype_ucs.result | 14 +- mysql-test/suite/binlog/r/flashback.result | 20 +- .../binlog_encryption/binlog_row_annotate.result | 36 +-- .../compat/oracle/r/binlog_ptr_mysqlbinlog.result | 100 ++++++ .../oracle/t/binlog_ptr_mysqlbinlog-master.opt | 1 + .../compat/oracle/t/binlog_ptr_mysqlbinlog.test | 117 +++++++ .../suite/rpl/r/rpl_blackhole_row_annotate.result | 22 +- mysql-test/suite/rpl/r/rpl_hrtime.result | 10 +- mysql-test/suite/rpl/r/rpl_mdev382.result | 10 +- mysql-test/suite/rpl/r/rpl_sp.result | 52 +-- sql/log_event.cc | 8 +- 28 files changed, 778 insertions(+), 560 deletions(-) diff --git a/mysql-test/include/binlog_parallel_replication_marks.test b/mysql-test/include/binlog_parallel_replication_marks.test index 3976088ca43..29e86d64d3e 100644 --- a/mysql-test/include/binlog_parallel_replication_marks.test +++ b/mysql-test/include/binlog_parallel_replication_marks.test @@ -80,7 +80,7 @@ while (<F>) { s/table id \d+/table id #/; s/mapped to number \d+/mapped to number #/; s/CRC32 0x[0-9a-f]+/CRC32 0x########/; - print if /\b(GTID|BEGIN|COMMIT|Table_map|Write_rows|Update_rows|Delete_rows|generated by server|40005 TEMPORARY)\b/; + print if /\b(GTID|START TRANSACTION|COMMIT|Table_map|Write_rows|Update_rows|Delete_rows|generated by server|40005 TEMPORARY)\b/; } close F; EOF diff --git a/mysql-test/main/mysqlbinlog-innodb.result b/mysql-test/main/mysqlbinlog-innodb.result index f9f48299f41..9d3bcc7c776 100644 --- a/mysql-test/main/mysqlbinlog-innodb.result +++ b/mysql-test/main/mysqlbinlog-innodb.result @@ -24,7 +24,7 @@ FLUSH LOGS; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; use `test`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -39,7 +39,7 @@ SET @@session.collation_database=DEFAULT/*!*/; INSERT INTO t1 VALUES (1) /*!*/; COMMIT/*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; INSERT INTO t1 VALUES (2) @@ -55,7 +55,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; use `foo`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -70,7 +70,7 @@ SET @@session.collation_database=DEFAULT/*!*/; INSERT INTO t1 VALUES (1) /*!*/; COMMIT/*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; INSERT INTO t1 VALUES (2) diff --git a/mysql-test/main/mysqlbinlog.result b/mysql-test/main/mysqlbinlog.result index decd095bbdf..4c82563e0f6 100644 --- a/mysql-test/main/mysqlbinlog.result +++ b/mysql-test/main/mysqlbinlog.result @@ -37,7 +37,7 @@ create table t1 (word varchar(20)) SET TIMESTAMP=1000000000/*!*/; create table t2 (id int auto_increment not null primary key) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; insert into t1 values ("abirvalg") @@ -45,7 +45,7 @@ insert into t1 values ("abirvalg") SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -54,7 +54,7 @@ insert into t2 values () SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -62,7 +62,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -70,7 +70,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -78,7 +78,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -97,7 +97,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; use `test`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -114,7 +114,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; insert into t1 values ("Alas") @@ -134,7 +134,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; SET @@session.pseudo_thread_id=999999999/*!*/; @@ -147,28 +147,28 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT @@ -184,7 +184,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; use `test`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -231,7 +231,7 @@ create table t1 (word varchar(20)) SET TIMESTAMP=1000000000/*!*/; create table t2 (id int auto_increment not null primary key) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; insert into t1 values ("abirvalg") @@ -239,7 +239,7 @@ insert into t1 values ("abirvalg") SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -248,7 +248,7 @@ insert into t2 values () SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -256,7 +256,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -264,7 +264,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -272,7 +272,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -291,7 +291,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; use `test`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -308,7 +308,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; insert into t1 values ("Alas") @@ -328,7 +328,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; SET @@session.pseudo_thread_id=999999999/*!*/; @@ -341,28 +341,28 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT @@ -378,7 +378,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; use `test`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -553,7 +553,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a varchar(64) character set utf8) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`a`) @@ -561,7 +561,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; SET @@session.collation_database=7/*!*/; @@ -570,7 +570,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; SET @@session.collation_database=DEFAULT/*!*/; @@ -579,7 +579,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`a`) @@ -587,7 +587,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; SET @@session.collation_database=7/*!*/; @@ -596,7 +596,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; SET @@session.collation_database=DEFAULT/*!*/; @@ -605,7 +605,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` CHARACTER SET koi8r FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`a`) diff --git a/mysql-test/main/mysqlbinlog_row_compressed.result b/mysql-test/main/mysqlbinlog_row_compressed.result index 94125352654..cb8d60704f9 100644 --- a/mysql-test/main/mysqlbinlog_row_compressed.result +++ b/mysql-test/main/mysqlbinlog_row_compressed.result @@ -53,7 +53,7 @@ CREATE TABLE t2 (pk INT PRIMARY KEY, f1 INT, f2 INT, f3 INT, f4 INT, f5 MEDIUMIN # at 727 #<date> server id 1 end_log_pos 769 CRC32 XXX GTID 0-1-3 /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 769 # at 843 @@ -82,7 +82,7 @@ COMMIT # at 1040 #<date> server id 1 end_log_pos 1082 CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1082 # at 1158 @@ -111,7 +111,7 @@ COMMIT # at 1354 #<date> server id 1 end_log_pos 1396 CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1396 # at 1474 @@ -140,7 +140,7 @@ COMMIT # at 1669 #<date> server id 1 end_log_pos 1711 CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1711 # at 1786 @@ -169,7 +169,7 @@ COMMIT # at 1982 #<date> server id 1 end_log_pos 2024 CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2024 # at 2078 @@ -231,7 +231,7 @@ COMMIT # at 2298 #<date> server id 1 end_log_pos 2340 CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2340 # at 2406 @@ -312,7 +312,7 @@ COMMIT # at 2634 #<date> server id 1 end_log_pos 2676 CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2676 # at 2713 @@ -374,7 +374,7 @@ COMMIT # at 2934 #<date> server id 1 end_log_pos 2976 CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2976 # at 3013 diff --git a/mysql-test/main/mysqlbinlog_row_minimal.result b/mysql-test/main/mysqlbinlog_row_minimal.result index 6f3dbab55a8..933457edc25 100644 --- a/mysql-test/main/mysqlbinlog_row_minimal.result +++ b/mysql-test/main/mysqlbinlog_row_minimal.result @@ -51,7 +51,7 @@ CREATE TABLE t2 (pk INT PRIMARY KEY, f1 INT, f2 INT, f3 INT, f4 INT, f5 MEDIUMIN # at 774 #<date> server id 1 end_log_pos 816 CRC32 XXX GTID 0-1-3 /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 816 # at 890 @@ -80,7 +80,7 @@ COMMIT # at 1088 #<date> server id 1 end_log_pos 1130 CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1130 # at 1206 @@ -109,7 +109,7 @@ COMMIT # at 1403 #<date> server id 1 end_log_pos 1445 CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1445 # at 1523 @@ -138,7 +138,7 @@ COMMIT # at 1719 #<date> server id 1 end_log_pos 1761 CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1761 # at 1836 @@ -167,7 +167,7 @@ COMMIT # at 2035 #<date> server id 1 end_log_pos 2077 CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2077 # at 2131 @@ -229,7 +229,7 @@ COMMIT # at 2427 #<date> server id 1 end_log_pos 2469 CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2469 # at 2535 @@ -262,7 +262,7 @@ COMMIT # at 2730 #<date> server id 1 end_log_pos 2772 CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2772 # at 2809 @@ -292,7 +292,7 @@ COMMIT # at 2992 #<date> server id 1 end_log_pos 3034 CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 3034 # at 3071 @@ -357,7 +357,7 @@ DELIMITER /*!*/; /*!100001 SET @@session.gtid_domain_id=0*//*!*/; /*!100001 SET @@session.server_id=1*//*!*/; /*!100001 SET @@session.gtid_seq_no=16*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 429 # at 543 diff --git a/mysql-test/main/mysqlbinlog_stmt_compressed.result b/mysql-test/main/mysqlbinlog_stmt_compressed.result index 42717bccd01..654dff9e90d 100644 --- a/mysql-test/main/mysqlbinlog_stmt_compressed.result +++ b/mysql-test/main/mysqlbinlog_stmt_compressed.result @@ -53,7 +53,7 @@ CREATE TABLE t2 (pk INT PRIMARY KEY, f1 INT, f2 INT, f3 INT, f4 INT, f5 MEDIUMIN # at 727 #<date> server id 1 end_log_pos 769 CRC32 XXX GTID 0-1-3 /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 769 #<date> server id 1 end_log_pos 897 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -68,7 +68,7 @@ COMMIT # at 970 #<date> server id 1 end_log_pos 1012 CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1012 #<date> server id 1 end_log_pos 1140 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -83,7 +83,7 @@ COMMIT # at 1213 #<date> server id 1 end_log_pos 1255 CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1255 #<date> server id 1 end_log_pos 1385 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -98,7 +98,7 @@ COMMIT # at 1458 #<date> server id 1 end_log_pos 1500 CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1500 #<date> server id 1 end_log_pos 1627 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -113,7 +113,7 @@ COMMIT # at 1700 #<date> server id 1 end_log_pos 1742 CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1742 #<date> server id 1 end_log_pos 1850 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -128,7 +128,7 @@ COMMIT # at 1923 #<date> server id 1 end_log_pos 1965 CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1965 #<date> server id 1 end_log_pos 2082 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -143,7 +143,7 @@ COMMIT # at 2155 #<date> server id 1 end_log_pos 2197 CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2197 #<date> server id 1 end_log_pos 2288 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -158,7 +158,7 @@ COMMIT # at 2361 #<date> server id 1 end_log_pos 2403 CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2403 #<date> server id 1 end_log_pos 2494 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 diff --git a/mysql-test/main/user_var-binlog.result b/mysql-test/main/user_var-binlog.result index 0d8732399c7..0a9fe956aee 100644 --- a/mysql-test/main/user_var-binlog.result +++ b/mysql-test/main/user_var-binlog.result @@ -23,7 +23,7 @@ flush logs; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET @`a b`:=_latin1 X'68656C6C6F' COLLATE `latin1_swedish_ci`/*!*/; use `test`/*!*/; @@ -41,7 +41,7 @@ INSERT INTO t1 VALUES(@`a b`) SET TIMESTAMP=10000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET @`var1`:=_latin1 X'273B616161' COLLATE `latin1_swedish_ci`/*!*/; SET @`var2`:=_binary X'61' COLLATE `binary`/*!*/; diff --git a/mysql-test/suite/binlog/r/binlog_mysqlbinlog2.result b/mysql-test/suite/binlog/r/binlog_mysqlbinlog2.result index 8ecd4d37b1f..2a0e1d3c810 100644 --- a/mysql-test/suite/binlog/r/binlog_mysqlbinlog2.result +++ b/mysql-test/suite/binlog/r/binlog_mysqlbinlog2.result @@ -34,7 +34,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -43,7 +43,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -52,7 +52,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -61,7 +61,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -70,7 +70,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -91,7 +91,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; use `test`/*!*/; @@ -109,7 +109,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -118,7 +118,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -127,7 +127,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -136,7 +136,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -157,7 +157,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -175,7 +175,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -184,7 +184,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -217,7 +217,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -226,7 +226,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -247,7 +247,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -274,7 +274,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -292,7 +292,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -301,7 +301,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -334,7 +334,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -343,7 +343,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -377,7 +377,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -386,7 +386,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -395,7 +395,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -404,7 +404,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -413,7 +413,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -424,7 +424,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -454,7 +454,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; use `test`/*!*/; @@ -472,7 +472,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -481,7 +481,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -490,7 +490,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -499,7 +499,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -510,7 +510,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -540,7 +540,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -558,7 +558,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -567,7 +567,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -578,7 +578,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -620,7 +620,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -629,7 +629,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -638,7 +638,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -647,7 +647,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -656,7 +656,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -667,7 +667,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; DELIMITER ; # End of log file @@ -681,7 +681,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -699,7 +699,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -708,7 +708,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -719,7 +719,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -761,7 +761,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -770,7 +770,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -803,7 +803,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -812,7 +812,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -821,7 +821,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -830,7 +830,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -839,7 +839,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -860,7 +860,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; use `test`/*!*/; @@ -878,7 +878,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -887,7 +887,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -896,7 +896,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -905,7 +905,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -925,7 +925,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -943,7 +943,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -952,7 +952,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -985,7 +985,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -994,7 +994,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1014,7 +1014,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -1041,7 +1041,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -1059,7 +1059,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1068,7 +1068,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1101,7 +1101,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1110,7 +1110,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1143,7 +1143,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1152,7 +1152,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1161,7 +1161,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -1170,7 +1170,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1179,7 +1179,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1190,7 +1190,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -1220,7 +1220,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; use `test`/*!*/; @@ -1238,7 +1238,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1247,7 +1247,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -1256,7 +1256,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1265,7 +1265,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1276,7 +1276,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -1305,7 +1305,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -1323,7 +1323,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1332,7 +1332,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1343,7 +1343,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -1385,7 +1385,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1394,7 +1394,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1403,7 +1403,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -1412,7 +1412,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1421,7 +1421,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1444,7 +1444,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -1462,7 +1462,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1471,7 +1471,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1482,7 +1482,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -1524,7 +1524,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1533,7 +1533,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1566,7 +1566,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1575,7 +1575,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1584,7 +1584,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -1593,7 +1593,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1602,7 +1602,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1611,7 +1611,7 @@ insert into t1 values(null, "e") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; SET TIMESTAMP=1773143/*!*/; diff --git a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result index 0cc1805eadc..469eb97badc 100644 --- a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result +++ b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result @@ -383,7 +383,7 @@ CREATE TABLE t1 (c01 BIT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -404,7 +404,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -441,7 +441,7 @@ CREATE TABLE t1 (c01 BIT(7)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -462,7 +462,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -483,7 +483,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -504,7 +504,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -525,7 +525,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -546,7 +546,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-11 /*!100001 SET @@session.gtid_seq_no=11*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -567,7 +567,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-12 /*!100001 SET @@session.gtid_seq_no=12*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -588,7 +588,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-13 /*!100001 SET @@session.gtid_seq_no=13*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -609,7 +609,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-14 /*!100001 SET @@session.gtid_seq_no=14*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -630,7 +630,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-15 /*!100001 SET @@session.gtid_seq_no=15*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -669,7 +669,7 @@ CREATE TABLE t1 (a BIT(20), b CHAR(2)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-18 /*!100001 SET @@session.gtid_seq_no=18*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -707,7 +707,7 @@ CREATE TABLE t1 (c02 BIT(64)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-21 /*!100001 SET @@session.gtid_seq_no=21*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -728,7 +728,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-22 /*!100001 SET @@session.gtid_seq_no=22*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -749,7 +749,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-23 /*!100001 SET @@session.gtid_seq_no=23*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -770,7 +770,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-24 /*!100001 SET @@session.gtid_seq_no=24*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -807,7 +807,7 @@ CREATE TABLE t1 (c03 TINYINT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-27 /*!100001 SET @@session.gtid_seq_no=27*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -834,7 +834,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-28 /*!100001 SET @@session.gtid_seq_no=28*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -855,7 +855,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-29 /*!100001 SET @@session.gtid_seq_no=29*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -878,7 +878,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-30 /*!100001 SET @@session.gtid_seq_no=30*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -915,7 +915,7 @@ CREATE TABLE t1 (c04 TINYINT UNSIGNED) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-33 /*!100001 SET @@session.gtid_seq_no=33*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -939,7 +939,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-34 /*!100001 SET @@session.gtid_seq_no=34*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -976,7 +976,7 @@ CREATE TABLE t1 (c06 BOOL) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-37 /*!100001 SET @@session.gtid_seq_no=37*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -997,7 +997,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-38 /*!100001 SET @@session.gtid_seq_no=38*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1034,7 +1034,7 @@ CREATE TABLE t1 (c07 SMALLINT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-41 /*!100001 SET @@session.gtid_seq_no=41*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1055,7 +1055,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-42 /*!100001 SET @@session.gtid_seq_no=42*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1092,7 +1092,7 @@ CREATE TABLE t1 (c08 SMALLINT UNSIGNED) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-45 /*!100001 SET @@session.gtid_seq_no=45*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1116,7 +1116,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-46 /*!100001 SET @@session.gtid_seq_no=46*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1139,7 +1139,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-47 /*!100001 SET @@session.gtid_seq_no=47*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1176,7 +1176,7 @@ CREATE TABLE t1 (c10 MEDIUMINT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-50 /*!100001 SET @@session.gtid_seq_no=50*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1197,7 +1197,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-51 /*!100001 SET @@session.gtid_seq_no=51*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1234,7 +1234,7 @@ CREATE TABLE t1 (c11 MEDIUMINT UNSIGNED) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-54 /*!100001 SET @@session.gtid_seq_no=54*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1258,7 +1258,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-55 /*!100001 SET @@session.gtid_seq_no=55*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1281,7 +1281,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-56 /*!100001 SET @@session.gtid_seq_no=56*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1318,7 +1318,7 @@ CREATE TABLE t1 (c13 INT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-59 /*!100001 SET @@session.gtid_seq_no=59*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1339,7 +1339,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-60 /*!100001 SET @@session.gtid_seq_no=60*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1376,7 +1376,7 @@ CREATE TABLE t1 (c14 INT UNSIGNED) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-63 /*!100001 SET @@session.gtid_seq_no=63*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1400,7 +1400,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-64 /*!100001 SET @@session.gtid_seq_no=64*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1423,7 +1423,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-65 /*!100001 SET @@session.gtid_seq_no=65*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1460,7 +1460,7 @@ CREATE TABLE t1 (c16 BIGINT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-68 /*!100001 SET @@session.gtid_seq_no=68*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1481,7 +1481,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-69 /*!100001 SET @@session.gtid_seq_no=69*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1518,7 +1518,7 @@ CREATE TABLE t1 (c17 BIGINT UNSIGNED) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-72 /*!100001 SET @@session.gtid_seq_no=72*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1542,7 +1542,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-73 /*!100001 SET @@session.gtid_seq_no=73*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1565,7 +1565,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-74 /*!100001 SET @@session.gtid_seq_no=74*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1602,7 +1602,7 @@ CREATE TABLE t1 (c19 FLOAT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-77 /*!100001 SET @@session.gtid_seq_no=77*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1623,7 +1623,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-78 /*!100001 SET @@session.gtid_seq_no=78*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1660,7 +1660,7 @@ CREATE TABLE t1 (c22 DOUBLE) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-81 /*!100001 SET @@session.gtid_seq_no=81*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1681,7 +1681,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-82 /*!100001 SET @@session.gtid_seq_no=82*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1718,7 +1718,7 @@ CREATE TABLE t1 (c25 DECIMAL(10,5)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-85 /*!100001 SET @@session.gtid_seq_no=85*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1739,7 +1739,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-86 /*!100001 SET @@session.gtid_seq_no=86*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1760,7 +1760,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-87 /*!100001 SET @@session.gtid_seq_no=87*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1797,7 +1797,7 @@ CREATE TABLE t1 (c28 DATE) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-90 /*!100001 SET @@session.gtid_seq_no=90*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1818,7 +1818,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-91 /*!100001 SET @@session.gtid_seq_no=91*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1855,7 +1855,7 @@ CREATE TABLE t1 (c29 DATETIME) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-94 /*!100001 SET @@session.gtid_seq_no=94*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1876,7 +1876,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-95 /*!100001 SET @@session.gtid_seq_no=95*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1913,7 +1913,7 @@ CREATE TABLE t1 (c30 TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURR # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-98 /*!100001 SET @@session.gtid_seq_no=98*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1935,7 +1935,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-99 /*!100001 SET @@session.gtid_seq_no=99*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1972,7 +1972,7 @@ CREATE TABLE t1 (c31 TIME) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-102 /*!100001 SET @@session.gtid_seq_no=102*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1993,7 +1993,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-103 /*!100001 SET @@session.gtid_seq_no=103*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2030,7 +2030,7 @@ CREATE TABLE t1 (c32 YEAR) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-106 /*!100001 SET @@session.gtid_seq_no=106*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2051,7 +2051,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-107 /*!100001 SET @@session.gtid_seq_no=107*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2088,7 +2088,7 @@ CREATE TABLE t1 (c33 CHAR) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-110 /*!100001 SET @@session.gtid_seq_no=110*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2109,7 +2109,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-111 /*!100001 SET @@session.gtid_seq_no=111*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2146,7 +2146,7 @@ CREATE TABLE t1 (c34 CHAR(0)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-114 /*!100001 SET @@session.gtid_seq_no=114*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2167,7 +2167,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-115 /*!100001 SET @@session.gtid_seq_no=115*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2204,7 +2204,7 @@ CREATE TABLE t1 (c35 CHAR(1)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-118 /*!100001 SET @@session.gtid_seq_no=118*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2225,7 +2225,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-119 /*!100001 SET @@session.gtid_seq_no=119*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2262,7 +2262,7 @@ CREATE TABLE t1 (c36 CHAR(255)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-122 /*!100001 SET @@session.gtid_seq_no=122*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2283,7 +2283,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-123 /*!100001 SET @@session.gtid_seq_no=123*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2320,7 +2320,7 @@ CREATE TABLE t1 (c37 NATIONAL CHAR) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-126 /*!100001 SET @@session.gtid_seq_no=126*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2341,7 +2341,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-127 /*!100001 SET @@session.gtid_seq_no=127*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2378,7 +2378,7 @@ CREATE TABLE t1 (c38 NATIONAL CHAR(0)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-130 /*!100001 SET @@session.gtid_seq_no=130*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2399,7 +2399,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-131 /*!100001 SET @@session.gtid_seq_no=131*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2436,7 +2436,7 @@ CREATE TABLE t1 (c39 NATIONAL CHAR(1)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-134 /*!100001 SET @@session.gtid_seq_no=134*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2457,7 +2457,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-135 /*!100001 SET @@session.gtid_seq_no=135*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2494,7 +2494,7 @@ CREATE TABLE t1 (c40 NATIONAL CHAR(255)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-138 /*!100001 SET @@session.gtid_seq_no=138*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2515,7 +2515,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-139 /*!100001 SET @@session.gtid_seq_no=139*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2536,7 +2536,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-140 /*!100001 SET @@session.gtid_seq_no=140*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2576,7 +2576,7 @@ CREATE TABLE t1 (c41 CHAR CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-143 /*!100001 SET @@session.gtid_seq_no=143*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2597,7 +2597,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-144 /*!100001 SET @@session.gtid_seq_no=144*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2634,7 +2634,7 @@ CREATE TABLE t1 (c42 CHAR(0) CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-147 /*!100001 SET @@session.gtid_seq_no=147*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2655,7 +2655,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-148 /*!100001 SET @@session.gtid_seq_no=148*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2692,7 +2692,7 @@ CREATE TABLE t1 (c43 CHAR(1) CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-151 /*!100001 SET @@session.gtid_seq_no=151*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2713,7 +2713,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-152 /*!100001 SET @@session.gtid_seq_no=152*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2750,7 +2750,7 @@ CREATE TABLE t1 (c44 CHAR(255) CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-155 /*!100001 SET @@session.gtid_seq_no=155*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2771,7 +2771,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-156 /*!100001 SET @@session.gtid_seq_no=156*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2792,7 +2792,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-157 /*!100001 SET @@session.gtid_seq_no=157*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2832,7 +2832,7 @@ CREATE TABLE t1 (c45 VARCHAR(0)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-160 /*!100001 SET @@session.gtid_seq_no=160*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2853,7 +2853,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-161 /*!100001 SET @@session.gtid_seq_no=161*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2890,7 +2890,7 @@ CREATE TABLE t1 (c46 VARCHAR(1)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-164 /*!100001 SET @@session.gtid_seq_no=164*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2911,7 +2911,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-165 /*!100001 SET @@session.gtid_seq_no=165*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2948,7 +2948,7 @@ CREATE TABLE t1 (c47 VARCHAR(255)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-168 /*!100001 SET @@session.gtid_seq_no=168*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2969,7 +2969,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-169 /*!100001 SET @@session.gtid_seq_no=169*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3006,7 +3006,7 @@ CREATE TABLE t1 (c48 VARCHAR(261)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-172 /*!100001 SET @@session.gtid_seq_no=172*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3027,7 +3027,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-173 /*!100001 SET @@session.gtid_seq_no=173*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3064,7 +3064,7 @@ CREATE TABLE t1 (c49 NATIONAL VARCHAR(0)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-176 /*!100001 SET @@session.gtid_seq_no=176*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3085,7 +3085,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-177 /*!100001 SET @@session.gtid_seq_no=177*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3122,7 +3122,7 @@ CREATE TABLE t1 (c50 NATIONAL VARCHAR(1)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-180 /*!100001 SET @@session.gtid_seq_no=180*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3143,7 +3143,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-181 /*!100001 SET @@session.gtid_seq_no=181*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3180,7 +3180,7 @@ CREATE TABLE t1 (c51 NATIONAL VARCHAR(255)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-184 /*!100001 SET @@session.gtid_seq_no=184*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3201,7 +3201,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-185 /*!100001 SET @@session.gtid_seq_no=185*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3222,7 +3222,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-186 /*!100001 SET @@session.gtid_seq_no=186*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3262,7 +3262,7 @@ CREATE TABLE t1 (c52 NATIONAL VARCHAR(261)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-189 /*!100001 SET @@session.gtid_seq_no=189*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3283,7 +3283,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-190 /*!100001 SET @@session.gtid_seq_no=190*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3304,7 +3304,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-191 /*!100001 SET @@session.gtid_seq_no=191*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3344,7 +3344,7 @@ CREATE TABLE t1 (c53 VARCHAR(0) CHARACTER SET ucs2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-194 /*!100001 SET @@session.gtid_seq_no=194*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3365,7 +3365,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-195 /*!100001 SET @@session.gtid_seq_no=195*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3402,7 +3402,7 @@ CREATE TABLE t1 (c54 VARCHAR(1) CHARACTER SET ucs2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-198 /*!100001 SET @@session.gtid_seq_no=198*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3423,7 +3423,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-199 /*!100001 SET @@session.gtid_seq_no=199*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3460,7 +3460,7 @@ CREATE TABLE t1 (c55 VARCHAR(255) CHARACTER SET ucs2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-202 /*!100001 SET @@session.gtid_seq_no=202*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3481,7 +3481,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-203 /*!100001 SET @@session.gtid_seq_no=203*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3518,7 +3518,7 @@ CREATE TABLE t1 (c56 VARCHAR(261) CHARACTER SET ucs2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-206 /*!100001 SET @@session.gtid_seq_no=206*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3539,7 +3539,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-207 /*!100001 SET @@session.gtid_seq_no=207*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3576,7 +3576,7 @@ CREATE TABLE t1 (c57 BINARY) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-210 /*!100001 SET @@session.gtid_seq_no=210*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3597,7 +3597,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-211 /*!100001 SET @@session.gtid_seq_no=211*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3618,7 +3618,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-212 /*!100001 SET @@session.gtid_seq_no=212*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3639,7 +3639,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-213 /*!100001 SET @@session.gtid_seq_no=213*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3676,7 +3676,7 @@ CREATE TABLE t1 (c58 BINARY(0)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-216 /*!100001 SET @@session.gtid_seq_no=216*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3697,7 +3697,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-217 /*!100001 SET @@session.gtid_seq_no=217*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3734,7 +3734,7 @@ CREATE TABLE t1 (c59 BINARY(1)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-220 /*!100001 SET @@session.gtid_seq_no=220*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3755,7 +3755,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-221 /*!100001 SET @@session.gtid_seq_no=221*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3776,7 +3776,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-222 /*!100001 SET @@session.gtid_seq_no=222*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3797,7 +3797,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-223 /*!100001 SET @@session.gtid_seq_no=223*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3834,7 +3834,7 @@ CREATE TABLE t1 (c60 BINARY(255)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-226 /*!100001 SET @@session.gtid_seq_no=226*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3855,7 +3855,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-227 /*!100001 SET @@session.gtid_seq_no=227*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3876,7 +3876,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-228 /*!100001 SET @@session.gtid_seq_no=228*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3897,7 +3897,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-229 /*!100001 SET @@session.gtid_seq_no=229*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3934,7 +3934,7 @@ CREATE TABLE t1 (c61 VARBINARY(0)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-232 /*!100001 SET @@session.gtid_seq_no=232*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3955,7 +3955,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-233 /*!100001 SET @@session.gtid_seq_no=233*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3992,7 +3992,7 @@ CREATE TABLE t1 (c62 VARBINARY(1)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-236 /*!100001 SET @@session.gtid_seq_no=236*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4013,7 +4013,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-237 /*!100001 SET @@session.gtid_seq_no=237*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4034,7 +4034,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-238 /*!100001 SET @@session.gtid_seq_no=238*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4055,7 +4055,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-239 /*!100001 SET @@session.gtid_seq_no=239*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4092,7 +4092,7 @@ CREATE TABLE t1 (c63 VARBINARY(255)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-242 /*!100001 SET @@session.gtid_seq_no=242*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4113,7 +4113,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-243 /*!100001 SET @@session.gtid_seq_no=243*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4134,7 +4134,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-244 /*!100001 SET @@session.gtid_seq_no=244*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4155,7 +4155,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-245 /*!100001 SET @@session.gtid_seq_no=245*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4192,7 +4192,7 @@ CREATE TABLE t1 (c65 TINYBLOB) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-248 /*!100001 SET @@session.gtid_seq_no=248*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4213,7 +4213,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-249 /*!100001 SET @@session.gtid_seq_no=249*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4250,7 +4250,7 @@ CREATE TABLE t1 (c68 BLOB) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-252 /*!100001 SET @@session.gtid_seq_no=252*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4271,7 +4271,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-253 /*!100001 SET @@session.gtid_seq_no=253*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4308,7 +4308,7 @@ CREATE TABLE t1 (c71 MEDIUMBLOB) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-256 /*!100001 SET @@session.gtid_seq_no=256*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4329,7 +4329,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-257 /*!100001 SET @@session.gtid_seq_no=257*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4366,7 +4366,7 @@ CREATE TABLE t1 (c74 LONGBLOB) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-260 /*!100001 SET @@session.gtid_seq_no=260*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4387,7 +4387,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-261 /*!100001 SET @@session.gtid_seq_no=261*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4424,7 +4424,7 @@ CREATE TABLE t1 (c66 TINYTEXT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-264 /*!100001 SET @@session.gtid_seq_no=264*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4445,7 +4445,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-265 /*!100001 SET @@session.gtid_seq_no=265*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4482,7 +4482,7 @@ CREATE TABLE t1 (c69 TEXT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-268 /*!100001 SET @@session.gtid_seq_no=268*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4503,7 +4503,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-269 /*!100001 SET @@session.gtid_seq_no=269*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4540,7 +4540,7 @@ CREATE TABLE t1 (c72 MEDIUMTEXT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-272 /*!100001 SET @@session.gtid_seq_no=272*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4561,7 +4561,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-273 /*!100001 SET @@session.gtid_seq_no=273*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4598,7 +4598,7 @@ CREATE TABLE t1 (c75 LONGTEXT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-276 /*!100001 SET @@session.gtid_seq_no=276*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4619,7 +4619,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-277 /*!100001 SET @@session.gtid_seq_no=277*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4656,7 +4656,7 @@ CREATE TABLE t1 (c67 TINYTEXT CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-280 /*!100001 SET @@session.gtid_seq_no=280*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4677,7 +4677,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-281 /*!100001 SET @@session.gtid_seq_no=281*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4714,7 +4714,7 @@ CREATE TABLE t1 (c70 TEXT CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-284 /*!100001 SET @@session.gtid_seq_no=284*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4735,7 +4735,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-285 /*!100001 SET @@session.gtid_seq_no=285*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4772,7 +4772,7 @@ CREATE TABLE t1 (c73 MEDIUMTEXT CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-288 /*!100001 SET @@session.gtid_seq_no=288*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4793,7 +4793,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-289 /*!100001 SET @@session.gtid_seq_no=289*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4830,7 +4830,7 @@ CREATE TABLE t1 (c76 LONGTEXT CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-292 /*!100001 SET @@session.gtid_seq_no=292*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4851,7 +4851,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-293 /*!100001 SET @@session.gtid_seq_no=293*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4888,7 +4888,7 @@ CREATE TABLE t1 (c77 ENUM('a','b','c')) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-296 /*!100001 SET @@session.gtid_seq_no=296*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4909,7 +4909,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-297 /*!100001 SET @@session.gtid_seq_no=297*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4946,7 +4946,7 @@ CREATE TABLE t1 (c78 SET('a','b','c','d','e','f')) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-300 /*!100001 SET @@session.gtid_seq_no=300*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4967,7 +4967,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-301 /*!100001 SET @@session.gtid_seq_no=301*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4988,7 +4988,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-302 /*!100001 SET @@session.gtid_seq_no=302*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5009,7 +5009,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-303 /*!100001 SET @@session.gtid_seq_no=303*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5030,7 +5030,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-304 /*!100001 SET @@session.gtid_seq_no=304*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5051,7 +5051,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-305 /*!100001 SET @@session.gtid_seq_no=305*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5072,7 +5072,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-306 /*!100001 SET @@session.gtid_seq_no=306*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5093,7 +5093,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-307 /*!100001 SET @@session.gtid_seq_no=307*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5138,7 +5138,7 @@ CREATE TABLE t2 (a int NOT NULL DEFAULT 0, b int NOT NULL DEFAULT 0) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-311 /*!100001 SET @@session.gtid_seq_no=311*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5160,7 +5160,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-312 /*!100001 SET @@session.gtid_seq_no=312*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5182,7 +5182,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-313 /*!100001 SET @@session.gtid_seq_no=313*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5204,7 +5204,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-314 /*!100001 SET @@session.gtid_seq_no=314*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5226,7 +5226,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-315 /*!100001 SET @@session.gtid_seq_no=315*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5294,7 +5294,7 @@ c_text_utf8 blob ) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-318 /*!100001 SET @@session.gtid_seq_no=318*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5347,7 +5347,7 @@ DELIMITER /*!*/; /*!100001 SET @@session.gtid_domain_id=0*//*!*/; /*!100001 SET @@session.server_id=1*//*!*/; /*!100001 SET @@session.gtid_seq_no=320*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_innodb.result b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_innodb.result index 22309e964be..d8a5864ba3d 100644 --- a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_innodb.result +++ b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_innodb.result @@ -2368,7 +2368,7 @@ crn INT -- row number # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 trans /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2550,7 +2550,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 trans /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2732,7 +2732,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 trans /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3080,7 +3080,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-5 trans /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3420,7 +3420,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 trans /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3765,7 +3765,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-7 trans /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4108,7 +4108,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-8 trans /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4454,7 +4454,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-9 trans /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4634,7 +4634,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-10 trans /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4811,7 +4811,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-11 trans /*!100001 SET @@session.gtid_seq_no=11*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4991,7 +4991,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-12 trans /*!100001 SET @@session.gtid_seq_no=12*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5293,7 +5293,7 @@ crn INT -- row number # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 trans /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5363,7 +5363,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 trans /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5442,7 +5442,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 trans /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5717,7 +5717,7 @@ c_3_n INT -- row number # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 trans /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5787,7 +5787,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-5 trans /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5857,7 +5857,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 trans /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5927,7 +5927,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-7 trans /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -6117,7 +6117,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-8 trans /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -6331,7 +6331,7 @@ c3 VARCHAR(60) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 trans /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_myisam.result b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_myisam.result index 8091242dab7..a75593055d7 100644 --- a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_myisam.result +++ b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_myisam.result @@ -2368,7 +2368,7 @@ crn INT -- row number # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2553,7 +2553,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2737,7 +2737,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3087,7 +3087,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3429,7 +3429,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3776,7 +3776,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4121,7 +4121,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4469,7 +4469,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4651,7 +4651,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4830,7 +4830,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-11 /*!100001 SET @@session.gtid_seq_no=11*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5012,7 +5012,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-12 /*!100001 SET @@session.gtid_seq_no=12*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5316,7 +5316,7 @@ crn INT -- row number # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5388,7 +5388,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5469,7 +5469,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5746,7 +5746,7 @@ c_3_n INT -- row number # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5818,7 +5818,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5890,7 +5890,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5962,7 +5962,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -6154,7 +6154,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -6370,7 +6370,7 @@ c3 VARCHAR(60) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_trans.result b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_trans.result index 28070fab374..545cb8f7857 100644 --- a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_trans.result +++ b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_trans.result @@ -172,7 +172,7 @@ c2 VARCHAR(20) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 trans /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -257,7 +257,7 @@ TRUNCATE TABLE t1 # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -287,7 +287,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -326,7 +326,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -348,7 +348,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-9 trans /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -433,7 +433,7 @@ TRUNCATE TABLE t2 # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-12 /*!100001 SET @@session.gtid_seq_no=12*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -463,7 +463,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-13 /*!100001 SET @@session.gtid_seq_no=13*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -502,7 +502,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-14 /*!100001 SET @@session.gtid_seq_no=14*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_row.result b/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_row.result index c4a1ba9b83b..bca96dd5fd0 100644 --- a/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_row.result +++ b/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_row.result @@ -38,13 +38,13 @@ disconnect tmp_con; connection default; FLUSH LOGS; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (1,0) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (2,0) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F @@ -52,13 +52,13 @@ COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl /* GTID */ ALTER TABLE t1 ADD c INT # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (3,0,0) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ UPDATE t1 SET b=1, c=1 WHERE a=2 # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Update_rows: table id # flags: STMT_END_F @@ -70,7 +70,7 @@ BEGIN # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ DELETE FROM t1 WHERE a=5 # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Delete_rows: table id # flags: STMT_END_F @@ -82,25 +82,25 @@ BEGIN # server id 1 end_log_pos # CRC32 0x######## Update_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (8, 5, 0) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (9, 5, 1) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (10, 6, 0) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (11, 7, 0) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F diff --git a/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_stm_mix.result b/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_stm_mix.result index c56dff9c8a0..4a6cd6f4ac0 100644 --- a/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_stm_mix.result +++ b/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_stm_mix.result @@ -38,21 +38,21 @@ disconnect tmp_con; connection default; FLUSH LOGS; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION /* GTID */ INSERT INTO t1 VALUES (1,0) COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION /* GTID */ INSERT INTO t1 VALUES (2,0) COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl /* GTID */ ALTER TABLE t1 ADD c INT # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION /* GTID */ INSERT INTO t1 VALUES (3,0,0) COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl -BEGIN +START TRANSACTION /* GTID */ UPDATE t1 SET b=1, c=1 WHERE a=2 /* GTID */ CREATE TEMPORARY TABLE t2 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB /* GTID */ INSERT INTO t2 VALUES (4,10), (5,20) @@ -63,7 +63,7 @@ COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl /* GTID */ CREATE TEMPORARY TABLE t3 (a INT PRIMARY KEY) ENGINE=InnoDB # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl -BEGIN +START TRANSACTION /* GTID */ DELETE FROM t1 WHERE a=5 /* GTID */ INSERT INTO t3 VALUES (7) /* GTID */ INSERT INTO t1 SELECT a, 4, 0 FROM t3 @@ -73,21 +73,21 @@ COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl /* GTID */ CREATE TEMPORARY TABLE t4 (a INT PRIMARY KEY) ENGINE=InnoDB # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION /* GTID */ INSERT INTO t1 VALUES (8, 5, 0) COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl /* GTID */ ALTER TABLE t4 ADD b INT # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION /* GTID */ INSERT INTO t1 VALUES (9, 5, 1) COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION /* GTID */ INSERT INTO t1 VALUES (10, 6, 0) COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl -BEGIN +START TRANSACTION /* GTID */ CREATE TEMPORARY TABLE t5 (a INT PRIMARY KEY) ENGINE=InnoDB /* GTID */ INSERT INTO t1 VALUES (11, 7, 0) COMMIT/*!*/; diff --git a/mysql-test/suite/binlog/r/binlog_row_annotate.result b/mysql-test/suite/binlog/r/binlog_row_annotate.result index bbdfe547b18..e65569853aa 100644 --- a/mysql-test/suite/binlog/r/binlog_row_annotate.result +++ b/mysql-test/suite/binlog/r/binlog_row_annotate.result @@ -144,7 +144,7 @@ CREATE DATABASE test3 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -171,7 +171,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -198,7 +198,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -225,7 +225,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -267,7 +267,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -294,7 +294,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -374,7 +374,7 @@ CREATE DATABASE test1 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -401,7 +401,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -414,7 +414,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -427,7 +427,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -458,7 +458,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -471,7 +471,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -542,7 +542,7 @@ CREATE DATABASE test3 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -567,7 +567,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -592,7 +592,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -617,7 +617,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -655,7 +655,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -680,7 +680,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -768,7 +768,7 @@ CREATE DATABASE test3 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -795,7 +795,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -822,7 +822,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -849,7 +849,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -891,7 +891,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -918,7 +918,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -998,7 +998,7 @@ CREATE DATABASE test1 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1025,7 +1025,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1038,7 +1038,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1051,7 +1051,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1082,7 +1082,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1095,7 +1095,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1166,7 +1166,7 @@ CREATE DATABASE test3 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test1`.`t1` mapped to number # @@ -1190,7 +1190,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test2`.`t2` mapped to number # @@ -1214,7 +1214,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test3`.`t3` mapped to number # @@ -1238,7 +1238,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test1`.`t1` mapped to number # @@ -1275,7 +1275,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test2`.`t2` mapped to number # @@ -1299,7 +1299,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test2`.`t2` mapped to number # diff --git a/mysql-test/suite/binlog/r/binlog_row_ctype_ucs.result b/mysql-test/suite/binlog/r/binlog_row_ctype_ucs.result index 9e95f0a365e..0fe726766d2 100644 --- a/mysql-test/suite/binlog/r/binlog_row_ctype_ucs.result +++ b/mysql-test/suite/binlog/r/binlog_row_ctype_ucs.result @@ -16,7 +16,7 @@ flush logs; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; # Annotate_rows: #Q> insert into t2 values (@v) @@ -104,7 +104,7 @@ CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf8) # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -125,7 +125,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -146,7 +146,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -167,7 +167,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -188,7 +188,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -209,7 +209,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/binlog/r/binlog_row_mysqlbinlog_options.result b/mysql-test/suite/binlog/r/binlog_row_mysqlbinlog_options.result index 8b138c461ab..0232443573a 100644 --- a/mysql-test/suite/binlog/r/binlog_row_mysqlbinlog_options.result +++ b/mysql-test/suite/binlog/r/binlog_row_mysqlbinlog_options.result @@ -62,7 +62,7 @@ CREATE TABLE t1 (a INT, b INT) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-2 /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -97,7 +97,7 @@ CREATE TABLE t2 (a INT) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -121,7 +121,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -152,7 +152,7 @@ CREATE TABLE t3 (a INT) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -176,7 +176,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -198,7 +198,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -237,7 +237,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -303,7 +303,7 @@ CREATE TABLE t1 (a INT, b INT) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-2 /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -338,7 +338,7 @@ CREATE TABLE t2 (a INT) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -362,7 +362,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -393,7 +393,7 @@ CREATE TABLE t3 (a INT) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -417,7 +417,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -439,7 +439,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -478,7 +478,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/binlog/r/binlog_stm_ctype_ucs.result b/mysql-test/suite/binlog/r/binlog_stm_ctype_ucs.result index 00f7ce6ead0..05b1520724a 100644 --- a/mysql-test/suite/binlog/r/binlog_stm_ctype_ucs.result +++ b/mysql-test/suite/binlog/r/binlog_stm_ctype_ucs.result @@ -15,7 +15,7 @@ flush logs; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET @`v`:=_ucs2 X'006100620063' COLLATE `ucs2_general_ci`/*!*/; use `test`/*!*/; @@ -106,7 +106,7 @@ CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf8) # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX Query thread_id=# exec_time=# error_code=0 @@ -121,7 +121,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX Query thread_id=# exec_time=# error_code=0 @@ -136,7 +136,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX Query thread_id=# exec_time=# error_code=0 @@ -151,7 +151,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX Query thread_id=# exec_time=# error_code=0 @@ -166,7 +166,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX Query thread_id=# exec_time=# error_code=0 @@ -181,7 +181,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX Query thread_id=# exec_time=# error_code=0 diff --git a/mysql-test/suite/binlog/r/flashback.result b/mysql-test/suite/binlog/r/flashback.result index bdffaabd673..eaced02b7a9 100644 --- a/mysql-test/suite/binlog/r/flashback.result +++ b/mysql-test/suite/binlog/r/flashback.result @@ -81,7 +81,7 @@ c08 TEXT # at # #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 trans /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -107,7 +107,7 @@ COMMIT/*!*/; # at # #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 trans /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -133,7 +133,7 @@ COMMIT/*!*/; # at # #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 trans /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -159,7 +159,7 @@ COMMIT/*!*/; # at # #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-5 trans /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -213,7 +213,7 @@ COMMIT/*!*/; # at # #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 trans /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -296,7 +296,7 @@ ROLLBACK/*!*/; # Number of rows: 3 #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Rotate to master-bin.000002 pos: 4 #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Xid = # -BEGIN/*!*/; +START TRANSACTION/*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Write_rows: table id # flags: STMT_END_F ### INSERT INTO `test`.`t1` ### SET @@ -331,7 +331,7 @@ BEGIN/*!*/; COMMIT /*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Xid = # -BEGIN/*!*/; +START TRANSACTION/*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Update_rows: table id # flags: STMT_END_F ### UPDATE `test`.`t1` ### WHERE @@ -374,7 +374,7 @@ BEGIN/*!*/; COMMIT /*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Xid = # -BEGIN/*!*/; +START TRANSACTION/*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Delete_rows: table id # flags: STMT_END_F ### DELETE FROM `test`.`t1` ### WHERE @@ -389,7 +389,7 @@ BEGIN/*!*/; COMMIT /*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Xid = # -BEGIN/*!*/; +START TRANSACTION/*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Delete_rows: table id # flags: STMT_END_F ### DELETE FROM `test`.`t1` ### WHERE @@ -404,7 +404,7 @@ BEGIN/*!*/; COMMIT /*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Xid = # -BEGIN/*!*/; +START TRANSACTION/*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Delete_rows: table id # flags: STMT_END_F ### DELETE FROM `test`.`t1` ### WHERE diff --git a/mysql-test/suite/binlog_encryption/binlog_row_annotate.result b/mysql-test/suite/binlog_encryption/binlog_row_annotate.result index b783488b07a..004f163f192 100644 --- a/mysql-test/suite/binlog_encryption/binlog_row_annotate.result +++ b/mysql-test/suite/binlog_encryption/binlog_row_annotate.result @@ -148,7 +148,7 @@ CREATE DATABASE test3 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -175,7 +175,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -202,7 +202,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -229,7 +229,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -271,7 +271,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -298,7 +298,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -381,7 +381,7 @@ CREATE DATABASE test1 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -408,7 +408,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -421,7 +421,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -434,7 +434,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -465,7 +465,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -478,7 +478,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -552,7 +552,7 @@ CREATE DATABASE test3 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test1`.`t1` mapped to number # @@ -576,7 +576,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test2`.`t2` mapped to number # @@ -600,7 +600,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test3`.`t3` mapped to number # @@ -624,7 +624,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test1`.`t1` mapped to number # @@ -661,7 +661,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test2`.`t2` mapped to number # @@ -685,7 +685,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test2`.`t2` mapped to number # diff --git a/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result b/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result new file mode 100644 index 00000000000..0656a685976 --- /dev/null +++ b/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result @@ -0,0 +1,100 @@ +SET @@SQL_MODE = 'ORACLE'; +########################################################################## +# Test verifies Gtid_log_event/Xid_log_event specific print # +########################################################################## +CREATE TABLE tm (f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +CREATE OR REPLACE PROCEDURE simpleproc (param1 OUT INT) AS +BEGIN +SELECT COUNT(*) INTO param1 FROM t; +END; +/ +CREATE FUNCTION f1 RETURN INT +AS +BEGIN +RETURN 10; +END; +/ +FLUSH LOGS; +########################################################################## +# Delete data from master so that it can be restored from binlog # +########################################################################## +DROP FUNCTION f1; +DROP PROCEDURE simpleproc; +DROP TABLE tm; +DROP TABLE t; +########################################################################## +# Post recovery using mysqlbinlog # +########################################################################## +SHOW TABLES; +Tables_in_test +t +tm +SELECT * FROM tm; +f +10 +SELECT * FROM t; +f +10 +SELECT f1(); +f1() +10 +CALL simpleproc(@a); +SELECT @a; +@a +1 +"***** Clean Up *****" +DROP TABLE t,tm; +DROP PROCEDURE simpleproc; +DROP FUNCTION f1; +RESET MASTER; +########################################################################## +# Test verifies Gtid_log_event/Xid_log_event/Qery_log_event # +# specific print along with flashback option # +########################################################################## +CREATE TABLE tm(f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); +INSERT INTO tm VALUES (20); +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +INSERT INTO t VALUES (20); +########################################################################## +# Initial data # +########################################################################## +SELECT * FROM tm; +f +10 +20 +SELECT * FROM t; +f +10 +20 +FLUSH LOGS; +DELETE FROM tm WHERE f=20; +DELETE FROM t WHERE f=20; +FLUSH LOGS; +########################################################################## +# Data after deletion # +########################################################################## +SELECT * FROM tm; +f +10 +SELECT * FROM t; +f +10 +FOUND 2 /START TRANSACTION/ in test.sql +########################################################################## +# Data after recovery using flashback # +########################################################################## +SELECT * FROM tm; +f +10 +20 +SELECT * FROM t; +f +10 +20 +"***** Clean Up *****" +DROP TABLE t,tm; diff --git a/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog-master.opt b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog-master.opt new file mode 100644 index 00000000000..8f0cc182f51 --- /dev/null +++ b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog-master.opt @@ -0,0 +1 @@ +--flashback diff --git a/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test new file mode 100644 index 00000000000..bda32af5d4e --- /dev/null +++ b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test @@ -0,0 +1,117 @@ +# ==== Purpose ==== +# +# Test verifies that point in time recovery of binary log works when +# sql_mode='ORACLE'. +# +# BEGIN statement is printed in three places +# 1) "Gtid_log_event::print" +# 2) "Xid_log_event::print" if flashback is enabled +# 3) "Query_log_event::print" if flashback is enabled and engine is +# non-transacional. +# +# Test verifies all these cases. +# +# ==== References ==== +# +# MDEV-23108: Point in time recovery of binary log fails when sql_mode=ORACLE +# +--source include/have_log_bin.inc +--source include/have_innodb.inc + +let $MYSQLD_DATADIR= `select @@datadir`; +SET @@SQL_MODE = 'ORACLE'; + +--echo ########################################################################## +--echo # Test verifies Gtid_log_event/Xid_log_event specific print # +--echo ########################################################################## +CREATE TABLE tm (f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); + +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); + +DELIMITER /; +CREATE OR REPLACE PROCEDURE simpleproc (param1 OUT INT) AS + BEGIN + SELECT COUNT(*) INTO param1 FROM t; + END; +/ +CREATE FUNCTION f1 RETURN INT +AS +BEGIN + RETURN 10; +END; +/ +DELIMITER ;/ + +FLUSH LOGS; +--echo ########################################################################## +--echo # Delete data from master so that it can be restored from binlog # +--echo ########################################################################## +DROP FUNCTION f1; +DROP PROCEDURE simpleproc; +DROP TABLE tm; +DROP TABLE t; + +--exec $MYSQL_BINLOG $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/test.sql +--exec $MYSQL < $MYSQLTEST_VARDIR/tmp/test.sql + +--echo ########################################################################## +--echo # Post recovery using mysqlbinlog # +--echo ########################################################################## +SHOW TABLES; +SELECT * FROM tm; +SELECT * FROM t; +--horizontal_results +SELECT f1(); +CALL simpleproc(@a); +SELECT @a; + +--echo "***** Clean Up *****" +DROP TABLE t,tm; +DROP PROCEDURE simpleproc; +DROP FUNCTION f1; +--remove_file $MYSQLTEST_VARDIR/tmp/test.sql +RESET MASTER; + +--echo ########################################################################## +--echo # Test verifies Gtid_log_event/Xid_log_event/Qery_log_event # +--echo # specific print along with flashback option # +--echo ########################################################################## +CREATE TABLE tm(f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); +INSERT INTO tm VALUES (20); +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +INSERT INTO t VALUES (20); +--echo ########################################################################## +--echo # Initial data # +--echo ########################################################################## +SELECT * FROM tm; +SELECT * FROM t; +FLUSH LOGS; +DELETE FROM tm WHERE f=20; +DELETE FROM t WHERE f=20; +FLUSH LOGS; + +--echo ########################################################################## +--echo # Data after deletion # +--echo ########################################################################## +SELECT * FROM tm; +SELECT * FROM t; +--exec $MYSQL_BINLOG --flashback $MYSQLD_DATADIR/master-bin.000002 > $MYSQLTEST_VARDIR/tmp/test.sql + +--let SEARCH_FILE=$MYSQLTEST_VARDIR/tmp/test.sql +--let SEARCH_PATTERN=START TRANSACTION +--source include/search_pattern_in_file.inc +--exec $MYSQL < $MYSQLTEST_VARDIR/tmp/test.sql + +--echo ########################################################################## +--echo # Data after recovery using flashback # +--echo ########################################################################## +SELECT * FROM tm; +SELECT * FROM t; + +--echo "***** Clean Up *****" +DROP TABLE t,tm; +--remove_file $MYSQLTEST_VARDIR/tmp/test.sql diff --git a/mysql-test/suite/rpl/r/rpl_blackhole_row_annotate.result b/mysql-test/suite/rpl/r/rpl_blackhole_row_annotate.result index 3c5711855d8..2e75080810b 100644 --- a/mysql-test/suite/rpl/r/rpl_blackhole_row_annotate.result +++ b/mysql-test/suite/rpl/r/rpl_blackhole_row_annotate.result @@ -230,7 +230,7 @@ ALTER TABLE t1 ENGINE=BLACKHOLE #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-3 trans /*!100001 SET @@session.server_id=1*//*!*/; /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -248,7 +248,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-4 trans /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -266,7 +266,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-5 trans /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -284,7 +284,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-6 trans /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -302,7 +302,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-7 trans /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -328,7 +328,7 @@ ALTER TABLE t1 ADD PRIMARY KEY pk_t1 (a,b) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-9 trans /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -346,7 +346,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-10 trans /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -364,7 +364,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-11 trans /*!100001 SET @@session.gtid_seq_no=11*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -390,7 +390,7 @@ ALTER TABLE t1 DROP PRIMARY KEY, ADD KEY key_t1 (a) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-13 trans /*!100001 SET @@session.gtid_seq_no=13*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -408,7 +408,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-14 trans /*!100001 SET @@session.gtid_seq_no=14*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -426,7 +426,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-15 trans /*!100001 SET @@session.gtid_seq_no=15*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/rpl/r/rpl_hrtime.result b/mysql-test/suite/rpl/r/rpl_hrtime.result index 7717dcf6700..d878a8d9df9 100644 --- a/mysql-test/suite/rpl/r/rpl_hrtime.result +++ b/mysql-test/suite/rpl/r/rpl_hrtime.result @@ -46,7 +46,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a timestamp(4) NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, b varchar(100), c datetime(2)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1293832861.123456/*!*/; SET @@session.time_zone='+03:00'/*!*/; @@ -55,7 +55,7 @@ insert t1 (b,c) values (now(6), now(6)) SET TIMESTAMP=1293832861.123456/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1293832861/*!*/; insert t1 values ('2010-10-10 10:10:10.101010','2010-10-10 10:10:10.101010','2010-10-10 10:10:10.101010') @@ -63,7 +63,7 @@ insert t1 values ('2010-10-10 10:10:10.101010','2010-10-10 10:10:10.101010','201 SET TIMESTAMP=1293832861/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1643756522.654321/*!*/; insert t1 (b,c) values (now(), now()) @@ -71,7 +71,7 @@ insert t1 (b,c) values (now(), now()) SET TIMESTAMP=1643756522.654321/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1643756522.654321/*!*/; insert t1 (b,c) values (0,0) @@ -79,7 +79,7 @@ insert t1 (b,c) values (0,0) SET TIMESTAMP=1643756522.654321/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1643756522.654321/*!*/; insert t1 (a,b,c) values (0,0,now(6)) diff --git a/mysql-test/suite/rpl/r/rpl_mdev382.result b/mysql-test/suite/rpl/r/rpl_mdev382.result index ee9377906ad..7e26e82d8c9 100644 --- a/mysql-test/suite/rpl/r/rpl_mdev382.result +++ b/mysql-test/suite/rpl/r/rpl_mdev382.result @@ -189,7 +189,7 @@ SET @@session.collation_database=DEFAULT/*!*/; CREATE TABLE `t``1` (`a``1` VARCHAR(4) PRIMARY KEY, `b``2` VARCHAR(3), `c``3` VARCHAR(7)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE '<name>' INTO TABLE `t``1` FIELDS TERMINATED BY ',' ENCLOSED BY '\'' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`a``1`, @`b```) SET `b``2`= @`b```, `c``3`= concat('|', "b""a'z", "!") @@ -200,7 +200,7 @@ COMMIT SET TIMESTAMP=1000000000/*!*/; truncate `t``1` /*!*/; -BEGIN +START TRANSACTION /*!*/; use `test`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -288,7 +288,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; CREATE TABLE t1 (a1 BIGINT PRIMARY KEY, a2 BIGINT, a3 BIGINT, a4 BIGINT UNSIGNED, b DOUBLE, c DECIMAL(65,10), d VARCHAR(100)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; INSERT INTO t1 VALUES (-9223372036854775808,42,9223372036854775807,18446744073709551615,-1234560123456789e110, -1234501234567890123456789012345678901234567890123456789.0123456789, REPEAT("x", 100)) @@ -296,7 +296,7 @@ INSERT INTO t1 VALUES (-9223372036854775808,42,9223372036854775807,1844674407370 SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET @`a``1`:=-9223372036854775808/*!*/; SET @`a``2`:=42/*!*/; @@ -383,7 +383,7 @@ master-bin.000002 # Query 1 # COMMIT /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; use `ts``et`/*!*/; SET TIMESTAMP=1000000000/*!*/; diff --git a/mysql-test/suite/rpl/r/rpl_sp.result b/mysql-test/suite/rpl/r/rpl_sp.result index 97cc4dad8ad..686380427b5 100644 --- a/mysql-test/suite/rpl/r/rpl_sp.result +++ b/mysql-test/suite/rpl/r/rpl_sp.result @@ -810,7 +810,7 @@ insert into t1 values (b); insert into t1 values (unix_timestamp()); end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values ( NAME_CONST('b',8)) @@ -818,7 +818,7 @@ insert into t1 values ( NAME_CONST('b',8)) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values (unix_timestamp()) @@ -826,7 +826,7 @@ insert into t1 values (unix_timestamp()) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete from t1 @@ -872,7 +872,7 @@ insert into t2 values(3); insert into t1 values (5); end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t2 values(3) @@ -880,7 +880,7 @@ insert into t2 values(3) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values (15) @@ -888,7 +888,7 @@ insert into t1 values (15) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t2 values(3) @@ -899,7 +899,7 @@ COMMIT SET TIMESTAMP=t/*!*/; alter procedure foo4 sql security invoker /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t2 values(3) @@ -907,7 +907,7 @@ insert into t2 values(3) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values (5) @@ -915,7 +915,7 @@ insert into t1 values (5) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete from t2 @@ -936,7 +936,7 @@ begin insert into t2 values(20),(20); end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t2 values(20),(20) @@ -964,7 +964,7 @@ insert into t1 values (x); return x+2; end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete t1,t2 from t1,t2 @@ -972,7 +972,7 @@ delete t1,t2 from t1,t2 SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; SELECT `mysqltest1`.`fn1`(20) @@ -980,7 +980,7 @@ SELECT `mysqltest1`.`fn1`(20) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t2 values(fn1(21)) @@ -998,7 +998,7 @@ begin return unix_timestamp(); end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete from t1 @@ -1006,7 +1006,7 @@ delete from t1 SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values(fn1()) @@ -1028,7 +1028,7 @@ begin return 0; end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete from t2 @@ -1049,7 +1049,7 @@ insert into t2 values(x),(x); return 10; end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; SELECT `mysqltest1`.`fn1`(100) @@ -1057,7 +1057,7 @@ SELECT `mysqltest1`.`fn1`(100) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; SELECT `mysqltest1`.`fn1`(20) @@ -1065,7 +1065,7 @@ SELECT `mysqltest1`.`fn1`(20) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete from t1 @@ -1076,7 +1076,7 @@ COMMIT SET TIMESTAMP=t/*!*/; CREATE DEFINER=`root`@`localhost` trigger trg before insert on t1 for each row set new.a= 10 /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values (1) @@ -1084,7 +1084,7 @@ insert into t1 values (1) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete from t1 @@ -1095,7 +1095,7 @@ COMMIT SET TIMESTAMP=t/*!*/; drop trigger trg /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values (1) @@ -1142,7 +1142,7 @@ CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW SET TIMESTAMP=t/*!*/; create table t1 (a int) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 (a) values (f1()) @@ -1169,7 +1169,7 @@ SET TIMESTAMP=t/*!*/; CREATE DEFINER=`root`@`localhost` PROCEDURE `p1`(arg VARCHAR(10)) INSERT INTO t1 VALUES(arg) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; INSERT INTO t1 VALUES( NAME_CONST('arg',_latin1'test' COLLATE 'latin1_swedish_ci')) @@ -1223,7 +1223,7 @@ SET TIMESTAMP=t/*!*/; CREATE DEFINER=`root`@`localhost` PROCEDURE `mysqltest`.`test`() begin end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t values ( 1 ) @@ -1238,7 +1238,7 @@ insert into t values (1); return 0; end /*!*/; -BEGIN +START TRANSACTION /*!*/; use `mysqltest`/*!*/; SET TIMESTAMP=t/*!*/; diff --git a/sql/log_event.cc b/sql/log_event.cc index 57a39085e86..964f451a766 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -5358,8 +5358,7 @@ bool Query_log_event::print(FILE* file, PRINT_EVENT_INFO* print_event_info) } else if (strcmp("COMMIT", query) == 0) { - if (my_b_write(&cache, (uchar*) "BEGIN", 5) || - my_b_printf(&cache, "\n%s\n", print_event_info->delimiter)) + if (my_b_printf(&cache, "START TRANSACTION\n%s\n", print_event_info->delimiter)) goto err; } } @@ -8254,7 +8253,8 @@ Gtid_log_event::print(FILE *file, PRINT_EVENT_INFO *print_event_info) goto err; } if (!(flags2 & FL_STANDALONE)) - if (my_b_printf(&cache, is_flashback ? "COMMIT\n%s\n" : "BEGIN\n%s\n", print_event_info->delimiter)) + if (my_b_printf(&cache, is_flashback ? "COMMIT\n%s\n" : + "START TRANSACTION\n%s\n", print_event_info->delimiter)) goto err; return cache.flush_data(); @@ -8937,7 +8937,7 @@ bool Xid_log_event::print(FILE* file, PRINT_EVENT_INFO* print_event_info) my_b_printf(&cache, "\tXid = %s\n", buf)) goto err; } - if (my_b_printf(&cache, is_flashback ? "BEGIN%s\n" : "COMMIT%s\n", + if (my_b_printf(&cache, is_flashback ? "START TRANSACTION%s\n" : "COMMIT%s\n", print_event_info->delimiter)) goto err;

1 0

[Maria-developers] Progress Report GSoC 2020 (6-12 Jule)
by Anton Mikhaylenko 16 Jul '20

16 Jul '20

Good day, everybody. In this post, I want to describe what I did this week, Jule 6-12. I was dealing with the benchmarking for a treadpool. I used https://github.com/vaintroub/benchmark_tools. I installed it and made initial launches. I had to run several commands to launch. 1. Compile the server with RelWithDebInfo. cmake --build. -config RelWithDebInfo -- -m 2. cd sql\RelWithDebInfo start mysqld.exe --defaults-file=C:\path\to\sysbench.ini 3. cd C:\path\to\benchmark_tools mysql -uroot -e "create database sbtest" 4. update_index prepare 1024 5. update_index run 1024 1024 is the number of clients. Regards, Anton MIkhailenko

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 14 Jul '20

14 Jul '20

Hi, This week - Started working on changing the build system to make the plugins call our functions instead of the library functions as per step 4 of MDEV-22895. - I faced a lot of issues with cmake, as INCLUDE_DIRECTORIES does not work with CHECK_INCLUDE_FILES, and setting CMAKE_REQUIRED_INCLUDES in cmake/plugins.cmake did not affect storage/innobase/lzma.cmake. The only thing that worked was STRING(APPEND CMAKE_C_FLAGS " -I${CMAKE_SOURCE_DIR}/include/mysql"), but that sets the flag globally, so is not ideal. - I spoke with some devs on the cmake IRC (#cmake), and it seems that InnoDB uses improper Cmake commands. It might be easier for the project to rewrite them instead of trying to work around the issues. Next week - Review the CMake files and finish updating the build system as per current standards. - Continue work on step 5 of MDEV-22895. Thanks, Kartik

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 06 Jul '20

06 Jul '20

Hi, This week - Started working on changing the build system to make the plugins call our functions instead of the library functions as per step 4 of MDEV-22895. - Worked on a proof-of-concept (poc-noservice-lzma) that bypassed the fake dynamic linker, and thus simplified the code greatly. Decided against using it because of stability and compatibility concerns. Next week - Finish tweaking the build system. - Start work step 5 on MDEV-22895. Thanks, Kartik

1 0

Re: [Maria-developers] cb2cd9c3a43: MDEV-22058: Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())' failed in Diagnostics_area::set_ok_status
by Sergei Golubchik 02 Jul '20

02 Jul '20

Hi, Rucha! On Jul 02, Rucha Deodhar wrote: > revision-id: cb2cd9c3a43 (mariadb-10.2.31-302-gcb2cd9c3a43) > parent(s): dba7e1e8e14 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-07-02 19:51:55 +0530 > message: > > MDEV-22058: Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())' failed in Diagnostics_area::set_ok_status > > Error state is not stored in check_and_do_in_subquery_rewrites() when there is > illegal combination of optimizer switches. So all the functions eventually > return false. Thus the assetion failure. > > diff --git a/sql/opt_subselect.cc b/sql/opt_subselect.cc > index 2fb5b88687f..e37b389836c 100644 > --- a/sql/opt_subselect.cc > +++ b/sql/opt_subselect.cc > @@ -559,6 +559,7 @@ bool is_materialization_applicable(THD *thd, Item_in_subselect *in_subs, > int check_and_do_in_subquery_rewrites(JOIN *join) > { > THD *thd=join->thd; > + int res= 0; > st_select_lex *select_lex= join->select_lex; > st_select_lex_unit* parent_unit= select_lex->master_unit(); > DBUG_ENTER("check_and_do_in_subquery_rewrites"); > @@ -703,7 +704,11 @@ int check_and_do_in_subquery_rewrites(JOIN *join) > /* Test if the user has set a legal combination of optimizer switches. */ > if (!optimizer_flag(thd, OPTIMIZER_SWITCH_IN_TO_EXISTS) && > !optimizer_flag(thd, OPTIMIZER_SWITCH_MATERIALIZATION)) > + { > my_error(ER_ILLEGAL_SUBQUERY_OPTIMIZER_SWITCHES, MYF(0)); I think this is very weird behavior. A user sets incorrect optimizer switch, SET statement succeeds. But later at some point SELECT returns an error. You have correctly found the place where the error status is not properly returned and fixed it. But I'd rather suggest to move the check to where it should've been. Here instead of the if() you put an assert: DBUG_ASSERT(optimizer_flag(thd, OPTIMIZER_SWITCH_IN_TO_EXISTS | OPTIMIZER_SWITCH_MATERIALIZATION)); and the if() with an error should be in sys_vars.cc. So that the user would get an error from the SET statement. Look for ON_CHECK in the sys_vars.cc > + res= -1; > + } > + > /* > Transform each subquery predicate according to its overloaded > transformer. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] cf0fa125877: MENT-471: Plugin implementation for the Hashicorp Vault KMS
by Sergei Golubchik 01 Jul '20

01 Jul '20

Hi, Julius! On Jul 01, Julius Goryavsky wrote: > revision-id: cf0fa125877 (mariadb-10.5.2-177-gcf0fa125877) > parent(s): e7ca9ec6f6b > author: Julius Goryavsky <julius.goryavsky(a)mariadb.com> > committer: Julius Goryavsky <julius.goryavsky(a)mariadb.com> > timestamp: 2020-04-20 15:28:49 +0200 > message: > > MENT-471: Plugin implementation for the Hashicorp Vault KMS > > - Authentication is done using the Hashicorp Vault's token > authentication method; > - If additional client authentication is required, then the > path to the CA authentication bundle file may be passed > as a plugin parameter; > - The creation of the keys and their management is carried > out using the Hashicorp Vault KMS and their tools; > - Key values stored as hexadecimal strings; > - Key values caching is supported. > - The plugin uses libcurl (https) as an interface to > the HashiCorp Vault server; > - JSON parsing is performed through the JSON service > (through the include/mysql/service_json.h); > - HashiCorp Vault 1.2.4 was used for development and testing. > > --- > mysql-test/suite/vault/disabled.def | 11 + > mysql-test/suite/vault/my.cnf | 1 + > mysql-test/suite/vault/r/hashicorp_encode.result | 45 ++ > mysql-test/suite/vault/suite.pm | 27 + > mysql-test/suite/vault/t/hashicorp_deinit.inc | 6 + > mysql-test/suite/vault/t/hashicorp_encode.opt | 5 + > mysql-test/suite/vault/t/hashicorp_encode.test | 3 + > mysql-test/suite/vault/t/hashicorp_goodtest.inc | 20 + > mysql-test/suite/vault/t/hashicorp_init.inc | 32 + > mysql-test/suite/vault/t/hashicorp_plugin.inc | 1 + please, put the plugin specific test suite into the plugin dir: plugin/hashicorp_key_management/mysql-test/vault > plugin/hashicorp_key_management/CMakeLists.txt | 11 + > .../hashicorp_key_management_plugin.cc | 899 +++++++++++++++++++++ > plugin/hashicorp_key_management/readme.txt | 138 ++++ > 14 files changed, 1203 insertions(+) > > diff --git a/mysql-test/suite/vault/disabled.def b/mysql-test/suite/vault/disabled.def > --- /dev/null > +++ b/mysql-test/suite/vault/disabled.def > @@ -0,0 +1,11 @@ > +############################################################################## > +# > +# List the test cases that are to be disabled temporarily. > +# > +# Separate the test case name and the comment with ':'. > +# > +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> > +# > +# Do not use any TAB characters for whitespace. > +# > +############################################################################## you don't have to create and commit this template > diff --git a/mysql-test/suite/vault/include/have_hashicorp_key_management_plugin.inc b/mysql-test/suite/vault/include/have_hashicorp_key_management_plugin.inc > --- /dev/null > +++ b/mysql-test/suite/vault/include/have_hashicorp_key_management_plugin.inc > @@ -0,0 +1,4 @@ > +if (`SELECT COUNT(*)=0 FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_NAME = 'hashicorp_key_management' AND PLUGIN_STATUS='ACTIVE'`) > +{ > + --skip Test requires active hashicorp_key_management plugin > +} > diff --git a/mysql-test/suite/vault/my.cnf b/mysql-test/suite/vault/my.cnf > --- /dev/null > +++ b/mysql-test/suite/vault/my.cnf > @@ -0,0 +1 @@ > +!include include/default_my.cnf and you definitely should not create suite my.cnf that only includes the default my.cnf. You only need a custom my.cnf if it adds some extra options that are not default. > diff --git a/mysql-test/suite/vault/suite.pm b/mysql-test/suite/vault/suite.pm > --- /dev/null > +++ b/mysql-test/suite/vault/suite.pm > @@ -0,0 +1,27 @@ > +package My::Suite::Vault; > + > +@ISA = qw(My::Suite); > + > +use strict; > + > +if (::IS_WINDOWS) { > + return "The test suite for Hashicorp Vault plugin currently does not support Windows"; > +} instead of a custom check, you'd better add source include/not_windows.inc into your have_hashicorp_key_management_plugin.inc > + > +return "Hashicorp Vault management utility not found" > + unless ::which("vault"); > + > +return "You need to set the value of the VAULT_ADDR variable" > + unless $ENV{VAULT_ADDR}; > + > +return "You need to set the value of the VAULT_TOKEN variable" > + unless $ENV{VAULT_TOKEN}; > + > +sub skip_combinations { > + my %skip; > + $skip{'include/have_hashicorp_key_management_plugin.inc'} = 'Needs dynamic hashicorp_key_management plugin' > + unless $ENV{HASHICORP_KEY_MANAGEMENT_SO}; this is rather redundant, isn't it? have_hashicorp_key_management_plugin.inc itself checks for the plugin and skips. If you remove your skip_combinations() function, the behavior will be exactly the same. Normally one needs to skip tests from suite.pm for conditions that cannot be checked from inside the test. > + %skip; > +} > + > +bless {}; > diff --git a/mysql-test/suite/vault/t/hashicorp_deinit.inc b/mysql-test/suite/vault/t/hashicorp_deinit.inc > --- /dev/null > +++ b/mysql-test/suite/vault/t/hashicorp_deinit.inc > @@ -0,0 +1,6 @@ > +perl; > + `vault secrets disable test`; 1. perl has "system" command, one normally uses backticks when one needs the output. 2. mysqltest will write ths small perl script into a file and will run perl on it. How is this overhead better than just exec vault secrets disable test; with no perl at all? 3. why did you create a separate hashicorp_deinit.inc if you only use it once? > + if (($? >> 8) != 0) { > + die "Vault: unable to destroy path for test keys"; > + } > +EOF > diff --git a/mysql-test/suite/vault/t/hashicorp_init.inc b/mysql-test/suite/vault/t/hashicorp_init.inc > --- /dev/null > +++ b/mysql-test/suite/vault/t/hashicorp_init.inc > @@ -0,0 +1,32 @@ > +perl; > +my $list = `vault secrets list`; > +if (($? >> 8) != 0) { > + die "Unable to retrieve secrets list - Hashicorp Vault not configured properly or not enough permissions"; > +} > +if ($list !~ /(^|\n|\r)test\/[\s\t]+kv[\s\t]+/) { > + `vault secrets enable -path /test -version=2 kv`; > + if (($? >> 8) != 0) { > + die "Vault: unable to create path for test keys"; > + } > + `vault kv put /test/1 data="123456789ABCDEF0123456789ABCDEF0"`; > + if (($? >> 8) != 0) { > + die "Vault: unable to create first secret key"; > + } > + `vault kv put /test/2 data="23456789ABCDEF0123456789ABCDEF01"`; > + if (($? >> 8) != 0) { > + die "Vault: unable to create second secret key"; > + } > + `vault kv put /test/33 data="00000000000000000000000000000000"`; > + if (($? >> 8) != 0) { > + die "Vault: unable to create third secret key"; > + } > + `vault kv put /test/33 data="3456789ABCDEF0123456789ABCDEF012"`; > + if (($? >> 8) != 0) { > + die "Vault: unable to update third secret key"; > + } > + `vault kv put /test/4 data="456789ABCDEF0123456789ABCDEF0123"`; > + if (($? >> 8) != 0) { > + die "Vault: unable to create fourth secret key"; > + } > +} > +EOF same as above. you shouldn't abuse backticks, don't need perl and you don't need a separate file. Use something like directly in your hashicorp_goodtest.test exec vault secrets disable test; # start from a clean state exec vault secrets enable -path /test -version=2 kv; exec vault kv put /test/1 data="123456789ABCDEF0123456789ABCDEF0"; exec vault kv put /test/2 data="23456789ABCDEF0123456789ABCDEF01"; exec vault kv put /test/33 data="00000000000000000000000000000000"; exec vault kv put /test/33 data="3456789ABCDEF0123456789ABCDEF012"; exec vault kv put /test/4 data="456789ABCDEF0123456789ABCDEF0123"; also, I'd recommend to use a more specific secret name, like mariadbtest > diff --git a/mysql-test/suite/vault/t/hashicorp_goodtest.inc b/mysql-test/suite/vault/t/hashicorp_goodtest.inc > --- /dev/null > +++ b/mysql-test/suite/vault/t/hashicorp_goodtest.inc > @@ -0,0 +1,20 @@ > +--source include/have_innodb.inc > +--source hashicorp_plugin.inc > + > +SHOW GLOBAL variables WHERE Variable_name LIKE "hashicorp%" AND Variable_name NOT LIKE "%vault_url"; normally it's done like replace_result $VAULT_ADDR VAULT_ADDR; SHOW GLOBAL variables LIKE "hashicorp%"; that is, you show all variables and replace away the dynamic part. > + > +create table t1(c1 bigint not null, b char(200)) engine=innodb encrypted=yes encryption_key_id=1; > +show create table t1; > +insert t1 values (12345, repeat('1234567890', 20)); > + > +alter table t1 encryption_key_id=2; > +show create table t1; > +--error ER_ILLEGAL_HA_CREATE_OPTION > +alter table t1 encryption_key_id=3; > +show create table t1; > +alter table t1 encryption_key_id=33; > +show create table t1; > +alter table t1 encryption_key_id=4; > +show create table t1; where's a test for key rotation? where's an encrypt/restart/decrypt test? > + > +drop table t1; > diff --git a/mysql-test/suite/vault/t/hashicorp_encode.opt b/mysql-test/suite/vault/t/hashicorp_encode.opt > --- /dev/null > +++ b/mysql-test/suite/vault/t/hashicorp_encode.opt > @@ -0,0 +1,5 @@ > +--plugin-load-add=$HASHICORP_KEY_MANAGEMENT_SO > +--loose-hashicorp-key-management > +--loose-hashicorp-key-management-vault-url="$VAULT_ADDR/v1/test/" > +--loose-hashicorp-key-management-token="$VAULT_TOKEN" > +--loose-hashicorp-key-management-timeout=60 > diff --git a/mysql-test/suite/vault/t/hashicorp_encode.test b/mysql-test/suite/vault/t/hashicorp_encode.test > --- /dev/null > +++ b/mysql-test/suite/vault/t/hashicorp_encode.test > @@ -0,0 +1,3 @@ > +source hashicorp_init.inc; > +source hashicorp_goodtest.inc; > +source hashicorp_deinit.inc; > diff --git a/mysql-test/suite/vault/t/hashicorp_plugin.inc b/mysql-test/suite/vault/t/hashicorp_plugin.inc > --- /dev/null > +++ b/mysql-test/suite/vault/t/hashicorp_plugin.inc > @@ -0,0 +1 @@ > +--source include/have_hashicorp_key_management_plugin.inc another file that it's only used once? what's the point? include have_hashicorp_key_management_plugin.inc directly into your hashicorp_encode.test > diff --git a/plugin/hashicorp_key_management/hashicorp_key_management_plugin.cc b/plugin/hashicorp_key_management/hashicorp_key_management_plugin.cc > --- /dev/null > +++ b/plugin/hashicorp_key_management/hashicorp_key_management_plugin.cc > @@ -0,0 +1,899 @@ > +/* Copyright (C) 2019 MariaDB Corporation 2019, 2020 > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */ > + > +#include <mysql/plugin_encryption.h> > +#include <mysqld_error.h> > +#include <string.h> > +#include <stdlib.h> > +#include <limits.h> > +#include <errno.h> > +#include <string> > +#include <sstream> > +#include <curl/curl.h> > +#ifdef _WIN32 > +#include <malloc.h> > +#define alloca _alloca > +#else > +#include <alloca.h> > +#endif > +#include <algorithm> > +#include <map> > +#include <mutex> > + > +static std::mutex mtx; > + > +/* Key information structure: */ > +typedef struct KEY_INFO > +{ > + unsigned int key_id; > + unsigned int key_version; > + unsigned int length; > + unsigned char data [MY_AES_MAX_KEY_LENGTH]; > +public: > + KEY_INFO() : key_id(0), key_version(0), length(0) {}; > +} KEY_INFO; > + > +/* Cache for the latest version, per key id: */ > +static std::map<unsigned int, unsigned int> latest_version_cache; > + > +/* Cache for key information: */ > +static std::map<unsigned long long, KEY_INFO> key_info_cache; why not unordered_map here and above? where do you use the ordering property? > + > +#define KEY_ID_AND_VERSION(key_id, version) \ > + ((unsigned long long)key_id << 32 | version) > + > +static void cache_add (unsigned int key_id, unsigned int key_version, > + const KEY_INFO* info) > +{ > + mtx.lock(); > + latest_version_cache[key_id]= > + std::max(latest_version_cache[key_id], key_version); > + key_info_cache[KEY_ID_AND_VERSION(key_id, key_version)]= *info; > + mtx.unlock(); > +} > + > +static unsigned int > + cache_get (unsigned int key_id, unsigned int key_version, > + unsigned char* data, unsigned int* buflen) > +{ > + unsigned int ver= key_version; > + mtx.lock(); > + if (key_version == ENCRYPTION_KEY_VERSION_INVALID) > + { > + ver= latest_version_cache[key_id]; > + if (ver == 0) > + { > + mtx.unlock(); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + } > + KEY_INFO info= key_info_cache[KEY_ID_AND_VERSION(key_id, ver)]; > + mtx.unlock(); > + unsigned int length= info.length; > + if (length == 0) > + { > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + unsigned int max_length = *buflen; > + *buflen = length; > + if (max_length < length) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Encryption key buffer too small", > + ME_ERROR_LOG_ONLY | ME_NOTE); I don't think you need an error here, this is not an error from the caller point of view, but just a way to query key length. > + return ENCRYPTION_KEY_BUFFER_TOO_SMALL; > + } > + memcpy(data, info.data, length); > + return 0; > +} > + > +static unsigned int cache_get_version (unsigned int key_id) > +{ > + unsigned int ver; > + mtx.lock(); > + ver= latest_version_cache[key_id]; > + mtx.unlock(); > + if (ver) > + { > + return ver; > + } > + else > + { > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > +} > + > +static char* vault_url; > +static char* token; > +static char* vault_ca; > +int timeout; > +int max_retries; > +char caching_enabled; > +char use_cache_on_timeout; why the other four aren't static? > + > +static MYSQL_SYSVAR_STR(vault_ca, vault_ca, > + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY, > + "Path to the Certificate Authority (CA) bundle (is a file " > + "that contains root and intermediate certificates)", > + NULL, NULL, ""); > + > +static MYSQL_SYSVAR_STR(vault_url, vault_url, > + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY, > + "HTTP[s] URL that is used to connect to the Hashicorp Vault server", > + NULL, NULL, "https://127.0.0.1:8200/v1/kv"); Is this a reasonable default for a vault url? I'd rather keep it empty. > + > +static MYSQL_SYSVAR_STR(token, token, > + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY | PLUGIN_VAR_NOSYSVAR, > + "Authentication token that passed to the Hashicorp Vault " > + "in the request header", > + NULL, NULL, ""); > + > +static MYSQL_SYSVAR_INT(timeout, timeout, > + PLUGIN_VAR_RQCMDARG, > + "Duration (in seconds) for the Hashicorp Vault server " > + "connection timeout", > + NULL, NULL, 15, 0, 86400, 1); > + > +static MYSQL_SYSVAR_INT(max_retries, max_retries, > + PLUGIN_VAR_RQCMDARG, > + "Number of server request retries in case of timeout", > + NULL, NULL, 3, 0, INT_MAX, 1); > + > +static MYSQL_SYSVAR_BOOL(caching_enabled, caching_enabled, > + PLUGIN_VAR_RQCMDARG, > + "Enable key caching (storing key values received from " > + "the Hashicorp Vault server in the local memory)", > + NULL, NULL, 1); > + > +static MYSQL_SYSVAR_BOOL(use_cache_on_timeout, use_cache_on_timeout, > + PLUGIN_VAR_RQCMDARG, > + "In case of timeout (when accessing the vault server) " > + "use the value taken from the cache", > + NULL, NULL, 0); > + > +static struct st_mysql_sys_var *settings[] = { > + MYSQL_SYSVAR(vault_url), > + MYSQL_SYSVAR(token), > + MYSQL_SYSVAR(vault_ca), > + MYSQL_SYSVAR(timeout), > + MYSQL_SYSVAR(max_retries), > + MYSQL_SYSVAR(caching_enabled), > + MYSQL_SYSVAR(use_cache_on_timeout), > + NULL > +}; > + > +static std::string get_error_from_curl (CURLcode curl_code, char *curl_errbuf) > +{ > + size_t len = strlen(curl_errbuf); > + std::ostringstream stream; > + if (curl_code != CURLE_OK) > + { > + stream << "CURL returned this error code: " << curl_code; > + stream << " with error message : "; > + if (len) > + stream << curl_errbuf; > + else > + stream << curl_easy_strerror(curl_code); > + } > + return stream.str(); > +} > + > +/* > + Reasonable length limit to protect against accidentally reading > + the wrong key or from trying to overload the server with unnecessary > + work to receive too long responses to requests: > +*/ > +#define max_response_size 131072 conventionally, it should be uppercase > + > +static size_t write_response_memory (void *contents, size_t size, size_t nmemb, > + void *userp) > +{ > + size_t realsize = size * nmemb; > + if (size != 0 && realsize / size != nmemb) > + return 0; // multiplication overflow detected this cannot happen, size is documented to be always 1, and the max data size is CURL_MAX_WRITE_SIZE anyway. > + std::ostringstream *read_data = static_cast<std::ostringstream *>(userp); > + size_t current_length = read_data->tellp(); > + if (current_length + realsize > max_response_size) > + return 0; // response size limit exceeded > + read_data->write(static_cast<char *>(contents), realsize); > + if (!read_data->good()) > + return 0; > + return realsize; > +} > + > +enum { > + OPERATION_OK, > + OPERATION_TIMEOUT, > + OPERATION_ERROR > +}; > + > +static CURLcode > + perform_with_retries (CURL *curl, std::ostringstream *read_data_stream) > +{ > + int retries= max_retries; > + CURLcode curl_res; > + do { > + curl_res= curl_easy_perform(curl); > + if (curl_res != CURLE_OPERATION_TIMEDOUT) > + { > + break; > + } > + read_data_stream->clear(); > + read_data_stream->str(""); I presume, this wasn't tested at all? > + } while (retries--); > + return curl_res; > +} > + > +#define max_token_size 32768 uppercase again > + > +static size_t token_len; > + > +static int curl_run (char *url, std::string *response, bool soft_timeout) > +{ > + const static char *x_vault_token = "X-Vault-Token:"; > + const static size_t x_vault_token_len = strlen(x_vault_token); > +/* > + Checking the maximum allowable length to protect > + against allocating too much memory on the stack: > +*/ > + if (token_len > max_token_size) 32K is a lot, tokens are much shorter and stack isn't big. You can safely reduce max_token_size to, say 8K. and why do you check it here in curl_run anyway? you can check it in hashicorp_key_management_plugin_init. once. > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Maximum allowed token length exceeded", > + ME_ERROR_LOG_ONLY); > + return OPERATION_ERROR; > + } > + char curl_errbuf[CURL_ERROR_SIZE]; > + std::ostringstream read_data_stream; > + long http_code = 0; > + struct curl_slist *list = NULL; > + CURLcode curl_res = CURLE_OK; > + CURL *curl = curl_easy_init(); > + if (curl == NULL) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Cannot initialize curl session", > + ME_ERROR_LOG_ONLY); > + return OPERATION_ERROR; > + } > + size_t buf_len = x_vault_token_len + token_len + 1; > + char *token_header = (char *) alloca(buf_len); > + snprintf(token_header, buf_len, "%s%s", x_vault_token, token); also there's no need to allocate the buffer on the stack and concatenate the header here every time, you can do it once in hashicorp_key_management_plugin_init too. > + curl_errbuf[0] = '\0'; > + if ((list= curl_slist_append(list, token_header)) == NULL || and you can prepare slist only once too (I think). > + (curl_res= curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_errbuf)) != > + CURLE_OK || > + (curl_res= curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, > + write_response_memory)) != CURLE_OK || > + (curl_res= curl_easy_setopt(curl, CURLOPT_WRITEDATA, > + &read_data_stream)) != > + CURLE_OK || > + (curl_res= curl_easy_setopt(curl, CURLOPT_HTTPHEADER, list)) != > + CURLE_OK || > + /* > + The options CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOST are > + set explicitly to withstand possible future changes in curl defaults: > + */ > + (curl_res= curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1)) != > + CURLE_OK || > + (curl_res= curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 2L)) != > + CURLE_OK || > + (strlen(vault_ca) != 0 && > + (curl_res= curl_easy_setopt(curl, CURLOPT_CAINFO, vault_ca)) != > + CURLE_OK) || > + (curl_res= curl_easy_setopt(curl, CURLOPT_USE_SSL, CURLUSESSL_ALL)) != > + CURLE_OK || > + (curl_res= curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L)) != > + CURLE_OK || > + (timeout && > + ((curl_res= curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, timeout)) != > + CURLE_OK || > + (curl_res= curl_easy_setopt(curl, CURLOPT_TIMEOUT, timeout)) != > + CURLE_OK)) || > + (curl_res = curl_easy_setopt(curl, CURLOPT_URL, url)) != CURLE_OK || > + (curl_res = perform_with_retries(curl, &read_data_stream)) != CURLE_OK || > + (curl_res = curl_easy_getinfo (curl, CURLINFO_RESPONSE_CODE, > + &http_code)) != CURLE_OK) > + { > + curl_slist_free_all(list); > + curl_easy_cleanup(curl); > + if (soft_timeout && curl_res == CURLE_OPERATION_TIMEDOUT) > + { > + return OPERATION_TIMEOUT; > + } > + my_printf_error(ER_UNKNOWN_ERROR, > + get_error_from_curl(curl_res, curl_errbuf).c_str(), really? strlen, ostringstream, append/append/append, then you return std::string and then print its c_str()? It's printf, you can do directly my_printf_error(ER_UNKNOWN_ERROR, "CURL returned this error code %u" " with error message: %s", curl_res, curl_errbuf); Which is much easier to read, doesn't do all these string operations and reallocs, and it's also safe, because your variant will crash if any of the curl error messages will include % sign. > + ME_ERROR_LOG_ONLY); > + return OPERATION_ERROR; > + } > + curl_slist_free_all(list); > + curl_easy_cleanup(curl); > + *response = read_data_stream.str(); > + bool is_error = http_code < 200 || http_code >= 300; > + if (is_error) > + { > + const char *res = response->c_str(); > + /* > + Error 404 requires special handling - in case the server > + returned an empty array of error strings (the value of the > + "error" object in JSON is equal to an empty array), we should > + ignore this error at this level, since this means the missing > + key (this problem is handled at a higher level), but if the > + error object contains anything other than empty array, then > + we need to print the error message to the log: > + */ > + if (http_code == 404) > + { > + const char *err; > + int err_len; > + if (json_get_object_key(res, res + strlen(res), > + "errors", &err, &err_len) == JSV_ARRAY) > + { > + const char *ev; > + int ev_len; > + if (json_get_array_item(err, err + err_len, 0, &ev, &ev_len) == > + JSV_NOTHING) > + { > + *response = std::string(""); > + is_error = false; > + } > + } > + } > + if (is_error) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Hashicorp server error: %d, response: %s", > + ME_ERROR_LOG_ONLY | ME_WARNING, http_code, res); > + } > + } > + return is_error ? OPERATION_ERROR : OPERATION_OK; > +} > + > +#define max_url_size 32768 uppercase > + > +static size_t vault_url_len; > + > +static unsigned int get_latest_version (unsigned int key_id) > +{ > + std::string response_str; > + const char *response; > + const char *js, *ver; > + int js_len, ver_len; > + size_t response_len; > +/* > + Checking the maximum allowable length to protect > + against allocating too much memory on the stack: > +*/ > + if (vault_url_len > max_url_size) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Maximum allowed vault URL length exceeded", > + 0); > + return true; > + } > +/* > + Maximum buffer length = url length plus 20 characters of > + a 64-bit unsigned integer, plus a slash character, plus > + a length of the "data/" string and plus a zero byte: > +*/ please, indent all comments properly (here and above). now they'll cause diffs to be rather unreadable. > + size_t buf_len = vault_url_len + (20 + 5 + 2); > + char *url = (char *) alloca(buf_len); > + if (vault_url_len && vault_url[vault_url_len - 1] != '/') > + { > + snprintf(url, buf_len, "%s/data/%u", vault_url, key_id); > + } > + else > + { > + snprintf(url, buf_len, "%sdata/%u", vault_url, key_id); > + } same comment as above about run-time checks. You can make sure that vault_url_len is within limits and vault_url doesn't end with a slash in hashicorp_key_management_plugin_init > + bool use_cache= caching_enabled && use_cache_on_timeout; > + int rc; > + if ((rc= curl_run(url, &response_str, use_cache)) != OPERATION_OK) would be good to have some cache expiration time and skip curl completely if the cached value is sufficiently new. e.g. instead of caching_enabled you can have caching_lifetime in seconds. Where 0 means "no caching". > + { > + if (rc == OPERATION_TIMEOUT) > + { > + unsigned int ver = cache_get_version(key_id); > + if (ver != ENCRYPTION_KEY_VERSION_INVALID) > + { > + return ver; > + } > + } > + my_printf_error(ER_UNKNOWN_ERROR, "Unable to get key data", 0); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + response = response_str.c_str(); > + response_len = response_str.size(); > + /* > + If the key is not found, this is not considered a fatal error, > + but we need to add an informational message to the log: > + */ > + if (response_len == 0) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Key not found", always specify the plugin name in any my_printf_error. Having only 2020-06-29 1:41:23 28 [ERRROR] Key not found is not very helpful. > + ME_ERROR_LOG_ONLY | ME_NOTE); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (json_get_object_key(response, response + response_len, "data", > + &js, &js_len) != JSV_OBJECT) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get data object (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (json_get_object_key(js, js + js_len, "metadata", > + &ver, &ver_len) != JSV_OBJECT) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get metadata object (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (json_get_object_key(ver, ver + ver_len, "version", > + &ver, &ver_len) != JSV_NUMBER) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get version number (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + /* > + Here we may use atoi, but it has undefined overflow behavior > + and does not work with unsigned integers, so use strtoul: > + */ No need to explain why you didn't use atoi, this is not helpful. Or, if you'd like to explain, explain why you didn't use atol, atof, strtod, and sscanf :) > + errno = 0; > + unsigned long version = strtoul(ver, NULL, 10); > + /* > + If the result of the conversion does not fit into an > + unsigned integer or if an error (for example, overflow) > + occurred during the conversion, then we need to return > + an error: > + */ thanks, but do not say in a comment what the code does. explain *why* it does it. But in this case both are very clear, so just remove the comment above. Thanks. > + if (version > UINT_MAX || (version == ULONG_MAX && errno)) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Integer conversion error (for version number)", no "http response is: %s" here? > + 0); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + unsigned int int_ver= (unsigned int) version; > + if (!caching_enabled) > + { > + return int_ver; > + } > + if (json_get_object_key(js, js + js_len, "data", > + &js, &js_len) != JSV_OBJECT) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get second-level data object " > + "(http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + const char* key; > + int key_len; > + if (json_get_object_key(js, js + js_len, "data", > + &key, &key_len) != JSV_STRING) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get data string (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + KEY_INFO info; > + if (((unsigned int) key_len >> 1) > sizeof(info.data)) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Encryption key data is too long", > + ME_ERROR_LOG_ONLY | ME_NOTE); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + unsigned int length = 0; > + while (key_len >= 2) > + { > + int c1 = key[0]; > + int c2 = key[1]; > + if (! isxdigit(c1) || ! isxdigit(c2)) > + { > + break; > + } > + c1 -= '0'; > + c2 -= '0'; > + if (c1 > 9) > + { > + c1 -= 'A' - '0'; > + if (c1 > 15) > + { > + c1 -= 'a' - 'A'; > + } > + } > + if (c2 > 9) > + { > + c2 -= 'A' - '0'; > + if (c1 > 15) > + { > + c2 -= 'a' - 'A'; > + } > + } please write a macro or an inline function for that. don't repeat the same code twice. E.g. static inline int c2xdigit(char *c) { return c >= 'a' ? c-'a'+10 : c >= 'A' ? c-'A'+10 : c - '0'; } > + info.data[length++] = (c1 << 4) + c2; > + key += 2; > + key_len -= 2; > + } > + if (key_len) > + { > + if (key_len != 1) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Syntax error - the key data should contain only " > + "hexadecimal digits", > + 0); > + } > + else > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Syntax error - extra character in the key data", > + 0); > + } > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + info.key_id = key_id; > + info.key_version = int_ver; > + info.length = length; > + cache_add(key_id, int_ver, &info); > + return int_ver; > +} > + > +static unsigned int get_key_from_vault (unsigned int key_id, > + unsigned int key_version, > + unsigned char *dstbuf, > + unsigned int *buflen) > +{ > + if (caching_enabled && > + cache_get(key_id, key_version, dstbuf, buflen) != > + ENCRYPTION_KEY_VERSION_INVALID) > + { > + return 0; > + } > + std::string response_str; > + const char *response; > + const char *js, *key; > + int js_len, key_len; > + size_t response_len; > +/* > + Checking the maximum allowable length to protect > + against allocating too much memory on the stack: > +*/ > + if (vault_url_len > max_url_size) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Maximum allowed vault URL length exceeded", > + 0); > + return true; > + } > +/* > + Maximum buffer length = url length plus 40 characters of the > + two 64-bit unsigned integers, plus a slash character, plus a > + question mark, plus length of the "data/" and the "version=" > + strings and plus a zero byte: > +*/ > + size_t buf_len = vault_url_len + (40 + 5 + 8 + 3); > + char *url = (char *) alloca(buf_len); > + if (vault_url_len && vault_url[vault_url_len - 1] != '/') > + { > + if (key_version != ENCRYPTION_KEY_VERSION_INVALID) > + snprintf(url, buf_len, "%s/data/%u?version=%u", vault_url, key_id, > + key_version); > + else > + snprintf(url, buf_len, "%s/data/%u", vault_url, key_id); > + } > + else > + { > + if (key_version != ENCRYPTION_KEY_VERSION_INVALID) > + snprintf(url, buf_len, "%sdata/%u?version=%u", vault_url, key_id, > + key_version); > + else > + snprintf(url, buf_len, "%sdata/%u", vault_url, key_id); > + } > + if (curl_run(url, &response_str, false) != OPERATION_OK) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "Unable to get key data", 0); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + response = response_str.c_str(); > + response_len = response_str.size(); > + /* > + If the key is not found, this is not considered a fatal error, > + but we need to add an informational message to the log: > + */ > + if (response_len == 0) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Key not found", > + ME_ERROR_LOG_ONLY | ME_NOTE); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (json_get_object_key(response, response + response_len, "data", > + &js, &js_len) != JSV_OBJECT) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get data object (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > +#ifndef NDEBUG > + unsigned long version; > +#else > + unsigned long version= key_version; > + if (caching_enabled && > + key_version == ENCRYPTION_KEY_VERSION_INVALID) > +#endif > + { > + const char *ver; > + int ver_len; > + if (json_get_object_key(js, js + js_len, "metadata", > + &ver, &ver_len) != JSV_OBJECT) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get metadata object (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (json_get_object_key(ver, ver + ver_len, "version", > + &ver, &ver_len) != JSV_NUMBER) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get version number (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + /* > + Here we may use atoi, but it has undefined overflow behavior > + and does not work with unsigned integers, so use strtoul: > + */ > + errno = 0; > + version = strtoul(ver, NULL, 10); > + /* > + If the result of the conversion does not fit into an > + unsigned integer or if an error (for example, overflow) > + occurred during the conversion, then we need to return > + an error: > + */ > + if (version > UINT_MAX || (version == ULONG_MAX && errno)) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Integer conversion error (for version number)", > + 0); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + } > +#ifndef NDEBUG > + /* > + An internal check that is needed only for debugging the plugin > + operation - in order to ensure that we get from the Hashicorp Vault > + server exactly the version of the key that is needed: > + */ > + if (key_version != ENCRYPTION_KEY_VERSION_INVALID && > + key_version != version) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "Key version mismatch", 0); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > +#endif > + if (json_get_object_key(js, js + js_len, "data", > + &js, &js_len) != JSV_OBJECT) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get second-level data object " > + "(http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (json_get_object_key(js, js + js_len, "data", > + &key, &key_len) != JSV_STRING) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get data string (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + unsigned int max_length = *buflen; > + if (((unsigned int) key_len >> 1) > max_length) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Encryption key buffer too small", > + ME_ERROR_LOG_ONLY | ME_NOTE); > + *buflen = key_len; > + return ENCRYPTION_KEY_BUFFER_TOO_SMALL; > + } > + unsigned int length = 0; > + while (key_len >= 2) > + { > + int c1 = key[0]; > + int c2 = key[1]; > + if (! isxdigit(c1) || ! isxdigit(c2)) > + { > + break; > + } > + c1 -= '0'; > + c2 -= '0'; > + if (c1 > 9) > + { > + c1 -= 'A' - '0'; > + if (c1 > 15) > + { > + c1 -= 'a' - 'A'; > + } > + } > + if (c2 > 9) > + { > + c2 -= 'A' - '0'; > + if (c1 > 15) > + { > + c2 -= 'a' - 'A'; > + } > + } > + dstbuf[length++] = (c1 << 4) + c2; > + key += 2; > + key_len -= 2; > + } > + *buflen = length; > + if (key_len) > + { > + if (key_len != 1) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Syntax error - the key data should contain only " > + "hexadecimal digits", > + 0); > + } > + else > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Syntax error - extra character in the key data", > + 0); > + } > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (caching_enabled) > + { > + unsigned int int_ver= (unsigned int) version; > + KEY_INFO info; > + info.key_id = key_id; > + info.key_version = int_ver; > + info.length = length; > + memcpy(info.data, dstbuf, length); > + cache_add(key_id, int_ver, &info); > + } > + return 0; 1. you don't update the latest_version_cache here. 2. this function has lots of code essentially the same as in the get_latest_version. Please reuse, don't copy-paste. > +} > + > +struct st_mariadb_encryption hashicorp_key_management_plugin= { > + MariaDB_ENCRYPTION_INTERFACE_VERSION, > + get_latest_version, > + get_key_from_vault, > + 0, 0, 0, 0, 0 > +}; > + > +#ifdef _MSC_VER > + > +static int setenv(const char *name, const char *value, int overwrite) > +{ > + if (!overwrite) > + { > + size_t len= 0; > + int rc= getenv_s(&len, NULL, 0, name); > + if (rc) > + { > + return rc; > + } > + if (len) > + { > + errno = EINVAL; > + return EINVAL; > + } > + } > + return _putenv_s(name, value); > +} > + > +#endif > + > +static char *local_token= NULL; > + > +static int hashicorp_key_management_plugin_init(void *p) > +{ > + char *token_env= getenv("VAULT_TOKEN"); > + token_len = strlen(token); > + if (token_len == 0) { > + if (token_env) > + { > + token_len = strlen(token_env); > + if (token_len != 0) > + { > + token = (char *) malloc(token_len + 1); > + memcpy(token, token_env, token_len + 1); > + local_token = token; > + } > + } > + if (token_len == 0) { > + my_printf_error(ER_UNKNOWN_ERROR, > + "The --hashicorp-key-management-token option value " > + "or the value of the corresponding parameter in the " > + "configuration file must be specified, otherwise the " > + "VAULT_TOKEN environment variable must be set", > + 0); > + return 1; > + } > + } > + else > + { > + /* > + If the VAULT_TOKEN environment variable is not set or > + is not equal to the value of the token parameter, then > + we must set (overwrite) it for correct operation of > + the mariabackup: > + */ > + bool not_equal= token_env != NULL && strcmp(token_env, token) != 0; > + if (token_env == NULL || not_equal) > + { > + setenv("VAULT_TOKEN", token, 1); > + if (not_equal) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "The --hashicorp-key-management-token option value " > + "or the value of the corresponding parameter is not " > + "equal to the value of the VAULT_TOKEN environment " > + "variable", > + ME_ERROR_LOG_ONLY | ME_WARNING); > + } what mariabackup child processes need it? > + } > + } > + curl_global_init(CURL_GLOBAL_ALL); > + vault_url_len = strlen(vault_url); > + return 0; > +} > + > +static int hashicorp_key_management_plugin_deinit(void *p) > +{ > + latest_version_cache.clear(); > + key_info_cache.clear(); > + curl_global_cleanup(); > + if (local_token) > + { > + free(local_token); > + } > + return 0; > +} > + > +/* > + Plugin library descriptor > +*/ > +maria_declare_plugin(hashicorp_key_management) > +{ > + MariaDB_ENCRYPTION_PLUGIN, > + &hashicorp_key_management_plugin, > + "hashicorp_key_management", > + "MariaDB Corporation", > + "HashiCorp Vault key management plugin", > + PLUGIN_LICENSE_GPL, > + hashicorp_key_management_plugin_init, > + hashicorp_key_management_plugin_deinit, > + 0x0102 /* 1.02 */, > + NULL, /* status variables */ > + settings, > + "1.02", > + MariaDB_PLUGIN_MATURITY_ALPHA > +} > +maria_declare_plugin_end; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 29 Jun '20

29 Jun '20

Hi, This week - I really struggled with the ABI check failing with the error: CMake Error at cmake/do_abi_check.cmake:82 (MESSAGE): ABI check found difference between /workspace/server/include/mysql/plugin_audit.h.pp and /workspace/server/abi_check.out Running make abi_update fixed the issue. - Using services means copying a large portion of the headers into the server. This may cause licensing issues. Overview: | Library | License | | AZlib | Zlib | | BZip2 | BSD-like | | LZ4 | Library files under BSD 2-Clause and other under GPL v2 | | LZMA | Public Domain | | LZO | GPL v2+ | | Snappy | Custom license | | Xpress | ??? | | Zlib | Zlib | | ZStandard | Dual: BSD + GPL v2 | - Tested the LZMA service by calling the function from a dynamic plugin as per step 2 of MDEV-22895. - Started working on making the service compatible with static plugins as per step 3 of MDEV-22895. Next week - Finish making the service compatible with static plugins. - Work on MDEV-22895. Thanks, Kartik

1 0

[Maria-developers] Progress report for MDEV-12326
by Mohammed Hammaad Mateen 28 Jun '20

28 Jun '20

Greetings, Hope you are safe and doing great, This post describes the things I've done during 22-28 June for Coding phase-1 under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20 to implement MDEV-12326. The tasks taken up for this week was to work towards validating the fields of the return list, requested by the client when it uses *multi-table DELETE....RETURNING *or* multi-table UPDATE....RETURNING*. If an invalid column name is requested in the return list then it should result in displaying: *ERROR 1054 (42S22): Unknown column '<invalid column>' in 'field list' *. Things I learnt while implementing this was, firstly that *setup_fields* and *setup_wild* functions validate the *returning_item_list* i.e. column names, expressions, etc. in *RETURNING* statement and understood that validation is done implicitly. Moreover, *setup_returning_fields* method could also be used interchangeably. Secondly, if we have to completely run a .test file that has many failing test cases then we should provide --force option twice, as in*:* *./mtr <file_name> --force --force. * I've also written and recorded the result for the above usecase, for *multi-table DELETE.... RETURNING *and will also do the same for *multi-table UPDATE.... RETURNING v*ery soon. All in all coding phase-1 was really informative and I would like to thank *MariaDB Foundation* for having me. All the code is uploaded up to date in these remote repositories, please feel free to comment. MULTI-TABLE UPDATE RETURNING: https://github.com/MariaDB/server/compare/10.5...MohammedHMateen:MDEV-12326… MULTI-TABLE DELETE RETURNING: https://github.com/MariaDB/server/compare/10.5...MohammedHMateen:MDEV-12326… Thank you Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] 97f7d4a9b4d: MDEV-22726: Add check that one can't change general or slow log to a
by Sergei Golubchik 26 Jun '20

26 Jun '20

Hi, Vlad! On Jun 26, Vlad Lesin wrote: > >> > >> + if (create_info->db_type == maria_hton && > >> + create_info->transactional != HA_CHOICE_NO) > >> + { > >> + my_error(ER_TRANSACTIONAL_ARIA_LOG_ENGINE, MYF(0)); > >> + DBUG_RETURN(true); > >> + } > > > > What about InnoDB? Or RocksDB? Or any other transactional engine? > > There is flag HTON_SUPPORT_LOG_TABLES, which is checked earlier in > mysql_alter_table(). It's not checked on CREATE, only on ALTER, so one can easily bypass this check. And, anyway, why an engine would not support log tables? Only because it's transactional? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 97f7d4a9b4d: MDEV-22726: Add check that one can't change general or slow log to a
by Sergei Golubchik 25 Jun '20

25 Jun '20

Hi, Vlad! On Jun 25, Vlad Lesin wrote: > revision-id: 97f7d4a9b4d (mariadb-10.2.31-290-g97f7d4a9b4d) > parent(s): e3104c4a8c6 > author: Vlad Lesin <vlad_lesin(a)mail.ru> > committer: Vlad Lesin <vlad_lesin(a)mail.ru> > timestamp: 2020-06-24 14:37:54 +0300 > message: > > MDEV-22726: Add check that one can't change general or slow log to a > transactional engine the commit subject should be in one line (and I personally only put one space between the MDEV number and the text, not a colon, because the subject length is quite limited to waste it on pretty delimiters that don't add any meaningful information) > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index ff495f69a4f..8db992c7501 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7731,3 +7731,5 @@ ER_GEOJSON_EMPTY_COORDINATES > > ER_MYROCKS_CANT_NOPAD_COLLATION > eng "MyRocks doesn't currently support collations with \"No pad\" attribute." > +ER_TRANSACTIONAL_ARIA_LOG_ENGINE > + eng "Only non-transactional Aria table can be used for logging" No, sorry, you cannot add new error messages in 10.2. Or 10.3 or 10.4. Because adding a new error message to anything below 10.5 would mean that 10.5 error numbers will shift and we aren't allowed to do it for compatibility reasons. The typical workaround is to use my_print_error(ER_something_else, "Only non-transactional Aria table can be used for logging"); (with the appropriate error number, MYF(0), etc) > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 7cce3bcc323..b3a600eec36 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -8925,6 +8925,13 @@ bool mysql_alter_table(THD *thd,char *new_db, char *new_name, > DBUG_RETURN(true); > } > > + if (create_info->db_type == maria_hton && > + create_info->transactional != HA_CHOICE_NO) > + { > + my_error(ER_TRANSACTIONAL_ARIA_LOG_ENGINE, MYF(0)); > + DBUG_RETURN(true); > + } What about InnoDB? Or RocksDB? Or any other transactional engine? > #ifdef WITH_PARTITION_STORAGE_ENGINE > if (alter_info->flags & Alter_info::ALTER_PARTITION) > { > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9bdf35e90f3: MDEV-18215: mariabackup does not report unknown command line options
by Sergei Golubchik 25 Jun '20

25 Jun '20

Hi, Vlad! Some comments about your command-line option handling code On Jun 25, Vlad Lesin wrote: > revision-id: 9bdf35e90f3 (mariadb-10.4.11-242-g9bdf35e90f3) > parent(s): ceaa8b647a1 > author: Vlad Lesin <vlad_lesin(a)mail.ru> > committer: Vlad Lesin <vlad_lesin(a)mail.ru> > timestamp: 2020-06-14 13:23:07 +0300 > message: > > MDEV-18215: mariabackup does not report unknown command line options > MDEV-21298: mariabackup doesn't read from the [mariadbd] and [mariadbd-X.Y] > server option groups from configuration files > MDEV-21301: mariabackup doesn't read [mariadb-backup] option group in > configuration file > > All three issues require to change the same code, that is why their > fixes are joined in one commit. > > The fix is in invoking load_defaults_or_exit() and handle_options() for > backup-specific groups separately from client-server groups to let the last > handle_options() call fail on unknown backup-specific options. > > The order of options procesing is the following: > 1) Load server groups and process server options, ignore unknown > options > 2) Load client groups and process client options, ignore unknown > options > 3) Load backup groups and process client-server options, exit on > unknown option > 4) Process --mysqld-args command line options, ignore unknown options > > New global flag my_handle_options_init_variables was added to have > ability to invoke handle_options() for the same allowed options set > several times without re-initialising previously set option values. > > --password value destroying is moved from option processing callback to > mariabackup's handle_options() function to have ability to invoke server's > handle_options() several times for the same possible allowed options > set. > > Galera invokes wsrep_sst_mariabackup.sh with mysqld command line > options to configure mariabackup as close to the server as possible. > It is not known what server options are supported by mariabackup when the > script is invoked. That is why new mariabackup option "--mysqld-args" is added, > all unknown options that follow this option will be silently ignored. > > wsrep_sst_mariabackup.sh was also changed to: > - use "--mysqld-args" mariabackup option to pass mysqld options, > - remove deprecated innobackupex mode, > - remove unsupported mariabackup options: > --encrypt > --encrypt-key > --rebuild-indexes > --rebuild-threads > > diff --git a/extra/mariabackup/xtrabackup.cc b/extra/mariabackup/xtrabackup.cc > index 852ef4efe56..19058398258 100644 > --- a/extra/mariabackup/xtrabackup.cc > +++ b/extra/mariabackup/xtrabackup.cc > @@ -5990,11 +5977,23 @@ void setup_error_messages() > die("could not initialize error messages"); > } > > -void > -handle_options(int argc, char **argv, char ***argv_client, char ***argv_server) > +/** Handle mariabackup options. The options are handled with the following > +order: > + > +1) Load server groups and process server options, ignore unknown options > +2) Load client groups and process client options, ignore unknown options > +3) Load backup groups and process client-server options, exit on unknown option > +4) Process --mysqld-args options, ignore unknown options > + > +@param[in] argc arguments count > +@param[in] argv arguments array > +@param[out] argv_server server options including loaded from server groups > +@param[out] argv_client client options including loaded from client groups > +@param[out] argv_backup backup options including loaded from backup groups */ > +void handle_options(int argc, char **argv, char ***argv_server, > + char ***argv_client, char ***argv_backup) could you please rename this function? handle_options() is part of my_getopt api, the main function to parse the argv array of options. this one has a different prototype, so C++ can distinguish between them, but it's very confusing, like having your own strcmp(int, char*, void*). > { > /* Setup some variables for Innodb.*/ > srv_operation = SRV_OPERATION_RESTORE; > > files_charset_info = &my_charset_utf8_general_ci; > @@ -6021,49 +6020,64 @@ handle_options(int argc, char **argv, char ***argv_client, char ***argv_server) > bool prepare = false; > > char conf_file[FN_REFLEN]; > - int argc_client = argc; > - int argc_server = argc; > - > - /* scan options for group and config file to load defaults from */ > - for (i = 1; i < argc; i++) { > - > - char *optend = strcend(argv[i], '='); > - > - if (strncmp(argv[i], "--defaults-group", > - optend - argv[i]) == 0) { > - defaults_group = optend + 1; > - append_defaults_group(defaults_group, > - xb_server_default_groups, > - array_elements(xb_server_default_groups)); > - } > > - if (strncmp(argv[i], "--login-path", > - optend - argv[i]) == 0) { > - append_defaults_group(optend + 1, > - xb_client_default_groups, > - array_elements(xb_client_default_groups)); > - } > + // array_elements() will not work for load_defaults, as it is defined > + // as external symbol, so let's use dynamic array to have ability to > + // add new server default groups > + std::vector<const char *> server_default_groups; > > - if (!strncmp(argv[i], "--prepare", > - optend - argv[i])) { > - prepare = true; > - } > + for (const char **default_group= load_default_groups; *default_group; > + ++default_group) > + server_default_groups.push_back(*default_group); > > - if (!strncmp(argv[i], "--apply-log", > - optend - argv[i])) { > - prepare = true; > - } > + std::vector<char *> mysqld_args; > + std::vector<char *> mariabackup_args; > + mysqld_args.push_back(argv[0]); > + mariabackup_args.push_back(argv[0]); > > - if (!strncmp(argv[i], "--target-dir", > - optend - argv[i]) && *optend) { > - target_dir = optend + 1; > - } > + /* scan options for group and config file to load defaults from */ > + for (i= 1; i < argc; i++) > + { > + char *optend= strcend(argv[i], '='); > + if (mysqld_args.size() > 1 || > + strncmp(argv[i], "--mysqld-args", optend - argv[i]) == 0) > + { > + mysqld_args.push_back(argv[i]); > + continue; > + } > + else > + mariabackup_args.push_back(argv[i]); > > - if (!*optend && argv[i][0] != '-') { > - target_dir = argv[i]; > - } > - } > + if (strncmp(argv[i], "--defaults-group", optend - argv[i]) == 0) > + { > + defaults_group= optend + 1; > + server_default_groups.push_back(defaults_group); > + } > + else if (strncmp(argv[i], "--login-path", optend - argv[i]) == 0) > + { > + append_defaults_group(optend + 1, xb_client_default_groups, > + array_elements(xb_client_default_groups)); > + } > + else if (!strncmp(argv[i], "--prepare", optend - argv[i])) > + { > + prepare= true; > + } > + else if (!strncmp(argv[i], "--apply-log", optend - argv[i])) > + { > + prepare= true; > + } > + else if (!strncmp(argv[i], "--target-dir", optend - argv[i]) && > + *optend) > + { > + target_dir= optend + 1; > + } I don't understand. Why is this manual parsing of options if you use my_getopt anyway? > + else if (!*optend && argv[i][0] != '-') > + { > + target_dir= argv[i]; > + } > + } > > + server_default_groups.push_back(NULL); > snprintf(conf_file, sizeof(conf_file), "my"); > > if (prepare && target_dir) { > @@ -6102,7 +6122,6 @@ handle_options(int argc, char **argv, char ***argv_client, char ***argv_server) > optp->u_max_value = (G_PTR *) &global_max_value; what is that??? > } > > - > /* Throw a descriptive error if --defaults-file or --defaults-extra-file > is not the first command line argument */ and what is that??? You know, if there are ~10 programs using load_defaults/my_getopt and not a single one of them is doing this, perhaps mariabackup shouldn't be doing it either? > for (int i = 2 ; i < argc ; i++) { > @@ -6143,18 +6163,76 @@ handle_options(int argc, char **argv, char ***argv_client, char ***argv_server) > xb_client_options, xb_get_one_option))) > exit(ho_error); > > + /* 3) Load backup groups and process client-server options, exit on > + unknown option */ > + > + load_defaults_or_exit(conf_file, backup_default_groups, &argc_backup, > + argv_backup); > + for (n= 0; (*argv_backup)[n]; n++) > + { > + }; > + argc_backup= n; > + > + my_handle_options_init_variables = FALSE; > + > + if (argc_backup > 0 && > + (ho_error= handle_options(&argc_backup, argv_backup, > + xb_server_options, xb_get_one_option))) > + exit(ho_error); I think this whole approach is wrong. And my_handle_options_init_variables highlights it. There should be only one invocation of handle_options(), otherwise you won't get the correct behavior of what options override what. You can call handle_options() separately for server and client options, because those are disjoint sets of options. But you cannot call it twice for xb_server_options. Now you call handle_options 5 times. I suggest to rewrite the code to do it only twice. As: read *all* groups with load_defaults_or_exit(), and read *all* options with one handle_options() call. This will be the actual parsing and setting values of variables, ignoring unknown options. But you still want to issue an error for unknown options, so truncate the list of command-line options before --mysqld_args, read only mariabackup groups, call handle_options again, erroring out on unknown options. this should not set any values, it's just for an error, so before that you reset all pointers to variables - just like you do now for --maximum. Because this approach parses all options at once, you'll get the same behavior regarding what options override what, as any other tool that uses my_getopt. > + > + /* Add back the program name handle_options removes */ > + ++argc_backup; > + --(*argv_backup); > + > + if (innobackupex_mode && argc_backup > 0 && > + !ibx_handle_options(&argc_backup, argv_backup)) > + exit(EXIT_FAILURE); > + > + my_getopt_skip_unknown = FALSE; > + > + if (argc_backup > 0 && > + (ho_error= handle_options(&argc_backup, argv_backup, > + xb_client_options, xb_get_one_option))) > + exit(ho_error); > + > + if (opt_password) > + { > + char *argument= opt_password; > + char *start= argument; > + opt_password= my_strdup(opt_password, MYF(MY_FAE)); > + while (*argument) > + *argument++= 'x'; // Destroy argument > + if (*start) > + start[1]= 0; > + } > + > + /* 4) Process --mysqld-args options, ignore unknown options */ > + > + my_getopt_skip_unknown = TRUE; > + > + int argc_mysqld = static_cast<int>(mysqld_args.size()); > + if (argc_mysqld > 1) > + { > + char **argv_mysqld= &mysqld_args[0]; > + if ((ho_error= handle_options(&argc_mysqld, &argv_mysqld, > + xb_server_options, xb_get_one_option))) > + exit(ho_error); > + } > + > + my_handle_options_init_variables = TRUE; > + > /* Reject command line arguments that don't look like options, i.e. are > not of the form '-X' (single-character options) or '--option' (long > options) */ > - for (int i = 0 ; i < argc_client ; i++) { > - const char * const opt = (*argv_client)[i]; > + for (int i = 0 ; i < argc_backup ; i++) { > + const char * const opt = (*argv_backup)[i]; > > if (strncmp(opt, "--", 2) && > !(strlen(opt) == 2 && opt[0] == '-')) { > bool server_option = true; > > - for (int j = 0; j < argc_server; j++) { > - if (opt == (*argv_server)[j]) { > + for (int j = 0; j < argc_backup; j++) { > + if (opt == (*argv_backup)[j]) { > server_option = false; > break; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] 258a0fb47b7: MDEV-22951: rpl.rpl_slave_alias_replica failed in buildbot with wrong result
by sujatha 24 Jun '20

24 Jun '20

revision-id: 258a0fb47b74363d0919d6a18b9b2e67d45d4cf7 (mariadb-10.5.2-490-g258a0fb47b7) parent(s): d1bb7f9143ceafe9cbdc1e53c4ffd4abe232cb97 author: Sujatha committer: Sujatha timestamp: 2020-06-24 15:33:45 +0530 message: MDEV-22951: rpl.rpl_slave_alias_replica failed in buildbot with wrong result Problem: ======== rpl.rpl_slave_alias_replica test fails sporadically on build bot as shown below. rpl.rpl_slave_alias_replica 'innodb,stmt' w4 [ fail ] +++ /usr/local/mariadb-10.5.4-linux-x86_64/mysql-test/suite/rpl/r/ rpl_slave_alias_replica.reject 2020-06-12 09:52:25.191267945 +0000 @@ -22,6 +22,7 @@ SHOW REPLICA HOSTS; Server_id Host Port Master_id 2 127.0.0.1 SLAVE_PORT 1 +2 127.0.0.1 SLAVE_PORT 1 Analysis: ======== The SHOW REPLICA HOSTS command reports an additional row for the same server occasionally. This happens because of zombie dump threads on the master. Error log from master also confirms that "A slave with the same server_uuid/server_id as this slave has connected to the master". Since SHOW REPLICA HOSTS goes through the list of dump threads and reports the corresponding hosts,the same host is reported twice. Fix: === Instead of capturing the SHOW REPLICA HOSTS command in result file use wait_show_condition.inc. This will ensure that SHOW REPLICA HOSTS command fetches slave host details. --- mysql-test/suite/rpl/r/rpl_slave_alias_replica.result | 5 ++--- mysql-test/suite/rpl/t/rpl_slave_alias_replica.test | 10 ++++++++-- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_slave_alias_replica.result b/mysql-test/suite/rpl/r/rpl_slave_alias_replica.result index 8262cf8212a..2a49e80075f 100644 --- a/mysql-test/suite/rpl/r/rpl_slave_alias_replica.result +++ b/mysql-test/suite/rpl/r/rpl_slave_alias_replica.result @@ -18,10 +18,9 @@ START REPLICA; include/wait_for_slave_io_to_start.inc include/wait_for_slave_sql_to_start.inc connection master; +connection slave; +connection master; "Command: SHOW SLAVE HOSTS --> SHOW REPLICA HOSTS" -SHOW REPLICA HOSTS; -Server_id Host Port Master_id -2 127.0.0.1 SLAVE_PORT 1 DROP USER 'repl_user'; connection slave; "Command: SHOW SLAVE IO/SQL THREAD --> SHOW REPLICA IO/SQL THREAD" diff --git a/mysql-test/suite/rpl/t/rpl_slave_alias_replica.test b/mysql-test/suite/rpl/t/rpl_slave_alias_replica.test index 8f2cd1a4eb3..324821a325f 100644 --- a/mysql-test/suite/rpl/t/rpl_slave_alias_replica.test +++ b/mysql-test/suite/rpl/t/rpl_slave_alias_replica.test @@ -45,10 +45,16 @@ START REPLICA; --source include/wait_for_slave_io_to_start.inc --source include/wait_for_slave_sql_to_start.inc +--connection master +--sync_slave_with_master + --connection master --echo "Command: SHOW SLAVE HOSTS --> SHOW REPLICA HOSTS" ---replace_result $SLAVE_MYPORT SLAVE_PORT -SHOW REPLICA HOSTS; +let $show_statement= SHOW REPLICA HOSTS; +let $field= Server_id; +# Slave's server_id 2 +let $condition= ='2'; +source include/wait_show_condition.inc; DROP USER 'repl_user'; --sync_slave_with_master

1 0

[Maria-developers] Progress Report GSoC 2020 (15-20 June)
by Anton Mikhaylenko 24 Jun '20

24 Jun '20

Good day, everybody. In this post, I want to describe what I did this week, June 15-20. I`m working with threadpool concurrency. Have made some fixes. Change type of adjustment interval. Used the existing time interval to adjust the throttling. Regards, Anton MIkhailenko

1 0

[Maria-developers] Review for: MDEV-17399 Add support for JSON_TABLE, part #6
by Sergey Petrunia 22 Jun '20

22 Jun '20

Hi Alexey, > revision-id: 0d342ab173cdbf5d35c5f2833c66b6004b9e908e (mariadb-10.5.2-330-g0d342ab173c) > parent(s): b1ab211dee599eabd9a5b886fafa3adea29ae041 > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2020-06-08 14:51:04 +0400 > message: > > MDEV-17399 Add support for JSON_TABLE. > > ha_json_table handler implemented. > JSON_TABLE() added to SQL syntax. > First, some general input: == Assertion with Prepared Statements == prepare s from 'select * from json_table(?, \'$[*]\' columns( color varchar(100) path \'$.color\', price text path \'$.price\', seq for ordinality ) ) as T order by color desc; ' mysqld: /home/psergey/dev-git2/10.5-hf-review/sql/item.h:1002: virtual bool Item::fix_fields(THD*, Item**): Assertion `basic_const_item()' failed. == Default value for TEXT == Default value for VARCHAR(100) is SQL NULL, while for TEXT it is empty string. Is this intentional? If yes, I would like to see the spec of the logic behind this. select * from json_table('[{"color": "blue", "price": 50}, {"color": "red", "price": 100}]', '$[*]' columns(color varchar(100) path '$.nonexistent', seq for ordinality ) ) as T; +-------+------+ | color | seq | +-------+------+ | NULL | 1 | | NULL | 2 | +-------+------+ select * from json_table('[{"color": "blue", "price": 50}, {"color": "red", "price": 100}]', '$[*]' columns(color text path '$.nonexistent', seq for ordinality ) ) as T; +-------+------+ | color | seq | +-------+------+ | | 1 | | | 2 | +-------+------+ == JSON_TABLE can have a non-unique alias? == create table t20(a varchar(100)); insert into t20 values (); create table t21(a varchar(100)); insert into t21 values (); select * from t20 as T, t21 as T; ERROR 1066 (42000): Not unique table/alias: 'T' Good. This is the expected behaviour. select * from t20 as T , json_table(T.a, '$[*]' columns(color varchar(100) path '$.nonexistent', seq for ordinality ) ) as T; Empty set (0.000 sec) No error. I think this is wrong. Why would JSON_TABLE be allowed to use an alias that conflicts with another table? == EXPLAIN doesn't show table function == I had requested this in previous review(s): Table function use should be shown in EXPLAIN [FORMAT=JSON]. Here's how MySQL does it: https://gist.github.com/spetrunia/257c929b0c04a866faf6428e733999ad Note that this can be extended to handle other table functions. if there are no other ideas, let's use the same syntax. == Comments == I still think the patch doesn't have good comments. Let me get back to you on this. > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 49cabec9916..3db438ee470 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7965,3 +7965,7 @@ ER_NOT_ALLOWED_IN_THIS_CONTEXT > eng "'%-.128s' is not allowed in this context" > ER_DATA_WAS_COMMITED_UNDER_ROLLBACK > eng "Engine %s does not support rollback. Changes where commited during rollback call" > +ER_JSON_TABLE_ERROR_ON_FIELD > + eng "Field '%s' can't be set for JSON_TABLE '%s'." > +ER_JSON_TABLE_DOUBLE_ON_RESPONSE > + eng "ON %s response specified more than once in JSON_TABLE definition." I agree that using this error solves the issue I've complained about, but this doesn't look like the right way to do it: * Why not just emit a parser error with YYERROR in this case? * Better: why not just remove the recursive definition from the grammar? ... > diff --git a/sql/table_function.cc b/sql/table_function.cc ... > + > +/* > + handler::print_error has a case statement for error numbers. > + This value is (10000) is far out of range and will envoke the > + default: case.(Current error range is 120-159 from include/my_base.h) > + */ > +#define HA_ERR_INVALID_JSON 10000 So, is this needed anymore? On the other hand, the new solution with returning HA_ERR_TABLE_IN_FK_CHECK doesn't look very good either. A plugin could register its own error codes, but this engine is not a plugin. Let's discuss with Serg and get back to this. > +static int print_path(String *str, const json_path_t *p) > +{ > + return str->append('\"') || > + str->append((const char *) p->s.c_str, p->s.str_end - p->s.c_str) || > + str->append('\"'); > +} The above doesn't do proper quoting, which is wrong. Testcase: select * from json_table('[{"co\\\\lor": "blue", "price": 50}]', '$[*]' columns( color varchar(100) path '$.co\\\\lor') ) as T; +-------+ | color | +-------+ | blue | +-------+ create view v2 as select * from json_table('[{"co\\\\lor": "blue", "price": 50}]', '$[*]' columns( color varchar(100) path '$.co\\\\lor') ) as T; select * from v2; +-------+ | color | +-------+ | NULL | +-------+ BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Progress report for MDEV-12326
by Mohammed Hammaad Mateen 22 Jun '20

22 Jun '20

Greetings, Hope you are safe and doing great, This post describes the things I've done during 15-21 June under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20 to implement MDEV-12326. The tasks taken up for this week was to work towards extending the parser for multi-table DELETE....RETURNING and multi-table UPDATE....RETURNING, side-by-side, thereby changing the sql_yacc.yy file, for incorporating opt_returning rule in respective query format. I've also updated the test suite for multi-table UPDATE.... RETURNING and extended the parser by adding select_result parameter in multi-update and multi-delete functions, so as to enable returning a result set possible, whenever we query with RETURNING clause. All the code is uploaded up to date in these remote repositories, please feel free to comment. MULTI-TABLE UPDATE RETURNING: https://github.com/MariaDB/server/compare/10.5...MohammedHMateen:MDEV-12326… MULTI-TABLE DELETE RETURNING: https://github.com/MariaDB/server/compare/10.5...MohammedHMateen:MDEV-12326… Thank you Regards, Mohammed Hammaad Mateen

1 0

[Maria-developers] 97e932919bd: MDEV-22665: Print ranges in the optimizer trace created for non-indexed columns when optimizer_use_condition_selectivity >2
by Sergey Petrunia 17 Jun '20

17 Jun '20

Hi Varun, Please find some minor comments below. Ok to push when addressed. > revision-id: 97e932919bd53b854e0cea0d0001d185d7e63cc0 (mariadb-10.4.11-199-g97e932919bd) > parent(s): 0a5668f5128c731a346abf41afdc6fed33164ffc > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-05-26 22:36:03 +0530 > message: > > MDEV-22665: Print ranges in the optimizer trace created for non-indexed columns when optimizer_use_condition_selectivity >2 > > Now the optimizer trace shows the ranges constructed while getting estimates from EITS > > --- > mysql-test/main/opt_trace.result | 94 ++++++++++++++++++++++++++++++ > mysql-test/main/opt_trace.test | 18 ++++++ > sql/field.cc | 40 +++++++++++++ > sql/field.h | 1 + > sql/opt_range.cc | 123 ++++++++++++++++++++++++++++----------- > 5 files changed, 241 insertions(+), 35 deletions(-) ... > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index 66c870dd2ac..2cef483ef28 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -15804,6 +15821,38 @@ void QUICK_GROUP_MIN_MAX_SELECT::dbug_dump(int indent, bool verbose) > > #endif /* !DBUG_OFF */ > > + > +/* > + @brief Print the comparison operator for the min range > +*/ > + > +static void print_min_range_operator(String *out, const key_range key) Do you really need to pass the whole struct key_range by value here? I think passing key.flag would be sufficient. > +{ > + if (key.flag == HA_READ_AFTER_KEY) > + out->append(STRING_WITH_LEN(" < ")); > + else if (key.flag == HA_READ_KEY_EXACT || > + key.flag == HA_READ_KEY_OR_NEXT) > + out->append(STRING_WITH_LEN(" <= ")); > + else > + out->append(STRING_WITH_LEN(" ? ")); > +} > + > + > +/* > + @brief Print the comparison operator for the max range > +*/ > + > +static void print_max_range_operator(String *out, const key_range key) > +{ Same as with print_min_range_operator: pass key.flag instead. > + if (key.flag == HA_READ_BEFORE_KEY) > + out->append(STRING_WITH_LEN(" < ")); > + else if (key.flag == HA_READ_AFTER_KEY) > + out->append(STRING_WITH_LEN(" <= ")); > + else > + out->append(STRING_WITH_LEN(" ? ")); > +} > + > + > static > void print_range(String *out, const KEY_PART_INFO *key_part, > KEY_MULTI_RANGE *range, uint n_key_parts) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Progress Report GSoC 2020 (8-13 June)
by Anton Mikhaylenko 17 Jun '20

17 Jun '20

Good day, everybody. In this post, I want to describe what I did this week, June 8-13. Added counting № of completed tasks for during period. During this week I tried to understand why threads dying fast. https://github.com/sapiest/server/commit/7ffa698044092e346353e4754d50188ef8… Regards, Anton MIkhailenko

1 0

Re: [Maria-developers] 3c81ed2f564: Add mytop to Client component for rpm package
by Sergei Golubchik 16 Jun '20

16 Jun '20

Hi, Anel! On Jun 16, Anel Husakovic wrote: > revision-id: 3c81ed2f564 (mariadb-10.5.2-359-g3c81ed2f564) > parent(s): a793ae5bc12 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-06-06 08:21:01 +0200 > message: > > Add mytop to Client component for rpm package > > Part of MDEV-22552 This is ok. But don't push it yet, let's have all commits ok to push them all at once Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 4cbbc796dcc: MDEV-22420 DDL on temporary object is prohibited when XA is in prepare state
by Sergei Golubchik 16 Jun '20

16 Jun '20

Hi, Andrei! Ok to push! Thanks! On Jun 16, Andrei Elkin wrote: > >> > What about RENAME? What other operations are there that don't > >> > open a table? > >> > >> So RENAME is in above and all that fits to > >> stmt_causes_implicit_commit() that is flagged CF_AUTO_COMMIT_TRANS > >> (that involves CF_IMPLICIT_COMMIT_BEGIN). That reads every DDL, > >> Admin, Account that are bin-logged. > > > > I mean RENAME for temporary tables. Does it also cause an implicit > > commit? > > Yes, it does, as any syntax whose lex > lex->tmp_table() > returns false here. > > RENAME specifically does not have 'temporary' qualifier. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 4cbbc796dcc: MDEV-22420 DDL on temporary object is prohibited when XA is in prepare state
by Sergei Golubchik 16 Jun '20

16 Jun '20

Hi, Andrei! On Jun 16, Andrei Elkin wrote: > > > What about RENAME? What other operations are there that don't open a table? > > So RENAME is in above and all that fits to stmt_causes_implicit_commit() > that is flagged CF_AUTO_COMMIT_TRANS (that involves > CF_IMPLICIT_COMMIT_BEGIN). > That reads every DDL, Admin, Account that are bin-logged. I mean RENAME for temporary tables. Does it also cause an implicit commit? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MariaDB support in WAL-G
by Кирилл Юлдашев 16 Jun '20

16 Jun '20

Hi! I implemented MariaDB support in the WAL-G backup system (Here is README https://github.com/wal-g/wal-g/blob/master/MySQL.md)) the implementation turned out to be compatible with MySQL. Can I ask someone from MariaDB to do a small design review and send it to me? I am defending my diploma on this topic on June 17 this year, and the developers ' opinion would be very useful. Integration tests: https://github.com/wal-g/wal-g/tree/master/docker/mariadb_tests/scripts/tes… Internal database stuff and command handlers: https://github.com/wal-g/wal-g/tree/master/internal/databases/mysql Commands: https://github.com/wal-g/wal-g/tree/master/cmd/mysql But PITR for MariaDB is not working yet, because of problems which occurred during the implementation stage due to the lack of documentation. I'll write about my problems later in another email. Thanks! Regards, Kirill

1 0

Re: [Maria-developers] 4cbbc796dcc: MDEV-22420 DDL on temporary object is prohibited when XA is in prepare state
by Sergei Golubchik 15 Jun '20

15 Jun '20

Hi, Andrei! On Jun 15, Andrei Elkin wrote: > revision-id: 4cbbc796dcc (mariadb-10.5.2-335-g4cbbc796dcc) > parent(s): a74e724e061 > author: Andrei Elkin <andrei.elkin(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2020-06-11 17:24:15 +0300 > message: > > MDEV-22420 DDL on temporary object is prohibited when XA is in prepare state > > The parser must reject DDL operations on temporary objects when > they may modify or alter such object, including temporary tables and sequences. > The rejection is regardless (has been already in place for bin-loggable DML:s) > of the binlogging capability of the server or connection. > > The patch implements the requirement. A binlog test is added. > > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 86f81dd001a..6ac0cf96024 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -4890,6 +4890,11 @@ mysql_execute_command(THD *thd) > } > else > { > + if (thd->transaction->xid_state.check_has_uncommitted_xa()) > + { > + thd->transaction->xid_state.er_xaer_rmfail(); > + goto error; > + } > status_var_decrement(thd->status_var.com_stat[lex->sql_command]); > status_var_increment(thd->status_var.com_drop_tmp_table); Two comments. 1. Where is this checked for non-temporary tables? check_has_uncommitted_xa is tested in open_tables(), and it's not called for DROP. What about RENAME? What other operations are there that don't open a table? 2. check_has_uncommitted_xa() already does er_xaer_rmfail() internally, you don't need to repeat it. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Progress report for MDEV-12326
by Mohammed Hammaad Mateen 15 Jun '20

15 Jun '20

Greetings, Hope you are safe and doing great, This post describes the things I've done during 8-14 June under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20 to implement MDEV-12326. The tasks taken up for this week was to work towards extending the parser for multi-table DELETE....RETURNING by changing the sql_yacc.yy file, for incorporating opt_returning rule in multi-delete query format. The RETURNING_SYM in opt_returning rule incorporates the RETURNING keyword and works completely fine with <column_names> but for some reason it fails and crashes the server for RETURNING * . I'll try to run it in a debugger and will try to fix it soon. Meanwhile, I'll also be pushing the code on this branch (https://github.com/MariaDB/server/pull/1590/commits) Thank you Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] c48b190824a: Fixed error messages from DROP VIEW to align with DROP TABLE
by Sergei Golubchik 14 Jun '20

14 Jun '20

Hi, Michael! On Jun 13, Michael Widenius wrote: > revision-id: c48b190824a (mariadb-10.5.2-405-gc48b190824a) > parent(s): 50bad912977 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-06-12 12:04:05 +0300 > message: > > Fixed error messages from DROP VIEW to align with DROP TABLE > > - Produce a "Note" for all wrongly dropped objects > (Like doing DROP VIEW on a table). > - IF EXISTS ends with a list of all not existing objects, instead of a > separate note for every not existing object. > > Other things: > - Fixed bug where one could do CREATE TEMPORARY SEQUENCE multiple times > and create multiple temporary sequences with the same name. > > diff --git a/mysql-test/main/create_drop_view.result b/mysql-test/main/create_drop_view.result > index 92fbf5ac9e5..7cfa818ccbf 100644 > --- a/mysql-test/main/create_drop_view.result > +++ b/mysql-test/main/create_drop_view.result > @@ -61,5 +61,6 @@ Warnings: > Note 4092 Unknown VIEW: 'test.v1' > DROP VIEW IF EXISTS t1; > Warnings: > +Warning 1347 'test.t1' is not of type 'VIEW' please, add a test for the case when a user has no privileges on test.t1, the object existence is not leaked out (if one has no privileges to know whether t1 exists or whether t1 is a view or a table, the error message should not divulge it). (the same for tables and for sequences in your next commit) > Note 4092 Unknown VIEW: 'test.t1' > DROP TABLE t1; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 250bc3b6d74: Ensure that table is truly dropped when using DROP TABLE
by Sergei Golubchik 14 Jun '20

14 Jun '20

Hi, Michael! On Jun 11, Michael Widenius wrote: > revision-id: 250bc3b6d74 (mariadb-10.5.2-390-g250bc3b6d74) > parent(s): ba2c2cfb20e > author: Michael Widenius <michael.widenus(a)gmail.com> > committer: Michael Widenius <michael.widenus(a)gmail.com> > timestamp: 2020-06-11 11:33:55 +0300 > message: > > Ensure that table is truly dropped when using DROP TABLE > > MDEV-11412 AliSQL: [Feature] Issue#34 Support force drop table please, put this line ^^^ first in the comment comment. > The used code is largely based on code from Tencent > > The problem is that in some rare cases there may be a conflict between .frm > files and the files in the storage engine. In this case the DROP TABLE > was not able to properly drop the table. > > Some MariaDB/MySQL forks has solved this by adding a FORCE option to TSQL? Or some other fork? > DROP TABLE. After some discussion among MariaDB developers, we concluded > that users expects that DROP TABLE should always work, even if the > table would not be consistent. There should not be a need to use a > separate keyword to ensure that the table is really deleted. > > The used solution is: > - If a .frm table doesn't exists, try dropping the table from all storage > engines. > - If the .frm table exists but the table does not exist in the engine > try dropping the table from all storage engines. I don't see why is that needed, if .frm exists, it tells exactly what storage engine to use, there is no point in trying other engines. > - Update storage engines using many table files (.CVS, MyISAM, Aria) to > succeed with the drop even if some of the files are missing. > - Add HTON_AUTOMATIC_DELETE_TABLE to handlerton's where delete_table() > is not needed and always succeed. This is used by ha_delete_table_force() > to know which handlers to ignore when trying to drop a table without > a .frm file. > > The disadvantage of this solution is that a DROP TABLE on a non existing > table will be a bit slower as we have to ask all active storage engines > if they know anything about the table. > > Other things: > - Added a new flag MY_IGNORE_ENOENT to my_delete() to not give an error > if the file doesn't exist. This simplifies some of the code. > - Don't clear thd->error in ha_delete_table() if there was an active > error. This is a bug fix. > - handler::delete_table() will not abort if first file doesn't exists. > This is bug fix to handle the case when a drop table was aborted in > the middle. > - Cleaned up mysql_rm_table_no_locks() to ensure that if_exists uses > same code path as when it's not used. > - Use non_existing_Table_error() to detect if table didn't exists. > Old code used different errors tests in different position. > - Table_triggers_list::drop_all_triggers() now drops trigger file if > it can't be parsed instead of leaving it hanging around (bug fix) > - InnoDB doesn't anymore print error about .frm file out of sync with > InnoDB directory if .frm file does not exists. This change was required > to be able to try to drop an InnoDB file when .frm doesn't exists. > - Fixed bug in mi_delete_table() where the .MYD file would not be dropped > if the .MYI file didn't exists. > - Fixed memory leak in Mroonga when deleting non existing table > - Fixed memory leak in Connect when deleting non existing table > > Bugs fixed introduced by the original version of this commit: > MDEV-22826 Presence of Spider prevents tables from being force-deleted from > other engines I don't think this part of the comment makes any sense, when one looks through git history there is no "original version of this commit". It's like a rolled back transaction, that value did not exist, unless you use READ UNCOMMITTED :) > diff --git a/mysql-test/main/drop_table_force.test b/mysql-test/main/drop_table_force.test > new file mode 100644 > index 00000000000..8fdd79465b7 > --- /dev/null > +++ b/mysql-test/main/drop_table_force.test > @@ -0,1 +1,197 @@ > +--source include/have_log_bin.inc > +--source include/have_innodb.inc > + > +# > +# This test is based on the orginal test from Tencent for DROP TABLE ... FORCE > +# In MariaDB we did reuse the code but MariaDB does not require the FORCE > +# keyword to drop a table even if the .frm file or some engine files are > +# missing. > +# To make it easy to see the differences between the orginal code and > +# the new one, we have left some references to the original test case > +# > + > +CALL mtr.add_suppression("Operating system error number"); > +CALL mtr.add_suppression("The error means the system cannot"); > +CALL mtr.add_suppression("returned OS error 71"); not sure it's a good idea, error numbers aren't very portable, this will blow up some day like a time bomb > + > +let $DATADIR= `select @@datadir`; > + > +--echo #Test1: table with missing .ibd can be dropped directly > +# drop table without ibd > +create table t1(a int)engine=innodb; > +--remove_file $DATADIR/test/t1.ibd > +drop table t1; > +--list_files $DATADIR/test/ > + > +# Original DROP TABLE .. FORCE required SUPER privilege. MariaDB doesn't > +--echo # Test droping table without frm without super privilege > + > +# create table t1 and rm frm > +create table t1(a int) engine=innodb; > +--remove_file $DATADIR/test/t1.frm > + > +# create test user > +create user test identified by '123456'; > +grant all privileges on test.t1 to 'test'@'%'identified by '123456' with grant option; why with grant option? better remove it if it's not essential for drop to work. if it is essential, why is it? > + > +# connect as test > +connect (con_test, localhost, test,'123456', ); > +--connection con_test > + > +# drop table with user test > +drop table t1; > +--error ER_BAD_TABLE_ERROR > +drop table t1; > + > +# connect as root > +--connection default > + > +--disconnect con_test > +drop user test; > + > +# check files in datadir about t1 > +--list_files $DATADIR/test/ > + > +--echo #Test4: drop table can drop consistent table as well > +create table t1(a int) engine=innodb; > +drop table t1; aha, so it works! :) > + > +# check files in datadir about t1 > +--list_files $DATADIR/test/ > + > +--echo #Test5: drop table with triger, and with missing frm > +# create table t1 with triger and rm frm > +create table t1(a int)engine=innodb; > +create trigger t1_trg before insert on t1 for each row begin end; > + > +let $DATADIR= `select @@datadir`; > +--remove_file $DATADIR/test/t1.frm > + > +drop table t1; > +--error ER_BAD_TABLE_ERROR > +drop table t1; > + > +# check files in datadir about t1 > +--list_files $DATADIR/test/ > + > +--echo #Test6: table with foreign key references can not be dropped > +# create table with foreign key reference and rm frm > +CREATE TABLE parent (id INT NOT NULL, PRIMARY KEY (id)) ENGINE=INNODB; > +CREATE TABLE child (id INT, parent_id INT, INDEX par_ind (parent_id), FOREIGN KEY (parent_id) REFERENCES parent(id) ON DELETE CASCADE) ENGINE=INNODB; > +--remove_file $DATADIR/test/parent.frm > + > +# parent can not be dropped when there are foreign key references > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent; > + > +# parent can be dropped when there are no foreign key references > +drop table child; > +drop table parent; > + > +# check files in datadir about child and parent > +--list_files $DATADIR/test/ > + > +--echo #Test7: drop table twice > +create table t1(a int)engine=innodb; > +--remove_file $DATADIR/test/t1.frm > + > +# first drop table will success > +drop table t1; > + > +# check files in datadir about t1 > +--list_files $DATADIR/test/ > + > +# second drop with if exists will also ok > +drop table if exists t1; > + > +# check files in datadir about t1 > +--list_files $DATADIR/test/ > + > +--echo #Test8: check compatibility with if exists > +create table t1(a int)engine=innodb; > +--remove_file $DATADIR/test/t1.frm > + > +# first drop will success > +drop table t1; > + > +# check files in datadir about t1 > +--list_files $DATADIR/test/ > + > +# second drop with if exists will success > +drop table if exists t1; you've just tested it in Test7, haven't you? > + > +--echo #Test9: check compatibility with restrict/cascade > +# create table with foreign key reference and rm frm > +CREATE TABLE parent (id INT NOT NULL, PRIMARY KEY (id)) ENGINE=INNODB; > +CREATE TABLE child (id INT, parent_id INT, INDEX par_ind (parent_id), FOREIGN KEY (parent_id) REFERENCES parent(id) ON DELETE CASCADE) ENGINE=INNODB; > + > +# parent can not be dropped when there are froeign key references typo "foreign" > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent; > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent restrict; > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent cascade; > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent; > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent restrict; > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent cascade; > + > +# parent can be dropped when there are no foreign key references > +drop table child; > +drop table parent; > + > +--echo #Test10: drop non-innodb engine table returns ok > +# create myisam table t1 and rm .frm > +create table t1(a int) engine=myisam; > +--remove_file $DATADIR/test/t1.frm > +--replace_result \\ / > +drop table t1; > + > +# create myisam table t1 and rm .MYD > +create table t1(a int) engine=myisam; > +--remove_file $DATADIR/test/t1.MYD > +--replace_result \\ / > +drop table t1; > + > +# create myisam table t1 and rm .MYI > +create table t1(a int) engine=myisam; > +--remove_file $DATADIR/test/t1.MYI > +--replace_result \\ / > +drop table t1; > +--list_files $DATADIR/test/ > + > +# create Aria table t1 and rm .frm and .MAD > +create table t1(a int) engine=aria; > +--remove_file $DATADIR/test/t1.frm > +--remove_file $DATADIR/test/t1.MAD > +--list_files $DATADIR/test/ > +--error ER_BAD_TABLE_ERROR > +drop table t1; > +--replace_result \\ / > +show warnings; > +--list_files $DATADIR/test/ > + > +# create Aria table t2 and rm .frm and .MAI > +create table t2(a int) engine=aria; > +flush tables; > +--remove_file $DATADIR/test/t2.frm > +--remove_file $DATADIR/test/t2.MAI > +--list_files $DATADIR/test/ > +--error ER_BAD_TABLE_ERROR > +drop table t2; > +--replace_result \\ / > +show warnings; > +--list_files $DATADIR/test/ > + > +# create Aria table t2 and rm .MAI and .MAD > +create table t2(a int) engine=aria; > +flush tables; > +--remove_file $DATADIR/test/t2.MAD > +--remove_file $DATADIR/test/t2.MAI > +--list_files $DATADIR/test/ > +--replace_result \\ / > +drop table t2; may be csv and archive tests too? > diff --git a/sql/handler.cc b/sql/handler.cc > index 1af0157783e..5dd02d057c7 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -2674,26 +2712,34 @@ const char *get_canonical_filename(handler *file, const char *path, > } > > > -/** delete a table in the engine > +/** > + Delete a table in the engine > > + @return 0 Table was deleted > + @return -1 Table didn't exists, no error given > + @return # Error from table handler > + > @note > ENOENT and HA_ERR_NO_SUCH_TABLE are not considered errors. > - The .frm file will be deleted only if we return 0. > + The .frm file should be deleted by the caller only if we return <= 0. > */ > + > int ha_delete_table(THD *thd, handlerton *table_type, const char *path, > - const LEX_CSTRING *db, const LEX_CSTRING *alias, bool generate_warning) > + const LEX_CSTRING *db, const LEX_CSTRING *alias, > + bool generate_warning) > { > handler *file; > char tmp_path[FN_REFLEN]; > int error; > TABLE dummy_table; > TABLE_SHARE dummy_share; > + bool is_error= thd->is_error(); > DBUG_ENTER("ha_delete_table"); > > /* table_type is NULL in ALTER TABLE when renaming only .frm files */ > if (table_type == NULL || table_type == view_pseudo_hton || > ! (file=get_new_handler((TABLE_SHARE*)0, thd->mem_root, table_type))) > - DBUG_RETURN(0); > + DBUG_RETURN(-1); "Table didn't exists, no error given"? if get_new_handler() fails it's OOM. > > bzero((char*) &dummy_table, sizeof(dummy_table)); > bzero((char*) &dummy_share, sizeof(dummy_share)); > @@ -2723,8 +2769,10 @@ int ha_delete_table(THD *thd, handlerton *table_type, const char *path, > } > if (intercept) > { > + /* Clear error if we got it in this function */ > + if (!is_error) > thd->clear_error(); this is generally a wrong approach. One should use push_internal_handler to silence errors, not thd->is_error() or thd->clear_error(). In this particilar case there will be no errors if there was no error before ha_delete_table() - which is correct. But if there was an error before, this will leave spurious and confusing errors in `SHOW WARNINGS` output. > - error= 0; > + error= -1; > } > } > delete file; > @@ -4365,45 +4413,63 @@ uint handler::get_dup_key(int error) > > @note > We assume that the handler may return more extensions than > - was actually used for the file. > + was actually used for the file. We also assume that the first > + extension is the most important one. If this exist and we can't delete > + that one we will abort the delete. after "the most important one" you can add "(see the comment near handlerton::tablefile_extensions)" > + If the first one doesn't exists, we have to try to delete all other "doesn't exist" > + extension as there is chance that the server had crashed between > + the delete of the first file and the next > > @retval > 0 If we successfully deleted at least one file from base_ext and > didn't get any other errors than ENOENT > + > @retval > !0 Error > */ > + > int handler::delete_table(const char *name) > { > - int saved_error= 0; > - int error= 0; > - int enoent_or_zero; > + int saved_error= ENOENT; > + bool abort_if_first_file_error= 1; > + bool some_file_deleted= 0; > + DBUG_ENTER("handler::delete_table"); > > + // For discovery tables, it's ok if first file doesn't exists why would that be ok? > if (ht->discover_table) > - enoent_or_zero= 0; // the table may not exist in the engine, it's ok > - else > - enoent_or_zero= ENOENT; // the first file of bas_ext() *must* exist > + { > + abort_if_first_file_error= 0; > + saved_error= 0; > + if (!bas_ext()) > + { > + DBUG_ASSERT(ht->flags & HTON_AUTOMATIC_DELETE_TABLE); > + DBUG_RETURN(0); // Drop succeded > + } > + } > > for (const char **ext= bas_ext(); *ext ; ext++) > { > + int error; > - if (mysql_file_delete_with_symlink(key_file_misc, name, *ext, 0)) > + if ((error= mysql_file_delete_with_symlink(key_file_misc, name, *ext, > + MYF(0)))) why do you need this `error` variable, if you never use it? > { > if (my_errno != ENOENT) > { > + saved_error= my_errno; > /* > - If error on the first existing file, return the error. > + If error other than file not found on the first existing file, > + return the error. > Otherwise delete as much as possible. > */ > - if (enoent_or_zero) > - return my_errno; > - saved_error= my_errno; > + if (abort_if_first_file_error) > + DBUG_RETURN(saved_error); > } > } > else > - enoent_or_zero= 0; // No error for ENOENT > - error= enoent_or_zero; > + some_file_deleted= 1; > + abort_if_first_file_error= 0; > } > - return saved_error ? saved_error : error; > + DBUG_RETURN(some_file_deleted && saved_error == ENOENT ? 0 : saved_error); > } > > > @@ -4439,5 +4505,20 @@ void handler::drop_table(const char *name) > > > /** > + Return true if the error from drop table means that the > + table didn't exists > +*/ > + > +bool non_existing_table_error(int error) > +{ > + return (error == ENOENT || error == HA_ERR_NO_SUCH_TABLE || > + error == HA_ERR_UNSUPPORTED || > + error == ER_NO_SUCH_TABLE || > + error == ER_NO_SUCH_TABLE_IN_ENGINE || > + error == ER_WRONG_OBJECT); > +} this should, I believe, be inline in handler.h > + > + > +/** > Performs checks upon the table. > > @@ -4899,5 +4981,89 @@ handler::ha_drop_table(const char *name) > > > /** > + Structure used during force drop table. > +*/ > + > +struct st_force_drop_table_params > +{ > + const char *path; > + const LEX_CSTRING *db; > + const LEX_CSTRING *alias; > + bool generate_warning; > + int error; > +}; > + > + > +/** > + Try to delete table from a given plugin > + Table types with discovery is ignored as these .frm files would have > + been created during discovery and thus doesn't need to be found > + for drop table force > +*/ > + > +static my_bool delete_table_force(THD *thd, plugin_ref plugin, void *arg) > +{ > + handlerton *hton = plugin_hton(plugin); > + st_force_drop_table_params *param = (st_force_drop_table_params *)arg; > + > + /* > + We have to ignore HEAP tables as these may not have been created yet > + We also remove engines that is using discovery (as these will recrate > + any missing .frm if needed) and tables marked with > + HTON_AUTOMATIC_DELETE_TABLE as for these we can't check if the table > + ever existed. > + */ > + if (!hton->discover_table && hton->db_type != DB_TYPE_HEAP && > + !(hton->flags & HTON_AUTOMATIC_DELETE_TABLE)) > + { > + int error; > + error= ha_delete_table(thd, hton, param->path, param->db, > + param->alias, param->generate_warning); This is doing a a lot of extra work. dummy_table/dummy_share, if()s. Allocating and constructing new handlers. I suggest the following: 1. Introduce new handlerton method hton->drop_table(). By default it'll be hton->get_new_handler() get_canonical_filename() file->ha_delete_table() like now, so all engines will still work. But later we can start moving drop table functionality out of the handler. In this commit only engines like blackhole will have a dummy drop_table() method. 2. Here you can do, basically hton->drop_table(hton, param->path) without checking for discovery, without a new flag HTON_AUTOMATIC_DELETE_TABLE. I'm not sure why you check for DB_TYPE_HEAP, could you please explain that? > + if (error > 0 && !non_existing_table_error(error)) > + param->error= error; > + if (error == 0) > + { > + param->error= 0; > + return TRUE; // Table was deleted > + } > + } > + return FALSE; > +} > + > +/** > + @brief > + Traverse all plugins to delete table when .frm file is missing. > + > + @return -1 Table was not found in any engine > + @return 0 Table was found in some engine and delete succeded > + @return # Error from first engine that had a table but didn't succeed to > + delete the table > + @return HA_ERR_ROW_IS_REFERENCED if foreign key reference is encountered, > + > +*/ > + > +int ha_delete_table_force(THD *thd, const char *path, const LEX_CSTRING *db, > + const LEX_CSTRING *alias, bool generate_warning) > +{ > + st_force_drop_table_params param; > + Table_exists_error_handler no_such_table_handler; > + DBUG_ENTER("ha_delete_table_force"); > + > + param.path= path; > + param.db= db; > + param.alias= alias; > + param.generate_warning= generate_warning; this is never used, you always invoke ha_delete_table_force() with generate_warning=0. > + param.error= -1; // Table not found > + > + thd->push_internal_handler(&no_such_table_handler); > + if (plugin_foreach(thd, delete_table_force, MYSQL_STORAGE_ENGINE_PLUGIN, > + &param)) > + param.error= 0; // Delete succeded > + thd->pop_internal_handler(); > + DBUG_RETURN(param.error); > +} > + > + > +/** > Create a table in the engine: public interface. > > diff --git a/sql/handler.h b/sql/handler.h > index f592c635c5d..f5ccdbe1ee4 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1780,6 +1780,13 @@ handlerton *ha_default_tmp_handlerton(THD *thd); > */ > #define HTON_TRANSACTIONAL_AND_NON_TRANSACTIONAL (1 << 17) > > +/* > + The engine doesn't keep track of tables, delete_table() is not > + needed and delete_table() always returns 0 (table deleted). This flag > + mainly used to skip storage engines in case of ha_delete_table_force() > +*/ > +#define HTON_AUTOMATIC_DELETE_TABLE (1 << 18) > + > class Ha_trx_info; > > struct THD_TRANS > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 61b1113f680..7c79acef3ea 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -2327,36 +2329,42 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST *tables, bool if_exists, > thd->find_temporary_table(table) && > table->mdl_request.ticket != NULL)); > > + /* First try to delete temporary tables and temporary sequences */ > - if (table->open_type == OT_BASE_ONLY || !is_temporary_table(table) || > - (drop_sequence && table->table->s->table_type != TABLE_TYPE_SEQUENCE)) > - error= 1; > - else > + if ((table->open_type != OT_BASE_ONLY && is_temporary_table(table)) && unnecessary parentheses > + (!drop_sequence || table->table->s->table_type == TABLE_TYPE_SEQUENCE)) > { > table_creation_was_logged= table->table->s->table_creation_was_logged; > if (thd->drop_temporary_table(table->table, &is_trans, true)) > { > + /* > + This is a very unlikely scenaro as dropping a temporary table > + should always work. Would be better if we tried to drop all > + temporary tables before giving the error. > + */ > error= 1; > goto err; > } > - error= 0; > table->table= 0; > + temporary_table_was_dropped= 1; > } > > - if ((drop_temporary && if_exists) || !error) > + if ((drop_temporary && if_exists) || temporary_table_was_dropped) > { > /* > This handles the case of temporary tables. We have the following cases: > > - . "DROP TEMPORARY" was executed and a temporary table was affected > - (i.e. drop_temporary && !error) or the if_exists was specified (i.e. > - drop_temporary && if_exists). > - > - . "DROP" was executed but a temporary table was affected (.i.e > - !error). > + - "DROP TEMPORARY" was executed and table was dropped > + temporary_table_was_dropped == 1 > + - "DROP TEMPORARY IF EXISTS" was specified but no temporary table > + existed > + temporary_table_was_dropped == 0 > */ > if (!dont_log_query && table_creation_was_logged) > { > /* > + DROP TEMPORARY succeded. For the moment when we only come > + here on success (error == 0) DBUG_ASSERT(error == 0) please > + > If there is an error, we don't know the type of the engine > at this point. So, we keep it in the trx-cache. > */ > @@ -2387,7 +2395,8 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST *tables, bool if_exists, > is no need to proceed with the code that tries to drop a regular > table. > */ > - if (!error) continue; > + if (temporary_table_was_dropped) can temporary_table_was_dropped be false here? > + continue; > } > else if (!drop_temporary) > { > @@ -2402,48 +2411,33 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST *tables, bool if_exists, > path_length= build_table_filename(path, sizeof(path) - 1, db.str, > alias.str, reg_ext, 0); > } > + > DEBUG_SYNC(thd, "rm_table_no_locks_before_delete_table"); > error= 0; > - if (drop_temporary || > - (ha_table_exists(thd, &db, &alias, &table_type, &is_sequence) == 0 && > + if (drop_temporary) > + { > + /* "DROP TEMPORARY" but a temporary table was not found */ > + error= ENOENT; > + } > + else if (((frm_exists= ha_table_exists(thd, &db, &alias, &table_type, > + &is_sequence)) == 0 && this is an overkill. ha_table_exists() will do a full table discovery, you don't need it here, because you'll try to delete anyway. So, basically you only need dd_frm_type() here. > table_type == 0) || > (!drop_view && (was_view= (table_type == view_pseudo_hton))) || > (drop_sequence && !is_sequence)) > { > /* > One of the following cases happened: > - . "DROP TEMPORARY" but a temporary table was not found. > . "DROP" but table was not found > . "DROP TABLE" statement, but it's a view. > . "DROP SEQUENCE", but it's not a sequence > */ > - was_table= drop_sequence && table_type; > - if (if_exists) > - { > - char buff[FN_REFLEN]; > - int err= (drop_sequence ? ER_UNKNOWN_SEQUENCES : > - ER_BAD_TABLE_ERROR); > - String tbl_name(buff, sizeof(buff), system_charset_info); > - tbl_name.length(0); > - tbl_name.append(&db); > - tbl_name.append('.'); > - tbl_name.append(&table->table_name); > - push_warning_printf(thd, Sql_condition::WARN_LEVEL_NOTE, > - err, ER_THD(thd, err), > - tbl_name.c_ptr_safe()); > - > - /* > - Our job is done here. This statement was added to avoid executing > - unnecessary code farther below which in some strange corner cases > - caused the server to crash (see MDEV-17896). > - */ > - goto log_query; > - } > - else > - { > - non_tmp_error = (drop_temporary ? non_tmp_error : TRUE); > - error= 1; > - } > + wrong_drop_sequence= drop_sequence && table_type; > + was_table|= wrong_drop_sequence; > + local_non_tmp_error= 1; > + error= -1; > + if ((!frm_exists && !table_type) || // no .frm > + if_exists) > + error= ENOENT; > } > else > { > @@ -2509,49 +2507,138 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST *tables, bool if_exists, > goto err; > } > } > - else > + > + /* > + Delete the .frm file if we managed to delete the table from the > + engine or the table didn't exists in the engine > + */ > + if (likely(!error) || non_existing_table_error(error)) > { > /* Delete the table definition file */ > strmov(end,reg_ext); > if (table_type && table_type != view_pseudo_hton && > - table_type->discover_table) > + (table_type->discover_table || error)) I don't think this new drop table logic needs to have any special handling of discovery. If the table didn't exist in engine - it's the normal case now, discovery or not, it's be handled normally. > { > /* > - Table type is using discovery and may not need a .frm file. > + Table type is using discovery and may not need a .frm file > + or the .frm file existed but no table in engine. > Delete it silently if it exists > */ > - (void) mysql_file_delete(key_file_frm, path, MYF(0)); > + if (mysql_file_delete(key_file_frm, path, > + MYF(MY_WME | MY_IGNORE_ENOENT))) > + error= my_errno; > } > else if (unlikely(mysql_file_delete(key_file_frm, path, > - MYF(MY_WME)))) > + !error ? MYF(MY_WME) : > + MYF(MY_WME | MY_IGNORE_ENOENT)))) > { > frm_delete_error= my_errno; > DBUG_ASSERT(frm_delete_error); > } > } > + frm_was_deleted= 1; > if (thd->replication_flags & OPTION_IF_EXISTS) > log_if_exists= 1; > > - if (likely(!error)) > + if (frm_delete_error) > { > - int trigger_drop_error= 0; > + /* > + Remember error if unexpected error from dropping the .frm file > + or we got an error from ha_delete_table() > + */ > + if (frm_delete_error != ENOENT) > + error= frm_delete_error; > + else if (if_exists && ! error) > + thd->clear_error(); no, please, don't do thd->clear_error(). I've seen and fixed so many bugs because of that. The last one - just an hour ago. Use push_internal_handler() instead. > + } > + if (likely(!error) || !frm_delete_error) > + non_tmp_table_deleted= TRUE; > > - if (likely(!frm_delete_error)) > + if (likely(!error) || non_existing_table_error(error)) > { > + trigger_drop_executed= 1; > + > + if (Table_triggers_list::drop_all_triggers(thd, &db, > + &table->table_name, > + MYF(MY_WME | > + MY_IGNORE_ENOENT))) > + error= error ? error : -1; > + } > + local_non_tmp_error|= MY_TEST(error); > + } > + > + /* > + If there was no .frm file and the table is not temporary, > + scan all engines try to drop the table from there. > + This is to ensure we don't have any partial table files left. > + > + We check for trigger_drop_executed to ensure we don't again try > + to drop triggers when it failed above (after sucecssfully dropping > + the table). > + */ > + if (non_existing_table_error(error) && !drop_temporary && > + table_type != view_pseudo_hton && !trigger_drop_executed && > + !wrong_drop_sequence) > + { > + char *end; > + int ferror= 0; > + > + /* Remove extension for delete */ > + *(end = path + path_length - reg_ext_length) = '\0'; > + ferror= ha_delete_table_force(thd, path, &db, > + &table->table_name, 0); > + if (!ferror) > + { > + /* Table existed and was deleted */ > non_tmp_table_deleted= TRUE; > - trigger_drop_error= > - Table_triggers_list::drop_all_triggers(thd, &db, > - &table->table_name); > + local_non_tmp_error= 0; > + error= 0; > } > + if (ferror <= 0) > + { > + ferror= 0; // Ignore table not found > > - if (unlikely(trigger_drop_error) || > - (frm_delete_error && frm_delete_error != ENOENT)) > - error= 1; > - else if (frm_delete_error && if_exists) > - thd->clear_error(); > + /* Delete the table definition file */ > + if (!frm_was_deleted) > + { > + strmov(end, reg_ext); > + if (mysql_file_delete(key_file_frm, path, > + MYF(MY_WME | MY_IGNORE_ENOENT))) > + ferror= my_errno; > } > - non_tmp_error|= MY_TEST(error); > + if (Table_triggers_list::drop_all_triggers(thd, &db, > + &table->table_name, > + MYF(MY_WME | > + MY_IGNORE_ENOENT))) > + ferror= -1; Why do you need two copies of code to drop triggers? Why cannot you do it once, on the first drop_all_triggers() place? > } > + if (!error) > + error= ferror; > + } > + > + /* > + Don't give an error if we are using IF EXISTS for a table that > + didn't exists > + */ > + > + if (if_exists && non_existing_table_error(error)) > + { > + char buff[FN_REFLEN]; > + int err= (drop_sequence ? ER_UNKNOWN_SEQUENCES : > + ER_BAD_TABLE_ERROR); > + String tbl_name(buff, sizeof(buff), system_charset_info); > + tbl_name.length(0); > + tbl_name.append(&db); > + tbl_name.append('.'); > + tbl_name.append(&table->table_name); > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_NOTE, > + err, ER_THD(thd, err), > + tbl_name.c_ptr_safe()); > + error= 0; > + local_non_tmp_error= 0; > + drop_table_not_done= 1; > + } > + non_tmp_error|= local_non_tmp_error; > > if (error) > { > diff --git a/storage/blackhole/ha_blackhole.cc b/storage/blackhole/ha_blackhole.cc > index 15548350b20..98589f1d043 100644 > --- a/storage/blackhole/ha_blackhole.cc > +++ b/storage/blackhole/ha_blackhole.cc > @@ -399,7 +399,7 @@ static int blackhole_init(void *p) > blackhole_hton= (handlerton *)p; > blackhole_hton->db_type= DB_TYPE_BLACKHOLE_DB; > blackhole_hton->create= blackhole_create_handler; > - blackhole_hton->flags= HTON_CAN_RECREATE; > + blackhole_hton->flags= HTON_CAN_RECREATE | HTON_AUTOMATIC_DELETE_TABLE; > > mysql_mutex_init(bh_key_mutex_blackhole, > &blackhole_mutex, MY_MUTEX_INIT_FAST); > diff --git a/storage/blackhole/ha_blackhole.h b/storage/blackhole/ha_blackhole.h > index 6ee30877b64..646fba6da9f 100644 > --- a/storage/blackhole/ha_blackhole.h > +++ b/storage/blackhole/ha_blackhole.h > @@ -95,6 +95,10 @@ class ha_blackhole: public handler > THR_LOCK_DATA **store_lock(THD *thd, > THR_LOCK_DATA **to, > enum thr_lock_type lock_type); > + int delete_table(const char *name) > + { > + return 0; > + } > private: > virtual int write_row(const uchar *buf); > virtual int update_row(const uchar *old_data, const uchar *new_data); > diff --git a/storage/connect/ha_connect.cc b/storage/connect/ha_connect.cc > index 0194b08c792..b1b44085e53 100644 > --- a/storage/connect/ha_connect.cc > +++ b/storage/connect/ha_connect.cc > @@ -5192,7 +5192,8 @@ int ha_connect::delete_or_rename_table(const char *name, const char *to) > } // endif pos > > } // endif open_table_def > - > + else > + rc= ENOENT; > free_table_share(share); > } else // Temporary file > ok= true; > diff --git a/storage/connect/mysql-test/connect/r/drop-open-error.result b/storage/connect/mysql-test/connect/r/drop-open-error.result > index f0ad8553d8b..34f58a845dc 100644 > --- a/storage/connect/mysql-test/connect/r/drop-open-error.result > +++ b/storage/connect/mysql-test/connect/r/drop-open-error.result > @@ -2,6 +2,8 @@ create table t1 (c varchar(8)); > create table tcon engine=connect table_type=mysql CONNECTION='mysql://root@localhost/test/t1' SRCDEF='select c from t1 where c in ("foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar")'; > ERROR HY000: Too long value for 'SRCDEF' > drop table mdev9949; > +Warnings: > +Warning 1017 Can't find file: './test/mdev9949.dos' (errno: 2 "No such file or directory") > drop table t1; > select @@secure_file_priv 'must be NULL'; > must be NULL > diff --git a/storage/federated/ha_federated.cc b/storage/federated/ha_federated.cc > index 00168791d35..b62b74d12bb 100644 > --- a/storage/federated/ha_federated.cc > +++ b/storage/federated/ha_federated.cc > @@ -484,7 +484,8 @@ int federated_db_init(void *p) > federated_hton->commit= federated_commit; > federated_hton->rollback= federated_rollback; > federated_hton->create= federated_create_handler; > - federated_hton->flags= HTON_ALTER_NOT_SUPPORTED | HTON_NO_PARTITION; > + federated_hton->flags= (HTON_ALTER_NOT_SUPPORTED | HTON_NO_PARTITION | > + HTON_AUTOMATIC_DELETE_TABLE); > > /* > Support for transactions disabled until WL#2952 fixes it. > diff --git a/storage/federated/ha_federated.h b/storage/federated/ha_federated.h > index 080d0ebd5f0..2e785f87858 100644 > --- a/storage/federated/ha_federated.h > +++ b/storage/federated/ha_federated.h > @@ -244,7 +244,10 @@ class ha_federated: public handler > void update_auto_increment(void); > int repair(THD* thd, HA_CHECK_OPT* check_opt); > int optimize(THD* thd, HA_CHECK_OPT* check_opt); > - > + int delete_table(const char *name) > + { > + return 0; > + } > int delete_all_rows(void); > int truncate(); > int create(const char *name, TABLE *form, > diff --git a/storage/federatedx/ha_federatedx.cc b/storage/federatedx/ha_federatedx.cc > index 6547964cc11..2370473236e 100644 > --- a/storage/federatedx/ha_federatedx.cc > +++ b/storage/federatedx/ha_federatedx.cc > @@ -438,7 +438,8 @@ int federatedx_db_init(void *p) > federatedx_hton->rollback= ha_federatedx::rollback; > federatedx_hton->discover_table_structure= ha_federatedx::discover_assisted; > federatedx_hton->create= federatedx_create_handler; > - federatedx_hton->flags= HTON_ALTER_NOT_SUPPORTED; > + federatedx_hton->flags= (HTON_ALTER_NOT_SUPPORTED | > + HTON_AUTOMATIC_DELETE_TABLE); > federatedx_hton->create_derived= create_federatedx_derived_handler; > federatedx_hton->create_select= create_federatedx_select_handler; > > diff --git a/storage/federatedx/ha_federatedx.h b/storage/federatedx/ha_federatedx.h > index a62456e1c33..67fe5f8cc22 100644 > --- a/storage/federatedx/ha_federatedx.h > +++ b/storage/federatedx/ha_federatedx.h > @@ -431,7 +431,10 @@ class ha_federatedx: public handler > void update_auto_increment(void); > int repair(THD* thd, HA_CHECK_OPT* check_opt); > int optimize(THD* thd, HA_CHECK_OPT* check_opt); > - > + int delete_table(const char *name) > + { > + return 0; > + } > int delete_all_rows(void); > int create(const char *name, TABLE *form, > HA_CREATE_INFO *create_info); //required > diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc > index 6940f8ff3bd..97cfdb01d9f 100644 > --- a/storage/innobase/handler/ha_innodb.cc > +++ b/storage/innobase/handler/ha_innodb.cc > @@ -13347,5 +13347,18 @@ ha_innobase::discard_or_import_tablespace( > } > > /** > + @return 1 if frm file exists > + @return 0 if it doesn't exists > +*/ > + > +static bool frm_file_exists(const char *path) > +{ > + char buff[FN_REFLEN]; > + strxnmov(buff, FN_REFLEN, path, reg_ext, NullS); > + return !access(buff, F_OK); > +} > + > + > +/** > Drops a table from an InnoDB database. Before calling this function, > MySQL calls innobase_commit to commit the transaction of the current user. > @@ -13446,7 +13459,9 @@ inline int ha_innobase::delete_table(const char* name, enum_sql_command sqlcom) > } > } > > - if (err == DB_TABLE_NOT_FOUND) { > + if (err == DB_TABLE_NOT_FOUND && > + frm_file_exists(name)) > + { > /* Test to drop all tables which matches db/tablename + '#'. > Only partitions can have '#' as non-first character in > the table name! > diff --git a/storage/mroonga/mrn_table.cpp b/storage/mroonga/mrn_table.cpp > index b1b2db6161b..037a6a59487 100644 > --- a/storage/mroonga/mrn_table.cpp > +++ b/storage/mroonga/mrn_table.cpp > @@ -1080,6 +1080,7 @@ TABLE_SHARE *mrn_create_tmp_table_share(TABLE_LIST *table_list, const char *path > if (open_table_def(thd, share, GTS_TABLE)) > { > *error = ER_CANT_OPEN_FILE; > + mrn_free_tmp_table_share(share); > DBUG_RETURN(NULL); > } > DBUG_RETURN(share); > diff --git a/storage/myisam/mi_delete_table.c b/storage/myisam/mi_delete_table.c > index 7990c3e8a80..d318b44720a 100644 > --- a/storage/myisam/mi_delete_table.c > +++ b/storage/myisam/mi_delete_table.c > @@ -28,19 +28,23 @@ > > int mi_delete_table(const char *name) > { > + int error= 0; > DBUG_ENTER("mi_delete_table"); > > #ifdef EXTRA_DEBUG > check_table_is_closed(name,"delete"); > #endif > > - if (mysql_file_delete_with_symlink(mi_key_file_kfile, name, MI_NAME_IEXT, MYF(MY_WME)) || > - mysql_file_delete_with_symlink(mi_key_file_dfile, name, MI_NAME_DEXT, MYF(MY_WME))) > - DBUG_RETURN(my_errno); > + if (mysql_file_delete_with_symlink(mi_key_file_kfile, name, MI_NAME_IEXT, > + MYF(MY_WME))) > + error= my_errno; > + if (mysql_file_delete_with_symlink(mi_key_file_dfile, name, MI_NAME_DEXT, > + MYF(MY_WME))) > + error= my_errno; > > // optionally present: > mysql_file_delete_with_symlink(mi_key_file_dfile, name, ".OLD", MYF(0)); > mysql_file_delete_with_symlink(mi_key_file_dfile, name, ".TMD", MYF(0)); > > - DBUG_RETURN(0); > + DBUG_RETURN(error); > } > diff --git a/storage/perfschema/ha_perfschema.cc b/storage/perfschema/ha_perfschema.cc > index 33d4c854c8b..c4c59d109f5 100644 > --- a/storage/perfschema/ha_perfschema.cc > +++ b/storage/perfschema/ha_perfschema.cc > @@ -95,10 +95,11 @@ static int pfs_init_func(void *p) > > pfs_hton->create= pfs_create_handler; > pfs_hton->show_status= pfs_show_status; > - pfs_hton->flags= HTON_ALTER_NOT_SUPPORTED | > + pfs_hton->flags= (HTON_ALTER_NOT_SUPPORTED | > HTON_TEMPORARY_NOT_SUPPORTED | > HTON_NO_PARTITION | > - HTON_NO_BINLOG_ROW_OPT; > + HTON_NO_BINLOG_ROW_OPT | > + HTON_AUTOMATIC_DELETE_TABLE); > > /* > As long as the server implementation keeps using legacy_db_type, > diff --git a/storage/sequence/sequence.cc b/storage/sequence/sequence.cc > index 8684a5c60b9..31522b8f3b7 100644 > --- a/storage/sequence/sequence.cc > +++ b/storage/sequence/sequence.cc > @@ -69,10 +69,15 @@ class ha_seq: public handler > > /* open/close/locking */ > int create(const char *name, TABLE *table_arg, > - HA_CREATE_INFO *create_info) { return HA_ERR_WRONG_COMMAND; } > + HA_CREATE_INFO *create_info) > + { return HA_ERR_WRONG_COMMAND; } > > int open(const char *name, int mode, uint test_if_locked); > int close(void); > + int delete_table(const char *name) > + { > + return 0; > + } > THR_LOCK_DATA **store_lock(THD *, THR_LOCK_DATA **, enum thr_lock_type); > > /* table scan */ > @@ -503,6 +508,7 @@ static int init(void *p) > hton->savepoint_set= hton->savepoint_rollback= hton->savepoint_release= > dummy_savepoint; > hton->create_group_by= create_group_by_handler; > + hton->flags= HTON_AUTOMATIC_DELETE_TABLE; > return 0; > } > > @@ -526,4 +532,3 @@ maria_declare_plugin(sequence) > MariaDB_PLUGIN_MATURITY_STABLE > } > maria_declare_plugin_end; > - > diff --git a/storage/sphinx/ha_sphinx.cc b/storage/sphinx/ha_sphinx.cc > index 8403c767796..d60a4d229e6 100644 > --- a/storage/sphinx/ha_sphinx.cc > +++ b/storage/sphinx/ha_sphinx.cc > @@ -696,7 +696,7 @@ handlerton sphinx_hton = > NULL, // create_cursor_read_view > NULL, // set_cursor_read_view > NULL, // close_cursor_read_view > - HTON_CAN_RECREATE > + HTON_CAN_RECREATE | HTON_AUTOMATIC_DELETE_TABLE > }; > #else > static handlerton * sphinx_hton_ptr = NULL; > @@ -749,7 +749,7 @@ static int sphinx_init_func ( void * p ) > hton->close_connection = sphinx_close_connection; > hton->show_status = sphinx_show_status; > hton->panic = sphinx_panic; > - hton->flags = HTON_CAN_RECREATE; > + hton->flags = HTON_CAN_RECREATE | HTON_AUTOMATIC_DELETE_TABLE; > #endif > } > SPH_RET(0); > diff --git a/storage/spider/spd_table.cc b/storage/spider/spd_table.cc > index 1607dd07902..583bfebbed6 100644 > --- a/storage/spider/spd_table.cc > +++ b/storage/spider/spd_table.cc > @@ -7249,7 +7249,7 @@ int spider_db_init( > DBUG_ENTER("spider_db_init"); > spider_hton_ptr = spider_hton; > > - spider_hton->flags = HTON_NO_FLAGS; > + spider_hton->flags = HTON_NO_FLAGS | HTON_AUTOMATIC_DELETE_TABLE; > #ifdef HTON_CAN_READ_CONNECT_STRING_IN_PARTITION > spider_hton->flags |= HTON_CAN_READ_CONNECT_STRING_IN_PARTITION; > #endif Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

[Maria-developers] Adding proper expressions to mysqltest
by Игорь Балашенко 14 Jun '20

14 Jun '20

----------------------------------------------------------- Sorry, I had to duplicate this message again to get more attention. I didn`t forget about theme this time ------------------------------------------------------------ Hello mariaDB developers! I am a participant of GSOC 2020 mentored by Nikita Malyavin and Sergey Golubchik. Currently I am working on mysqltest parser improvements and I have a question to discuss with everyone: To be short: Currently there are expressions in if and while to be calculated i.e if(<expr>) code But it is written using switch and works only in condition block. Code like: let $a= 1 + 2 Makes $a = "1 + 2" Obviously mysqltest syntax needs proper expressions thats why I decided to make them. There are following questions: What syntax do you want to see? let $a = 1 + 2 $a = 3 Is a bad idea because we will break lots of previous test cases. let $a = $((1 + 2)) That is done in a bash style, so it seems to be more comfortable to everyone. Another question is what operators should be in expressions? To begin with I want to implement: +, -. *, /, ++, --, <, >, ==, != To implement expressions I want to use Byson to make it easier to rewrite parser if needed I would be glad to read feedback and critique. ------------------------------

1 0

[Maria-developers] (no subject)
by Игорь Балашенко 12 Jun '20

12 Jun '20

Hello mariaDB developers! I am a participant of GSOC 2020 mentored by Nikita Malyavin and Sergey Golubchik. Currently I am working on mysqltest parser improvements and I have a question to discuss with everyone: To be short: Currently there are expressions in if and while to be calculated i.e if(<expr>) code But it is written using switch and works only in condition block. Code like: let $a= 1 + 2 Makes $a = "1 + 2" Obviously mysqltest syntax needs proper expressions thats why I decided to make them. There are following questions: What syntax do you want to see? let $a = 1 + 2 $a = 3 Is a bad idea because we will break lots of previous taest cases. let $a = $((1 + 2)) That is done in a bash style, so it seems to be more comfortable to everyone. Another question is what operators should be in expressions? To begin with I want to implement: +, -. *, /, ++, --, <, >, ==, != To implement expressions I want to use Byson to make it easier to rewrite parser if needed I would be glad to read feedback and critique. ------------------------------ GSOC2020 participant Balashenko Igor

3 2

Re: [Maria-developers] 2849076672d: MDEV-16470: switch off user variables (and fixes of its support)
by Sergei Golubchik 12 Jun '20

12 Jun '20

Hi, Oleksandr! On Jun 12, Oleksandr Byelkin wrote: > revision-id: 2849076672d (mariadb-10.5.2-322-g2849076672d) > parent(s): bb47050e1fd > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-06-12 08:32:56 +0200 > message: > > MDEV-16470: switch off user variables (and fixes of its support) > > diff --git a/client/mysqltest.cc b/client/mysqltest.cc > index 55c4a66a89d..24e0cd25108 100644 > --- a/client/mysqltest.cc > +++ b/client/mysqltest.cc > &data, &data_length)) > { > dynstr_append(ds, "-- "); > - switch (type) > + if (type <= SESSION_TRACK_END) > { > - case SESSION_TRACK_SYSTEM_VARIABLES: > - dynstr_append(ds, "Tracker : SESSION_TRACK_SYSTEM_VARIABLES\n"); > - break; > - case SESSION_TRACK_SCHEMA: > - dynstr_append(ds, "Tracker : SESSION_TRACK_SCHEMA\n"); > - break; > - case SESSION_TRACK_STATE_CHANGE: > - dynstr_append(ds, "Tracker : SESSION_TRACK_STATE_CHANGE\n"); > - break; > - case SESSION_TRACK_GTIDS: > - dynstr_append(ds, "Tracker : SESSION_TRACK_GTIDS\n"); > - break; > - case SESSION_TRACK_TRANSACTION_CHARACTERISTICS: > - dynstr_append(ds, "Tracker : SESSION_TRACK_TRANSACTION_CHARACTERISTICS\n"); > - break; > - case SESSION_TRACK_TRANSACTION_TYPE: > - dynstr_append(ds, "Tracker : SESSION_TRACK_TRANSACTION_TYPE\n"); > - break; > - default: > - DBUG_ASSERT(0); > - dynstr_append(ds, "\n"); > + dynstr_append(ds, trking_info_desc[type]); > + } > + else > + { > + DBUG_ASSERT(0); > + dynstr_append(ds, "Trecker???\n"); "Tracker" > } > - > > dynstr_append(ds, "-- "); > dynstr_append_mem(ds, data, data_length); > diff --git a/libmariadb b/libmariadb > index ce74fd0c400..db7adf63fbf 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit ce74fd0c4009ed9f4bcbdb4a01e96c823e961dc3 > +Subproject commit db7adf63fbf242a07c355f192a4d8c60323b5d89 No, I don't see any need for that now. C/C does not know anything about SESSION_TRACK_USER_VARIABLES, let's just keep it that way. > diff --git a/sql/item_func.cc b/sql/item_func.cc > index da185f9878b..448bfd380a2 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -4702,10 +4702,12 @@ update_hash(user_var_entry *entry, bool set_null, void *ptr, size_t length, > entry->unsigned_flag= unsigned_arg; > } > entry->type=type; > +#ifdef USER_VAR_TACKING TRACKING > #ifndef EMBEDDED_LIBRARY > THD *thd= current_thd; > thd->session_tracker.user_variables.mark_as_changed(thd, entry); > #endif > +#endif // USER_VAR_TACKING > return 0; > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 4f528bb514c: MDEV-22011: DISTINCT with JSON_ARRAYAGG gives wrong results
by Sergey Petrunia 11 Jun '20

11 Jun '20

Hi Varun, I only have input about comments/variable names. Please find it below. On Tue, Mar 24, 2020 at 10:14:38AM +0530, Varun wrote: > revision-id: 4f528bb514c607d848b30119cea135e37e0d9c69 (mariadb-10.5.0-425-g4f528bb514c) > parent(s): 1cd8e8213b6019794ca7f01e8276cfd1ba6408cf > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-03-24 10:13:28 +0530 > message: > > MDEV-22011: DISTINCT with JSON_ARRAYAGG gives wrong results > > For JSON_ARRAYAGG with DISTINCT also store the null bytes for the record > inside the Unique object > > --- > mysql-test/main/func_json.result | 4 +- > sql/item_jsonfunc.cc | 36 ++++++++++++-- > sql/item_jsonfunc.h | 9 ++-- > sql/item_sum.cc | 101 ++++++++++++++++++++++++++++++++++----- > sql/item_sum.h | 13 +++-- > 5 files changed, 133 insertions(+), 30 deletions(-) > > diff --git a/sql/item_jsonfunc.cc b/sql/item_jsonfunc.cc > index a3b5d3b7fac..ed496df8169 100644 > --- a/sql/item_jsonfunc.cc > +++ b/sql/item_jsonfunc.cc > @@ -1452,6 +1452,31 @@ static int append_json_value(String *str, Item *item, String *tmp_val) > } > > > +bool append_json_value(String *str, String *res, Item *item, bool null_value) There is another append_json_value right above this one. That function has no comments but let's not follow that bad pattern. - Please add a function comment describing this function: @brief should describe this function @detail should describe the difference from the one above. It is very counter-intuitive that the parameter named "res" does not hold the result of this function. Please rename it (to value?) (can it be changed to be const String* ?) > +{ > + if (null_value) > + return str->append("null", 4); > + > + if (item->type_handler()->is_bool_type()) > + { > + return ((*res)[0] == '1') ? > + str->append("true", 4) : > + str->append("false", 5); > + } > + > + if (item->is_json_type()) > + return str->append(res->ptr(), res->length()); > + > + if (item->result_type() == STRING_RESULT) > + { > + return str->append("\"", 1) || > + st_append_escaped(str, res) || > + str->append("\"", 1); > + } > + return st_append_escaped(str, res); This made me wonder why something is escaped if it is not enclosed in quotes, but now I see this is just copied from above. > +} > + > + > static int append_json_keyname(String *str, Item *item, String *tmp_val) > { > String *sv= item->val_str(tmp_val); > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index 5c732095573..a04dbdc19e2 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -3552,6 +3552,54 @@ int group_concat_key_cmp_with_distinct(void* arg, const void* key1, > } > > > +int group_concat_key_cmp_with_distinct_with_nulls(void* arg, > + const void* key1_arg, > + const void* key2_arg) > +{ > + Item_func_group_concat *item_func= (Item_func_group_concat*)arg; > + > + uchar *key1= (uchar*)key1_arg + item_func->table->s->null_bytes; > + uchar *key2= (uchar*)key2_arg + item_func->table->s->null_bytes; > + > + for (uint i= 0; i < item_func->arg_count_field; i++) This loop is actually not needed, right? JSON_ARRAYAGG only accepts one argument. Please add a note. > + { > + Item *item= item_func->args[i]; > + /* > + If item is a const item then either get_tmp_table_field returns 0 > + or it is an item over a const table. > + */ > + if (item->const_item()) > + continue; > + /* > + We have to use get_tmp_table_field() instead of > + real_item()->get_tmp_table_field() because we want the field in > + the temporary table, not the original field > + */ > + Field *field= item->get_tmp_table_field(); > + > + if (!field) > + continue; > + > + if (field->is_null_in_record((uchar*)key1_arg) && > + field->is_null_in_record((uchar*)key2_arg)) > + continue; > + > + if (field->is_null_in_record((uchar*)key1_arg)) > + return -1; > + > + if (field->is_null_in_record((uchar*)key2_arg)) > + return 1; > + > + uint offset= (field->offset(field->table->record[0]) - > + field->table->s->null_bytes); > + int res= field->cmp(key1 + offset, key2 + offset); > + if (res) > + return res; > + } > + return 0; > +} > + > + > /** > function of sort for syntax: GROUP_CONCAT(expr,... ORDER BY col,... ) > */ > @@ -3650,9 +3699,11 @@ int dump_leaf_key(void* key_arg, element_count count __attribute__((unused)), > result->append(*item->separator); > > > + bool is_null= false; > for (; arg < arg_end; arg++) > { > String *res; > + is_null= false; (1) > /* > We have to use get_tmp_table_field() instead of > real_item()->get_tmp_table_field() because we want the field in > @@ -3661,7 +3712,11 @@ int dump_leaf_key(void* key_arg, element_count count __attribute__((unused)), > because it contains both order and arg list fields. > */ > if ((*arg)->const_item()) > + { > res= (*arg)->val_str(&tmp); > + if ((*arg)->null_value) > + is_null= TRUE; (2) Please either use true/false (preferred) or TRUE / FALSE. > + } > else > { > Field *field= (*arg)->get_tmp_table_field(); ... > diff --git a/sql/item_sum.h b/sql/item_sum.h > index b10d9f5974d..908dbe3cce9 100644 > --- a/sql/item_sum.h > +++ b/sql/item_sum.h > @@ -1897,14 +1899,19 @@ class Item_func_group_concat : public Item_sum > */ > Item_func_group_concat *original; > > + bool exclude_nulls; > + > /* > Used by Item_func_group_concat and Item_func_json_arrayagg. The latter > needs null values but the former doesn't. > */ Is this comment relevant here? I think it's better to remove this and instead add a comment for the exclude_nulls above. > - bool add(bool exclude_nulls); > + bool add(); > > friend int group_concat_key_cmp_with_distinct(void* arg, const void* key1, > const void* key2); > + friend int group_concat_key_cmp_with_distinct_with_nulls(void* arg, > + const void* key1, > + const void* key2); > friend int group_concat_key_cmp_with_order(void* arg, const void* key1, > const void* key2); > friend int dump_leaf_key(void* key_arg, BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Improving `source` in mysqltest
by Nikita Malyavin 11 Jun '20

11 Jun '20

Hello mariadb developers! This year we have a student on mysqltest parser improvements, mentored by me and @serg. The major improvement will be proper expressions support, which you'll soon receive an email about. Another a bit more minor-ish improvement we want our student Igor to make is parameters support for source command, which I want to ask you about, which way is better to implement. It's quite spread in the tests to parametrize source with variables, for example > --let SEARCH_FILE=$MYSQLTEST_VARDIR/log/mysqld.1.err > --let SEARCH_PATTERN= This connection closed normally without > authentication > --source include/search_pattern_in_file.inc Another example is period/create_triggers.inc. So, we want make it possible to write > --source do_something.inc A=1 B=foobar And the question is, how should it behave. One way is to make it a strict equivalent of > let A=1; > let B=foobar; > source do_something.inc; This allows to reuse previously set values, for example: > --source do_something.inc A=1 B=foobar > ... > --source do_something.inc A=2 Which literally means > let A=1; > let B=foobar; > --source do_something.inc; ... let A=2; > --source do_something.inc; Another way is to make it work like local arguments, so the arguments are put in the stack, and as a result > let A=qwe; > --source do_something.inc A=1 B=foobar > --echo $A; outputs qwe. What do you think of this, what's more suitable for us? This is an important question, since all of us are using mysqltest, and we probably wouldn't change the behavior ever later. -- Yours truly, Nikita Malyavin

1 0

Re: [Maria-developers] MDEV-21831: Assertion `length == pack_length()' failed in Field_inet6::sort_string upon INSERT into RocksDB table
by Sergey Petrunia 11 Jun '20

11 Jun '20

On Thu, Jun 11, 2020 at 03:41:24PM +0530, Varun Gupta wrote: > > /* > > pack_length() returns size (in bytes) used to store field data in > > memory > > (i.e. it returns the maximum size of the field in a row of the table, > > which is located in RAM). > > */ > > virtual uint32 pack_length() const { return (uint32) field_length; } > > > > so, it is size in bytes, while strnxfrmlen() accepts length *in > > characters*. > > That is, strnxfrmlen(field->pack_length()) is certainly not correct. > > > > > This is not correct, strnxfrmlen accepts length in bytes and there we > change the length to characters and then calculate the weights. You're right. I've missed this part. The patch is ok to push, then. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Progress report week 5
by Mohammed Hammaad Mateen 10 Jun '20

10 Jun '20

Greetings, Hope you are safe and doing great, This post describes the things I've done during 1-7 June under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. The tasks taken up for this week Greetings mentors, I've realized that to extend the parser for multi delete, I should be changing the sql_yacc.yy file, for incorporating opt_returning rule in multi-delete. opt_returning rule works completely fine with single table delete, but to tweak it to adapt for multi-delete, required me to learn some bison grammar syntax and semantics and that took me a couple of days. Moreover everytime I made changes to the grammar, the server used to crash rendering all changes invalid. I'll try to push the code that was triggering crashes, shortly. Thank you Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] 5ff9525ff86: MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE
by Sergei Golubchik 10 Jun '20

10 Jun '20

Hi, Sachin! On Jun 10, Sachin wrote: > revision-id: 5ff9525ff86 (mariadb-10.5.2-290-g5ff9525ff86) > parent(s): ff881056b8d > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-07 16:45:01 +0530 > message: > > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 5910ece12f0..ffda352c866 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -5570,6 +5570,12 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > { > master_had_triggers= table->master_had_triggers; > bool transactional_table= table->file->has_transactions_and_rollback(); > + if (get_general_type_code() == WRITE_ROWS_EVENT) > + table->file->prepare_for_insert(0); > + else if (get_general_type_code() == UPDATE_ROWS_EVENT || > + get_general_type_code() == DELETE_ROWS_EVENT) > + table->file->prepare_for_insert(1); > + You can simplify it as table->file->prepare_for_insert(get_general_type_code() != WRITE_ROWS_EVENT); otherwise ok to push Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-21831: Assertion `length == pack_length()' failed in Field_inet6::sort_string upon INSERT into RocksDB table
by Sergey Petrunia 09 Jun '20

09 Jun '20

Hi Varun, > MDEV-21831: Assertion `length == pack_length()' failed in Field_inet6::sort_string upon INSERT into RocksDB table > > For INET6 columns the values are stored as BINARY columns and returned to the client in TEXT format. > For rocksdb the indexes store mem-comparable images for columns, so use the pack_length() to store > the mem-comparable form for INET6 columns. This would also remain consistent with CHAR columns. > > diff --git a/storage/rocksdb/rdb_datadic.cc b/storage/rocksdb/rdb_datadic.cc > index 21a9b57b555e..6e73535d1de7 100644 > --- a/storage/rocksdb/rdb_datadic.cc > +++ b/storage/rocksdb/rdb_datadic.cc > @@ -3317,7 +3317,9 @@ bool Rdb_field_packing::setup(const Rdb_key_def *const key_descr, > field->field_length = field->char_length() * cs->mbmaxlen. > */ > const CHARSET_INFO *cs = field->charset(); > - m_max_image_len = cs->strnxfrmlen(field->field_length); > + m_max_image_len = cs->strnxfrmlen(type == MYSQL_TYPE_STRING ? > + field->pack_length() : > + field->field_length); Field::pack_length is defined as: /* pack_length() returns size (in bytes) used to store field data in memory (i.e. it returns the maximum size of the field in a row of the table, which is located in RAM). */ virtual uint32 pack_length() const { return (uint32) field_length; } so, it is size in bytes, while strnxfrmlen() accepts length *in characters*. That is, strnxfrmlen(field->pack_length()) is certainly not correct. Rdb_field_packing::m_max_image_len is defined as: /* Length of mem-comparable image of the field, in bytes */ int m_max_image_len; while pack_length() seems to describe table->record[0] format (or is it KeyTupleFormat) ? Field::sort_length() doesn't have a definition but it seems to be a closer match as it describes the length of the mem-comparable form? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 06942363ebc: MDEV-14347 CREATE PROCEDURE returns no error when using an unknown variable
by Sergei Golubchik 09 Jun '20

09 Jun '20

Hi, Alexander! On Jun 07, Alexander Barkov wrote: > revision-id: 06942363ebc (mariadb-10.5.2-311-g06942363ebc) > parent(s): 0d6d63e1505 > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2020-06-03 12:11:07 +0400 > message: > > MDEV-14347 CREATE PROCEDURE returns no error when using an unknown variable > > diff --git a/sql/item.cc b/sql/item.cc > index 8ea6366e6c4..5800b9bc95e 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -764,6 +764,29 @@ bool Item_field::collect_item_field_processor(void *arg) > } > > > +void Item_ident::undeclared_spvar_error() const Currently it's used only in Item_field::unknown_splocal_processor(). Do you plan to use this somewhere else? Why is it a separate method and in Item_ident? > +{ > + /* > + We assume this is an unknown SP variable, possibly a ROW variable. > + Print the leftmost name in the error: > + SET var=a; -> a > + SET var=a.b; -> a > + SET var=a.b.c; -> a > + */ > + my_error(ER_SP_UNDECLARED_VAR, MYF(0), db_name.str ? db_name.str : > + table_name.str ? table_name.str : > + field_name.str); > +} > + > +bool Item_field::unknown_splocal_processor(void *arg) > +{ > + DBUG_ENTER("Item_field::unknown_splocal_processor"); > + DBUG_ASSERT(type() == FIELD_ITEM); > + undeclared_spvar_error(); > + DBUG_RETURN(true); > +} > + > + > bool Item_field::add_field_to_set_processor(void *arg) > { > DBUG_ENTER("Item_field::add_field_to_set_processor"); > @@ -8012,6 +8035,27 @@ void Item_ref::cleanup() > } > > > +bool Item_ref::unknown_splocal_processor(void *arg) > +{ > + DBUG_ENTER("Item_ref::unknown_splocal_processor"); > + DBUG_ASSERT(type() == REF_ITEM); > + DBUG_ASSERT(ref_type() == REF); > + undeclared_spvar_error(); > + DBUG_RETURN(true); > +} > + > + > +bool Item_ref::walk(Item_processor processor, bool walk_subquery, void *arg) > +{ > + if (processor == &Item::unknown_splocal_processor) > + return unknown_splocal_processor(arg); Why is that? This is very unusual, normally walk() should not change its behavior based on the processor. > + if (ref && *ref) > + return (*ref)->walk(processor, walk_subquery, arg) || > + (this->*processor)(arg); > + return false; > +} > + > + > /** > Transform an Item_ref object with a transformer callback function. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] JSON aggregate functions working on JSON documents
by Sergey Petrunia 08 Jun '20

08 Jun '20

On Mon, Jun 08, 2020 at 04:05:48PM +0400, Alexey Botchkov wrote: > So the problem is that we (MariaDB) doesn't consider the JSON field content > as a JSON, but as a plain text. > That affects not only aggregate functions, but any other expecting JSON > arguments. > To me it's the bug, should be fixed. Will you report it, Sergei? Filed: https://jira.mariadb.org/browse/MDEV-22837 BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Progress Report GSoC 2020 (1-6 June)
by Anton Mikhaylenko 08 Jun '20

08 Jun '20

Good day, everybody. In this post, I want to describe what I did this week, June 1-6. I have added the initial implementation of the hill_climbing class. It includes an update function - the main function that changes the required number of threads. To implement this function, I added the Complex class - the base class for working with complex numbers. https://github.com/sapiest/server/commits/10.5-update_threadpool_concurrency The main work is not completed yet, as the algorithm requires adjustments and adaptation in MariaDB environment. Regards, Anton MIkhailenko

1 0

[Maria-developers] JSON aggregate functions working on JSON documents
by Sergey Petrunia 08 Jun '20

08 Jun '20

Hi Alexey, I'm looking at how JSON_ARRAYAGG and JSON_OBJECTAGG work at processing JSON documents. I'm trying these examples, first on MySQL-8: create table t30 (col1 json); insert into t30 values('{"color":"red", "size":1}' ); insert into t30 values('{"color":"blue", "size":2}' ); select JSON_ARRAYAGG(col1) from t30; +-------------------------------------------------------------+ | JSON_ARRAYAGG(col1) | +-------------------------------------------------------------+ | [{"size": 1, "color": "red"}, {"size": 2, "color": "blue"}] | +-------------------------------------------------------------+ Good. create table t31 (name varchar(100), contents json); insert into t31 values('obj1', '{"color":"blue", "size":2}' ); insert into t31 values('obj2', '{"color":"red", "size":1}' ); select JSON_OBJECTAGG(name, contents) from t31; +-----------------------------------------------------------------------------+ | JSON_OBJECTAGG(name, contents) | +-----------------------------------------------------------------------------+ | {"obj1": {"size": 2, "color": "blue"}, "obj2": {"size": 1, "color": "red"}} | +-----------------------------------------------------------------------------+ Good. Now, trying these on MariaDB, I get: MariaDB [test]> select JSON_ARRAYAGG(col1) from t30; +------------------------------------------------------------------------+ | JSON_ARRAYAGG(col1) | +------------------------------------------------------------------------+ | ["{\"color\":\"red\", \"size\":1}","{\"color\":\"blue\", \"size\":2}"] | +------------------------------------------------------------------------+ MariaDB [test]> select JSON_OBJECTAGG(name, contents) from t31; +---------------------------------------------------------------------------------------+ | JSON_OBJECTAGG(name, contents) | +---------------------------------------------------------------------------------------+ | {"obj1":"{\"color\":\"blue\", \"size\":2}", "obj2":"{\"color\":\"red\", \"size\":1}"} | +---------------------------------------------------------------------------------------+ The JSON documents are coverted into strings. I think, the users will prefer MySQL's behavior. But which behaivor is "correct"? (i.e. required by the standard, etc) Is MariaDBs's behaivor a bug? Is this expected? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] MDEV-11563: GROUP_CONCAT(DISTINCT ...) may produce a non-distinct list
by Sergey Petrunia 07 Jun '20

07 Jun '20

Hi Varun, > revision-id: c408c41230c4a9de0114d13aaf861d3ced65f4c7 (mariadb-10.5.0-424-gc408c41230c) > parent(s): 95da2113a050ad739fdaf60ee871329468a01554 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-03-31 11:57:37 +0530 > message: > > MDEV-11563: GROUP_CONCAT(DISTINCT ...) may produce a non-distinct list > The commit message should describe the error and the fix made. (especially important here as this patch fixes only a part of the issue with GROUP_CONCAT) > --- > mysql-test/main/func_gconcat.result | 53 ++++++++++++++++++++++--------------- > mysql-test/main/func_gconcat.test | 7 +++++ > sql/item_sum.cc | 40 +++++++++++++++++----------- > sql/item_sum.h | 3 ++- > 4 files changed, 65 insertions(+), 38 deletions(-) > > diff --git a/sql/item_sum.h b/sql/item_sum.h > index e221d04397d..b10d9f5974d 100644 > --- a/sql/item_sum.h > +++ b/sql/item_sum.h > @@ -1879,7 +1879,8 @@ class Item_func_group_concat : public Item_sum > bool warning_for_row; > bool always_null; > bool force_copy_fields; > - bool no_appended; > + /** True if result has been written to output buffer. */ > + bool m_result_finalized; Why does this variable use the m_ convention when others do not? I think this is confusing (I stop for a moment to wonder how is it different from all others every time I see the name). Please remove this. What's "output buffer"? this->result ? And what does "result" mean here - the entire return value of GROUP_CONCAT function or some part of it? Please improve the comment. > /** Limits the rows in the result */ > Item *row_limit; > /** Skips a particular number of rows in from the result*/ Ok to push after this is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Is it okay that JSON_OBJECTAGG creates objects with duplicate elements?
by Sergey Petrunia 07 Jun '20

07 Jun '20

Hi Alexey, Consider this example: create table t20 (a varchar(10), b int); insert into t20 values ('red', 1), ('blue', 2), ('red',1001), ('blue', 1002); select json_objectagg(a,b) from t20; +----------------------------------------------+ | json_objectagg(a,b) | +----------------------------------------------+ | {"red":1, "blue":2, "red":1001, "blue":1002} | +----------------------------------------------+ the created object has multiple members with the same name. Is this expected or should be reported as a bug? In MySQL-8, the above example produces this: +-----------------------------+ | json_objectagg(a,b) | +-----------------------------+ | {"red": 1001, "blue": 1002} | +-----------------------------+ BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 0f91c62d332: MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE
by Sergei Golubchik 07 Jun '20

07 Jun '20

Hi, Sachin! Two questions: 1. This looks like there was no replication tests for periods of long uniques at all? Please add them (create an MDEV, like "replication tests of long unique") and add various relevant replication tests in 10.4. 2. It feels a bit risky to backport my changes to 10.4. At least you definitely should take the latest code from 10.5. Not cherry-pick one commit, but copy handler::prepare_for_insert() and other relevant code from 10.5. May be it'd be safer not to backport? What would your fix look like if you wouldn't have prepare_for_insert()? Also, a couple of comments to the code, see below. On Jun 03, Sachin wrote: > revision-id: 0f91c62d332 (mariadb-10.4.11-225-g0f91c62d332) > parent(s): c01b3d91690 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-02 08:24:24 +0530 > message: > > MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE > > Add missing call for handler->prepare_for_insert() in rows event. > > diff --git a/mysql-test/main/long_unique_bugs_replication.test b/mysql-test/main/long_unique_bugs_replication.test > new file mode 100644 > index 00000000000..5348be3e7d3 > --- /dev/null > +++ b/mysql-test/main/long_unique_bugs_replication.test > @@ -0,0 +1,22 @@ > +# > +# Long unique bugs related to master slave replication > +# > + > +# > +# MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE > +# > + > +--source include/master-slave.inc > +--source include/have_binlog_format_row.inc master-slave.inc should always be included last, after all have_xxx.inc files that may cause the test to be skipped. > + > +create table t1 (i1 int, a1 text, unique key i1 (a1)) engine=myisam; > +insert into t1 values (1,1); > +update t1 set a1 = 'd' limit 1; > + > +sync_slave_with_master; > +connection slave; > + > +connection master; > +drop table t1; > + > +--source include/rpl_end.inc > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 3707f73a716..f4c0c546a38 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -11568,6 +11568,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > { > master_had_triggers= table->master_had_triggers; > bool transactional_table= table->file->has_transactions(); > + table->file->prepare_for_insert(); > /* > table == NULL means that this table should not be replicated > (this was set up by Table_map_log_event::do_apply_event() > @@ -14344,6 +14345,7 @@ int Rows_log_event::find_row(rpl_group_info *rgi) > issue_long_find_row_warning(get_general_type_code(), m_table->alias.c_ptr(), > is_index_scan, rgi); > table->default_column_bitmaps(); > + table->file->prepare_for_insert(); Why do you need it here, I'd think that do_apply_event() should be enough? > DBUG_RETURN(error); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] MDEV-16230: Server crashes when Analyze format=json is run with a window function with empty PARTITION BY and ORDER BY clauses
by Sergey Petrunia 03 Jun '20

03 Jun '20

Hi Varun, * Please limit the commit comment to ~80 chars wide. * The .result file has r_total_time_ms. Use analyze-format.inc to avoid this. Ok to push after this is addressed. > commit 9e755e21799695b9a837a240a86ba5ef863ae7b3 > Author: Varun Gupta <varun.gupta(a)mariadb.com> > Date: Tue Jun 2 16:31:53 2020 +0530 > > MDEV-16230: Server crashes when Analyze format=json is run with a window function with empty PARTITION BY and ORDER BY clauses > > Currently when both PARTITION BY and ORDER BY clauses are empty then we create a Item with the first field > in the select list and sort with that field. It should be created as an Item_temptable_field instead of > Item_field because the print() function continues to work even if the table has been dropped. > ... BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] e14dcfe0bc7: MDEV-22715: SIGSEGV in radixsort_for_str_ptr and in native_compare/my_qsort2 (optimized builds)
by Sergey Petrunia 03 Jun '20

03 Jun '20

Hi Varun, Please add a comment above the declaration of Field::sort_string() that some implementations actually may write up to 8 bytes regardless of what size was requested. Ok to push after that. > revision-id: e14dcfe0bc743aa6b4ed2b3c4f7e9314aa01d6eb (mariadb-10.1.43-158-ge14dcfe0bc7) > parent(s): db1f73b26edaace41345f19bca507a7848aa39a8 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-05-29 21:13:38 +0530 > message: > > MDEV-22715: SIGSEGV in radixsort_for_str_ptr and in native_compare/my_qsort2 (optimized builds) > > For DECIMAL[(M[,D])] datatype max_sort_length was not being honoured which was leading to buffer > overflow while making the sort key. The fix to this problem would be to create sort keys for decimals > with atmost max_sort_key bytes > > Important: > The minimum value of max_sort_length has been raised to 8 (previously was 4), > so fixed size datatypes like DOUBLE and BIGINIT are not truncated for > lower values of max_sort_length. > ... BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 7df57cc8206: MDEV-22719 Long unique keys are not created when individual key_part->length < max_key_length but SUM(key_parts->length) > max_key_length
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Sachin! ok to push On Jun 03, Sachin wrote: > revision-id: 7df57cc8206 (mariadb-10.4.11-226-g7df57cc8206) > parent(s): 0f91c62d332 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-03 13:47:39 +0530 > message: > > MDEV-22719 Long unique keys are not created when individual key_part->length < max_key_length but SUM(key_parts->length) > max_key_length > > Make UNIQUE HASH key in case when key_info->key_length > max_key_length > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1e9ea675a18: MDEV-21282 Assertion 'mariadb_table' failed in gcol.innodb_virtual_debug_purge
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Thirunarayanan! Sure, it's purely InnoDB functionality. If you say that purge thread already hold MDL for the table, then open_purge_table() doesn't need to. Ok to push as you see fit. On Jun 03, Thirunarayanan Balathandayuthapani wrote: > revision-id: 1e9ea675a18 (mariadb-10.5.2-304-g1e9ea675a18) > parent(s): f22093ad392 > author: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > committer: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > timestamp: 2020-06-01 18:02:17 +0530 > message: > > MDEV-21282 Assertion 'mariadb_table' failed in gcol.innodb_virtual_debug_purge > > - commit ea37b144094a0c2ebfc6774047fd473c1b2a8658 (MDEV-16678) caused > a regression. when purge thread tries to open the table for virtual > column computation, there is no need to acquire MDL for the table. > Because purge thread already hold MDL for the table Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c01b3d91690: MDEV-22551 Server crashes in skip_trailing_space upon ANALYZE on table with long unique, or Assertion `marked_for_read()' fails
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Sachin! On Jun 03, Sachin wrote: > revision-id: c01b3d91690 (mariadb-10.4.11-224-gc01b3d91690) > parent(s): 0552ed86782 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-02 08:24:13 +0530 > message: > > MDEV-22551 Server crashes in skip_trailing_space upon ANALYZE on table with long unique, or Assertion `marked_for_read()' fails > > If Blob(or parent) field read_set is not set then Do not calc hash in > that case. No, I think this is wrong. The problem, again, is that read_set bits are set incorrectly, and the fix is, again, to set them correctly, not to perform more read_set checks at run time. Think about what's happening here. The server tries to calculate a generated column value (because gcol is present in read_set) when not all argument columns were read (because they are not in read_set). This is an impossible situation, a gcol can _only_ be in read_set if all its dependencies are also there, you've seen in TABLE::mark_virtual_columns_for_write() that for every marked gcol all dependencies are also recursively marked. So in this case ANALYZE breaks this rule and should be fixed not to do it. There are two possible fixes - either ANALYZE should mark dependencies for all gcols (marking blobs in this bug) or ANALYZE should skip and not mark gcols. Better ask somebody who's more famiiar with EITS about what fix is preferred. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e2017fc3a5f: MDEV-22441 implement a generic way to change a value of a variable in scope
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Eugene! On Jun 03, Eugene Kosov wrote: > revision-id: e2017fc3a5f (mariadb-10.2.31-219-ge2017fc3a5f) This should go into 10.5, not into old-old GA 10.2. Besides, 10.2 cannot use C++11 features, 10.5 can. > parent(s): 1b3adaab25c > author: Eugene Kosov <claprix(a)yandex.ru> > committer: Eugene Kosov <eugene.kosov(a)mariadb.com> > timestamp: 2020-05-27 12:45:41 +0300 > message: > > MDEV-22441 implement a generic way to change a value of a variable in scope > > Scope_value<T> class changes a variable value to some temporary value while > it exits and restores an old value in it's destructor. > > make_scope_value() is a function targeted for C++11 which allows to omit > a template parameter which is required by Scope_value<T> I think I'd name it backup_and_change_scoped() or something like that. But it's just a suggestion, feel free to keep make_scope_value() name if you like it better. > Sql_mode_save removed and the only usage is replaced with Scope_value<T> > > Sql_mode_save removal should not be merged in newer branches! > > --- > include/scope.h | 46 ++++++++++++++++++++++++++++++++++++++++++++++ > sql/sql_acl.cc | 7 ++++--- > sql/sql_class.h | 16 ---------------- > 3 files changed, 50 insertions(+), 19 deletions(-) > > diff --git a/include/scope.h b/include/scope.h > new file mode 100644 > index 00000000000..81061ceed37 > --- /dev/null > +++ b/include/scope.h > @@ -0,0 +1,46 @@ > +/* > + Copyright (c) 2020, MariaDB Corporation. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA */ > + > +#pragma once use include guards, please, not a pragma > + > +namespace maria and, as you remember, we didn't quite decide on a global server-wide namespace, so it's not the MDEV that should introduce it. > +{ > + > +template <typename T> class Scope_value > +{ > +public: > + Scope_value(T &variable, const T &scope_value) > + : variable_(variable), saved_value_(variable) > + { > + variable= scope_value; > + } > + > + ~Scope_value() { variable_= saved_value_; } > + > +private: > + T &variable_; > + T saved_value_; > +}; > + > +// Starting from C++11 this allows to omit template argument like this: > +// auto _ = make_scope_value(var, tmp_value); > +template <typename T> > +Scope_value<T> make_scope_value(T &variable, const T &scope_value) > +{ > + return Scope_value<T>(variable, scope_value); > +} as you've seen from Sql_mode_save usage, sometimes the value is assigned later, not when it's saved. Please, add another helper that doesn't require to provide a value. Like template <typename T> Scope_value<T> backup_scoped(T &variable) { return Scope_value<T>(variable, variable); } > + > +} // namespace maria Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 0552ed86782: MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Sachin! ok to push with one change, see below. On Jun 03, Sachin wrote: > revision-id: 0552ed86782 (mariadb-10.4.11-223-g0552ed86782) > parent(s): 61b2cd38d48 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-02 08:24:02 +0530 > message: > > MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB > > Problem:- Calling mark_columns_per_binlog_row_image() earlier may change the > result of mark_virtual_columns_for_write() , Since it can set the bitmap on > for virtual column, and henceforth mark_virtual_column_deps(field) will > never be called in mark_virtual_column_with_deps. > > This bug is not specific for long unique, It also fails for this case > create table t2(id int primary key, a blob, b varchar(20) as (LEFT(a,2))); > > diff --git a/sql/table.cc b/sql/table.cc > index bc8ddbc0f9d..8edbbe8d599 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -7040,7 +7040,6 @@ void TABLE::mark_columns_needed_for_update() > DBUG_ENTER("TABLE::mark_columns_needed_for_update"); > bool need_signal= false; > > - mark_columns_per_binlog_row_image(); > > if (triggers) > triggers->mark_fields_used(TRG_EVENT_UPDATE); > @@ -7048,6 +7047,7 @@ void TABLE::mark_columns_needed_for_update() > mark_default_fields_for_write(FALSE); > if (vfield) > need_signal|= mark_virtual_columns_for_write(FALSE); > + mark_columns_per_binlog_row_image(); I think that's too early. In that if() below you can see > if (file->ha_table_flags() & HA_REQUIRES_KEY_COLUMNS_FOR_DELETE) > { > KEY *end= key_info + s->keys; ... if (bitmap_fast_test_and_set(read_set, idx)) continue; if (field[idx]->vcol_info) mark_virtual_col(field[idx]); and mark_columns_per_binlog_row_image() might break that. Better to put mark_columns_per_binlog_row_image() at the very end, just before signalling. > @@ -7132,7 +7132,6 @@ void TABLE::mark_columns_needed_for_update() > void TABLE::mark_columns_needed_for_insert() > { > DBUG_ENTER("mark_columns_needed_for_insert"); > - mark_columns_per_binlog_row_image(); > > if (triggers) > { > @@ -7152,6 +7151,7 @@ void TABLE::mark_columns_needed_for_insert() > /* Mark virtual columns for insert */ > if (vfield) > mark_virtual_columns_for_write(TRUE); > + mark_columns_per_binlog_row_image(); this looks ok. > if (check_constraints) > mark_check_constraint_columns_for_read(); > DBUG_VOID_RETURN; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Progress report - Week 4
by Mohammed Hammaad Mateen 02 Jun '20

02 Jun '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my fourth week [25-31 May] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. The tasks taken up for this week was to study and analyze *INSERT SELECT RETURNING* and developing a test suite for *MULTIPLE TABLE DELETE RETURNING.* The following attachment contains the test cases for evaluating Multiple Delete Returning, as they are developed before actual implementation of code, please do notify me of any improvement needed, I'd be glad to resolve. Regards, Mohammed Hammaad Mateen

2 1

Re: [Maria-developers] fab8349af86: MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB
by Sergei Golubchik 01 Jun '20

01 Jun '20

Hi, Sachin! On May 31, Sachin wrote: > revision-id: fab8349af86 (mariadb-10.4.11-223-gfab8349af86) > parent(s): 61b2cd38d48 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-05-29 11:56:45 +0530 > message: > > MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB > > Problem:- > In the case of BINLOG_ROW_IMAGE_NOBLOB table->readset is PKE + non blob fields > But while updating virtual fields we need to read the blob field, which will > generate assert failure. > Solution:- > If binlog_row_image == NOBLOB, then set read_bit for the blob field. > This bug is not specific for long unique, It also fails for this case > create table t2(id int primary key, a blob, b varchar(20) as (LEFT(a,2))); > > diff --git a/sql/field.cc b/sql/field.cc > index 0a8fdc3d3f5..bcad056e76a 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -8467,6 +8467,10 @@ longlong Field_blob::val_int(void) > String *Field_blob::val_str(String *val_buffer __attribute__((unused)), > String *val_ptr) > { > + ulong binlog_row_image= table->in_use->variables.binlog_row_image; > + if (table->read_set && !bitmap_is_set(table->read_set, field_index) && > + binlog_row_image == BINLOG_ROW_IMAGE_NOBLOB) > + bitmap_set_bit(table->read_set, field_index); > DBUG_ASSERT(marked_for_read()); > char *blob; > memcpy(&blob, ptr+packlength, sizeof(char*)); A couple of red flags here. First, you don't fix BINLOG_ROW_IMAGE_MINIMAL. And generally you should not modify read/write sets from val_str, if should be done earlier before the execution phase. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Review for: MDEV-17399 Add support for JSON_TABLE, part #5
by Sergey Petrunia 26 May '20

26 May '20

Hi Alexey, On Fri, May 15, 2020 at 04:26:02PM +0400, Alexey Botchkov wrote: > See the next iteration here > https://github.com/MariaDB/server/commit/692cb566096d61b240ec26e84fc7d3c7d1… > So the ha_json_table;:position() was implemented. > The size of the reference depends on the nested path depth - each level > adds some 9 bytes to the initial 5. > That can be decreased but i decided to keep it simple initially and i doubt > we're going to have really deep > nesting in realistic scenarios. This is ok. I'm not sure if the server has a limit on the rowid size. Perhaps there is, and in that case we just need to limit the depth we allow. > And i'd like to ask you for some testing > queries here. Or just the model how > to produce queries that will be using the ::position() extensively. I've provided a testcase for the position() call in my previous email. https://gist.github.com/spetrunia/a905d51731c58f5439bd9f70c64cdc43 As for rnd_pos(), one case that I'm aware of is when the query does a filesort, and the row being sorted either includes a blob, or has the total length exceeding certain limit. I've tried constructing an example with blobs, and it fails with an assertion before the execution reaches rnd_pos() calls: select * from json_table('[{"color": "blue", "price": 50}, {"color": "red", "price": 100}, {"color": "rojo", "price": 10.0}, {"color": "blanco", "price": 11.0}]', '$[*]' columns( color varchar(100) path '$.color', price text path '$.price', seq for ordinality ) ) as T order by color desc; fails an assertion: mysqld: /home/psergey/dev-git/10.5-json/sql/field.cc:8309: virtual int Field_blob::store(const char*, size_t, CHARSET_INFO*): Assertion `marked_for_write_or_computed()' failed. Once that is fixed, this should use position() and rnd_pos() calls. This will likely expose more issues with rnd_pos(), see my comments to the Json_table_nested_path::set_position below. == A problem with VIEWs == mysql> select * -> from -> json_table( -> '[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]', -> '$[*]' columns( color varchar(100) path '$.color')) as `T_A` where T_A.color<>'azul' ; +-------+ | color | +-------+ | red | | black | +-------+ 2 rows in set (6.34 sec) Good so far. Now, let's try creating a VIEW from this: create view v1 as select * from json_table( '[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]', '$[*]' columns( color varchar(100) path '$.color')) as `T_A` where T_A.color<>'azul' ; select * from v1; ERROR 4041 (HY000): Unexpected end of JSON path in argument 2 to function 'JSON_TABLE' Examining the .frm file, I see something that looks like garbage data: query=select `T_A`.`color` AS `color` from JSON_TABLE(\'[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]\', \'\'\0<D8>7^A<9C><FF>^?\0\0\0\0\0\0\0\0\0\0\0\0^A<A5><A5><A5><A5><A5>^H<D0>kWUU\0\0[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]\0$[*]\'\' COLUMNS (`color` PATH \'^A<9C><FF>^?\0\0\0\0\0\0^A\0\0\0\0\0\0\0<A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5>\0\0\0\0<A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5> <A5><A5><A5><A5><A5><A5> D^A<9C><FF>^?\0\0H:^A<9C><FF>^?\0\0<A5><A5><A5><A5><A5><A5><A5><A5>100\0<8F><8F><8F><8F>$.color\')) T_A where `T_A`.`color` <> \'azul\' EXPLAIN EXTENDED ...; SHOW WARNINGS; - also print something odd. == EXPLAIN [FORMAT=JSON] == I think EXPLAIN output should provide an indication that a table function is used. MySQL does it like so: <TODO> == Unneeded recursive rules in grammar == I've already complained about this: the grammar has recursive ON EMPTY, ON ERROR rules, which cause the following to be accepted (note the two ON EMPTY clauses) : select * from json_table('[{"color": "blue", "price": 50},{"color": "red"}]', '$[*]' columns( price varchar(255) path '$.price' default 'xyz' on empty default 'abc' on empty ) ) as T; > commit 692cb566096d61b240ec26e84fc7d3c7d13f024c > Author: Alexey Botchkov <holyfoot(a)askmonty.org> > Date: Fri May 15 15:25:28 2020 +0400 > > MDEV-17399 Add support for JSON_TABLE. > > Syntax for JSON_TABLE added. > The ha_json_table handler added. Executing the JSON_TABLE we > create the temporary table of the ha_json_table, add dependencies > of other tables and sometimes conditions to WHERE. > and sometimes conditions to WHERE. I think this is not true anymore? ... > diff --git a/include/my_base.h b/include/my_base.h > index 7efa5eb9673..89ef3e8e7c1 100644 > --- a/include/my_base.h > +++ b/include/my_base.h > @@ -523,6 +523,13 @@ enum ha_base_keytype { > #define HA_ERR_TABLESPACE_MISSING 194 /* Missing Tablespace */ > #define HA_ERR_SEQUENCE_INVALID_DATA 195 > #define HA_ERR_SEQUENCE_RUN_OUT 196 > + > +/* > + Share the error code to not increment the HA_ERR_LAST for now, > + as it disturbs some storage engine's tests. > + Probably should be fixed later. > +*/ > +#define HA_ERR_INVALID_JSON HA_ERR_TABLE_IN_FK_CHECK We definitely can't have this in the final patch. We need to either: A. Use an engine-specific error code. Check out MyRocks and ha_rocksdb::get_error_message() for an example of how this is done B. Indeed introduce a generic "Invalid JSON" error code. I'm hesitant about B, let's discuss this with other developers. ... > diff --git a/sql/table_function.cc b/sql/table_function.cc > new file mode 100644 > index 00000000000..71f2378ce7d > --- /dev/null > +++ b/sql/table_function.cc ... > + > +class ha_json_table: public handler > +{ Please add comments about these > +protected: > + Table_function_json_table *m_jt; > + String m_tmps; > + String *m_js; > + uchar *m_cur_pos; > +public: > + ha_json_table(TABLE_SHARE *share_arg, Table_function_json_table *jt): > + handler(&table_function_hton.m_hton, share_arg), m_jt(jt) > + { > + /* > + set the mark_trx_read_write_done to avoid the > + handler::mark_trx_read_write_internal() call. > + It relies on &ha_thd()->ha_data[ht->slot].ha_info[0] to be set. > + But we don't set the ha_data for the ha_json_table, and > + that call makes no sence for ha_json_table. > + */ > + mark_trx_read_write_done= 1; > + ref_length= (jt->m_depth+1) * (4 + 1) + > + jt->m_depth * (sizeof(Json_table_nested_path *)); > + } > + ~ha_json_table() {} > + handler *clone(const char *name, MEM_ROOT *mem_root) { return NULL; } > + const char *index_type(uint inx) { return "NONE"; } > + /* Rows also use a fixed-size format */ > + enum row_type get_row_type() const { return ROW_TYPE_FIXED; } > + ulonglong table_flags() const > + { > + return (HA_FAST_KEY_READ | /*HA_NO_BLOBS |*/ HA_NULL_IN_KEY | > + HA_CAN_SQL_HANDLER | > + HA_REC_NOT_IN_SEQ | HA_NO_TRANSACTIONS | > + HA_HAS_RECORDS | HA_CAN_HASH_KEYS); > + } > + ulong index_flags(uint inx, uint part, bool all_parts) const > + { > + return HA_ONLY_WHOLE_INDEX | HA_KEY_SCAN_NOT_ROR; > + } > + uint max_supported_keys() const { return 1; } > + uint max_supported_key_part_length() const { return MAX_KEY_LENGTH; } > + double scan_time() { return 1000000.0; } > + double read_time(uint index, uint ranges, ha_rows rows) { return 0.0; } The above two functions are never called. Please * add a comment saying that. * add a DBUG_ASSERT() into them to back the point made by the comment :-) > + int open(const char *name, int mode, uint test_if_locked); > + int close(void) { return 0; } > + int rnd_init(bool scan); > + int rnd_next(uchar *buf); > + int rnd_pos(uchar * buf, uchar *pos); > + void position(const uchar *record); > + int can_continue_handler_scan() { return 1; } > + int info(uint); > + int extra(enum ha_extra_function operation); > + THR_LOCK_DATA **store_lock(THD *thd, THR_LOCK_DATA **to, > + enum thr_lock_type lock_type) > + { return NULL; } > + int create(const char *name, TABLE *form, HA_CREATE_INFO *create_info) > + { return 1; } > +private: > + void update_key_stats(); > +}; > + ... > +void Json_table_nested_path::set_position(const char *j_start, const uchar *pos) > +{ * This needs a comment. * I don't see where the value of m_ordinality_counter is restored? * The same about m_null - its value is not restored either? > + if (m_nested) > + { > + memcpy(&m_cur_nested, pos, sizeof(m_cur_nested)); > + pos+= sizeof(m_cur_nested); > + } > + > + m_engine.s.c_str= (uchar *) j_start + sint4korr(pos); > + m_engine.state= (int) pos[4]; > + if (m_cur_nested) > + m_cur_nested->set_position(j_start, pos); > +} > + ... > +int ha_json_table::info(uint) > +{ > + /* We don't want 0 or 1 in stats.records. */ > + stats.records= 4; > + return 0; Does this value matter? As far as I understand it doesn't, as the optimizer will use the estimates obtained from Table_function_json_table::get_estimates. Please add a comment about this. > +} > + ... Please document this function. What's last_column? Why does print() method get it and return it? > +int Json_table_nested_path::print(THD *thd, TABLE_LIST *sql_table, String *str, > + List_iterator_fast<Json_table_column> &it, > + Json_table_column **last_column) > +{ > + Json_table_nested_path *c_path= this; ... > diff --git a/sql/table_function.h b/sql/table_function.h > new file mode 100644 > index 00000000000..49d650cb7b2 > --- /dev/null > +++ b/sql/table_function.h > @@ -0,0 +1,168 @@ ... > +#include <json_lib.h> > + > +/* > + The Json_table_nested_path represents the 'current nesting' level > + for a set of JSON_TABLE columns. > + Each column (Json_table_column instance) is linked with corresponding > + 'nested path' object and gets it's piece of JSON to parse during the computation > + phase. > + The root 'nested_path' is always present as a part of Table_function_json_table, > + then other 'nested_paths' can be created and linked into a tree structure when new > + 'NESTED PATH' is met. The nested 'nested_paths' are linked with 'm_nested', the same-level > + 'nested_paths' are linked with 'm_next_nested'. > + So for instance > + JSON_TABLE( '...', '$[*]' > + COLUMNS( a INT PATH '$.a' , > + NESTED PATH '$.b[*]' COLUMNS (b INT PATH '$', > + NESTED PATH '$.c[*]' COLUMNS(x INT PATH '$')), > + NESTED PATH '$.n[*]' COLUMNS (z INT PAHT '$')) > + results in 4 'nested_path' created: > + root nested_b nested_c nested_n > + m_path '$[*]' '$.b[*]' '$.c[*]' '$.n[*] > + m_nested &nested_b &nested_c NULL NULL > + n_next_nested NULL &nested_n NULL NULL > + > +and 4 columns created: > + a b x z > + m_nest &root &nested_b &nested_c &nested_n > +*/ > + > + > +class Json_table_column; > + > +class Json_table_nested_path : public Sql_alloc > +{ > +public: > + bool m_null; > + json_path_t m_path; > + json_engine_t m_engine; > + json_path_t m_cur_path; > + > + /* Counts the rows produced. Value is set to the FOR ORDINALITY coluns */ > + longlong m_ordinality_counter; > + > + Json_table_nested_path *m_parent; > + Json_table_nested_path *m_nested, *m_next_nested; > + Json_table_nested_path **m_nested_hook; > + Json_table_nested_path *m_cur_nested; Please add documentation about the above members. I see the diagram above, but I think text descriptions are also needed for each member. What's m_nested_hook? > + Json_table_nested_path(Json_table_nested_path *parent_nest): > + m_parent(parent_nest), m_nested(0), m_next_nested(0), > + m_nested_hook(&m_nested) {} > + int set_path(THD *thd, const LEX_CSTRING &path); > + void scan_start(CHARSET_INFO *i_cs, const uchar *str, const uchar *end); > + int scan_next(); > + int print(THD *thd, TABLE_LIST *sql_table, String *str, > + List_iterator_fast<Json_table_column> &it, > + Json_table_column **last_column); > + void get_current_position(const char *j_start, uchar *pos) const; > + void set_position(const char *j_start, const uchar *pos); > +}; ... > +class Table_function_json_table : public Sql_alloc Please document the class and the data members. > +{ > +public: > + Item *m_json; > + Json_table_nested_path m_nested_path; > + List<Json_table_column> m_columns; > + table_map m_dep_tables; > + uint m_depth, m_cur_depth; > + > + Table_function_json_table(Item *json): m_json(json), m_nested_path(0), > + m_depth(0), m_cur_depth(0) {} > + > + /* > + Used in sql_yacc.yy. > + Represents the current NESTED PATH level being parsed. > + */ > + Json_table_nested_path *m_sql_nest; > + void add_nested(Json_table_nested_path *np); > + void leave_nested(); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 070df171c1e: MDEV-16937 Strict SQL with system versioned tables causes issues
by Sergei Golubchik 26 May '20

26 May '20

Hi, Aleksey! ok to push On May 26, Aleksey Midenkov wrote: > revision-id: 070df171c1e (mariadb-10.3.21-91-g070df171c1e) > parent(s): ad41da5c934 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-05-25 14:47:26 +0300 > message: > > MDEV-16937 Strict SQL with system versioned tables causes issues > > Respect system fields in NO_ZERO_DATE mode. > > This is the subject for refactoring in MDEV-19597 > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] cd9cab54aac: MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field
by Sergei Golubchik 26 May '20

26 May '20

Hi, Aleksey! Ok to push, thanks! On May 26, Aleksey Midenkov wrote: > revision-id: cd9cab54aac (mariadb-10.2.31-123-gcd9cab54aac) > parent(s): d275ecbd208 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-05-25 20:56:31 +0300 > message: > > MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field > > update_virtual_field() is called as part of index rebuild in > ha_myisam::repair() (MDEV-5800) which is done on bulk INSERT finish. > > Assertion in update_virtual_field() was put as part of MDEV-16222 > because update_virtual_field() returns in_use->is_error(). The idea: > wrongly mixed semantics of error status before update_virtual_field() > and the status returned by update_virtual_field(). The former can > falsely influence the latter. > > diff --git a/sql/table.cc b/sql/table.cc > index d6d86d96016..2429bb12abe 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -7707,15 +7707,17 @@ int TABLE::update_virtual_fields(handler *h, enum_vcol_update_mode update_mode) > > int TABLE::update_virtual_field(Field *vf) > { > - DBUG_ASSERT(!in_use->is_error()); > - Query_arena backup_arena; > DBUG_ENTER("TABLE::update_virtual_field"); > + Query_arena backup_arena; > + Counting_error_handler count_errors; > + in_use->push_internal_handler(&count_errors); > in_use->set_n_backup_active_arena(expr_arena, &backup_arena); > bitmap_clear_all(&tmp_set); > vf->vcol_info->expr->walk(&Item::update_vcol_processor, 0, &tmp_set); > vf->vcol_info->expr->save_in_field(vf, 0); > in_use->restore_active_arena(expr_arena, &backup_arena); > - DBUG_RETURN(in_use->is_error()); > + in_use->pop_internal_handler(); > + DBUG_RETURN(count_errors.errors); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 7d593466a22: MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field
by Sergei Golubchik 25 May '20

25 May '20

Hi, Aleksey! I don't see what you've changed. We've discussed that fix and that one isn't supposed to swap Diagnostics_area's like that. And in your new patch you do exactly the same. Possible correct approaches: * don't return in_use->is_error(), return the return value of vf->vcol_info->expr->walk() || vf->vcol_info->expr->save_in_field() This means that Item_field::update_vcol_processor() should also do the same, I suspect * Use thd->push_internal_handler() and Counting_error_handler. Or, better, Turn_errors_to_warnings_handler with counting. This is the simplest one. there's a third option: * always return 0, because, looking how it's used, I don't really see how update_virtual_field() can ever get an error. But it's not a particularly future-proof approach. And I just might be wrong about errors. On Apr 23, Aleksey Midenkov wrote: > revision-id: 7d593466a22 (mariadb-10.2.28-4-g7d593466a22) > parent(s): 7bc26de591c > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2019-11-07 10:45:21 +0300 > message: > > MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field > > Preserve and restore statement DA. > > update_virtual_field() is called as part of index rebuild in > ha_myisam::repair() (MDEV-5800) which is done on bulk INSERT finish. > > Assertion in update_virtual_field() was put as part of MDEV-16222 > because update_virtual_field() returns in_use->is_error(). The idea: > wrongly mixed semantics of error status before update_virtual_field() > and the status returned by update_virtual_field(). The former can > falsely influence the latter. > > Preserve global error status and run update_virtual_field() with clear > DA since no matter how SQL command is finished it must update the > index after bulk INSERT. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] Progress Report - Week 3
by Mohammed Hammaad Mateen 25 May '20

25 May '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my third week [ 18-24 May ] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. The tasks taken up for this week was to study and analyze: *INSERT.... RETURNING:* • *TableList* consists of 1 table instance to use, where each instance is a structure consisting of db argument, table name argument, alias argument and lock type argument. *Referenced this from the table.h file.* • In *sql_parse.cc* <http://sql_parse.cc> under *case SQLCOM_INSERT*: we do a DBUG_ASSERT where first_table == all_table && first_table!=0 // we check if only 1 table is used and the table used is already created. • if lex-> has_returning() we increment the system status var by 1 and perform analyze.. insert.. returning. • we compute result by mysql_insert function by passing the following arguments. - Thread Handler thd - all table instances - list of all the fields used - the values we want to insert - the list of update feilds - and update value list - duplicate flag - ignore - result of select • if Inserting fails due to some reason we equate result to send_explain(thd) • we also update the MYSQL_INSERT_DONE with result of insert and the the row count. *INSERT.. SELECT.. RETURNING:* • We fix the lock on first table. • lock other tables until command is written to the binary log. • the procedure is same as discussed earlier with respect to insert.... returning. • To switch to the second table we traverse from first_table to (->) next_local and we compute select result with the help of select insert function. • we can now unlock the tables and we also need to check if something changed after unlocking, of that happens we should invalidate the table from the query cache using query_cache_invalidate3 function. • Manual cleaning of select result obtained from select insert must be done. Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] MDEV-22461: JOIN::make_aggr_tables_info(): Assertion `select_options & (1ULL << 17)' failed.
by Sergey Petrunia 24 May '20

24 May '20

Hi Varun, - Please add a testcase to the testsuite - Please address a few cosmetic comments below - Ok to push after addressed. > commit 5e448b77a3812e65623c0a1214049322ace3aacf > Author: Varun Gupta <varun.gupta(a)mariadb.com> > Date: Tue May 5 20:44:43 2020 +0530 > > MDEV-22461: JOIN::make_aggr_tables_info(): Assertion `select_options & (1ULL << 17)' failed. > > A temporary table is needed for window function computation but if only a NAMED WINDOW SPEC > is used and there is no window function, then there is no need to create a temporary > table as there is no stage to compute WINDOW FUNCTION > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index c601946cfa0..0a1d0c2dbcc 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2014,11 +2014,16 @@ JOIN::optimize_inner() > } > > need_tmp= test_if_need_tmp_table(); > - //TODO this could probably go in test_if_need_tmp_table. > - if (this->select_lex->window_specs.elements > 0) { > - need_tmp= TRUE; > + > + /* > + If window functions are present then we can't have simple_order set to > + TRUE as the window function needs a temp table for compuatation. typo: compuatation. > + ORDER BY is computed after the window function computation is done, so > + the sort would be done on the temp table. > + */ > + if (this->select_lex->have_window_funcs()) Is there a need to have "this->select_lex"? I think not, please remove, as it only confuses the reader. > simple_order= FALSE; > - } > + > > /* > If the hint FORCE INDEX FOR ORDER BY/GROUP BY is used for the table > diff --git a/sql/sql_select.h b/sql/sql_select.h > index 0e011c9267a..7a892c1af89 100644 > --- a/sql/sql_select.h > +++ b/sql/sql_select.h > @@ -1645,7 +1645,8 @@ class JOIN :public Sql_alloc > ((select_distinct || !simple_order || !simple_group) || > (group_list && order) || > MY_TEST(select_options & OPTION_BUFFER_RESULT))) || > - (rollup.state != ROLLUP::STATE_NONE && select_distinct)); > + (rollup.state != ROLLUP::STATE_NONE && select_distinct) || > + select_lex->have_window_funcs()); Please amend the function comment above this code to cover the new condition as well. > } > bool choose_subquery_plan(table_map join_tables); > void get_partial_cost_and_fanout(int end_tab_idx, > BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] b22a28c2295: fixup! 3fe5cd5e1785e3e8de7add9977a1c2ddd403538b
by Michael Widenius 22 May '20

22 May '20

Hi! On Fri, May 22, 2020 at 3:27 PM Andrei Elkin <andrei.elkin(a)mariadb.com> wrote: <cut> CORNER CASES: read-only, pure myisam, binlog-*, @@skip_log_bin, etc > > Aria just makes yet another previously unknown use case of an engine > that produces THD::ha_info but does not support 2pc, which the assert > implied. > > To explain more, the original block > > #ifndef DBUG_OFF > for (ha_info= thd->transaction.all.ha_list; rw_count > 1 && ha_info; > ha_info= ha_info->next()) > DBUG_ASSERT(ha_info->ht() != binlog_hton); > #endif > > claims there most be no binlog hton in a transaction consisting of more > than 1 hton:s, *when* (at this point) this transaction has not been > binlogged yet. > So combination of binlog + Innodb hton would be raise the assert, to > question "why the heck binlogging has not been done yet?!". > > Aria is different from Innodb in this context in that binlogging > was done at the end of the statement, so to miss > `cache_mngr->need_unlog' flagging (which is at xa prepare time logging). Thanks for the explanation, we should have had that in the code. > I think we should fix the assert rather than to remove. This way: > > cat > assert.diff <<. > diff --git a/sql/log.cc b/sql/log.cc > index 792c6bb1a99..aaf1fae1cd6 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -10124,6 +10124,16 @@ int TC_LOG_BINLOG::unlog_xa_prepare(THD *thd, bool all) > Ha_trx_info *ha_info; > uint rw_count= ha_count_rw_all(thd, &ha_info); > bool rc= false; > +#ifndef DBUG_OFF > + bool no_binlog= true, exist_no_2pc= false; > + for (ha_info= thd->transaction->all.ha_list; rw_count > 1 && ha_info; > + ha_info= ha_info->next()) > + { > + no_binlog= no_binlog && ha_info->ht() != binlog_hton; > + exist_no_2pc= exist_no_2pc || !ha_info->ht()->prepare; > + } > + DBUG_ASSERT(no_binlog || exist_no_2pc); > +#endif > > if (rw_count > 0) > { > . > > I tested it briefly with running XA:s on combination of engines > including. On which tree did you test? bb-10.5-aria? In the end, after discussions on slack, we ended with: #ifndef DBUG_OFF if (rw_count > 1) { /* There must be no binlog_hton used in a transaction consisting of more than 1 engine, *when* (at this point) this transaction has not been binlogged. The one exception is if there is an engine without a prepare method, as in this case the engine doesn't support XA and we have to ignore this check. */ bool binlog= false, exist_hton_without_prepare= false; for (ha_info= thd->transaction->all.ha_list; ha_info; ha_info= ha_info->next()) { if (ha_info->ht() == binlog_hton) binlog= true; if (!ha_info->ht()->prepare) exist_hton_without_prepare= true; } DBUG_ASSERT(!binlog || exist_hton_without_prepare); } #endif > > The reason it was not hit > > is that before Aria was not treated as transactional we could only > > come here in case of errors > > and that code was was apparently not tested, at least with binary logging on. > > > > > With Aria we could come here in case of rollback and we got assert for > > cases that was perfectly ok. > > Well, what 'rollback' do you mean? The function is invoked only > for ha_prepare(). As far as I remember, I come into this code in some edge case when something failed that normally never fails. I think the issue was that Aria doesn't have a prepare handler, which caused issues in ha_prepare(), but not sure. Anyway, we now have a solution for this. Regards, Monty

1 0