July 2010 - developers - lists.mariadb.org

Re: [Maria-developers] Repeated Error in accept: Bad file descriptor
by Paul McCullagh 16 Jul '10

16 Jul '10

Hi All, Yes, PBXT uses a lot of file handles: generally 3 per table. It opens all tables before recovery. Unfortunately, I have not worked on optimizing this behavior. Although the only reason for this is to resolve possible FK/PK relationships. So I think this should be considered a PBXT bug, and the problem should be fixed. Anyway this will certainly be the problem in this case. As Kristian says, select() is severely limited in that it only works with the low numbered file handles. To get the server up and running again you can just delete the tables manually (the table files: .frm, .xtd, .xtr, .xti), when MySQL is not running. PBXT will complain on startup, but should recover anyway. Best regards, Paul On Jul 15, 2010, at 9:48 AM, Michael Widenius wrote: > > Hi! > >>>>>> "Time" == Time Less <timelessness(a)gmail.com> writes: > >>> I think it is very likely that you are hitting this bug: >>> >>> http://bugs.mysql.com/bug.php?id=48929 >>> > > Time> Ah, yes, could be. Though my strace was different than this > bug shows. > > >>> The problem is that MySQL/MariaDB is using select() to accept new >>> connections. But select() has a hard-coded limit of 1024 on the >>> max number >>> of >>> open files it can support. It seems PBXT uses an open file >>> descriptor per >>> table, > > > Time> Not two per table? It has it looks like many log files, then > also a data and > Time> index file per table. On my system where I'm trying to use > 1,000 tables, I > Time> expect about 2,000+[mumble] file handles. > >>> Maybe MariaDB should backport the fix, it is actually a buffer >>> overflow >>> (though it is hard to see how it could be exploitable), but >>> perhaps more >>> relevant it is a rather nasty state for the server to get into, >>> and not >>> really >>> clear how to get it out of it again :-(. >>> > > Time> You can't get out of the state again. The server won't accept > connections, > Time> so you can't drop any tables. You just have to wipe the DB and > start over. > Time> If PBXT is resilient to its tables disappearing between server > restarts, you > Time> could rm files from the data directory. > > I didn't know that PBXT would open up all files at startup. If that's > the case, we should switch to use poll in MariaDB 5.3 ASAP (and > provide a patch for those that wants it for MariaDB 5.1). > > Paul, can you verify the above is the case for PBXT (ie, that PBXT > uses one file descriptor per table and don't free these at all?) > > Regards, > Monty > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-developers > Post to : maria-developers(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-developers > More help : https://help.launchpad.net/ListHelp

1 0

[Maria-developers] Rev 2801: Fixed an error in the creation of REF access method for materialized in file:///home/tsk/mprog/src/5.3/
by timour＠askmonty.org 15 Jul '10

15 Jul '10

At file:///home/tsk/mprog/src/5.3/ ------------------------------------------------------------ revno: 2801 revision-id: timour(a)askmonty.org-20100715135910-y1gvcc3d63sod6xt parent: sanja(a)askmonty.org-20100710103730-ayy6a61pdibspf4o committer: timour(a)askmonty.org branch nick: 5.3 timestamp: Thu 2010-07-15 16:59:10 +0300 message: Fixed an error in the creation of REF access method for materialized subquery execution, where the the REF buffer format was mistaken to be in record format instead of key format. The error was that the null byte for all fields of the record was in the front of the buffer, and not before each field data. === modified file 'sql/item_subselect.cc' --- a/sql/item_subselect.cc 2010-07-10 10:37:30 +0000 +++ b/sql/item_subselect.cc 2010-07-15 13:59:10 +0000 @@ -3957,8 +3957,6 @@ Item_iterator_row it(item_in->left_expr); /* The only index on the temporary table. */ KEY *tmp_key= tmp_table->key_info; - /* Number of keyparts in tmp_key. */ - uint tmp_key_parts= tmp_key->key_parts; JOIN_TAB *tab; DBUG_ENTER("subselect_hash_sj_engine::make_unique_engine"); === modified file 'sql/sql_select.cc' --- a/sql/sql_select.cc 2010-07-10 10:37:30 +0000 +++ b/sql/sql_select.cc 2010-07-15 13:59:10 +0000 @@ -7925,7 +7925,7 @@ use that information instead. */ cur_ref_buff + null_count, - null_count ? key_buff : 0, + null_count ? cur_ref_buff : 0, cur_key_part->length, items[i], value); cur_ref_buff+= cur_key_part->store_length; }

1 0

Re: [Maria-developers] Repeated Error in accept: Bad file descriptor
by Michael Widenius 15 Jul '10

15 Jul '10

Hi! >>>>> "Time" == Time Less <timelessness(a)gmail.com> writes: >> I think it is very likely that you are hitting this bug: >> >> http://bugs.mysql.com/bug.php?id=48929 >> Time> Ah, yes, could be. Though my strace was different than this bug shows. >> The problem is that MySQL/MariaDB is using select() to accept new >> connections. But select() has a hard-coded limit of 1024 on the max number >> of >> open files it can support. It seems PBXT uses an open file descriptor per >> table, Time> Not two per table? It has it looks like many log files, then also a data and Time> index file per table. On my system where I'm trying to use 1,000 tables, I Time> expect about 2,000+[mumble] file handles. >> Maybe MariaDB should backport the fix, it is actually a buffer overflow >> (though it is hard to see how it could be exploitable), but perhaps more >> relevant it is a rather nasty state for the server to get into, and not >> really >> clear how to get it out of it again :-(. >> Time> You can't get out of the state again. The server won't accept connections, Time> so you can't drop any tables. You just have to wipe the DB and start over. Time> If PBXT is resilient to its tables disappearing between server restarts, you Time> could rm files from the data directory. I didn't know that PBXT would open up all files at startup. If that's the case, we should switch to use poll in MariaDB 5.3 ASAP (and provide a patch for those that wants it for MariaDB 5.1). Paul, can you verify the above is the case for PBXT (ie, that PBXT uses one file descriptor per table and don't free these at all?) Regards, Monty

2 1

Re: [Maria-developers] Repeated Error in accept: Bad file descriptor
by Kristian Nielsen 14 Jul '10

14 Jul '10

Time Less <timelessness(a)gmail.com> writes: >> The problem is that MySQL/MariaDB is using select() to accept new >> connections. But select() has a hard-coded limit of 1024 on the max number >> of >> open files it can support. It seems PBXT uses an open file descriptor per >> table, > > > Not two per table? It has it looks like many log files, then also a data and > index file per table. On my system where I'm trying to use 1,000 tables, I > expect about 2,000+[mumble] file handles. Sure could be, I don't know the details, just that it seems to open >1000 files simultaneously. > You can't get out of the state again. The server won't accept connections, > so you can't drop any tables. You just have to wipe the DB and start over. > If PBXT is resilient to its tables disappearing between server restarts, you > could rm files from the data directory. One way is to use the --bootstrap parameter to mysqld (after stopping the server). This allows to start the server, run a set of commands, and shut down, without needing to connect. Something like (echo "DROP TABLE t1;" ; echo "DROP TABLE t2;"; ...) | mysqld --defaults-file=/etc/my.cnf --bootstrap (I actually tried this). One way or the other, it's a nasty bug. - Kristian.

1 0

Re: [Maria-developers] Repeated Error in accept: Bad file descriptor
by Michael Widenius 14 Jul '10

14 Jul '10

Hi! I am resending this to maria-developers, as this is a more appropriate address for PBXT issues. >>>>> "Time" == Time Less <timelessness(a)gmail.com> writes: Time> Just thought I'd try out PBXT. I created a database instance with 10 Time> databases, inside each 100 tables. They were MyISAM tables. Then I did an Time> "alter table <name> engine=pbxt" across all of them. That worked fine. Then Time> I restarted MySQL, and now it won't come up. Error logs are filled with Time> thousands of: Time> # tail /var/log/mysql.err Time> 100713 12:33:33 [ERROR] Error in accept: Bad file descriptor Time> 100713 12:33:33 [ERROR] Error in accept: Bad file descriptor Time> 100713 12:33:33 [ERROR] Error in accept: Bad file descriptor Time> To the tune of tens (or hundreds?) per second. I've tried restarting mysqld Time> with *ulimit -n 32768* and with my.cnf setting *open_files_limit = 32768* to Time> no effect. Does anyone have experience with this? Extensive Google searches Time> turn up nothing except esoteric OpenBSD problems or such. Time> MariaDB is MariaDB-server-5.1.44-75.el5. Sorry, but this is an issue that I haven't heard about before. However, I am sure that the PBXT team will have some suggestions for you how to solve this. Regards, Monty

2 1

Re: [Maria-developers] [Fwd: [Commits] bzr commit into Mariadb 5.2, with Maria 2.0:maria/5.2 branch (igor:2822) Bug#603654]
by Sergey Petrunya 13 Jul '10

13 Jul '10

Hello Igor, Ok to push. On Mon, Jul 12, 2010 at 07:08:58PM -0700, Igor Babaev wrote: > Please review this patch for the 5.2 tree. > > Regards, > Igor. > > > -------- Original Message -------- > Subject: [Commits] bzr commit into Mariadb 5.2, with Maria 2.0:maria/5.2 > branch (igor:2822) Bug#603654 > Date: Mon, 12 Jul 2010 19:05:37 -0700 (PDT) > From: Igor Babaev <igor(a)askmonty.org> > Reply-To: maria-developers(a)lists.launchpad.net > To: commits(a)mariadb.org > > #At lp:maria/5.2 based on > revid:igor@askmonty.org-20100713012307-rnom77fx57ef900o > > 2822 Igor Babaev 2010-07-12 > Fixed bug #603654. > If a virtual column was used in the ORDER BY clause of a query > and some of the columns this virtual column was based upon were > not referred anywhere in the query then the execution of the > query could cause an assertion failure. > It happened because in this case the bitmap of the columns used > for ordering keys was not formed correctly. > modified: > mysql-test/suite/vcol/r/vcol_misc.result > mysql-test/suite/vcol/t/vcol_misc.test > sql/filesort.cc > > === modified file 'mysql-test/suite/vcol/r/vcol_misc.result' > --- a/mysql-test/suite/vcol/r/vcol_misc.result 2010-07-13 01:23:07 +0000 > +++ b/mysql-test/suite/vcol/r/vcol_misc.result 2010-07-13 02:05:28 +0000 > @@ -35,3 +35,13 @@ a int NOT NULL DEFAULT '0', > v double AS ((1, a)) VIRTUAL > ); > ERROR HY000: Expression for computed column cannot return a row > +CREATE TABLE t1 ( > +a CHAR(255) BINARY NOT NULL DEFAULT 0, > +b CHAR(255) BINARY NOT NULL DEFAULT 0, > +v CHAR(255) BINARY AS (CONCAT(a,b)) VIRTUAL ); > +INSERT INTO t1(a,b) VALUES ('4','7'), ('4','6'); > +SELECT 1 AS C FROM t1 ORDER BY v; > +C > +1 > +1 > +DROP TABLE t1; > > === modified file 'mysql-test/suite/vcol/t/vcol_misc.test' > --- a/mysql-test/suite/vcol/t/vcol_misc.test 2010-07-13 01:23:07 +0000 > +++ b/mysql-test/suite/vcol/t/vcol_misc.test 2010-07-13 02:05:28 +0000 > @@ -30,6 +30,19 @@ CREATE TABLE t1 ( > v double AS ((1, a)) VIRTUAL > ); > > +# > +# Bug#603654: Virtual column in ORDER BY, no other references of table > columns > +# > + > +CREATE TABLE t1 ( > + a CHAR(255) BINARY NOT NULL DEFAULT 0, > + b CHAR(255) BINARY NOT NULL DEFAULT 0, > + v CHAR(255) BINARY AS (CONCAT(a,b)) VIRTUAL ); > +INSERT INTO t1(a,b) VALUES ('4','7'), ('4','6'); > +SELECT 1 AS C FROM t1 ORDER BY v; > + > +DROP TABLE t1; > + > > > > > === modified file 'sql/filesort.cc' > --- a/sql/filesort.cc 2010-06-01 19:52:20 +0000 > +++ b/sql/filesort.cc 2010-07-13 02:05:28 +0000 > @@ -1009,7 +1009,14 @@ static void register_used_fields(SORTPAR > if ((field= sort_field->field)) > { > if (field->table == table) > - bitmap_set_bit(bitmap, field->field_index); > + { > + if (field->vcol_info) > + { > + Item *vcol_item= field->vcol_info->expr_item; > + vcol_item->walk(&Item::register_field_in_read_map, 1, (uchar > *) 0); > + } > + bitmap_set_bit(bitmap, field->field_index); > + } > } > else > { // Item > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergey -- Sergey Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Separate builddir from src dir with cmake
by Bo Thorsen 12 Jul '10

12 Jul '10

Hi people, Our cmake scripts weren't able to build outside the source dir. This is bad for 3 reasons: - it's supposed to work - cmake recommends building outside the src dir - on Windows, cmake can't build 32 and 64 bit in the same solution, so patches has to be manually copied to test compilation on the other The attached patch fixes all the issues. I ran a find on the source dir before and after the the build, and they were identical. I didn't actually check that no existing files we're modified, but bzr di claim this isn't the case. I didn't port the zip creation script, but that shouldn't be a problem. src != blddir is more a development thing, and the buildbot slave producing the zip file will just continue to build in the same dir. The cpack installer builds just fine. OK to push to 5.1? Should I send this patch to the Oracle as well to minimize the drift between the cmake files? Bo Thorsen. Monty Program AB. -- MariaDB: MySQL replacement Community developed. Feature enhanced. Backward compatible.

1 0

[Maria-developers] SphinxSE pushed into 5.2
by Kristian Nielsen 09 Jul '10

09 Jul '10

Hi, I have pushed SphinxSE into MariaDB 5.2. It should be available in the upcoming MariaDB 5.2.2, which is currently planned within a few weeks. For now, only the code for the storage engine is pushed, not for the mysql-test-run.pl stuff. Sergei Golubchik wanted to modify mysql-test-run.pl first so that engine-specific code can be added in a more modular fashion (he will do it when he returns from vacation). For now, the engine is by default built as a loadable .so plugin. This is the general policy for MariaDB for new plugins. I think the idea is that engines that prove to be stable and well supported by their maintainer can get promoted to build statically by default (we are still flexing out the new policy). Note that the engine will still be available by default, the user will just need to do a one-time INSTALL PLUGIN command. Note also that SphinxSE can still be built statically with ./configure --with-plugin-sphinx, only the default is dynamic. Daniel, you should probably look into documenting the SphinxSE for MariaDB 5.2.2. Andrew kindly gave us permission to use what we need from the Sphinx documentation: http://sphinxsearch.com/docs/current.html#sphinxse Andrew suggested including also a link to the Sphinx webpage in the MariaDB documentation for SphinxSE, which is of course a very good idea. Also you should document how to do the INSTALL PLUGIN command (it's similar to for OQGraph). Note that the sphinx plugin will be included in eg. deb packages and so on and installed by default, just the INSTALL PLUGIN command is needed to enable it. Andrew, thanks for all your work with this! It is very good to have SphinxSE included, there have been several requests for this. We (mainly Sergei) made some changes to the code during review. I attach a patch for all the changes since your original import, in case you are interested. Note that I re-added the code when I merged to 5.2 to get a clean history, so future work on SphinxSE in MariaDB should be based on lp:maria/5.2, not our old sphinx bzr trees. - Kristian.

1 0

[Maria-developers] Get rid of most 32 bit Visual Studio compiler warnings
by Bo Thorsen 08 Jul '10

08 Jul '10

The attached patch reduces the warnings in a 32 bit build with Visual Studio 2008 to a set in the flex/bison generated code. I'll handle those later. I'll add comments on some of the changes here. The libmysqld.def file: The description line isn't supported in the more recent compilers. In fsp0fsp.c, there is an explicit cast to ullint. In this case, the code does what's intended, and the compiler warning is one of the "are you sure this is right" warnings. C4334 gives a warning if you make a 1u << 10 and store that in a 64 bit variable, because you could have meant 1i64 << 10. The change in i_s.cc is the one I'm most worried about. It used to store the unsigned long long in a double. The change I did can be wrong. But even if it's not, I'm worried what happens when this runs on an existing set of tables. We had a very long discussion about the cast in row0sel.c. The conclusion was that * auto_increment on doubles or floats is a very odd case * there is a compiler bug in Visual Studio for very large values (above around 2^53 it converts doubles wrong to uint64 I'd like to separate this discussion from this patch, though, and submit a bug report on the innodb code on mysql. I assume there will be something here I should change, but if parts of the patch are ready for pushing, let me know that. Bo Thorsen. Monty Program AB. -- MariaDB: MySQL replacement Community developed. Feature enhanced. Backward compatible.

1 1

[Maria-developers] How MySQL/MariaDB handles aborted DDL
by Arjen Lentz 07 Jul '10

07 Jul '10

Hi Philip, all Philip's bug on recovery from a kill -9 during DML reminded me... from my experience, aborting a DDL query (ALTER TABLE - things like adding an index usually) can be tricky and more often than not causes hassles. By abort in this case I mean killing the connection/thread that does the DDL. It would be great to somehow have a few test cases in the suite and it needs to be tested with all storage engines. Provided the table it operates on is big enough, it should be possible to just zap it by timing. Thoughts? Cheers, Arjen.

2 2