- developers - lists.mariadb.org

[Maria-developers] Missing locking around THD::set_db() ?
by Kristian Nielsen 07 May '13

07 May '13

Hi, I was looking at a Valgrind warning in Buildbot (appended below). What happens seems to be this: Thread 1 is running SHOW PROCESSLIST, it grabs the pointer THD::db to the current database of thread 2. Thread 2 then does THD::set_db(), freeing the old THD::db pointer and allocating a new one with the new data. Thread 1 then resumes, doing strdup() of the _old_, now invalid, THD::db pointer, which reads garbage data (or could even segfault if we get really unlucky). This seems like a genuine bug. I see absolutely no locking protecting against this race :-( Any suggestions for how to deal with this? The only locking I see in mysqld_list_processes() is holding LOCK_thread_count, but taking that for every THD::set_db() sounds quite bad for scalability (for example, set_db() is run by every query replicated by a slave thread). Should we use LOCK_thd_data also for THD::db ? Or maybe we could have a fixed buffer in THD for the db? Seems we would need 193 bytes. Then SHOW PROCESSLIST could sometimes show a random mix of old and new db name if we race, but at least we would not read invalid memory or crash due to mummap() ... Any better ideas? - Kristian. ----------------------------------------------------------------------- Original valgrind warning, for reference: rpl.rpl_stop_slave 'mix,xtradb' w5 [ fail ] Found warnings/errors in server log file! Test ended at 2013-04-22 22:05:12 line ==9897== Thread 20: ==9897== Invalid read of size 1 ==9897== at 0x4C2826B: memcpy (mc_replace_strmem.c:878) ==9897== by 0xBAF9E8: strmake_root (my_alloc.c:424) ==9897== by 0x63C74E: mysqld_list_processes(THD*, char const*, bool) (sql_class.h:771) ==9897== by 0x5C8574: mysql_execute_command(THD*) (sql_parse.cc:3149) ==9897== by 0x5CC844: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:5759) ==9897== by 0x5CDDC9: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1068) ==9897== by 0x5CE4BD: do_command(THD*) (sql_parse.cc:794) ==9897== by 0x6A4E23: do_handle_one_connection(THD*) (sql_connect.cc:1266) ==9897== by 0x6A4F23: handle_one_connection (sql_connect.cc:1181) ==9897== by 0x8F0027: pfs_spawn_thread (pfs.cc:1015) ==9897== by 0x4E3506F: start_thread (in /lib64/libpthread-2.9.so) ==9897== by 0x62F713C: clone (in /lib64/libc-2.9.so) ==9897== Address 0x1a06a973 is 3 bytes inside a block of size 5 free'd ==9897== at 0x4C257D8: free (vg_replace_malloc.c:446) ==9897== by 0xBB96BB: my_free (my_malloc.c:115) ==9897== by 0x838C86: Query_log_event::do_apply_event(Relay_log_info const*, char const*, unsigned int) (sql_class.h:2846) ==9897== by 0x53904C: apply_event_and_update_pos(Log_event*, THD*, Relay_log_info*) (log_event.h:1230) ==9897== by 0x5446DD: exec_relay_log_event(THD*, Relay_log_info*) (slave.cc:2766) ==9897== by 0x545BD9: handle_slave_sql (slave.cc:3627) ==9897== by 0x8F0027: pfs_spawn_thread (pfs.cc:1015) ==9897== by 0x4E3506F: start_thread (in /lib64/libpthread-2.9.so) ==9897== by 0x62F713C: clone (in /lib64/libc-2.9.so)

4 4

Re: [Maria-developers] [Gsoc] Regex enhancements Project
by Sergei Golubchik 04 May '13

04 May '13

Hi, Sudheera! On Apr 19, Sudheera Palihakkara wrote: > Hi, > I went through other threads on this topic. In one thread you mentioned to > choose a suitable regex library. > > *( Preliminary research - only about chosing a regex library to use in > MariaDB. You should be able to explain why we should use this library and > not some other one.) > > * > What do you mean by "choosing"? don't we have to enhance the exiting regex > library? Or choose from exiting already implemented libraries which are > free to use? sorry if it's a stupid question, but I'm confused. :O Enhancing our old regex library to support all modern features and multiple charsets is complex and bug-prone work. I don't see why we should bother doing it, when there are plenty of regex libraries available. There's PHP's mb_regex, there's prce, and many others too. We'd better just pick one that works better for MariaDB, and put it instead of Henry Spencer's library. Regards, Sergei P.S. Please, don't reply to me only, use reply-to-all, so that your mails appear on the mailing list.

3 9

[Maria-developers] Fwd: MariaDB release numbering
by Jorge Manuel B. S. Vicetto 03 May '13

03 May '13

Hi. I've emailed Kristian about this subject and he suggested I sent this to your mailing list to get a more complete picture. On our work packaging MariaDB for Gentoo, me and Robin need to decide how to deal with the packaging of MySQL vs MariaDB and to that it would be really helpful to have an idea of how you plan to number your future releases. AFAIK, currently we can compare MySQL vs MariaDB as: MySQL MariaDB 5.1 5.1 / 5.2 / 5.3 5.5 5.5 (including MariaDB features from 5.2/5.3) 5.6 10.0 (based on MySQL 5.5 with backports from MySQL 5.6 and features from previous MariaDB versions) 5.7 ??? The first question for us is what version number will you use for a newer version based on MySQL-5.7. Another question is if you plan to port all your work to the 5.7 codebase or if you'll keep working on the current 10.0 codebase and do backports from 5.7 (like you did for MySQL 5.6). A second question is whether you still claim binary compatibility with MySQL[1] and how would you compare the latest versions of MariaDB and MySQL (10.0 vs 5.6, 5.7). [1] - https://kb.askmonty.org/en/mariadb-versus-mysql-compatibility/ Any info you can share on the release versioning plans would be really helpful. If this is still too early for any definitions or comments, is there any way we can follow or learn about such decisions? Thank you and regards, Jorge Manuel B. S. Vicetto Gentoo Developer

2 1

[Maria-developers] Forked mytop - conflicts, issues
by Honza Horak 03 May '13

03 May '13

Hi guys, MariaDB forked mytop utility source some time back, but it causes some issues for us in Fedora for example (and similarly in other distributions I guess) -- particularly the MariaDB's binary mytop conflicts with the same utility from separate package mytop. So, I'd like to take a look at possible solutions. Since there was a small commit in mytop upstream few weeks back, it doesn't really look like totally dead upstream. So the ideal solution from my POV would be including enhancements that MariaDB did to mytop back to upstream and not including mytop utility in MariaDB any more. Would something like that be possible from the perspective of MariaDB? Cheers, Honza

4 4

[Maria-developers] xa transaction cause replication broken
by jhx1008 02 May '13

02 May '13

hello I have another problem witch seems like a bug(Server version: 5.5.5-10.0.2-MariaDB-log MariaDB Server). the problem is about the replication for the xa transactions when the master crashed and then restarted. I know when we do a transaction, the queries will be stored in the binlog cache, after commit, flush the binlog cache into the binlog file, any normal un-committed transactions will be rollbacked when the server crashed then restarted, but the prepared xa transactions are not. Prepared xa transactions can be recovered use the command 'xa recover', then committed use 'xa commit xid', but the binlog cache is empty when the master restart, so if we do the 'xa commit xid', nothing to be recorded into the binlog files, the replication will be broken. test: master mysql> create table t(id int auto_increment primary key, a int) engine=innodb; mysql> xa start '111'; mysql> insert into t(a) values(1); mysql> xa end '111'; mysql> xa prepare '111'; kill -9 master_pid restart the master mysql> xa recover; +----------+--------------+--------------+------+ | formatID | gtrid_length | bqual_length | data | +----------+--------------+--------------+------+ | 1 | 3 | 0 | 111 | +----------+--------------+--------------+------+ mysql> xa commit '111'; mysql> select * from t; Empty set (0.00 sec) why? I did the xa commit '111', but no result return, maybe another bug? do the test continue mysql> xa start '222'; mysql> insert into t(a) values(2); mysql> xa end '222'; mysql> prepare '222'; mysql> xa commit '111'; mysql> select * from t; +----+------+ | id | a | +----+------+ | 1 | 1 | | 2 | 2 | +----+------+ value(1,1) return at this time, why? what's hanppen on the slave? mysql> select * from t; +----+------+ | id | a | +----+------+ | 2 | 2 | +----+------+ the replication is broken now!! by the way, when the xa transactions preapred, if the client exit, the transactions will be rollbacked in the THD::cleanup function, maybe the server rollback all the xa prepared transcations when restart is a good method? 2013-05-02 jhx1008 @netease

2 2

[Maria-developers] GTIDs, events sequence and alternate futures
by Pavel Ivanov 01 May '13

01 May '13

Kristian, I can't find the answer to my questions in the code and don't know if there's some documentation I can use (is there such thing?), so I'm writing my questions here. As I understand GTID is constructed as <domain id>-<server id>-<sequence number>. Is it guaranteed that sequence numbers are always increasing inside the domain id? So let's say two servers have GTID 0-1-100 and if they have alternate futures then one server will have GTIDs 0-1-101, 0-1-102 and so on, while second server will have 0-2-101, 0-2-102 etc. Is that correct? BTW, where in the code is sequence number increased exactly? Thank you, Pavel

2 4

Re: [Maria-developers] Installing jemalloc on build VMs
by Kristian Nielsen 26 Apr '13

26 Apr '13

Daniel Bartholomew <dbart(a)mariadb.org> writes: > I'm getting ready to install jemalloc on the buildslaves for TODO-459 - > https://mariadb.atlassian.net/browse/TODO-459 , and wanted to know if > there was a preferred way to do it. Since we want to use a specific > version of jemalloc (3.1.0), I figured I would build from source on all > of the build machines. Has this been thought through properly with respect to packaging? My guess is it has not. For .debs, which are what I know about, it is generally expected that one can rebuild from `apt-get source mariadb-server`. This requires proper build dependencies. It doesn't work to install random stuff on random machines. It does not appear that jemalloc 3.1 is available on most .deb distros we support. Wheezy seems to have 3.0.0. It is said that tokutek "requires" jemalloc - maybe it would be better to fix tokutek to work with standard libraries? The proper way to produce .debs with custom jemalloc requires creating new custom packages for libjemalloc and including them in our repos. - Kristian.

8 11

Re: [Maria-developers] MariaDB and jemalloc
by Axel Schwenke 24 Apr '13

24 Apr '13

Hi Sanja, Oleksandr Byelkin wrote: > 24.04.2013 19:00, Axel Schwenke пишет: > [skip] >> >> OLTP ro transactions per second relative to glibc malloc >> ---------------------------------------------------------------------- >> Threads jemalloc-3.3.1 glibc malloc tcmalloc jemalloc-3.1.0 >> 4 +3.2% +0.0% +11.6% +61.9% >> 8 +0.9% +0.0% -2.3% +8.0% >> 16 +9.6% +0.0% +4.7% +12.9% >> 32 +1.8% +0.0% +0.9% +5.8% >> 64 -5.1% +0.0% +2.1% +5.4% >> 128 -4.8% +0.0% +0.2% +4.5% >> 256 -4.2% +0.0% +0.7% +5.0% >> 512 -13.3% +0.0% +1.8% +4.1% > > Strange that results jumps up and down when percona had quite smooth > one. Is it possible that computer was doing something else? or we need > more tests to take average? It's not so "jumpy" when you look at the absolute numbers. The difference to Percona is, that they tested their own server, where MariaDB tries to minimize the number of malloc() calls. I wouldn't give too much on the numbers for 4 threads, since this is the buffer pool warmup phase. Also I've run each test for only 100 seconds to get results fast. Re "jumpy" - see the attached pdf with the scatter plot. Each point represents the tps from a 5 second interval. Numbers are in fact rather smooth, except for 512 threads. The main takeaway is, that I can reproduce the regression in jemalloc 3.3. With TokuDB there is no way around jemalloc anyway. XL

2 1

[Maria-developers] MariaDB and jemalloc
by Axel Schwenke 24 Apr '13

24 Apr '13

Hi, Monty asked me to do a quick check how MariaDB performs with jemalloc. This is in the light of the upcoming integration of TokuDB into the MariaDB tree. TokuDB requires jemalloc. While I did use tcmalloc in the connect speed benchmarks with good results, I did no use jemalloc before. This article from Percona http://www.mysqlperformanceblog.com/2013/03/08/mysql-performance-impact-of-… suggests that jemalloc and tcmalloc perform very comparable, but also that jemalloc shows a regression in 3.2.0 and later. The benchmark I used is sysbench OLTP ro, 16 tables, data fitting into buffer pool. I used the new 64 (virtual, 32 real) core machine and compared 4 different malloc implementations as below: OLTP ro transactions per second relative to glibc malloc ---------------------------------------------------------------------- Threads jemalloc-3.3.1 glibc malloc tcmalloc jemalloc-3.1.0 4 +3.2% +0.0% +11.6% +61.9% 8 +0.9% +0.0% -2.3% +8.0% 16 +9.6% +0.0% +4.7% +12.9% 32 +1.8% +0.0% +0.9% +5.8% 64 -5.1% +0.0% +2.1% +5.4% 128 -4.8% +0.0% +0.2% +4.5% 256 -4.2% +0.0% +0.7% +5.0% 512 -13.3% +0.0% +1.8% +4.1% (more data in attached spread sheet) Observations: - the effect is rather limited - jemalloc 3.1.0 clearly wins - jemalloc 3.3.1 performs bad at higher concurrency Conclusion It's fine to build MariaDB with jemalloc in the future. However we mustn't use jemalloc-3.3.1, but rather jemalloc-3.1.0. XL

1 0

[Maria-developers] Fwd: Fwd: gsoc13
by Samarth Mathur 24 Apr '13

24 Apr '13

Hi! I have a few doubts. Going through the internet, I've come across two major implementations of LDAP protocol The openLDAP, and the libpam_ldap library. Both have a detailed information of client side implementation. However, openLDAP gives the slapd which is a standalone server daemon, whereas from the libpam_ldap library, I am able to gather the existence of an inbuilt ldap authentication system. Am I correct about that? Which do you think will be useful to implement as a plugin for mariadb? Thanks!

2 2