- developers - lists.mariadb.org

[Maria-developers] audit plugin help with MDEV-575 (fail2ban like)
by Roberto Spadim 09 Jun '13

09 Jun '13

hi guys i want get users / ip that try to connect to server with bad password/user and retry very frequently (normally a brute force attack) some things i need: 1)get information about connection when it fail or not (maybe an audit plugin could do this?), information that i need: 1.1)protocol (tcp/unix/others) 1.2)user 1.3)remote ip (when tcp only) it must be executed in every login 2) log this information maybe we could use CSV,aria, myisam, to log bad logins and put it in a log table: PROTOCOL, USER, REMOTE IP, DATATIME 3) add columns to user BAD_LOGIN_MAX_TRIES, BAD_LOGIN_LOG_TIME, BAD_LOGIN_MIN_RETRY_TIME 4) add server global bad_login_tcp_log_time, bad_login_tcp_max_errors 5) implement checks before connection: when openned a connection to server via TCP execute something like: SELECT COUNT(*) FROM bad_login_log WHERE remote_ip="remote_ip" AND datetime>NOW() - @@bad_login_tcp_log_time if COUNT(*) > @@bad_login_tcp_max_erros disconnect 6) implement user checks: SELECT COUNT(*),(SELECT BAD_LOGIN_MAX_TRIES FROM users WHERE user="user") FROM bad_login_log WHERE user="user" AND datetime>NOW() - (SELECT BAD_LOGIN_LOG_TIME FROM users WHERE user="user") if count(*) > bad_login_max_tries form that user disconnect that's all -- Roberto Spadim SPAEmpresarial

1 0

[Maria-developers] Help with MDEV-4419, ENUM optimization
by Roberto Spadim 09 Jun '13

09 Jun '13

Hi guys, i want to optimize fields with ENUM() in item comparation the point is: SELECT COUNT(*) FROM table WHERE enum_field!='x' column 'enum_field' is ENUM('x,'y','z') NOT NULL DEFAULT 'x' i want to rewrite the query in a way that we can execute it with index: SELECT COUNT(*)FROM table WHERE enum_field IN ('y','z','') the "" is not allowed values ('a' for example) the feature is 'easy' to understand but i don't know where to implement it will read possible values of enum fields, and rewrite "!=","<","<=",">=",">" operators to IN operator with fields that match the condition of the primary operator some think like: column "operator" const value if(operator in ('!=','<','<=','>=','>' AND column is ENUM){ get all possible values from column copy all values to a return variable execute the "operator" in all values with const values if operator return false remove the value from the return variable change the "operator" and "const value" to : "IN" and return variable comma separated }else{ no optimization leave operation as it is } example: column = enum('x','y','z'), operator = ">" const value = "3" if(column is ENUM and operator = ">"){ possible values => "x","y","z","" (check that "" is used when insert into, value = "a" for example) return variable = "x","y","z", "" for each value.... "x" > "3" ? yes, ASCII 120 > ASCII 51 -> do nothing "y" > "3" ? yes, ASCII 121 > ASCII 51 -> do nothing "z" > "3" ? yes, ASCII 122 > ASCII 51 -> do nothing "" > "3" ? no, "" < ASCII 51 -> remove the value "" from return variable end for now the return variable is : "x","y","z" the "" was removed rewrite the operator to: "column IN ('x','y','z')" } ----------------------- this will optimize a lot: using != in a big table (446577 rows) result = Using where, 446577 rows, type= ALL using IN() in a big table: result = Using where, 28 rows, key = index, type = range in other words, a lot of optimization with ENUM fields, since they are a pseudo "index" of what we can found in the column =) thank guys any help is wellcome to make a patch -- Roberto Spadim SPAEmpresarial

1 0

[Maria-developers] GSoC MariaDB Kerberos Authentication Plugin Proposal
by QIU Shuang 06 Jun '13

06 Jun '13

Hi Sergei, Thank you and the community for accepting my proposal of this year's GSoC: Kerberize MariaDB -- add kerberos authentication support to MariaDB. I'm Shuang Qiu, a second year graduate student from Shanghai Jiao Tong University, Shanghai, China. My field of research is distributed computing, especially distributed storage. I just get in touch with my project mentor Mr. Vaintroub and join the #maria IRC channel at freenode.net, where I can be found as shuang. Thank you again and hope we can have a happy and productive summer! Cheers! Sincerely, Shuang

1 0

Re: [Maria-developers] [Maria-discuss] Clear-text plugin
by Peter Laursen 06 Jun '13

06 Jun '13

"no support for dlopen()" . How does this apply to Windows clients? We always build first on Windows and once we have no errors of any kind we build on Linux if the application exists for that platform (what MONyog does and SQLyog does not) and fix what issues may occur there. This issue has occurred with SQLyog (and as such for a Windows client). I am not coder/developer myself, but on Windows I think we have no other option than statically linking? We could of could compile the API as a .dll, but we don't (because we had a lot of problems with it earlier). -- Peter On Thu, Jun 6, 2013 at 3:25 PM, Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, Peter! > > On Jun 04, Peter Laursen wrote: > > We have the same problem. We cannot test either. But we have a few user > > reports. We will discuss what we can do. > > > > Is there any parameter or compile-time option we should be aware of? Any > > specific documentation somewhere? Anything? > > Not really. If the cleartext plugin is built and installed, it'll be > used when the server requests it. > > I've just checked that this plugin is present both in rpm and deb > packages. I didn't check binary tarballs, but it should be there too. > > So, there's nothing to configure at run-time or compile-time, everything > should work automatically and transparently. > > Hmm, unless you build a completely static client library with no support > for dlopen(). > > Regards, > Sergei >

2 1

Re: [Maria-developers] [Maria-discuss] Clear-text plugin
by Peter Laursen 04 Jun '13

04 Jun '13

We have the same problem. We cannot test either. But we have a few user reports. We will discuss what we can do. Is there any parameter or compile-time option we should be aware of? Any specific documentation somewhere? Anything? -- Peter On Tue, Jun 4, 2013 at 2:19 PM, Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, Peter! > > On Jun 04, Peter Laursen wrote: > > For clarity: a 'dummy' parameter doing nothing does not solve our > > problem. We actually want to build clients that can (also) connect to > > Oracle MySQL Enterprise servers configured to make use of Oracle's PAM > > implementation. > > We build and distribute a cleartext plugin. And it can be used if the > server requests. In a sense, in MariaDB a cleartext plugin is always > enabled. So a dummy option is needed only to restore the binary protocol > compatibility. That's my understanding, at least, but I don't have > Oracle's PAM plugin, so I cannot test it. > > But if you have verified that your client with our C connector can not > connect to Oracle MySQL Enterprise server with the Oracle's PAM > implementation - please submit a bug report, and we will fix it. > > Regards, > Sergei > >

1 0

Re: [Maria-developers] [Maria-discuss] Clear-text plugin
by Peter Laursen 04 Jun '13

04 Jun '13

For clarity: a 'dummy' parameter doing nothing does not solve our problem. We actually want to build clients that can (also) connect to Oracle MySQL Enterprise servers configured to make use of Oracle's PAM implementation. -- Peter On Tue, Jun 4, 2013 at 11:05 AM, Peter Laursen <peter_laursen(a)webyog.com>wrote: > yes .. in the Connector/C. We use C/C++ for both SQLyog and MONyog. > > -- Peter > > > On Tue, Jun 4, 2013 at 10:17 AM, Sergei Golubchik <serg(a)mariadb.org>wrote: > >> Hi, Peter! >> >> On Jun 04, Peter Laursen wrote: >> > This is a continuation of >> > https://mariadb.atlassian.net/browse/CONC-25. We requested support for >> > the clear-text plugin (or a workaround of any kind) that would allow a >> > client compiled with the MariaDB connector to use PAM authentication >> > with MySQL Enterprise servers from Oracle. >> >> Another related issue is https://mariadb.atlassian.net/browse/MDEV-4574 >> which should be fixed in 5.5.32, next 5.5 release. >> >> But you, probably, want it in Connector/C, don't you? >> >> Regards, >> Sergei >> >> >

1 0

[Maria-developers] Clear-text plugin
by Peter Laursen 04 Jun '13

04 Jun '13

This is a continuation of https://mariadb.atlassian.net/browse/CONC-25. We requested support for the clear-text plugin (or a workaround of any kind) that would allow a client compiled with the MariaDB connector to use PAM authentication with MySQL Enterprise servers from Oracle. I will not dispute Sergei's opinions about Oracle's implementation. But I believe it misses the point. The question is if the MariaDB connector should be a replacement for the C-IPA from Oracle. And the question is nothing else than that. Not the quality of solutions bundled with Enterprise servers from Oracle. I would like to hear opinions not only from 'core developers' like Sergei, but also from wider circles in MariaDB organization (management, marketing) as well as the user community. For us the problem is like this: We introduced support for the clear-text plugin in SQLyog 11.1. In 11.12 beta2 we replaced Oracles connector with MariaDB (and had to removed it again). We had a few crashes and memory leaks with mariaDB Connector but that is getting in control now (ther were bugs in both the API and in oru code). We probably could patch the MariaDB code without much problems to support the full-text plugin - but we don't have a MySQL Enterprise server to test with and we cannot release without proper testing. We have already started getting complaints from users/customers about that what we introduced in 11.1 has been removed in current development tree. It seems that we have quite a lot of customers using Oracle's Enterprise servers wiht Oracle's PAM authentication. A solution/workaround would very much be appreciated. Regards Peter Laursen Webyog

1 0

Re: [Maria-developers] MDEV-4568 - Port Percona response time distribution as audit plugin
by Sergey Vojtovich 03 Jun '13

03 Jun '13

Hi Sergei, 03.06.2013, в 16:36, Sergei Golubchik <serg(a)mariadb.org> написал(а): > Hi, Sergey! > > On May 31, Sergey Vojtovich wrote: >> revno: 3753 >> revision-id: svoj(a)mariadb.org-20130531131833-zs83x5wntvl1i15f >> parent: wlad(a)montyprogram.com-20130523125248-22m56m01r24t0dmb >> committer: Sergey Vojtovich <svoj(a)mariadb.org> >> branch nick: maria-qrtp >> timestamp: Fri 2013-05-31 17:18:33 +0400 >> message: >> MDEV-4568 - Port Percona response time distribution as audit plugin >> >> sql/sp_head.cc: >> Trigger MYSQL_AUDIT_GENERAL_STATUS event for each statement in stored >> procedures. Not strictly needed for query_response_time plugin, but >> makes it behave more like Percona patch. > > I'm sorry, I couldn't quite understand what you have changed, as > compared to the first patch (and the first was not in emails, so I > couldn't compare them). The difference is rather minor: - renamed query_response_time_exec_time to query_response_time_exec_time_debug - updated plugin author and added GPL header to plugin.cc - fixed I_S plugin description What I didn't touch were query_response_time.cc and .h, but below you agree that we shouldn't touch them. > Looks pretty much the same as the previous one, although you need to run > the tests and to see that they pass (you've changed plugin description > in the code, but the test result wasn't changed accordingly). Hmm… you're right, somehow I forgot to update basic test. Other tests are up to date. > Otherwise, looks ok to push, I suppose we can let it be as close to > Percona's version as possible. I don't think you need to rewrite it, > it's in a plugin now anyway. > > Regards, > Sergei Thanks, Sergey

1 0

[Maria-developers] Partition prune and lock
by Roberto Spadim 02 Jun '13

02 Jun '13

Hi guys i'm studing mariadb code and reading user manuals to know more about features. My last study was about query cache with many ideas in some MDEV and some patchs sent Now i'm reading about partition My questions are 1)when a update or a delete is done on a expecific partition like: Update table set field=2 where partition=1 How lock is done? Lock is done at table level or partition level? 2)Today we have vertical partition, could we implement 'virtual' horizontal partition update and delete in some group of fields (horizontal partition) only lock the horizontal partition being changed and used in where clausule? Any idea and problems on how to develop it? 3) can query cache be partition pruned? instead of removing entire table, just remove the partition that was changed It will consume more memory but will do a better work in cache Maybe a new key like SQL_CACHE_PARTITON or a new variable, query_cache_min_unit=PARTITION or TABLE could change how the query will be cached and pruned Thanks guys

1 1

Re: [Maria-developers] [codership-team] Hanging Galera Cluster, looking for the cause
by Jan Kirchhoff 01 Jun '13

01 Jun '13

I have seen exactly the same issue again, second day in a row. This time at 7:30 in the morning and again, no logs or anything that would give me an idea of what could have caused this. "killall -9 mysqld" and "service mysql start" on the hanging node is the only solution I have so far... I am crossposting this to maria-developers, maybe somebody on that list has an idea where to start? Jan Am 31.05.2013 10:29, schrieb Jan Kirchhoff: > Hi, > > we have been testing MariaDB 5.5.29 with Galera > (wsrep_provider_version 23.2.4(r147)) for a few months now and have had > a freeze of the cluster for the second time now. I didn't report the > first hang about 6 weeks ago as I wanted to be sure it wasn't our fault, > but I can't find any type of network issue or such in our monitoring > system and nothing in the logs that could be related to this problem: > > We have 2 database servers running Ubuntu 12.10 with Kernel 3.5.0 > (names: db8, db9), with a bunch of 15k-disks, some SSDs, 128GB RAM and a > test database of about 600-700GB. The amount of binlogs written every > day is around 500GB, so you get an idea of the workload. > db8 is a slave (traditional replication) of our main database. We have a > garbd running on another server as a third galera instance. > > Everything runs smooth for some weeks, but since this morning at 3:50am > db9 seems to be frozen. The server itself looks good, mysqld is taking > about 50% of one CPU core, everything else looks normal. Nothing in the > logs. Connecting to mysql works, but any query (like "show status" or > normal selects) hang forever. Only "show engine innodb status" returns. > There are almost 3000 processes that piled up like this: > > ---TRANSACTION 2F89D7CB, not started > MySQL thread id 113575, OS thread handle 0x7fab31821700, query id > 45249946 cl4.logical-line.de 192.168.0.16 stats_pflege sleeping > > but none of them ever gets any further. I can issue a "kill 113575" and > the prompt returns, but nothing happens. > > This is what db8's global status says: > wsrep_local_state_uuid 41d7259d-7200-11e2-0800-943dd0e2f659 > wsrep_protocol_version 4 > wsrep_last_committed 179518511 > wsrep_replicated 53590833 > wsrep_replicated_bytes 14768686288759 > wsrep_received 894995 > wsrep_received_bytes 7161712 > wsrep_local_commits 53583808 > wsrep_local_cert_failures 60 > wsrep_local_bf_aborts 0 > wsrep_local_replays 0 > wsrep_local_send_queue 8 > wsrep_local_send_queue_avg 0.000000 > wsrep_local_recv_queue 0 > wsrep_local_recv_queue_avg 0.000000 > wsrep_flow_control_paused 1.000000 > wsrep_flow_control_sent 0 > wsrep_flow_control_recv 0 > wsrep_cert_deps_distance 73.229730 > wsrep_apply_oooe 0.000000 > wsrep_apply_oool 0.000000 > wsrep_apply_window 0.000000 > wsrep_commit_oooe 0.000000 > wsrep_commit_oool 0.000000 > wsrep_commit_window 0.000000 > wsrep_local_state 4 > wsrep_local_state_comment Synced > wsrep_cert_index_size 25606 > wsrep_causal_reads 0 > wsrep_incoming_addresses 192.168.10.30:3306,192.168.10.31:3306, > wsrep_cluster_conf_id 15 > wsrep_cluster_size 3 > wsrep_cluster_state_uuid 41d7259d-7200-11e2-0800-943dd0e2f659 > wsrep_cluster_status Primary > wsrep_connected ON > wsrep_local_index 0 > wsrep_provider_name Galera > wsrep_provider_vendor Codership Oy <info(a)codership.com> > wsrep_provider_version 23.2.4(r147) > wsrep_ready ON > > > > > Here is the innodb status from db9 (the one that hangs). This is the > only output I could get out of db9 at all: > > Type Name Status > InnoDB > ===================================== > 130531 10:02:08 INNODB MONITOR OUTPUT > ===================================== > Per second averages calculated from the last 18 seconds > ----------------- > BACKGROUND THREAD > ----------------- > srv_master_thread loops: 907033 1_second, 906789 sleeps, 89831 > 10_second, 22491 background, 22489 flush > srv_master_thread log flush and writes: 478222 > ---------- > SEMAPHORES > ---------- > OS WAIT ARRAY INFO: reservation count 17017782, signal count 287266881 > Mutex spin waits 98936299, rounds 91970366, OS waits 868715 > RW-shared spins 139527705, rounds 455067309, OS waits 6488041 > RW-excl spins 58122849, rounds 842712424, OS waits 8565312 > Spin rounds per wait: 0.93 mutex, 3.26 RW-shared, 14.50 RW-excl > -------- > FILE I/O > -------- > I/O thread 0 state: waiting for completed aio requests (insert buffer > thread) > I/O thread 1 state: waiting for completed aio requests (log thread) > I/O thread 2 state: waiting for completed aio requests (read thread) > I/O thread 3 state: waiting for completed aio requests (read thread) > I/O thread 4 state: waiting for completed aio requests (read thread) > I/O thread 5 state: waiting for completed aio requests (read thread) > I/O thread 6 state: waiting for completed aio requests (write thread) > I/O thread 7 state: waiting for completed aio requests (write thread) > I/O thread 8 state: waiting for completed aio requests (write thread) > I/O thread 9 state: waiting for completed aio requests (write thread) > Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , > ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 > Pending flushes (fsync) log: 0; buffer pool: 0 > 49405970 OS file reads, 277420410 OS file writes, 6653801 OS fsyncs > 1 pending preads, 0 pending pwrites > 49.44 reads/s, 16384 avg bytes/read, 0.61 writes/s, 0.61 fsyncs/s > ------------------------------------- > INSERT BUFFER AND ADAPTIVE HASH INDEX > ------------------------------------- > Ibuf: size 1, free list len 698, seg size 700, 300226 merges > merged operations: > insert 410879, delete mark 8032461, delete 82075 > discarded operations: > insert 0, delete mark 0, delete 0 > Hash table size 212497717, node heap has 450852 buffer(s) > 16846.79 hash searches/s, 77045.28 non-hash searches/s > --- > LOG > --- > Log sequence number 18522888544071 > Log flushed up to 18522888544071 > Last checkpoint at 18522888544051 > Max checkpoint age 2035833201 > Checkpoint age target 1972213414 > Modified age 20 > Checkpoint age 20 > 0 pending log writes, 0 pending chkp writes > 6895012 log i/o's done, 0.39 log i/o's/second > ---------------------- > BUFFER POOL AND MEMORY > ---------------------- > Total memory allocated 110310195200; in additional pool allocated 0 > Internal hash tables (constant factor + variable factor) > Adaptive hash index 9086745120 \t(1699981736 + 7386763384) > Page hash 106249768 (buffer pool 0 only) > Dictionary cache 426512815 \t(424997488 + 1515327) > File system 429392 \t(82672 + 346720) > Lock system 296805680 \t(295674296 + 1131384) > Recovery system 0 \t(0 + 0) > Dictionary memory allocated 1515327 > Buffer pool size 6553599 > Buffer pool size, bytes 107374166016 > Free buffers 0 > Database pages 6102747 > Old database pages 2252752 > Modified db pages 1 > Pending reads 1 > Pending writes: LRU 0, flush list 0, single page 0 > Pages made young 48135625, not young 0 > 64.83 youngs/s, 0.00 non-youngs/s > Pages read 49405737, created 7213276, written 266046626 > 49.44 reads/s, 0.00 creates/s, 0.11 writes/s > Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 > Pages read ahead 0.00/s, evicted without access 40.11/s, Random read > ahead 0.00/s > LRU len: 6102747, unzip_LRU len: 0 > I/O sum[2662]:cur[22], unzip sum[0]:cur[0] > -------------- > ROW OPERATIONS > -------------- > 1 queries inside InnoDB, 0 queries in queue > 2 read views open inside InnoDB > ---OLDEST VIEW--- > Normal read view > Read view low limit trx n:o 2F754555 > Read view up limit trx id 2F754555 > Read view low limit trx id 2F754555 > Read view individually stored trx ids: > ----------------- > Main thread process no. 15888, id 140381271201536, state: sleeping > Number of rows inserted 46059494, updated 2501543763, deleted 28801859, > read 17046037029 > 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 48965.50 reads/s > ------------ > TRANSACTIONS > ------------ > Trx id counter 2F89DD34 > Purge done for trx's n:o < 2F754554 undo n:o < 0 > History list length 140 > LIST OF TRANSACTIONS FOR EACH SESSION: > ---TRANSACTION 0, not started > MySQL thread id 113586, OS thread handle 0x7fab314fe700, query id > 45250432 localhost root sleeping > SHOW engine INNODB STATUS > ---TRANSACTION 2F89DC7B, not started > MySQL thread id 113583, OS thread handle 0x7fab315d9700, query id > 45250367 cl4.logical-line.de 192.168.0.16 stats_pflege sleeping > ---TRANSACTION 2F89DB4F, not started > MySQL thread id 113582, OS thread handle 0x7fab31622700, query id > 45250261 cl4.logical-line.de 192.168.0.16 stats_pflege sleeping > ---TRANSACTION 2F89DA23, not started > MySQL thread id 113578, OS thread handle 0x7fab31746700, query id > 45250159 cl4.logical-line.de 192.168.0.16 stats_pflege sleeping > ---TRANSACTION 0, not started > <many many rows like these, around 3000 although max_connections is 300?! > > > As soon as I trigger the restart of mysql on db9 (service mysql > restart), I see normal log entries like this: > May 31 10:16:59 db9 mysqld: 130531 10:16:59 [Note] WSREP: Stop replication > May 31 10:16:59 db9 mysqld: 130531 10:16:59 [Note] WSREP: Closing send > monitor... > May 31 10:16:59 db9 mysqld: 130531 10:16:59 [Note] WSREP: Closed send > monitor. > May 31 10:16:59 db9 mysqld: 130531 10:16:59 [Note] WSREP: gcomm: > terminating thread > > and db8 is accepting inserts again and catches up with replication. > > meanwhile, db9 says: > May 31 10:17:02 db9 mysqld: 130531 10:17:02 [Note] WSREP: killing local > connection: 113529 > May 31 10:17:02 db9 mysqld: 130531 10:17:02 [Note] WSREP: killing local > connection: 113199 > May 31 10:17:02 db9 mysqld: 130531 10:17:02 [Note] WSREP: killing local > connection: 110221 > > and nothing happens any more. No CPU and no IO load coming from the > mysqld processes. > After 5 minutes I killed the mysqld with "killall -9 mysqld" and then > started it again the normal way. > db9 syncs via IST and everything is fine again. But I have no idea what > caused all this. According to a colleague killing mysql on db9 was the > thing that helped 6 weeks ago, too. > > Thanks for any pointers where we should look further. > > Jan Kirchhoff >

1 0

[Maria-developers] Open Questions in the knowledgebase
by Ian Gilfillan 31 May '13

31 May '13

I've been cleaning up the questions on the knowledgebase, but there are still a few open ones that need a response. If anyone can help, please pay a visit. Here are the ones still needing feedback: Multiple error logs: https://kb.askmonty.org/en/multiple-err-logs/ Query optimizer: https://kb.askmonty.org/en/does-the-query-optimize-a-low-cardinality-index-… Strange SSL messages: https://kb.askmonty.org/en/strange-ssl-messages-in-error-log/ Multi-source replication and time zones: https://kb.askmonty.org/en/time-zones-multi-source-replication/ And of course, the old faithful, when will it be released (10.x as stable) - https://kb.askmonty.org/en/release-date-for-10x-stable-version/ Also, a comment asking for specifics on JSON implementation going forward: https://kb.askmonty.org/en/column_json/ thanks, ian

1 0

[Maria-developers] a bug also in mariadb
by xiaobing jiang 30 May '13

30 May '13

hi all: I submit a bug: http://bugs.mysql.com/bug.php?id=69352 . I review the mariadb code, I think it is also in mariadb. I have read the page. https://kb.askmonty.org/en/reporting-bugs/ should I report this bug to mariadb? Thanks!

1 0

[Maria-developers] GSoC 2013 - Interested Student - Roles Project
by Vicentiu Ciorbaru 28 May '13

28 May '13

Hello, My name is Vicențiu Ciorbaru. I am 3'rd year undergraduate student at the Polytechnic University of Bucharest, Romania and I am very interested in contributing to MariaDB as part of Google Summer of Code. I have not used MariaDB previously but I have taken it for a spin just now. I have participated in GSoC once before and would like to do so again. I have contributed to other open source organisations (Mozilla, Samba, ROSEdu) and I submit patches when time permits. The project that interests me the most is MDEV-4397: Roles. I want to ask you what would be a good place to start in the codebase to help me get an idea what I need to do for this project? Looking forward to hearing from you, Vicențiu

2 7

Re: [Maria-developers] GTID work priorities
by Kristian Nielsen 28 May '13

28 May '13

Pavel Ivanov <pivanof(a)google.com> writes: > we would really love to see "strict gtid mode" and related stuff next. I've now pushed strict mode to 10.0-base, and documented it in the knowledgebase. However, one thing was not clear from your description in MDEV-4478: > 4) Probably with a separate flag we want to prohibit any writes to binlog on > the slave except from the slave thread(s), i.e. whenever MASTER_HOST is > configured server works as if sql_log_bin = 0 globally except for the slave > thread(s) which work as if sql_log_bin = 1 and SELECT @@global.sql_log_bin > still returns 1. Can you elaborate exactly what semantics you have in mind for this new option? And why a new option is needed/how it differs from the already existing read-only and sql-log-bin options? It sounds to me what you want is simply to be able to start the server with --sql-log-bin=0 and still have the slave sql thread log its events to the binlog (if --log-slave-updates=1)? I would expect that to work already, but maybe there is a bug (that we could fix), or maybe you had something else in mind? - Kristian.

2 1

[Maria-developers] Query cache information plugin - MDEV-4516
by Roberto Spadim 28 May '13

28 May '13

Hi guys i'm reading query cache code, and i have some doubts... mariadb-10.0.1 source: 1)PROTOCOL_LOCAL: at line: 1412 (Query_cache::store_query) we have this comment: /* PROTOCOL_LOCAL results are not cached. */ what's PROTOCOL_LOCAL ? it's the unix socket? windows pipe? embedded? memcache? mysql? another thing like this? or it's the internal mariadb source running queries? 2)the query_size=0 (MDEV-4516) i was reading at line 1680 if (is_disabled() || thd->locked_tables_mode || thd->variables.query_cache_type == 0) goto err; and line : 1694 DBUG_ASSERT(query_cache_size != 0); // otherwise cache would be disabled why it's isn't used in query_cache_information plugin? *check that in line 1680 it test the "query_cache_type" too, * *in MDEV-4516 report about size=0 we didn't tested this case!!!* 3) i was at line 1886: DBUG_PRINT("qcache", ("\ long %d, 4.1: %d, bin_proto: %d, more results %d, pkt_nr: %d, \ CS client: %u, CS result: %u, CS conn: %u, limit: %lu, TZ: 0x%lx, \ sql mode: 0x%llx, sort len: %lu, conncat len: %lu, div_precision: %lu, \ def_week_frmt: %lu, in_trans: %d, autocommit: %d", is this the "primary key" of the query cache!?!?!?!?!?!?! could we use a sha1 or another "primary key" method (index [] of the query in cache + '';'+ 5 bytes of a hash key to compare if the index still with the same query) to easly show query entry "number", and allow a "delete from query_cache_information where key=xxx" function? at line 1938 we have this maybe the query position on cache ("primary key of query cache")?: DBUG_PRINT("qcache", ("Query have result 0x%lx", (ulong) query)); 4) there's some more ideas about query cache information plugin at MDEV-4571, if anyone could read it =) well that's all =] -- Roberto Spadim

2 19

[Maria-developers] GTID replication and relay_log.info
by Pavel Ivanov 27 May '13

27 May '13

Kristian, There's a well-known problem in MySQL that relay_log.info is not always in sync with the database state, and killing and restarting server at an unfortunate time will cause it to re-execute last statement and potentially break replication. Did you fix this situation with GTIDs and rpl_slave_state table? It looks like with GTIDs it's really easy to solve the problem -- rpl_slave_state has GTID of the last executed transaction, so on restart one needs just to ignore all relay logs and re-connect to master with the last GTID. Did you do that? If not are you planning to do that? Thank you, Pavel

2 11

Re: [Maria-developers] MariaDB 5.5 EOL plans
by Honza Horak 27 May '13

27 May '13

On 05/24/2013 11:04 PM, Rasmus Johansson wrote: > Hi, > > On Tue, May 14, 2013 at 5:57 PM, Honza Horak <hhorak(a)redhat.com > <mailto:hhorak@redhat.com>> wrote: > > Hi guys, > > I'm not able to find any info about life cycle of MariaDB 5.5 or > even other versions. Particularly if the MariaDB 5.5 life cycle will > somehow follow the MySQL's 5.5 life cycle or if there is some > another vision/rough plans when supporting of 5.5 will end? > > > Thanks for bringing this up! It has been on the table quite some times, > but now finally I think we got it documented. So here it is: > Thank you very much, that looks really great. Honza

1 0

[Maria-developers] Syncs server status variable
by Ian Gilfillan 26 May '13

26 May '13

What does the Syncs server status variable record, and when was it introduced? I can't seem to find this documented anywhere. thanks, ian gilfillan

2 1

[Maria-developers] MariaDB 5.5 EOL plans
by Honza Horak 25 May '13

25 May '13

Hi guys, I'm not able to find any info about life cycle of MariaDB 5.5 or even other versions. Particularly if the MariaDB 5.5 life cycle will somehow follow the MySQL's 5.5 life cycle or if there is some another vision/rough plans when supporting of 5.5 will end? Cheers, Honza

3 3

[Maria-developers] query cache restrictions
by Roberto Spadim 24 May '13

24 May '13

hi guys, why we can't cache "union" queries with query cache? http://www.mysqlperformanceblog.com/2006/07/27/mysql-query-cache/ *Caching full queries only *– Meaning it does not work for subselects, inline views, parts of the UNION. This is also common missunderstanding. -- Roberto Spadim

1 1

Re: [Maria-developers] To Sergei:
by Sergei Golubchik 24 May '13

24 May '13

Hi, Samarth! On May 24, Samarth Mathur wrote: > I put the <plugin>.c file and the CmakeLists.txt in a folder on my desktop > and ran > *cmake .* > > CMake Warning (dev) in CMakeLists.txt: > No cmake_minimum_required command is present. A line of code such as > > cmake_minimum_required(VERSION 2.8) > > should be added at the top of the file. The version specified may be > lower > if you wish to support older CMake versions for this project. For more > information run "cmake --help-policy CMP0000". > This warning is for project developers. Use -Wno-dev to suppress it. > > then i ran > *cmake . -Wno-dev* > > CMake Error at CMakeLists.txt:1 (MYSQL_ADD_PLUGIN): > Unknown CMake command "MYSQL_ADD_PLUGIN". You need to do that from the MariaDB source tree. Put your plugin in the plugin/ directory, where other plugins are. Regards, Sergei

1 0

[Maria-developers] Fwd: To Sergei:
by Samarth Mathur 24 May '13

24 May '13

I put the <plugin>.c file and the CmakeLists.txt in a folder on my desktop and ran *cmake .* CMake Warning (dev) in CMakeLists.txt: No cmake_minimum_required command is present. A line of code such as cmake_minimum_required(VERSION 2.8) should be added at the top of the file. The version specified may be lower if you wish to support older CMake versions for this project. For more information run "cmake --help-policy CMP0000". This warning is for project developers. Use -Wno-dev to suppress it. then i ran *cmake . -Wno-dev* CMake Error at CMakeLists.txt:1 (MYSQL_ADD_PLUGIN): Unknown CMake command "MYSQL_ADD_PLUGIN". -- Configuring incomplete, errors occurred! Thanks On Fri, May 24, 2013 at 10:40 PM, Sergei Golubchik <serg(a)askmonty.org>wrote: > Hi, Samarth! > > On May 24, Samarth Mathur wrote: > > Hi, > > > > How do I go about compiling the plugin in mariadb? I searched online, but > > only found the mysql method : > > https://dev.mysql.com/doc/refman/5.5/en/compiling-plugin-libraries.html > > I'm getting some errors there, but, I wanna know, if it's done > differently > > or like this? > > Like in the manual, at the link above. > What errors are you getting? > > Regards, > Sergei >

1 0

Re: [Maria-developers] To Sergei:
by Samarth Mathur 24 May '13

24 May '13

Hi, How do I go about compiling the plugin in mariadb? I searched online, but only found the mysql method : https://dev.mysql.com/doc/refman/5.5/en/compiling-plugin-libraries.html I'm getting some errors there, but, I wanna know, if it's done differently or like this? Thanks On Thu, May 23, 2013 at 6:28 PM, Sergei Golubchik <serg(a)askmonty.org> wrote: > Hi, Samarth! > > On May 23, Samarth Mathur wrote: > > 2 questions: > > > > to make .so file from .c, is it necessary for me to compile mariadb > > after each change? Or is there a shorter way as far as plugins are > > concerned? > > You only need to compile the plugin, you don't need to compile MariaDB > at all, you can use a binary from our site or as provided by your OS. > > > Is there any specific tip you would have as far as coding is > > concerned? Or does the normal convention of good code, good comments > > and indentation work alright? > > Just normal convention. The usual - see how it's done elsewhere in > MariaDB and do the same. > > Regards, > Sergei > >

2 1

[Maria-developers] Fusion-io atomic writes patch comments
by Laurynas Biveinis 24 May '13

24 May '13

Hi - We ported your patch to Percona Server, meanwhile we collected the following comments which perhaps are useful for you. These are addressed in our XtraDB port, so you will pick them up, but since you patched InnoDB as well, you might consider addressing them there as well. I tried to order them roughly in the order of importance. The comments below refer to rev 3720 of 5.5 tree (and one comment to MariaDB 10 trunk). - Where are the testcases? sys_vars test doesn't count :) - fil_extend_space_to_desired_ size() fails to consider that it might be extending a single node on a multi-node tablespace. It will calculate the desired size as if the last node was the only one, overextending the tablespace by the size of all the previous nodes. Fix is replacing offset_high = size_after_extend * page_size / (4ULL*1024*1024*1024); offset_low = size_after_extend * page_size % (4ULL*1024*1024*1024); with offset_high = (size_after_extend - file_start_page_no) * page_size / (4ULL * 1024 * 1024 * 1024); offset_low = (size_after_extend - file_start_page_no) * page_size % (4ULL * 1024 * 1024 * 1024); - In the same function in InnoDB 5.6 / MariaDB 10.0 it is necessary to set node->being_extended = FALSE; before jumping to complete_io. - In the same function, complete_io: label may skip over the fil_node_complete_io() call, which is required for os_aio() tablespace-extending call only. - os_file_set_atomic_writes() should return not an int, but bool or ibool instead, also its return value currently is raw ioctl() result, which is used as a bool at callers anyway and reversed from the InnoDB convention of true/non-zero meaning success. While at that, it's also nice to add __attribute__((warn_unused_result)) to it. - If ioctl() failed, then it's customary for the failed I/O call handling to call one of the os_file_handle_error... functions, here os_file_handle_error_no_exit() looks like a good fit. - If atomic writes are unsupported, then I'm not sure why set errno / LastError. No syscall has failed, the atomic writes unsupported-ness is communicated by the os_file_set_atomic_writes() return value. - os_file_set_atomic_writes() declaration should not depend on __linux__ etc, thus we pushed the #ifdef to function body. - InnoDB uses UNIV_LINUX instead of __linux__, the build sys checks for the presence of sys/ioctl.h and defines corresponding define, thus we replaced #ifdef __linux__ #include <sys/ioctl.h> #ifndef DFS_IOCTL_ATOMIC_WRITE_SET #define DFS_IOCTL_ATOMIC_WRITE_SET _IOW(0x95, 2, uint) #endif with #if defined(UNIV_LINUX) && defined(HAVE_SYS_IOCTL_H) # include <sys/ioctl.h> # ifndef DFS_IOCTL_ATOMIC_WRITE_SET # define DFS_IOCTL_ATOMIC_WRITE_SET _IOW(0x95, 2, uint) # endif #endif at the top of the file, and the #ifdef inside the function is #ifdef DFS_IOCTL... - Is this bit inside __WIN__ in os_file_create_func() actually needed? if (srv_use_atomic_writes && type == OS_DATA_FILE && os_file_set_atomic_writes(file, name)) { CloseHandle(file); *success = FALSE; file = INVALID_HANDLE_VALUE; } - os_file_set_size() could call os_file_handle_error_no_exit() for failed posix_fallocate() instead of printing the errno itself. - Comments don't follow InnoDB style, missing header function comments, too long lines, spurious whitespace, etc. Our code is at https://bazaar.launchpad.net/~percona-core/percona-server/5.5/revision/518 https://bazaar.launchpad.net/~percona-core/percona-server/5.6/revision/353 We haven't addressed the testcase issue yet neither. Regards, -- Laurynas

2 2

Re: [Maria-developers] [Commits] Rev 3770: MDEV-4042 & MDEV-4536 fix. in file:///home/bell/maria/bzr/work-maria-5.5-MDEV-4536/
by Sergei Petrunia 24 May '13

24 May '13

Hi Sanja, The patch doesn't take one thing into account. Consider a query: explain select * from t12 where b>3 or exists (select max(t10.b) Z from t10 where t10.a in ( select t11.a from t11) group by t10.c having Z<100); +--+------------+-----------+------+-------------+------------+-------+----+----+-------------------------------+ |id|select_type |table |type |possible_keys|key |key_len|ref |rows|Extra | +--+------------+-----------+------+-------------+------------+-------+----+----+-------------------------------+ |1 |PRIMARY |t12 |ALL |NULL |NULL |NULL |NULL|10 | | |2 |SUBQUERY |t10 |ALL |NULL |NULL |NULL |NULL|6 |Using temporary; Using filesort| |2 |SUBQUERY |<subquery3>|eq_ref|distinct_key |distinct_key|4 |func|1 | | |3 |MATERIALIZED|t11 |ALL |NULL |NULL |NULL |NULL|6 | | +--+------------+-----------+------+-------------+------------+-------+----+----+-------------------------------+ Here, the EXISTS subquery is the one that will have the problem. I have added "in ( select t11.a from t11)" so that EXISTS subquery's join_tab array is not linear. The problem is not visible to the user, but it can be seen in debugger. Put a breakpoint in JOIN::cleanup(), with these commands: p table->alias->Ptr p table->file Run the query and you'll see: Breakpoint 5, st_join_table::cleanup (this=0x7fff280dbea0) at /home/psergey/dev2/5.5/sql/sql_select.cc:10248 $30 = 0x7fff28000af0 "t12" $31 = (ha_innobase *) 0x7fff28047198 ... Breakpoint 5, st_join_table::cleanup (this=0x7fff28053078) at /home/psergey/dev2/5.5/sql/sql_select.cc:10248 $32 = 0xd05418 "" $33 = (ha_heap *) 0x7fff2805ab98 ... Breakpoint 5, st_join_table::cleanup (this=0x7fff28051ef8) at /home/psergey/dev2/5.5/sql/sql_select.cc:10248 $36 = 0x7fff28025720 "t10" $37 = (ha_innobase *) 0x7fff28009558 ... Breakpoint 5, st_join_table::cleanup (this=0x7fff28052218) at /home/psergey/dev2/5.5/sql/sql_select.cc:10248 $38 = 0xd30a69 "sj-materialize" $39 = (ha_heap *) 0x7fff28054398 ... Note that in subquery join_tab->cleanup() was called for t10, and for sj-materialize. For t11 (which is inside the sj-materialize) it was not called. This caused by this loop being flat loop: > @@ -10646,6 +10646,12 @@ void JOIN::cleanup(bool full) > { > tab->cleanup(); > } > + if (join_tab != table_access_tabs) > + { > + for (uint i= 0; i < top_table_access_tabs_count; i++)a ^^^^^^^^^^^ here. > + table_access_tabs[i].cleanup(); > + table_access_tabs= join_tab; // avoid second cleanup() > + } > cleaned= true; > } > else It turns out, I was incorrect with the statement that "table_access_tabs array is always flat". On Tue, May 21, 2013 at 08:30:35AM +0300, sanja(a)montyprogram.com wrote: > At file:///home/bell/maria/bzr/work-maria-5.5-MDEV-4536/ > > ------------------------------------------------------------ > revno: 3770 > revision-id: sanja(a)montyprogram.com-20130521053031-xe34093yhvey0l5o > parent: monty(a)askmonty.org-20130515132812-pnjhza9iugz89356 > committer: sanja(a)montyprogram.com > branch nick: work-maria-5.5-MDEV-4536 > timestamp: Tue 2013-05-21 08:30:31 +0300 > message: > MDEV-4042 & MDEV-4536 fix. > > Optimization JOIN tabs also should be cleaned up (in case of EXPLAIN when we cleaning up everythibng at the end of execution). > === added file 'mysql-test/r/innodb_explain.result' > --- a/mysql-test/r/innodb_explain.result 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/r/innodb_explain.result 2013-05-21 05:30:31 +0000 > @@ -0,0 +1,34 @@ > +drop table if exists t1,t2,t3; > +# > +# MDEV-4536: sql_base.cc:1598: bool close_thread_table(THD*, TABLE**): > +# Assertion `table->key_read == 0' failed. > +# > +set optimizer_switch="index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off"; > +CREATE TABLE t1 (pk int auto_increment primary key, `col_int_key` int(11), > +key `col_int_key` (`col_int_key`),`col_varchar_key` varchar(128), key > +(col_varchar_key)) engine=innodb; > +EXPLAIN SELECT 1 FROM t1 AS alias1 WHERE EXISTS ( SELECT SQ2_alias1 . col_int_key AS SQ2_field1 FROM ( t1 AS SQ2_alias1 RIGHT OUTER JOIN t1 AS SQ2_alias2 ON (SQ2_alias2 . col_int_key = SQ2_alias1 . col_int_key ) ) GROUP BY SQ2_field1 HAVING SQ2_alias1 . col_int_key >= 7 ); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE > +2 SUBQUERY SQ2_alias2 index NULL col_int_key 5 NULL 1 Using index; Using temporary; Using filesort > +2 SUBQUERY SQ2_alias1 ref col_int_key col_int_key 5 test.SQ2_alias2.col_int_key 1 Using where; Using index > +set optimizer_switch=default; > +drop table t1; > +# > +# MDEV-4042: Assertion `table->key_read == 0' fails in > +# close_thread_table on EXPLAIN with GROUP BY and HAVING in > +# EXISTS SQ, InnoDB > +# > +CREATE TABLE t1 (a INT) ENGINE=InnoDB; > +CREATE TABLE t2 (b INT PRIMARY KEY, c INT) ENGINE=InnoDB; > +CREATE TABLE t3 (d INT) ENGINE=InnoDB; > +EXPLAIN > +SELECT * FROM t1 > +WHERE EXISTS ( SELECT b FROM t2, t3 > +GROUP BY b > +HAVING b != 3 ); > +id select_type table type possible_keys key key_len ref rows Extra > +1 PRIMARY NULL NULL NULL NULL NULL NULL NULL Impossible WHERE > +2 SUBQUERY t2 index NULL PRIMARY 4 NULL 1 Using index; Using temporary; Using filesort > +2 SUBQUERY t3 ALL NULL NULL NULL NULL 1 Using join buffer (flat, BNL join) > +drop table t1,t2,t3; > > === added file 'mysql-test/t/innodb_explain.test' > --- a/mysql-test/t/innodb_explain.test 1970-01-01 00:00:00 +0000 > +++ b/mysql-test/t/innodb_explain.test 2013-05-21 05:30:31 +0000 > @@ -0,0 +1,41 @@ > +# > +# Innodb specific explains > + > +--disable_warnings > +drop table if exists t1,t2,t3; > +--enable_warnings > + > +--source include/have_innodb.inc > + > +--echo # > +--echo # MDEV-4536: sql_base.cc:1598: bool close_thread_table(THD*, TABLE**): > +--echo # Assertion `table->key_read == 0' failed. > +--echo # > +set optimizer_switch="index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off"; > + > +CREATE TABLE t1 (pk int auto_increment primary key, `col_int_key` int(11), > +key `col_int_key` (`col_int_key`),`col_varchar_key` varchar(128), key > +(col_varchar_key)) engine=innodb; > + > +EXPLAIN SELECT 1 FROM t1 AS alias1 WHERE EXISTS ( SELECT SQ2_alias1 . col_int_key AS SQ2_field1 FROM ( t1 AS SQ2_alias1 RIGHT OUTER JOIN t1 AS SQ2_alias2 ON (SQ2_alias2 . col_int_key = SQ2_alias1 . col_int_key ) ) GROUP BY SQ2_field1 HAVING SQ2_alias1 . col_int_key >= 7 ); > + > + > +set optimizer_switch=default; > +drop table t1; > + > +--echo # > +--echo # MDEV-4042: Assertion `table->key_read == 0' fails in > +--echo # close_thread_table on EXPLAIN with GROUP BY and HAVING in > +--echo # EXISTS SQ, InnoDB > +--echo # > +CREATE TABLE t1 (a INT) ENGINE=InnoDB; > +CREATE TABLE t2 (b INT PRIMARY KEY, c INT) ENGINE=InnoDB; > +CREATE TABLE t3 (d INT) ENGINE=InnoDB; > + > +EXPLAIN > +SELECT * FROM t1 > +WHERE EXISTS ( SELECT b FROM t2, t3 > +GROUP BY b > +HAVING b != 3 ); > + > +drop table t1,t2,t3; > > === modified file 'sql/sql_select.cc' > --- a/sql/sql_select.cc 2013-05-07 11:05:09 +0000 > +++ b/sql/sql_select.cc 2013-05-21 05:30:31 +0000 > @@ -3680,8 +3680,8 @@ make_join_statistics(JOIN *join, List<TA > if (pull_out_semijoin_tables(join)) > DBUG_RETURN(TRUE); > > - join->join_tab=stat; > - join->top_join_tab_count= table_count; > + join->table_access_tabs= join->join_tab=stat; > + join->top_table_access_tabs_count= join->top_join_tab_count= table_count; > join->map2table=stat_ref; > join->table= table_vector; > join->const_tables=const_count; > @@ -10646,6 +10646,12 @@ void JOIN::cleanup(bool full) > { > tab->cleanup(); > } > + if (join_tab != table_access_tabs) > + { > + for (uint i= 0; i < top_table_access_tabs_count; i++) > + table_access_tabs[i].cleanup(); > + table_access_tabs= join_tab; // avoid second cleanup() > + } > cleaned= true; > } > else > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer Monty Program AB, http://askmonty.org Blog: http://s.petrunia.net/blog

1 0