March 2015 - developers - lists.mariadb.org

[Maria-developers] GSoC 2015: Table functions (a.k.a UDF returning arrays / result set) MDEV-5199
by Diwas Joshi 25 Mar '15

25 Mar '15

Hello, I am Diwas, a Masters' student at IIIT Hyderabad, India. I am interested to work on "Table functions (a.k.a UDF returning arrays / result set)". I have downloaded and built MariaDB 10.1 source code from github. I started going through the code. I realized UDF's for scalar values is already implemented in the code, so i began to analyze from there. I got myself acquainted with the query work-flow. I believe the work will be similar to create_function from sql_udf.cc for some part. I understand that I'll need some real storage mechanism for executing the functions in queries so I'll need to use the handler class for this. Please tell me how much of the project I have understood correctly and if I am on the right lines here and how can i proceed towards writing my proposal for this project. regards Diwas Joshi

2 1

[Maria-developers] Port InnoDB memcached interface to MariaDB (GSOC 2015)
by Aalekh Nigam 24 Mar '15

24 Mar '15

Hello, I am Aalekh Nigam, 3rd Year Undergraduate student at JIIT Noida, India. I would like to propose my participation for Port InnoDB MemcacheInterface to MariaDB (https://mariadb.atlassian.net/browse/MDEV-4674) as GSOC 2015 Student. For the purpose of security following is the approach I have planned to accomplish for this project: The plugin will contain a xtradb_memcached.sql table for configuation option, these configuration options will also contain a username, password , tcp/udp address, port no. as column objects which will help us authenticate the connection for a specified user, thus on authentication the server would fetch corresponding tcp/udp address and port to connect with the memcached server. Hence this will not just prove an improvement to the existing mysql plugin but also provide a better security to the database. Nextly after connecting with a memcached server, our database will act as a NoSql system providing rapid user query transaction. Also please follow my proposal here: https://www.google-melange.com/gsoc/proposal/review/student/google/gsoc2015… Waiting for your comments regarding the approach. ---Aalekh

1 1

Re: [Maria-developers] I wonder what elements add to CMakeLists.txt
by 김경남 24 Mar '15

24 Mar '15

Hi, Sergei! Thanks for your advice and I agree with you. Accoring to your advice, It's not a good plan to pass some data to my plugin agent from sql_parser.cc because it needs to compile mariadb server. So I hava a another concept. Instead of put some code to sql_parse.cc, plugin collects a some data like below sample conceptual code. I_List_iterator<THD> it(threads); while ((tmp_thd=it++)) { if ((tmp_thd->get_command() == COM_QUERY) ||(tmp_thd->get_command() == COM_STMT_EXECUTE) ) continue; char* sql = tmp_thd->query(); if(sql) { ... send_data(); } } In this case, I though that it doesn't need to compile maria server, but it need to recompile whenever structure of THD is changed. Can you accept that? Otherwise, if i hardcode directly into the server, can I contribute this community ? Additionally, you asked me about what information I want to collect. I want to collect data like below. * events_statements_current * events_stages_current * events_waits_current * events_statements_summary_by_digest These data easily collected by sql . but it's hard to collect by access memory.(THD or something else.) I wonder if it is possible to collect these data without using pfs (or psi) data structure. If it is possible, Is THD class is entry point or something else? If you don't mind, please let me know the best way to collect these data. Dear, kyungnam. ps: my email address changed from gmail to my company's mail address. ------------------------------------------------------------------------ ---- From: Sergei Golubchik <serg(a)mariadb.org> Date: 2015-03-24 0:11 GMT+09:00 Subject: Re: [Maria-developers] I wonder what elements add to CMakeLists.txt To: kyung nam Kim <knamkim(a)gmail.com> Cc: maria-developers(a)lists.launchpad.net Hi, kyung! On Mar 23, kyung nam Kim wrote: > Hi, Sergei! > Thanks for your reply. > You asked me about information which I want to send to my plugin. > I have a plan to develop monitoring tool for mariadb. > I want to develop a agent as a mariadb plugin. > Agent collect data from sql_parse.cc (I' don't know exactly where more > infomation about performance .) and sends that data to APM (Application > Performance Monitoring) Server. and Users use databoard to see that data > via APM server. > > We have already have a APM Server, Dashboard and many kinds of agent. (for > server, was). I want to add agent for mariaDB and contribute this community > if possible. Right. That's exactly why I ask. We cannot accept your plugin and it'll be of little use to the community (not "of no use", it'll be used, but less than it could've been) if one would need to modify and recompile MariaDB to use your plugin. If you'd like, let's try to figure out whether you can collect your infomation without patching the server. So what data do you need, exactly? > Your advice is right. It's easier my header file to sql folder. > But When I want to develop my agent as a plugin type, I meet error as > follows. > ============================================ > [ 98%] Built target mysqlserver > Linking CXX shared library libmysqld.so > ../../sql_parse.cc:linenum: undefined reference to 'Mymon::Mymon()' > ../../sql_parse.cc:linenum: undefined reference to 'Mymon::putData(THD* > thd); Of course. Because you want sql_parse.cc to call your plugin, your plugin *always must be in the server*. You cannot have it as a separate shared object. To do that, add the MANDATORY keyword to your MYSQL_ADD_PLUGIN() declaration in CMakeLists.txt Regards, Sergei _____ 상기 메시지와 첨부화일 내에는 비밀정보가 포함되어 있을 수 있으며, 지정된 수신자에 한하여 조회 및 사용될 수 있습니다. 만약 송신자의 실수로 인하여 상기 메시지를 수신하였다면, 송신자에게 메시지를 반송해 주시고, 원본 메시지와 모든 사본을 폐기해 주시기 바랍니다. 상기 메시지의 전체 또는 일부에 대해 무단 열람, 사용, 공개, 배포하는 것은 금지되어 있습니다.(주)LG CNS. This message and its attachments may contain confidential information, and they are intended to be viewed or used by only the individuals specified in the message. If you have received this message in an error from the sender, please contact the sender immediately to notify the error and delete all of the message and its copies. It is prohibited to view, use, make public and/or distribute part or whole of this message without written permission.

2 3

[Maria-developers] I wonder what elements add to CMakeLists.txt
by kyung nam Kim 23 Mar '15

23 Mar '15

Dears. I'm a newer in maria develper community. I'm develping a simple plugin for monitoring. If I access from other module to my plugin interface, I wonder how and what elements add to CMakeLists.txt. for example, My plugin is daemon type and myplugin.h has a definition of public methodA. I want to use methodA in my plugin from sql/sql_parse.cc . I think that information of library is needed at build time. ( -L, -l option) I don't know how to add that elements to CMakeLists.txt. plz,let me know. Thanks in advance.

2 5

Re: [Maria-developers] [server] Execute sync_with_master before SELECT in rpl_gtid_basic.test (#31)
by Kristian Nielsen 23 Mar '15

23 Mar '15

f4rnham <notifications(a)github.com> writes: > Usually slave synchronized on its own before select, but not always. Those two lines were probably swapped by accident > You can view, comment on, or merge this pull request online at: Indeed, silly mistake... This was already fixed in 10.0, so it will be merged to 10.1 that way. But thanks for looking into this anyway, it's good to be aware of these kinds of races in replication tests, they are quite common. - Kristian.

1 0

Re: [Maria-developers] [JIRA] (MDEV-7557) Option to put into slow log any transaction that was retried in parallel replication
by Kristian Nielsen 23 Mar '15

23 Mar '15

"Sergei Golubchik (JIRA)" <jira(a)mariadb.atlassian.net> writes: >> Key: MDEV-7557 >> URL: https://mariadb.atlassian.net/browse/MDEV-7557 >> In optimistic parallel replication (MDEV-6676), we speculatively apply >> transactions in parallel, even if they might conflict. If too many conflicts >> happen, excessive retry might degrade performance. >> The DBA needs some way to monitor for such problems. One good way would seem >> to have an option to add to the slow query log any transactions that were >> retried due to parallel replication conflicts. Possibly with extra information >> such as duration of transaction and similar. > I'm not sure that slow log is a good match for this kind of events Right. So do you have a suggestion for a better way? I was thinking on --log-queries-not-using-indexes, which similarly uses the slow query log to expose certain kinds of problematic queries. So queries that cause deadlocks could be another useful filter for the slow log (it should then probably be for all queries that deadlock, not just replication queries). In non-parallel replication, we put deadlock errors as warnings in the error log. But I disabled that for parallel replication. In non-parallel replication, you normally do not expect any deadlocks (since there is only one thread, how can it deadlock?). But in parallel replication, deadlocks are generally unavoidable, and not harmful. And DBAs do not like noise in the error logs that does not indicate a problem and cannot be removed. One option might be to log deadlocks as INFO level in the error log, and then users can temporarily raise verbosity level to see them? Another option could be to add an option to enable/disable logging deadlocks as warnings in parallel replication. (Though probably you will think as me, that this feels a bit like too ad-hoc an option to add.) Any other ideas? - Kristian.

2 1

[Maria-developers] GSoC 2015: Indexes on Virtual Columns and Array Based UDFs
by Richa Sehgal 20 Mar '15

20 Mar '15

Dear mentors, I am currently pursuing Master's in University of Illinois at Urbana Champaign, USA and completed by B.Tech from Indian Institute of Technology - Delhi (IIT- Delhi) This is regarding GSoC 2015. I am really interested in databases, and I was very excited to see all these projects listed here. The exciting part was that some of the projects are really “hard” as in they have challenged the database community since a long time, and thus it would be very interesting to solve some of these challenges as part of GSoC. I want to discuss 2 projects: A. Indexes on virtual columns Materialization gives us two things: 1. A name to the column which we can use in queries 2. A formal "regular" column which is stored and indexed in the regular fashion - Disadvantage: Extra memory requirements for the materialized column. My initial thoughts on this project are the following: We do need the name of the column which can be used to query. So maybe we can expose a command such as: create virtual_index <name> on <column_name> <expression> What this would do would run a regular query which evaluates expressions (like in WHERE clause) and the feed the result into the indexer. This index can then be stored in the regular fashion. B. Having UDFs returning an array/set There are three approaches that I can think of: 1. Supporting array/set as native datatype inside MariaDB (like int64, double, etc) - This might be hard and touches all levels of stack. 2. Have the array/set pass in serialized form to the above node of query execution and have appropriate deserializer when we want to interpret the result - Coming up with ser/deser strategy might be tough and this would be expensive too. 3. The query execution would be in a Tree structure where each node must be exposing functions like init(), next(), read(int col_index), etc. Maybe we can use this to emulate the evaluation of UDF against row. I think this is the suggestion that is listed in the project. I would like to get some direction on this from the mentors. I would like to discuss these and then decide on one of them. Am I approaching this in the right direction? Can you please point me to the next steps? Thanks Richa

3 9

[Maria-developers] Tasks for newcomer
by Martin Kaluznik 19 Mar '15

19 Mar '15

Hello, I am student interested in GSoC project Automatic provisioning of slave ( https://mariadb.atlassian.net/browse/MDEV-7502 ). Who would like to contribute to codebase. I will appreciate if somebody could suggest me something to work on. I found something really small for myself (https://mariadb.atlassian.net/browse/MDEV-7780 ) and implemented it in most basic way. https://github.com/f4rnham/server/commits/MDEV-7780 Config entry for case specified in report would be fake_version=5.5.25 It will probably require test case too, and I am not sure if this feature will be actually safe, it may create some inconsistencies. I have never worked with bzr before, but if it will be required, I will look into that. Thanks for feedback, Martin Kaluznik ( martin.kaluznik(a)gmail.com ). I can be also found on IRC under nick Farnham

2 1

Re: [Maria-developers] 1cda762: MDEV-6069: Remove old logic for 3.23-to-higher upgrades from upgrade SQL scripts
by Sergei Golubchik 19 Mar '15

19 Mar '15

Hi, Nirbhay! On Mar 17, Nirbhay Choubey wrote: > revision-id: 1cda762d3f1132e6696e1c451435746db0a1420a > parent(s): b0542b78c866fb32383f759914b10f060d0e14cd > committer: Nirbhay Choubey > branch nick: server > timestamp: 2015-03-17 11:20:49 -0400 > message: > > MDEV-6069: Remove old logic for 3.23-to-higher upgrades from upgrade SQL scripts > > Fix for failing tests. > > * Update mysql_system_tables_fix.sql to makeup the differences in system > tables in 5.1.17 (main.system_mysql_db_fix50117) I don't quite understand that. Why these ALTER TABLEs were not needed before? > * Removed system_mysql_db tests for versions 5.0.30 & 4.1.23. ok Regards, Sergei

3 3

[Maria-developers] [GSoC] MDEV-7502 Automatic provisioning of slave
by Martin Kaluznik 18 Mar '15

18 Mar '15

Hello, I am student interested in GSoC project Automatic provisioning of slave ( https://mariadb.atlassian.net/browse/MDEV-7502 ). I have read current code and put together my idea of how it could be implemented. Please correct me if there is better solution for some steps or I just misunderstood something. Slave starts regular sql thread and something really similar to current io thread (maybe best to begin with will be copy of handle_slave_io and related functions, it may still end with some duplicate code, which at the end can be moved to new common functions). Slave will receive gtids of current master as response to copy-modified version of COM_BINLOG_DUMP, they will mark position from which binlogs will be sent to slave. Master proceeds with building of list containing what needs to be sent to slave. Here exact implementation will depend on whether the master itself will block any DDL queries during provisioning process or not. If he will (probably won’t – it’s not necessary and it will just slow him down), we will have guaranteed that internal containers with databases/tables won’t change and they can be just iterated through safely (not true for procedure/trigger containers?). In other case, it will be probably required to prefetch list of databases/tables to copy to slave (it can be done lazily with tables/triggers/procedures/…, holding only list of tables/… for database which is currently being processed). In the second case, slave will have to detect DDL event and interrupt connection. The range scans on primary key mentioned in description, I assume it would look like SELECT * FROM tbl WHERE pk > @lastChunkKey ORDER BY pk LIMIT @chunkSize; just done with internal functions more efficiently, and the result will be converted to Rows_log_event which will be sent to slave. To restrictions I would add at least one more, provisioning cannot be continued if master is restarted during process (or can we actually store information about provisioning process in database?). Another possible problem could be slave reconnect. I found kill_zombie_dump_threads function, in this case, new thread could take information about provisioning progress from old thread and with help of fake gtids bundled with chunks somehow figure out which chunk was last successfully received by slave before disconnect. It would require additional memory to store few (or more?) most recently sent chunks and wouldn't guarantee success. Great way to solve this would be, to be able to map gtid sequence number to exact chunk of data, but I can’t see how it can be done without assuming something about maximum amount of rows in table / number of tables / triggers / … Thanks for feedback, Martin Kaluznik ( martin.kaluznik [at] gmail.com ). I can be also found on IRC under nick Farnham

2 1