- developers - lists.mariadb.org

[Maria-developers] [GSoC] MDEV-4674 Port InnoDB memcached interface to MariaDB - Midterm report
by Piotr Jurkiewicz 02 Jul '15

02 Jul '15

Hi all, as the midterm is approaching, I am posting a report on what I have done so far. The project is going as scheduled. You can see the timeline in my proposal: http://www.google-melange.com/gsoc/proposal/public/google/gsoc2015/piotrjur… All goals planned before the midterm have been reached. InnoDB Memcached plugin ported from MySQL works now with MariaDB (both with InnoDB and XtraDB engines). That means that MariaDB is now on par with MySQL on this issue. My repository with the code is available here: https://github.com/piotrjurkiewicz/mariadb-server What have been done in details: 1. Code of InnoDB Memcached plugin has been imported from MySQL into the MariaDB tree. 2. There were some differences between MariaDB and MySQL, which blocked the plugin from out-of-the-box compilation and correct operation. They have been resolved. 3. Test suite for the plugin has been imported from MySQL. Compatibility changes were introduced into these test cases which required them. The test suite was modified to assign memcached port number dynamically from mtr worker's pool, what allows to run the same test in parallel for many combinations. The plugin fully passes the test suite, both for 'innodb_plugin' and 'xtradb' combinations. 4. Some enhancements and bug fixes have been introduced into the plugin: - detection of memcached daemon initialization has been fixed -- this eliminated unnecessary wait (15 seconds) on plugin shutdown and memory leak (in cases when plugin failed to initialize properly before) - memcached arithmetic commands (incr/decr) handling has been improved -- now numeric columns values are handled in a consistent and predictable way - incorrect handling of unsigned not null integers columns has been fixed For details of all changes please see commit messages associated with them: https://github.com/piotrjurkiewicz/mariadb-server/commits/10.1-memcached After the midterm, I plan to focus on further enhancements of the plugin, as stated in my proposal: 1. Change plugin architecture to "one table per socket", as originally proposed in MDEV-4674 bug report. This will allow usage of well known security solutions (firewalls, unix file permissions) to restrict access to different tables for different users. 2. Implement a new interface inside the Memcached plugin, which will use Handler API for database access. This will make possible to use the plugin with database engines other than InnoDB/XtraDB. Ultimately, this interface will replace existing InnoDB direct API interface (if there will be no significant performance differences). Therefore, I would like to as you to point me to Handler API documentation (if documentation such exists). I know that HandlerSocket plugin uses the Handler API to access databases. Is HandlerSocket plugin a good example of Handler API usage to follow? Piotr

1 0

[Maria-developers] How to read of innodb_locks table by using low level interface?
by 김경남 01 Jul '15

01 Jul '15

Hi, Sergei I'm developing a mariadb monitoring plugin type agent. We're planning to contribute our monitoring solution for mariadb in one or two months. Our solution is consist of plugin type agent, APM Server and Dashboard. Our plugin type agent collect data by using low level interface instead of using query. I can collect a information_schema.global_status and performance_schema.event_xxxx_table for your help in the past. By referencing the source of feedback.cc, I can collect data from global_status using fill_stauts function. I hava a question. I need to collect data from information_schema.innodb_locks for lock monitoring. But I found that I must use trx_i_s_common_fill_table in i_s.cc for reading of innodb_locks table and that fuction is static. so I only access that function by function pointer. Debugging source for many days, I guess that I must use ST_SCHEMA TABLE's fill_table member variable for pointing to trx_i_s_common_fille_table function. Right? But I don't know how to that. I attemped in various ways for many days. but I failed. If possible. please let me know how to read data of information_schema.innodb_locks using native api. Thanks in advance. Regards, Sergei. _____ 상기 메시지와 첨부화일 내에는 비밀정보가 포함되어 있을 수 있으며, 지정된 수신자에 한하여 조회 및 사용될 수 있습니다. 만약 송신자의 실수로 인하여 상기 메시지를 수신하였다면, 송신자에게 메시지를 반송해 주시고, 원본 메시지와 모든 사본을 폐기해 주시기 바랍니다. 상기 메시지의 전체 또는 일부에 대해 무단 열람, 사용, 공개, 배포하는 것은 금지되어 있습니다.(주)LG CNS. This message and its attachments may contain confidential information, and they are intended to be viewed or used by only the individuals specified in the message. If you have received this message in an error from the sender, please contact the sender immediately to notify the error and delete all of the message and its copies. It is prohibited to view, use, make public and/or distribute part or whole of this message without written permission.

1 0

[Maria-developers] sprint tasks for the next week
by Alexander Barkov 01 Jul '15

01 Jul '15

Hi Sergei, I'd like to do these tasks: - MDEV-8215 Asian MB3 charsets: compare broken bytes as "greater than any non-broken character" - MDEV-8036 Fix all collations to compare broken bytes as "greater than any non-broken character" to close the group of tasks related to handling bad bytes in comparison. Any objections? How do I mark tasks as "sprint"? Thanks.

1 0

Re: [Maria-developers] MDEV-8356 analysis and ideas
by Sergey Petrunia 29 Jun '15

29 Jun '15

Hi Diwas, Nice analysis. Please find my comments below. On Mon, Jun 29, 2015 at 10:01:56AM +0530, Diwas Joshi wrote: > hello sergey, so i was reading further on MDEV-8356 last night. There are > two main things here: type of sf and return type of sf. Right now the > create function... is giving errors at > > store_failed= store_failed || > table->field[MYSQL_PROC_MYSQL_TYPE]-> > store((longlong)type, TRUE); > > in sf_create_routine(), which is right since we have been been using > TYPE_ENUM_TABLE for m_type so far and mysql.proc takes > enum('FUNCTION','PROCEDURE') for type. This can be solved by either > changing this enum in mysql.proc definition and including TABLE, or storing > table_valued functions as FUNCTION as well. I think first option is better, > including TABLE in the enum, since this will help us in distinguishing > these functions while executing later if needed. It is not clear for me which option is better. On one hand, there is your argument that adding 'TABLE' into the enum will make it easier to tell table functions from regular function. On the other hand, try grepping for 'TYPE_ENUM_FUNCTION' in sql/*.{h,cc}. You find a lot of code that checks if m_type == TYPE_ENUM_FUNCTION. You will need to review each check to see what should be done for TYPE_ENUM_FUNCTION. If we can't decide towards one option or another, let's use whatever you prefer right now. If we discover that that works poorly, we will be able to reverse the decision later on. > The second thing is about return type of function. Right now > sp_returns_type() method is being used to store the return type of > functions. But in our case we have a list of create_fields: > List<Create_field> m_cols_list keeping list of attribute names and their > types for the table. We need to store all these, for this we can either > alter mysql.proc definition to make a field appropriate for storing this > list. The type of the field storing return type right now is longblob which > as i read further on it is a binary large object that can hold a variable > amount of data and are treated as binary strings (byte strings). Maybe we > can try to find a way to store all the attribute information in this field > only, maybe putting in some delimiters between different field types or > some other way since after all its a string. I am still not sure about this > problem. I tried creating this function: CREATE FUNCTION f2(a INT, b VARCHAR(11)) RETURNS VARCHAR(32) DETERMINISTIC BEGIN set @a=3; RETURN @a; END| and mysql.proc.returns has: returns: varchar(32) CHARSET latin1 So, it is a text defition of the return type. It seems natural if for table functions we Is it a problem to do similar thing for Table functions? e.g for a function CREATE FUNCTION f1(a INT, b VARCHAR(11)) RETURNS TABLE t1(id INT, name VARCHAR(11)) BEGIN set @a=3; END| store 'TABLE t1(id INT, name VARCHAR(11)' ? I guess it depends on where the value of mysql.proc.returns is used. Please research this. > > Also, another thing is to store the table alias m_table_alias, because > queries inside the function will be using it. There is no field present > specifically for this in mysql.proc. So either we can create one. There is > a field specific_name in it, as i saw in the code is right now storing the > same thing as name field. So maybe we can use this field to store the table > alias. I would prefer that m_table_alias was stored as part of the expression in mysql.proc.returns field. Can we try that option first? If we figure that can't work, we can think of other options. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB multi-source replication testing at Booking.com
by Karoly Nagy 29 Jun '15

29 Jun '15

Dear MariaDB developers, I'm Károly Nagy working for Booking.com currently testing multi-source replication functionality of MariaDB. Kristian suggested I should reach out to you on this mailing list regarding my questions. We're seeing very high and fluctuating mutex contentions while replicating from two sources (Oracle MySQL 5.6) to a single MariaDB slave. You can see that on the graphs below. The spin waits are relatively [1] aligned but the mutex rounds [2] are 5-10 times higher than it is on the two sources combined together and not consistent. The sources have a relatively constant pattern while the target has dips around 2.5k and spikes up to 8k. The os waits are in completely different order of magnitude [3]. The scenario where values were captured: * Multi-source target is replicating the full dataset of `source 2` and a subset of `source 1` (the hot data) - MariaDB 10.0.16 * Both sources are MySQL 5.6 being part of their replication chain as slaves with log_slave_updates * Source 2 is in normal mode - Oracle MySQL 5.6.17 * Source 1 is catching up from a 1 day replication delay - Oracle MySQL 5.6.24 * All the slaves are warm having the buffer pool fully populated Is this behavior expected? Could you give us some insights on why we could see these results? If there is any more information you need please let me know. Thank you for your help in advance! Every metric is on /10 seconds basis. [1] Mutex spin waits [2] Mutex rounds [3] Mutex OS waits Best regards, -- Károly Nagy System engineer Booking.com <http://booking.com/> BV Rembrandt Square Office, Herengracht 597, 1017 CE Amsterdam Direct +31 (0)20 715 8403

3 4

Re: [Maria-developers] [Commits] 132935a: MDEV-8339 - Server crash during table cache eviction
by Sergei Golubchik 27 Jun '15

27 Jun '15

Hi, Sergey! On Jun 25, svoj(a)mariadb.org wrote: > revision-id: 132935a3ef6578f5fabc765ff6ac966cd3798819 > parent(s): 1f4a89b10ca8df61ec51fcc87e1203dc72579759 > committer: Sergey Vojtovich > branch nick: mariadb > timestamp: 2015-06-25 14:21:16 +0400 > message: > > MDEV-8339 - Server crash during table cache eviction > > TDC_element::free_tables_back() had pre-lfhash leftover code, which referenced > TDC_element::free_tables via TDC_element::share. This is not correct as share > may be NULL (newly inserted or to be removed), instead access free_tables > directly. ok to push Regards, Sergei

1 0

[Maria-developers] MDEV-8214 Asian MB2 charsets: compare broken bytes as "greater than any non-broken character"
by Alexander Barkov 26 Jun '15

26 Jun '15

Hi Sergei, For simplicity I decided to split: MDEV-8036 Fix all collations to compare broken bytes as "greater than any non-broken character" into sub-tasks. Please review a patch for the first sub-task MDEV-8214. The overall plan is: - MDEV-8214, for the Asian charsets with mbmaxlen=2 - MDEV-8215, for the Asian charsets with mbmaxlen=3, based on MDEV-8214 - MDEV-8036, the rest (Unicode character sets) Thanks.

2 4

[Maria-developers] MDEV-7502 Automatic provisioning of slave - error reporting
by Martin Kaluznik 26 Jun '15

26 Jun '15

Hello Kristian, with general idea working (sending of database / table structures + table data), my next plan is to improve / introduce error handling in existing code along with error handling on slave side (ddl events not from provisioning, ignore errors when updating not yet existing rows, ...). What are general guidelines for introduction of new error codes? Is there anything else I need to do except adding it to /sql/share/errmsg-utf8.txt ? I would create only one new error code - ER_PROVISIONING_FAILED - and place specific messages inside formatted text - same way as ER_MASTER_FATAL_ERROR_READING_BINLOG works. It doesn't go well with translations, but... you can decide. Maybe create different error codes at least for errors, where we are expecting user to fix them - table without index, statement mode. Also, how to tell slave from master that provisioning finished? Is introduction of new error code ER_PROVISIONING_DONE acceptable solution? The error itself would never produce error message in logs. Alternatively I can reuse / create similar event to Stop_log_event, note - provisioning is completed after SQL thread processes whole relay log, not when IO thread receives this event, so even reuse would require additional code. And side question, is there any log event other than Query_log_event which can produce ddl statement? Martin

2 1

Re: [Maria-developers] [Commits] 2cf747e: MDEV-8219: enforce_storage_engine cannot be set globally
by Sergei Golubchik 26 Jun '15

26 Jun '15

Hi, Jan! On Jun 26, Jan Lindström wrote: > revision-id: 2cf747e885603122b2adbefc131516cd12dc953c > parent(s): d3b7eb7b99ef7dae79c20b5beb63acdfdf6d8046 > committer: Jan Lindström > branch nick: 10.1-bugs > timestamp: 2015-06-26 08:56:50 +0300 > message: > > MDEV-8219: enforce_storage_engine cannot be set globally > > Change session only variable enforce_storage_engine to be > session variable and make sure that also global value > is used if session variable is not set. Ok to push, thanks! Regards, Sergei

1 0

[Maria-developers] Please review: MDEV-7824 [Bug #68041] Zero date can be inserted in strict no-zero mode through a default value
by Alexander Barkov 25 Jun '15

25 Jun '15

Hi Sergei, Please review a patch for mdev-7824. It's based on a MySQL patch for http://bugs.mysql.com/bug.php?id=68041 and is a blocker for: MDEV-3929 Add full support for auto-initialized/updated timestamp and datetime Thanks.

2 9