March 2016 - developers - lists.mariadb.org

Re: [Maria-developers] [Maria-discuss] Collations, trailing spaces and unique indexes
by Alexander Barkov 17 Mar '16

17 Mar '16

Hi Binarus, On 03/16/2016 06:13 PM, Binarus wrote: >> I just wanted to split the task into these steps: >> >> 1. Default and _bin collations. >> This is what MDEV-9711 is about. >> >> 2. Unicode Collation Algorithm (UCA) based collations >> (e.g. utf8_unicode_nopad_ci) >> >> >> 3. The rest (not covered by the previous steps) >> This will include latin1_general_nopad_cs, latin2_czech_nopad_cs >> >> >> This is why asked in the previous letter if utf8_unicode_nopad_ci would >> work for you. >> Also, which MariaDB version do you need this for? Will 10.1 work? >> > > Thank you very much for this explanation. > > Regarding the version: Our O/S usually is Debian which generally uses software which is quite old. This means that I will have to compile MariaDB myself in every case. > The current project where this new collations are needed will probably take about three months before being finished. It would be very nice if we could have the new collations until then. > In the meantime, we'll continue development with MySQL and switch to MariaDB as soon as the new collations are available. Excellent. The fact that you compile from sources makes the things easier. I guess we can add new collations officially into 10.2. Then after 3 month if you'll feel that 10.2 is not mature enough for you, we can make a custom non-official 10.1 tree with the new collations back-ported (it should be trivial to do). > > ci collations would be sufficient as a starting point. Nevertheless, I think there must be a nopad collation for every collation which already exists. > Personally, since we are using utf8 (and no other encoding) in every part of every project, I would prefer to have *all utf8* nopad collations before >having any other nopad collation. But that's just my personal opinion. >We will be grateful for any progress regardless of the order of things being implemented. Thanks. Then we're flexible to implement #2 first, and wait for #1 to be done under terms of Google Summer Of Code 2016. > > Binarus > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp >

1 0

Re: [Maria-developers] [Maria-discuss] Collations, trailing spaces and unique indexes
by Binarus 17 Mar '16

17 Mar '16

Alexander, I'd like to continue our conversation from Maria-discuss. >> - I have found my_strnxfrm_unicode in the same file and will need more time to make my opinion of how difficult it will be >> (I don't know what a weight is, so I currently try to understand what the function does at all). > > This function is used to create sort keys for non-indexed ORDER BY, > for these cases: > > - ORDER BY on an expression > - ORDER BY on a column that does not have an index > > > The idea is exactly the same with the C function strxfrm. > See "man strxfrm". > > The code implements non-indexed ORDER BY in filesort.cc > as follows: > > 1. It calls *_strnxfrm_* functions for all records and converts > CHAR/VARCHAR/TEXT values into their fixed length binary sortable keys. > > 2 . Then executes binary sorting on these keys. > > By the way, fixing this function might be tricky. > > > Currently my_strnxfrm_unicode() pads the tail using weights of the SPACE > character. > The NO PAD version will need to pad the tail using a weight which is > less than the weight of the smallest possible character. > > This should be easy for UCA bases collations (e.g. > utf8_unicode_nopad_ci), because the smallest possible > character in UCA collations is "U+0009 HORIZONTAL TABULATION", > and its weight is 0x0201. So we can just pad the sort key > using a smaller value 0x0200. > > > But I'm not sure yet what to do with 8-bit collations, > which usually use 0x00 as weight for the smallest character. > So we don't have a smaller value. > There are two options here: > > 1. Pad with 0x00. But this will mean that 'aaa<min>' and just 'aaa' > will have unpredictable order when doing ORDER BY without an index > (where <min> is the smallest possible character in the collation). > > As the smallest character in non-UCA collations is usually > "U+0000 NULL", this will mean that 'aaa\0' and just 'aaa' > will have unpredictable order. > > 2. Reserve extra bytes at the end of the key, to store the true length, so > - 'aaa\0' will have the key '4141410004' > - 'aaa' will have the key '4141410003', and therefore will always > be sorted before 'aaa\0'. > > I'm inclined towards #2, to have consistent ORDER BY behavior > with and without indexes. Wow, thanks for explaining. I now have learned that I probably can't assist you in solving that problem (I haven't been in unicode, weights, sorting and character sets very deep until now). But if it helps, you eventually could break the problem into two steps: 1) Fix the problem with unique indexes 2) Fix ordering for any possible situation The problems we have encountered in real life so far are all related to 1), i.e. we would be happy if unique indexes would consider 'a' and 'a ' being different values. In contrast, ordering is not that important for us. I don't know what others are thinking about that, though. >> >> - My main problem: I did not find my_strnncollsp_utf8_general_ci anywhere (nor in the same neither in any other file). Where is it? > > The function name is just "my_strnncollsp_utf8". OK, thank you very much. I must have misunderstood something. >> Furthermore, studying the code has led to some questions; for example, there already seems to be a #define which controls the padding-when-comparing mode, but only for the _cs collations? > > Can you please clarify which lines do you mean? I am referring to the file ctype-utf8.c, line 5698. There is: #ifndef VARCHAR_WITH_DIFF_ENDSPACE_ARE_DIFFERENT_FOR_UNIQUE diff_if_only_endspace_difference= 0; #endif I haven't been able yet to thoroughly analyze where and how this is used, but at least somebody obviously already had some thoughts whether or not trailing spaces should make a difference in UNIQUEs :-) Regards, Binarus

2 1

Re: [Maria-developers] bf2eaa7: MDEV-9382: After updating mariadb server apt-configure fails
by Sergei Golubchik 17 Mar '16

17 Mar '16

Hi, Nirbhay! On Mar 02, Nirbhay Choubey wrote: > revision-id: bf2eaa7c1548f07ce5de13a406ff9183f082b8b3 (mariadb-10.1.12-4-gbf2eaa7) > parent(s): fd741c1a12e78035dcef9a5684b530fe09cbfdde > author: Nirbhay Choubey > committer: Nirbhay Choubey > timestamp: 2016-03-02 09:41:03 -0500 > message: > > MDEV-9382: After updating mariadb server apt-configure fails > > When acting as a Galera receiver node, server startup may take > more than 30 secs (the current default) as it has to wait for > SST/IST operation to complete besides spending some time doing > wsrep recovery. > > In specific case of this MDEV, the server startup timedout as > it took ~48 secs for the node to fully start. > > A viable workaround would be to adjust the MYSQLD_STARTUP_TIMEOUT > env to a higher value, but raising the default to 60 secs should > fix scenarios like these. > > --- > debian/mariadb-server-10.1.mysql.init | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/debian/mariadb-server-10.1.mysql.init b/debian/mariadb-server-10.1.mysql.init > index 9e098b4..005d0e8 100644 > --- a/debian/mariadb-server-10.1.mysql.init > +++ b/debian/mariadb-server-10.1.mysql.init > @@ -109,7 +109,7 @@ case "${1:-''}" in > /usr/bin/mysqld_safe "${@:2}" > /dev/null 2>&1 & > > # 6s was reported in #352070 to be too little > - for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-30}"); do > + for i in $(seq 1 "${MYSQLD_STARTUP_TIMEOUT:-60}"); do > sleep 1 This doesn't fix much. Next time somebody will need 61 second and you'll adjust to 120? SST might take many minutes and it's not a good idea to have every user wait this long. Instead you should do what MDEV-8509 says. Add a line like [ -r /etc/default/mariadb ] && . /etc/default/mariadb then one can have a very personal setting of MYSQLD_STARTUP_TIMEOUT in /etc/default/mariadb Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

3 3

[Maria-developers] [GSoC 2016] NO PAD Collations
by Даниил Медведев 16 Mar '16

16 Mar '16

Hi, I am Daniil Medvedev from Lomonosov Moscow State University, a 3rd year student, pursuing my bachelor’s degree in Computer Science. I was interested in project idea NO PAD Collations, and I wish to contribute to MariaDB using my skills and ideas. Technologies that i know: C/C++, git, linux, shell, cmake, some databases. My experience: Some projects on github: https://github.com/medvdanil And I contributed one open source NoSQL DBMS on github last summer. Also I am ready to perform some opening task or do commit in your repo, if it helps me to join GSoC. Thank you, Daniil Medvedev

3 2

[Maria-developers] Gsoc 2016 Mdev 371 Unique index for blob
by Sachin Setia 16 Mar '16

16 Mar '16

Dear Developers, I was doing prototype for this project as Mr Sergei Golubchik suggested "This task is about creating MI_UNIQUEDEF "uniques" instead of MI_KEYDEF "keys" for long unique constraints." Here is my output MariaDB [sachin]> create table tbl(int_key int primary key , blob_key blob unique); Query OK, 0 rows affected (0.03 sec) MariaDB [sachin]> insert into tbl values(1,1); Query OK, 1 row affected (0.01 sec) MariaDB [sachin]> insert into tbl values(2,1); ERROR 1062 (23000): Duplicate entry '1' for key 'blob_key' MariaDB [sachin]> insert into tbl values(2,2); Query OK, 1 row affected (0.00 sec) MariaDB [sachin]> insert into tbl values(3,3); Query OK, 1 row affected (0.00 sec) MariaDB [sachin]> select * from tbl; +---------+----------+ | int_key | blob_key | +---------+----------+ | 1 | 1 | | 2 | 2 | | 3 | 3 | +---------+----------+ 3 rows in set (0.00 sec) I am assuming that unique blob will have 0 zero key . Should i mail you the source code.Please let me know Regards sachin

2 4

[Maria-developers] Introduction - GSOF16
by Chandu Maddila 16 Mar '16

16 Mar '16

Hi Team, I'm Chandra Sekhar Maddila from India and currently masters student of BITS-Pilani. I'm pretty interested in working with MariaDB team as part of GSOF. Some of the items that I'm interested in are MXS-1, MDEV-8100, MDEV-5662, MDEV-371 Here's a quick summary of my work experience and Skill set. - Currently a student of BITS-Pilani <http://www.bits-pilani.ac.in/> (Masters, Computer Science). - Total experience of 6 years in IT industry, as a software developer (My last stint was in *Microsoft Research *[MSR] - India as a research SDE). - Have been working with designing and development of RDBMS systems : With Oracle earlier and SQL Server after joining Microsoft - Involved in various projects like :* Implementing column store indexes in SQL Server, Designing a cloud based monitoring systems for database performance monitoring, tools to migrate data from Oracle to SQL Server, Implementing HA Solution (AlwaysOn) for SQL Server etc...* - Languages/Frameworks: - Asp.Net, C#, Jquery, Javascript, HTML5, CSS, Bootstrap, Knockout, SOAP, REST, JSON, Web Services - WCF, MVVM, XAML, Windows App Development, Azure Cloud services, Azure web services - Java, J2EE, UNIX Shell scripting I worked extensively on both *UNIX and Windows platforms*. *Database domain expertise:* *Oracle* + In depth understanding of Oracle database internals and architecture (Memory management, storage engine and optimizer) + *Performance tuning*: Instance and SQL level tuning with the help of *AWR, Statspack, TKPROF* reports and Explain plans and tools like *OEM* and *TOAD* + Oracle *11g, 10g, 9i* Database Setup, Administration, Monitoring on various flavors of UNIX: RHEL, AIX, SUN Solaris, HPUX – Itanium chipset; Good understanding of Oracle internals and Oracle system architecture + Oracle *RAC, ASM and Grid infrastructure* setup, maintenance and administration; Non RAC to RAC, Non-ASM to ASM conversions; Familiarity with OS level clustering techniques like *HACMP* + All kinds of Database *Backup and Recovery* methodologies: *RMAN *(Hot, Cold, Active Database Duplication)*, Datapump, SQL Loader* etc…; Familiarity with 3rd party backup tools like *CA ArcServe* *SQL Server* + Working experience with SQL 2014, 2012, 2008R2, 2008 and 2005 on different versions of Windows + Very good at *troubleshooting complex issues and working on outages/showstoppers*. + Very good in *database performance and query tuning*: Analyzing *execution plans* and rewriting queries + Administering SQL Server High Availability and Disaster Recovery technologies like *AlwaysOn, Failover Clustering *and *Database mirroring*. + Proficiency and expertise in all varieties of *SQL Server Replication*: *Snapshot Replication, Transactional replication, Merge replication and Change tracking* + Expertise in handling all sorts of *backup technologies: Native, VSS, VDI, Backup to URL etc…* + Expert in handling *database corruption cases and recovering* the databases + Expertise in T-SQL writing stored procedures, triggers and functions. For any queries, feel free to reach out to me @ chandu.maddila(a)gmail.com Thanks, Chandra

2 1

Re: [Maria-developers] Gsoc 2016 Mdev 371 Unique index for blob
by Sergei Golubchik 15 Mar '16

15 Mar '16

Hi, Sachin! First: please send questions like this to maria-developers(a)lists.launchpad.net it's a public mailing list dedicated to MySQL and MariaDB internals, source code, and related things. I am subscribed, so I'll see you mail there, and you may be sure I will, because it won't be accidentally catched by my spam filter, or sorted out in some obscure folder. Furthermore other subscribers will see your question and could reply if I will be not available (e.g. I could be travelling). Thank you. On Mar 14, Sachin Setia wrote: > Thank you sir for your reply but my question is what this pack flag will do > and basically what is pack record will you please suggest source file in > which i can look into to get more understanding of pack length ,pack flag > and pack record There isn't much I can answer - your question is too generic. "pack_flag" is a set of flags that is stored per field in the frm file and it describes some aspects of the field. Search for "pack_flag" in the source tree. If you ask a specific question, I'll be able to provide a more detailed answer. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] a0c06ba: Preliminary implementation for the aggregate sum function as a window function
by Sergey Petrunia 14 Mar '16

14 Mar '16

Hi Vicentiu, I think it add_helper() is a really poor choice of name when the function can do removal. Maybe add_or_remove() name would be better? On Mon, Mar 14, 2016 at 03:44:48PM +0200, Vicentiu Ciorbaru wrote: > revision-id: a0c06ba1edb54c8c4705189c0455137a85658297 (mariadb-10.1.8-154-ga0c06ba) > parent(s): ce8a0d8e19e7bf1486a2c80ff6b7a30ef35bf99f > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2016-03-14 15:42:00 +0200 > message: > > Preliminary implementation for the aggregate sum function as a window function > > This implementation does not deal with the case where removal of > elements from the window frame causes the item to turn to a null value. > > --- > mysql-test/r/win_sum.result | 42 ++++++++++++++++++++++++++++++++++++++++++ > mysql-test/t/win_sum.test | 32 ++++++++++++++++++++++++++++++++ > sql/item_sum.cc | 29 +++++++++++++++++++++++++---- > sql/item_sum.h | 4 ++++ > sql/sql_window.cc | 1 + > 5 files changed, 104 insertions(+), 4 deletions(-) > > diff --git a/mysql-test/r/win_sum.result b/mysql-test/r/win_sum.result > new file mode 100644 > index 0000000..1db6c6e > --- /dev/null > +++ b/mysql-test/r/win_sum.result > @@ -0,0 +1,42 @@ > +create table t1 ( > +pk int primary key, > +a int, > +b int, > +c real > +); > +insert into t1 values > +(101 , 0, 10, 1.1), > +(102 , 0, 10, 2.1), > +(103 , 1, 10, 3.1), > +(104 , 1, 10, 4.1), > +(108 , 2, 10, 5.1), > +(105 , 2, 20, 6.1), > +(106 , 2, 20, 7.1), > +(107 , 2, 20, 8.15), > +(109 , 4, 20, 9.15), > +(110 , 4, 20, 10.15), > +(111 , 5, NULL, 11.15), > +(112 , 5, 1, 12.25), > +(113 , 5, NULL, 13.35), > +(114 , 5, NULL, 14.50), > +(115 , 5, NULL, 15.65); > +select pk, a, b, sum(b) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING), > +sum(c) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) > +from t1; > +pk a b sum(b) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) sum(c) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) > +101 0 10 20 3.2 > +102 0 10 20 3.2 > +103 1 10 20 7.199999999999999 > +104 1 10 20 7.199999999999999 > +105 2 20 40 13.2 > +106 2 20 60 21.35 > +107 2 20 50 20.35 > +108 2 10 30 13.250000000000002 > +109 4 20 40 19.3 > +110 4 20 40 19.3 > +111 5 NULL 1 23.4 > +112 5 1 1 36.75 > +113 5 NULL 1 40.1 > +114 5 NULL 0 43.5 > +115 5 NULL 0 30.15 > +drop table t1; > diff --git a/mysql-test/t/win_sum.test b/mysql-test/t/win_sum.test > new file mode 100644 > index 0000000..3c12b08 > --- /dev/null > +++ b/mysql-test/t/win_sum.test > @@ -0,0 +1,32 @@ > +create table t1 ( > + pk int primary key, > + a int, > + b int, > + c real > +); > + > + > +insert into t1 values > +(101 , 0, 10, 1.1), > +(102 , 0, 10, 2.1), > +(103 , 1, 10, 3.1), > +(104 , 1, 10, 4.1), > +(108 , 2, 10, 5.1), > +(105 , 2, 20, 6.1), > +(106 , 2, 20, 7.1), > +(107 , 2, 20, 8.15), > +(109 , 4, 20, 9.15), > +(110 , 4, 20, 10.15), > +(111 , 5, NULL, 11.15), > +(112 , 5, 1, 12.25), > +(113 , 5, NULL, 13.35), > +(114 , 5, NULL, 14.50), > +(115 , 5, NULL, 15.65); > + > +--sorted_result > +select pk, a, b, sum(b) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING), > + sum(c) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) > + > +from t1; > + > +drop table t1; > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index c78c206..3f4853a 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -1318,25 +1318,39 @@ void Item_sum_sum::fix_length_and_dec() > bool Item_sum_sum::add() > { > DBUG_ENTER("Item_sum_sum::add"); > + add_helper(false); > + DBUG_RETURN(0); > +} > + > +void Item_sum_sum::add_helper(bool perform_removal) > +{ > + DBUG_ENTER("Item_sum_sum::add_helper"); > if (Item_sum_sum::result_type() == DECIMAL_RESULT) > { > my_decimal value; > const my_decimal *val= aggr->arg_val_decimal(&value); > if (!aggr->arg_is_null(true)) > { > - my_decimal_add(E_DEC_FATAL_ERROR, dec_buffs + (curr_dec_buff^1), > - val, dec_buffs + curr_dec_buff); > + if (perform_removal) > + my_decimal_sub(E_DEC_FATAL_ERROR, dec_buffs + (curr_dec_buff ^ 1), > + dec_buffs + curr_dec_buff, val); > + else > + my_decimal_add(E_DEC_FATAL_ERROR, dec_buffs + (curr_dec_buff ^ 1), > + val, dec_buffs + curr_dec_buff); > curr_dec_buff^= 1; > null_value= 0; > } > } > else > { > - sum+= aggr->arg_val_real(); > + if (perform_removal) > + sum-= aggr->arg_val_real(); > + else > + sum+= aggr->arg_val_real(); > if (!aggr->arg_is_null(true)) > null_value= 0; > } > - DBUG_RETURN(0); > + DBUG_VOID_RETURN; > } > > > @@ -1386,6 +1400,13 @@ my_decimal *Item_sum_sum::val_decimal(my_decimal *val) > return val_decimal_from_real(val); > } > > +void Item_sum_sum::remove() > +{ > + DBUG_ENTER("Item_sum_sum::remove"); > + add_helper(true); > + DBUG_VOID_RETURN; > +} > + > /** > Aggregate a distinct row from the distinct hash table. > > diff --git a/sql/item_sum.h b/sql/item_sum.h > index f1f5b22..24dfd9b 100644 > --- a/sql/item_sum.h > +++ b/sql/item_sum.h > @@ -773,6 +773,10 @@ class Item_sum_sum :public Item_sum_num, > return has_with_distinct() ? "sum(distinct " : "sum("; > } > Item *copy_or_same(THD* thd); > + void remove(); > + > +private: > + void add_helper(bool perform_removal); > }; > > > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index df9899e..f0d74b1 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -1584,6 +1584,7 @@ bool JOIN::process_window_functions(List<Item> *curr_fields_list) > } > case Item_sum::COUNT_FUNC: > case Item_sum::SUM_BIT_FUNC: > + case Item_sum::SUM_FUNC: > { > /* > Frame-aware window function computation. It does one pass, but > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Gsoc 2016 Mdev 371 Unique index for blob
by Sachin Setia 14 Mar '16

14 Mar '16

Hello Developers My name is sachin , a third year undergrad student in India.I am interested in gsoc 2016 I was debugging the maria db for following query create table c4(a1 int primary key,b1 int unique,c1 blob) engine=myisam; In mysql_prepare_create_table there is a function name prepare_create_field which sets the sql_field pack flag . for a1 field it sets the pack flag =27 . do not quiet get what it means.

2 1

Re: [Maria-developers] c0b1148: MDEV-9659 : AWS KMS encryption plugin
by Sergei Golubchik 12 Mar '16

12 Mar '16

Hi, Wlad! Just a couple of comments: On Mar 11, wlad(a)mariadb.com wrote: > + # Relax AWS C++ SDK unreasonably high requirements for CMake version. Use replace utility (from MariaDB build) > + # to patch their CMakeLists.txt Nice solution. But you might need to add a dependency, to ensure that this external project target is built after replace. Unless cmake can, somehow, derive it automatically (you can test it by building just this specific target in a clean tree). > + MY_DIR *dirp = my_dir(".", MYF(0)); > + if (!dirp) > + { > + sql_print_error("Can't scan current directory"); > + DBUG_RETURN(-1); > + } > + for (unsigned int i=0; i < dirp->number_of_files; i++) > + { > + > + KEY_INFO info; > + if (extract_id_and_version(dirp->dir_entry[i].name, &info.key_id, &info.key_version) == 0) > + { > + key_info_cache[KEY_ID_AND_VERSION(info.key_id, info.key_version)]= info; > + latest_version_cache[info.key_id]= max(info.key_version, latest_version_cache[info.key_id]); > + } > + } > + DBUG_RETURN(0); my_dir() uses malloc(), don't forget my_dirend() to free the memory. Regards, Sergei

2 1