July 2013 - developers - lists.mariadb.org

[Maria-developers] MariaDB 10/trunk (re)install destructively overwrites existing my.cnf* @ spec'd SYSCONFDIR
by darx＠sent.com 14 Jul '13

14 Jul '13

Hi, I've a custom my.cnf ls -al /usr/local/etc/mariadb/my.cnf -rw-r--r--+ 1 root root 7.9K Jul 13 13:03 /usr/local/etc/mariadb/my.cnf If I (re)install mariadb 10/trunk from a src build with SYSCONFDIR spec'd for that dir, as done similarly to a MySQL build, cd mariadb/bld cmake .. \ ... -DINSTALL_SYSCONFDIR=/usr/local/etc/mariadb \ ... make make install The INSTALL process *destructively* overwrites the existing my.cnf ... -- Up-to-date: /usr/local/etc/mariadb/logrotate.d/mysql -- Up-to-date: /usr/local/etc/mariadb/init.d/mysql -- Installing: /usr/local/etc/mariadb/my.cnf <================================ !! -- Up-to-date: /usr/local/etc/mariadb/my.cnf.d/client.cnf -- Up-to-date: /usr/local/etc/mariadb/my.cnf.d/mysql-clients.cnf -- Up-to-date: /usr/local/etc/mariadb/my.cnf.d/server.cnf ... with an older default cnf, ls -al /usr/local/etc/mariadb/my.cnf -rw-r--r--+ 1 root root 202 Jul 12 11:31 /usr/local/etc/mariadb/my.cnf with no backup made of the existing cnf, find /usr/local/etc/mariadb -type f -name 'my.cnf*' /usr/local/etc/mariadb/my.cnf This is critically different behavior that MySQL. If that's intended, how are we to exec a mariadb install without overwriting our config files/dirs? If it's not, I suspect this should be fixed. Thanks.

2 2

[Maria-developers] building MariaDB 10/trunk with external SSL generates errors/warnings. Docs are inconsistent. What's the right procedure?
by darx＠sent.com 13 Jul '13

13 Jul '13

hi, building cd /usr/local/src/mariadb bzr log | head -n 5 ------------------------------------------------------------ revno: 3766 [merge] committer: Alexander Barkov <bar(a)mariadb.org> branch nick: maria-10.0.ms timestamp: Wed 2013-07-10 18:46:33 +0400 reading ./BUILD/SETUP.sh ... 120 # SSL library to use.--with-ssl will select our bundled yaSSL # implementation of SSL. To use OpenSSL you will need to specify # the location of OpenSSL headers and libs on your system. # Ex --with-ssl=/usr <================================= OK, do this SSL_LIBRARY=--with-ssl ... configuring @ `cmake` cd /usr/local/src/mariadb rm -rf bld mkdir bld cd bld cmake .. \ -DWITH_SSL=/usr/local/ssl \ -DOPENSSL_INCLUDE_DIR=/usr/local/ssl/include \ -DOPENSSL_SSL_LIBRARY=/usr/local/ssl/lib64/libssl.so \ -DOPENSSL_CRYPTO_LIBRARY=/usr/local/ssl/lib64/libcrypto.so fails, returning, ... CMake Error at cmake/ssl.cmake:90 (MESSAGE): Wrong option for WITH_SSL. Valid values are : yes, no, bundled Call Stack (most recent call first): CMakeLists.txt:260 (MYSQL_CHECK_SSL) ... -- Configuring incomplete, errors occurred! vi ../CmakeLists.txt ... # Add bundled or system zlib. MYSQL_CHECK_ZLIB_WITH_COMPRESS() # Optionally add bundled yassl/taocrypt or system openssl. 260 MYSQL_CHECK_SSL() # Add readline or libedit. MYSQL_CHECK_READLINE() checking vi cmake/ssl.cmake ... # MYSQL_CHECK_SSL # # Provides the following configure options: # WITH_SSL=[yes|no|bundled] MACRO (MYSQL_CHECK_SSL) IF(NOT WITH_SSL) IF(WIN32) CHANGE_SSL_SETTINGS("bundled") ELSE() CHANGE_SSL_SETTINGS("no") ENDIF() ENDIF() IF(WITH_SSL STREQUAL "bundled") MYSQL_USE_BUNDLED_SSL() ELSEIF(WITH_SSL STREQUAL "system" OR WITH_SSL STREQUAL "yes") # Check for system library SET(OPENSSL_FIND_QUIETLY TRUE) INCLUDE(FindOpenSSL) FIND_LIBRARY(CRYPTO_LIBRARY crypto) MARK_AS_ADVANCED(CRYPTO_LIBRARY) INCLUDE(CheckSymbolExists) SET(CMAKE_REQUIRED_INCLUDES ${OPENSSL_INCLUDE_DIR}) CHECK_SYMBOL_EXISTS(SHA512_DIGEST_LENGTH "openssl/sha.h" HAVE_SHA512_DIGEST_LENGTH) SET(CMAKE_REQUIRED_INCLUDES) IF(OPENSSL_FOUND AND CRYPTO_LIBRARY AND HAVE_SHA512_DIGEST_LENGTH) SET(SSL_SOURCES "") SET(SSL_LIBRARIES ${OPENSSL_LIBRARIES} ${CRYPTO_LIBRARY}) SET(SSL_INCLUDE_DIRS ${OPENSSL_INCLUDE_DIR}) SET(SSL_INTERNAL_INCLUDE_DIRS "") SET(SSL_DEFINES "-DHAVE_OPENSSL") CHANGE_SSL_SETTINGS("system") ELSE() IF(WITH_SSL STREQUAL "system") MESSAGE(SEND_ERROR "Cannot find appropriate system libraries for SSL. Use WITH_SSL=bundled to enable SSL su ENDIF() MYSQL_USE_BUNDLED_SSL() ENDIF() ELSEIF(NOT WITH_SSL STREQUAL "no") MESSAGE(SEND_ERROR "Wrong option for WITH_SSL. Valid values are : yes, no, bundled") ENDIF() ENDMACRO() there's no accommodation for "--with-ssl=/path/to/ssl/install". If I instead cmake .. \ -DWITH_SSL=yes \ -DOPENSSL_INCLUDE_DIR=/usr/local/ssl/include \ -DOPENSSL_SSL_LIBRARY=/usr/local/ssl/lib64/libssl.so \ -DOPENSSL_CRYPTO_LIBRARY=/usr/local/ssl/lib64/libcrypto.so it returns, ... -- Running cmake version 2.8.11.1 -- MariaDB 10.0.3 -- Packaging as: mariadb-10.0.3-Linux-x86_64 -- Found OpenSSL: /usr/local/ssl/lib64/libssl.so;/usr/local/ssl/lib64/libcrypto.so (found version "1.0.1e") ... -- Configuring done then a bunch of these messages: ... CMake Warning at cmake/plugin.cmake:184 (ADD_LIBRARY): Cannot generate a safe runtime search path for target feedback because files in some directories may conflict with libraries in implicit directories: runtime library [libcrypto.so.1.0.0] in /usr/lib64 may be hidden by files in: /usr/local/ssl/lib64 Some of these libraries may not be found correctly. Call Stack (most recent call first): plugin/feedback/CMakeLists.txt:20 (MYSQL_ADD_PLUGIN) CMake Warning at cmake/libutils.cmake:259 (ADD_LIBRARY): Cannot generate a safe runtime search path for target libmysql because files in some directories may conflict with libraries in implicit directories: runtime library [libcrypto.so.1.0.0] in /usr/lib64 may be hidden by files in: /usr/local/ssl/lib64 Some of these libraries may not be found correctly. Call Stack (most recent call first): libmysql/CMakeLists.txt:374 (MERGE_LIBRARIES) CMake Warning at client/CMakeLists.txt:78 (ADD_EXECUTABLE): Cannot generate a safe runtime search path for target async_example because files in some directories may conflict with libraries in implicit directories: runtime library [libcrypto.so.1.0.0] in /usr/lib64 may be hidden by files in: /usr/local/ssl/lib64 Some of these libraries may not be found correctly. ... What's the right method to find/use external SSL libs/headers in a MariaDB build without errors or warnings? --darx

2 1

[Maria-developers] Doubts about index and optimization
by Roberto Spadim 13 Jul '13

13 Jul '13

Hi guys i have two doubts... First the r-tree index Could i index two integer or float or decimal columns using rtree And queries like X>=1 and x<=2 and y>=1 and y<=2 Be optimized using rtree? Another doubt... i dont have a mariadb now to test, i'm in cellphone... this kind of query is optimized if i have a x,y btree index? Btree will use x only, or x and y? Second doubt I'm thinking about enum rewrite optimzation and i didn't write anything yet, just studying mariadb code... When using queries like this Where x<=5 Or where x>=5 Or without = part Does mysql know what's the max and min value of x, and rewrite it to something X<=5. To. X in (0,1,2,3,4,5) when x is a integer for example and min(x)=0 I'm thinking about this, in conditions like the rtree optimization, example... we could optimize x and y with something like X>1 and x<5 and y>1 and y<5 to X in (2,3,4) and y in (2,3,4) It's something like enum optimization but using integers and max min informations Well i don't know the full code but i didn't found this information (i don't remember at least) Anyone know something about this in source? If not i will consider this optimization when doing the enum optimization (i don't remember the mdev number now, but anyone interested in this optimization can search in jira and find it, i reported and done any job yet just the raw and high implementation idea) Thanks guys

1 0

[Maria-developers] OLAP
by Roberto Spadim 12 Jul '13

12 Jul '13

Hi guys, there's any good OLAP tool for mariadb? maybe one that we could package with mariadb as default? -- Roberto Spadim

2 5

[Maria-developers] Status on MDEV-4506, parallel replication
by Kristian Nielsen 12 Jul '13

12 Jul '13

So I have been working for some weeks now on implementing MDEV-4506. This task is about making the slave apply (some) events in parallel threads, to speed up replication and reduce the risk of slave not being able to keep up with a busy master. Events are applied in parallel on the slave if they were group-committed together on the master. This is an easy way to detect transactions that are independent. Note that this is transparent to applications; while transactions are executed in parallel on the slave, they are still committed in the same order as on the master. I also added parallel execution of events with different GTID domain id. This makes testing a lot easier (no need to carefully arrange timing to get a specific group commit on the master), and also really is the whole point of much of my hard work on GTID. So if we have multi-source M1->S1, M2->S1, and S1->S2, then S2 will be able to execute events from M1 in parallel with those from M2, just like S1 can. And the user can explicitly set the domain_id different for eg. a long-running ALTER or UPDATE, and this way get it to run in parallel and not cause a huge replication delay. ---- On the master, I added to each GTID event a commit_id. If two transactions group-commit together, they are binlogged with the same commit_id; if not, they get different commit_ids. Thus, the slave can detect the possibility of executing two transactions in parallel by checking if the commit_ids are equal. On the master, I implemented --binlog-commit-wait-count=N and --binlog-commit-wait-usec=T. A transaction will wait at most T microseconds for at least N transactions to queue up and be ready for group commit. This allows to deliberately delay transactions on the master in order to get bigger group commits and thus better opportunity for parallel execution (and again it makes testing easier). On the slave, I implemented --slave-parallel-threads=N. If N>0, that many threads will be spawned, and events will be executed in parallel (if possible) by those threads. ---- The current code is pushed here (it is based on 10.0-base): lp:~maria-captains/maria/10.0-knielsen It is still far from finished, but it now works sufficiently that I could do some basic benchmarking (just on my laptop, completely unscientifically). First, I prepared a binlog on the master with plenty of opportunity for parallel replication. I started the master with --binlog-commit-wait-count=20 --binlog-commit-wait-usec=1000000. I then ran this Gypsy script: ----------------------------------------------------------------------- i|1|DROP TABLE IF EXISTS t1 i|1|CREATE TABLE t1 (a INT PRIMARY KEY, b VARCHAR(100)) ENGINE=InnoDB p|1|REPLACE INTO t1 (a,b) VALUES (? MOD 10000, ?)|int,varchar /home/knielsen/my/gypsy/words ----------------------------------------------------------------------- gypsy --queryfile=simple_replace_load.gypsy --duration=20 --threads=40 This results in a binlog with about 65k updates to the table, group-committed in batches of 20 transactions. I then started a fresh slave and let it replicate everything with START SLAVE UNTIL; the time to replicate all the events is then easy to see in the slave error log. The time to replicate everything with unmodified 10.0-base was 99 seconds. With --slave-parallel-threads=25, it was just 22 seconds. So that is a 4.5 times speedup, which is quite promising. Also note that at 22 seconds, the slave is within 10% of the speed of the master. ---- But as I said, there is still significant work left to do. I put a ToDo list at the top of sql/rpl_parallel.cc. Some of the big remaining issues: 1. The existing code is not thread-safe for class Relay_log_info. This class contains a bunch of stuff that is specific to executed transactions, not related to relay-log at all. This needs to be moved to the new struct rpl_group_info I introduced, and all code updated to pass around a pointer to that struct instead. There may also be a need to add additional locking on Relay_log_info, existing code needs review for this. 2. Error handling needs to be implemented, it is rather more complex in the parallel case. If one transaction fails (and retry also fails), then we need to somehow get hold of all later transactions that are in the process of parallel execution, and abort them + roll them back. Otherwise we get inconsistent binlog position for the next slave restart. 3. In the old code, when the SQL thread is stopped, it has logic to let the current event group (=transaction) replicate to completeness first, with a timeout to force a stop in the middle of the event group if eg. the master has disappeared. This logic needs to be re-implemented to work when having any number of event groups executing in parallel. (It is important to let the groups complete execution when doing non-transactional stuff that cannot be rolled back, otherwise again the slave position becomes inconsistent for next slave restart). So as you see, there is quite a bit of work left on this (as well as on GTID). So I would very much welcome any help on this to avoid causing delays for 10.0-GA ... - Kristian.

1 0

[Maria-developers] a bug of ding-qing parallel replication
by nanyi607rao 11 Jul '13

11 Jul '13

hi guys, I find a bug of ding-qing parallel replication. let's consider a table like this: create table t1( a int(11) NOT NULL DEFAULT '0', b varchar(10), PRIMARY KEY (a) )ENGINE=InnoDB if we do transactions on master(binlog_format=ROW) like this: trans 1: begin; insert into t1 values(1, "tt"); update t1 set a=2 where a=1; commit; trans 2; begin; update t1 set b="kk" where a=2; commit; then this two transactions binlog transmit to the slave, because trans 1 and trnas 2 has different primary keys, it would disaptch in different threads like this: thread 1 do trans 1 thread 2 do trans 2 thread 2 may update t1 before thread 1, then it will fail. maybe thread 2 can retry sever times ,but it don't solve this problem compeletly.. 2013-07-11 nanyi607rao

3 2

[Maria-developers] deferred_event
by nanyi607rao 10 Jul '13

10 Jul '13

Hi guys, as I know, when replication filter used, the defered events is a container hold on events(such as Intvar-, Rand-, User_var- log_event)that be applied delay. why those events need be applied delay? and what is parent Query mentioned in codes? Thanks! 2013-07-10 nanyi607rao

2 1

[Maria-developers] buildbot config
by Leif Walsh 09 Jul '13

09 Jul '13

Hi, I'm learning buildbot and I was wondering if I could take a look at the master.cfg for buildbot.askmonty.org as a reference for learning common practices and things like that. Is it available somewhere? -- Cheers, Leif

2 2

[Maria-developers] merge for MySQL56 FSP data types
by Alexander Barkov 09 Jul '13

09 Jul '13

Hi Serg, Please review the patch merging MySQL-5.6 fractional second precision enabled data types. Thanks.

3 7

[Maria-developers] Connect: compilation warnings
by Alexander Barkov 08 Jul '13

08 Jul '13

Hi Oliver, I fixed almost all compilation warnings that happened in Connect on my Fedora 17 laptop with gcc-4.7.2. Most of them were "variable is initialized but not used" and "no previous declaration". There were not serious, but very unpleasant for developers. (Note, we try to keep the tree in a shape that it compiles without warnings. Connect must follow this rule as well!!!) I think there was a bug in one place: how SQLSetStmtAttr() was called. See "bzr diff -c 3827" for details. The value of m_RowsetSize was erroneously passed, but SQLSetStmtAttr() expects the address of a SQLUINT variable (not a value!). See the ODBC docs for the details: http://msdn.microsoft.com/en-us/library/windows/desktop/ms712631%28v=vs.85%… One compilation warning still remains: maria-10.0-connect/storage/connect/tabsys.cpp:279:26: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast] It looks like a bug. Cast from 4-byte integer to 8-byte pointer is obviously wrong. Please address the problem. Thanks.

1 0