- developers - lists.mariadb.org

[Maria-developers] MariaDB 10.4 on AIX. Issue with: HAVE_POOL_OF_THREADS
by REIX, Tony 18 Apr '19

18 Apr '19

Hi, In my environment (AIX & MariaDB v10.4.3 or 10.4.4), I have an issue with HAVE_POOL_OF_THREADS. When building, I have the following trace and error (full line of g++ compilation at end of email) : [ 44%] Linking CXX executable explain_filename-t /opt/freeware/bin/cmake -E cmake_link_script CMakeFiles/explain_filename-t.dir/link.txt --verbose=1 /opt/freeware/bin/g++ ........... -o explain_filename-t .... ld: 0711-317 ERROR: Undefined symbol: .TP_pool_generic::TP_pool_generic() ld: 0711-317 ERROR: Undefined symbol: .TP_pool_generic::init() This seems to be due to the following lines of the file: sql/CMakeLists.txt : IF (CMAKE_SYSTEM_NAME MATCHES "Linux" OR CMAKE_SYSTEM_NAME MATCHES "Windows" OR CMAKE_SYSTEM_NAME MATCHES "SunOS" OR HAVE_KQUEUE) ADD_DEFINITIONS(-DHAVE_POOL_OF_THREADS) IF(WIN32) SET(SQL_SOURCE ${SQL_SOURCE} threadpool_win.cc) ENDIF() SET(SQL_SOURCE ${SQL_SOURCE} threadpool_generic.cc) ENDIF() In case the Operating System is not Linux, Windows, or SunOS, then there is NO -DHAVE_POOL_OF_THREADS and the file: threadpool_generic.cc is NOT compiled. However, the C++ code of MariaDB still looks at least for 2 routines of threadpool_generic.cc : TP_pool_generic::TP_pool_generic() TP_pool_generic::init() but these routines are not available since threadpool_generic.cc was not compiled. So, in my opinion, there are some #ifdef HAVE_POOL_OF_THREADS lines that are missing in common threadpool code that would prevent to use threadpool generic routines. However, I do not master this code. Why not implement on AIX the generic pool of thread and thus add AIX to the above IF( ) test of sql/CMakeLists.txt ?? The reason if that, out of the 4 possibilities listed in sql/threadpool_generic.cc : epoll, KQUEUE, port, and IOCP, none of them are available on AIX with the EXACT required behavior. We have to implement something miming epoll(), we think. Before fixing this performance improvement, we'd like to be able to build & test MariaDB for all its other features. sql/threadpool_generic.cc : #ifdef __linux__ #include <sys/epoll.h> typedef struct epoll_event native_event; #elif defined(HAVE_KQUEUE) #include <sys/event.h> typedef struct kevent native_event; #elif defined (__sun) #include <port.h> typedef port_event_t native_event; #elif defined (HAVE_IOCP) typedef OVERLAPPED_ENTRY native_event; #else #error threadpool is not available on this platform <<---- AIX #endif Regards, Tony /opt/freeware/bin/g++ -O2 -g -L/opt/freeware/lib64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -maix64 -Wl,-bbigtoc -fPIC -O2 -g -L/opt/freeware/lib64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -maix64 -Wl,-bbigtoc -fPIC -pie -fPIC -fno-rtti -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized -D_FORTIFY_SOURCE=2 -DDBUG_OFF -Wl,-blibpath:/opt/freeware/lib:/opt/freeware/lib64:/usr/lib:/lib -lpgport CMakeFiles/explain_filename-t.dir/explain_filename-t.cc.o -o explain_filename-t -Wl,-bnoipath -Wl,-blibpath:/opt/freeware/lib64:/opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/8.2.0/ppc64:/opt/freeware/lib/ppc64:/opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/8.2.0:/opt/freeware/lib:/usr/lib:/lib -lpthreads ../../sql/libsql.a ../mytap/libmytap.a ../../storage/archive/libarchive.a ../../storage/blackhole/libblackhole.a ../../storage/csv/libcsv.a ../../storage/federatedx/libfederatedx.a ../../storage/heap/libheap.a ../../storage/innobase/libinnobase.a -llz4 ../../storage/maria/libaria.a ../../storage/myisam/libmyisam.a ../../storage/myisammrg/libmyisammrg.a ../../storage/sequence/libsequence.a ../../plugin/feedback/libfeedback.a ../../plugin/user_variables/libuser_variables.a ../../plugin/userstat/libuserstat.a ../../sql/libpartition.a ../../sql/libsql_sequence.a ../../mysys/libmysys.a ../../mysys_ssl/libmysys_ssl.a ../../dbug/libdbug.a ../../mysys/libmysys.a ../../mysys_ssl/libmysys_ssl.a ../../dbug/libdbug.a /usr/lib64/libz.so -lm ../../extra/yassl/libyassl.a ../../extra/yassl/taocrypt/libtaocrypt.a ../../strings/libstrings.a ../../vio/libvio.a ../../pcre/libpcre.a ../../wsrep-lib/src/libwsrep-lib.a -lpthread -ldl ../../wsrep-lib/wsrep-API/libwsrep_api_v26.a -lpthreads

2 5

Re: [Maria-developers] [JIRA] (MDEV-17260) Memory leaks in mysqlbinlog
by andrei.elkin＠pp.inet.fi 17 Apr '19

17 Apr '19

Sujatha, hello. The patches look good though I have a concern in .. > [ https://jira.mariadb.org/browse/MDEV-17260?page=com.atlassian.jira.plugin.s… ] > > Sujatha Sivakumar commented on MDEV-17260: > ------------------------------------------ > > In the above report there are three different kinds of leaks were observed. > > In 5.5: > ===== > CURRENT_TEST: rpl.rpl_row_mysqlbinlog > ================================================================ > ==8988==ERROR: LeakSanitizer: detected memory leaks > Direct leak of 112 byte(s) in 1 object(s) allocated from: > #0 0x4eff87 in __interceptor_malloc (/dev/shm/5.5/client/mysqlbinlog+0x4eff87) > #1 0x60eaab in my_malloc /mariadb/5.5/mysys/my_malloc.c:41:10 > #2 0x5300dd in Log_event::read_log_event(char const*, unsigned int, char const**, Format_description_log_event const*, char) /mariadb/5.5/sql/log_event.cc:1568:12 > #3 0x564a9c in dump_remote_log_entries(st_print_event_info*, char const*) /mariadb/5.5/client/mysqlbinlog.cc:1978:17 > #4 0x536408 in dump_log_entries(char const*) /mariadb/5.5/client/mysqlbinlog.cc:1794:21 > #5 0x535e4b in main /mariadb/5.5/client/mysqlbinlog.cc:2505:18 > #6 0x7f71ea0a1b16 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x22b16) > > This issue exists only in 5.5. It is specific to fake rotate events while reading binary log from a remote server. > This issues is already fixed in MariaDB 10.0.23 and higher versions as part of > commit c3018b0ff4fb02c029787d03867adf0530607bab > > 10.2 and above: > ============= > 1) Leak in Log_event::read_log_event(char const*, unsigned int, char const**, > Format_description_log_event const*, char) > File: sql/log_event.cc:2150 .. in 2). > 2) read_remote_annotate_event(unsigned char*, unsigned long, char const**) > File: client/mysqlbinlog.cc:194 > What would you think could be the reason of the being deleted two-liner: @@ -1539,10 +1539,11 @@ Exit_status process_event(PRINT_EVENT_INFO *print_event_info, Log_event *ev, } } - if (remote_opt) - ev->temp_buf= 0; I see there were done as a part of the flashback feature. Do we have the flashback and remote combination in any test? Cheers, Andrei > These were caused because of flashback feature and its related bug fixes. > > To reproduce the failures use: > > ASAN_OPTIONS=abort_on_error=1 ./mtr --parallel=auto binlog.binlog_row_mysqlbinlog_verbose binlog_encryption.binlog_row_annotate binlog.binlog_row_mysqlbinlog_options binlog.binlog_row_annotate encryption.tempfiles binlog.binlog_mysqlbinlog_row_myisam binlog.binlog_mysqlbinlog_row binlog.binlog_mysqlbinlog_base64 binlog_encryption.rpl_skip_replication binlog.binlog_mysqlbinlog_row_innodb binlog.flashback > >> Memory leaks in mysqlbinlog >> --------------------------- >> >> Key: MDEV-17260 >> URL: https://jira.mariadb.org/browse/MDEV-17260 >> Project: MariaDB Server >> Issue Type: Bug >> Components: Scripts & Clients >> Affects Versions: 10.0, 10.1, 10.2, 10.3, 10.4, 5.5.62 >> Reporter: Marko Mäkelä >> Assignee: Sujatha Sivakumar >> Priority: Major >> Labels: ASAN, Memory_leak, valgrind >> Fix For: 10.1, 10.2, 10.3, 10.4 >> >> Attachments: mdev-17260-wip.patch >> >> Time Spent: 5h >> Remaining Estimate: 0d >> >> The {{mysqlbinlog}} tool is leaking memory, causing failures in various tests when compiling and testing with AddressSanitizer or LeakSanitizer like this: >> {code:sh} >> cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_ASAN:BOOL=ON /path/to/source >> make -j$(nproc) >> cd mysql-test >> ASAN_OPTIONS=abort_on_error=1 ./mtr --parallel=auto >> {code} >> {noformat:title=5.5 e07118946a82af60c7cc3804c321d3fd9a49f128} >> CURRENT_TEST: rpl.rpl_row_mysqlbinlog >> ================================================================= >> ==8988==ERROR: LeakSanitizer: detected memory leaks >> Direct leak of 112 byte(s) in 1 object(s) allocated from: >> #0 0x4eff87 in __interceptor_malloc (/dev/shm/5.5/client/mysqlbinlog+0x4eff87) >> #1 0x60eaab in my_malloc /mariadb/5.5/mysys/my_malloc.c:41:10 >> #2 0x5300dd in Log_event::read_log_event(char const*, unsigned int, char const**, Format_description_log_event const*, char) /mariadb/5.5/sql/log_event.cc:1568:12 >> #3 0x564a9c in dump_remote_log_entries(st_print_event_info*, char const*) /mariadb/5.5/client/mysqlbinlog.cc:1978:17 >> #4 0x536408 in dump_log_entries(char const*) /mariadb/5.5/client/mysqlbinlog.cc:1794:21 >> #5 0x535e4b in main /mariadb/5.5/client/mysqlbinlog.cc:2505:18 >> #6 0x7f71ea0a1b16 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x22b16) >> {noformat} > > > > -- > This message was sent by Atlassian Jira > (v7.13.1#713001)

1 0

[Maria-developers] MariaDB 10.4 on AIX. Issue with: sql/wsrep_mysqld.cc : ip_len
by REIX, Tony 16 Apr '19

16 Apr '19

Hi, In my environment (AIX & MariaDB v10.4.3 or 10.4.4), the following line of sql/wsrep_mysqld.cc : size_t const ip_len= wsrep_host_len(node_addr.c_str(), node_addr.size()); is preprocessed (gcc -E) as: size_t const ip_ff.ip_vhltl.ip_x.ip_xlen = wsrep_host_len(node_addr.c_str(), node_addr.size()); where "ip_len" becomes: ip_ff.ip_vhltl.ip_x.ip_xlen , which is wrong and breaks. This is due to the following macro in AIX /usr/include/netinet/ip.h : #define ip_len ip_ff.ip_flen The attached patch (replace ip_len by ip_len_mdb) is an easy work-around for fixing the issue. However, it may not be the perfect appropriate change. Since I do not master MariaDB, I'd like someone to propose a fully correct fix. Thanks/Regards, Tony Reix

1 0

[Maria-developers] MariaDB 10.4 on AIX. Issue with: #define SI_KERNEL in include/my_pthread.h
by REIX, Tony 16 Apr '19

16 Apr '19

Hi, In my environment (AIX & MariaDB v10.4.3 or 10.4.4), I have: include/config.h: #define HAVE_SIGWAITINFO 1 and thus in the file : include/my_pthread.h : static inline int my_sigwait(sigset_t *set, int *sig, int *code) { #ifdef HAVE_SIGWAITINFO siginfo_t siginfo; *sig= sigwaitinfo(set, &siginfo); *code= siginfo.si_code; return *sig < 0 ? errno : 0; #else #define SI_KERNEL 128 *code= 0; return sigwait(set, sig); #endif } the SI_KERNEL macro is NOT defined. Then, I get the following error at compilation: mariadb-10.4.3/sql/mysqld.cc:3272:36: error: 'SI_KERNEL' was not declared in this scope if (!abort_loop && origin != SI_KERNEL) The attached patch is an easy work-around for fixing the issue. However, it may not be the perfect appropriate change. Since I do not master MariaDB, I'd like someone to propose a fully correct fix. Thanks/Regards, Tony Reix tony.reix(a)atos.net ATOS / Bull SAS ATOS Expert IBM Coop Architect & Technical Leader Office : +33 (0) 4 76 29 72 67 1 rue de Provence - 38432 Échirolles - France www.atos.net

1 0

Re: [Maria-developers] 404 galera-arbitrator-3_25.3.26-xenial_amd64.deb
by Jongmin Kim 12 Apr '19

12 Apr '19

Ping to the Cc'ed folks, and add more Cc: our new mailing list (harukasan-ftp(a)lists.launchpad.net) On Thu, Apr 11, 2019 at 11:33:33PM +0900, Jongmin Kim wrote: > Hello, > > Thanks for reporting your failure! > > On Thu, Apr 11, 2019 at 03:51:17PM +0300, Sergei Shvarts wrote: > > Hi guys! > > > > looks like there is no galera-arbitraro-3 for xenial > > > > ERROR: unable to update: download errors: > > HTTP code 404 while fetching http://ftp.harukasan.org/mariadb/repo/10.3/ubuntu/pool/main/g/galera-3/gale… > > HTTP code 404 while fetching http://ftp.harukasan.org/mariadb/repo/10.3/ubuntu/pool/main/g/galera-3/gale… > > > > We digged some, and we found that: > - the package with 'Filename' is provided in Xenial 'Packages' [1], > - but the file is not exist at 'Filename' [2]. > We don't have idea about this issue. An official upstream mirror does have same issue [3,4]. > > [1] http://ftp.harukasan.org/mariadb/repo/10.3/ubuntu/dists/xenial/main/binary-… > [2] http://ftp.harukasan.org/mariadb/repo/10.3/ubuntu/pool/main/g/galera-3/ > [3] http://ftp.osuosl.org/pub/mariadb/repo/10.3/ubuntu/dists/xenial/main/binary… > [4] http://ftp.osuosl.org/pub/mariadb/repo/10.3/ubuntu/pool/main/g/galera-3/ > > So I'm forwarding this error to the package maintainer of 'galera-arbitrator-3': > - Codership Oy <info(a)codership.com> > > Also forwarding to the repository master of MariaDB repository: > - MariaDB Developers <maria-developers(a)lists.launchpad.net> > > Dear the package maintainer and MariaDB Developers, would you please dig this problem? > > Thank you! -- Jongmin Kim OpenPGP key located at https://jongmin.dev/pgp OpenPGP fingerprint: 012E 4A06 79E1 4EFC DAAE 9472 D39D 8D29 BAF3 6DF8

2 3

[Maria-developers] GSoC 2019 proposal
by Rajiv nayan 09 Apr '19

09 Apr '19

Hello. My name is Rajiv Nayan. I would like to contribute to mariaDB during GSoC 2019. More information can be found in my draft of proposal on the attached google doc. Thanks Proposal for "Implement UPDATE with result set, and Implement multiple-table UPDATE/DELETE returning a result set” for GSoC’19 https://docs.google.com/document/d/1vb9ixQlhOcHj4RpmaAuhMNNHw4jE_j1wjnzKuNW…

1 0

[Maria-developers] (no subject)
by Rajiv nayan 09 Apr '19

09 Apr '19

Hello. My name is Rajiv Nayan. I would like to contribute to mariaDB during GSoC 2019. More information can be found in my draft of proposal on the attached google doc. Thanks Proposal for "Implement UPDATE with result set, and Implement multiple-table UPDATE/DELETE returning a result set” for GSoC’19 https://docs.google.com/document/d/1vb9ixQlhOcHj4RpmaAuhMNNHw4jE_j1wjnzKuNW…

1 0

[Maria-developers] set_time() on slave and unversioned -> versioned replication
by Aleksey Midenkov 08 Apr '19

08 Apr '19

Hello, Sergei! In unversioned -> versioned scenario in the code below it first gets to Set time 4, creates some records (on slave) and then seconds on slave increased (X+1) while on master seconds are not yet increased (X). Then we get to Set time 3 and reset time on slave to X.0 therefore we get back in time and all stored records with timestamp X.n will be in future. 'n' came from ++system_time.sec_part in Set time 4. Why did you decided to use such logic of getting seconds from master and microseconds from slave? Since microseconds sooner or later reset to 0 it's not better than just assigning some random number. What sending microseconds from master conditionally only is good for? And for RBR since you don't send microseconds at all I see no good solution for this. inline void set_time(my_time_t t, ulong sec_part) { if (opt_secure_timestamp > (slave_thread ? SECTIME_REPL : SECTIME_SUPER)) { set_time(); // note that BINLOG itself requires SUPER DBUG_EXECUTE("time", print_start_time("Set time 1");); } else { if (sec_part <= TIME_MAX_SECOND_PART) { start_time= system_time.sec= t; start_time_sec_part= system_time.sec_part= sec_part; DBUG_EXECUTE("time", print_start_time("Set time 2");); } else if (t != system_time.sec) { DBUG_EXECUTE("time", print_system_time("System time"););; start_time= system_time.sec= t; start_time_sec_part= system_time.sec_part= 0; DBUG_EXECUTE("time", print_start_time("Set time 3");); } else { start_time= t; start_time_sec_part= ++system_time.sec_part; DBUG_EXECUTE("time", print_start_time("Set time 4");); } user_time.val= hrtime_from_time(start_time) + start_time_sec_part; PSI_CALL_set_thread_start_time(start_time); start_utime= utime_after_lock= microsecond_interval_timer(); } } -- All the best, Aleksey Midenkov @midenok

2 7

[Maria-developers] GSoC'19 proposal
by Rucha Deodhar 05 Apr '19

05 Apr '19

Hello. My name is Rucha Deodhar. I would like to contribute to mariaDB during GSoC 2019. More information can be found in my draft of proposal on the attached google doc. Any feedback would be appreciated. Thanks Proposal for “ADD RETURNING TO INSERT” for GSoC’19 <https://docs.google.com/document/d/1vE52kqZI-bhbRKbGBxeIrQKss8xXaGUU-0_PJHq…>

1 0

[Maria-developers] MDEV-18822: possible slowdown with optimizer trace
by Sergey Petrunia 04 Apr '19

04 Apr '19

Hi Varun, So I was thinking of the impact of the optimizer trace, and decided to look at how the most common check "if optimizer trace is enabled" is done. The code at SQL layer does this kind of check if (thd->trace_started()) (it is wrapped somewhere with "unlikely" but this is not consistent) THD::trace_started is defined as: bool THD::trace_started() { return opt_trace.is_started(); } which opt_trace here is Opt_trace_context. looking there: class Opt_trace_context { ... bool is_started() { return current_trace && is_enabled(); } following: bool Opt_trace_context::is_enabled() { if (current_trace) return current_trace->is_enabled(); return false; } current_trace is Opt_trace_stmt, looking there: bool Opt_trace_stmt::is_enabled() { return I_S_disabled == 0; } This is quite convoluted, and could probably made shorter. Disassembling sql_select.cc, I can see that the calls to thd->trace_started() are not inlined: if (unlikely(thd->trace_started())) 10ac9: 4c 89 ff mov rdi,r15 10acc: e8 00 00 00 00 call 10ad1 <_ZL32best_extension_by_limited_searchP4JOINyjddjjj+0x1a1> 10ad1: 84 c0 test al,al 10ad3: 0f 85 fc 02 00 00 jne 10dd5 <_ZL32best_extension_by_limited_searchP4JOINyjddjjj+0x4a5> if (unlikely(thd->trace_started())) 1197b: 4c 89 f7 mov rdi,r14 1197e: e8 00 00 00 00 call 11983 <_Z11choose_planP4JOINy+0x8a3> 11983: 84 c0 test al,al 11985: 0f 85 2f 01 00 00 jne 11aba <_Z11choose_planP4JOINy+0x9da> if (thd->trace_started()) 2d6f7: 48 8b bd b0 fc ff ff mov rdi,QWORD PTR [rbp-0x350] 2d6fe: e8 00 00 00 00 call 2d703 <_ZN4JOIN14optimize_innerEv+0x1963> 2d703: 84 c0 test al,al 2d705: 0f 85 6c 09 00 00 jne 2e077 <_ZN4JOIN14optimize_innerEv+0x22d7> if (thd->trace_started()) 2d7da: 48 8b bd b0 fc ff ff mov rdi,QWORD PTR [rbp-0x350] 2d7e1: e8 00 00 00 00 call 2d7e6 <_ZN4JOIN14optimize_innerEv+0x1a46> 2d7e6: 84 c0 test al,al 2d7e8: 0f 85 ff 1a 00 00 jne 2f2ed <_ZN4JOIN14optimize_innerEv+0x354d> But does it all matter? I think the action plan should be: 1. Reproduce the slowdown as it was reported in MDEV-18822. 2. If #1 succeeds in several re-runs, declare THD::trace_started as: class THD { public: bool trace_started; // initialize this to false in constructor and never touch it inline bool trace_started() const { return trace_started; } }; and then use search-replace to change "thd->trace_started()" to "unlikely(thd->trace_started)" and then re-run the benchmark. Any thoughts? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0