[Maria-discuss] Segfaulting with signal 6
I've run into another weird problem where MariaDB is segfaulting and dieing on Gentoo Hardened Linux. I can't figure out if its a weird situation with my Gentoo environment on the machine or something with MariaDB itself. I don't have this problem on the other machine running MariaDB. It seems to happen under load about once or twice a day. I have a nearly identical machine that doesn't have any of these problems. I'm running mariadb-5.1.50 under Hardened Gentoo. The compiled flags are: --- Configuration summary for MariaDB Server version 5.1.50-MariaDB * Installation prefix: /usr * System type: pc-linux-gnu * Host CPU: x86_64 * C Compiler: x86_64-pc-linux-gnu-gcc (Gentoo Hardened 4.4.4-r2 p1.2, pie-0.4.5) 4.4.4 * C++ Compiler: x86_64-pc-linux-gnu-g++ (Gentoo Hardened 4.4.4-r2 p1.2, pie-0.4.5) 4.4.4 * Debug enabled: no * Community Features: yes --- $ ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --libexecdir=/usr/sbin --sysconfdir=/etc/mysql --localstatedir=/var/lib/mysql --sharedstatedir=/usr/share/mysql --libdir=/usr/lib64/mysql --includedir=/usr/include/mysql --with-low-memory --with-client-ldflags=-lstdc++ --enable-thread-safe-client --with-comment=Gentoo Linux mariadb-5.1.50 --without-docs --without-big-tables --enable-local-infile --with-extra-charsets=all --with-mysqld-user=mysql --with-server --with-unix-socket-path=/var/run/mysqld/mysqld.sock --without-libwrap --enable-shared --enable-static --without-debug --with-charset=utf8 --with-collation=utf8_general_ci --without-embedded-privilege-control --without-embedded-server --with-ssl=/usr --enable-assembler --with-geometry --with-readline --with-zlib-dir=/usr/ --without-pstack --with-plugindir=/usr/lib64/mysql/plugin --enable-community-features --disable-profiling --with-maria-tmp-tables --with-libevent --with-plugins=csv,myisam,myisammrg,heap,archive,blackhole,maria,xtradb --without-plugin-example --without-plugin-ibmdb2i --without-plugin-partition --without-plugin-federated --without-plugin-ndbcluster --without-plugin-pbxt I haven't opened a bug with Gentoo yet (I'm actually developer myself). I figured I'd ask here first. Any help would be great. Thanks! Here's the output I have from the log file: mysqld: malloc.c:5153: malloc_consolidate: Assertion `nextchunk->fd_nextsize->bk_nextsize == nextchunk' failed. 110329 13:11:25 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=134217728 read_buffer_size=262144 max_used_connections=376 max_threads=401 threads_connected=80 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 445322 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd: 0x3518dccbaa0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x3519ebd3ea0 thread_stack 0x3c000 /usr/lib64/mysql/libmysys.so.0(my_print_stacktrace+0x3c) [0x3547393ab0c] /usr/sbin/mysqld(handle_segfault+0x3cc) [0x7d0ebd1c1c] /lib/libpthread.so.0(+0xf010) [0x35473f86010] /lib/libc.so.6(gsignal+0x35) [0x35472264135] /lib/libc.so.6(abort+0x180) [0x35472265550] /lib/libc.so.6(+0x72c6a) [0x354722a4c6a] /lib/libc.so.6(+0x72f2d) [0x354722a4f2d] /lib/libc.so.6(+0x751e4) [0x354722a71e4] /lib/libc.so.6(__libc_malloc+0x70) [0x354722a9900] /usr/sbin/mysqld(+0x5ad9cf) [0x7d0ee529cf] /usr/sbin/mysqld(+0x5f3bfb) [0x7d0ee98bfb] /usr/sbin/mysqld(+0x5ed71b) [0x7d0ee9271b] /usr/sbin/mysqld(+0x588aa3) [0x7d0ee2daa3] /usr/sbin/mysqld(rr_sequential(READ_RECORD*)+0x29) [0x7d0ecc4b69] /usr/sbin/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x99) [0x7d0ec430b9] /usr/sbin/mysqld(+0x39e524) [0x7d0ec43524] /usr/sbin/mysqld(JOIN::exec()+0xbd7) [0x7d0ec53b47] /usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st _order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x1b2) [0x7d0ec551a 2] /usr/sbin/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x19c) [0x7d0ec55c5c] /usr/sbin/mysqld(+0x33955a) [0x7d0ebde55a] /usr/sbin/mysqld(mysql_execute_command(THD*)+0x405) [0x7d0ebe07f5] /usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x2f3) [0x7d0ebe5cf3] /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x53a) [0x7d0ebe6b2a] /usr/sbin/mysqld(libevent_thread_proc+0x182) [0x7d0ebda682] /lib/libpthread.so.0(+0x68c4) [0x35473f7d8c4] /lib/libc.so.6(clone+0x6d) [0x35472302b4d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x3518424f018 is an invalid pointer thd->thread_id=14399 thd->killed=NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. -- Lance Albertson Systems Administrator / Architect Open Source Lab Network Services Oregon State University
Hi!
"Lance" == Lance Albertson <lance@osuosl.org> writes:
Lance> I've run into another weird problem where MariaDB is segfaulting Lance> and dieing on Gentoo Hardened Linux. I can't figure out if its a weird Lance> situation with my Gentoo environment on the machine or something with Lance> MariaDB itself. I don't have this problem on the other machine running Lance> MariaDB. It seems to happen under load about once or twice a day. I have Lance> a nearly identical machine that doesn't have any of these problems. Lance> I'm running mariadb-5.1.50 under Hardened Gentoo. The compiled flags are: Lance> --- Lance> Configuration summary for MariaDB Server version 5.1.50-MariaDB <cut> Lance> mysqld: malloc.c:5153: malloc_consolidate: Assertion Lance> `nextchunk->fd_nextsize->bk_nextsize == nextchunk' failed. Lance> 110329 13:11:25 [ERROR] mysqld got signal 6 ; This looks like memory overrun, something that we have not seen in MariaDB for a long time. (We are testing all builds with valgrind to ensure we don't have any memory overruns). <cut> Is there a way we could install a debug binary on the machine in question so that we could be able to debug this? (We would need a binary compiled with debug + a core to be able to find out what's going on). The easyest way to get a debug binary is to get the source and compile it with: ./BUILD/compile-pentium-debug-max and then just copy it over your original mysqld binary. If you can get us access to the system, we can help you with this. You can find us at '#maria' channel on freenode if you want to discuss this real time. Regards, Monty
On 3/30/2011 10:58 AM, Michael Widenius wrote:
Hi!
"Lance" == Lance Albertson <lance@osuosl.org> writes: Lance> I've run into another weird problem where MariaDB is segfaulting Lance> and dieing on Gentoo Hardened Linux. I can't figure out if its a weird Lance> situation with my Gentoo environment on the machine or something with Lance> MariaDB itself. I don't have this problem on the other machine running Lance> MariaDB. It seems to happen under load about once or twice a day. I have Lance> a nearly identical machine that doesn't have any of these problems.
Lance> I'm running mariadb-5.1.50 under Hardened Gentoo. The compiled flags are:
Note that under Gentoo Linux, core dumps are disabled by default. The Gentoo approved reference of getting good traces and enabling cores is here: http://www.gentoo.org/proj/en/qa/backtraces.xml
Is there a way we could install a debug binary on the machine in question so that we could be able to debug this?
(We would need a binary compiled with debug + a core to be able to find out what's going on).
The easyest way to get a debug binary is to get the source and compile it with:
./BUILD/compile-pentium-debug-max
and then just copy it over your original mysqld binary.
If you can get us access to the system, we can help you with this.
You can find us at '#maria' channel on freenode if you want to discuss this real time.
Regards, Monty
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
On 03/30/2011 07:58 AM, Michael Widenius wrote:
Hi!
"Lance" == Lance Albertson <lance@osuosl.org> writes:
Lance> I've run into another weird problem where MariaDB is segfaulting Lance> and dieing on Gentoo Hardened Linux. I can't figure out if its a weird Lance> situation with my Gentoo environment on the machine or something with Lance> MariaDB itself. I don't have this problem on the other machine running Lance> MariaDB. It seems to happen under load about once or twice a day. I have Lance> a nearly identical machine that doesn't have any of these problems.
Lance> I'm running mariadb-5.1.50 under Hardened Gentoo. The compiled flags are:
Lance> --- Lance> Configuration summary for MariaDB Server version 5.1.50-MariaDB
<cut>
Lance> mysqld: malloc.c:5153: malloc_consolidate: Assertion Lance> `nextchunk->fd_nextsize->bk_nextsize == nextchunk' failed. Lance> 110329 13:11:25 [ERROR] mysqld got signal 6 ;
This looks like memory overrun, something that we have not seen in MariaDB for a long time. (We are testing all builds with valgrind to ensure we don't have any memory overruns).
<cut>
Is there a way we could install a debug binary on the machine in question so that we could be able to debug this?
(We would need a binary compiled with debug + a core to be able to find out what's going on).
The easyest way to get a debug binary is to get the source and compile it with:
./BUILD/compile-pentium-debug-max
and then just copy it over your original mysqld binary.
I will try and get it compiled with debug symbols under Gentoo using their documentation. If that doesn't work I'll do it manually.
If you can get us access to the system, we can help you with this.
If we get to the point where I can't get it farther with it, I'll do that but for now I feel more comfortable doing it myself.
You can find us at '#maria' channel on freenode if you want to discuss this real time.
Thanks for that tip! So far the machine has been running smoothly. I just did a minor glibc upgrade to see if that helped it or not. I have monit monitoring mysqld so that it will kick it more automatically. Cheers- -- Lance Albertson Systems Administrator / Architect Open Source Lab Network Services Oregon State University
participants (3)
-
Brian Evans -
Lance Albertson -
Michael Widenius