[Maria-discuss] error on show status query
Have the following error trace from running SHOW STATUS LIKE 'wsrep_%' Any idea what could cause it? 2016-12-21 3:39:24 139668819036928 [Note] InnoDB: Online DDL : End of applying row log 2016-12-21 3:39:24 139668819036928 [Note] InnoDB: Online DDL : Completed 2016-12-21 3:43:29 139668818733824 [Note] InnoDB: Online DDL : Start 2016-12-21 3:43:29 139668818733824 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files 2016-12-21 03:43:29 7f072e53ab00 InnoDB: Error: Write to file (merge) failed at offset 0. InnoDB: 1048576 bytes should have been written, only 901120 were written. InnoDB: Operating system error number 12. InnoDB: Check that your OS and file system support files of this size. InnoDB: Check also that the disk is not full or a disk quota exceeded. InnoDB: Error number 12 means 'Cannot allocate memory'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 2016-12-21 3:43:29 139668818733824 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files java.sql.SQLNonTransientConnectionException: (conn:17945) Could not read resultset: unexpected end of stream, read 0 bytes from 4 Query is : SHOW STATUS LIKE 'wsrep_%' at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:132) at org.mariadb.jdbc.internal.util.ExceptionMapper.getException(ExceptionMapper.java:101) at org.mariadb.jdbc.internal.util.ExceptionMapper.throwAndLogException(ExceptionMapper.java:77) at org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:224) at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:258) at org.mariadb.jdbc.MariaDbStatement.executeQuery(MariaDbStatement.java:371) at com.verizon.mesos.Dbcheck.getStatus(Dbcheck.java:153) at com.verizon.mesos.Dbcheck.run(Dbcheck.java:103) at com.verizon.mesos.MariadbTask.handleHealthCheck(MariadbTask.java:310) at com.verizon.mesos.MariadbTask.executeCmd(MariadbTask.java:384) at com.verizon.mesos.MariadbTask.run(MariadbTask.java:450) at com.verizon.mesos.ExecutorTask.executeCmd(ExecutorTask.java:50) at com.verizon.mesos.ExecutorTask.run(ExecutorTask.java:115) at java.lang.Thread.run(Thread.java:745)
"SHOW STATUS" fails because server closes the client socket. "Could not read resultset: unexpected end of stream, read 0 bytes from 4" is a slightly weird message, but client reads 0 bytes instead of packet header(4 bytes). Reading 0 bytes which means socket was closed orderly (as opposed to e.g aborting connection with SO_LINGER or server maybe server crash). Why does server closes the socket, and what happened otherwise, I do not know. The error log hints at memory shortage. On 21.12.2016 20:45, l vic wrote:
Have the following error trace from running SHOW STATUS LIKE 'wsrep_%' Any idea what could cause it?
2016-12-21 3:39:24 139668819036928 [Note] InnoDB: Online DDL : End of applying row log 2016-12-21 3:39:24 139668819036928 [Note] InnoDB: Online DDL : Completed 2016-12-21 3:43:29 139668818733824 [Note] InnoDB: Online DDL : Start 2016-12-21 3:43:29 139668818733824 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files 2016-12-21 03:43:29 7f072e53ab00 InnoDB: Error: Write to file (merge) failed at offset 0. InnoDB: 1048576 bytes should have been written, only 901120 were written. InnoDB: Operating system error number 12. InnoDB: Check that your OS and file system support files of this size. InnoDB: Check also that the disk is not full or a disk quota exceeded. InnoDB: Error number 12 means 'Cannot allocate memory'. InnoDB: Some operating system error numbers are described at InnoDB:http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 2016-12-21 3:43:29 139668818733824 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files java.sql.SQLNonTransientConnectionException: (conn:17945) Could not read resultset: unexpected end of stream, read 0 bytes from 4 Query is : SHOW STATUS LIKE 'wsrep_%' at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:132) at org.mariadb.jdbc.internal.util.ExceptionMapper.getException(ExceptionMapper.java:101) at org.mariadb.jdbc.internal.util.ExceptionMapper.throwAndLogException(ExceptionMapper.java:77) at org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:224) at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:258) at org.mariadb.jdbc.MariaDbStatement.executeQuery(MariaDbStatement.java:371) at com.verizon.mesos.Dbcheck.getStatus(Dbcheck.java:153) at com.verizon.mesos.Dbcheck.run(Dbcheck.java:103) at com.verizon.mesos.MariadbTask.handleHealthCheck(MariadbTask.java:310) at com.verizon.mesos.MariadbTask.executeCmd(MariadbTask.java:384) at com.verizon.mesos.MariadbTask.run(MariadbTask.java:450) at com.verizon.mesos.ExecutorTask.executeCmd(ExecutorTask.java:50) at com.verizon.mesos.ExecutorTask.run(ExecutorTask.java:115) at java.lang.Thread.run(Thread.java:745)
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
I run this query every minute and I have this exception after couple of hours... Class.forName("org.mariadb.jdbc.Driver"); conn = DriverManager.getConnection(DB_URL, USER, PASS); stmt = conn.createStatement(); String sql = "SHOW STATUS LIKE 'wsrep_%'"; rs = stmt.executeQuery(sql); Is there something i can do to get root cause of that exception? Thank you, vic On Wed, Dec 21, 2016 at 2:55 PM, Vladislav Vaintroub <vvaintroub@gmail.com> wrote:
"SHOW STATUS" fails because server closes the client socket.
"Could not read resultset: unexpected end of stream, read 0 bytes from 4" is a slightly weird message, but client reads 0 bytes instead of packet header(4 bytes). Reading 0 bytes which means socket was closed orderly (as opposed to e.g aborting connection with SO_LINGER or server maybe server crash).
Why does server closes the socket, and what happened otherwise, I do not know. The error log hints at memory shortage.
On 21.12.2016 20:45, l vic wrote:
Have the following error trace from running SHOW STATUS LIKE 'wsrep_%' Any idea what could cause it?
2016-12-21 3:39:24 139668819036928 [Note] InnoDB: Online DDL : End of applying row log 2016-12-21 3:39:24 139668819036928 [Note] InnoDB: Online DDL : Completed 2016-12-21 3:43:29 139668818733824 [Note] InnoDB: Online DDL : Start 2016-12-21 3:43:29 139668818733824 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files 2016-12-21 03:43:29 7f072e53ab00 InnoDB: Error: Write to file (merge) failed at offset 0. InnoDB: 1048576 bytes should have been written, only 901120 were written. InnoDB: Operating system error number 12. InnoDB: Check that your OS and file system support files of this size. InnoDB: Check also that the disk is not full or a disk quota exceeded. InnoDB: Error number 12 means 'Cannot allocate memory'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 2016-12-21 3:43:29 139668818733824 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files java.sql.SQLNonTransientConnectionException: (conn:17945) Could not read resultset: unexpected end of stream, read 0 bytes from 4 Query is : SHOW STATUS LIKE 'wsrep_%' at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:132) at org.mariadb.jdbc.internal.util.ExceptionMapper.getException(ExceptionMapper.java:101) at org.mariadb.jdbc.internal.util.ExceptionMapper.throwAndLogException(ExceptionMapper.java:77) at org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:224) at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:258) at org.mariadb.jdbc.MariaDbStatement.executeQuery(MariaDbStatement.java:371) at com.verizon.mesos.Dbcheck.getStatus(Dbcheck.java:153) at com.verizon.mesos.Dbcheck.run(Dbcheck.java:103) at com.verizon.mesos.MariadbTask.handleHealthCheck(MariadbTask.java:310) at com.verizon.mesos.MariadbTask.executeCmd(MariadbTask.java:384) at com.verizon.mesos.MariadbTask.run(MariadbTask.java:450) at com.verizon.mesos.ExecutorTask.executeCmd(ExecutorTask.java:50) at com.verizon.mesos.ExecutorTask.run(ExecutorTask.java:115) at java.lang.Thread.run(Thread.java:745)
_______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp
On 22.12.2016 17:38, l vic wrote:
I run this query every minute and I have this exception after couple of hours... Class.forName("org.mariadb.jdbc.Driver"); conn = DriverManager.getConnection(DB_URL, USER, PASS); stmt = conn.createStatement(); String sql = "SHOW STATUS LIKE 'wsrep_%'"; rs = stmt.executeQuery(sql);
Is there something i can do to get root cause of that exception? Thank you, vic
It should be some something on the server, but I do not have a better idea that to look in server's error log file. Maybe periodic "show processlist" can also sched some light on what happens right before server would close the connection. As I mentioned, error log seems to indicate memory pressure of some sort.
Seems like it only happens during or after SST. I wonder if this may happen because of data inconsistency... On Thu, Dec 22, 2016 at 2:27 PM, Vladislav Vaintroub <vvaintroub@gmail.com> wrote:
On 22.12.2016 17:38, l vic wrote:
I run this query every minute and I have this exception after couple of hours... Class.forName("org.mariadb.jdbc.Driver"); conn = DriverManager.getConnection(DB_URL, USER, PASS); stmt = conn.createStatement(); String sql = "SHOW STATUS LIKE 'wsrep_%'"; rs = stmt.executeQuery(sql);
Is there something i can do to get root cause of that exception? Thank you, vic
It should be some something on the server, but I do not have a better idea that to look in server's error log file. Maybe periodic "show processlist" can also sched some light on what happens right before server would close the connection. As I mentioned, error log seems to indicate memory pressure of some sort.
participants (2)
-
l vic -
Vladislav Vaintroub