- developers - lists.mariadb.org

[Maria-developers] WL#196 New (by Mdcallag): add is_connection_alive for server-side checks to find disconnected clients
by worklog-noreply＠askmonty.org 08 Apr '11

08 Apr '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: add is_connection_alive for server-side checks to find disconnected clients CREATION DATE..: Fri, 08 Apr 2011, 14:06 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 196 (http://askmonty.org/worklog/?tid=196) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: There are a few places in the server where the thread for a user connection will wait. With upcoming features in the FB patch there will be more places. I don't want a thread to continue waiting after the client has disconnected. Is it possible to check whether the client has disconnected without reading/writing more than 0 bytes from the socket? Assuming the check is is_connection_alive(), it can return TRUE when the client has disconnected (but not all of the time). But it must not return FALSE when the client has not disconnected. The check does not have to be cheap. It can require a system call as it won't be made frequently. For example, assume that a row lock wait is implemented as a sequent of timed pthread condition variable waits (1 second each). At the end of each 1 second timeout there can be a check for whether the connection was killed and a check for whether the client has disconnected. Disconnected clients on the server-side can be a serious problem for high throughput servers. The worst case is when the client starts a long-running and blocking query (external sort) that won't return a row for a long time and thus won't detect the disconnected client for a long time. I have used tool in the past that join output from netstat with SHOW PROCESSLIST to find sockets for disconnected clients. This works but is very expensive. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#195 New (by Mdcallag): SHOW QUERY STATUS
by worklog-noreply＠askmonty.org 07 Apr '11

07 Apr '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: SHOW QUERY STATUS CREATION DATE..: Thu, 07 Apr 2011, 13:54 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 195 (http://askmonty.org/worklog/?tid=195) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: I want some performance counters to be maintained per-query in addition to per- session. This could then be used to implement SHOW QUERY STATUS. It could also be used to augment slow query log output with additional performance data per statement. The Facebook patch and probably Percona already put more data per statement in the slow query log. With the Facebook patch the option --log-slow- extra enables this. Unfortunately, that uses the session counters and can be confusing. Some details on the use of this in the facebook patch are at: https://www.facebook.com/note.php?note_id=390420710932 ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] Summary of the storage engine API changes for group commit
by Kristian Nielsen 07 Apr '11

07 Apr '11

Hi Serg, As promised, here is a summary of the changes to the storage engine API that I made as part of group commit. Two new handlerton methods are added: void (*prepare_ordered)(handlerton *hton, THD *thd, bool all); void (*commit_ordered)(handlerton *hton, THD *thd, bool all); commit_ordered() will be called just before commit(). It should commit the transaction in memory; calls to commit_ordered() will be serialised, so order of commit is determined by order of calls to commit_ordered(). MVCC engines should make the transaction visible. No time-consuming operations like flushing logs to disk should be done; this should happen in commit(). prepare_ordered() will be called just after prepare(). Calls will again be serialised and happen in commit order, so an engine can do here any part of the prepare step that depends on commit order. Again no time-consuming operations like I/O should be done. Order of calls to the new methods is guaranteed to be consistent between engines and with commit order in the binlog. Locking is done so that START TRANSACTION WITH CONSISTENT SNAPSHOT becomes consistent between storage engines. The methods are optional; a storage engine is free to not set them. Group commit will still work, but there will be no guarantee that commits happen in the same order in engine and in binlog, nor that START TRANSACTION WITH CONSISTENT SNAPSHOT will be consistent between engines/binlog. (The prepare_commit_mutex in built-in InnoDB will still ensure consistent commit order, and will still destroy group commit). There are no other changes to the storage engine API by me. Hope this helps, - Kristian.

3 4

[Maria-developers] Default parameter settings / Windows MSI installer
by Vladislav Vaintroub 07 Apr '11

07 Apr '11

Hello, After recent discussion about appropriate or not appropriate default setting on in my.cnf , I have created a prototype minimal settings for Windows installer. Here is the picture of this attempt http://cid-ff0c950417b4f8a4.office.live.com/self.aspx/share/dbsettings.png For those who have familiar the 5.2 MSI installer already, what I have added is a checkbox "Standard configuration", explanation text, and input field for innodb bufferpool size (preset with RAM/8, in my case it is 8191/8=1023). "Standard configuration" is on by default, if it is "on", then 4 parameters are added to the my.ini file 1) default_storage_engine=innodb 2) innodb_buffer_pool_size=[VALUE_IN_EDIT_BOX]M 3) innodb_log_file_size= min(innodb_buffer_pool_size/4, 50) I made this formula almost out of thin air and using some blog reading and comments in the templates. It is likely that 50M is more than the average user needs, however it only takes some disk space, and If user has to increase from default 5MB , the procedure is awkward. I checked, MySQL Config Wizard set this parameter to 54M on my box, so it is in the same range. 4) sql_mode=STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION If "Standard configuration" is off, hen input field for bufferpool size is disabled, and none of those 4 parameters is added to my.ini Do you think it is useful in this form? So far I tried to avoid "calibration" topics altogether since it is a controversial topic where people never have same opinions. Also I tried to avoid it because I cannot write GUIs . However, this one change is small in and it does not introduce additional dialogs. It makes some important defaults the same as "standard configuration" in existing MySQL installers. I think it was PeterL who already asked for non-default buffer pool size already, also Monty also spoke about it in Lisbon. Comments welcome, also those on the picture above - I'm not sure if wording and use of brackets and quotes is correct. Wlad

5 8

Re: [Maria-developers] dist-hook in top-level Makefile.am
by Kristian Nielsen 06 Apr '11

06 Apr '11

"Vladislav Vaintroub" <wlad(a)montyprogram.com> writes: > Kristian, now that Windows builds in 5.2+ are self-sufficient and can > produce different packages from bzr, do you see any reason to keep this hook > in the top-level Makefile.am? > > If not, I can remove, or you can remove it if you prefer. Will make our > source distribution smaller/cleaner , if nothing else. > > I'm talking about this place in Makefile.am > > # Remove BK's "SCCS" subdirectories from source distribution > # Create initial database files for Windows installations and check them. > dist-hook: > mkdir -p $(distdir)/win > scripts/mysql_install_db --no-defaults --cross-bootstrap \ > --builddir=$(top_builddir) \ > --datadir=$(distdir)/win/data \ > --srcdir=$(top_srcdir) > storage/myisam/myisamchk --silent --fast > $(distdir)/win/data/mysql/*.MYI I do not have time to look deeply into this at the moment, but I did not want you to have to wait any longer for an answer, sorry for taking so long already. It is very good news that Windows builds are now self-sufficient! This need to compile the whole server just to produce a source tarball has been nagging me forever, and I suppose this has also been an annoyance on the Windows side when installing from source. You need to be careful when you remove it. Update any documentation that refers to the old way of installing from source on Windows, making sure existing stuff still works, etc. Well, I guess you are already aware of that. It would be nice to also fix the autotools builds so that `make dist` to produce a source tarball no longer needs to compile the whole server. But then, as we are phasing out autotools, it is probably not worth it. So I suppose just removing the dist-hook is a reasonable step if you make sure it does not break anything. I guess the CMake system already has this done properly, so that a source tarball is produced without needing to build the server (and without generating these obsolete database files for Windows)? - Kristian.

2 1

Re: [Maria-developers] [Commits] Rev 2864: MWL#136: Cross-engine consistency for START TRANSACTION WITH CONSISTENT SNAPSHOT in http://bazaar.launchpad.net/~maria-captains/maria/5.1
by Michael Widenius 05 Apr '11

05 Apr '11

Hi! >>>>> "knielsen" == knielsen <knielsen(a)knielsen-hq.org> writes: knielsen> At http://bazaar.launchpad.net/~maria-captains/maria/5.1 knielsen> ------------------------------------------------------------ knielsen> revno: 2864 knielsen> revision-id: knielsen(a)knielsen-hq.org-20101107213743-3luszsivft2vt7t6 knielsen> parent: knielsen(a)knielsen-hq.org-20101103155438-vdou0fngj6hpgpsd knielsen> committer: knielsen(a)knielsen-hq.org knielsen> branch nick: work-5.1-mwl136 knielsen> timestamp: Sun 2010-11-07 22:37:43 +0100 knielsen> message: knielsen> MWL#136: Cross-engine consistency for START TRANSACTION WITH CONSISTENT SNAPSHOT <cut> knielsen> === modified file 'client/mysqldump.c' <cut> knielsen> -static int do_show_master_status(MYSQL *mysql_con) knielsen> +static int do_show_master_status(MYSQL *mysql_con, int consistent_binlog_pos) knielsen> { knielsen> MYSQL_ROW row; knielsen> MYSQL_RES *master; knielsen> + char binlog_pos_file[FN_REFLEN]; knielsen> + char binlog_pos_offset[LONGLONG_LEN+1]; knielsen> + char *file, *offset; knielsen> const char *comment_prefix= knielsen> (opt_master_data == MYSQL_OPT_MASTER_DATA_COMMENTED_SQL) ? "-- " : ""; knielsen> - if (mysql_query_with_error_report(mysql_con, &master, "SHOW MASTER STATUS")) knielsen> + knielsen> + if (consistent_binlog_pos) knielsen> { knielsen> - return 1; knielsen> + if(!check_consistent_binlog_pos(binlog_pos_file, binlog_pos_offset)) knielsen> + return 1; knielsen> + file= binlog_pos_file; knielsen> + offset= binlog_pos_offset; knielsen> } knielsen> else knielsen> { knielsen> + if (mysql_query_with_error_report(mysql_con, &master, "SHOW MASTER STATUS")) knielsen> + return 1; knielsen> + knielsen> row= mysql_fetch_row(master); knielsen> if (row && row[0] && row[1]) knielsen> { knielsen> + file= row[0]; knielsen> + offset= row[1]; knielsen> } knielsen> + else knielsen> { You need a mysql_free_result(master) here. knielsen> + if (!ignore_errors) knielsen> + { knielsen> + /* SHOW MASTER STATUS reports nothing and --force is not enabled */ knielsen> + my_printf_error(0, "Error: Binlogging on server not active", knielsen> + MYF(0)); knielsen> + maybe_exit(EX_MYSQLERR); knielsen> + return 1; knielsen> + } knielsen> + else knielsen> + { knielsen> + return 0; knielsen> + } knielsen> } knielsen> } <cut> knielsen> === modified file 'sql/log.cc' knielsen> --- a/sql/log.cc 2010-11-02 07:40:27 +0000 knielsen> +++ b/sql/log.cc 2010-11-07 21:37:43 +0000 knielsen> @@ -62,6 +62,7 @@ static int binlog_savepoint_rollback(han knielsen> static int binlog_commit(handlerton *hton, THD *thd, bool all); knielsen> static int binlog_rollback(handlerton *hton, THD *thd, bool all); knielsen> static int binlog_prepare(handlerton *hton, THD *thd, bool all); knielsen> +static int binlog_start_consistent_snapshot(handlerton *hton, THD *thd); knielsen> /** knielsen> Silence all errors and warnings reported when performing a write knielsen> @@ -155,9 +156,10 @@ class binlog_trx_data { knielsen> public: knielsen> binlog_trx_data() knielsen> : at_least_one_stmt_committed(0), incident(FALSE), m_pending(0), knielsen> - before_stmt_pos(MY_OFF_T_UNDEF), commit_bin_log_file_pos(0), using_xa(0) knielsen> + before_stmt_pos(MY_OFF_T_UNDEF), last_commit_pos_offset(0), using_xa(0) knielsen> { knielsen> trans_log.end_of_file= max_binlog_cache_size; knielsen> + strcpy(last_commit_pos_file, ""); strcpy() -> strmov() or even better last_commit_pos_file[0]= 0; (I want to get rid of all strcpy() and replace with strmov() as strcpy() is often used wrongly; I would like to have this thing changed to not get 'false positives' when searching for strcpy() to remove. knielsen> } knielsen> ~binlog_trx_data() knielsen> @@ -215,7 +217,8 @@ public: knielsen> incident= FALSE; knielsen> trans_log.end_of_file= max_binlog_cache_size; knielsen> using_xa= FALSE; knielsen> - commit_bin_log_file_pos= 0; knielsen> + strcpy(last_commit_pos_file, ""); strcpy -> strmov or last_commit_pos_file[0]= 0; knielsen> + last_commit_pos_offset= 0; knielsen> DBUG_ASSERT(empty()); knielsen> } knielsen> /* knielsen> Write a table map to the binary log. knielsen> @@ -4337,6 +4369,9 @@ MYSQL_BIN_LOG::flush_and_set_pending_row knielsen> rotate_and_purge(RP_LOCK_LOG_IS_ALREADY_LOCKED); knielsen> } Add a comment that this mutex is a protection for partial reads of last_commit_pos_offset on 32 bit systems. (at least I can't come up with any better reason for this mutex). Maybe we should introduce pthread_mutex_lock_for_32_bit() that would be void on 64 bit systems in cases like this. knielsen> + pthread_mutex_lock(&LOCK_commit_ordered); knielsen> + last_commit_pos_offset= my_b_tell(&log_file); knielsen> + pthread_mutex_unlock(&LOCK_commit_ordered); knielsen> pthread_mutex_unlock(&LOCK_log); knielsen> } knielsen> @@ -4526,7 +4561,12 @@ bool MYSQL_BIN_LOG::write(Log_event *eve knielsen> err_unlock: knielsen> if (file == &log_file) knielsen> + { knielsen> + pthread_mutex_lock(&LOCK_commit_ordered); knielsen> + last_commit_pos_offset= my_b_tell(&log_file); knielsen> + pthread_mutex_unlock(&LOCK_commit_ordered); knielsen> pthread_mutex_unlock(&LOCK_log); knielsen> + } Same comment needed for this. knielsen> err: knielsen> if (error) knielsen> @@ -4827,6 +4867,9 @@ bool MYSQL_BIN_LOG::write_incident(THD * knielsen> signal_update(); knielsen> rotate_and_purge(RP_LOCK_LOG_IS_ALREADY_LOCKED); knielsen> } knielsen> + pthread_mutex_lock(&LOCK_commit_ordered); knielsen> + last_commit_pos_offset= my_b_tell(&log_file); knielsen> + pthread_mutex_unlock(&LOCK_commit_ordered); knielsen> pthread_mutex_unlock(&LOCK_log); and here... knielsen> static ulonglong binlog_status_var_num_commits; knielsen> static ulonglong binlog_status_var_num_group_commits; knielsen> +static char binlog_trx_file[FN_REFLEN]; knielsen> +static ulonglong binlog_trx_position; knielsen> static SHOW_VAR binlog_status_vars_detail[]= knielsen> { knielsen> @@ -6562,12 +6609,16 @@ static SHOW_VAR binlog_status_vars_detai knielsen> (char *)&binlog_status_var_num_commits, SHOW_LONGLONG}, knielsen> {"group_commits", knielsen> (char *)&binlog_status_var_num_group_commits, SHOW_LONGLONG}, knielsen> + {"trx_file", knielsen> + (char *)&binlog_trx_file, SHOW_CHAR}, knielsen> + {"trx_position", knielsen> + (char *)&binlog_trx_position, SHOW_LONGLONG}, knielsen> {NullS, NullS, SHOW_LONG} knielsen> }; knielsen> static int show_binlog_vars(THD *thd, SHOW_VAR *var, char *buff) knielsen> { knielsen> - mysql_bin_log.set_status_variables(); knielsen> + mysql_bin_log.set_status_variables(thd); knielsen> var->type= SHOW_ARRAY; knielsen> var->value= (char *)&binlog_status_vars_detail; knielsen> return 0; knielsen> @@ -6606,17 +6657,31 @@ static struct st_mysql_sys_var *binlog_s knielsen> This is called only under LOCK_status, so we can fill in a static array. knielsen> */ knielsen> void knielsen> -TC_LOG_BINLOG::set_status_variables() knielsen> +TC_LOG_BINLOG::set_status_variables(THD *thd) knielsen> { knielsen> - ulonglong num_commits, num_group_commits; knielsen> + binlog_trx_data *trx_data; knielsen> + knielsen> + if (thd) knielsen> + trx_data= (binlog_trx_data*) thd_get_ha_data(thd, binlog_hton); knielsen> + else knielsen> + trx_data= NULL; knielsen> pthread_mutex_lock(&LOCK_commit_ordered); knielsen> - num_commits= this->num_commits; knielsen> - num_group_commits= this->num_group_commits; knielsen> + binlog_status_var_num_commits= this->num_commits; knielsen> + binlog_status_var_num_group_commits= this->num_group_commits; knielsen> + if (!trx_data || 0 == strcmp(trx_data->last_commit_pos_file, "")) -> if (!trx_data || trx_data->last_commit_pos_file[0] == 0) knielsen> + { knielsen> + strmake(binlog_trx_file, last_commit_pos_file, sizeof(binlog_trx_file)-1); knielsen> + binlog_trx_position= last_commit_pos_offset; knielsen> + } knielsen> pthread_mutex_unlock(&LOCK_commit_ordered); knielsen> + if (trx_data && 0 != strcmp(trx_data->last_commit_pos_file, "")) -> if (trx_data && trx_data->last_commit_pos_file[0] != 0) It was a bit tricky to see that this was a negative test for the earlier one. I would sugges to change first to: -> if ((binlog_pos_set= (!trx_data || trx_data->last_commit_pos_file[0] == 0)) and the later to if (!binlog_pos_set) knielsen> + { knielsen> + strmake(binlog_trx_file, trx_data->last_commit_pos_file, knielsen> + sizeof(binlog_trx_file)-1); knielsen> + binlog_trx_position= trx_data->last_commit_pos_offset; knielsen> + } knielsen> } <cut> Regards, Monty

4 4

[Maria-developers] WL#194 New (by Sergei): remove the difference between standalone and bundled plugins
by worklog-noreply＠askmonty.org 04 Apr '11

04 Apr '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: remove the difference between standalone and bundled plugins CREATION DATE..: Mon, 04 Apr 2011, 13:29 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: Sergei CATEGORY.......: Server-RawIdeaBin TASK ID........: 194 (http://askmonty.org/worklog/?tid=194) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 20 (hours remain) ORIG. ESTIMATE.: 20 PROGRESS NOTES: DESCRIPTION: Now, standalone plugins needs to be distributed witha different set of files, as compared to bundled plugins. For example, standalone plugins that use autotools need NEWS, AUTHORS, README, configure.ac, Makefile.am, ChangeLog. Bundled plugins need Makefile.am and plug.in - or CMakeLists.txt for 5.5. It means that standalone plugin cannot be built from within the tree, and bundled plugin - outside of it. It would be beneficial for plugin authors if we remove this difference. Speaking of 5.5 only, it means providing cmake include file (?) that would allow to use MYSQL_ADD_PLUGIN() macro from outside the source tree. Such a cmake file would be installed with the mysql headers and libraries. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] WL#193 New (by Sergei): Logging service
by worklog-noreply＠askmonty.org 04 Apr '11

04 Apr '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Logging service CREATION DATE..: Mon, 04 Apr 2011, 12:47 SUPERVISOR.....: Sergei IMPLEMENTOR....: COPIES TO......: Sergei CATEGORY.......: Server-RawIdeaBin TASK ID........: 193 (https://askmonty.org/worklog/?tid=193) VERSION........: Benchmarks-3.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 80 (hours remain) ORIG. ESTIMATE.: 80 PROGRESS NOTES: DESCRIPTION: logging service for plugins. destinations: * syslog * file. can be automatically rotated. * stderr, stdout (or FILE? or handle?) * table * event log on windows rewrite general, error, slow logs to use that. not all destinations and not all features need to be implemented in the first version. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0

[Maria-developers] MWL#116: Efficient group commit, ready for code review
by Kristian Nielsen 04 Apr '11

04 Apr '11

Hi Serg, Please find attached the patch for code review of MWL#116: Efficient group commit. I went through all your comments in the architecture review and updated the worklog description and the code accordingly. I hope I got everything, otherwise I hope you have the patience to point out any omissions once again! The patch is pushed here if you want to see the complete code: lp:~maria-captains/maria/mariadb-5.1-mwl116 Once the code passes code review, I think the plan is to include this feature in MariaDB 5.3. Please let me know if you need anything else from me for the review. - Kristian.

2 3

[Maria-developers] WL#192 New (by Mdcallag): optional nonblocking client API
by worklog-noreply＠askmonty.org 31 Mar '11

31 Mar '11

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: optional nonblocking client API CREATION DATE..: Wed, 30 Mar 2011, 22:13 SUPERVISOR.....: IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Client-RawIdeaBin TASK ID........: 192 (http://askmonty.org/worklog/?tid=192) VERSION........: WorkLog-4.0 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: We want some of the calls in libmysqlclient to optionally be non-blocking so it can be used with libevent as described by http://jan.kneschke.de/2008/9/9/async- mysql-queries-with-c-api. We prefer to extend libmysqlclient rather than replace it. Is this possible? ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v4.0.0)

1 0