- developers - lists.mariadb.org

[Maria-developers] Bazaar-problem
by AugustQ 06 Sep '13

06 Sep '13

Hi, I'm struggling with Bazaar. Maybe you can help me. I've made some modifications to the source-code and committed these to the local repository. Next I tried to upload them via push. This is what I receive: Cannot lock LockDir(chroot-70341520:///% 2Bbranch/maria/.bzr/branch/lock): Transport operation not possible: readonly transport As I'm using Eclipse I found this within Eclipse on the console: Error while executing push Cannot lock LockDir(chroot-70341520:///% 2Bbranch/maria/.bzr/branch/lock): Transport operation not possible: readonly transport ------ XmlRpcClient Executing: push "bzr+ssh://bazaar.launchpad.net/+branch/maria/" Can you please give me a tip. Thanks AugustQ

5 9

[Maria-developers] Incorrect format description event skipping in queue_event()
by Pavel Ivanov 04 Sep '13

04 Sep '13

Kristian, You have the following comment in the queue_event() in sql/slave.cc: /* Do not queue any format description event that we receive after a reconnect where we are skipping over a partial event group received before the reconnect. (If we queued such an event, and it was the first format_description event after master restart, the slave SQL thread would think that the partial event group before it in the relay log was from a previous master crash and should be rolled back). */ I don't understand which failure scenario you are talking about here and I claim that this bypassing of queuing into relay log is incorrect. My reasoning is: relay log can always be rotated in the middle of event group. During rotation when new relay log file is created format description event is always written at the beginning of the file. So in this case SQL thread will see format description event in the middle of event group and according to my testing it doesn't rollback the active transaction because of that. So when do you think it will rollback? Now, you may say that there's no problem in skipping the format description event because everything works. But it's only for now. When IO thread is reconnecting it rotates relay log and as I said it writes format description event at the beginning of the new file. But it writes an event that it created itself, i.e. not the one that master have sent. And as format description event from master is not written into relay log SQL thread from this point on starts to use format description generated by slave which may be different from the one generated by master. It may lead to a broken replication and SQL thread may not recognize events in the relay log if difference between master's and slave's format descriptions is significant. Another somewhat related question: Gtid_log_event::peek() (as well as Gtid_log_event constructor from const char* buf) is implemented with assumption that Format_description_log_event::common_header_len is always equal to LOG_EVENT_HEADER_LEN. While currently it's true I believe common_header_len can be increased in future MariaDB versions. And when it happens old slave won't be able to replicate from a new master. I'd think this is a bug. What do you think? Pavel

2 10

Re: [Maria-developers] Rev 3876: MDEV-4472 Audit-plugin. Server-related part of the task
by Sergei Golubchik 04 Sep '13

04 Sep '13

Hi, Holyfoot! Mostly ok. There are few comments, see below: On Sep 03, holyfoot(a)askmonty.org wrote: > ------------------------------------------------------------ > revno: 3876 > revision-id: holyfoot(a)askmonty.org-20130903082521-xpia2wond6oggpsk > parent: igor(a)askmonty.org-20130831163309-vqtls4oubhvz9zcz > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > branch nick: 55-4472 > timestamp: Tue 2013-09-03 13:25:21 +0500 > message: > MDEV-4472 Audit-plugin. Server-related part of the task. > file_logger became the service. > Data like query_id now are sent to the audit plugin. > Fix for MDEV-4770 ported from 10.0. > Fix added for the read_maria_plugin_info(). > === modified file 'include/mysql/plugin_audit.h' > --- a/include/mysql/plugin_audit.h 2013-04-19 10:50:16 +0000 > +++ b/include/mysql/plugin_audit.h 2013-09-03 08:25:21 +0000 > @@ -25,7 +25,7 @@ > > #define MYSQL_AUDIT_CLASS_MASK_SIZE 1 > > -#define MYSQL_AUDIT_INTERFACE_VERSION 0x0301 > +#define MYSQL_AUDIT_INTERFACE_VERSION 0x0302 > > > /************************************************************************* > @@ -59,6 +59,10 @@ struct mysql_event_general > struct charset_info_st *general_charset; > unsigned long long general_time; > unsigned long long general_rows; > + /* Added for 0x302 */ A bit confusing. Better "Added in version 0x0302" > + long long query_id; should it rather be unsigned? > + const char *database; > + unsigned int database_length; > }; > > > @@ -140,6 +144,8 @@ struct mysql_event_table > unsigned int new_database_length; > const char *new_table; > unsigned int new_table_length; > + /* Added for 0x302 */ > + long long query_id; > }; > > /************************************************************************* > > === renamed file 'plugin/sql_errlog/sql_logger.cc' => 'mysys/file_logger.c' > --- a/plugin/sql_errlog/sql_logger.cc 2013-06-22 12:02:03 +0000 > +++ b/mysys/file_logger.c 2013-09-03 08:25:21 +0000 > @@ -51,6 +53,12 @@ LOGGER_HANDLE *logger_open(const char *p > unsigned int rotations) > { > LOGGER_HANDLE new_log, *l_perm; > + char *local_mysql_data_home; > +#ifdef _WIN32 > + local_mysql_data_home= (char *)GetProcAddress(0, "mysql_data_home"); > +#else > + local_mysql_data_home= mysql_data_home; > +#endif /*_WIN32*/ You shouldn't need it, file_logger.c is part of the server, not a dynamically loaded plugin. > > /* > I don't think we ever need more rotations, > === modified file 'sql/sql_audit.h' > --- a/sql/sql_audit.h 2013-04-19 10:50:16 +0000 > +++ b/sql/sql_audit.h 2013-09-03 08:25:21 +0000 > @@ -93,7 +93,8 @@ void mysql_audit_general_log(THD *thd, t > > mysql_audit_notify(thd, MYSQL_AUDIT_GENERAL_CLASS, MYSQL_AUDIT_GENERAL_LOG, > 0, time, user, userlen, cmd, cmdlen, > - query, querylen, clientcs, 0); > + query, querylen, clientcs, (ha_rows) 0, > + thd->db, thd->db_length); no query_id? > } > } > > @@ -139,7 +140,8 @@ void mysql_audit_general(THD *thd, uint > > mysql_audit_notify(thd, MYSQL_AUDIT_GENERAL_CLASS, event_subtype, > error_code, time, user, userlen, msg, msglen, > - query.str(), query.length(), query.charset(), rows); > + query.str(), query.length(), query.charset(), rows, > + thd->db, thd->db_length); no query_id? > } > } > > === modified file 'sql/sql_plugin.cc' > --- a/sql/sql_plugin.cc 2013-06-13 18:19:11 +0000 > +++ b/sql/sql_plugin.cc 2013-09-03 08:25:21 +0000 > @@ -2645,13 +2647,16 @@ static void update_func_longlong(THD *th > static void update_func_str(THD *thd, struct st_mysql_sys_var *var, > void *tgt, const void *save) > { > - char *old= *(char **) tgt; > - *(char **)tgt= *(char **) save; > + char *value= *(char**) save; > if (var->flags & PLUGIN_VAR_MEMALLOC) > { > - *(char **)tgt= my_strdup(*(char **) save, MYF(0)); > + char *old= *(char**) tgt; > + if (value) > + *(char**) tgt= my_strdup(value, MYF(0)); add else *(char**) tgt= 0; > my_free(old); > } > + else > + *(char**) tgt= value; > } > Regards, Sergei

2 3

[Maria-developers] RFC - CREATE PROCEDURE clauses (question about docs from mariadb-discuss)
by Roberto Spadim 02 Sep '13

02 Sep '13

Hi mariadb-devel guys, could anyone help here in a documentation problem? (maybe not a problem...) *Please answer the BOLD questions* We are talking about CREATE PROCEDURE , about DETERMINISTIC / NO SQL http://dev.mysql.com/doc/refman/5.6/en/create-procedure.html *characteristic*: COMMENT '*string*' | LANGUAGE SQL | *[NOT] DETERMINISTIC* | { CONTAINS SQL | *NO SQL* | READS SQL DATA | MODIFIES SQL DATA } | SQL SECURITY { DEFINER | INVOKER } *--------* *from mariadb-discuss:* *--------* Hi Federico, sorry i read some points and start writing, i will read and write after... i will answer with my experience as an user, not as a mariadb/mysql developer... 2013/9/2 Federico Razzoli <federico_raz(a)yahoo.it> > Hi Roberto > I'll try to be more clear, please read carefully before answering > ok I know what a deterministic expression is - but *what kinds of output must > be deterministic for a procedure to be defined as DETERMINISTIC?* > hummm i don't know, but for procedure, there isn't a deterministic output right? just for funcitons... maybe it's a parser problem, allow deterministic keyword for procedures (must check) i'm thinking about, *why we use deterministic procs, and non deterministic procs?* *this change something important in mysql server? replication, query cache? other performace considerations?* > There are many types of output for a proc: > * OUT params > * SELECTs > * writes on tables > * SET @var > hum.... from what i remember procedure ~= function without return, something like a "void FUNCTION( any parameters )" function ~= a procedure that return values, in other words it must be different from void... that's why i think it's a problem at parser allowing deterministic keyword at procedure, but must check... i don't know if this keyword change something in mysql/mariadb code > For a deterministic proc, do all of these need to be deterministic? > well, for me, procs are always non deterministics since it don't change query cache invalidation and replication (must be done in binary i think) > * I guess the answer isn't easy. > * *yeah, i see, we need a simple answer not a simplified answer* > -- > For "NO SQL" a complete list of SQL statements would be needed. If we use > this as a reference: > http://dev.mysql.com/doc/refman/5.6/en/sql-syntax.html > , even RETURN seems to be a SQL statement, yes, maybe the point isn't the SQL statements list, but other functions that mysql/mariadb server do? i don't know... NO SQL maybe is different from NOSQL (memcache,mongo,etc...) since nosql is newer than mysql procedures the point is *what's NO SQL in CREATE PROCEDURE ?* but of course it can't be true, because it's impossible to write a func > without RETURN. > yeah, maybe we have a problem at docs... let's see if anyone else know about this... maybe we need help from mariadb-devel list, i will include in this email... > > Greetings > Federico > bye =) thanks guys -- Roberto Spadim

1 0

Re: [Maria-developers] FWD: Status on MDEV-4506, parallel replication
by Kristian Nielsen 02 Sep '13

02 Sep '13

Michael Widenius <monty(a)askmonty.org> writes: >> On the master, I implemented --binlog-commit-wait-count=N and >> --binlog-commit-wait-usec=T. A transaction will wait at most T microseconds >> for at least N transactions to queue up and be ready for group commit. This >> allows to deliberately delay transactions on the master in order to get bigger >> group commits and thus better opportunity for parallel execution (and again it >> makes testing easier). > > Do you think the above helps in any real world case, except testing? > > Assuming we are have N=4 and T=2 > > - We have 3 threads ready to commit > - We wait 2 milliseconds and get 2 more threads. > - We commit now 5 threads. During this time you get 3 other threads > waiting to commit. > - We wait again... > > In a scenario without waiting: > > - We have 3 threads ready to commit > - We commit 3 threads. During this time we get 5 more threads waiting > to commit. > - We commit 5 threads. > > In other words, as the group commit will anyway take up 50 > microseconds on a hard disk, and thus automaticly group things > together for the next commit, why do we need ever to wait more? > > I agree that with --binlog-commit-wait-count you may get less sync > calls, but at an expense that a lot of threads took up to T > microseconds longer to execute. > > The worst case scenario is when you have only one user doing a lot of > inserts with auto-commit. In this case using wait will slow down the > server with T microseconds for every query. > > Have you been able to run any kind of benchmark where using > binglog-commit-wait-count will give more performance? > > I agree that the options are good for testing. The main question I > have is if we want to have the variables in the production server and > how we should document when and how a user should use the variables. I agree that with respect to group commit, the --binlog-commit-wait-count in many cases will not improve performance (that is why I did not implement it earlier). However, for parallel replication things are a bit different. Suppose that the application is doing C transaction commits per second, and that the disk system is capable of F binlog fsyncs per second. Now, if C is significantly bigger than F, then things are as you describe. Generally several transactions will queue up while the previous group commit runs, and there will be sufficient parallelism without using --bicommit_wait-*. This will be typical for eg. a simple harddisk-based system (F=40 commits/second perhaps) where all data is cached in the InnoDB buffer pool (eg. C>500 transactions/second). On the other hand, if C is smaller than F (or of similar magnitide), then usually only few or no new transactions will have time to queue up while the previous transaction is committing. So there will not be much parallelism for parallel replication to exploit without using --binlog-commit-wait-*. This will be typical for eg. a good-quality server with battery-backup RAID controller (F > 1000 commits/second) where data is too big to fit in buffer pool and every update requires disk access to complete (C < 500 transactions/second for example). And in fact, it is the second case, where random I/O is the bottleneck and multiple disk spindels are needed to improve I/O throughput, that single-threaded slave hurts the most, and where increasing --binlog-commit-wait-usec can be used with least penalty. I agree we need to document clearly the risk that --binlog-commit-wait-* will decrease performance on the master. Basically, the user can look at the ratio of Binlog_commits and Binlog_group_commits to check if enough transactions are part of each group commit for parallel replication to be effective. >> 1. The existing code is not thread-safe for class Relay_log_info. This class >> contains a bunch of stuff that is specific to executed transactions, not >> related to relay-log at all. This needs to be moved to the new struct >> rpl_group_info I introduced, and all code updated to pass around a pointer to >> that struct instead. There may also be a need to add additional locking on >> Relay_log_info, existing code needs review for this. > > I could take a look working on the above tomorrow and next week. Ok, great. Ping me when you can so we can coordinate, I may have some partial patches for this lying around. - Kristian.

1 0

[Maria-developers] Fwd: Bazaar-problem
by Nitish Upreti 31 Aug '13

31 Aug '13

I cant even clone the repository. Bazaar crashes with a bug midway. Can we have an active git mirror of the project for people just starting out? On Sat, Aug 31, 2013 at 2:13 PM, Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, AugustQ! > > On Aug 31, AugustQ wrote: > > I'm struggling with Bazaar. Maybe you can help me. > > > > I've made some modifications to the source-code and committed these to > > the local repository. Next I tried to upload them via push. This is what > > I receive: > > > > Cannot lock LockDir(chroot-70341520:///% > > 2Bbranch/maria/.bzr/branch/lock): Transport operation not possible: > > readonly transport > > You don't have write access to the mariadb source repository. > That's why you can only pull, but you cannot push. > > Regards, > Sergei > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-developers > Post to : maria-developers(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-developers > More help : https://help.launchpad.net/ListHelp >

2 1

[Maria-developers] introduction
by AugustQ 29 Aug '13

29 Aug '13

Hi, i've joined this list some days ago because I want to participate in the discussions here. The following links give you an overview of what i've done in the past: without registration: https://www.xing.com/profiles/August_Quint with registration: http://www.linkedin.com/profile/view?id=9080630 unfortunately the texts are in German, so in short: i've worked as a software-developer mainly in the backend area. I've made some small modifications on the source code but currently I'm struggling with Bazaar. This should be solved soon and I will then upload some code to the repository. I hope I can help you a bit. AugustQ PS: i found this: in MySQL I call cmake with an argument: -DWITH_DEBUG=1 in MariaDB this is: -DCMAKE_BUILD_TYPE=Debug is this correct?

2 1

[Maria-developers] alter online table documentation
by Roberto Spadim 26 Aug '13

26 Aug '13

Hi guys, i was searching alter online docs, at mariadb https://kb.askmonty.org/en/alter-table-statement/ but didn't found anything, could we add ? thanks! -- Roberto Spadim

4 5

Re: [Maria-developers] Calll for testing: MariaDB 5.5.32 packages
by Otto Kekäläinen 23 Aug '13

23 Aug '13

2013/8/23 Сергей Войтович <svojtovich(a)gmail.com>: > thanks for packaging MariaDB. I did a few basic tests with your packages and I confirm that they work well. > The only minor note I have is that I was asked for root password 3x during installation, which is a bit too much. Thanks! I haven't noticed any extra password asking during installation. It should ask for MariaDB root password in two screens, and of course if you run sudo apt-get it will ask once for your user password in the beginning. Could you clarify a little what happened in you installation?

2 3

[Maria-developers] Additional documentation for GTID
by Kristian Nielsen 23 Aug '13

23 Aug '13

Hi Daniel, Ian, I have implemented a new system variable for global transaction ID and pushed it into 10.0-base (so it should appear in 10.0.5 I suppose). I have written some text for documenting the new variable, appended below. Can one of you help me get this text into the Knowledgebase with proper formatting and cross-referencing across relevant pages? - Kristian. ----------------------------------------------------------------------- Variable: gtid_binlog_state Scope: global Dynamic: Yes Type: String The variable gtid_binlog_state holds the internal state of the binlog. The state consists of the last GTID ever logged to the binary log for every combination of domain_id and server_id. This information is used by the master to determine whether a given GTID has been logged to the binlog in the past, even if it has later been deleted due to binlog purge. For each domain_id, the last entry in @@gtid_binlog_state is the last GTID logged into binlog, ie. this is the value that appears in @@gtid_binlog_pos. Normally this internal state is not needed by users, as @@gtid_binlog_pos is more useful in most cases. The main usage of @@gtid_binlog_state is to restore the state of the binlog after RESET MASTER (or equivalently if the binlog files are lost). If the value of @@gtid_binlog_state is saved before RESET MASTER and restored afterwards, the master will retain information about past history, same as if PURGE BINARY LOGS had been used (of course the actual events in the binary logs are still deleted). Note that to set the value of @@gtid_binlog_state, the binary log must be empty, that is it must not contain any GTID events and the previous value of @@gtid_binlog_state must be the empty string. If not, then RESET MASTER must be used first to erase the binary log first. For completeness, note that setting @@gtid_binlog_state internally executes a RESET MASTER. This is normally not noticable as it can only be changed when the binlog is empty of GTID events. However, if executed eg. immediately after upgrading to MariaDB 10, it is possible that the binlog is non-empty but without any GTID events, in which case all such events will be deleted, just as if RESET MASTER had been run.

2 1