January 2010 - developers - lists.mariadb.org

[Maria-developers] bzr commit into MariaDB 5.1, with Maria 1.5:maria branch (knielsen:2803)
by knielsen＠knielsen-hq.org 27 Jan '10

27 Jan '10

#At lp:maria 2803 knielsen(a)knielsen-hq.org 2010-01-27 Fix test failures due to previous change of not setting TZ by default. Fix by explicitly setting timezone for a few more tests that need it. (We avoid setting TZ everywhere by default as this breaks some tests on windows). Also add fix of two other windows failures due to non-portable suppressions, thanks to Alex Budovski. added: mysql-test/t/mysqlbinlog_row-master.opt mysql-test/t/mysqlbinlog_row_innodb-master.opt mysql-test/t/mysqlbinlog_row_myisam-master.opt mysql-test/t/mysqlbinlog_row_trans-master.opt modified: mysql-test/mysql-test-run.pl mysql-test/suite/maria/r/maria-recover.result mysql-test/suite/maria/t/maria-recover.test mysql-test/suite/parts/inc/partition_timestamp.inc mysql-test/suite/parts/r/partition_recover_myisam.result mysql-test/suite/parts/t/partition_recover_myisam.test per-file messages: mysql-test/mysql-test-run.pl Don't let --timezone option from one test stray into the next test (mostly to make results more deterministic rather than depend on whatever test happens to run just before). mysql-test/suite/maria/r/maria-recover.result Fix suppression pattern to also work with Windows \ path separator. mysql-test/suite/maria/t/maria-recover.test Fix suppression pattern to also work with Windows \ path separator. mysql-test/suite/parts/inc/partition_timestamp.inc Set timezone explicitly for test that needs it. mysql-test/suite/parts/r/partition_recover_myisam.result Fix suppression pattern to also work with Windows \ path separator. mysql-test/suite/parts/t/partition_recover_myisam.test Fix suppression pattern to also work with Windows \ path separator. mysql-test/t/mysqlbinlog_row-master.opt Set timezone explicitly for test that needs it. mysql-test/t/mysqlbinlog_row_innodb-master.opt Set timezone explicitly for test that needs it. mysql-test/t/mysqlbinlog_row_myisam-master.opt Set timezone explicitly for test that needs it. mysql-test/t/mysqlbinlog_row_trans-master.opt Set timezone explicitly for test that needs it. === modified file 'mysql-test/mysql-test-run.pl' --- a/mysql-test/mysql-test-run.pl 2010-01-19 10:36:52 +0000 +++ b/mysql-test/mysql-test-run.pl 2010-01-27 17:41:05 +0000 @@ -3457,6 +3457,8 @@ sub run_testcase ($$) { my $timezone= timezone($tinfo); if ($timezone ne 'DEFAULT') { $ENV{'TZ'}= $timezone; + } else { + delete($ENV{'TZ'}); } mtr_verbose("Setting timezone: $timezone"); === modified file 'mysql-test/suite/maria/r/maria-recover.result' --- a/mysql-test/suite/maria/r/maria-recover.result 2009-02-19 09:01:25 +0000 +++ b/mysql-test/suite/maria/r/maria-recover.result 2010-01-27 17:41:05 +0000 @@ -1,6 +1,3 @@ -call mtr.add_suppression("Checking table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Recovering table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Table '.\/mysqltest\/t_corrupted2' is marked as crashed and should be repaired"); select @@global.maria_recover; @@global.maria_recover BACKUP === modified file 'mysql-test/suite/maria/t/maria-recover.test' --- a/mysql-test/suite/maria/t/maria-recover.test 2009-10-09 08:09:24 +0000 +++ b/mysql-test/suite/maria/t/maria-recover.test 2010-01-27 17:41:05 +0000 @@ -2,9 +2,13 @@ --source include/have_maria.inc -call mtr.add_suppression("Checking table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Recovering table: '.\/mysqltest\/t_corrupted2'"); -call mtr.add_suppression("Table '.\/mysqltest\/t_corrupted2' is marked as crashed and should be repaired"); +--disable_query_log +# Note: \\. matches a single period. We use '.' as directory separator to +# account for Unix and Windows variation. +call mtr.add_suppression("Checking table: '\\..mysqltest.t_corrupted2'"); +call mtr.add_suppression("Recovering table: '\\..mysqltest.t_corrupted2'"); +call mtr.add_suppression("Table '\\..mysqltest.t_corrupted2' is marked as crashed and should be repaired"); +--enable_query_log # Note: we're setting an environment variable (not prefixing it by $), # so that the perl code below can access it. === modified file 'mysql-test/suite/parts/inc/partition_timestamp.inc' --- a/mysql-test/suite/parts/inc/partition_timestamp.inc 2009-10-28 07:52:34 +0000 +++ b/mysql-test/suite/parts/inc/partition_timestamp.inc 2010-01-27 17:41:05 +0000 @@ -48,6 +48,7 @@ show create table t3; let $count=12; --echo $count inserts; --disable_query_log +SET TIME_ZONE= '+03:00'; begin; while ($count) { === modified file 'mysql-test/suite/parts/r/partition_recover_myisam.result' --- a/mysql-test/suite/parts/r/partition_recover_myisam.result 2009-08-29 21:29:47 +0000 +++ b/mysql-test/suite/parts/r/partition_recover_myisam.result 2010-01-27 17:41:05 +0000 @@ -1,5 +1,3 @@ -call mtr.add_suppression("./test/t1_will_crash"); -call mtr.add_suppression("Got an error from unknown thread, ha_myisam.cc"); CREATE TABLE t1_will_crash (a INT, KEY (a)) ENGINE=MyISAM; INSERT INTO t1_will_crash VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10), (11); FLUSH TABLES; === modified file 'mysql-test/suite/parts/t/partition_recover_myisam.test' --- a/mysql-test/suite/parts/t/partition_recover_myisam.test 2009-12-03 11:19:05 +0000 +++ b/mysql-test/suite/parts/t/partition_recover_myisam.test 2010-01-27 17:41:05 +0000 @@ -1,7 +1,9 @@ # test the auto-recover (--myisam-recover) of partitioned myisam tables -call mtr.add_suppression("./test/t1_will_crash"); +--disable_query_log +call mtr.add_suppression("..test.t1_will_crash"); call mtr.add_suppression("Got an error from unknown thread, ha_myisam.cc"); +--enable_query_log --source include/have_partition.inc --disable_warnings === added file 'mysql-test/t/mysqlbinlog_row-master.opt' --- a/mysql-test/t/mysqlbinlog_row-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_innodb-master.opt' --- a/mysql-test/t/mysqlbinlog_row_innodb-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_innodb-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_myisam-master.opt' --- a/mysql-test/t/mysqlbinlog_row_myisam-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_myisam-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3 === added file 'mysql-test/t/mysqlbinlog_row_trans-master.opt' --- a/mysql-test/t/mysqlbinlog_row_trans-master.opt 1970-01-01 00:00:00 +0000 +++ b/mysql-test/t/mysqlbinlog_row_trans-master.opt 2010-01-27 17:41:05 +0000 @@ -0,0 +1 @@ +--timezone=GMT-3

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2803: YaSSL cert info buffer overflow fix
by noreply＠launchpad.net 27 Jan '10

27 Jan '10

------------------------------------------------------------ revno: 2803 committer: Sergei Golubchik <sergii(a)pisem.net> branch nick: maria-5.1 timestamp: Wed 2010-01-27 11:38:29 +0100 message: YaSSL cert info buffer overflow fix Apply a diff from [Yassl-commit] yassl/taocrypt/src asn.cpp,1.13,1.14 [Yassl-commit] yassl/taocrypt/include asn.hpp,1.9,1.10 Original patch http://lists.mysql.com/commits/96697 modified: extra/yassl/taocrypt/include/asn.hpp extra/yassl/taocrypt/src/asn.cpp -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] [Branch ~maria-captains/maria/5.1] Rev 2802: Fix for https://bugs.launchpad.net/maria/+bug/509795
by noreply＠launchpad.net 26 Jan '10

26 Jan '10

------------------------------------------------------------ revno: 2802 committer: Hakan Kuecuekyilmaz <hakan(a)askmonty.org> branch nick: maria timestamp: Fri 2010-01-22 22:19:21 +0100 message: Fix for https://bugs.launchpad.net/maria/+bug/509795 Result of reverse_lookup("localhost") is system depended. Therefore we disable the result of it. modified: mysql-test/r/udf.result mysql-test/t/udf.test -- lp:maria https://code.launchpad.net/~maria-captains/maria/5.1 Your team Maria developers is subscribed to branch lp:maria. To unsubscribe from this branch go to https://code.launchpad.net/~maria-captains/maria/5.1/+edit-subscription.

1 0

[Maria-developers] BuildBot Status Checker
by Adam M. Dutko 26 Jan '10

26 Jan '10

I realized my buildbots went offline earlier today but wasn't notified when they did. I've worked to rectify that situation by writing a small PHP script. The lastest commit is at http://git.thefuturegrid.org/?p=mariadb-scripts;a=commit;h=3d26dd7d842e2e5a… . It currently does the following: $ php checkStatus.php -h Welcome to the MariaDB BuildBot Checker: usage: php checkStatus.php ${URL} ${BUILDBOT} (show a bots info) php checkStatus.php ${URL} 1 (show all bots) note: A local file can be used in place of a URL. You can check individual buildbot info: $ php checkStatus.php http://askmonty.org/buildbot/buildslaves adutko-centos5-amd64 adutko-centos5-amd64: -------------- Used by Builders: centos5-amd64-minimal Slave is currently connected Admin: Kristian Nielsen Last heard from: about 34 minutes ago (2010-Jan-12 05:22:32) or get a list of all buildbots: $ php checkStatus.php http://askmonty.org/buildbot/buildslaves 1 All buildbots ... FOUND BOT NAME --> adutko-centos5-amd64 FOUND BOT NAME --> adutko-ultrasparc3 FOUND BOT NAME --> archivist-amd64 FOUND BOT NAME --> archivist-cnc FOUND BOT NAME --> bo-win-1 FOUND BOT NAME --> centos5 FOUND BOT NAME --> hakan-mac-g5 FOUND BOT NAME --> knielsen-kvm-x86 FOUND BOT NAME --> mariadb-brs FOUND BOT NAME --> opensolaris-x86 FOUND BOT NAME --> pcrews-win FOUND BOT NAME --> psergey-foxhole-x86 FOUND BOT NAME --> psergey-pslp2-x86 FOUND BOT NAME --> psergey-pylon-amd64 FOUND BOT NAME --> psergey-pylon-x86 FOUND BOT NAME --> psergey-win32box FOUND BOT NAME --> sanja-gentoo-x86_64 FOUND BOT NAME --> sanja-mac-x86 FOUND BOT NAME --> toby-sol10 FOUND BOT NAME --> toby-sol10-sparc FOUND BOT NAME --> valgrind FOUND BOT NAME --> vm-win-1 FOUND BOT NAME --> work-opensuse-amd64 You can also use a static file of the page you want to crawl as opposed to an online URL: $ php checkStatus.php file://buildslaves.htm adutko-centos5-amd64 adutko-centos5-amd64: -------------- Used by Builders: centos5-amd64-minimal Slave is currently connected Admin: Kristian Nielsen Last heard from: 0 seconds ago (2010-Jan-12 03:27:32) I am going to try to add an e-mail notification piece so I can be notified when a buildbot goes offline so "ideally" one could minimize downtime. I hope someone else finds this useful. -Adam

2 2

[Maria-developers] Patch for Windows problem with setting TZ
by Kristian Nielsen 26 Jan '10

26 Jan '10

Hi, Alex Budovski has done some work to debug why some tests (binlog tests in particular) are failing on Windows, and found that this is because mysql-test-run.pl set the TZ variable. This interacts poorly with the Windows Operating System. It's actually only a few test cases that need to set TZ. Here is a quick patch that avoids the need to set TZ except for tests that specifically requests it (with the already existing --timezone=XXX mechanism in -master.opt). Opinions? - Kristian.

3 6

[Maria-developers] Rev 2802: Fix for https://bugs.launchpad.net/maria/+bug/509795 in file:///Users/hakan/work/monty_program/maria/
by Hakan Kuecuekyilmaz 22 Jan '10

22 Jan '10

At file:///Users/hakan/work/monty_program/maria/ ------------------------------------------------------------ revno: 2802 revision-id: hakan(a)askmonty.org-20100122211921-e177xzarwy7l8j5y parent: knielsen(a)knielsen-hq.org-20100118125610-s55p0dsjofl9eclf committer: Hakan Kuecuekyilmaz <hakan(a)askmonty.org> branch nick: maria timestamp: Fri 2010-01-22 22:19:21 +0100 message: Fix for https://bugs.launchpad.net/maria/+bug/509795 Result of reverse_lookup("localhost") is system depended. Therefore we disable the result of it. === modified file 'mysql-test/r/udf.result' --- a/mysql-test/r/udf.result 2010-01-11 13:15:28 +0000 +++ b/mysql-test/r/udf.result 2010-01-22 21:19:21 +0000 @@ -38,8 +38,6 @@ select reverse_lookup("127.0.0.1"); select reverse_lookup(127,0,0,1); select reverse_lookup("localhost"); -reverse_lookup("localhost") -NULL select avgcost(); ERROR HY000: Can't initialize function 'avgcost'; wrong number of arguments: AVGCOST() requires two arguments select avgcost(100,23.76); === modified file 'mysql-test/t/udf.test' --- a/mysql-test/t/udf.test 2010-01-11 13:15:28 +0000 +++ b/mysql-test/t/udf.test 2010-01-22 21:19:21 +0000 @@ -56,9 +56,9 @@ --disable_result_log select reverse_lookup("127.0.0.1"); select reverse_lookup(127,0,0,1); +select reverse_lookup("localhost"); --enable_result_log -select reverse_lookup("localhost"); --error ER_CANT_INITIALIZE_UDF select avgcost(); --error ER_CANT_INITIALIZE_UDF

1 0

[Maria-developers] Updated (by Knielsen): Add support for google protocol buffers (34)
by worklog-noreply＠askmonty.org 22 Jan '10

22 Jan '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add support for google protocol buffers CREATION DATE..: Tue, 21 Jul 2009, 21:11 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 34 (http://askmonty.org/worklog/?tid=34) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Knielsen - Fri, 22 Jan 2010, 11:38)=-=- Low Level Design modified. --- /tmp/wklog.34.old.29965 2010-01-22 11:38:57.000000000 +0200 +++ /tmp/wklog.34.new.29965 2010-01-22 11:38:57.000000000 +0200 @@ -2,3 +2,12 @@ and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. +* We should have both server-side support and client-side support (client side + means functions in libmysqlclient so that user can select the full BLOB and + extract fields in the application). + +* Add some kind of header to the GPB blob to support versioning and future + extensibility. + +* Add complete syntax description (update, add, drop, exists, ...). + -=-=(Psergey - Tue, 21 Jul 2009, 21:13)=-=- Low Level Design modified. --- /tmp/wklog.34.old.6462 2009-07-21 21:13:13.000000000 +0300 +++ /tmp/wklog.34.new.6462 2009-07-21 21:13:13.000000000 +0300 @@ -1 +1,4 @@ +* GPB tarball contains a protocol definition for .proto file structure itself + and a parser for text form of .proto file which then exposes the parsed + file via standard GPB message navigation API. -=-=(Psergey - Tue, 21 Jul 2009, 21:12)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.6399 2009-07-21 21:12:23.000000000 +0300 +++ /tmp/wklog.34.new.6399 2009-07-21 21:12:23.000000000 +0300 @@ -1 +1,78 @@ +<contents> +1. GPB Encoding overview +2. GPB in an SQL database +2.1 Informing server about GPB field names and types +2.2 Addressing GPB fields +2.2.1 Option1: SQL Function +2.2.2 Option2: SQL columns +</contents> + + +1. GPB Encoding overview +======================== + +GBB is a compact encoding for structured and typed data. A unit of GPB data +(it is called message) is only partially self-describing: it's possible to +iterate over its parts, but, quoting the spec + +http://code.google.com/apis/protocolbuffers/docs/encoding.html: + " the name and declared type for each field can only be determined on the + decoding end by referencing the message type's definition (i.e. the .proto + file). " + +2. GPB in an SQL database +========================= + +It is possible to store GPB data in MariaDB today - one can declare a binary +blob column and use it to store GPB messages. Storing and retrieving entire +messages will be the only available operations, though, as the server has no +idea about the GPB format. +It is apparent that ability to peek inside GPB data from SQL layer would be of +great advantage: one would be able to +- select only certain fields or parts of GPB messages +- filter records based on the values of GPB fields +- etc +performing such operations at SQL layer will allow to reduce client<->server +traffic right away, and will open path to getting the best possible +performance. + +2.1 Informing server about GPB field names and types +---------------------------------------------------- +User-friendly/meaningful access to GPB fields requires knowledge of GPB field +names and types, which are not available from GPB message itself (see "GPB +encoding overview" section). + +So the first issue to be addressed is to get the server to know the definition +of stored messages. We intend to assume that all records have GPB messages +that conform to a certain single definition, which gives one definition per +GPB field. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify either the +definition itself or path to .proto file with the definition. + +2.2 Addressing GPB fields +------------------------- +We'll need to provide a way to access GPB fields. This can be complicated as +structures that are encoded in GPB message can be nested and recursive. + +2.2.1 Option1: SQL Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Introduce an SQL function GPB_FIELD(path) which will return contents of the +field. +- Return type of the function will be determined from GPB message definition. +- For path, we can use XPath selector (a subset of XPath) syntax. + +(TODO ^ the above needs to be specified in more detail. is the selector as +simple as filesystem path or we allow quantifiers (with predicates?)?) + +2.2.2 Option2: SQL columns +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make GPB columns to be accessible as SQL columns. +This approach has problems: +- It might be hard to implement code-wise + - (TODO will Virtual columns patch help??) +- It is not clear how to access fields from nested structures. Should we allow + quoted names like `foo/bar[2]/baz' ? + DESCRIPTION: Add support for Google Protocol Buffers (further GPB). It should be possible to have columns that store GPB-encoded data, as well as use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. Any support for indexing GPB data is outside of scope of this WL entry. HIGH-LEVEL SPECIFICATION: <contents> 1. GPB Encoding overview 2. GPB in an SQL database 2.1 Informing server about GPB field names and types 2.2 Addressing GPB fields 2.2.1 Option1: SQL Function 2.2.2 Option2: SQL columns </contents> 1. GPB Encoding overview ======================== GBB is a compact encoding for structured and typed data. A unit of GPB data (it is called message) is only partially self-describing: it's possible to iterate over its parts, but, quoting the spec http://code.google.com/apis/protocolbuffers/docs/encoding.html: " the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file). " 2. GPB in an SQL database ========================= It is possible to store GPB data in MariaDB today - one can declare a binary blob column and use it to store GPB messages. Storing and retrieving entire messages will be the only available operations, though, as the server has no idea about the GPB format. It is apparent that ability to peek inside GPB data from SQL layer would be of great advantage: one would be able to - select only certain fields or parts of GPB messages - filter records based on the values of GPB fields - etc performing such operations at SQL layer will allow to reduce client<->server traffic right away, and will open path to getting the best possible performance. 2.1 Informing server about GPB field names and types ---------------------------------------------------- User-friendly/meaningful access to GPB fields requires knowledge of GPB field names and types, which are not available from GPB message itself (see "GPB encoding overview" section). So the first issue to be addressed is to get the server to know the definition of stored messages. We intend to assume that all records have GPB messages that conform to a certain single definition, which gives one definition per GPB field. DecisionToMake: How to pass the server the GPB definition? First idea: add a CREATE TABLE parameter which will specify either the definition itself or path to .proto file with the definition. 2.2 Addressing GPB fields ------------------------- We'll need to provide a way to access GPB fields. This can be complicated as structures that are encoded in GPB message can be nested and recursive. 2.2.1 Option1: SQL Function ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Introduce an SQL function GPB_FIELD(path) which will return contents of the field. - Return type of the function will be determined from GPB message definition. - For path, we can use XPath selector (a subset of XPath) syntax. (TODO ^ the above needs to be specified in more detail. is the selector as simple as filesystem path or we allow quantifiers (with predicates?)?) 2.2.2 Option2: SQL columns ~~~~~~~~~~~~~~~~~~~~~~~~~~ Make GPB columns to be accessible as SQL columns. This approach has problems: - It might be hard to implement code-wise - (TODO will Virtual columns patch help??) - It is not clear how to access fields from nested structures. Should we allow quoted names like `foo/bar[2]/baz' ? LOW-LEVEL DESIGN: * GPB tarball contains a protocol definition for .proto file structure itself and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. * We should have both server-side support and client-side support (client side means functions in libmysqlclient so that user can select the full BLOB and extract fields in the application). * Add some kind of header to the GPB blob to support versioning and future extensibility. * Add complete syntax description (update, add, drop, exists, ...). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): Add support for google protocol buffers (34)
by worklog-noreply＠askmonty.org 22 Jan '10

22 Jan '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add support for google protocol buffers CREATION DATE..: Tue, 21 Jul 2009, 21:11 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 34 (http://askmonty.org/worklog/?tid=34) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Knielsen - Fri, 22 Jan 2010, 11:38)=-=- Low Level Design modified. --- /tmp/wklog.34.old.29965 2010-01-22 11:38:57.000000000 +0200 +++ /tmp/wklog.34.new.29965 2010-01-22 11:38:57.000000000 +0200 @@ -2,3 +2,12 @@ and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. +* We should have both server-side support and client-side support (client side + means functions in libmysqlclient so that user can select the full BLOB and + extract fields in the application). + +* Add some kind of header to the GPB blob to support versioning and future + extensibility. + +* Add complete syntax description (update, add, drop, exists, ...). + -=-=(Psergey - Tue, 21 Jul 2009, 21:13)=-=- Low Level Design modified. --- /tmp/wklog.34.old.6462 2009-07-21 21:13:13.000000000 +0300 +++ /tmp/wklog.34.new.6462 2009-07-21 21:13:13.000000000 +0300 @@ -1 +1,4 @@ +* GPB tarball contains a protocol definition for .proto file structure itself + and a parser for text form of .proto file which then exposes the parsed + file via standard GPB message navigation API. -=-=(Psergey - Tue, 21 Jul 2009, 21:12)=-=- High-Level Specification modified. --- /tmp/wklog.34.old.6399 2009-07-21 21:12:23.000000000 +0300 +++ /tmp/wklog.34.new.6399 2009-07-21 21:12:23.000000000 +0300 @@ -1 +1,78 @@ +<contents> +1. GPB Encoding overview +2. GPB in an SQL database +2.1 Informing server about GPB field names and types +2.2 Addressing GPB fields +2.2.1 Option1: SQL Function +2.2.2 Option2: SQL columns +</contents> + + +1. GPB Encoding overview +======================== + +GBB is a compact encoding for structured and typed data. A unit of GPB data +(it is called message) is only partially self-describing: it's possible to +iterate over its parts, but, quoting the spec + +http://code.google.com/apis/protocolbuffers/docs/encoding.html: + " the name and declared type for each field can only be determined on the + decoding end by referencing the message type's definition (i.e. the .proto + file). " + +2. GPB in an SQL database +========================= + +It is possible to store GPB data in MariaDB today - one can declare a binary +blob column and use it to store GPB messages. Storing and retrieving entire +messages will be the only available operations, though, as the server has no +idea about the GPB format. +It is apparent that ability to peek inside GPB data from SQL layer would be of +great advantage: one would be able to +- select only certain fields or parts of GPB messages +- filter records based on the values of GPB fields +- etc +performing such operations at SQL layer will allow to reduce client<->server +traffic right away, and will open path to getting the best possible +performance. + +2.1 Informing server about GPB field names and types +---------------------------------------------------- +User-friendly/meaningful access to GPB fields requires knowledge of GPB field +names and types, which are not available from GPB message itself (see "GPB +encoding overview" section). + +So the first issue to be addressed is to get the server to know the definition +of stored messages. We intend to assume that all records have GPB messages +that conform to a certain single definition, which gives one definition per +GPB field. + +DecisionToMake: How to pass the server the GPB definition? +First idea: add a CREATE TABLE parameter which will specify either the +definition itself or path to .proto file with the definition. + +2.2 Addressing GPB fields +------------------------- +We'll need to provide a way to access GPB fields. This can be complicated as +structures that are encoded in GPB message can be nested and recursive. + +2.2.1 Option1: SQL Function +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Introduce an SQL function GPB_FIELD(path) which will return contents of the +field. +- Return type of the function will be determined from GPB message definition. +- For path, we can use XPath selector (a subset of XPath) syntax. + +(TODO ^ the above needs to be specified in more detail. is the selector as +simple as filesystem path or we allow quantifiers (with predicates?)?) + +2.2.2 Option2: SQL columns +~~~~~~~~~~~~~~~~~~~~~~~~~~ +Make GPB columns to be accessible as SQL columns. +This approach has problems: +- It might be hard to implement code-wise + - (TODO will Virtual columns patch help??) +- It is not clear how to access fields from nested structures. Should we allow + quoted names like `foo/bar[2]/baz' ? + DESCRIPTION: Add support for Google Protocol Buffers (further GPB). It should be possible to have columns that store GPB-encoded data, as well as use SQL constructs to extract parts of GPB data for use in select list, for filtering, and so forth. Any support for indexing GPB data is outside of scope of this WL entry. HIGH-LEVEL SPECIFICATION: <contents> 1. GPB Encoding overview 2. GPB in an SQL database 2.1 Informing server about GPB field names and types 2.2 Addressing GPB fields 2.2.1 Option1: SQL Function 2.2.2 Option2: SQL columns </contents> 1. GPB Encoding overview ======================== GBB is a compact encoding for structured and typed data. A unit of GPB data (it is called message) is only partially self-describing: it's possible to iterate over its parts, but, quoting the spec http://code.google.com/apis/protocolbuffers/docs/encoding.html: " the name and declared type for each field can only be determined on the decoding end by referencing the message type's definition (i.e. the .proto file). " 2. GPB in an SQL database ========================= It is possible to store GPB data in MariaDB today - one can declare a binary blob column and use it to store GPB messages. Storing and retrieving entire messages will be the only available operations, though, as the server has no idea about the GPB format. It is apparent that ability to peek inside GPB data from SQL layer would be of great advantage: one would be able to - select only certain fields or parts of GPB messages - filter records based on the values of GPB fields - etc performing such operations at SQL layer will allow to reduce client<->server traffic right away, and will open path to getting the best possible performance. 2.1 Informing server about GPB field names and types ---------------------------------------------------- User-friendly/meaningful access to GPB fields requires knowledge of GPB field names and types, which are not available from GPB message itself (see "GPB encoding overview" section). So the first issue to be addressed is to get the server to know the definition of stored messages. We intend to assume that all records have GPB messages that conform to a certain single definition, which gives one definition per GPB field. DecisionToMake: How to pass the server the GPB definition? First idea: add a CREATE TABLE parameter which will specify either the definition itself or path to .proto file with the definition. 2.2 Addressing GPB fields ------------------------- We'll need to provide a way to access GPB fields. This can be complicated as structures that are encoded in GPB message can be nested and recursive. 2.2.1 Option1: SQL Function ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Introduce an SQL function GPB_FIELD(path) which will return contents of the field. - Return type of the function will be determined from GPB message definition. - For path, we can use XPath selector (a subset of XPath) syntax. (TODO ^ the above needs to be specified in more detail. is the selector as simple as filesystem path or we allow quantifiers (with predicates?)?) 2.2.2 Option2: SQL columns ~~~~~~~~~~~~~~~~~~~~~~~~~~ Make GPB columns to be accessible as SQL columns. This approach has problems: - It might be hard to implement code-wise - (TODO will Virtual columns patch help??) - It is not clear how to access fields from nested structures. Should we allow quoted names like `foo/bar[2]/baz' ? LOW-LEVEL DESIGN: * GPB tarball contains a protocol definition for .proto file structure itself and a parser for text form of .proto file which then exposes the parsed file via standard GPB message navigation API. * We should have both server-side support and client-side support (client side means functions in libmysqlclient so that user can select the full BLOB and extract fields in the application). * Add some kind of header to the GPB blob to support versioning and future extensibility. * Add complete syntax description (update, add, drop, exists, ...). ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Knielsen): Re-merge PBXT into MariaDB with proper history using bzr merge-into (76)
by worklog-noreply＠askmonty.org 22 Jan '10

22 Jan '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Re-merge PBXT into MariaDB with proper history using bzr merge-into CREATION DATE..: Fri, 22 Jan 2010, 10:03 SUPERVISOR.....: Knielsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-BackLog TASK ID........: 76 (http://askmonty.org/worklog/?tid=76) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: PBXT is maintained in its own repository separate from MariaDB (as it is also used for MySQL and Drizzle). Merging new PBXT features into MariaDB is currently a manual process where a copy inside the MariaDB tree is manually patched to match the latest version in the real PBXT repository. This is a pity, as it requires extra work, and the PBXT revision history is not available in the MariaDB tree. Bzr actually has a feature to properly merge one repository into another with preservation of full history and bzr-assisted merging. This is used for merging XtraDB into MariaDB. Some information on this here: http://askmonty.org/wiki/index.php/Merging_into_MariaDB#Merging_from_XtraDB… We want to do this for PBXT also. However, this is currently blocked on a bug in bzr: bug: https://bugs.launchpad.net/bzr/+bug/375898 Once this bug is resolved, we would want to re-do the inclusion of PBXT into MariaDB, using `bzr merge-into` and following the procedure in the bug report. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Knielsen): Extend build to create a shared libmysqld.so library (75)
by worklog-noreply＠askmonty.org 22 Jan '10

22 Jan '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Extend build to create a shared libmysqld.so library CREATION DATE..: Fri, 22 Jan 2010, 09:39 SUPERVISOR.....: Knielsen IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 75 (http://askmonty.org/worklog/?tid=75) VERSION........: Server-9.x STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Currently, the embedded library libmysqld is only built as a static library, libmysqld.a. However, there is also a need for a shared embedded library, libmysqld.so. A good example is Amarok, which is now using MySQL (and by default an embedded libmysqld) for storing the user's music collection. Amarok is designed to load a number of modules as .so plugins, and the code using libmysqld is one such plugin. Any code that is to be loaded as a shared object on Linux must be built with -fPIC (position independent code). Any library used must thus also be built with -fPIC and preferably itself be linked statically. Amarok is a widely used application (end-user desktop). It could be a good leverage for making libmysqld more popular eg. in distros. However, currently the distros need to resort to various hacks to make things work with libmysqld and Amarok. Some links: Fedora patches .spec to add -fPIC in the build, extract all objects from libmysqld.a, and re-link them as libmysqld.so. http://cvs.fedoraproject.org/viewvc/devel/mysql/mysql.spec?r1=1.108&r2=1.109 Gentoo apparently have a patch for MySQL to build libmysqld.so: http://bugs.gentoo.org/attachment.cgi?id=188057 Debian seems to suggest building both libmysqld.a and libmysqld_pic.a, the latter to be used for shared objects linking with embedded server: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=508406#52 It would be nice if the main build system was able to build properly libmysqld.so, so different distros would not need to resort to different hacks to get things working. The normal way to build both .a and .so as far as I know is to use libtool; it will then build each object twice (with and without -fPIC) if needed. This one might want to make optional to reduce build times. It would also need to be done for each storage engine. An alternative would be to build everything with -fPIC. This should probably be optional also, as on some architectures (ELF x86 32-bit), there is some speed penalty for -fPIC. The libmysqld.so would then be made only in this case. Distro packages could do a separate build with -fPIC to make libmysqld.so (to not get -fPIC into the main server code). There is a MySQL bug for this: http://bugs.mysql.com/bug.php?id=39288 ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0