----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Add Sphinx storage engine to MariaDB CREATION DATE..: Mon, 10 Aug 2009, 23:57 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-Sprint TASK ID........: 42 (http://askmonty.org/worklog/?tid=42) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 9 ESTIMATE.......: 7 (hours remain) ORIG. ESTIMATE.: 16 PROGRESS NOTES: -=-=(Knielsen - Mon, 07 Jun 2010, 07:12)=-=- Help Andrew with the integration. Worked 2 hours and estimate 7 hours remain (original estimate unchanged). -=-=(Knielsen - Mon, 31 May 2010, 06:49)=-=- Wrote patch that allows to test SphinxSE in mysql-test-run, using external Sphinx daemon. Worked 7 hours and estimate 9 hours remain (original estimate unchanged). -=-=(Knielsen - Fri, 28 May 2010, 07:49)=-=- High-Level Specification modified. --- /tmp/wklog.42.old.4369 2010-05-28 07:49:13.000000000 +0000 +++ /tmp/wklog.42.new.4369 2010-05-28 07:49:13.000000000 +0000 @@ -49,6 +49,10 @@ might be possible to pre-generate the necessary data/index files and store them in the source tree. +I pushed a proof-of-concept patch for this here: + + lp:~knielsen/maria/5.2-sphinxse + Here is a sample test case using this: --source include/have_sphinx.inc -=-=(Knielsen - Fri, 28 May 2010, 06:31)=-=- High-Level Specification modified. --- /tmp/wklog.42.old.32746 2010-05-28 06:31:24.000000000 +0000 +++ /tmp/wklog.42.new.32746 2010-05-28 06:31:24.000000000 +0000 @@ -1 +1,63 @@ +Code +---- + +Andrew Aksyonoff from Sphinx is helping to integrate the SphinxSE plugin into +the MariaDB tree. + +It is a plugin, so it can be added to the tree just by including the +sub-directory storage/sphinx/. + +The Sphinx plugin is already of some maturity, having been used with MySQL for +some time. + + +Testing +------- + +To get testing in the mysql-test-run framework, some extensions are needed. + +To use the Sphinx storage engine, the external Sphinx search daemon needs to +be running with some data directory containing indexed data. It also needs to +be allocated a port. + +This is the indended approach: + +1. Testing will use an external Sphinx setup installed on the machine. Sphinx +binaries will be searched in typical locations (eg. /usr/bin, /usr/local/bin), +or can be specified explicitly in the environment with SPHINXSEARCH_INDEXER +and SPHINXSEARCH_SEARCHD for the two required binaries. If the external Sphinx +binaries can not be found, then Sphinx tests will be disabled (using some +--source include/have_sphinx.inc in the test cases). + +2. The mysql-test-run framework will install Sphinx search data and start/stop +the Sphinx search daemon for the test cases, similarly how it is done for the +other servers mysqld, ndbd, etc. We will run the Sphinx search daemon with +options --console, --config, and --pidfile. + +3. The mysql-test-run framework will generate a Sphinx config file from a +template in mysql-test/suite/sphinx/my.cnf. This config file will allocate +ports and data directories appropriate for avoiding conflicts between multiple +simultaneous mysql-test-run executions. The Sphinx config file is sufficiently +similar to MySQL my.cnf that we can use the existing framework for generating +config file, with just a slightly modified variant of the code writing the +file to disk. + +4. The mysql-test-run framework will pre-load the mysql database with tables +and data for Sphinx to index. It will then run the `indexer` program to +generate the indexes, and then start the `searchd` daemon. These three steps +must be done in order, as each step depends on the previous. ALTERNATIVE: it +might be possible to pre-generate the necessary data/index files and store +them in the source tree. + +Here is a sample test case using this: + +--source include/have_sphinx.inc +--source include/have_sphinxse.inc + +--replace_result $SPHINXSEARCH_PORT SPHINXSEARCH_PORT +eval create table ts ( id int unsigned not null, w int not null, q varchar(255) +not null, index(q) ) engine=sphinx +connection="sphinx://127.0.0.1:$SPHINXSEARCH_PORT/*"; +select * from ts where q='test'; +drop table ts; -=-=(Knielsen - Fri, 28 May 2010, 06:07)=-=- Version updated. --- /tmp/wklog.42.old.32184 2010-05-28 06:07:00.000000000 +0000 +++ /tmp/wklog.42.new.32184 2010-05-28 06:07:00.000000000 +0000 @@ -1 +1 @@ -9.x +Server-5.2 -=-=(Knielsen - Fri, 28 May 2010, 06:06)=-=- Category updated. --- /tmp/wklog.42.old.32171 2010-05-28 06:06:23.000000000 +0000 +++ /tmp/wklog.42.new.32171 2010-05-28 06:06:23.000000000 +0000 @@ -1 +1 @@ -Maria-BackLog +Server-Sprint -=-=(Knielsen - Fri, 28 May 2010, 06:06)=-=- Version updated. --- /tmp/wklog.42.old.32171 2010-05-28 06:06:23.000000000 +0000 +++ /tmp/wklog.42.new.32171 2010-05-28 06:06:23.000000000 +0000 @@ -1 +1 @@ -Maria-2.0 +9.x -=-=(Knielsen - Fri, 28 May 2010, 06:06)=-=- Status updated. --- /tmp/wklog.42.old.32171 2010-05-28 06:06:23.000000000 +0000 +++ /tmp/wklog.42.new.32171 2010-05-28 06:06:23.000000000 +0000 @@ -1 +1 @@ -Un-Assigned +Assigned -=-=(Guest - Tue, 15 Sep 2009, 02:25)=-=- no Reported zero hours worked. Estimate unchanged. -=-=(Guest - Tue, 15 Sep 2009, 02:24)=-=- Version updated. --- /tmp/wklog.42.old.13241 2009-09-15 02:24:07.000000000 +0300 +++ /tmp/wklog.42.new.13241 2009-09-15 02:24:07.000000000 +0300 @@ -1 +1 @@ -Connector/.NET-5.1 +Maria-2.0 DESCRIPTION: Add the Sphinx storage engine to the MariaDB tree HIGH-LEVEL SPECIFICATION: Code ---- Andrew Aksyonoff from Sphinx is helping to integrate the SphinxSE plugin into the MariaDB tree. It is a plugin, so it can be added to the tree just by including the sub-directory storage/sphinx/. The Sphinx plugin is already of some maturity, having been used with MySQL for some time. Testing ------- To get testing in the mysql-test-run framework, some extensions are needed. To use the Sphinx storage engine, the external Sphinx search daemon needs to be running with some data directory containing indexed data. It also needs to be allocated a port. This is the indended approach: 1. Testing will use an external Sphinx setup installed on the machine. Sphinx binaries will be searched in typical locations (eg. /usr/bin, /usr/local/bin), or can be specified explicitly in the environment with SPHINXSEARCH_INDEXER and SPHINXSEARCH_SEARCHD for the two required binaries. If the external Sphinx binaries can not be found, then Sphinx tests will be disabled (using some --source include/have_sphinx.inc in the test cases). 2. The mysql-test-run framework will install Sphinx search data and start/stop the Sphinx search daemon for the test cases, similarly how it is done for the other servers mysqld, ndbd, etc. We will run the Sphinx search daemon with options --console, --config, and --pidfile. 3. The mysql-test-run framework will generate a Sphinx config file from a template in mysql-test/suite/sphinx/my.cnf. This config file will allocate ports and data directories appropriate for avoiding conflicts between multiple simultaneous mysql-test-run executions. The Sphinx config file is sufficiently similar to MySQL my.cnf that we can use the existing framework for generating config file, with just a slightly modified variant of the code writing the file to disk. 4. The mysql-test-run framework will pre-load the mysql database with tables and data for Sphinx to index. It will then run the `indexer` program to generate the indexes, and then start the `searchd` daemon. These three steps must be done in order, as each step depends on the previous. ALTERNATIVE: it might be possible to pre-generate the necessary data/index files and store them in the source tree. I pushed a proof-of-concept patch for this here: lp:~knielsen/maria/5.2-sphinxse Here is a sample test case using this: --source include/have_sphinx.inc --source include/have_sphinxse.inc --replace_result $SPHINXSEARCH_PORT SPHINXSEARCH_PORT eval create table ts ( id int unsigned not null, w int not null, q varchar(255) not null, index(q) ) engine=sphinx connection="sphinx://127.0.0.1:$SPHINXSEARCH_PORT/*"; select * from ts where q='test'; drop table ts; ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)