- developers - lists.mariadb.org

[Maria-developers] Updated (by Psergey): Subqueries: Inside-out execution for non-semijoin materialized subqueries that are AND-parts of the WHERE (92)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: Inside-out execution for non-semijoin materialized subqueries that are AND-parts of the WHERE CREATION DATE..: Sun, 28 Feb 2010, 14:02 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Psergey, Timour CATEGORY.......: Server-RawIdeaBin TASK ID........: 92 (http://askmonty.org/worklog/?tid=92) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: -=-=(Psergey - Sun, 28 Feb 2010, 14:03)=-=- Observers changed: Psergey,Igor,Timour DESCRIPTION: Consider the following case: SELECT * FROM big_table WHERE oe IN (SELECT ie FROM table_with_few_groups WHERE ... GROUP BY group_col) AND ... Here the best way to execute the query is: Materialize the subquery; # now run the join: for each record R1 in materialized table for each record R2 in big_table such that oe=R1 pass R2 to output Semi-join materialization supports such strategy with SJM-Scan strategy. This WL entry is about adding support for such strategies for non-semijoin subqueries. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Subqueries: Inside-out execution for non-semijoin materialized subqueries that are AND-parts of the WHERE (92)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: Inside-out execution for non-semijoin materialized subqueries that are AND-parts of the WHERE CREATION DATE..: Sun, 28 Feb 2010, 14:02 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 92 (http://askmonty.org/worklog/?tid=92) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Consider the following case: SELECT * FROM big_table WHERE oe IN (SELECT ie FROM table_with_few_groups WHERE ... GROUP BY group_col) AND ... Here the best way to execute the query is: Materialize the subquery; # now run the join: for each record R1 in materialized table for each record R2 in big_table such that oe=R1 pass R2 to output Semi-join materialization supports such strategy with SJM-Scan strategy. This WL entry is about adding support for such strategies for non-semijoin subqueries. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Subqueries: Inside-out execution for non-semijoin materialized subqueries that are AND-parts of the WHERE (92)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: Inside-out execution for non-semijoin materialized subqueries that are AND-parts of the WHERE CREATION DATE..: Sun, 28 Feb 2010, 14:02 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 92 (http://askmonty.org/worklog/?tid=92) VERSION........: WorkLog-3.4 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 0 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: Consider the following case: SELECT * FROM big_table WHERE oe IN (SELECT ie FROM table_with_few_groups WHERE ... GROUP BY group_col) AND ... Here the best way to execute the query is: Materialize the subquery; # now run the join: for each record R1 in materialized table for each record R2 in big_table such that oe=R1 pass R2 to output Semi-join materialization supports such strategy with SJM-Scan strategy. This WL entry is about adding support for such strategies for non-semijoin subqueries. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Subqueries: cost-based choice between Materialization and IN->EXISTS transformation (90)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: cost-based choice between Materialization and IN->EXISTS transformation CREATION DATE..: Sun, 28 Feb 2010, 13:45 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Psergey, Timour CATEGORY.......: Server-RawIdeaBin TASK ID........: 90 (http://askmonty.org/worklog/?tid=90) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: -1 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: For uncorrelated IN subqueries that can't be converted to semi-joins it is necessary to make a cost-based choice between IN->EXISTS and Materialization strategies. Both strategies handle two cases: 1. A simple case w/o NULLs handling 2. Handling NULLs. This WL is about making cost-based decision for #1. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Subqueries: cost-based choice between Materialization and IN->EXISTS transformation (90)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: cost-based choice between Materialization and IN->EXISTS transformation CREATION DATE..: Sun, 28 Feb 2010, 13:45 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Psergey, Timour CATEGORY.......: Server-RawIdeaBin TASK ID........: 90 (http://askmonty.org/worklog/?tid=90) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: -1 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: For uncorrelated IN subqueries that can't be converted to semi-joins it is necessary to make a cost-based choice between IN->EXISTS and Materialization strategies. Both strategies handle two cases: 1. A simple case w/o NULLs handling 2. Handling NULLs. This WL is about making cost-based decision for #1. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Subqueries: cost-based choice between Materialization and IN->EXISTS transformation (90)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: cost-based choice between Materialization and IN->EXISTS transformation CREATION DATE..: Sun, 28 Feb 2010, 13:45 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Psergey, Timour CATEGORY.......: Server-RawIdeaBin TASK ID........: 90 (http://askmonty.org/worklog/?tid=90) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: -1 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: For uncorrelated IN subqueries that can't be converted to semi-joins it is necessary to make a cost-based choice between IN->EXISTS and Materialization strategies. Both strategies handle two cases: 1. A simple case w/o NULLs handling 2. Handling NULLs. This WL is about making cost-based decision for #1. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Psergey): Subqueries: cost-based choice between Materialization and IN->EXISTS transformation (90)
by worklog-noreply＠askmonty.org 28 Feb '10

28 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Subqueries: cost-based choice between Materialization and IN->EXISTS transformation CREATION DATE..: Sun, 28 Feb 2010, 13:45 SUPERVISOR.....: Monty IMPLEMENTOR....: COPIES TO......: Igor, Psergey, Timour CATEGORY.......: Server-RawIdeaBin TASK ID........: 90 (http://askmonty.org/worklog/?tid=90) VERSION........: Server-5.3 STATUS.........: Un-Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: -1 (hours remain) ORIG. ESTIMATE.: 0 PROGRESS NOTES: DESCRIPTION: For uncorrelated IN subqueries that can't be converted to semi-joins it is necessary to make a cost-based choice between IN->EXISTS and Materialization strategies. Both strategies handle two cases: 1. A simple case w/o NULLs handling 2. Handling NULLs. This WL is about making cost-based decision for #1. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] New (by Knielsen): Update packaging scripts for MariaDB 5.2 (88)
by worklog-noreply＠askmonty.org 27 Feb '10

27 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Update packaging scripts for MariaDB 5.2 CREATION DATE..: Sat, 27 Feb 2010, 16:39 SUPERVISOR.....: Knielsen IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-RawIdeaBin TASK ID........: 88 (http://askmonty.org/worklog/?tid=88) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 0 ESTIMATE.......: 30 (hours remain) ORIG. ESTIMATE.: 30 PROGRESS NOTES: DESCRIPTION: The packaging scripts need to be updated to work for MariaDB 5.2 Currently, 5.2 package builds fail in Buildbot. The .debs are missing a debian-5.2 subdirectory. The .rpm also need to be checked. Buildbot needs to be updated to do the new upgrade tests (mariadb-5.1 -> mariadb 5.2) ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): Enable the use of libstdc++ in MariaDB (63)
by worklog-noreply＠askmonty.org 27 Feb '10

27 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Enable the use of libstdc++ in MariaDB CREATION DATE..: Wed, 11 Nov 2009, 13:19 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-BackLog TASK ID........: 63 (http://askmonty.org/worklog/?tid=63) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 4 ESTIMATE.......: 10 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: -=-=(Knielsen - Sat, 27 Feb 2010, 16:02)=-=- Status updated. --- /tmp/wklog.63.old.30948 2010-02-27 16:02:20.000000000 +0000 +++ /tmp/wklog.63.new.30948 2010-02-27 16:02:20.000000000 +0000 @@ -1 +1 @@ -Un-Assigned +Assigned -=-=(Knielsen - Thu, 14 Jan 2010, 13:46)=-=- Research and updated description Worked 4 hours and estimate 10 hours remain (original estimate increased by 14 hours). -=-=(Knielsen - Thu, 14 Jan 2010, 13:45)=-=- High-Level Specification modified. --- /tmp/wklog.63.old.13967 2010-01-14 11:45:49.000000000 +0000 +++ /tmp/wklog.63.new.13967 2010-01-14 11:45:49.000000000 +0000 @@ -1 +1,37 @@ +I did some investigation into this. + +The simple way to do this is to simply use g++ to link C++ objects. So this +issue is really restricted to GCC compilation where we by default prefer to +link with gcc even for C++ code. So this means building with + + CXX=g++ + +The consequences of doing this for the binaries is the addition of two +additional run-time .so dependencies: libstdc++.so and libgcc_s.so. + +It still needs to be investigated if these additional dependencies are a +problem for binary tarball packages, or if the ABI for those libraries are now +as stable as libc.so. + +The libgcc_s.so is needed as a dependency to support exceptions between +different object files, as they need to use the same code for stack unwinding. +libstdc++.so is of course needed for access to C++ runtime. + +I researched into the possibility to instead link only specific plugins with +g++, and continue to link the rest of the server with gcc. Unfortunately, this +seems really hard to do in a proper way due to the way autotools works. At +configure time, a script ./libtool is created with hardcoded compiler commands +derived from $CC and $CXX. This script is then used to do the actual linking +in Makefiles generated by Automake. I thus did not find a way to change the +linker command on a per-makefile basis, as libtool is global to the project. + +One option would be to use separate configure.in for plugins, but this is +quite an intrusive change. + +My conclusion is that the best way is to start using g++ for linking the +entire server. This is no problem for binaries made for a specific +distribution (Ubuntu, Debian, Centos), where the dependencies are handled by +the package manager. If it is a big problem for binary tarball releases, at +worst we can build multiple binary tarball releases for the different library +versions we need to support. -=-=(Guest - Tue, 12 Jan 2010, 16:26)=-=- Version updated. --- /tmp/wklog.63.old.19522 2010-01-12 16:26:23.000000000 +0200 +++ /tmp/wklog.63.new.19522 2010-01-12 16:26:23.000000000 +0200 @@ -1 +1 @@ -Connector/.NET-5.2 +Server-5.2 -=-=(Guest - Tue, 12 Jan 2010, 16:26)=-=- Category updated. --- /tmp/wklog.63.old.19506 2010-01-12 16:26:15.000000000 +0200 +++ /tmp/wklog.63.new.19506 2010-01-12 16:26:15.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-BackLog DESCRIPTION: Enable the use of libstdc++ in MariaDB. As time goes on, more and more plugins and external code/library will need linking to libstdc++ for stuff it uses. I have already seen this happen several times, with extra work needed to integrate things properly. It would be nice to have a general solution for this so that it is not necessary to spend time on individual solutions in each case. It also needs to be considered what the impact of this will be for the server in terms of binary compatibility, performance etc. I think it should be mostly ok, except that it might introduce a problem for bintar packages with an external dependency on libstdc++. HIGH-LEVEL SPECIFICATION: I did some investigation into this. The simple way to do this is to simply use g++ to link C++ objects. So this issue is really restricted to GCC compilation where we by default prefer to link with gcc even for C++ code. So this means building with CXX=g++ The consequences of doing this for the binaries is the addition of two additional run-time .so dependencies: libstdc++.so and libgcc_s.so. It still needs to be investigated if these additional dependencies are a problem for binary tarball packages, or if the ABI for those libraries are now as stable as libc.so. The libgcc_s.so is needed as a dependency to support exceptions between different object files, as they need to use the same code for stack unwinding. libstdc++.so is of course needed for access to C++ runtime. I researched into the possibility to instead link only specific plugins with g++, and continue to link the rest of the server with gcc. Unfortunately, this seems really hard to do in a proper way due to the way autotools works. At configure time, a script ./libtool is created with hardcoded compiler commands derived from $CC and $CXX. This script is then used to do the actual linking in Makefiles generated by Automake. I thus did not find a way to change the linker command on a per-makefile basis, as libtool is global to the project. One option would be to use separate configure.in for plugins, but this is quite an intrusive change. My conclusion is that the best way is to start using g++ for linking the entire server. This is no problem for binaries made for a specific distribution (Ubuntu, Debian, Centos), where the dependencies are handled by the package manager. If it is a big problem for binary tarball releases, at worst we can build multiple binary tarball releases for the different library versions we need to support. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0

[Maria-developers] Updated (by Knielsen): Enable the use of libstdc++ in MariaDB (63)
by worklog-noreply＠askmonty.org 27 Feb '10

27 Feb '10

----------------------------------------------------------------------- WORKLOG TASK -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- TASK...........: Enable the use of libstdc++ in MariaDB CREATION DATE..: Wed, 11 Nov 2009, 13:19 SUPERVISOR.....: Monty IMPLEMENTOR....: Knielsen COPIES TO......: CATEGORY.......: Server-BackLog TASK ID........: 63 (http://askmonty.org/worklog/?tid=63) VERSION........: Server-5.2 STATUS.........: Assigned PRIORITY.......: 60 WORKED HOURS...: 4 ESTIMATE.......: 10 (hours remain) ORIG. ESTIMATE.: 10 PROGRESS NOTES: -=-=(Knielsen - Sat, 27 Feb 2010, 16:02)=-=- Status updated. --- /tmp/wklog.63.old.30948 2010-02-27 16:02:20.000000000 +0000 +++ /tmp/wklog.63.new.30948 2010-02-27 16:02:20.000000000 +0000 @@ -1 +1 @@ -Un-Assigned +Assigned -=-=(Knielsen - Thu, 14 Jan 2010, 13:46)=-=- Research and updated description Worked 4 hours and estimate 10 hours remain (original estimate increased by 14 hours). -=-=(Knielsen - Thu, 14 Jan 2010, 13:45)=-=- High-Level Specification modified. --- /tmp/wklog.63.old.13967 2010-01-14 11:45:49.000000000 +0000 +++ /tmp/wklog.63.new.13967 2010-01-14 11:45:49.000000000 +0000 @@ -1 +1,37 @@ +I did some investigation into this. + +The simple way to do this is to simply use g++ to link C++ objects. So this +issue is really restricted to GCC compilation where we by default prefer to +link with gcc even for C++ code. So this means building with + + CXX=g++ + +The consequences of doing this for the binaries is the addition of two +additional run-time .so dependencies: libstdc++.so and libgcc_s.so. + +It still needs to be investigated if these additional dependencies are a +problem for binary tarball packages, or if the ABI for those libraries are now +as stable as libc.so. + +The libgcc_s.so is needed as a dependency to support exceptions between +different object files, as they need to use the same code for stack unwinding. +libstdc++.so is of course needed for access to C++ runtime. + +I researched into the possibility to instead link only specific plugins with +g++, and continue to link the rest of the server with gcc. Unfortunately, this +seems really hard to do in a proper way due to the way autotools works. At +configure time, a script ./libtool is created with hardcoded compiler commands +derived from $CC and $CXX. This script is then used to do the actual linking +in Makefiles generated by Automake. I thus did not find a way to change the +linker command on a per-makefile basis, as libtool is global to the project. + +One option would be to use separate configure.in for plugins, but this is +quite an intrusive change. + +My conclusion is that the best way is to start using g++ for linking the +entire server. This is no problem for binaries made for a specific +distribution (Ubuntu, Debian, Centos), where the dependencies are handled by +the package manager. If it is a big problem for binary tarball releases, at +worst we can build multiple binary tarball releases for the different library +versions we need to support. -=-=(Guest - Tue, 12 Jan 2010, 16:26)=-=- Version updated. --- /tmp/wklog.63.old.19522 2010-01-12 16:26:23.000000000 +0200 +++ /tmp/wklog.63.new.19522 2010-01-12 16:26:23.000000000 +0200 @@ -1 +1 @@ -Connector/.NET-5.2 +Server-5.2 -=-=(Guest - Tue, 12 Jan 2010, 16:26)=-=- Category updated. --- /tmp/wklog.63.old.19506 2010-01-12 16:26:15.000000000 +0200 +++ /tmp/wklog.63.new.19506 2010-01-12 16:26:15.000000000 +0200 @@ -1 +1 @@ -Server-RawIdeaBin +Server-BackLog DESCRIPTION: Enable the use of libstdc++ in MariaDB. As time goes on, more and more plugins and external code/library will need linking to libstdc++ for stuff it uses. I have already seen this happen several times, with extra work needed to integrate things properly. It would be nice to have a general solution for this so that it is not necessary to spend time on individual solutions in each case. It also needs to be considered what the impact of this will be for the server in terms of binary compatibility, performance etc. I think it should be mostly ok, except that it might introduce a problem for bintar packages with an external dependency on libstdc++. HIGH-LEVEL SPECIFICATION: I did some investigation into this. The simple way to do this is to simply use g++ to link C++ objects. So this issue is really restricted to GCC compilation where we by default prefer to link with gcc even for C++ code. So this means building with CXX=g++ The consequences of doing this for the binaries is the addition of two additional run-time .so dependencies: libstdc++.so and libgcc_s.so. It still needs to be investigated if these additional dependencies are a problem for binary tarball packages, or if the ABI for those libraries are now as stable as libc.so. The libgcc_s.so is needed as a dependency to support exceptions between different object files, as they need to use the same code for stack unwinding. libstdc++.so is of course needed for access to C++ runtime. I researched into the possibility to instead link only specific plugins with g++, and continue to link the rest of the server with gcc. Unfortunately, this seems really hard to do in a proper way due to the way autotools works. At configure time, a script ./libtool is created with hardcoded compiler commands derived from $CC and $CXX. This script is then used to do the actual linking in Makefiles generated by Automake. I thus did not find a way to change the linker command on a per-makefile basis, as libtool is global to the project. One option would be to use separate configure.in for plugins, but this is quite an intrusive change. My conclusion is that the best way is to start using g++ for linking the entire server. This is no problem for binaries made for a specific distribution (Ubuntu, Debian, Centos), where the dependencies are handled by the package manager. If it is a big problem for binary tarball releases, at worst we can build multiple binary tarball releases for the different library versions we need to support. ESTIMATED WORK TIME ESTIMATED COMPLETION DATE ----------------------------------------------------------------------- WorkLog (v3.5.9)

1 0