Re: [Maria-developers] [Merge] lp:~maria-captains/maria/maria-xtradb into lp:maria
Percona <launchpad@percona.com> writes:
Percona has proposed merging lp:~maria-captains/maria/maria-xtradb into lp:maria.
Requested reviews: Maria-captains (maria-captains)
Proposal to merge replacement InnoDB->XtraDB
Thanks a lot for your efforts in this! I branched the tree and took a look. There are a couple of issues that I think need to be resolved before we can merge it into MariaDB. I have some questions below, but please don't hesitate to ask me for any kind of help needed to move this forward.
=== modified file 'storage/innobase/include/sync0rw.h' --- storage/innobase/include/sync0rw.h 2008-02-19 16:44:09 +0000 +++ storage/innobase/include/sync0rw.h 2009-03-31 04:19:17 +0000
+#ifndef INNODB_RW_LOCKS_USE_ATOMICS +#error INNODB_RW_LOCKS_USE_ATOMICS is not defined. Do you use enough new GCC or compatibles? +#error Or do you use exact options for CFLAGS? +#error e.g. (for x86_32): "-m32 -march=i586 -mtune=i686" +#error e.g. (for Sparc_64): "-m64 -mcpu=v9" +#error Otherwise, this build may be slower than normal version. +#endif +
My attempt to build (BUILD/compile-pentium64-max) failed with this error. There were also other build errors (I assume places where the atomics-using code has not been extended with a part that works without the availability of atomics). The reason is that in the MariaDB tree, HAVE_GCC_ATOMIC_BUILTINS is disabled, which caused XtraDB to disable INNODB_RW_LOCKS_USE_ATOMICS, which triggers the above error. I think I understand why this makes sense for Percona, after all using these better synchronisation primitives is part of the reason for using the Percona server in the first place. Can you tell me if Percona has decided not to maintain XtraDB working without the availability of atomic operations? Or if it is just an oversight? I need to discuss with other MariaDB people whether XtraDB for MariaDB should be maintained working without the atomic operations (if so, we should of course be willing to do the work/effort required). So, any thoughts about the best way to deal with this? Should the above #error be removed and XtraDB extended to work without atomics in the MariaDB tree? And is this something Percona wants to do, or should I look into it? Also, Sergei Golubchik told me that HAVE_GCC_ATOMIC_BUILTINS is for my_atomic_ops, and InnoDB/XtraDB shouldn't really be using it. But I need to look more into the code to understand what the problem is, if any.
=== added directory 'storage/innobase/mysql-test'
=== added directory 'storage/innobase/mysql-test/patches'
When I ran the test suite, I got test failures in test main.innodb. I see that the patch contains patches for main MySQL test cases in mysql-test/t/*.test, and also seems to add separate test cases in the storage/innobase/mysql-test/ directory. Do you know what the status is of these test suite modifications? Do the patches need to be applied to the existing test suite, and/or should the extra test cases be used to add to/overwrite the existing tests? We would need to get the test suite to run without problems before merging. Does Percona run the test suite with no failures? Can you suggest which directions I should work in to solve the test failures? Ie. I'm unsure to what extent the extra test cases/patches have already been applied in main MySQL sources, and whether failures are expected or are just due to not being adapted for current MariaDB source changes. Any help with the above would be great. I plan to continue working with you on this so we can get it merged without unnecessary delays. - Kristian.
Kristian, Uh, Thank you for looking into this. You ask good question re: INNODB_RW_LOCKS_USE_ATOMICS, this is actually one of main points of XtraDB to improve scalability on 8CPU+ boxes, so we just rely on it. XtraDB can be built without INNODB_RW_LOCKS_USE_ATOMICS, but effect may be opposite - we can see slowdown instead of improvement. I am not sure what to do with this, I also need to look into InnoDB-plugin 1.0.3, what solution they have and may be use new InnoDB rw_locks instead of ours. Anyway - this is something to discuss. About tests you should execute setup.sh from storage/innobase before running test, as it patches main test suite by InnoDB specific tests. I did not want to put them into main test suite to keep all changes only to storage/innobase repository. In our tests only two tests do not pass after patches - it is main.variables_big and main.read_many_rows_innodb, but they also do not pass in native InnoDB-plugin. Again what to do with tests it is topic to discuss, I do not see obvious way for now... Thanks! Vadim Kristian Nielsen wrote:
Percona <launchpad@percona.com> writes:
Percona has proposed merging lp:~maria-captains/maria/maria-xtradb into lp:maria.
Requested reviews: Maria-captains (maria-captains)
Proposal to merge replacement InnoDB->XtraDB
Thanks a lot for your efforts in this!
I branched the tree and took a look. There are a couple of issues that I think need to be resolved before we can merge it into MariaDB. I have some questions below, but please don't hesitate to ask me for any kind of help needed to move this forward.
=== modified file 'storage/innobase/include/sync0rw.h' --- storage/innobase/include/sync0rw.h 2008-02-19 16:44:09 +0000 +++ storage/innobase/include/sync0rw.h 2009-03-31 04:19:17 +0000
+#ifndef INNODB_RW_LOCKS_USE_ATOMICS +#error INNODB_RW_LOCKS_USE_ATOMICS is not defined. Do you use enough new GCC or compatibles? +#error Or do you use exact options for CFLAGS? +#error e.g. (for x86_32): "-m32 -march=i586 -mtune=i686" +#error e.g. (for Sparc_64): "-m64 -mcpu=v9" +#error Otherwise, this build may be slower than normal version. +#endif +
My attempt to build (BUILD/compile-pentium64-max) failed with this error. There were also other build errors (I assume places where the atomics-using code has not been extended with a part that works without the availability of atomics).
The reason is that in the MariaDB tree, HAVE_GCC_ATOMIC_BUILTINS is disabled, which caused XtraDB to disable INNODB_RW_LOCKS_USE_ATOMICS, which triggers the above error.
I think I understand why this makes sense for Percona, after all using these better synchronisation primitives is part of the reason for using the Percona server in the first place.
Can you tell me if Percona has decided not to maintain XtraDB working without the availability of atomic operations? Or if it is just an oversight?
I need to discuss with other MariaDB people whether XtraDB for MariaDB should be maintained working without the atomic operations (if so, we should of course be willing to do the work/effort required).
So, any thoughts about the best way to deal with this? Should the above #error be removed and XtraDB extended to work without atomics in the MariaDB tree? And is this something Percona wants to do, or should I look into it?
Also, Sergei Golubchik told me that HAVE_GCC_ATOMIC_BUILTINS is for my_atomic_ops, and InnoDB/XtraDB shouldn't really be using it. But I need to look more into the code to understand what the problem is, if any.
=== added directory 'storage/innobase/mysql-test'
=== added directory 'storage/innobase/mysql-test/patches'
When I ran the test suite, I got test failures in test main.innodb.
I see that the patch contains patches for main MySQL test cases in mysql-test/t/*.test, and also seems to add separate test cases in the storage/innobase/mysql-test/ directory.
Do you know what the status is of these test suite modifications? Do the patches need to be applied to the existing test suite, and/or should the extra test cases be used to add to/overwrite the existing tests?
We would need to get the test suite to run without problems before merging. Does Percona run the test suite with no failures? Can you suggest which directions I should work in to solve the test failures? Ie. I'm unsure to what extent the extra test cases/patches have already been applied in main MySQL sources, and whether failures are expected or are just due to not being adapted for current MariaDB source changes.
Any help with the above would be great. I plan to continue working with you on this so we can get it merged without unnecessary delays.
- Kristian.
-- Vadim Tkachenko, CTO Percona Inc. ICQ: 369-510-335, Skype: vadimtk153, Phone +1-888-401-3403 MySQL Performance Blog - http://www.mysqlperformanceblog.com MySQL Consulting http://www.percona.com/ Attend the 2009 Percona Performance Conference April 22-23 - http://conferences.percona.com/
Vadim Tkachenko <vadim@percona.com> writes:
You ask good question re: INNODB_RW_LOCKS_USE_ATOMICS, this is actually one of main points of XtraDB to improve scalability on 8CPU+ boxes, so we just rely on it.
Yes, understood.
XtraDB can be built without INNODB_RW_LOCKS_USE_ATOMICS, but effect may be opposite - we can see slowdown instead of improvement.
Yes. I think there are two issues here. One is that we are replacing InnoDB with XtraDB in MariaDB. Thus we need XtraDB to build and work on all the platforms that MariaDB should work on. The other is to get good performance/scalability of XtraDB. This is very important on the major platforms, but maybe not on all platforms. So it seems that right now XtraDB relies on GCC intrinsics for the performance improvements. I personally think this is fine. We can build with GCC on the main performance platforms I think. The main non-GCC platform currently is Windows, so we need XtraDB to build and work with non-GCC, but I think for now 8CPU+ scalability on Windows is not the most important priority. Later we can then add Windows method for scalability, or maybe switch to also build with GCC on Windows.
I am not sure what to do with this, I also need to look into InnoDB-plugin 1.0.3, what solution they have and may be use new InnoDB rw_locks instead of ours. Anyway - this is something to discuss.
Ok. I will also look closer as time permits, checking the ./configure tests (already discussed this a bit with Sergei Golubchik) and the build failures when INNODB_RW_LOCKS_USE_ATOMICS is not defined.
About tests you should execute setup.sh from storage/innobase before running test, as it patches main test suite by InnoDB specific tests.
I did not want to put them into main test suite to keep all changes only to storage/innobase repository.
Ok. I understand why you keep this separate for XtraDB, thanks for the info. I will try this. It seems to me that once we replace InnoDB with XtraDB in the MariaDB tree, we can commit the changes done by setup.sh, do you agree?
In our tests only two tests do not pass after patches - it is main.variables_big and main.read_many_rows_innodb, but they also do not pass in native InnoDB-plugin.
Ok, I can hopefully take a look later, maybe we need to disable them until Oracle/Innobase team fixes them. - Kristian.
Kristian Nielsen <knielsen@knielsen-hq.org> writes:
I think there are two issues here.
One is that we are replacing InnoDB with XtraDB in MariaDB. Thus we need XtraDB to build and work on all the platforms that MariaDB should work on.
The other is to get good performance/scalability of XtraDB. This is very important on the major platforms, but maybe not on all platforms.
So it seems that right now XtraDB relies on GCC intrinsics for the performance improvements. I personally think this is fine. We can build with GCC on the main performance platforms I think. The main non-GCC platform currently is Windows, so we need XtraDB to build and work with non-GCC, but I think for now 8CPU+ scalability on Windows is not the most important priority.
Later we can then add Windows method for scalability, or maybe switch to also build with GCC on Windows.
I am not sure what to do with this, I also need to look into InnoDB-plugin 1.0.3, what solution they have and may be use new InnoDB rw_locks instead of ours. Anyway - this is something to discuss.
Ok.
I will also look closer as time permits, checking the ./configure tests (already discussed this a bit with Sergei Golubchik) and the build failures when INNODB_RW_LOCKS_USE_ATOMICS is not defined.
Just to avoid duplicate work, let me just mention that I am now starting to look into this issue. I will try to find a good way to make XtraDB compile/work on all the platforms/compilers we want to support, and report back what I discover. - Kristian.
Kristian, Ok, thank you. Please let me know if some code changes required from our side! Kristian Nielsen wrote:
Kristian Nielsen <knielsen@knielsen-hq.org> writes:
I think there are two issues here.
One is that we are replacing InnoDB with XtraDB in MariaDB. Thus we need XtraDB to build and work on all the platforms that MariaDB should work on.
The other is to get good performance/scalability of XtraDB. This is very important on the major platforms, but maybe not on all platforms.
So it seems that right now XtraDB relies on GCC intrinsics for the performance improvements. I personally think this is fine. We can build with GCC on the main performance platforms I think. The main non-GCC platform currently is Windows, so we need XtraDB to build and work with non-GCC, but I think for now 8CPU+ scalability on Windows is not the most important priority.
Later we can then add Windows method for scalability, or maybe switch to also build with GCC on Windows.
I am not sure what to do with this, I also need to look into InnoDB-plugin 1.0.3, what solution they have and may be use new InnoDB rw_locks instead of ours. Anyway - this is something to discuss. Ok.
I will also look closer as time permits, checking the ./configure tests (already discussed this a bit with Sergei Golubchik) and the build failures when INNODB_RW_LOCKS_USE_ATOMICS is not defined.
Just to avoid duplicate work, let me just mention that I am now starting to look into this issue. I will try to find a good way to make XtraDB compile/work on all the platforms/compilers we want to support, and report back what I discover.
- Kristian.
-- Vadim Tkachenko, CTO Percona Inc. ICQ: 369-510-335, Skype: vadimtk153, Phone +1-888-401-3403 MySQL Performance Blog - http://www.mysqlperformanceblog.com MySQL Consulting http://www.percona.com/
Kristian Nielsen <knielsen@knielsen-hq.org> writes:
Just to avoid duplicate work, let me just mention that I am now starting to look into this issue. I will try to find a good way to make XtraDB compile/work on all the platforms/compilers we want to support, and report back what I discover.
I got some good progress on this. I re-did the merge using the merge-into bzr plugin (advise from bzr developers). This should allow us to merge directly from the lp:percona-xtradb tree into lp:maria in the future. I placed the files in storage/xtradb/ and left storage/innodb/ in place. This should hopefully make merging from MySQL slightly easier, and seems to work well (and it is kind of nice also). I integrated the XtraDB testsuite into the tree so there is no need to run setup.sh or anything. I also fixed a number of test suite failures, and even a small bug or two. I fixed the code to compile without GCC atomic operation intrinsics. Performance will be worse of course, but it is good to have it at least working. I still need to fix Maria configure.in to not disable GCC atomics for xtradb by default. I pushed the result so far to Launchpad: lp:~maria-captains/maria/mariadb-xtradb-merge2 In particular you might be interested in the changeset 2702: http://bazaar.launchpad.net/~maria-captains/maria/mariadb-xtradb-merge2/revi... Maybe you will want to merge some of that back into XtraDB? Comments are welcome, of course. I think that after fixing GCC atomics in configure this should be ready to push into MariaDB 5.1. Would be good to get a review of especially the 2702 commit. - Kristian.
Kristian, Thank you for looking into this! I will look how we can backport your changes. We need to have xtradb compiled against standard 5.1, if it is not broken - it should be not problem to include your fixes. Thanks! Vadim Kristian Nielsen wrote:
Kristian Nielsen <knielsen@knielsen-hq.org> writes:
Just to avoid duplicate work, let me just mention that I am now starting to look into this issue. I will try to find a good way to make XtraDB compile/work on all the platforms/compilers we want to support, and report back what I discover.
I got some good progress on this. I re-did the merge using the merge-into bzr plugin (advise from bzr developers). This should allow us to merge directly from the lp:percona-xtradb tree into lp:maria in the future.
I placed the files in storage/xtradb/ and left storage/innodb/ in place. This should hopefully make merging from MySQL slightly easier, and seems to work well (and it is kind of nice also).
I integrated the XtraDB testsuite into the tree so there is no need to run setup.sh or anything. I also fixed a number of test suite failures, and even a small bug or two.
I fixed the code to compile without GCC atomic operation intrinsics. Performance will be worse of course, but it is good to have it at least working. I still need to fix Maria configure.in to not disable GCC atomics for xtradb by default.
I pushed the result so far to Launchpad:
lp:~maria-captains/maria/mariadb-xtradb-merge2
In particular you might be interested in the changeset 2702:
http://bazaar.launchpad.net/~maria-captains/maria/mariadb-xtradb-merge2/revi...
Maybe you will want to merge some of that back into XtraDB?
Comments are welcome, of course. I think that after fixing GCC atomics in configure this should be ready to push into MariaDB 5.1. Would be good to get a review of especially the 2702 commit.
- Kristian.
-- Vadim Tkachenko, CTO Percona Inc. ICQ: 369-510-335, Skype: vadimtk153, Phone +1-888-401-3403 MySQL Performance Blog - http://www.mysqlperformanceblog.com MySQL Consulting http://www.percona.com/
Regarding the XtraDB merge into MariaDB, I am now finally getting ready to push the merge, after lots of small fixes in different areas. The code is available in the mariadb-xtradb-merge2 branch: https://code.launchpad.net/~maria-captains/maria/mariadb-xtradb-merge2 The buildbot testing is here: https://askmonty.org/buildbot/waterfall?branch=mariadb-xtradb-merge2 Vadim, there is one remaining issue that I need Percona to look into. This is a hang in the innodb_xtradb_bug317074 test case on host hardy-amd64-dbg. Test output is included at the end of this mail, and full log is here: https://askmonty.org/buildbot/builders/hardy-amd64-dbg/builds/69/steps/test/... Please take a look and see what it could be. The failure seems to be repeatable on that host. If needed, we can probably talk to archivist (who runs that build slave) on getting access to debug etc. I would also like your opinion on whether this is something that should block pushing the merge into MariaDB, as it is the last remaining known issue. Also, I was wondering if there is some way that Percona and MariaDB engineers could be on the same IRC channel(s)? That might be useful for some of the discussions. MariaDB engineers already hang out in #maria on FreeNode. Vadim Tkachenko <vadim@percona.com> writes:
I will look how we can backport your changes. We need to have xtradb compiled against standard 5.1, if it is not broken - it should be not problem to include your fixes.
I am pretty sure there are no changes that would not work with 5.1. There are of course a number of changes that should _not_ go into the Percona XtraDB repository, like the replacing of innodb with xtradb etc. Here is an annotated list of the changes I think you should consider including upstream in XtraDB. If you choose to include any of them, it would be a help if you can do them in one (or a few) separate commits without any other changes. This will allow a simple null-merge to deal with merge conflits in any following xtradb->mariadb merges. Revision: 2716 http://bazaar.launchpad.net/~maria-captains/maria/mariadb-xtradb-merge2/revi... The fix in mysql-test/t/innodb_information_schema.test would be relevant to merge into XtraDB. Revision: 2714 http://bazaar.launchpad.net/~maria-captains/maria/mariadb-xtradb-merge2/revi... Here the test case fixes may be relevant. And there is a bug fix for ALTER TABLE in ha_innodb.cc ha_innobase::check_if_incompatible_data() that is needed to work with latest MySQL 5.1 sources (should also be ok with older 5.1 versions). There are also a few minor merges of changes from MySQL 5.1 to innodb not in the innodb plugin, which may or may not make sense for XtraDB to merge (probably doesn't matter). Revision: 2697.3.4 http://bazaar.launchpad.net/~maria-captains/maria/mariadb-xtradb-merge2/revi... This is fixes for test suite failures. I think most are not relevant, however changes to these files fixes a test failure: mysql-test/t/innodb-zip.test mysql-test/r/innodb-zip.result As far as I can tell, remaining changes are either outside the innobase storage engine code (so cannot be meaningfully merged into XtraDB), or fixes mainly relevant for the merge into MariaDB. - Kristian. ----------------------------------------------------------------------- main.innodb_xtradb_bug317074 [ fail ] timeout after 900 seconds Test ended at 2009-06-23 15:20:52 Test case timeout after 900 seconds == /home/archivist/archivist-amd64/archivist-amd64/build/mysql-test/var/1/log/innodb_xtradb_bug317074.log == SET @save_innodb_file_format=@@global.innodb_file_format; SET @save_innodb_file_format_check=@@global.innodb_file_format_check; SET @save_innodb_file_per_table=@@global.innodb_file_per_table; SET GLOBAL innodb_file_format='Barracuda'; SET GLOBAL innodb_file_per_table=ON; == /home/archivist/archivist-amd64/archivist-amd64/build/mysql-test/var/1/tmp/analyze-timeout-mysqld.1.err == mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local MySQL server through socket '/home/archivist/archivist-amd64/archivist-amd64/build/mysql-test/var/tmp/1/mysqld.1.sock' (111) - saving '/home/archivist/archivist-amd64/archivist-amd64/build/mysql-test/var/1/log/main.innodb_xtradb_bug317074/' to '/home/archivist/archivist-amd64/archivist-amd64/build/mysql-test/var/log/main.innodb_xtradb_bug317074/' - found 'core.21388' (0/5) Trying 'dbx' to get a backtrace Trying 'gdb' to get a backtrace Core generated by '/home/archivist/archivist-amd64/archivist-amd64/build/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- warning: Can't read pathname for load map: Input/output error. Core was generated by `/home/archivist/archivist-amd64/archivist-amd64/build/sql/mysqld --defaults-gro'. Program terminated with signal 6, Aborted. [New process 21388] [New process 24743] [New process 21619] [New process 21618] [New process 21600] [New process 21598] [New process 21535] [New process 21534] [New process 21533] [New process 21532] [New process 21520] [New process 21519] [New process 21518] [New process 21500] [New process 21483] [New process 21466] #0 0x00007fb3097939c2 in pthread_kill () from /lib/libpthread.so.0 #0 0x00007fb3097939c2 in pthread_kill () from /lib/libpthread.so.0 #1 0x0000000000b21d9f in my_write_core (sig=6) at stacktrace.c:310 #2 0x0000000000668653 in handle_segfault (sig=6) at mysqld.cc:2597 #3 <signal handler called> #4 0x00007fb308803db2 in select () from /lib/libc.so.6 #5 0x0000000000668f81 in handle_connections_sockets (arg=0x0) at mysqld.cc:5080 #6 0x000000000066c932 in main (argc=8, argv=0x7fff11dd9428) at mysqld.cc:4566 Thread 16 (process 21466): #0 0x00007fb309792e1d in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x0000000000b27e74 in safe_cond_timedwait (cond=0x198a0a0, mp=0x198a000, abstime=0x414ef0c0, file=0xcdbad6 "ma_checkpoint.c", line=715) at thr_mutex.c:549 #2 0x00000000009a1c9f in ma_checkpoint_background (arg=0x1e) at ma_checkpoint.c:715 #3 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #4 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #5 0x0000000000000000 in ?? () Thread 15 (process 21483): #0 0x00007fb309792e1d in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00000000009f2d3b in xt_timed_wait_cond (self=0x0, cond=0x198a6f8, mutex=0x198a6d0, milli_sec=500) at thread_xt.cc:1945 #2 0x00000000009ded3d in tabc_fr_wait_for_cache (self=0x1aaf930, msecs=500) at tabcache_xt.cc:818 #3 0x00000000009df89b in tabc_fr_main (self=0x1aaf930) at tabcache_xt.cc:1114 #4 0x00000000009dfa15 in tabc_fr_run_thread (self=0x1aaf930) at tabcache_xt.cc:1131 #5 0x00000000009f5436 in thr_main (data=0x7fff11dd8bc0) at thread_xt.cc:1005 #6 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #7 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () Thread 14 (process 21500): #0 0x00007fb309792e1d in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00000000009f2d3b in xt_timed_wait_cond (self=0x1f0d080, cond=0x1b22f38, mutex=0x1b22f10, milli_sec=100) at thread_xt.cc:1945 #2 0x00000000009fca42 in xn_sw_wait_for_xact (self=0x1f0d080, db=0x1ad24b0, hsecs=10) at xaction_xt.cc:2578 #3 0x00000000009ff328 in xn_sw_main (self=0x1f0d080) at xaction_xt.cc:2481 #4 0x00000000009ff4e1 in xn_sw_run_thread (self=0x1f0d080) at xaction_xt.cc:2525 #5 0x00000000009f5436 in thr_main (data=0x7fff11dd6a10) at thread_xt.cc:1005 #6 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #7 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () Thread 13 (process 21518): #0 0x00007fb309792e1d in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00000000009f2d3b in xt_timed_wait_cond (self=0x1f64820, cond=0x1b23040, mutex=0x1b23018, milli_sec=120000) at thread_xt.cc:1945 #2 0x0000000000a1b003 in dl_co_wait (self=0x1f64820, db=0x1ad24b0, secs=120) at datalog_xt.cc:1620 #3 0x0000000000a1f3a7 in dl_co_main (self=0x1f64820, once_off=0) at datalog_xt.cc:1919 #4 0x0000000000a1f4fa in dl_run_co_thread (self=0x1f64820) at datalog_xt.cc:1956 #5 0x00000000009f5436 in thr_main (data=0x7fff11dd6a10) at thread_xt.cc:1005 #6 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #7 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () Thread 12 (process 21519): #0 0x00007fb309792e1d in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00000000009f2d3b in xt_timed_wait_cond (self=0x1f7bfb0, cond=0x1b22fc0, mutex=0x1b22f98, milli_sec=500) at thread_xt.cc:1945 #2 0x0000000000a05683 in xlog_wr_wait_for_log_flush (self=0x1f7bfb0, db=0x1ad24b0) at xactlog_xt.cc:2260 #3 0x0000000000a05dca in xlog_wr_main (self=0x1f7bfb0) at xactlog_xt.cc:2451 #4 0x0000000000a05f59 in xlog_wr_run_thread (self=0x1f7bfb0) at xactlog_xt.cc:2476 #5 0x00000000009f5436 in thr_main (data=0x7fff11dd6a10) at thread_xt.cc:1005 #6 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #7 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () Thread 11 (process 21520): #0 0x00007fb309792e1d in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x00000000009f2d3b in xt_timed_wait_cond (self=0x1fd3750, cond=0x1b36d28, mutex=0x1b36d00, milli_sec=400) at thread_xt.cc:1945 #2 0x00000000009d236b in xres_cp_wait_for_log_writer (self=0x1fd3750, db=0x1ad24b0, milli_secs=400) at restart_xt.cc:2491 #3 0x00000000009d3e6e in xres_cp_main (self=0x1fd3750) at restart_xt.cc:2567 #4 0x00000000009d3ffb in xres_cp_run_thread (self=0x1fd3750) at restart_xt.cc:2594 #5 0x00000000009f5436 in thr_main (data=0x7fff11dd6a10) at thread_xt.cc:1005 #6 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #7 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () Thread 10 (process 21532): #0 0x00007fb309792b99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x0000000000b27ba1 in safe_cond_wait (cond=0x29fb1d0, mp=0x29fb120, file=0xcfec48 "os/os0sync.c", line=438) at thr_mutex.c:495 #2 0x0000000000a8fa2f in os_event_wait_low (event=0x29fb120, reset_sig_count=0) at os/os0sync.c:438 #3 0x0000000000a8e6ca in os_aio_simulated_handle (global_segment=0, message1=0x444f50f8, message2=0x444f50f0, type=0x444f50e8) at os/os0file.c:3971 #4 0x0000000000a6ab2c in fil_aio_wait (segment=0) at fil/fil0fil.c:4472 #5 0x0000000000ac61ee in io_handler_thread (arg=0x198d720) at srv/srv0start.c:464 #6 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #7 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () Thread 9 (process 21533): #0 0x00007fb309792b99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x0000000000b27ba1 in safe_cond_wait (cond=0x29fb2d0, mp=0x29fb220, file=0xcfec48 "os/os0sync.c", line=438) at thr_mutex.c:495 #2 0x0000000000a8fa2f in os_event_wait_low (event=0x29fb220, reset_sig_count=0) at os/os0sync.c:438 #3 0x0000000000a8e6ca in os_aio_simulated_handle (global_segment=1, message1=0x44cf60f8, message2=0x44cf60f0, type=0x44cf60e8) at os/os0file.c:3971 #4 0x0000000000a6ab2c in fil_aio_wait (segment=1) at fil/fil0fil.c:4472 #5 0x0000000000ac61ee in io_handler_thread (arg=0x198d728) at srv/srv0start.c:464 #6 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #7 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () Thread 8 (process 21534): #0 0x00007fb309792b99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x0000000000b27ba1 in safe_cond_wait (cond=0x29fb3d0, mp=0x29fb320, file=0xcfec48 "os/os0sync.c", line=438) at thr_mutex.c:495 #2 0x0000000000a8fa2f in os_event_wait_low (event=0x29fb320, reset_sig_count=0) at os/os0sync.c:438 #3 0x0000000000a8e6ca in os_aio_simulated_handle (global_segment=2, message1=0x454f70f8, message2=0x454f70f0, type=0x454f70e8) at os/os0file.c:3971 #4 0x0000000000a6ab2c in fil_aio_wait (segment=2) at fil/fil0fil.c:4472 #5 0x0000000000ac61ee in io_handler_thread (arg=0x198d730) at srv/srv0start.c:464 #6 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #7 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () Thread 7 (process 21535): #0 0x00007fb309792b99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #1 0x0000000000b27ba1 in safe_cond_wait (cond=0x29fb4d0, mp=0x29fb420, file=0xcfec48 "os/os0sync.c", line=438) at thr_mutex.c:495 #2 0x0000000000a8fa2f in os_event_wait_low (event=0x29fb420, reset_sig_count=0) at os/os0sync.c:438 #3 0x0000000000a8e6ca in os_aio_simulated_handle (global_segment=3, message1=0x45cf80f8, message2=0x45cf80f0, type=0x45cf80e8) at os/os0file.c:3971 #4 0x0000000000a6ab2c in fil_aio_wait (segment=3) at fil/fil0fil.c:4472 #5 0x0000000000ac61ee in io_handler_thread (arg=0x198d738) at srv/srv0start.c:464 #6 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #7 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #8 0x0000000000000000 in ?? () Thread 6 (process 21598): #0 0x00007fb308803db2 in select () from /lib/libc.so.6 #1 0x0000000000a9019e in os_thread_sleep (tm=1000000) at os/os0thread.c:288 #2 0x0000000000ac4071 in srv_lock_timeout_and_monitor_thread (arg=0x0) at srv/srv0srv.c:2035 #3 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #4 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #5 0x0000000000000000 in ?? () Thread 5 (process 21600): #0 0x00007fb308803db2 in select () from /lib/libc.so.6 #1 0x0000000000a9019e in os_thread_sleep (tm=1000000) at os/os0thread.c:288 #2 0x0000000000ac4583 in srv_error_monitor_thread (arg=0x0) at srv/srv0srv.c:2268 #3 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #4 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #5 0x0000000000000000 in ?? () Thread 4 (process 21618): #0 0x00007fb308803db2 in select () from /lib/libc.so.6 #1 0x0000000000a9019e in os_thread_sleep (tm=1000000) at os/os0thread.c:288 #2 0x0000000000ac47a8 in srv_master_thread (arg=0x0) at srv/srv0srv.c:2401 #3 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #4 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #5 0x0000000000000000 in ?? () Thread 3 (process 21619): #0 0x00007fb3097964f9 in do_sigwait () from /lib/libpthread.so.0 #1 0x00007fb30979659d in sigwait () from /lib/libpthread.so.0 #2 0x00000000006679c5 in signal_hand (arg=0x0) at mysqld.cc:2801 #3 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #4 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #5 0x0000000000000000 in ?? () Thread 2 (process 24743): #0 0x00007fb309795d07 in fsync () from /lib/libpthread.so.0 #1 0x0000000000a8c6e3 in os_file_fsync (file=12) at os/os0file.c:1843 #2 0x0000000000a8c7a6 in os_file_flush (file=12) at os/os0file.c:1940 #3 0x0000000000a6ad9e in fil_flush (space_id=4294967280) at fil/fil0fil.c:4583 #4 0x0000000000a80eac in log_write_up_to (lsn=22725942, wait=92, flush_to_disk=1) at log/log0log.c:1468 #5 0x0000000000ad7ce1 in trx_commit_off_kernel (trx=0x2d52150) at trx/trx0trx.c:935 #6 0x0000000000ad8de5 in trx_commit_for_mysql (trx=0x2d52150) at trx/trx0trx.c:1582 #7 0x0000000000a2b9b3 in innobase_commit_low (trx=0x2d52150) at handler/ha_innodb.cc:2463 #8 0x0000000000a2f627 in innobase_commit (hton=0x1ac7168, thd=0x2f60ff8, all=false) at handler/ha_innodb.cc:2592 #9 0x00000000007ae3c7 in ha_commit_one_phase (thd=0x2f60ff8, all=false) at handler.cc:1214 #10 0x00000000007ae8d1 in ha_commit_trans (thd=0x2f60ff8, all=false) at handler.cc:1183 #11 0x00000000007aea99 in ha_autocommit_or_rollback (thd=0x2f60ff8, error=0) at handler.cc:1345 #12 0x00000000006cbb28 in close_thread_tables (thd=0x2f60ff8) at sql_base.cc:1284 #13 0x000000000083bb39 in sp_lex_keeper::reset_lex_and_exec_core ( this=0x3113838, thd=0x2f60ff8, nextp=0x40c1ef88, open_tables=false, instr=0x31137f8) at sp_head.cc:2740 #14 0x0000000000841bad in sp_instr_stmt::execute (this=0x31137f8, thd=0x2f60ff8, nextp=0x40c1ef88) at sp_head.cc:2846 #15 0x000000000083dcd0 in sp_head::execute (this=0x2c6f780, thd=0x2f60ff8) at sp_head.cc:1252 #16 0x000000000083ea74 in sp_head::execute_procedure (this=0x2c6f780, thd=0x2f60ff8, args=0x2f63550) at sp_head.cc:1981 #17 0x00000000006818e2 in mysql_execute_command (thd=0x2f60ff8) at sql_parse.cc:4345 #18 0x0000000000683a99 in mysql_parse (thd=0x2f60ff8, inBuf=0x2db5590 "call insert_many(100000)", length=24, found_semicolon=0x40c20c30) at sql_parse.cc:5924 #19 0x000000000068477b in dispatch_command (command=COM_QUERY, thd=0x2f60ff8, packet=0x2f58ba9 "call insert_many(100000)", packet_length=24) at sql_parse.cc:1224 #20 0x0000000000685bd0 in do_command (thd=0x2f60ff8) at sql_parse.cc:865 #21 0x0000000000671c68 in handle_one_connection (arg=0x2f60ff8) at sql_connect.cc:1118 #22 0x00007fb30978e3f7 in start_thread () from /lib/libpthread.so.0 #23 0x00007fb30880ab3d in clone () from /lib/libc.so.6 #24 0x0000000000000000 in ?? () Thread 1 (process 21388): #0 0x00007fb3097939c2 in pthread_kill () from /lib/libpthread.so.0 #1 0x0000000000b21d9f in my_write_core (sig=6) at stacktrace.c:310 #2 0x0000000000668653 in handle_segfault (sig=6) at mysqld.cc:2597 #3 <signal handler called> #4 0x00007fb308803db2 in select () from /lib/libc.so.6 #5 0x0000000000668f81 in handle_connections_sockets (arg=0x0) at mysqld.cc:5080 #6 0x000000000066c932 in main (argc=8, argv=0x7fff11dd9428) at mysqld.cc:4566
Kristian, We are about to finalize xtradb-release6, I think it makes sense to push this new release. I am trying to incorporate your changes, but bzr branch lp:~maria-captains/maria/mariadb-xtradb-merge2 takes ages and it failed twice for me due "broken connection" error, so I had no success here yet. Kristian Nielsen wrote:
Regarding the XtraDB merge into MariaDB, I am now finally getting ready to push the merge, after lots of small fixes in different areas.
The code is available in the mariadb-xtradb-merge2 branch:
https://code.launchpad.net/~maria-captains/maria/mariadb-xtradb-merge2
The buildbot testing is here:
https://askmonty.org/buildbot/waterfall?branch=mariadb-xtradb-merge2
Vadim, there is one remaining issue that I need Percona to look into. This is a hang in the innodb_xtradb_bug317074 test case on host hardy-amd64-dbg. Test output is included at the end of this mail, and full log is here:
https://askmonty.org/buildbot/builders/hardy-amd64-dbg/builds/69/steps/test/...
Please take a look and see what it could be. The failure seems to be repeatable on that host. If needed, we can probably talk to archivist (who runs that build slave) on getting access to debug etc.
I would also like your opinion on whether this is something that should block pushing the merge into MariaDB, as it is the last remaining known issue.
Also, I was wondering if there is some way that Percona and MariaDB engineers could be on the same IRC channel(s)? That might be useful for some of the discussions. MariaDB engineers already hang out in #maria on FreeNode.
Vadim Tkachenko <vadim@percona.com> writes:
I will look how we can backport your changes. We need to have xtradb compiled against standard 5.1, if it is not broken - it should be not problem to include your fixes.
I am pretty sure there are no changes that would not work with 5.1. There are of course a number of changes that should _not_ go into the Percona XtraDB repository, like the replacing of innodb with xtradb etc.
Here is an annotated list of the changes I think you should consider including upstream in XtraDB. If you choose to include any of them, it would be a help if you can do them in one (or a few) separate commits without any other changes. This will allow a simple null-merge to deal with merge conflits in any following xtradb->mariadb merges.
Revision: 2716 http://bazaar.launchpad.net/~maria-captains/maria/mariadb-xtradb-merge2/revi...
The fix in mysql-test/t/innodb_information_schema.test would be relevant to merge into XtraDB.
Revision: 2714 http://bazaar.launchpad.net/~maria-captains/maria/mariadb-xtradb-merge2/revi...
Here the test case fixes may be relevant. And there is a bug fix for ALTER TABLE in ha_innodb.cc ha_innobase::check_if_incompatible_data() that is needed to work with latest MySQL 5.1 sources (should also be ok with older 5.1 versions).
There are also a few minor merges of changes from MySQL 5.1 to innodb not in the innodb plugin, which may or may not make sense for XtraDB to merge (probably doesn't matter).
Revision: 2697.3.4 http://bazaar.launchpad.net/~maria-captains/maria/mariadb-xtradb-merge2/revi...
This is fixes for test suite failures. I think most are not relevant, however changes to these files fixes a test failure:
mysql-test/t/innodb-zip.test mysql-test/r/innodb-zip.result
As far as I can tell, remaining changes are either outside the innobase storage engine code (so cannot be meaningfully merged into XtraDB), or fixes mainly relevant for the merge into MariaDB.
- Kristian.
----------------------------------------------------------------------- main.innodb_xtradb_bug317074 [ fail ] timeout after 900 seconds Test ended at 2009-06-23 15:20:52
Test case timeout after 900 seconds
== /home/archivist/archivist-amd64/archivist-amd64/build/mysql-test/var/1/log/innodb_xtradb_bug317074.log == SET @save_innodb_file_format=@@global.innodb_file_format; SET @save_innodb_file_format_check=@@global.innodb_file_format_check; SET @save_innodb_file_per_table=@@global.innodb_file_per_table; SET GLOBAL innodb_file_format='Barracuda'; SET GLOBAL innodb_file_per_table=ON;
-- Vadim Tkachenko, CTO Percona Inc. ICQ: 369-510-335, Skype: vadimtk153, Phone +1-888-401-3403 MySQL Performance Blog - http://www.mysqlperformanceblog.com MySQL Consulting http://www.percona.com/
Vadim Tkachenko <vadim@percona.com> writes:
We are about to finalize xtradb-release6, I think it makes sense to push this new release.
Yes, that sounds reasonable. Unless there are substantial changes compared to latest bzr a few days ago, it should be simple to merge it.
I am trying to incorporate your changes, but bzr branch lp:~maria-captains/maria/mariadb-xtradb-merge2 takes ages and it failed twice for me due "broken connection" error, so I had no success here yet.
It will greatly help if you run the branch command in a shared repository that already has a branch of mariadb (or failing that, mysql). This will make it only need to download the changes, not the whole history. (A shared repository is created with `bzr init-repo` if you didn't know already; it makes working with Launchpad much less painful.) But Launchpad seems to have been somewhat unstable these last few days, so that may also be the cause of the trouble. - Kristian.
Kristian, I made push to lp:~maria-captains/maria/mariadb-xtradb6-merge (not proposed for merge yet). If everything is fine we can merge it to maria. Thanks, Vadim Kristian Nielsen wrote:
Vadim Tkachenko <vadim@percona.com> writes:
We are about to finalize xtradb-release6, I think it makes sense to push this new release.
Yes, that sounds reasonable. Unless there are substantial changes compared to latest bzr a few days ago, it should be simple to merge it.
I am trying to incorporate your changes, but bzr branch lp:~maria-captains/maria/mariadb-xtradb-merge2 takes ages and it failed twice for me due "broken connection" error, so I had no success here yet.
It will greatly help if you run the branch command in a shared repository that already has a branch of mariadb (or failing that, mysql). This will make it only need to download the changes, not the whole history.
(A shared repository is created with `bzr init-repo` if you didn't know already; it makes working with Launchpad much less painful.)
But Launchpad seems to have been somewhat unstable these last few days, so that may also be the cause of the trouble.
- Kristian.
-- Vadim Tkachenko, CTO Percona Inc. ICQ: 369-510-335, Skype: vadimtk153, Phone +1-888-401-3403 MySQL Performance Blog - http://www.mysqlperformanceblog.com MySQL Consulting http://www.percona.com/
Vadim Tkachenko <vadim@percona.com> writes:
I made push to
lp:~maria-captains/maria/mariadb-xtradb6-merge
(not proposed for merge yet).
If everything is fine we can merge it to maria.
I am now looking into this. I have a hard time understanding this merge. It seems you just applied manually a patch against latest lp:maria, omitting most (but not all?) of the necessary fixes I already made in the tree lp:~maria-captains/maria/mariadb-xtradb-merge2 I think we need to work instead from my merged tree, as that includes the full bzr history of both lp:maria and lp:percona-xtradb. This is necessary to allow smooth merging of future changes in XtraDB (and MariaDB). Basically, the merge should be a simple `bzr merge` from the xtradb6 branch into my mariadb-xtradb-merge2 branch, and then a merge of that result into lp:maria. However, I do not know which branch contains the xtradb6 code (is it pushed to Launchpad yet?). I assumed 'lp:percona-xtradb/release-6', but that code seems to be missing the changes listed below compared to mariadb-xtradb-merge. Can you tell me which branch to merge to get the correct changes? - Kristian. ----------------------------------------------------------------------- diff -u --recursive ./buf/buf0rea.c ../mariadb-xtradb6-merge/storage/xtradb/buf/buf0rea.c --- ./buf/buf0rea.c 2009-07-07 14:08:24.000000000 +0200 +++ ../mariadb-xtradb6-merge/storage/xtradb/buf/buf0rea.c 2009-07-07 14:02:21.000000000 +0200 @@ -134,6 +134,46 @@ bpage = buf_page_init_for_read(err, mode, space, zip_size, unzip, tablespace_version, offset); if (bpage == NULL) { + /* bugfix: http://bugs.mysql.com/bug.php?id=43948 */ + if (recv_recovery_is_on() && *err == DB_TABLESPACE_DELETED) { + /* hashed log recs must be treated here */ + recv_addr_t* recv_addr; + + mutex_enter(&(recv_sys->mutex)); + + if (recv_sys->apply_log_recs == FALSE) { + mutex_exit(&(recv_sys->mutex)); + goto not_to_recover; + } + + /* recv_get_fil_addr_struct() */ + recv_addr = HASH_GET_FIRST(recv_sys->addr_hash, + hash_calc_hash(ut_fold_ulint_pair(space, offset), + recv_sys->addr_hash)); + while (recv_addr) { + if ((recv_addr->space == space) + && (recv_addr->page_no == offset)) { + break; + } + recv_addr = HASH_GET_NEXT(addr_hash, recv_addr); + } + + if ((recv_addr == NULL) + || (recv_addr->state == RECV_BEING_PROCESSED) + || (recv_addr->state == RECV_PROCESSED)) { + mutex_exit(&(recv_sys->mutex)); + goto not_to_recover; + } + + fprintf(stderr, " (cannot find space: %lu)", space); + recv_addr->state = RECV_PROCESSED; + + ut_a(recv_sys->n_addrs); + recv_sys->n_addrs--; + + mutex_exit(&(recv_sys->mutex)); + } +not_to_recover: return(0); } @@ -784,11 +824,11 @@ while (buf_pool->n_pend_reads >= recv_n_pool_free_frames / 2) { os_aio_simulated_wake_handler_threads(); - os_thread_sleep(500000); + os_thread_sleep(10000); count++; - if (count > 100) { + if (count > 5000) { fprintf(stderr, "InnoDB: Error: InnoDB has waited for" " 50 seconds for pending\n" diff -u --recursive ./handler/innodb_patch_info.h ../mariadb-xtradb6-merge/storage/xtradb/handler/innodb_patch_info.h --- ./handler/innodb_patch_info.h 2009-07-07 14:08:24.000000000 +0200 +++ ../mariadb-xtradb6-merge/storage/xtradb/handler/innodb_patch_info.h 2009-07-07 14:02:21.000000000 +0200 @@ -37,5 +37,6 @@ {"innodb_dict_size_limit","Limit dictionary cache size","Variable innodb_dict_size_limit in bytes","http://www.percona.com/docs/wiki/percona-xtradb"}, {"innodb_split_buf_pool_mutex","More fix of buffer_pool mutex","Spliting buf_pool_mutex and optimizing based on innodb_opt_lru_count","http://www.percona.com/docs/wiki/percona-xtradb"}, {"innodb_stats","Additional features about InnoDB statistics/optimizer","","http://www.percona.com/docs/wiki/percona-xtradb"}, +{"innodb_recovery_patches","Bugfixes and adjustments about recovery process","","http://www.percona.com/docs/wiki/percona-xtradb"}, {NULL, NULL, NULL, NULL} }; diff -u --recursive ./include/univ.i ../mariadb-xtradb6-merge/storage/xtradb/include/univ.i --- ./include/univ.i 2009-07-07 14:08:24.000000000 +0200 +++ ../mariadb-xtradb6-merge/storage/xtradb/include/univ.i 2009-07-07 14:02:43.000000000 +0200 @@ -35,7 +35,7 @@ #define INNODB_VERSION_MAJOR 1 #define INNODB_VERSION_MINOR 0 #define INNODB_VERSION_BUGFIX 3 -#define PERCONA_INNODB_VERSION 5a +#define PERCONA_INNODB_VERSION 6 /* The following is the InnoDB version as shown in SELECT plugin_version FROM information_schema.plugins; diff -u --recursive ./log/log0recv.c ../mariadb-xtradb6-merge/storage/xtradb/log/log0recv.c --- ./log/log0recv.c 2009-07-07 14:08:24.000000000 +0200 +++ ../mariadb-xtradb6-merge/storage/xtradb/log/log0recv.c 2009-07-07 14:02:21.000000000 +0200 @@ -110,7 +110,7 @@ use these free frames to read in pages when we start applying the log records to the database. */ -UNIV_INTERN ulint recv_n_pool_free_frames = 256; +UNIV_INTERN ulint recv_n_pool_free_frames = 1024; /* The maximum lsn we see for a page during the recovery process. If this is bigger than the lsn we are able to scan up to, that is an indication that @@ -1225,6 +1225,8 @@ buf_block_get_page_no(block)); if ((recv_addr == NULL) + /* bugfix: http://bugs.mysql.com/bug.php?id=44140 */ + || (recv_addr->state == RECV_BEING_READ && !just_read_in) || (recv_addr->state == RECV_BEING_PROCESSED) || (recv_addr->state == RECV_PROCESSED)) { diff -u --recursive ./mysql-test/patches/events_stress.diff ../mariadb-xtradb6-merge/storage/xtradb/mysql-test/patches/events_stress.diff --- ./mysql-test/patches/events_stress.diff 2009-07-07 14:08:24.000000000 +0200 +++ ../mariadb-xtradb6-merge/storage/xtradb/mysql-test/patches/events_stress.diff 2009-07-07 14:03:00.000000000 +0200 @@ -1,5 +1,5 @@ ---- mysql-test/t/events_stress.test.orig 2009-04-16 19:39:47.000000000 +0000 -+++ mysql-test/t/events_stress.test 2009-04-16 19:41:16.000000000 +0000 +--- mysql-test/t/events_stress.test.orig 2009-07-05 10:29:14.000000000 +0000 ++++ mysql-test/t/events_stress.test 2009-07-05 10:30:49.000000000 +0000 @@ -61,6 +61,7 @@ } --enable_query_log @@ -8,13 +8,15 @@ SET GLOBAL event_scheduler=on; --sleep 2.5 DROP DATABASE events_conn1_test2; -@@ -135,3 +136,4 @@ - # - +@@ -137,5 +138,5 @@ DROP DATABASE events_test; + + # Cleanup +-SET GLOBAL event_scheduler=off; +SET GLOBAL event_scheduler=@old_event_scheduler; ---- mysql-test/r/events_stress.result.orig 2009-04-16 19:41:48.000000000 +0000 -+++ mysql-test/r/events_stress.result 2009-04-16 19:42:07.000000000 +0000 + --source include/check_events_off.inc +--- mysql-test/r/events_stress.result.orig 2009-07-05 10:54:30.000000000 +0000 ++++ mysql-test/r/events_stress.result 2009-07-05 10:54:48.000000000 +0000 @@ -32,6 +32,7 @@ SELECT COUNT(*) FROM INFORMATION_SCHEMA.EVENTS WHERE EVENT_SCHEMA='events_conn1_test2'; COUNT(*) @@ -23,8 +25,9 @@ SET GLOBAL event_scheduler=on; DROP DATABASE events_conn1_test2; SET GLOBAL event_scheduler=off; -@@ -63,3 +64,4 @@ +@@ -63,4 +64,4 @@ DROP TABLE fill_it2; DROP TABLE fill_it3; DROP DATABASE events_test; +-SET GLOBAL event_scheduler=off; +SET GLOBAL event_scheduler=@old_event_scheduler;
Kristian, Ok, let's try to figure out how to merge it properly :) I tried to include all your changes, but seems failed. Actually all latest changes should be in lp:xtradb. Will it work for simple merge ? Thanks, Vadim Kristian Nielsen wrote:
Vadim Tkachenko <vadim@percona.com> writes:
I made push to
lp:~maria-captains/maria/mariadb-xtradb6-merge
(not proposed for merge yet).
If everything is fine we can merge it to maria.
I am now looking into this.
I have a hard time understanding this merge. It seems you just applied manually a patch against latest lp:maria, omitting most (but not all?) of the necessary fixes I already made in the tree
lp:~maria-captains/maria/mariadb-xtradb-merge2
I think we need to work instead from my merged tree, as that includes the full bzr history of both lp:maria and lp:percona-xtradb. This is necessary to allow smooth merging of future changes in XtraDB (and MariaDB).
Basically, the merge should be a simple `bzr merge` from the xtradb6 branch into my mariadb-xtradb-merge2 branch, and then a merge of that result into lp:maria. However, I do not know which branch contains the xtradb6 code (is it pushed to Launchpad yet?). I assumed 'lp:percona-xtradb/release-6', but that code seems to be missing the changes listed below compared to mariadb-xtradb-merge.
Can you tell me which branch to merge to get the correct changes?
- Kristian.
-- Vadim Tkachenko, CTO Percona Inc. ICQ: 369-510-335, Skype: vadimtk153, Phone +1-888-401-3403 MySQL Performance Blog - http://www.mysqlperformanceblog.com MySQL Consulting http://www.percona.com/
Vadim Tkachenko <vadim@percona.com> writes:
Ok, let's try to figure out how to merge it properly :)
I tried to include all your changes, but seems failed.
Actually all latest changes should be in lp:xtradb.
Hm, we must have been misunderstanding each other? lp:~maria-captains/maria/mariadb-xtradb6-merge is missing all of the changes outside of storage/xtradb (except one CMakeLists.txt). These missing changes should not go in stand-alone xtradb, but are needed in MariaDB. lp:xtradb does not exist. lp:percona-xtradb/release-6 exists, but seems to be missing some things that are in lp:~maria-captains/maria/mariadb-xtradb-merge2, including things like #define PERCONA_INNODB_VERSION 6, /* bugfix: http://bugs.mysql.com/bug.php?id=43948 */, ... So I am confused about what needs to be merged ... What I need from you is the name of the branch that contains the stand-alone XtraDB 6 release. That is the crucial thing. - Kristian.
Kristian Nielsen wrote:
Vadim Tkachenko <vadim@percona.com> writes:
I made push to
lp:~maria-captains/maria/mariadb-xtradb6-merge
(not proposed for merge yet).
If everything is fine we can merge it to maria.
I am now looking into this.
I have a hard time understanding this merge. It seems you just applied manually a patch against latest lp:maria, omitting most (but not all?) of the necessary fixes I already made in the tree
lp:~maria-captains/maria/mariadb-xtradb-merge2
I think we need to work instead from my merged tree, as that includes the full bzr history of both lp:maria and lp:percona-xtradb. This is necessary to allow smooth merging of future changes in XtraDB (and MariaDB).
Basically, the merge should be a simple `bzr merge` from the xtradb6 branch into my mariadb-xtradb-merge2 branch, and then a merge of that result into lp:maria. However, I do not know which branch contains the xtradb6 code (is it pushed to Launchpad yet?). I assumed 'lp:percona-xtradb/release-6', but that code seems to be missing the changes listed below compared to mariadb-xtradb-merge.
Can you tell me which branch to merge to get the correct changes?
- Kristian.
-- Vadim Tkachenko, CTO Percona Inc. ICQ: 369-510-335, Skype: vadimtk153, Phone +1-888-401-3403 MySQL Performance Blog - http://www.mysqlperformanceblog.com MySQL Consulting http://www.percona.com/
Kristian Nielsen <knielsen@knielsen-hq.org> writes:
Vadim Tkachenko <vadim@percona.com> writes:
Ok, let's try to figure out how to merge it properly :)
I tried to include all your changes, but seems failed.
Actually all latest changes should be in lp:xtradb.
Hm, we must have been misunderstanding each other?
lp:~maria-captains/maria/mariadb-xtradb6-merge is missing all of the changes outside of storage/xtradb (except one CMakeLists.txt). These missing changes should not go in stand-alone xtradb, but are needed in MariaDB.
lp:xtradb does not exist. lp:percona-xtradb/release-6 exists, but seems to be missing some things that are in lp:~maria-captains/maria/mariadb-xtradb-merge2, including things like #define PERCONA_INNODB_VERSION 6, /* bugfix: http://bugs.mysql.com/bug.php?id=43948 */, ...
So I am confused about what needs to be merged ...
What I need from you is the name of the branch that contains the stand-alone XtraDB 6 release. That is the crucial thing.
And to clarify further, once we have that branch (say "lp:XXX"), here is the exact procedure that will be needed to do the merge: bzr branch lp:~maria-captains/maria/mariadb-xtradb-merge2 mybranch cd mybranch bzr merge lp:XXX # Resolve any conflicts... # Compile + run tests, and fix any errors... bzr commit -m "Merge XtraDB 6 into MariaDB." bzr push lp:~maria-captains/maria/mariadb-xtradb-merge2 So as you see, I just need to know what to replace for "XXX". After that, it would be good if you could look at the test failure in main.innodb_xtradb_bug317074 seen in our buildbot here: https://askmonty.org/buildbot/builders/hardy-amd64-dbg/builds/69/steps/test/... But let's get the merge confusion solved first :) - Kristian.
Kristian Nielsen <knielsen@knielsen-hq.org> writes:
lp:~maria-captains/maria/mariadb-xtradb6-merge is missing all of the changes outside of storage/xtradb (except one CMakeLists.txt). These missing changes should not go in stand-alone xtradb, but are needed in MariaDB.
lp:xtradb does not exist. lp:percona-xtradb/release-6 exists, but seems to be missing some things that are in lp:~maria-captains/maria/mariadb-xtradb-merge2, including things like #define PERCONA_INNODB_VERSION 6, /* bugfix: http://bugs.mysql.com/bug.php?id=43948 */, ...
So I am confused about what needs to be merged ...
What I need from you is the name of the branch that contains the stand-alone XtraDB 6 release. That is the crucial thing.
Vadim, any updates on this? All I need is the name of the branch of XtraDB you used as the base when you created lp:~maria-captains/maria/mariadb-xtradb6-merge, in other words the branch that contains release 6 of the stand-alone XtraDB. No extra work needed on your part (except to push it to Launchpad if you forgot) :-) Would be good to get this today, as I'm about to start a three week vacation, and I really want to get XtraDB merged ... - Kristian.
Kristian Nielsen <knielsen@knielsen-hq.org> writes:
I got some good progress on this. I re-did the merge using the merge-into bzr plugin (advise from bzr developers). This should allow us to merge directly from the lp:percona-xtradb tree into lp:maria in the future.
I just tested this, and it seems to work ok. I was able to merge latest XtraDB with just `bzr merge lp:parcona-xtradb`. And merging that into latest MariaDB also worked fine with just `bzr merge`. There were a number of conflicts in the merges. But I think that is only expected, and unavoidable in fact. Eg. when same problem was fixed in different ways in the different trees, or changes to InnoDB plugin files removed in MariaDB tree. Nothing serious, and we can reduce some of this by cooperating closely between XtraDB and MariaDB (and Sun and Oracle if they are willing). - Kristian.
On Tue, Jun 09, 2009 at 05:30:04PM +0200, Kristian Nielsen wrote:
I fixed the code to compile without GCC atomic operation intrinsics. Performance will be worse of course, but it is good to have it at least working. I still need to fix Maria configure.in to not disable GCC atomics for xtradb by default.
We have some headers in drizzle so we get atomic ops through the SunStudio provided way as well (or gcc). may want to look it up for better Solaris perf. -- Stewart Smith
participants (3)
-
Kristian Nielsen -
Stewart Smith -
Vadim Tkachenko