- developers - lists.mariadb.org

[Maria-developers] Patch for MDEV-10574
by jerome brauge 19 Sep '17

19 Sep '17

Alexander, Here is the patch I was talking about. It not address all aspects of MDEV-10574, especially, empty strings in table column are not view as null. But as in oracle mode, no empty string in columns should be created, an fully oracle application will never fall in this case. Regards, Jérôme. > -----Message d'origine----- > De : jerome brauge > Envoyé : mardi 19 septembre 2017 14:13 > À : 'Alexander Barkov' > Objet : RE: Oracle strings functions compatibility : substr > > Hello Alexander, > Yes I'm aware. > This point will be treated by my next patch which will also affect the others > string functions (a part of MDEV-10574). > I will refresh this new patch with the current development branch, and I > submit to you. > Regards. > Jérôme. > > > -----Message d'origine----- > > De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : mardi 19 > > septembre 2017 13:16 À : jerome brauge Objet : Re: Oracle strings > > functions compatibility : substr > > > > Hi Jerome, > > > > I noticed one more difference in SUBSTR: > > > > > > SELECT SUBSTR('aaa',1,-1) FROM DUAL; > > > > MariaDB returns an empty string. > > Oracle returns NULL. > > > > Here's Oracle's documentation: > > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions162.ht > > m > > > > > > I think we should do this in the same patch, to avoid behavior change > > in the future. > > > > Would you like to try doing this additional change? > > Or would you like me to make an incremental patch on top of yours? > > > > Thanks! > > > > On 09/19/2017 10:33 AM, jerome brauge wrote: > > > Hello Alexander, > > > It's done. > > > > > > Do you have news from Sergei for MDEV-12874, MDEV-13417 and MDEV- > > 13418 ? > > > > I asked Sergei, awaiting for his answer. I'll let you know. > > > > > > > > Thank you very much. > > > > > > > > > > > >> -----Message d'origine----- > > >> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 18 > > >> septembre 2017 18:12 À : jerome brauge Objet : Re: Oracle strings > > >> functions compatibility : substr > > >> > > >> Hello Jerome, > > >> > > >> > > >> On 09/18/2017 05:14 PM, jerome brauge wrote: > > >>> Hello Alexander, > > >>> I don't understand how I've missed such an obvious solution ! > > >>> Here is the new patch. > > >> > > >> Thanks. Now it looks simpler. > > >> > > >> Can you please do one small thing: > > >> move get_position() from "public:" to "protected:". > > >> > > >> Ok to make a pull request after this change. > > >> > > >> Thank you very much! > > >> > > >> > > >>> > > >>> Regards. > > >>> Jérôme. > > >>> > > >>> > > >>>> -----Message d'origine----- > > >>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 18 > > >>>> septembre 2017 11:30 À : jerome brauge Objet : Re: Oracle strings > > >>>> functions compatibility : substr > > >>>> > > >>>> Hello Jerome, > > >>>> > > >>>> > > >>>> On 08/17/2017 04:43 PM, jerome brauge wrote: > > >>>>> Hello Alexander, > > >>>>> Here is a patch for function SUBSTR in sql_mode=oracle (If > > >>>>> position is 0, > > >>>> then it is treated as 1). > > >>>> > > >>>> I'm thinking of how to get a smaller patch. > > >>>> > > >>>> Wouldn't it be possible to introduce: > > >>>> > > >>>> virtual longlong get_position(); > > >>>> > > >>>> > > >>>> > > >>>> longlong Item_func_substr::get_position() { > > >>>> return args[1]->val_int(); > > >>>> } > > >>>> > > >>>> longlong Item_func_substr_oracle::get_position() > > >>>> { > > >>>> longlong pos= args[1]->val_int(); > > >>>> return pos == 0 ? 1 : pos; > > >>>> } > > >>>> > > >>>> > > >>>> and just replace all calls for args[1]->val_int() to get_position() ? > > >>>> > > >>>> Thanks! > > >>>> > > >>>> > > >>>>> > > >>>>> Regards, > > >>>>> Jérôme. > > >>>>>

1 0

Re: [Maria-developers] NUMA Support in MariaDB
by serg＠mariadb.org 19 Sep '17

19 Sep '17

Hi, Sumit! On Sep 16, Sumit Lakra wrote: > The project 'NUMA Support in MariaDB' is still incomplete and like I > mentioned in my mails to my mentors, I am still willing to work on it > and see it through to the end. Great, thanks! > Most of the minor tasks along with some important tasks are already > complete. The important part that remains is to work on the task-queue > interface between SQL and InnoDB. I was unable to implement this for > the NUMA case and I won't be able to complete it without some help. > So, do you think you can connect me with someone from the > MariaDB/InnoDB/MySQL team who is an expert of this interface? I will > need someone who has a very good understanding of the present > structure to help me come up with a way to implement this for the NUMA > case. If you do find someone who can be of help, please see to it that > since GSoC is over (and otherwise as well), it shouldn't be a > Student-Mentor kind of thing, but more like two or more developers > working together trying to come up with a solution to this problem. Why wouldn't you ask the question on the maria-developers@ mailing list? I'll try to make sure that it'll be answered. > However, if you are unable to connect me with someone who has > expertise in the handler interface, I am afraid I won't be able to > complete this project on my own. In this case, would you like me to > create a pull request of the work done so far ( > https://github.com/theGodlessLakra/server/tree/NUMA_Support) Although > NUMA Support is incomplete, merging this code to the MariaDB code base > will allow other developers to continue working on this. Is it sufficiently complete to be useful as is? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 6a72d15: MDEV-10767 /tmp/wsrep_recovery.${RANDOM} file created in unallowed SELinux context
by Sergei Golubchik 13 Sep '17

13 Sep '17

Hi, Sachin! On Aug 31, sachin wrote: > revision-id: 6a72d154d6ec231eeff861496e4d07101ac41553 (mariadb-10.1.26-16-g6a72d15) > parent(s): dda40b930498b70bb5546f857b27744039a5649d > author: sachin > committer: sachin > timestamp: 2017-08-31 12:17:10 +0530 > message: > > MDEV-10767 /tmp/wsrep_recovery.${RANDOM} file created in unallowed SELinux context > > Solution:- Allowed mysqld_t to open file with context initrc_tmp_t. Why does the server need it? I've only found scripts/galera_recovery.sh, that does: log_file=$(mktemp /tmp/wsrep_recovery.XXXXXX) /usr/sbin/mysqld $cmdline_args --user=$user --wsrep_recover --log-error="$log_file" For that use case, I wouldn't bother with a new selinux policy, it'd be safer to do log_file=$(mktemp /tmp/wsrep_recovery.XXXXXX) /usr/sbin/mysqld $cmdline_args --user=$user --wsrep_recover --disable-log-error 2>"$log_file" Note, that the similar piece of code is used in mysqld_safe.sh, only there it uses DATADIR/wsrep_recovery.XXXXXX, apparently somebody's attempt to work around selinux. Better to make it identical to galera_recovery.sh, I'd say. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] [External] Obsolete GTID domain delete on master (MDEV-12012, MDEV-11969)
by andrei.elkin＠pp.inet.fi 13 Sep '17

13 Sep '17

Hello, Simon, Kristian. (The mail was meant to be sent out yesterday, but it got stuck in my outgoing box). > Simon Mudd <simon.mudd(a)booking.com> writes: > >> ids. Obviously once all appropriate bin logs have been purged >> (naturally by other means) then no special processing will be needed. > > Right. Hence my original idea (which was unfortunately never implemented so > far). If at some point a domain has been unused for so long that all GTIDs > in that domain are gone, it is relatively safe to pretend that the domain > never existed. > > I would like to understand if you can think of significant use cases where > the DBA needs to have active binlog files in the master containing some > domain, while simultaneously pretending that this domain never existed. > > Or if it is more of a general concern, and the inconvenience for users to > have to save old binlogs somewhere else than the master's data directory and > binlog index (SHOW BINARY LOGS). > >> removing old binary logs should _not_ IMO be done as a way of >> forgetting the past obsolete domains. >> BINLOGS are important so throwing them away is an issue. I think >> that somehow the code needs >> to be aware of the cut-off point and when the “stale domain ids” are removed.) Simon, initially I thought of masking out the problematic domain so that the most recent binlog file would not have it in its Gtid_list header. Yet I've given up that idea to have agreed the strict setup on master weighs much more. > > I understand the desire to not delete binlog files. And the mdev-12012 use case might even not require to conduct this purging/flush-delete-domain procedure if IGNORE_DOMAIN_IDS (for the zero id domain in question) would do. There seems to be MDEV-9108 in the way though, but conceptually the DBA may have a way to stay with the binlog files even having a problematic domain. > > The problem is: If you want to have GTIDs with some domain in your active > binlog files, _and_ you also want to pretend that this domain never existed, > what does it mean? What is the semantics? It creates a lot of complexities > for defining the semantics, for documenting it, for the users to understand > it, and for the code to implement it correctly. > > So basically, I do not understand what is the intended meaning of FLUSH > BINARY LOGS DELETE DOMAIN d _and_ at the same time keeping GTIDs with domain > d around in active binlog files? In what respects is the domain deleted, and > in what respects not? > > For the master, the binlog files are mainly used to stream to connecting > slaves. Deleting a domain means replacing the conceptual binlog history with > one in which that domain never existed. So that domain will be ignored in a > connecting slaves position, assuming it is served by another multi-source > master. If a new GTID in that domain appears later, it will be considered > the very first GTID ever in that domain. > > So consider what happens if there is anyway GTIDs in that domain deeper in > the binlog: > > 1. An already connected slave may be happily replicating those GTIDs. If > that slave reconnects (temporary network error for example), it will instead > fail with unknown GTID, or perhaps just start silently ignoring all further > GTIDs in that domain. This kind of unpredictable behaviour seems bad. > > 2. Suppose a slave connects with a position without the deleted domain. The > master starts reading the binlog from some point. What happens if a GTID is > encountered that contains the deleted domain? The slave will start > replicating that domain from some arbitrary point that depends on where it > happened to be in other domains at the last disconnect. This also seems > undesirable. > > There may be other scenarios that I did not think about. > >> DBAs do not like to remove bin logs “early" as unless you keep a copy >> somewhere you may lose valuable information, >> for recovery, for backups etc. Not everyone will be making automatic >> copies (as MySQL does not provide an automatic way to do this) > > Understood. Maybe what is needed is a PURGE BINARY LOGS that removes the > entries from the binlog index (SHOW BINARY LOGS), but leaves the files in > the file system for the convenience of the sysadmin? (Well, you can just > hand-edit binlog.index, but that requires master restart I think). Like I said above, a filtering solution could be helpful. > >> The other comment I see mentioned here was “make sure all slaves are >> up to date”. That’s going to be hard. The master can only be >> aware of “connected slaves” and if you have intermediate masters, or a > > Indeed, the master cannot ensure this. The idea is that the DBA, who decides > to delete a domain, must understand that this should not be done if any > slave still needs GTIDs from that domain. This is similar to configuring > normal binlog purge, where the DBA needs to ensure that binlogs are kept > long enough for the needs of the slowest slave. > >> FWIW expiring old domains is good to do. There’s a similar FR for > >> completely different the problem space is the same. Coming up with a >> solution which is simple to use and understand and also >> avoids where that’s possible making mistakes which may break >> replication is good. So thanks for looking at this. > > Indeed. And the input from people like you with strong operational > experience is very valuable to end up with a good solution, hence my request > for additional input. > > - Kristian. I hope clarifications given by Kristian back up the new feature idea. On the other hand it would be good to have a way for DBA to cope with the likes of mdev-12012 (there are few reports on this matter) without purging. I rely on a solution to MDEV-9108 in that regard. Cheers, Andrei

1 0

[Maria-developers] A problem with implementing Group Commit with Binlog with MyRocks
by Sergey Petrunia 12 Sep '17

12 Sep '17

Hello, This is about https://jira.mariadb.org/browse/MDEV-11934. I've encountered an insteresting issue here, so I thought I would consult on both MyRocks and MariaDB lists. == Some background == "group commit with binlog" feature needs to accomplish two goals: 1. Keep the binlog and the storage engine in sync. This is done by employing XA between the binlog and the storage engine. It works by making these calls: /* Make the transaction's changes to be ready to be committed (no conflicts with other transactions, etc) but do not commit them yet. The effects of the prepare operation must be synced to disk, as the storage engine needs to be able to recover (i.e. commit) the prepared transaction after a crash */ storage_engine->prepare(sync=true); /* After this call, the transaction is considered committed. In case of a crash, the recovery process will use the contents of the binlog to determine which of the prepared transactions are to be committed and which are to be rolled back. */ binlog->write(sync=true); /* Commit the transaction in the storage engine. This makes its changes visible to other transactions (and also releases its locks and so forth) Note that most of the time(*) we don't need to sync there. In case of a crash we will be able to recover using the binlog. */ storage_engine->commit(); 2. The second goal is to make operation performant. We need two coordinated disk flushes per transaction, the idea is to do "Group Commit" where multiple transactions share disk flushes. So, we need to do group commit and keep the storage engine and the binlog in sync while doing that. == Group Commit with Binlog in MySQL == MySQL (and fb/mysql-5.6 in particular) does in the following phases: Phase #1: Call storage_engine->prepare() for all transactions in the group. The call itself is not persistent. Phase #2: Call storage->engine->flush_logs(). This makes the effect of all Prepare operations from Phase#1 persistent. Phase #3: Write and sync the binary log. Phase #4: Call storage_engine->commit(). This does not need to be persistent. MyRocks implements them. == Group Commit with Binlog in MariaDB == MariaDB does not have these phases described above: > Phase #1: > Call storage_engine->prepare() for all transactions in the group. > The call itself is not persistent. > > Phase #2: Call storage->engine->flush_logs(). > This makes the effect of all Prepare operations from Phase#1 persistent. A quote from Kristian's description at https://lists.launchpad.net/maria-developers/msg10832.html >> So the idea is to do group prepare with the same group of transactions that >> will later group commit to the binlog. In MariaDB, this concept does not >> exist. Storage engine prepares are allowed to run in parallel and in any >> order compared to binlog commit. Initially this looked like it could work for MyRocks. MyRocks has a group commit implementation, both Prepare() and Commit() operations participate in groups. However when I implemented a group commit implementation I found its performance to be close to what one would expect if there was no commit grouping, and commit() call flushed to disk https://jira.mariadb.org/browse/MDEV-11934 has the details. == The issue == (I'm 95% certain about this. It's not 100% yet but it is very likely) RocksDB's Group Write (see rocksdb/rocksdb/db/db_impl_write.cc, DBImpl::WriteImpl function) handles both Prepare() and Commit() commands and does the following: 1. Controls writing the commited data into the MemTable 2. Writes transactions to WAL 3. Syncs the WAL. All three steps are done for the whole group. This has a consequence: a Commit() operation that does not need to sync the WAL will still be delayed if another operation in the group needs the WAL to be synced. This delay has a disastrous effect, because SQL layer tries to have the same order of transactions in the storage engine and in the binlog. In order to do that, it calls rocksdb_commit_ordered() for each transaction sequentially. Delaying one transaction causes a delay of the entire SQL-level commit group. == Possible solutions == I am not sure what to do. - Make the SQL layer's Group Commit implementation invoke hton->flush_logs() explicitly, like MySQL does? - Modify RocksDB so that Transaction::Commit(sync=false) do not use Group Write? I am not sure if this is possible: Group Write is not about only performance, it's about preventing concurrent MemTable writes. AFAIU one cannot just tell a certain DBImpl::WriteImpl() call to not participate in write groups and work as if there was no other activity. - Modify RocksDB so that Transaction::Commit(sync=false) does not wait until its write group finishes WAL sync? This could be doable but is potentially complex. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

3 5

[Maria-developers] ha_innobase::info_low() n_rows hack
by Aleksey Midenkov 11 Sep '17

11 Sep '17

In ha_innobase::info_low() there is following dirty hack: /* The MySQL optimizer seems to assume in a left join that n_rows is an accurate estimate if it is zero. Of course, it is not, since we do not have any locks on the rows yet at this phase. Since SHOW TABLE STATUS seems to call this function with the HA_STATUS_TIME flag set, while the left join optimizer does not set that flag, we add one to a zero value if the flag is not set. That way SHOW TABLE STATUS will show the best estimate, while the optimizer never sees the table empty. */ if (n_rows == 0 && !(flag & HA_STATUS_TIME)) { n_rows++; } It is very old (from 5.0 or earlier) and bug-prone. Because in ha_innobase::open(): info(HA_STATUS_NO_LOCK | HA_STATUS_VARIABLE | HA_STATUS_CONST); every opened empty table will be non-empty! I don't know what is the problem with join optimizer, but having storage engine to handle it seems not the right thing to do. Moreover, relying on HA_STATUS_TIME in this is definitely wrong. We can make join optimizer to ignore "0 rows case" for all storage engines. Is it big win from "1 row case" anyway? Or we can make new flag HA_JOIN_STAT and use it in make_join_statistics().

2 3

Re: [Maria-developers] A problem with implementing Group Commit with Binlog with MyRocks
by Sergey Petrunia 11 Sep '17

11 Sep '17

Hi Kristian, On Mon, Sep 11, 2017 at 01:02:31PM +0200, Kristian Nielsen wrote: > Sergey Petrunia <sergey(a)mariadb.com> writes: > > > This is about https://jira.mariadb.org/browse/MDEV-11934. I've encountered > > an insteresting issue here, so I thought I would consult on both MyRocks and > > MariaDB lists. > > Is your current code available somewhere? > No, it wasn't. Just pushed the current code into bb-10.2-mdev11934 branch: https://github.com/MariaDB/server/tree/bb-10.2-mdev11934 All changes are in the last commit. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] 2nd 2017 Developers Unconference
by Ian Gilfillan 11 Sep '17

11 Sep '17

The dates for the Developers Unconference in Shenzhen, China, are coming up fast. This is our first official event in Asia, and will be held from 13 November - 17 November. 13 November - new contributor day 14-15 November - traditional developers unconference 16-17 November - patch days As always, it's free to attend, and open to anyone interested in MariaDB development. Details are at https://mariadb.org/2017-2-developers-unconference-and-related-events-shenz… and signup on the Meetup page: https://www.meetup.com/MariaDB-Developers/events/241873269/

1 0

[Maria-developers] GSoC NUMA Evaluation Student failed Mail 2
by Sumit Lakra 06 Sep '17

06 Sep '17

Hello, I am not aware of what information regarding this project has been exchanged between my mentor Daniel Black and others including serg, so I am sharing this here for everyone to see. My last mail contains the conversations I had with my mentors. Some quotes from the final evaluation - "Your questions in the last week have been exceptionally keen even though these seem to be the same questions from 1-2 months ago" Remind me when you (Daniel Black) answered them. "Sumit, overall we are happy with the quality of work. Unfortunately the quantity of work showed a lack of putting in hours and limited test cases fell short of the project plan and organisational expectations" What limited test cases ? I wrote all the test cases you (Daniel Black) and JAN Lindstrom asked me to write. Which test case did I miss ? I even ran a bench-marking test with all possible values and sent you the results. However, testing the actual performance difference is something that can only be done in a real NUMA Machine. Sorry, I don't have that. And have you ever even tested my code in a real NUMA Machine ? I too deserve to know if my work has made any real difference on the machine it is supposed to work on. And I have asked you to run it on a real machine more than once over the past months. You are happy with the quality of the work. Good. You are unhappy with the quantity. How do you expect me to increase the quantity when I already completed the tasks I promised in my GSoC proposal except for those that you removed from the list. What project plan are you referring to ? My GSoC proposal - https://docs.google.com/document/d/1UbOQysgOOzCM7z5FPC7gmuCpyxPmB7RGg7y9Ogq… Tasks spreadsheet - https://docs.google.com/spreadsheets/d/1nE-qFXhwwhF0hpOX3wUlczPjlPZTK_UzBu_… The tasks can be categorised as follows. [1] The tasks mentioned in my GSoC proposal which were not removed / changed later. - I completed all these. [2] The tasks mentioned in my GSoC proposal which were crossed out / (under consideration to be dropped / unnecessary) / or were implemented differently. - I either did not complete these for obvious reasons or implemented some of them differently. [3] The tasks you added (in the last month) to the spreadsheet originally under the title 'Out of Time' in the last month. - I completed some of these too. Then when these were done you moved a task (the queue per node task) from the 'Out of Time' to the third month group. We both agreed it was an important part for the completion of the project and I tried my best to implement it. I spent days trying to figure it out, asked you for help, didn't get any, tried again and yet failed because I am no expert of either InnoDB or SQl side of MariaDB. Which organisational expectations did you ask me to fulfil ? When ? I don't remember ever being told about any organisational expectations. Perhaps these were your personal expectations of having a complete NUMA Support in MariaDB by the end of GSoC. If so, then I am really sorry I wasn't able to fulfill those expectations because I got stuck at an important part of the project which also was a seemingly impossible task to do and *YOU FAILED TO HELP ME* with it even though I repeatedly asked (begged actually) you for help. You just kept reminding me that to do the task was my job. That's some real good mentoring right there. Another quote from the final evaluation "Passing to get your continued work isn't honoring the rules of the project or being fair to Google" Yeah, How about showing some good mentor ship by helping me to complete the project while we had the time to do it within Google's timelines in stead. Oh! wait a sec... you never went through the code.. until 3-4 days ago. I was determined to complete it if you had helped me. I have been mentioning it in all mails since the mail I wrote to you and Jan on : 22nd Aug - "Last but not least, if you think it can still be done and have a idea, I will be more than willing to attempt it. After all, GSoC has only got me started to contribute to open-source and this evaluation won't be the end of it." 25th Aug - "I am sorry I am unable to create an implementation. It's not like I don't want to. I just can't completely understand how it has been implemented currently. I have tried to follow the code before, and I did again the whole day today, but I can't come up with an idea at all." 25th Aug - "Like I said, I am not unwilling to complete this task. But I can't even think of where to start. Since, the two of you have greater experience with InnoDB, maybe you should give it a shot. You will definitely understand the present structure better than I do. And if you can come up with even a verbal solution that sounds like it could work, I will implement it asap." 25th Aug - "I spent all day trying to think of any way to implement a queue-per-node thing, and I confess I failed to come up with one. So, in stead of spending more time on that, I will work on migrating these commits to a new branch today. And I also urge you to give this task a try yourselves." 25th Aug - " I haven't been able to implement the queue per node structure for background threads, and I am afraid I won't be able to do it without your help. It is beyond me. I gave it a few attempts and I have failed. It's not the implementation part but the 'coming up with a way to implement it' part that I haven't been able to figure out yet. I need you to know that I am trying my best here." 25th Aug - "But anyway I kindly request you two to try it out yourselves now. You don't have to do the work. I just urge you to take some time out this weekend and go through the code. Let me know how you think it may be possible to implement it. I will do it. I have 4 days 2 hrs before the deadline to submit the evaluation, ends. If you can come up with an implementation plan within the next 2 or 2.5 days, I assure you I will code the implementation within a day of that and probably commit it before submitting the final evaluation as well. I am willing to take this risk." On Aug 28th you wrote a mail and asked me "Knowing what plans you had and how they failed would have been good to know." So, on the same day I wrote you a mail with a detailed explanation of the attempts I had made in trying to complete the task, asking you again to try it yourself and help me if you could come up with an idea. You never did. 31st Aug http://marialog.archivist.info/2017-08-31.txt Me - "did you have an attempt at the task ? come up with any ideas yet ?" dragonheart - been a bit busy with my own work so far. I know I'm going to have to look at a lot of stuff to finish the final GSoC evaluation 4th Sept http://marialog.archivist.info/2017-09-04.txt Me - "You must have seen my mails. Did you have an attempt at the queue per node thing ? Any ideas ?" dragonheart - well, implementing it was your task. but no, haven't found the hours to look though the code again. As per our conversation on IRC clearly Daniel Black hadn't even gone through the code by 4th Sept or attempted to solve the problem. How did he hope to complete the final evaluation. Even if he went through the code a day before submitting his evaluation, isn't this too late for a mentor to go through his/her student's code. Here is a list of some questions I would like MariaDB staff to ponder upon especially serg. [1] Was the task doable at all ? [A] Not unless you are an expert of both the SQL side and the InnoDB side, expert enough to re-write a major part of the interfaace between them. Jan himself acknowledged he wouldn't be able to help me with this, because he has no experience with the interface. I really appreciate that. [2] Even if the task was doable by a team of SQL and InnoDB experts, was it doable by an individual student working on MariaDB for the past four months ? [A] Nope. [3] Why didn't Daniel Black help me with the task even when I clearly mentioned I would need help with the task weeks ago ? [A] ??? [4] Did my mentor Daniel Black even try to help me with this particular task ? Ever ? [A] Nope. [5] Did Daniel Black evaluate me on the basis of the number of commits or the tasks accomplished by those commits ? [A] Based on the evaluation, he clearly has been unhappy with the decreased number of commits, not the content, as all the tasks mentioned in the original proposal have been achieved along with a few more. [6] Can Daniel Black himself complete this task, and thus the project ? [A] I bet not. No doubt the project is incomplete because it involves doing a task that is nearly impossible to do, at least for me, especially without any help. But does GSoC means that the student has to necessarily complete the entire project. I have gone through the code submitting guidelines plenty of times. It clearly states that we must state which tasks were done, and which tasks remain. Obviously not all projects are complete, some only complete the project partially, but mention everything and continue working on them. I did the same, and mentioned in my evaluation that almost all tasks were complete, in stead of all, and I have always been willing to complete this project but man, I can't do something impossible, especially when the people I am supposed to ask for help themselves fail to help their students. Thanks, Sumit

1 0

[Maria-developers] GSoC NUMA Evaluation Student failed Mail 1
by Sumit Lakra 06 Sep '17

06 Sep '17

Hello Serg, I am sorry to bug you with this mail but I am not sure what discussions you and my mentors have had before I was failed in GSoC. I completed all the tasks I had mentioned in my proposal but the project is incomplete. No doubts about that, but it is so because I got stuck with a major task (added in the last month) of the project for which I received no help from my mentors. These are the conversations I had with my mentors recently regarding the same task. Thanks, Sumit ---------- Forwarded message ---------- From: Sumit Lakra <sumitlakradev(a)gmail.com> Date: Tue, Aug 22, 2017 at 1:04 AM Subject: GSoC Final Evaluation To: Daniel Black <daniel.black(a)au1.ibm.com>, Jan Lindström < jan.lindstrom(a)mariadb.com> Hello, I went through the code that the InnoDB background threads use to pick tasks and execute them, in detail. This is how they seem to work. The reader/writer threads are passed an integer which acts as a segment. They then call fil_aio_wait(segment) which calls os_aio_handler(segment, &node, &message, &type). The control then goes to the os_aio_simulated_handler(segment, m1, m2, request) where the code gets more complicated with AIO arrays and slots. It gets harder to understand how they choose their tasks. It is definitely not a simple queue structure from which they pick their tasks. Also, which buffer pool the task is related to can only be figured out quite later.. based on the value of m2, which stores the address of a bpage. A simple queue could have been easily replaced with multiple queues, i.e. a queue per numa-node like we had once discussed on IRC. Lastly, all these procedures are common for log threads as well. Another thing, you mentioned more than once you wanted the reader threads to look for a bpage in their local nodes before looking them up in other nodes, but they use the bpage structure itself like I mentioned. Obviously neither of us had a proper understanding of how InnoDB worked in these aspects when we started the project. These threads seem to operate on bpages mostly rather than buf_pools, which makes numa mapping even harder (buf_pools to numa nodes would comparatively be easier), but is definitely more efficient than buf_pools and hence shouldn't be changed. Then again there were cases when the tasks assigned to the background threads were done by other server threads as well, especially in case of flushing, but I successfully restricted them to their nodes. If we were to create queues per node, which queues would these other threads work on. In other words the way in which InnoDB was initially written and expanded has made it significantly difficult to be adapted for explicit NUMA support. It wouldn't just require restructuring and changing the way these threads use parameters, pick tasks etc, but may also require re-ordering the order in which they are executed. For example, the background threads are created before the user threads, and trying to use the NUMA node from user THD later on would mean more system calls and probably thread migrations. When you added the task in the spreadsheet, you were right to anticipate that it could require large cleanup of the InnoDB structure, but I am beginning to think it will be way more complicated. Also, as you once mentioned that most architectures have 2 to 4 NUMA nodes. No doubt, if we could implement a support for NUMA architectuure such that InnoDB would make the best use out of it, there would be a performance difference but it would still be quite negligible in a very fast computer, and I really don't think making such big changes to InnoDB would be worth the effort and risk. Trying to bring a major change to a working and verified code (written by someone else) is more error-prone than adding some modular code and functions to support a new feature. I hope you agree. Last but not least, if you think it can still be done and have a idea, I will be more than willing to attempt it. After all, GSoC has only got me started to contribute to open-source and this evaluation won't be the end of it. And since there's no other database software out there which supports NUMA (let's ignore numa-interleave) and since I was the first to start working on this, I will be really proud to see it through to the end. Some Evaluation stuff : I see only one text field in the form for a link to the work done, and it takes a single url. Reserving this for the github link, where do I add a link to the spreadsheet or documentation ? Assuming you have already verified the work done so far, I will be creating a pull request shortly, so I can add whether it was merged or not, in the evaluation. Thanking You, Sumit ---------- Forwarded message ---------- From: Daniel Black <daniel.black(a)au1.ibm.com> Date: Thu, Aug 24, 2017 at 12:11 PM Subject: Re: GSoC Final Evaluation To: Sumit Lakra <sumitlakradev(a)gmail.com>, Jan Lindström < jan.lindstrom(a)mariadb.com> On 22/08/17 05:34, Sumit Lakra wrote: > Hello, > > I went through the code that the InnoDB background threads use to pick > tasks and execute them, in detail. This is how they seem to work. > > The reader/writer threads are passed an integer which acts as a segment. > They then call fil_aio_wait(segment) which calls os_aio_handler(segment, > &node, &message, &type). The control then goes to the > os_aio_simulated_handler(segment, m1, m2, request) where the code gets > more complicated with AIO arrays and slots. It gets harder to understand > how they choose their tasks. It is definitely not a simple queue > structure from which they pick their tasks. Also, which buffer pool the > task is related to can only be figured out quite later.. based on the > value of m2, which stores the address of a bpage. > A simple queue could > have been easily replaced with multiple queues, i.e. a queue per > numa-node like we had once discussed on IRC. Yes, you know where requests are created and where the job message ends up so you should be able to create an implementation. Even if it doesn't preserver the existing behaviour it should be enough to test it. > Lastly, all these > procedures are common for log threads as well. relevance? > Another thing, you mentioned more than once you wanted the reader SQL threads? Or is this the same thing? > threads to look for a bpage in their local nodes before looking them up > in other nodes, but they use the bpage structure itself like I > mentioned > Obviously neither of us had a proper understanding of how > InnoDB worked in these aspects when we started the project. No project knows all the details. > These > threads seem to operate on bpages mostly rather than buf_pools, which > makes numa mapping even harder You could add a node_id to the class buf_page_t? > (buf_pools to numa nodes would > comparatively be easier), but is definitely more efficient than > buf_pools and hence shouldn't be changed. > Then again there were cases when the tasks assigned to the background > threads were done by other server threads as well, especially in case of > flushing, but I successfully restricted them to their nodes. If we were > to create queues per node, which queues would these other threads work on. Yes, which is why this issue was raised with you months ago. > In other words the way in which InnoDB was initially written and > expanded has made it significantly difficult to be adapted for explicit > NUMA support. The split of buffer pool instances was in appreciation that there there are hot locks causes by a large number of threads. It just needed more thought than implementing a few thread bindings. I'm sorry you though it was that easy. It wouldn't just require restructuring and changing the > way these threads use parameters, pick tasks etc, You've changed very few parameters so far. https://github.com/MariaDB/server/compare/10.2...theGodlessLakra:numa_one but may also require > re-ordering the order in which they are executed. I can't think of one example of this. > For example, the > background threads are created before the user threads, and trying to > use the NUMA node from user THD later on would mean more system calls > and probably thread migrations. Huh, what? "may..require..(bad design)". Which is why a mapping of THD to known background threads per node was desired, to avoid syscall thread migrations. > When you added the task in the spreadsheet, you were right to anticipate > that it could require large cleanup of the InnoDB structure, but I am > beginning to think it will be way more complicated. I mentioned in on 2017-07-10 too (http://marialog.archivist.info/2017-07-10.txt) too and you said you'd give it a try. > Also, as you once > mentioned that most architectures have 2 to 4 NUMA nodes. No doubt, if > we could implement a support for NUMA architectuure such that InnoDB > would make the best use out of it, there would be a performance > difference but it would still be quite negligible in a very fast > computer, Cross node cache-coherency takes significant clock cycles compared to local access. What you'd end up with is a system that has idle time that can't be used. By finishing this task this could be measured. local/remote perf access can be measure (perhaps even with fake_numa). https://joemario.github.io/blog/2016/09/01/c2c-blog/ > and I really don't think making such big changes to InnoDB > would be worth the effort and risk. There is no gain without risk. Small changes with careful review, running full tests between them is they way large changes are made low risk. > Trying to bring a major change to a > working and verified code (written by someone else) is more error-prone > than adding some modular code and functions to support a new feature. I > hope you agree. When developing you are always developing for current and future use. As such architectural changes need to be commiserate to the functional changes. Jan and I have placed emphasis on writing test cases. These ensure that changes. > Last but not least, if you think it can still be done and have a idea, I > will be more than willing to attempt it. After all, GSoC has only got me > started to contribute to open-source and this evaluation won't be the > end of it. > And since there's no other database software out there which > supports NUMA (let's ignore numa-interleave) and since I was the first > to start working on this There is, found https://github.com/aerospike/aerospike-server/blob/master/cf/src/hardware.c the other day. Wouldn't have helped much as it has a different architecture and getting on top of one was hard enough. >, I will be really proud to see it through to > the end. This week looks like you've done one commit that is largely a revert that Jan asked for. You have slowed down considerably. > Some Evaluation stuff : > I see only one text field in the form for a link to the work done, and > it takes a single url. Reserving this for the github link, where do I > add a link to the spreadsheet or documentation ? > > Assuming you have already verified the work done so far, I will be > creating a pull request shortly, so I can add whether it was merged or > not, in the evaluation. Looks like the fake_numa stuff is incomplete. Pinning of background threads looks odd as they aren't big allocators of memory and aren't associated with SQL threads/buffer pool instance nodes. I have looked though it. As a pull request 60 commits with 26(?) of them fixing prior commits will look messy for anyone trying to look at the changes. If you are up for trying to clean it up please give it a go. Doing it in a copied repository is a good idea as you've seen before it can get a bit hard to keep track of. > Thanking You, > Sumit > ---------- Forwarded message ---------- From: Sumit Lakra <sumitlakradev(a)gmail.com> Date: Fri, Aug 25, 2017 at 3:03 AM Subject: Re: GSoC Final Evaluation To: Daniel Black <daniel.black(a)au1.ibm.com> Cc: Jan Lindström <jan.lindstrom(a)mariadb.com> On Thu, Aug 24, 2017 at 12:11 PM, Daniel Black <daniel.black(a)au1.ibm.com> wrote: > > > On 22/08/17 05:34, Sumit Lakra wrote: > > Hello, > > > > I went through the code that the InnoDB background threads use to pick > > tasks and execute them, in detail. This is how they seem to work. > > > > The reader/writer threads are passed an integer which acts as a segment. > > They then call fil_aio_wait(segment) which calls os_aio_handler(segment, > > &node, &message, &type). The control then goes to the > > os_aio_simulated_handler(segment, m1, m2, request) where the code gets > > more complicated with AIO arrays and slots. It gets harder to understand > > how they choose their tasks. It is definitely not a simple queue > > structure from which they pick their tasks. Also, which buffer pool the > > task is related to can only be figured out quite later.. based on the > > value of m2, which stores the address of a bpage. > > > A simple queue could > > have been easily replaced with multiple queues, i.e. a queue per > > numa-node like we had once discussed on IRC. > > Yes, you know where requests are created and where the job message ends > up so you should be able to create an implementation. Even if it doesn't > preserver the existing behaviour it should be enough to test it. > I am sorry I am unable to create an implementation. It's not like I don't want to. I just can't completely understand how it has been implemented currently. I have tried to follow the code before, and I did again the whole day today, but I can't come up with an idea at all. > > > Lastly, all these > > procedures are common for log threads as well. > > relevance? The reader/writer/log threads all start by executing fil_aio_wait(segment), in srv0start.cc > > > Another thing, you mentioned more than once you wanted the reader > > SQL threads? Or is this the same thing? > > > threads to look for a bpage in their local nodes before looking them up > > in other nodes, but they use the bpage structure itself like I > > mentioned > > Obviously neither of us had a proper understanding of how > > InnoDB worked in these aspects when we started the project. > > No project knows all the details. > > > These > > threads seem to operate on bpages mostly rather than buf_pools, which > > makes numa mapping even harder > > You could add a node_id to the class buf_page_t? > Okay, but how to use it ? where ? In which functions ? > > > (buf_pools to numa nodes would > > comparatively be easier), but is definitely more efficient than > > buf_pools and hence shouldn't be changed. > > > > Then again there were cases when the tasks assigned to the background > > threads were done by other server threads as well, especially in case of > > flushing, but I successfully restricted them to their nodes. If we were > > to create queues per node, which queues would these other threads work > on. > > Yes, which is why this issue was raised with you months ago. > > > In other words the way in which InnoDB was initially written and > > expanded has made it significantly difficult to be adapted for explicit > > NUMA support. > > The split of buffer pool instances was in appreciation that there there > are hot locks causes by a large number of threads. > > It just needed more thought than implementing a few thread bindings. I'm > sorry you though it was that easy. > The split of buffer pool is not problematic to NUMA implementation. Its the task queues for the background threads that's wrecking my brain. > > It wouldn't just require restructuring and changing the > > way these threads use parameters, pick tasks etc, > > > You've changed very few parameters so far. > > https://github.com/MariaDB/server/compare/10.2...theGodlessLakra:numa_one > > but may also require > > re-ordering the order in which they are executed. > > I can't think of one example of this. > > For example, the > > background threads are created before the user threads, and trying to > > use the NUMA node from user THD later on would mean more system calls > > and probably thread migrations. > > Huh, what? "may..require..(bad design)". > One way (a really bad design), would be to create background threads when a request is made by a SQL thread, and then bind it to the node depending on the request i.e. the buffer pool instance in which the associated bpage belongs, which would mean more system calls and migrations. Obviously a very bad design. > > Which is why a mapping of THD to known background threads per node was > desired, to avoid syscall thread migrations. > The SQL threads were expected to be bound to numa nodes, which I did, but I can't think of how to map them to background threads. Can you think of a way to do this ? Let me know how you think this can be done. I will implement it asap. > When you added the task in the spreadsheet, you were right to anticipate > > that it could require large cleanup of the InnoDB structure, but I am > > beginning to think it will be way more complicated. > > I mentioned in on 2017-07-10 too > (http://marialog.archivist.info/2017-07-10.txt) too and you said you'd > give it a try. > > > Also, as you once > > mentioned that most architectures have 2 to 4 NUMA nodes. No doubt, if > > we could implement a support for NUMA architectuure such that InnoDB > > would make the best use out of it, there would be a performance > > difference but it would still be quite negligible in a very fast > > computer, > > Cross node cache-coherency takes significant clock cycles compared to > local access. What you'd end up with is a system that has idle time that > can't be used. > > By finishing this task this could be measured. local/remote perf access > can be measure (perhaps even with fake_numa). > https://joemario.github.io/blog/2016/09/01/c2c-blog/ > > > and I really don't think making such big changes to InnoDB > > would be worth the effort and risk. > > There is no gain without risk. Small changes with careful review, > running full tests between them is they way large changes are made low > risk. > Agreed, but I honestly can't make out the heads and tails of the changes that will be required to replace the present task queue with a queue-per-node structure :( > > > Trying to bring a major change to a > > working and verified code (written by someone else) is more error-prone > > than adding some modular code and functions to support a new feature. I > > hope you agree. > > When developing you are always developing for current and future use. As > such architectural changes need to be commiserate to the functional > changes. Jan and I have placed emphasis on writing test cases. These > ensure that changes. > Like I said, I am not unwilling to complete this task. But I can't even think of where to start. Since, the two of you have greater experience with InnoDB, maybe you should give it a shot. You will definitely understand the present structure better than I do. And if you can come up with even a verbal solution that sounds like it could work, I will implement it asap. > > > Last but not least, if you think it can still be done and have a idea, I > > will be more than willing to attempt it. After all, GSoC has only got me > > started to contribute to open-source and this evaluation won't be the > > end of it. > > > > And since there's no other database software out there which > > supports NUMA (let's ignore numa-interleave) and since I was the first > > to start working on this > > There is, found > https://github.com/aerospike/aerospike-server/blob/master/cf > /src/hardware.c > the other day. Wouldn't have helped much as it has a different > architecture and getting on top of one was hard enough. > > >, I will be really proud to see it through to > > the end. > > This week looks like you've done one commit that is largely a revert > that Jan asked for. You have slowed down considerably. > You have a good reason to be upset at me and I am not complaining. But although it doesn't look from Github like much work due to no changes, I spent a good deal of time brainstorming on the implementation of the first task from the spreadsheet under the "should be done" title, unfortunately to no success. > > > Some Evaluation stuff : > > I see only one text field in the form for a link to the work done, and > > it takes a single url. Reserving this for the github link, where do I > > add a link to the spreadsheet or documentation ? > > > > Assuming you have already verified the work done so far, I will be > > creating a pull request shortly, so I can add whether it was merged or > > not, in the evaluation. > > Looks like the fake_numa stuff is incomplete. Pinning of background > threads looks odd as they aren't big allocators of memory and aren't > associated with SQL threads/buffer pool instance nodes. > Fixed the fake numa test which was failing in non-debug builds. Most of the memory allocation is done during the buffer pool creation, and these have been managed properly while creating buffer pools per numa node. What other allocators of memory do you wish to be managed ? and how ? (Non-pinned threads should probably be allowed to allocate memory from any node) > > I have looked though it. As a pull request 60 commits with 26(?) of them > fixing prior commits will look messy for anyone trying to look at the > changes. If you are up for trying to clean it up please give it a go. > Doing it in a copied repository is a good idea as you've seen before it > can get a bit hard to keep track of. > I spent all day trying to think of any way to implement a queue-per-node thing, and I confess I failed to come up with one. So, in stead of spending more time on that, I will work on migrating these commits to a new branch today. And I also urge you to give this task a try yourselves. Thanks, Sumit ---------- Forwarded message ---------- From: Sumit Lakra <sumitlakradev(a)gmail.com> Date: Fri, Aug 25, 2017 at 6:52 PM Subject: Re: GSoC Final Evaluation To: Daniel Black <daniel.black(a)au1.ibm.com> Cc: Jan Lindström <jan.lindstrom(a)mariadb.com> Hello, I am currently working on copying the changes from numa_one branch to a different branch in a more orderly fashion. I haven't been able to implement the queue per node structure for background threads, and I am afraid I won't be able to do it without your help. It is beyond me. I gave it a few attempts and I have failed. It's not the implementation part but the 'coming up with a way to implement it' part that I haven't been able to figure out yet. I need you to know that I am trying my best here. Back in http://marialog.archivist.info/2017-07-10.txt, you also said "warning the whole io read/write thread segements and scheduling is really messy and could do with a significant rework", and you also mentioned that you and Jan would have an attempt at it yourselves, if I am unable to do it and I move on. Well, I regret moving on to SQL threads back then without informing you that I wasn't successfull. A 'queue-per-numa-node' is easier said than done. But anyway I kindly request you two to try it out yourselves now. You don't have to do the work. I just urge you to take some time out this weekend and go through the code. Let me know how you think it may be possible to implement it. I will do it. I have 4 days 2 hrs before the deadline to submit the evaluation, ends. If you can come up with an implementation plan within the next 2 or 2.5 days, I assure you I will code the implementation within a day of that and probably commit it before submitting the final evaluation as well. I am willing to take this risk. However, if you are unable to come up with an implementation plan, we don't really have to hurry and stress ourselves. We can work together (I can't do this on my own) on this particular task later on and come up with something, assuming I am not failed in the final evaluation. Happy Weekend, Sumit ---------- Forwarded message ---------- From: Daniel Black <daniel.black(a)au1.ibm.com> Date: Mon, Aug 28, 2017 at 7:56 AM Subject: Re: GSoC Final Evaluation To: Sumit Lakra <sumitlakradev(a)gmail.com> Cc: Jan Lindström <jan.lindstrom(a)mariadb.com> On 25/08/17 23:22, Sumit Lakra wrote: > Hello, > > I am currently working on copying the changes from numa_one branch to a > different branch in a more orderly fashion. This is looking good: https://github.com/theGodlessLakra/server/tree/NUMA_Support Don't be afraid to be very descriptive in the commit message especially about the problem being solved and what functionality it implements/changes. Don't re-say what the code is doing, just cover its meaning at a high level. > I haven't been able to > implement the queue per node structure for background threads, and I am > afraid I won't be able to do it without your help. It is beyond me. > I > gave it a few attempts and I have failed. It's not the implementation > part but the 'coming up with a way to implement it' part that I haven't > been able to figure out yet. I need you to know that I am trying my best > here. Knowing what plans you had and how they failed would have been good to know. > Back in http://marialog.archivist.info/2017-07-10.txt, > you also said "warning the whole io read/write thread segements and > scheduling is really messy and could do with a significant rework", To do it properly yes. > and > you also mentioned that you and Jan would have an attempt at it > yourselves, if I am unable to do it and I move on. Well, I regret moving > on to SQL threads back then without informing you that I wasn't > successfull. I probably should of harassed you a bit too. Acknowledging failures is an important part to learning, even if you just acknowledge to yourself. How did SQL threads end up? > A 'queue-per-numa-node' is easier said than done. But > anyway I kindly request you two to try it out yourselves now. You don't > have to do the work. I just urge you to take some time out this weekend > and go through the code. Let me know how you think it may be possible to > implement it. I will do it. Having only just read this, this is a bit late notice. The aspect of understanding the interoperation between parts is also a fundamental aspect of getting a plan to implement it. > I have 4 days 2 hrs before the deadline to submit the evaluation, ends. > If you can come up with an implementation plan within the next 2 or 2.5 > days, I assure you I will code the implementation within a day of that > and probably commit it before submitting the final evaluation as well. I > am willing to take this risk. I thought your submission of a URL left it open to still committing on it. Please don't risk missing it. Making sure the existing code/comments/documentation is as clean as possible is better than tackling new work now. > However, if you are unable to come up with an implementation plan, we > don't really have to hurry and stress ourselves. We can work together (I > can't do this on my own) on this particular task later on and come up > with something, assuming I am not failed in the final evaluation. > > Happy Weekend, > Sumit ---------- Forwarded message ---------- From: Sumit Lakra <sumitlakradev(a)gmail.com> Date: Mon, Aug 28, 2017 at 9:50 PM Subject: Re: GSoC Final Evaluation To: Daniel Black <daniel.black(a)au1.ibm.com> Cc: Jan Lindström <jan.lindstrom(a)mariadb.com> Hello, The following is a detailed explanation of what I did, how and why, while trying to come up with a solution to the task queue problem. However, before you go through this I would still request you to have an attempt at it yourselves, so your perspectives won't be affected by my ideas and you may come up with something different. I had once tried this before in https://github.com/MariaDB/ server/commit/4b8436d3f7c573668a0d37b54dab2d05e8e102be. I even pointed this commit out to once on IRC, (can't access logs of this time). Anyway I didn't put much focus on this as I was onto some other task. Before trying to replace the present queue structure with a queue-per-node, I first tried to let all the threads access this queue as before but only pick tasks dealing with their local nodes. Neither this queue nor these tasks seem to be easily distinguishable in code. The threads start by running fil_aio_wait(segment). This function calls os_aio_handler(segment, &node, &message, &type). This calls os_aio_simulated_handler(segment, m1, m2, request). This calls AIO::get_array_and_local_segment(&array, global_segment), and also creates SimulatedAIOHandler handler(array, segment), followed by calling handler.check_pending(global_segment, event), handler.init(n_slots), and slot = handler.check_completed(&n_reserved). Now, in order to skip a task to be done by other thread, what a thread must do is pick the task, check whether the task is related to the same node or not. If the task is concerned with the same node execute it or else leave it in the queue. This is where the problem occurs. In order to figure out whether a task is concerned with the same node or not, it must know which buffer pool instance the task is associated to. This can only be figured out after the task picks up the bpage it has to deal with. The bpage already has this information by now. Anyway, the bpage a thread has to deal with can only be found out by buf_page_t* bpage = static_cast<buf_page_t*>(*m2). Also, the task is not picked up at any particular place, but is picked up in parts in different functions. All these function calls seem to make some or other kind of changes to the segment/slot/array they are picking up the task from. So, when a thread finally gets a task and finds it concerned with a different node, then in order to safely return the task back to the queue, it must also undo the changes made by all these procedures so far. This is the difficult part. Compare this with pc_flush_slot() where I successfully implemented this method of skipping tasks if not related to local pool. In the above commit you will find I have a few lines like 'ib::info() << thread_name << " accessing buf_page on node : " << node_of_pool << " in fil_aio_wait()";' If you compile and run this code you can see in real time which threads are accessing which pools. I have given proper names to them. Also, when I tried to skip tasks which were not of the same queue by returning from the function os_aio_simulated_handler(), which partly worked but on trying to return from the funtion fil_aio_wait(), there seems to be either an infinite loop or a deadlock somewhere, as the server stops midway (doesn't abort) while starting up. Hence I have commented out those lines on lines 5463 and 5464 in fil0fil.cc in the above commit. Another thing is the role of segments here. Are these segments related to buffer pool instances somehow ? Because all these threads including the log threads start there work by executing fil_aio_wait(segment), but segment here seems to be just an incrementing integer. At one point assuming these segments could contain tasks related with different pools/nodes, I added a loop such that both the read and write threads loop around fil_aio_wait(segment) calling it with all valid segments. Didn't work out. These were attempted by me when I pushed this commit weeks ago in the temp branch. And even though I pointed it out to you, I think I should have given much more importance to this task. Over the past few days, when I tried to give this task another try, I couldn't come up with any idea either to replace this queue with a queue-per-node thing or even to complete this skip-if-diff-node thing.

1 0