- developers - lists.mariadb.org

Can find sponsor: Would anyone be interested in patching mha to be MariaDB gtid compatible?
by highclass99 22 Dec '24

22 Dec '24

Hello, Would anyone be interested in patching mha to be MariaDB gtid compatible? I think I would be able to find a sponsor to pay a bit for it. reference links) https://github.com/yoshinorim/mha4mysql-manager https://github.com/yoshinorim/mha4mysql-node https://github.com/charlesdirk/MariaDB-MHA -> could not get this to work: if anyone got it to work please let me know how to get it to work. Thank you. Have a nice day.

3 3

Re: 3297680579f: Remove end newlines from CHECK TABLE output.
by Sergei Golubchik 21 Dec '24

21 Dec '24

Hi, Michael, On Dec 21, Michael Widenius wrote: > revision-id: 3297680579f (mariadb-10.6.20-59-g3297680579f) > parent(s): 4038ec2cc2c > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:25 +0200 > message: > > Remove end newlines from CHECK TABLE output. > > InnoDB has in some cases (not always) an newline at the end of error > messages. This messes up the CHECK TABLE output. > Fixed CHECK TABLE to remove the end newlines in its result set. I'd rather fix InnoDB to not do that, instead of working around InnoDB bug that garbles the CHECK TABLE output Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 6c5c1d0bf00: MDEV-35469 Heap tables are calling mallocs to often
by Sergei Golubchik 21 Dec '24

21 Dec '24

Hi, Monty, On Dec 21, Michael Widenius wrote: > revision-id: 6c5c1d0bf00 (mariadb-10.6.20-57-g6c5c1d0bf00) > parent(s): 671f80c7388 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-12-17 10:45:25 +0200 > message: > > MDEV-35469 Heap tables are calling mallocs to often > > Heap tables are allocated blocks to store rows according to > my_default_record_cache (mapped to the server global variable > read_buffer_size). > This causes performance issues when the record length is big > (> 1000 bytes) and the my_default_record_cache is small. > > Changed to instead split the default heap allocation to 1/16 of the > allowed space and not use my_default_record_cache anymore when creating > the heap. The allocation is also aligned to be just under a power of 2. > > For some test that I have been running, which was using record length=633, > the speed of the query doubled thanks to this change. > > Other things: > - Fixed calculation of max_records passed to hp_create() to take > into account padding between records. > > diff --git a/storage/heap/hp_create.c b/storage/heap/hp_create.c > index c07a1e968c4..fcfcf607822 100644 > --- a/storage/heap/hp_create.c > +++ b/storage/heap/hp_create.c ... > + of the block. > + */ > + records_in_block= > + (my_round_up_to_next_power((uint32) > + (records_in_block * recbuffer + > + sizeof(HP_PTRS) + MALLOC_OVERHEAD)/2) / > + recbuffer); Strictly speaking, it should be (my_round_up_to_next_power(records_in_block * recbuffer + extra) - extra ) / recbuffer And why /2 ? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: e0afb5f08b5: cleanup: further replace thd->alloc() with operator new
by Sergei Golubchik 10 Dec '24

10 Dec '24

Hi, Nikita, Despite the commit has in the title, comments below apply to four commits: $ git log --oneline 7d5165894282^..0f6fd23e1604 0f6fd23e160 cleanup: add new(thd) to Sql_alloc and Item e0afb5f08b5 cleanup: further replace thd->alloc() with operator new 8daea2e14b2 cleanup: add operator new(size_t, const THD*) for single-object allocations 7d516589428 cleanup: add operator new[](size_t, const THD*) On Dec 06, Nikita Malyavin wrote: > diff --git a/sql/sql_class.h b/sql/sql_class.h > index aff02df69ec..299c8298606 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1279,11 +1279,5 @@ class Query_arena > { return state == STMT_CONVENTIONAL_EXECUTION; } > > template <typename T=char> > - inline T* alloc(size_t size) const > - { > - return (T*)alloc_root(mem_root, sizeof(T)*size); > - } > - > - template <typename T=char> > inline T* calloc(size_t size) const Hmm, you didn't remove thd->calloc<>() I understand why, but that kind of makes the API inconsistent, perhaps it would've been cleaner to keep thd->alloc<>() and only implement new (thd) and new (arena) for Sql_alloc? May be not only for Sql_alloc. There are 1. pod types. they can be alloc'ed an calloc'ed, I think thd->alloc/thd->calloc is fine for them 2. non-pod types, can be allocated on the heap and on a mem_root. your new (thd)/delete (thd) operators are best for them. 3. non-pod types that can only be allocated on a mem_root. Sql_alloc can be used for them, if it'll disallow non-placement new. I feel like we don't need to make a distinction between 2 and 3, in that case Sql_alloc can simply go away. > { > diff --git a/plugin/feedback/feedback.cc b/plugin/feedback/feedback.cc > index bc8a98c11ae..dcc2e4417e5 100644 > --- a/plugin/feedback/feedback.cc > +++ b/plugin/feedback/feedback.cc > @@ -103,22 +103,22 @@ static COND* make_cond(THD *thd, TABLE_LIST *tables, LEX_STRING *filter) > nrc->resolve_in_table_list_only(tables); > nrc->select_lex= tables->select_lex; > > - res= new (thd) Item_cond_or(thd); > + res= new (thd) Item_cond_or(thd); > if (!res) > return OOM; > > for (; filter->str; filter++) > { > - Item_field *fld= new (thd) Item_field(thd, nrc, db, table, > + Item_field *fld= new (thd) Item_field(thd, nrc, db, table, > field); > - Item_string *pattern= new (thd) Item_string(thd, filter->str, > + Item_string *pattern= new (thd) Item_string(thd, filter->str, > (uint) filter->length, cs); you might want to go through the diff and fix the indentation > - Item_string *escape= new (thd) Item_string(thd, "\\", 1, cs); > + Item_string *escape= new (thd) Item_string(thd, "\\", 1, cs); > > if (!fld || !pattern || !escape) > return OOM; > > - Item_func_like *like= new (thd) Item_func_like(thd, fld, pattern, > + Item_func_like *like= new (thd) Item_func_like(thd, fld, pattern, > escape, 0); > > if (!like) > diff --git a/sql/item.h b/sql/item.h > index 631fd1b19f8..b0cf03c1435 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -857,8 +857,9 @@ class Item :public Value_source, > static void *operator new(size_t size); > > public: > - static void *operator new(size_t size, MEM_ROOT *mem_root) throw () > + static void *operator new(size_t size, MEM_ROOT *mem_root) noexcept > { return alloc_root(mem_root, size); } > + static void *operator new(size_t size, const THD *thd) noexcept; > static void operator delete(void *ptr,size_t size) { TRASH_FREE(ptr, size); } > static void operator delete(void *ptr, MEM_ROOT *mem_root) {} Better inherit it from Sql_alloc > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index d86fdf5c5d6..e0adc5c3bb4 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -2007,7 +2007,7 @@ bool Item_func_interval::fix_length_and_dec(THD *thd) > > if (not_null_consts) > { > - intervals= thd->alloc<interval_range>(rows - 1); > + intervals= new(current_thd) interval_range[rows - 1]; current_thd ? > if (!intervals) > return true; > > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index 008384a6590..21364286260 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -6086,8 +6086,8 @@ bool Ordered_key::init(int col_idx) > > // TIMOUR: check for mem allocation err, revert to scan > > - key_columns= thd->alloc<Item_field*>(1); > - compare_pred= thd->alloc<Item_func_lt*>(1); > + key_columns= new(thd) Item_field*();; > + compare_pred= new(thd) Item_func_lt*();; ;; > > key_columns[0]= new (thd) Item_field(thd, tbl->field[col_idx]); > /* Create the predicate (tmp_column[i] < outer_ref[i]). */ > diff --git a/sql/opt_subselect.h b/sql/opt_subselect.h > index 8b4dd1f0092..37902c6eebe 100644 > --- a/sql/opt_subselect.h > +++ b/sql/opt_subselect.h > @@ -357,7 +357,7 @@ uint get_number_of_tables_at_top_level(JOIN *join); > the tuple. > */ > > -class SJ_TMP_TABLE : public Sql_alloc > +class SJ_TMP_TABLE why? > { > public: > /* > diff --git a/sql/sql_alloc.h b/sql/sql_alloc.h > index f5d2d4e8b1a..e643a9f817d 100644 > --- a/sql/sql_alloc.h > +++ b/sql/sql_alloc.h > @@ -23,17 +23,21 @@ > class Sql_alloc > { > public: > - static void *operator new(size_t size) throw () > + static void *operator new(size_t size) noexcept > { > return thd_alloc(_current_thd(), size); > } can this be disallowed completely? > - static void *operator new[](size_t size) throw () > + static void *operator new[](size_t size) noexcept > { > return thd_alloc(_current_thd(), size); > } > - static void *operator new[](size_t size, MEM_ROOT *mem_root) throw () > + > + static void *operator new[](size_t size, const THD *thd) noexcept; > + static void *operator new(size_t size, const THD *thd) noexcept; > + > + static void *operator new[](size_t size, MEM_ROOT *mem_root) noexcept > { return alloc_root(mem_root, size); } > - static void *operator new(size_t size, MEM_ROOT *mem_root) throw() > + static void *operator new(size_t size, MEM_ROOT *mem_root) noexcept > { return alloc_root(mem_root, size); } > static void operator delete(void *ptr, size_t size) { TRASH_FREE(ptr, size); } > static void operator delete(void *, MEM_ROOT *){} > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 7847389dabc..37869da63ae 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -8831,3 +8831,46 @@ LEX_CSTRING make_string(THD *thd, const char *start_ptr, > size_t length= end_ptr - start_ptr; > return {strmake_root(thd->mem_root, start_ptr, length), length}; > } > +void* operator new(size_t size, const Query_arena *thd) noexcept > +{ > + return alloc_root(thd->mem_root, size); > +} > + > +void operator delete[](void *ptr, const Query_arena *thd) noexcept > +{} > + > +void operator delete(void *ptr, const Query_arena *thd) noexcept > +{} if you put them into sql_class.h - will they be inlined? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

MDEV-34705: Storing binlog in InnoDB
by Kristian Nielsen 06 Dec '24

06 Dec '24

I am in the early stages of a project to hopefully implement a new binlog format that is managed transactionally by InnoDB. This is a complex change, and I want to solicit comments early, so I encourage feedback. The TLDR; of this: - The current binlog format stores events separately from InnoDB. This requires a complex and _very_ constly 2-phase commit protocol between the binlog and InnoDB for every commit. - The binlog format is rather naive, and not well suited as a high-performance transactional log. It exposes internal implementation detail to the slaves, and its flat-file nature makes it inflexible to read (for example finding the last event sequential start from the start). Replacing it with something new has the potential to reap many benefits. - This project proposes to extend the storage engine API to allow an engine to implement binlog storage. It will be implemented in InnoDB so that binlog events are stored in InnoDB tablespaces, and the standard InnoDB transactional and recovery mechanisms are used to provide crash safety and optionally durability. - This mechanism will allow full crash-recovery and consistency without any fsync per commit (--innodb-flush-log-at-trx-commit=0), providing huge improvement in throughput for applications that do not require strict durability requirements. And it will reduce fsync requirement to one per commit for durable config, with greatly improved opportunities for group commit. Below a long and somewhat raw write-up of my current state of design. There was some interest expressed in learning more about this work, an interest that I'm very pleased with ;-). The current state of my prototype implementation is available in the branch knielsen_binlog_in_engine: https://github.com/MariaDB/server/commits/knielsen_binlog_in_engine Architectural considerations. The fundamental problem with the binlog is that it functions as a transactional log, but it is separate from the InnoDB transactional log. This is very unoptimal. To ensure consistency in case of crash, a two-phase commit protocol is needed that requires two fsync per commit, which is very costly in terms of performance. And it causes a lot of complexity in the code as well. We should make it so that there is only a single transactional log ("single source of truth") used, at least for the case where only a single storage engine is used (eg. InnoDB) for all tables. Some different options for doing this can be considered (explained in detail below): 1. Implement a new binlog format that is stored in a new InnoDB tablespace type (or more generally in any storage engine that implements the appropriate new API for binlog). 2. Keep the current binlog format, extend the InnoDB redo log format with write-ahead records for binlog writes. Integrate binlog writes with InnoDB checkpointing and buffer pool flush. 3. Store the binlog events directly inside the InnoDB redo log. Implement log archiving of the InnoDB log to preserve old binlog data as long as desired. 4. (Mentioned only for completeness): Implement a general redo log facility on the server level, and change both InnoDB and binlog to use that facility. I am currently working on option (1), inspired by talks I had with Marko Mäkelä at the MariaDB fest last Autumn. He suggested that a new type of tablespace be added to InnoDB which would hold the binlog, and explained that this would require relative small changes in the InnoDB code. I think this is the best option. The InnoDB log and tablespace implementation is very mature, it is very good to be able to re-use it for the binlog. The integration into InnoDB looks to be very clean, and all the existing infrastructure for log write, recovery, checkpointing, buffer pool can be re-used directly. All the low-level detail of how the binlog is stored on disk can be handled efficiently inside the InnoDB code. This option will require a lot of changes in the binlog and replication code, which is the main cost of choosing this option. I have a _lot_ of experience with the MariaDB replication code, and I think this is feasible. And I think there are another set of huge benefits possible from implementing a new binlog format. The existing binlog format is very naive and neither very flexible, nor well suited as a high-performance transactional log. So changing the binlog format is something that should be done eventually anyway. Option (2) would also be feasible I think, and has the advantage of needing less changes to the replication code. Monty mentioned seeing a patch that sounded something like this. The main disadvantage is that I think this will require complex interaction between low-level InnoDB checkpointing code and low-level binlog writing code, which I fear will eventually lead to complex code that will be hard to maintain and will stiffle further innovation on either side. It also doesn't address the other problems with the current binlog format. To me, letting the storage engine control how it stores data transactionally - including the binlog events - is the "right" solution, and worth persuing over a quicker hack that tries to avoid changing code. Option (3) is also interesting. The idea would be that the binlog becomes identical to the InnoDB redo log, storing the binlog data interleaved with the InnoDB log records. This is conceptually simpler than options (1) and (2), in the sense that we don't have to first write-ahead-log a record for modifying binlog data, and then later write the actual data modification to a separate tablespace/file. But the InnoDB log is cyclic, and old data is overwritten as the log LSN wraps over the end. Thus, this option (3) will require implementing log archiving in InnoDB, which I am not sure how much work will be required. Either the binlog data could be archived by copying out asynchroneously to persistent binlog files, which becomes somewhat similar to option (1) or (2) then perhaps. Or InnoDB could be changed to create a new log file at wrap-over, renaming the old one to archive it, though this will leave the binlog data somewhat "dilluted", mixed up with InnoDB internal redo records. One pragmatic reason I've chosen option (1) over this option (3) for now is that I am more comfortable with doing large changes to replication code than to InnoDB code. Also, option (3) would more drastically change the way users can work with the binlog files, as a large amount of the most recent data would be sitting inside the InnoDB redo log and not in dedicated binlog files, unlike options (1) and (2). Still, option (3) is an interesting idea and has the potential to reduce the write amplification that will occur from (1) and (2). So I'm eager to hear any suggestions around this. Option (4) I mention only for completeness. Somehow, the "clean" way to handle transactional logging would be if the server layer provides a general logging service, and all the components would then use this for logging. InnoDB and the binlog, but also all the other logs or log-like things we have, eg. Aria, DDL log, partitioning, etc. However, I don't think it makes sense to try to discard the whole InnoDB write-ahead logging code and replace it with some new implementation done on the server level. And I'm not even sure it makes sense from a theoretical point of view - somehow, the logging method used is intrinsically a property of the storage engine; it seems dubious if a general server-level log facility could be designed that would be optimal for both InnoDB and RocksDB, for example. High-level design. Introduce a new option --binlog-storage-engine=innodb. Extend the storage engine API with the option for an engine to provide a binlog implementation, supported by InnoDB initially. Initially the option cannot be changed dynamically. When --binlog-storage-engine=innodb, the binlog files will use a different format, stored as a new type of InnoDB tablespace. It should support that the old part of the binlog is the legacy format and the new is the InnoDB format, to facilitate migrations. Maybe also support going back again, though this is less critical initially. A goal is to clean up some of the unnecessary complexity of the legacy binlog format. The old format will be supported for the foreseeable future, so the new format can break backwards compatibility. For example, the name of binlog files will be fixed, binlog-<NNNNNN>.ibb, so each file can be identified solely by its number NNNNNN. The option to decide the directory in which to store the binlog files should be supported though. I think we can require the slaves to use GTID when --binlog-storage-engine=innodb is enabled on the master. This way we can avoid exposing slaves to internal implementation details of the binlog, and the slaves no longer need to update the file relay-log.info at every commit. We should also be able to get rid of the Format_description_log_event and the Rotate_log_event at the start of every binlog file, so that the slave can view the events from the master as one linear stream and not care about how the data is split into separate binlog files on the master. The binlog format will be page based. This will allow pre-allocating the binlog files and efficiently writing them page by page. And it will be possible to access the files without scanning them sequentially from the start; eg. we can find the last event by reading the last page of the file, binary-searching for the last used page if the binlog file is partially written. The GTID indexes can be stored inside the same tablespace as the binlog data (eg. at the end of the tablespace), avoiding the need for a separate index file. GTID index design is still mostly TBD, but I think it can be implemented so that indexes are updated transactionally as part of the InnoDB commit and no separate crash recovery of GTID indexes is needed. With GTID indexes being guaranteed available, we can use them to obtain the GTID state at the start of each binlog file, and avoid the need for the Gtid_list_log_event at the start of the binlog. With a page-based log with extensible format, metadata can be added to the binlog that is only used on the master without having to introduce new replication events that are not relevant to the replication on the slave. This can be used eg. to eliminate the Binlog_checkpoint_log_event, for example. Possibly the binlog checkpoint mechanism can be completely avoided for the --binlog-storage-engine=innodb case, since there is no more 2-phase commit needed in the common case of an InnoDB-only transaction. The existing binlog checksum and encryption will no longer be used, instead the standard InnoDB checksums and encryption will be reused. Implementation plan. The storage engine API must be extended to provide a facilities for writing to the binlog and for reading from the binlog. The design of the API is TBD, should be discussed to try to make it generic and suitable for other engines than InnoDB. When writing to binlog in InnoDB, the central idea is to use the same mini-transaction (mtr) that marks the transaction as committed in InnoDB. This is what makes the binlog data guaranteed consistent with the InnoDB table data without the need for two-phase commit. The initial version I think can use the existing binlog group commit framework; this will simplify the implementation. This will thus keep the LOCK_commit_ordered and the queue_for_group_commit() mechanisms. Later work can then be to see if this can be even further improved in terms of scalability. I want to implement that large event groups can be split into multiple chunks in the binlog that no longer need to be consecutive. The final chunk will be the one that contains the GTID and marks the event group binlogged; this final chunk will then refer back to the other parts. This way, a large transactions can be binlogged without stalling the writing of other (smaller) transactions in parallel, which is a bottleneck in the legacy binlog. And it avoids problems with exceeding the maximum size of an mtr. In the first version, I think the binlog reader in the dump thread will collect and concatenate the different chunks before sending them to the slave, so simplify the initial implementation. But a later change could allow the slave to receive the different chunks interleaved between different event groups. This can even eventually allow the slave to speculatively execute events even before the transaction has committed on the master, to potentially reduce slave lag to less than the time of even a single transaction; this would be a generalisation of the --binlog-alter-two-phase feature. The binlog will be stored in separate tablespace files, each of size --max-binlog-size. The binlog files will be pre-allocated by a background thread. Since event groups can spill over to the next file, each file can be a fixed size, and hopefully also the rotate events will no longer be necessary. A busy server will quickly cycle through tablespace files, and we want to avoid "using up" a new tablespace ID for each file (InnoDB tablespace IDs are limited to 2**32 and are not reused). Two system tablespace IDs will be reserved for the binlog, and new binlog files will alternate between them. This way, the currently written binlog can be active while the previous one is being flushed to disk and the remaining writes checkpointed. Once the previous log has been flushed completely, its tablespace ID can be re-used for the next, pre-allocated binlog file and be ready for becoming active. This way, the switching to the next binlog file should be able to occur seamlessly, without any stalls, as long as page flushing can keep up. The flushing of binlog pages will be prioritised, to avoid stalling binlog writes, to free up buffer pool pages that can be used more efficiently than holding binlog data, and to quickly make the binlog files readable from outside the server (eg. with mysqlbinlog). The binlog dump thread that reads from the binlog and sends the data to slaves will use a binlog reading API implemented in InnoDB. I will prefer to read directly from the binlog files, in order to reduce pressure on the InnoDB buffer pool etc. A slave that connects from an old position back in time may need to read a lot of old data from the binlogs; there is little value in loading this data into the buffer pool, evicting other more useful pages. The reader can lookup in the InnoDB buffer pool with the BUF_GET_IF_IN_POOL flag. This way, the data can be accessed from the buffer pool if it is present. If not present, we can be sure that the data will be in the data file, and can read it from the file directly. If the data is known to be already flushed to disk before the specific binlog position, then the buffer pool lookup can be skipped altogether. The mysqlbinlog program will need to be extended somehow to be able to read from an InnoDB tablespace (or in general other storage engine). I think this means mysqlbinlog needs some kind of engine plugin facility for reading binlog. Note that the -read-from-remote-server option will also be available to read data from a mysqlbinlog that doesn't the new format, or to read the very newest data before it gets written to disk. Final words. This is still an early draft of the feature, as it is being worked on and refined; things are subject to change. And this also means that things are *open* to change, according to any suggestions with a good rationale.

2 7

Review of MDEV-6017 Add support for Indexes on Expressions
by Michael Widenius 03 Dec '24

03 Dec '24

Hi! Here is the review for MDEV-6017 Add support for Indexes on Expressions commit c1e7574c799163f5d8a26ca8e0f91af6c91f7077 Instead of adding bool function virtual bool vcol_subst_analyzer(uchar **) { return false; } Why not add Item_base_t flag: IS_SIMPLE_BOOL_FUNC This takes one bit instead of increasing ALL created item base structures with 8 bytes. As we have 100+ items, adding a virtual function increases code space with 1K or more + the functions. You can also replace vcol_subst_analyzer() with a top level function: bool Item::vcol_subst_analyzer/( { return (type() == Item::FUNC_ITEM && (((Item_func*) this)->bitmap_bit() & (BITMAP_EQ | BITMAP_EQUAL | BITMAP_LT | BITMAP_GT | BITMAP_LE | BITMAP_GE | BITMAP_BETWEEN | BITMAP_IN | BITMAP_ISNULL | BITMAP_ISNOTNULL ) } For this to work, you would have to add the last two bitmaps to enum Bitmap in item_func.h @@ -3272,6 +3292,7 @@ class Item_cond :public Item_bool_func bool excl_dep_on_table(table_map tab_map) override; bool excl_dep_on_grouping_fields(st_select_lex *sel) override; + bool vcol_subst_analyzer(uchar **) override { return true; } I assume you need the above to be able to traverse AND and OR expressions for WHERE. Please add a comment that this is the intent. If you decide to use the bitmaps, add an explanation why AND and OR is in the vcol_subst_analyzer list. ---- +void collect_indexed_vcols_for_table(TABLE *tbl, List<Field> *vcol_fields) +{ Please use table instead of tbl. We use table as a varible much more than tbl: ((/my/maria-10.5/sql)) grep tbl *.* | wc 1143 5371 73297 ((/my/maria-10.5/sql)) grep table *.* | wc 40886 236754 2832202 + for (uint i=0; i < tbl->s->keys; i++) + { + // note: we could also support histograms here + if (!tbl->keys_in_use_for_query.is_set(i)) + continue; I wonder if it would be a good idea to extend bitmap to have an iterator like we have for tables where we only loop over the set bits. From sql_bitmap.h: class Table_map_iterator { ulonglong bmp; public: Table_map_iterator(ulonglong t): bmp(t){} uint next_bit() { if (!bmp) return BITMAP_END; uint bit= my_find_first_bit(bmp); bmp &= ~(1ULL << bit); return bit; } int operator++(int) { return next_bit(); } enum { BITMAP_END= 64 }; }; Would be quite trival to do also for keys. This would help tables with many keys and also would allows us to remove the 'call' to if (!tbl->keys_in_use_for_query.is_set(i)) We could also do an optimized version where we implement bitmap_get_next_set() where we do not have to reset the previous bits for each call. --------- + if (field->vcol_info) + vcol_fields->push_back(field); You have to take care of error handling here. Please change the functions collect_indexed_vcols_for_table(), collect_indexed_vcols_for_join() and other functions that calls push_back() to bool and return 1 in case of error. + item->top_level_compile(ctx->thd, + &Item::vcol_subst_analyzer, &yes, + &Item::vcol_subst_transformer, (uchar*)ctx); You also have to check the return of the above for errors. Another option is to just check thd->fatal_error() as this will be set in case of OOM. .... + { + if (auto nested_join= table->nested_join) + subst_vcols_in_join_list(ctx, &nested_join->join_list); PLEASE do not use auto. I want to know the type while reading and debugging the code! auto can also introduce bugs if a type changes. + @todo: this cannot handle VIEW columns currently. View columns are wrapped + in Item_direct_view_ref objects which do not comsider themselves equal to + Item_field objects they're wrapping. Please add this also to the commit comment. + /* + Do not do the substitution if the datatypes do not match. + Virtual column's datatype can be different from the expression, for + example: + + col3 INT AS (CONCAT(col1, col2)) + + Do not allow substitutions that change the semantics of comparison. + At the moment, we require that datatypes are the same. + This probably could be relaxed. + + For strings: we allow two cases: + - vcol_expr and vcol_field have the same collation. + - vcol_field has the same collation as the comparison collation. + + (Note: MySQL calls resolve_type() after it has done the substitution. + This can potentially update the comparator. The idea is that this + shouldn't be necessary as we do not want to change the comparator. + Changing the comparator will change the semantics of the condition, + our point is that this must not happen) + */ Move to a function comment. It is hard to read read the function code with this long comment in the middle. + if (vcol_expr->type_handler_for_comparison() != + vcol_field->type_handler_for_comparison() || + vcol_expr->maybe_null() != vcol_field->maybe_null()) + fail_cause="type mismatch"; Why cannot we handle this case if vcol_expr->maybe_null is 0 and vcol_field->maybe_null() is set? In this case we know that the vcol_field's we are going to encounter can never be null, which should satisfy our requirements. + Item_field *itf= new (thd->mem_root) Item_field(thd, vcol_field); + if (!itf) + return false; Better to return true in case of errors, like we do in most of the other code. Remove and a blank line instead + //Item *const_expr; ... remove + //const_expr= args[0]; ----- +Item* Item_func_in::vcol_subst_transformer(THD *thd, uchar *arg) +{ + Vcol_subst_context *ctx= (Vcol_subst_context*)arg; + if (!compatible_types_scalar_bisection_possible()) Add comment /* Check that all arguments inside IN() are constants */ Regards, Monty

1 0

Re: 5b6b23563bf: MDEV-34979 generate SBOM from server builds
by Sergei Golubchik 02 Dec '24

02 Dec '24

Hi, Vladislav, Thanks! See comments below On Dec 02, Vladislav Vaintroub wrote: > revision-id: 5b6b23563bf (mariadb-11.7.1-5-g5b6b23563bf) > parent(s): eff9c198e32 > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2024-11-25 19:04:18 +0100 > message: > > MDEV-34979 generate SBOM from server builds > > This commit adds the capability to generate a Software Bill of Materials > (SBOM) from server builds. > > It introduces a new WITH_SBOM variable, which defaults to ON for package > builds (i.e if BUILD_CONFIG is used) and to OFF otherwise. > > When enabled, the build process will produce an sbom.json document in > CycloneDX format, capturing information about various dependencies, > which is gathered from various sources. > > We use git submodule information and CMake external projects properties > to gather version information for 3rd party code, but also handle > dependencies if external code is part of our repository > (zlib, or Connect storage engine's minizip) > > The SBOM document is stored in the root build directory in sbom.json file, > but is not currently installed. > > diff --git a/CMakeLists.txt b/CMakeLists.txt > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -581,6 +581,13 @@ ENDIF() > > INCLUDE(build_depends) > > +INCLUDE(CMakeDependentOption) > +CMAKE_DEPENDENT_OPTION(WITH_SBOM "Generate Software Bill of Materials (SBOM)" OFF "BUILD_CONFIG" ON) > +IF(WITH_SBOM) > + INCLUDE(generate_sbom) > + GENERATE_SBOM() > +ENDIF() This doesn't work for me, I mean, cmake -DBUILD_CONFIG=mysql_release . does not generate the file, but cmake -DBUILD_CONFIG=mysql_release . -DWITH_SBOM=ON does. May be it'd be more robust to use OPTION() and set it from cmake/build_configuration/mysql_release.cmake? Like it's done for other options. > + > INCLUDE(CPack) > > IF(WIN32 AND SIGNCODE) > diff --git a/cmake/generate_submodule_info.cmake b/cmake/generate_submodule_info.cmake > --- /dev/null > +++ b/cmake/generate_submodule_info.cmake > @@ -0,0 +1,73 @@ ... > + FOREACH(s ${out_list}) > + IF(${counter} EQUAL 0) > + IF (NOT("${s}" MATCHES "Entering '")) does it depend on the locale? > + MESSAGE(FATAL "Unexpected output ${outvar}") > + ENDIF() > + STRING(LENGTH "${s}" slen) > + MATH(EXPR substr_len "${slen} - 11") ... > diff --git a/cmake/generate_sbom.cmake b/cmake/generate_sbom.cmake > --- /dev/null > +++ b/cmake/generate_sbom.cmake > @@ -0,0 +1,240 @@ > +INCLUDE(generate_submodule_info) > +INCLUDE(ExternalProject) > + > +# Remove all matches from a cmake list > +FUNCTION (REMOVE_ALL_MATCHES list_var_name match_str) > + SET(new_list) > + FOREACH(el ${${list_var_name}}) > + IF(NOT("${el}" MATCHES "${match_str}")) > + LIST(APPEND new_list ${el}) > + ENDIF() > + ENDFOREACH() > + SET(${list_var_name} ${new_list} PARENT_SCOPE) > +ENDFUNCTION() you could've used LIST(FILTER ...) instead. (but not LIST(REMOVE_ITEM ...) which is 3.16+) > + > +# Extract user name and repository name from a github URL. > +FUNCTION (EXTRACT_REPO_NAME_AND_USER repo_url repo_name_var repo_user_var) > + IF(repo_url MATCHES "^git@") > + # normalize to https-style URLs > + STRING(REGEX REPLACE "^git@([^:]+):(.*)$" "https://\\1/\\2" repo_url "${repo_url}") I don't think we have that, is it even tested? :) note, I'm not saying it isn't needed, better to have it than not. > + ENDIF() > + # Extract the repository user > + STRING(REGEX REPLACE "https://([^/]+)/([^/]+)/.*" "\\2" repo_user "${repo_url}") > + > + STRING(REGEX REPLACE ".*/([^/]*)$" "\\1" repo_name "${repo_url}") > + STRING(REGEX REPLACE "\\.git$" "" repo_name "${repo_name}") > + > + SET(${repo_name_var} ${repo_name} PARENT_SCOPE) > + SET(${repo_user_var} ${repo_user} PARENT_SCOPE) > +ENDFUNCTION() > + > +# Add a known 3rd party dependency for SBOM generation > +# Currently used for "vendored" (part of our repository) source code we know about > +# such as zlib, as well ExternalProject_Add() projects > +MACRO(ADD_THIRD_PARTY_DEPENDENCY name url tag rev version description) > + LIST(FIND ALL_THIRD_PARTY ${name} idx) > + IF (idx GREATER -1) > + MESSAGE(FATAL_ERROR "${name} is already in ALL_THIRD_PARTY") > + ENDIF() > + SET(${name}_URL ${url}) > + SET(${name}_TAG ${tag}) > + SET(${name}_REVISION ${rev}) > + SET(${name}_DESCRIPTION "${description}") > + SET(${name}_VERSION "${version}") > + LIST(APPEND ALL_THIRD_PARTY ${name}) > +ENDMACRO() > + > +# Match third party component with supplier > +# CyclonDX documentation says it is > +# "The organization that supplied the component. > +# The supplier may often be the manufacturer, but may also be a distributor or repackager." > +# > +# Perhaps it can always be "MariaDB", but security team recommendation is different > +# more towards "author" > +FUNCTION (sbom_get_supplier repo_name repo_user varname) why a function? just curious > + IF("${repo_name_SUPPLIER}") > + SET(${varname} "${repo_name_SUPPLIER}" PARENT_SCOPE) ... > + # It is possible to provide EXTRA_SBOM_DEPENDENCIES > + # and accompanying per-dependency data, to extend generared sbom > + # document. > + # Example below injects an extra "ncurses" dependency using several > + # command line parameters for CMake. > + # -DEXTRA_SBOM_DEPENDENCIES=ncurses > + # -Dncurses_URL=https://github.com/mirror/ncurses > + # -Dncurses_TAG=v6.4 > + # -Dncurses_VERSION=6.4 > + # -Dncurses_DESCRIPTION="A fake extra dependency" > + SET(ALL_THIRD_PARTY ${ALL_SUBMODULES} ${EXTRA_SBOM_DEPENDENCIES}) okay. Though I personally wouldn't use ALL_SUBMODULES and EXTRA_SBOM_DEPENDENCIES and instead would scan GET_CMAKE_PROPERTY(VARS VARIABLES) for *_TAG or something > + > + IF(TARGET ha_connect OR TARGET connect) > + ADD_THIRD_PARTY_DEPENDENCY(minizip > + "https://github.com/zlib-ng/minizip-ng" > + "" 252588f "" > + "Vendored minizip, inside connect storage engine, storage/connect/zip.c et al") did you verify that our file matches the content of the file from that repo at that commit? also, a though - how can we prevent our minigzip.c from being modified without updaing the dependency? Suggestion, check the commit hash, like git log -n1 --format=%h storage/connect/zip.c > + ENDIF() > + > + IF(TARGET libfmt) > + ExternalProject_GET_PROPERTY(libfmt URL) > + STRING(REGEX MATCH "[0-9]+\\.[0-9]+\\.[0-9]+" libmt_TAG "${URL}") > + ADD_THIRD_PARTY_DEPENDENCY(libfmt > + "https://github.com/fmtlib/fmt" > + "${libmt_TAG}" "${libmt_TAG}" "${libmt_TAG}" > + "header only library, used in server") > + ENDIF() I wonder, wouldn't it be simpler and more robust to overwrite or wrap ExternalProject() to do the above automatically? (using the same name or a different one like MyExternalProject) ... Also, I noticed that the server entry gets "purl": "pkg:github//@5b6b23563bf" perhaps it should have a repo name here? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 534dda1234b: MDEV-35522: MariaDB Audit does not detect all DCLs forms when masking password
by Sergei Golubchik 02 Dec '24

02 Dec '24

Hi, Oleksandr, Ok to push. But do you no longer delete server_audit.log after the test? On Dec 02, Oleksandr Byelkin wrote: > revision-id: 534dda1234b (mariadb-10.5.27-64-g534dda1234b) > parent(s): fdb6db6b47f > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2024-11-29 11:45:01 +0100 > message: > > MDEV-35522: MariaDB Audit does not detect all DCLs forms when masking password > > 1. skip OR REPLACE > (to make it filter_query_type made recursive) > > 2. skip SET STATEMENT ... FOR before checking > statements with passwords > > diff --git a/mysql-test/suite/plugins/t/server_audit_pwd_mask.test b/mysql-test/suite/plugins/t/server_audit_pwd_mask.test > --- a/mysql-test/suite/plugins/t/server_audit_pwd_mask.test > +++ b/mysql-test/suite/plugins/t/server_audit_pwd_mask.test > --echo # cleaunup > +DROP SERVER s1; > DROP USER u1; > DROP USER u2; > set global server_audit_logging=off; > ---remove_file $SEARCH_FILE > +##--remove_file $SEARCH_FILE why? > --disable_warnings > UNINSTALL PLUGIN ed25519; > UNINSTALL PLUGIN server_audit; > --enable_warnings Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: ff3d06f6601: MDEV-22217 Make OS character sets "utf8" and "utf-8" map to MariaDB character set "utf8mb4"
by Sergei Golubchik 16 Nov '24

16 Nov '24

Hi, Alexander, Of course, there can be no comments about the code change here. But the question is, is it something we want to do? 1. Is utf8/utf-8 locale indeed 4-byte utf8? 2. Why is hard-coding to utf8mb4 better than having it dependent on UTF8_IS_UTF8MB3 ? On Nov 16, Alexander Barkov wrote: > revision-id: ff3d06f6601 (mariadb-11.6.1-44-gff3d06f6601) > parent(s): 4b6922a315f > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-10-28 14:11:40 +0400 > message: > > MDEV-22217 Make OS character sets "utf8" and "utf-8" map to MariaDB character set "utf8mb4" > > diff --git a/mysql-test/main/mysql_locale_posix.test b/mysql-test/main/mysql_locale_posix.test > index c047b7b1ae9..b37d04baca2 100644 > --- a/mysql-test/main/mysql_locale_posix.test > +++ b/mysql-test/main/mysql_locale_posix.test > @@ -203,3 +203,20 @@ DROP DATABASE `Ã³`; > + > +--exec LC_ALL=en_US.utf8 $MYSQL --default-character-set=auto test -e "SELECT @@character_set_client" > +--exec LC_ALL=en_US.utf8 $MYSQL --default-character-set=auto test -e "SELECT @@character_set_connection" > +--exec LC_ALL=en_US.utf8 $MYSQL --default-character-set=auto test -e "SELECT @@character_set_results" may be do it in one SELECT? or in SHOW VARIABLES LIKE 'character_set%' ? to have it a tiny bit faster? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

2024 Q4 release part 2 (11.6.2, 11.7,1)
by Daniel Black 12 Nov '24

12 Nov '24

All, The 11.6.2 and 11.7.1 releases are now built: * 11.6.2 d8dad8c3b54cd09fefce7bc3b9749f427eed9709 (GA) * 11.7.1 74331a48dd3dadfe5041f3440b81a7183e9fd4a8 release notes still work in progress: * https://mariadb.com/kb/en/mariadb-11-6-2-release-notes/ * 11.7.1 still being prepared but will be 11.6.2 like + the features of 11.7.0 release notes + Vector search, + bugs fixes since 11.7.0. Both are rolling releases and once GA, like 11.6.2, will be the last of the release series. containers for testing are available: * quay.io/mariadb-foundation/mariadb-devel:11.6 * quay.io/mariadb-foundation/mariadb-devel:11.7 Build packages for RPM/Deb distros repo/source files: https://ci.mariadb.org/ * 11.6.2 - bb-11.6-release-latest* * 11.7.1 - bb-11.7-release-latest* Sources: * https://ci.mariadb.org/51755/ (11.6.2) * https://ci.mariadb.org/51770/ (11.7.1) Now begins the process of loading packages to mirrors and Docker Official Images releases.

1 0

Contribute to MariaDB compatibility in open source projects?
by Robert Silén 06 Nov '24

06 Nov '24

All, We have been working on community outreach as described in an earlier blog Improving MariaDB support in open source projects <https://mariadb.org/improving-mariadb-support-in-open-source-projects/>. Many projects have made efforts to improve their MariaDB compatibility with or without our help, and also documented compatibility more visibly in READMEs and documentation. However, we at the Foundation don't have the knowhow or resources to help out everybody, so I thought I'd share a list of some issues. Would somebody like to contribute to any of them? Any comment on the issues, or code contributions to them would be much appreciated. Thank you. Do share any thoughts or actions on this. Robert Silén, community advocacy at MariaDB Foundation robert.silen(a)mariadb.org - https://github.com/SOCI/soci/issues/1150: add MariaDB to README - https://github.com/tinode/chat/pull/924: Help set up CI for MariaDB (and MySQL) - https://github.com/inspircd/inspircd-docs/issues/239: tls and srv not available in MariaDB - https://github.com/umami-software/umami/discussions/2943: BIN_TO_UUID() not available in MariaDB (MDEV-15854 <https://jira.mariadb.org/browse/MDEV-15854>) - https://github.com/diesel-rs/diesel/issues/1882: Create a distinct backend for MariaDB - https://github.com/grafana/grafana: Create a dedicated DataStore for MariaDB - https://github.com/drizzle-team/drizzle-orm/pull/1692: WIP: Add support for Maria DB (core API implementation) - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1079753: Create mariadb specific dbconfig-mariadb - https://core.trac.wordpress.org/ticket/60037: Fix version checking in next release - https://gitlab.com/zoph-project/zoph/-/issues/267: give a thumbs up to their own addition of MariaDB to documentation (preferably recommend MariaDB)

2 1

Re: e8021aaf28e: auth: refactor MYSQL_AUTHENTICATION_INTERFACE_VERSION naming
by Sergei Golubchik 28 Oct '24

28 Oct '24

Hi, Nikita, This looks very confusing, this is not how versions work. What are you trying to achieve by that? On Oct 07, Nikita Malyavin wrote: > revision-id: e8021aaf28e (mariadb-11.6.1-9-ge8021aaf28e) > parent(s): a5b80531fbb > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-10-04 00:16:17 +0200 > message: > > auth: refactor MYSQL_AUTHENTICATION_INTERFACE_VERSION naming > > Rename MYSQL_AUTHENTICATION_INTERFACE_VERSION to > MYSQL_AUTH_INTERFACE_VERSION_2_02. > This is the current latest version, and it will be fixed to all auth plugins > for current implementation. > MYSQL_AUTHENTICATION_INTERFACE_VERSION will stay for compatibility with 3-rd > party auth plugins and will be fixed to V. 2.02. > Add MYSQL_AUTHENTICATION_INTERFACE_VERSION_LAST that will be updated as needed. > Add MYSQL_AUTH_INTERFACE_MAJOR_MASK to identify major version. > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 5

Re: b17308e254a: MDEV-34854 Parsec sends garbage when using an empty password
by Sergei Golubchik 28 Oct '24

28 Oct '24

Hi, Nikita, On Oct 07, Nikita Malyavin wrote: > revision-id: b17308e254a (mariadb-11.6.1-10-gb17308e254a) > parent(s): e8021aaf28e > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-10-04 01:33:47 +0200 > message: > > MDEV-34854 Parsec sends garbage when using an empty password > > When an empty password is set, the server doesn't call > st_mysql_auth::hash_password and leaves > MYSQL_SERVER_AUTH_INFO::auth_string empty. > > Fix: > generate hashes for empty passwords as well. This breaks some auth > plugins, so we increment interface version and do it only from Auth V. > 2.03. > > Some empty passwords could be already stored with no though. The user "with no though" ? > will have to call SET PASSWORD once again, anyway the authentication > wouldn't have worked for such password. ok, I presume you mean ed25519 only, because mysql_native_password worked with an empty password and it generates an empty hash for it. > diff --git a/mysql-test/suite/plugins/r/parsec.result b/mysql-test/suite/plugins/r/parsec.result > index 512c066e2d7..b7e3537af29 100644 > --- a/mysql-test/suite/plugins/r/parsec.result > +++ b/mysql-test/suite/plugins/r/parsec.result ... let's add ed25519 test too, for completeness. > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 2722ea2ea19..ba05a5656c5 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2402,7 +2402,10 @@ static int set_user_auth(THD *thd, const LEX_CSTRING &user, > res= ER_NOT_VALID_PASSWORD; > goto end; > } > - if (pwtext.length) > + > + // Starting from version 2.03 we also generate hash for empty passwords. > + if ((info->interface_version >= MYSQL_AUTH_INTERFACE_VERSION_2_03 I don't understand this MYSQL_AUTH_INTERFACE_VERSION_2_03 thing. First, again, that's not how a version is supposed to work. Second, this empty-password change isn't a change in the API. You can simply start calling hash_password() for empty passwords and it'll just work. I've tried :) > + && pwtext.str) || pwtext.length) > { > if (info->hash_password) > { Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

2024 Q4 release
by Daniel Black 27 Oct '24

27 Oct '24

All, Just letting you know that we're in the progress of getting the releases ready. The issue where this is tracked is: * https://jira.mariadb.org/browse/MDEV-35030 Branches being released are at the tips of these branches: bb-10.5-release (LTS) bb-10.6-release (LTS) bb-10.11-release (LTS) bb-11.2-release bb-11.4-release (LTS) bb-11.6-release bb-11.7-release So far the code is merged up to bb-11.4-release. Later branches will be updated, check history before using. There are two blocking bugs on bb-11.2-release onwards affecting compilation of the columnstore and connect storage engine. In addition to compiling the source yourselves; Containers: quay.ion/mariadb-foundation/mariadb-devel:{release} list: https://quay.io/repository/mariadb-foundation/mariadb-devel?tab=tags note: * 11.4 is from the 11.4 branch rather than bb-11.4-release, * 11.2 didn't build due to blocker bugs). Repos for testing: https://ci.mariadb.org/ contains for deb and rpm distros a repo file in the form: bb-{version}-release-latest-{arch}-{distro}-{version}-{deb|rpm}-autobake.{sources|repo} Please report any serious regressions with a Blocker priority bug, and mark this issue blocking MDEV-35030.

1 0

Re: 692ca003f8c: MDEV-33658 2/2 Cannot add a foreign key on a table with a matching long UNIQUE
by Sergei Golubchik 23 Oct '24

23 Oct '24

Hi, Nikita, On Oct 18, Nikita Malyavin wrote: > revision-id: 692ca003f8c (mariadb-10.5.26-169-g692ca003f8c) > parent(s): ba988513ec7 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-10-13 19:33:14 +0200 > message: > > MDEV-33658 2/2 Cannot add a foreign key on a table with a matching long UNIQUE > > Cannot add a foreign key on a table with a long UNIQUE multi-column index, that > contains a foreign key as a prefix. > > Check that index algorithms match during the "generated" keys duplicate removal. > > diff --git a/mysql-test/main/long_unique_bugs.test b/mysql-test/main/long_unique_bugs.test > index 2f85328bda3..08b49b6c5da 100644 > --- a/mysql-test/main/long_unique_bugs.test > +++ b/mysql-test/main/long_unique_bugs.test > @@ -706,4 +706,19 @@ alter table t1 enable keys; > insert into t1 values (2); > drop table t1; > > +--echo # > +--echo # Cannot add a foreign key on a table with a long UNIQUE multi-column > +--echo # index, that contains a foreign key as a prefix. > +--echo # > +create table a (id int(19) primary key) engine = innodb; > +create table b (id int primary key, > + long_text varchar(1000), > + constraint unique_b unique key (id, long_text) > + ) engine = innodb default charset utf8mb4; > + > +alter table b add constraint b_fk_ida foreign key (id) references a (id); > +show create table b; > +drop table b; > +drop table a; Two problems with this test case. A minor one - it doesn't start with the --echo # MDEV-33658 Cannot add a foreign key on a table with a matching long UNIQUE line. Please add it. A more serious problem - it doesn't fail on a vanilla 10.5. The original set of SQL commands from MDEV-33658 fails with "Foreign key constraint is incorrectly formed", but your test doesn't. You might've oversimplified it. Please, fix the test case to fail on vanilla 10.5 with the above error. > + > --echo # End of 10.5 tests > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 99ba9e3587b..3d01b9d5584 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -203,7 +203,7 @@ Foreign_key::Foreign_key(const Foreign_key &rhs, MEM_ROOT *mem_root) > > /* > Test if a foreign key (= generated key) is a prefix of the given key > - (ignoring key name, key type and order of columns) > + (ignoring key name, key type and order of columns, but minding the algorithm) it doesn't ignore the order of columns and "key type" is basically algorithm, so the above can simply be "(ignoring key name)" > > NOTES: > This is only used to test if an index for a FOREIGN KEY exists > @@ -218,6 +218,17 @@ Foreign_key::Foreign_key(const Foreign_key &rhs, MEM_ROOT *mem_root) > > bool is_foreign_key_prefix(Key *a, Key *b) > { > + ha_key_alg a_alg= a->key_create_info.algorithm; > + ha_key_alg b_alg= b->key_create_info.algorithm; > + > + bool is_a_btree= a_alg == HA_KEY_ALG_BTREE || a_alg == HA_KEY_ALG_UNDEF; > + bool is_b_btree= b_alg == HA_KEY_ALG_BTREE || b_alg == HA_KEY_ALG_UNDEF; > + > + // Either the algorithms are total match, or they both are btree > + // in order to be able to prefix > + if (a_alg != b_alg && !(is_a_btree && is_b_btree)) > + return false; I don't see why algorithms can be not b-tree, if they match. What use case did you have in mind? > + > /* Ensure that 'a' is the generated key */ > if (a->generated) > { > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Update documentation for non-blocking client library
by Kristian Nielsen 22 Oct '24

22 Oct '24

Hi Georg, Following MDEV-34859, I wrote a small update for the documentation on the new cmake option -DWITH_BOOST_CONTEXT=ON cmake option, patch included below. However, I am not sure how I can include it in the documentation? Feel free to just take it yourself, or let me know a preferred way to submit it. - Kristian. diff --git a/configuration_options.md b/configuration_options.md index 6e8da57..726b4f8 100644 --- a/configuration_options.md +++ b/configuration_options.md @@ -27,6 +27,11 @@ If you want to use a different generator, e.g. for nmake on Windows, you need to |WITH_OPENSSL|ON| Possible values are ON or OFF. Not supported anymore since Connector/C 3.0| |WITH_SSL|SCHANNEL (windows), otherwise OPENSSL|Specifies type of TLS/SSL library. E.g. GNUTLS, OPENSSL or SCHANNEL (Windows only). OFF disables TLS/SSL functionality| +### Non-blocking client library options +| Option | Default | Description | +|-|-|- +|WITH_BOOST_CONTEXT|OFF| Use `boost::context` instead of `ucontext` for the non-blocking client API. Can be used to build the non-blocking API on platforms that do not have `ucontext`. Note that on x86_64 (aka amd64), i386, and aarch64 (aka arm64), a native implementation is always used over `ucontext` or `boost::context`. (Added in 3.3.12)| + ### Client plugins Client plugins can be configured as dynamic plugins (DYNAMIC) or built-in plugins (STATIC) by specifying the plugin name followed by suffix `_PLUGIN_TYPE` as key, and `DYNAMIC` or `STATIC` as value.

2 1

Community request: audit the PARSEC plugin
by Andrew Hutchings 16 Oct '24

16 Oct '24

Hi all, There is a new authentication plugin called PARSEC that is set to become the default authentication plugin in MariaDB soon (MDEV-32618 and MDEV-12320). The current documentation for it can be found here: https://mariadb.com/kb/en/authentication-plugin-parsec/ I personally think that using a single salt and sending it to the client could be a problem. I do not work in the security industry and the documentation has a few errors in it and pieces missing, so I'm unclear how certain things work. I think the ext-salt requirements will likely add complications to load balancing scenarios and replication. This is a request for anyone in the community to please audit this plugin before it becomes the default. Once it is the default, it will be difficult to change it. Kind Regards -- Andrew (LinuxJedi) Hutchings Chief Contributions Officer MariaDB Foundation

2 2

Re: 778c67f7af0: MDEV-18151 Skipped error returning for GRANT/SET PASSWORD
by Sergei Golubchik 09 Oct '24

09 Oct '24

Hi, Oleksandr, On Oct 09, Oleksandr Byelkin wrote: > revision-id: 778c67f7af0 (mariadb-10.5.26-154-g778c67f7af0) > parent(s): e9c999caf48 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2024-10-04 09:28:46 +0200 > message: > > MDEV-18151 Skipped error returning for GRANT/SET PASSWORD > > Make error issueing for GRANT and SET PASSWORD the same. typo. "issuing" > Report errors wich were skipped before. > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2370,6 +2369,21 @@ static int set_user_auth(THD *thd, const LEX_CSTRING &user, > > res= 0; > end: > + switch(res) > + { > + case ER_OUTOFMEMORY: // should be reported by my_malloc > + case ER_NOT_VALID_PASSWORD: // should be reported by plugin > + case 0: > + break; > + case ER_SET_PASSWORD_AUTH_PLUGIN: > + my_error(res, MYF(0), plugin_name); > + break; > + case ER_PASSWD_LENGTH: > + my_error(ER_PASSWD_LENGTH, MYF(0), SCRAMBLED_PASSWORD_CHAR_LENGTH); this is also reported by the plugin. The caller doesn't know what the correct length is, it's not SCRAMBLED_PASSWORD_CHAR_LENGTH, this value is different for different plugins. > + break; > + default: > + DBUG_ASSERT(0); > + } > if (unlock_plugin) > plugin_unlock(thd, plugin); > return res; > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

PRs with one approval pending the second
by Otto Kekäläinen 04 Oct '24

04 Oct '24

Hi *all* developers, Hopefully someone else than Andrew or Sergei replies as a project of the size of MariaDB should have multiple developers who are capable of reviewing PRs, and not overload a few. These PRs already have one approval, but would need a second to be merged. If you feel you are knowledgeable in the domain of the PRs, please help out by reviewing them. Thanks! https://github.com/MariaDB/server/pull/3225: MDEV-29955: add ${PCRE_LIBRARY_PATH} to link directories https://github.com/MariaDB/server/pull/3540: Add MTRs to further test coverage around TLS functionality

1 0

PRs with two approvals but not merged
by Otto Kekäläinen 04 Oct '24

04 Oct '24

Hi, Here are two PRs that have already two approvals, yet they have not been merged for many weeks. Can you please state explicitly on the PR as a comment what the submitter needs to do, or if there is nothing to do, just merge it? * https://github.com/MariaDB/server/pull/3208 * https://github.com/MariaDB/server/pull/2556 Thanks

3 3

More extensive major version upgrade testing
by Otto Kekäläinen 04 Oct '24

04 Oct '24

Hi, If any dev here is interested in the topic of ensuring the major version upgrades (and essentially running mariadb-upgrade) continues to work flawlessly and regressions are less likely to surface unnoticed, please review and consider adding your approvals to https://github.com/MariaDB/server/pull/3503 and share feedback on WIP https://github.com/MariaDB/server/pull/3563. Thanks!

2 1

Re: [PATCH] MDEV-28413 System versioning is incorrect in Spider
by Yuchen Pei 03 Oct '24

03 Oct '24

Hi Aleksey, Thanks for the patch. Please see below for some initial review comments. There are some todos for myself noted below which I will look into tomorrow. > From b4465fc6b51dfc4e645569201e6026ec6405367b Mon Sep 17 00:00:00 2001 > From: Aleksey Midenkov <midenok(a)gmail.com> > Date: Mon, 30 Sep 2024 22:26:51 +0300 > Subject: [PATCH] MDEV-28413 System versioning is incorrect in Spider > Basic functioning of system versioning in Spider: UPDATE, DELETE, > DELETE HISTORY, AS OF, FROM .. TO, BETWEEN .. AND. > More testing should be done for: > LOAD DATA, joins, derived tables, multi-update/delete and what else? > Note that we are still using HA_CAN_DIRECT_UPDATE_AND_DELETE, but not > for periods in DELETE (fwiw there is condition > !table_list->has_period(), but it is missing in UDPATE). > --- > .../mysql-test/spider/r/versioning.result | 142 ++++++++++++++++ > .../mysql-test/spider/t/versioning.test | 151 ++++++++++++++++++ > storage/spider/spd_db_conn.cc | 19 ++- > storage/spider/spd_db_include.h | 1 + > storage/spider/spd_db_mysql.cc | 25 ++- > 5 files changed, 335 insertions(+), 3 deletions(-) > create mode 100644 storage/spider/mysql-test/spider/r/versioning.result > create mode 100644 storage/spider/mysql-test/spider/t/versioning.test > [... 149 lines elided] > diff --git a/storage/spider/mysql-test/spider/t/versioning.test b/storage/spider/mysql-test/spider/t/versioning.test > new file mode 100644 > index 00000000000..6093169bff4 > --- /dev/null > +++ b/storage/spider/mysql-test/spider/t/versioning.test > @@ -0,0 +1,151 @@ > +--source include/have_innodb.inc > +--source suite/versioning/common.inc > + > +install plugin spider soname 'ha_spider'; For init the spider plugin in spider tests we commonly do --disable_query_log --disable_result_log --source ../t/test_init.inc --enable_result_log --enable_query_log > +set spider_same_server_link= on; > + > +--replace_result $MASTER_1_MYPORT MASTER_1_MYPORT > +eval create server s foreign data wrapper mysql options > +(host '127.0.0.1', database 'test', user 'root', port $MASTER_1_MYPORT); Use evalp instead of eval and you can save the --replace_result line > + > + > +create table t1 ( > + x int, y int > +) with system versioning; > + > +create table t2 ( > + x int, y int, > + row_start timestamp(6) as row start invisible, > + row_end timestamp(6) as row end invisible, > + period for system_time (row_start, row_end) > +) with system versioning; > + > +create or replace table t1_sp ( > + x int, y int) > +engine=spider comment='wrapper "mysql", srv "s", table "t1"' > +with system versioning; > + > +create or replace table t2_sp ( > + x int, y int, > + row_start timestamp(6) as row start invisible, > + row_end timestamp(6) as row end invisible, > + period for system_time (row_start, row_end)) > +engine=spider comment='wrapper "mysql", srv "s", table "t2"' > +with system versioning; > + > +--echo # Timestamps are not propagated (subject of MDEV-16546, but not for implicit system fields) > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); I tried executing directly using select spider_direct_sql('set timestamp= unix_timestamp("2000-01-01 00:00:00");', '', 'srv "s"'); But then I get extra rows in subsequent select queries and I think it is caused by the timezones. It could be a separate task to fix this, but we should warn users about it. > +insert into t1_sp values (1, 1), (2, 2); It may be obvious to you, but I find the timestamps at times hard to keep track of, so I suggest adding a comment to each DML statement indicating the time, as well as the select row_start / row_end statements. For example, change the line above to insert into t1_sp values (1, 1), (2, 2); # T1 > +insert into t2_sp values (1, 1), (2, 2); and change this line to insert into t2_sp values (1, 1), (2, 2); # T2 > +--sleep 0.01 > + > +set timestamp= unix_timestamp('2000-01-01 00:11:11'); > +--echo # check dml generating history > +delete from t1_sp where x = 1; add # T3 > +--sleep 0.01 > +delete from t2_sp where x = 1; # T4 > +--sleep 0.01 > + > +select row_end into @t1 from t2 for system_time all where x = 1; And so we can annotate this line with the timestamp above, like so: select row_end into @t1 from t2 for system_time all where x = 1; # @t1 = T4 > + > +update t1_sp set y= y + 1; # T5 > +select row_start into @t2 from t1 where x = 2; # @t2 = T5 > +--sleep 0.01 > +update t2_sp set y= y + 1; # T6 With the annotations I suggested, results from the following block of selects becomes easier to understand. > + > +select * from t1_sp; > +select * from t2_sp; > +select * from t1_sp for system_time as of timestamp @t1; > +select * from t2_sp for system_time as of timestamp @t1; > +select * from t1_sp for system_time as of timestamp @t2; > +select * from t2_sp for system_time as of timestamp @t2; > +select * from t1_sp for system_time from timestamp @t1 to timestamp @t2; > +select * from t2_sp for system_time from timestamp @t1 to timestamp @t2; > +select * from t1_sp for system_time between timestamp @t1 and timestamp @t2; > +select * from t2_sp for system_time between timestamp @t1 and timestamp @t2; > + > +--echo # spider cannot call functions, it fails in spider_create_group_by_handler() > + # at: > + # > + # 1555 if (spider_db_print_item_type(item, null, spider, null, null, 0, > + # 1556 roop_count, true, fields_arg)) > + # 1557 { > + # 1558 dbug_print("info",("spider dbton_id=%d can't create select", roop_count)); > + # 1559 spider_clear_bit(dbton_bitmap, roop_count); > + # 1560 keep_going = false; > + # 1561 break; > + # 1562 } > + # This is fine. The Spider group-by-handler (GBH) is not meant to handle every situation. Either remove this block starting from the --echo, or make it more succinct by replacing it with something like (Note that GBH supports certain functions like trim): # The spider group-by-handler does not support the check_row function > +--echo # but somehow it doesn't fail the query and continues without FOR SYSTEM_TIME ALL clause > +select *, check_row(row_start, row_end) from t1_sp for system_time all order by row_end, y, x; > +select *, check_row(row_start, row_end) from t2_sp for system_time all order by row_end, y, x; I will check why the execution without the group-by-handler results in wrong results. > +--echo # SELECT from base table works as usual: > +select *, check_row(row_start, row_end) from t1 for system_time all order by row_end, y, x; > +select *, check_row(row_start, row_end) from t2 for system_time all order by row_end, y, x; > + > +--echo # here it works nice (append_table_list() has the chance to add FOR SYSTEM_TIME ALL) > + > +# Usage: SHOW_TIMESTAMPS=1 mtr versioning.spider Shouldn't the test name be spider.versioning? > +if ($SHOW_TIMESTAMPS) > +{ > + select @t1, @t2; > +} > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t1_sp for system_time all order by row_end, y, x; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t2_sp for system_time all order by row_end, y, x; > + > +delete history from t1_sp before system_time @t1; > +delete history from t2_sp before system_time @t1; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t1_sp for system_time all order by row_end, y, x; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t2_sp for system_time all order by row_end, y, x; > + > +select * from t1 for system_time all; > +select * from t2 for system_time all; > + > +delete history from t1_sp; > +delete history from t2_sp; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t1_sp for system_time all order by row_end, y, x; > + > +if (!$SHOW_TIMESTAMPS) > +{ > +--replace_regex /\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d.\d\d\d\d\d\d/TIMESTAMP/ > +} > +select *, row_start, row_end from t2_sp for system_time all order by row_end, y, x; > + > +select * from t1 for system_time all; > +select * from t2 for system_time all; > + > +drop tables t1, t2, t1_sp, t2_sp; > +set timestamp= default; > +--source suite/versioning/common_finish.inc > + > +uninstall plugin spider; > +drop table mysql.spider_link_failed_log, mysql.spider_link_mon_servers, mysql.spider_tables, mysql.spider_table_crd, mysql.spider_table_position_for_recovery, mysql.spider_table_sts, mysql.spider_xa, mysql.spider_xa_failed_log, mysql.spider_xa_member; > +drop function spider_direct_sql; > +drop function spider_bg_direct_sql; > +drop function spider_ping_table; > +drop function spider_copy_tables; > +drop function spider_flush_table_mon_cache; For deinit the spider plugin in spider tests we commonly do --disable_query_log --disable_result_log --source ../t/test_deinit.inc --enable_result_log --enable_query_log Please replace the lines from "uninstall plugin spider;" to the end of the test with this block. > diff --git a/storage/spider/spd_db_conn.cc b/storage/spider/spd_db_conn.cc > index 8750e87237e..896fce8adf5 100644 > --- a/storage/spider/spd_db_conn.cc > +++ b/storage/spider/spd_db_conn.cc > @@ -1515,9 +1515,8 @@ int spider_db_append_key_where_internal( > ) { > SPIDER_RESULT_LIST *result_list = &spider->result_list; > SPIDER_SHARE *share = spider->share; > -#ifndef DBUG_OFF > TABLE *table = spider->get_table(); > -#endif Why remove the #ifndef and #endif? > [... 99 lines elided] Best, Yuchen

1 1

[Maria-developers] MariaDB // Temporal tables
by Robert Palm 02 Oct '24

02 Oct '24

2 2

Help wanted: Encryption Plugin
by ssss.aaa2002＠gmail.com 20 Sep '24

20 Sep '24

Hello I wonder if anyone can help me out for this case I'm trying to build an encryption plugin for mariaDB and I built this https://gist.github.com/Mahmoudgalalz/ae1de7a57f8b9f37056efc2018575ce4 I managed to compile it, but I'm running into an issue when I'm trying to use encrypted option MariaDB [test]> CREATE TABLE test_encrypted (id INT PRIMARY KEY, sensitive_data VARCHAR(255)) ENGINE=InnoDB ENCRYPTED=YES; ERROR 2013 (HY000): Lost connection to server during query MariaDB [test]> CREATE TABLE t1 (a VARCHAR(8)) ENGINE=InnoDB ENCRYPTED=YES ENCRYPTION_KEY_ID=1; ERROR 2006 (HY000): Server has gone away No connection. Trying to reconnect... Connection id: 31 Current database: test ERROR 2013 (HY000): Lost connection to server during query so if anyone can help me out to figure out what is wrong and should I correct or consider would be very much appreciated.

2 4

Help wanted: Encryption Plugin
by ssss.aaa2002＠gmail.com 19 Sep '24

19 Sep '24

Hello I wonder if anyone can help me out for this case I'm trying to build an encryption plugin for mariaDB and I built this https://gist.github.com/Mahmoudgalalz/ae1de7a57f8b9f37056efc2018575ce4 I managed to compile it, but I'm running into an issue when I'm trying to use encrypted option MariaDB [test]> CREATE TABLE test_encrypted (id INT PRIMARY KEY, sensitive_data VARCHAR(255)) ENGINE=InnoDB ENCRYPTED=YES; ERROR 2013 (HY000): Lost connection to server during query MariaDB [test]> CREATE TABLE t1 (a VARCHAR(8)) ENGINE=InnoDB ENCRYPTED=YES ENCRYPTION_KEY_ID=1; ERROR 2006 (HY000): Server has gone away No connection. Trying to reconnect... Connection id: 31 Current database: test ERROR 2013 (HY000): Lost connection to server during query so if anyone can help me out to figure out what is wrong and should I correct or consider would be very much appreciated.

1 0

Re: [PATCH] MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction
by Kristian Nielsen 10 Sep '24

10 Sep '24

> From: Libing Song <anders.slb(a)alibaba-inc.com> > > Description > =========== > When a transaction commits, it copies the binlog events from > binlog cache to binlog file. Very large transactions > (eg. gigabytes) can stall other transactions for a long time > because the data is copied while holding LOCK_log, which blocks > other commits from binlogging. Hi Libing Song, Here is my full review of the patch for MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction. First let me say that thanks, reviewing this patch was a pleasant surprise. The binlog code that you are integrating this new feature in is complex to work with, but overall I think you have managed to do it a pretty clean way. That is great work, and I think a lot of effort must have gone into getting it this way. I have a number of comments and requests for changes below, so I wanted to give this overall good impression up-front. Also thanks to Brandon for doing first review. My apologies if I am repeating something already discussed, but Github pull request reviews of non-trivial patches are almost impossible to read, with reviews being split in fragments, patches disappearing after force-pushes, etc. The root cause for the problem being addressed by the patch is a fundamental limitation in the current binlog format. It requires all transactions to be binlogged as a single event group, using consecutive bytes in a single binlog file. Thus, while writing the large transaction to the binlog during commit, all other transaction commits will have to wait before they can commit. Eventually, we should move towards an enhanced binlog format that doesn't have this limitation, so that transactions can be partially binlogged during execution, avoiding this bottlenect. There are a number of other problems caused by this limitation that are not addressed by this patch. Changing the binlog format is a complex task not easily done, so this patch should be seen as a temporary solution to one specific bottleneck, until if/when this underlying limitation can hopefully be removed. First some overall comments: As Brandon also remarked, the name "binlog_free_flush" does not describe the feature very clearly. I would suggest to use the term "rotate" in the name, as the concept of binlog rotation is well known in MariaDB replication, and the patch effectively performs large commits by doing it as a binlog rotation instead of a simple write to the existing binlog file. So I suggest to name the option --binlog-commit-by-rotate-threshold. And I suggest in the code (names and comments) to use the phrase "rotate_by_commit" instead of "free_flush". The patch enables the feature by default, with a threshold of 128MB. We need to consider if that is appropriate, or if the feature should be disabled by default. Reasons for disabling by default: - The feature causes some binlog files to be cut short of the configured --max-binlog-size, which will be a surprise to users. - The feature adds an overhead of 4096 pre-allocated bytes at the start of every trx cache (though the overhead is minimal). - A default of 128MB is somewhat arbitrary. - Releasing this feature disabled by default is somewhat less risky wrt. regressions for users that upgrade. Reasons to enable by default: - Most users will otherwise not know the feature exists, and will thus not benefit from this work. - It will get much less testing "in the wild". So the decision will be a compromise between these two. If I had to make the choice, I would default to this feature being disabled. But I'm open to hear opinions the other way. The patch calls the trx_group_commit_leader() function. That is ackward, as commit-by-rotate does not actually participate in group commit. But the problem is with the existing function trx_group_commit_leader(), which does too much in a single function. So please do a small refactoring of trx_group_commit_leader() and keep only the first part of that function that fetches the group commit queue. Move the second part to a new function trx_group_commit_with_engines() that is called by trx_group_commit_leader() passing the "queue" and "last_in_queue" values. And assert in trx_group_commit_with_engines() that LOCK_log is held on entry and comment that this function will release the lock. Then you can from your Binlog_free_flush::commit() function call into trx_group_commit_with_engines() directly, avoiding the need for adding a true/false flag, and also avoiding that the trx_group_commit_leader() function is sometimes called with LOCK_log held and sometimes not. Same situation with MYSQL_BIN_LOG::write_transaction_to_binlog_events(), this function is also doing too much and makes integrating your code ackward. Please refactor the MYSQL_BIN_LOG::write_transaction_to_binlog_events() function something like this: bool MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) { if (binlog_commit_by_rotate.should_commit_by_rotate(entry) && binlog_commit_by_rotate.commit(entry)) { /* Commit done by renaming the trx/stmt cache. */ } else { if (write_transaction_with_group_commit(entry)) return true; /* Error */ } if (unlikely(entry->error)) { write_transaction_handle_error(entry); return 1; } return 0; } Introducing two new functions: bool write_transaction_with_group_commit(group_commit_entry *entry) This function contains the code from write_transaction_to_binlog_events() up to and including this: if (likely(!entry->error)) return entry->thd->wait_for_prior_commit(); You don't need the wait_for_prior_commit() in the commit-by-rotate case, as you already did this in Binlog_free_flush::commit()). You also don't need to handle the (!opt_optimize_thread_scheduling) case for commit-by-rotate, this is old code that is being deprecated. Just add a condition in should_free_flush() to skip if opt_optimize_thread_scheduling is set. Other new function: void write_transaction_handle_error(group_commit_entry *entry) This function contains the remainder of the write_transaction_to_binlog_events() function starting from "switch (entry->error) ...". This way, the commit-by-rotate case is cleanly separated from the normal group commit case and the code of write_transaction_to_binlog_events() becomes much cleaner. Now follows more detailed comments on specific parts of the patch: > diff --git a/sql/log.cc b/sql/log.cc > index 34f9ad745fc..3dc57b21c05 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -163,6 +163,111 @@ static SHOW_VAR binlog_status_vars_detail[]= > + void set_reserved_bytes(uint32 header_len) > + { > + // Gtid event length > + header_len+= LOG_EVENT_HEADER_LEN + Gtid_log_event::max_data_length + > + BINLOG_CHECKSUM_LEN; > + header_len= header_len - (header_len % IO_SIZE) + IO_SIZE; This has an off-by-one error, it will round up IO_SIZE to 2*IO_SIZE. Use eg. header_len = (header_len + (IO_SIZE - 1)) & ~(IO_SIZE - 1) You can use static_assert((IO_SIZE & (IO_SIZE - 1)) == 0, "power of two") if you want to catch if anyone would ever change IO_SIZE to be not a power of two. +private: + Binlog_free_flush &operator=(const Binlog_free_flush &); + Binlog_free_flush(const Binlog_free_flush &); Add before this a comment "singleton object, prevent copying by mistake". > + char m_cache_dir[FN_REFLEN]; This member appears to be unused? > + /** The cache_data which will be renamed to binlog. */ > + binlog_cache_data *m_cache_data{nullptr}; Also appears unused. Maybe add a comment that m_entry and m_replaced are protected by LOCK_log? > +static Binlog_free_flush binlog_free_flush; > +ulonglong opt_binlog_free_flush_threshold= 10 * 1024 * 1024; This initializes the value to 10 MB, but the default value in sys_vars.cc is 128MB, so that seems inconsistent. > @@ -4126,8 +4238,7 @@ bool MYSQL_BIN_LOG::open(const char *log_name, > /* Notify the io thread that binlog is rotated to a new file */ > if (is_relay_log) > signal_relay_log_update(); > - else > - update_binlog_end_pos(); > + Why do you remove update_binlog_end_pos() here? Doesn't the binlog dump threads need this update? > @@ -8703,7 +8821,16 @@ MYSQL_BIN_LOG::queue_for_group_commit(group_commit_entry *orig_entry) > bool > MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) > { > - int is_leader= queue_for_group_commit(entry); > + int is_leader; > + > + if (binlog_free_flush.should_free_flush(entry) && > + binlog_free_flush.commit(entry)) > + { > + is_leader= 1; > + goto commit; > + } > + > + is_leader= queue_for_group_commit(entry); As described above, re-factor the write_transaction_to_binlog_events() function to avoid this assignment of is_leader and the goto. > @@ -8863,6 +8992,16 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) > uint64 commit_id; > DBUG_ENTER("MYSQL_BIN_LOG::trx_group_commit_leader"); > > + /* > + When move a binlog cache to a binlog file, the transaction itself is > + a group. > + */ > + if (unlikely(is_free_flush)) > + { > + last_in_queue= leader; > + queue= leader; > + } > + else As described above, re-factor trx_group_commit_leader() to avoid this check in trx_group_commit_leader(), instead calling directly into the relevant code now in trx_group_commit_with_engines(). > +inline bool Binlog_free_flush::should_free_flush( > + const MYSQL_BIN_LOG::group_commit_entry *entry) const > +{ > + /* Do not free flush if total_bytes smaller than limit size. */ > + if (likely(cache_data->get_byte_position() <= > + opt_binlog_free_flush_threshold)) > + return false; Put this check as the first check in the function. So the common case of a small transaction does not need to run any of the other checks. And as suggested above, put a check: if (unlikely(!opt_optimize_thread_scheduling)) return false; (This is not critical, but just to make things a bit easier; there is no reason for you/us to worry about making commit-by-rotate work with disabled optimize_thread_scheduling, this option isn't used any more. Otherwise you'd need to test that your patch works correctly with optimize_thread_scheduling disabled, which would just be a waste of time.) > +bool Binlog_free_flush::commit(MYSQL_BIN_LOG::group_commit_entry *entry) > +{ > + // It will be released by trx_group_commit_leader > + mysql_mutex_lock(&mysql_bin_log.LOCK_log); After suggested changes, comment should read "It will be released by trx_group_commit_with_engines()." > + mysql_bin_log.trx_group_commit_leader(entry, true /* is_free_flush */); As discussed above, change this to call a new function mysql_bin_log.trx_group_commit_with_engines(), which contains the second part of trx_group_commit_leader() that you need to run. Then there is no longer a need to pass a `true` flag here to skip the first part. Also, before the call, this is the place to set entry->next to nullptr, to convert the stand-alone entry into a (singleton) list (instead of initializing it in MYSQL_BIN_LOG::write_transaction_to_binlog().) > @@ -8301,6 +8418,7 @@ MYSQL_BIN_LOG::write_transaction_to_binlog(THD *thd, > DBUG_RETURN(0); > } > > + entry.next= nullptr; > entry.thd= thd; > entry.cache_mngr= cache_mngr; > entry.error= 0; Do you need to initialize entry->next here, and if so why? At this point, the entry is not a list, it is a stand-alone struct that is not yet linked into any list, the next pointer should be set appropriately when/if the struct is later put into a list. More important than the minor cost of initializing, by redundantly initializing here we are hiding bugs (eg. Valgrind/msan/asan) if some code would incorrectly read the next pointer. > +bool Binlog_free_flush::replace_binlog_file() > +{ > + size_t binlog_size= my_b_tell(&mysql_bin_log.log_file); > + size_t required_size= binlog_size; > + // space for Gtid_log_event > + required_size+= LOG_EVENT_HEADER_LEN + Gtid_log_event::max_data_length + > + BINLOG_CHECKSUM_LEN; Just a remark, here you actually know the real required size of the GTID event, so you don't need to conservatively estimate Gtid_log_event::max_data_length. But I think doing it this way is fine, it is simpler and in most cases we are rounding up to IO_SIZE anyway, so I'm fine with keeping it this way. > + sql_print_information("Could not rename binlog cache to binlog, " > + "require %llu bytes but only %llu bytes reserved.", > + required_size, m_cache_data->file_reserved_bytes()); This log message will perhaps appear a bit out-of-context in the error log. Suggested clarification: "Could not rename binlog cache to binlog (as requested by --binlog-commit-by-rotate-threshold). Required ..." > + // Set the cache file as binlog file. > + mysql_file_close(mysql_bin_log.log_file.file, MYF(MY_WME)); Do not to pass MY_WME here (instead pass MYF(0)). Since you are ignoring the return of the function, so we don't want mysql_file_close to call my_error() leaving an error code set even though the operation succeeds. > +size_t Binlog_free_flush::get_gtid_event_pad_size() > +{ > + size_t begin_pos= my_b_tell(&mysql_bin_log.log_file); > + size_t pad_to_size= > + m_cache_data->file_reserved_bytes() - begin_pos - LOG_EVENT_HEADER_LEN; > + > + if (binlog_checksum_options != BINLOG_CHECKSUM_ALG_OFF) > + pad_to_size-= BINLOG_CHECKSUM_LEN; Adjusting with BINLOG_CHECKSUM_LEN is to account for the 4 bytes of checksum at the end of the GTID_EVENT, right? But why do you subtract LOG_EVENT_HEADER_LEN when calculating pad_to_size? (I mean, it is probably correct, otherwise nothing would work I think. I just did not understand it, maybe you can explain in a short comment. Or maybe Gtid_log_event::write() could be modified to do the adjusting for LOG_EVENT_HEADER_LEN and BINLOG_CHECKSUM_LEN). > + // offset must be saved before replace_binlog_file(), it will update the pos > + my_off_t offset= my_b_tell(&log_file); Suggest to clarify comment to "..., it will update the file position". > @@ -756,18 +759,20 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log > new_file() is locking. new_file_without_locking() does not acquire > LOCK_log. > */ > - int new_file_impl(); > + int new_file_impl(bool is_free_flush= false); > void do_checkpoint_request(ulong binlog_id); > - int write_transaction_or_stmt(group_commit_entry *entry, uint64 commit_id); > + int write_transaction_or_stmt(group_commit_entry *entry, uint64 commit_id, > + bool is_free_flush= false); > int queue_for_group_commit(group_commit_entry *entry); > bool write_transaction_to_binlog_events(group_commit_entry *entry); > - void trx_group_commit_leader(group_commit_entry *leader); > + void trx_group_commit_leader(group_commit_entry *leader, > + bool is_free_flush= false); > bool is_xidlist_idle_nolock(); > void update_gtid_index(uint32 offset, rpl_gtid gtid); > > public: > void purge(bool all); > - int new_file_without_locking(); > + int new_file_without_locking(bool is_free_flush= false); > /* > A list of struct xid_count_per_binlog is used to keep track of how many > XIDs are in prepared, but not committed, state in each binlog. And how > @@ -997,7 +1002,8 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log > enum cache_type io_cache_type_arg, > ulong max_size, > bool null_created, > - bool need_mutex); > + bool need_mutex, > + bool is_free_flush = false); > bool open_index_file(const char *index_file_name_arg, > const char *log_name, bool need_mutex); > /* Use this to start writing a new log file */ > @@ -1037,7 +1043,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private Event_log > bool is_active(const char* log_file_name); > bool can_purge_log(const char *log_file_name, bool interactive); > int update_log_index(LOG_INFO* linfo, bool need_update_threads); > - int rotate(bool force_rotate, bool* check_purge); > + int rotate(bool force_rotate, bool* check_purge, bool is_free_flush= false); > void checkpoint_and_purge(ulong binlog_id); > int rotate_and_purge(bool force_rotate, DYNAMIC_ARRAY* drop_gtid_domain= NULL); > /** Style preference: please add the these new boolean parameters without default value, and instead change existing callers to pass false explicitly. (And the parameter will be "commit_by_rotate" after name change.) > + > +extern ulonglong opt_binlog_free_flush_threshold; > +static Sys_var_ulonglong Sys_binlog_free_flush_threshold( > + "binlog_free_flush_threshold", > + "Try to rename the binlog cache temporary file of the commiting " > + "transaction to a binlog file when its binlog cache size " > + "is bigger than the value of this variable", > + GLOBAL_VAR(opt_binlog_free_flush_threshold), > + CMD_LINE(REQUIRED_ARG), VALID_RANGE(10 * 1024 * 1024, ULLONG_MAX), > + DEFAULT(128 * 1024 * 1024), BLOCK_SIZE(1)); As described above: - Suggested name: "binlog_commit_by_rotate_threshold". - Suggested text: "For transactions that exceed this size in the binlog, commit by a more efficient method that forces a binlog rotation and re-uses the pre-existing temporary file as the next binlog file." - Let me hear opinions on whether this should default to 128 MB or should default to eg. ULONGLONG_MAX to disable the feature by default. > diff --git a/mysql-test/main/tmp_space_usage.test b/mysql-test/main/tmp_space_usage.test > index af7b295f343..1685dbbc450 100644 > --- a/mysql-test/main/tmp_space_usage.test > +++ b/mysql-test/main/tmp_space_usage.test > @@ -215,7 +215,8 @@ select count(distinct concat(seq,repeat('x',1000))) from seq_1_to_1000; > > set @save_max_tmp_total_space_usage=@@global.max_tmp_total_space_usage; > set @@global.max_tmp_total_space_usage=64*1024*1024; > -set @@max_tmp_session_space_usage=1179648; > +# Binlog cache reserve 4096 bytes at the begin of the temporary file. > +set @@max_tmp_session_space_usage=1179648+65536; > select @@max_tmp_session_space_usage; > set @save_aria_repair_threads=@@aria_repair_threads; > set @@aria_repair_threads=2; > @@ -224,6 +225,7 @@ set @@max_heap_table_size=16777216; > > CREATE TABLE t1 (a CHAR(255),b INT,INDEX (b)); > INSERT INTO t1 SELECT SEQ,SEQ FROM seq_1_to_100000; > +set @@max_tmp_session_space_usage=1179648; > --error 200 > SELECT * FROM t1 UNION SELECT * FROM t1; > DROP TABLE t1; > @@ -266,11 +268,15 @@ SELECT MIN(VARIABLE_VALUE) OVER (), NTILE(1) OVER (), MAX(VARIABLE_NAME) OVER () > > connect(c1, localhost, root,,); > set @@binlog_format=row; > -CREATE OR REPLACE TABLE t1 (a DATETIME) ENGINE=MyISAM; > +CREATE OR REPLACE TABLE t1 (a DATETIME) ENGINE=InnoDB; > +# Use the transaction to keep binlog cache temporary file large enough > +BEGIN; > INSERT INTO t1 SELECT NOW() FROM seq_1_to_6000; > + > SET max_tmp_session_space_usage = 64*1024; > --error 200 > SELECT * FROM information_schema.ALL_PLUGINS LIMIT 2; > +ROLLBACK; > drop table t1; Can you explain why you need these changes to the test case? I was not able to understand that from the patch. With these changes, I approve of this patch / pull-request to merge into the next MariaDB release. - Kristian.

3 3

Re: bd616a3733c: Auth: set thd->scramble in all cases during writing Initial Handshake Packet
by Sergei Golubchik 10 Sep '24

10 Sep '24

Hi, Nikita, On Sep 09, Nikita Malyavin wrote: > revision-id: bd616a3733c (mariadb-11.6.1-13-gbd616a3733c) > parent(s): bbbb429a1eb > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-09-07 20:46:19 +0200 > message: > > Auth: set thd->scramble in all cases during writing Initial Handshake Packet > > --- > sql/sql_acl.cc | 19 +++++++++++++------ > 1 file changed, 13 insertions(+), 6 deletions(-) > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 93efc5806cc..7845af55413 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -13504,17 +13504,21 @@ static bool send_server_handshake_packet(MPVIO_EXT *mpvio, > mpvio->cached_server_packet.pkt_len= data_len; > } > > - if (data_len < SCRAMBLE_LENGTH) > + if (thd->scramble[SCRAMBLE_LENGTH] != 0) old code was obviously wrong but the new one makes no sense either. How can thd->scramble[SCRAMBLE_LENGTH] be not zero at this point? I'd think you need to execute the code below unconditionally. > { > + DBUG_ASSERT(thd->scramble[SCRAMBLE_LENGTH] == 1); // Sanity > + > if (data_len) > { > /* > the first packet *must* have at least 20 bytes of a scramble. > - if a plugin provided less, we pad it to 20 with zeros > + if a plugin provided less, we pad it to 20 with zeros, > + plus extra zero termination sign is put in thd->scramble. > + If more is provided, we'll use only 20 bytes as a handshake scramble. Not sure it's a good idea. Why not to send the complete scramble? True, it won't fit on the client side into mysql->scramble[]. Two ways to fix it: * Write the first 20 bytes. The server will only store the first 20 bytes too, and for COM_CHANGE_USER they'll use the first 20 bytes only (for the login auth they can use the complete scramble). That's a rather hackish fix, a proper fix would be * store the complete scramble in MYSQL. The first 20 bytes in MYSQL::scramble_buff, the rest in MySQL::extension.scramble_suffix > */ > - memcpy(scramble_buf, data, data_len); > - bzero(scramble_buf + data_len, SCRAMBLE_LENGTH - data_len); > - data= scramble_buf; > + size_t fill_size= MY_MIN(SCRAMBLE_LENGTH, data_len); > + memcpy(thd->scramble, data, fill_size); > + bzero(thd->scramble + fill_size, SCRAMBLE_LENGTH - fill_size + 1); > } Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

Re: 982bf06d560: MDEV-12320 configurable default authentication plugin for the server
by Sergei Golubchik 09 Sep '24

09 Sep '24

Hi, Nikita, This looks good. Minor comments below. On Sep 09, Nikita Malyavin wrote: > revision-id: 982bf06d560 (mariadb-11.6.1-14-g982bf06d560) > parent(s): bd616a3733c > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-09-07 21:44:13 +0200 > message: > > MDEV-12320 configurable default authentication plugin for the server > > * Add a new cmdline-only variable "default_auth_plugin". > * A default plugin is locked at the server init and unlocked at the deinit > stages. This means that mysql_native_password and old_password_plugin, when > default, are locked/unlocked twice. doesn't matter, compiled-in plugins are only locked in debug builds, otherwise it's a no-op. > > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -4538,6 +4538,14 @@ static Sys_var_plugin Sys_enforce_storage_engine( > DEFAULT(&enforced_storage_engine), NO_MUTEX_GUARD, NOT_IN_BINLOG, > ON_CHECK(check_has_super)); > > +extern const char *default_auth_plugin_name; > +extern LEX_CSTRING native_password_plugin_name; Is it ok? The correct type is Lex_ident_plugin. > +static Sys_var_charptr Sys_default_auth_plugin( there's also Sys_var_lexstring, if you prefer that. > + "default_auth_plugin", "Default plugin, that will be tried first when authenticating new connections", reformat the long line, please > + READ_ONLY GLOBAL_VAR(default_auth_plugin_name), CMD_LINE(OPT_ARG), > + DEFAULT(native_password_plugin_name.str), > + NO_MUTEX_GUARD, NOT_IN_BINLOG); > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2543,9 +2542,20 @@ bool acl_init(bool dont_read_acl_tables) > old_password_plugin= my_plugin_lock_by_name(0, > &old_password_plugin_name, MYSQL_AUTHENTICATION_PLUGIN); > > + Lex_cstring_strlen def_plugin_name(default_auth_plugin_name); > + default_auth_plugin= my_plugin_lock_by_name(NULL, &def_plugin_name, > + MYSQL_AUTHENTICATION_PLUGIN); > + > if (!native_password_plugin || !old_password_plugin) > DBUG_RETURN(1); > > + if (!default_auth_plugin) > + { > + sql_print_error("Default plugin %s could not be loaded", > + default_auth_plugin_name); see init_default_storage_engine() in mysqld.cc - it's for --default-storage-engine option. let's use similar wording for consistency: sql_print_error("Unknown/unsupported authentication plugin: %s", default_auth_plugin_name); > + DBUG_RETURN(1); > + } > + > if (dont_read_acl_tables) > { > DBUG_RETURN(0); /* purecov: tested */ Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: bbbb429a1eb: Auth: release the plugins from responsibility to fill scramble_buff.
by Sergei Golubchik 09 Sep '24

09 Sep '24

Hi, Nikita, Conceptually ok, see one omission below. On Sep 09, Nikita Malyavin wrote: > revision-id: bbbb429a1eb (mariadb-11.6.1-12-gbbbb429a1eb) > parent(s): db5d1cde450 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-09-07 20:44:02 +0200 > message: > > Auth: release the plugins from responsibility to fill scramble_buff. > > Fill it once during the Initial Handshake Packet parsing. This uses the fact > that the server guarantees first 20 bytes to be the scramble. > > Submodule libmariadb de6305915f8..12d78187061: > diff --git a/libmariadb/libmariadb/mariadb_lib.c b/libmariadb/libmariadb/mariadb_lib.c > index 78195d44..07953ea4 100644 > --- a/libmariadb/libmariadb/mariadb_lib.c > +++ b/libmariadb/libmariadb/mariadb_lib.c > @@ -1954,6 +1954,8 @@ restart: > goto error; > } > } > + memmove(mysql->scramble_buff, scramble_data, SCRAMBLE_LENGTH); > + mysql->scramble_buff[SCRAMBLE_LENGTH]= 0; > /* Set character set */ > if (mysql->options.charset_name) > diff --git a/libmariadb/plugins/auth/my_auth.c b/libmariadb/plugins/auth/my_auth.c > index a2fd519d..47d26150 100644 > --- a/libmariadb/plugins/auth/my_auth.c > +++ b/libmariadb/plugins/auth/my_auth.c > @@ -126,10 +126,6 @@ static int native_password_auth_client(MYSQL_PLUGIN_VIO *vio, MYSQL *mysql) > > if (pkt_len != SCRAMBLE_LENGTH + 1) > return CR_SERVER_HANDSHAKE_ERR; > - > - /* save it in MYSQL */ > - memmove(mysql->scramble_buff, pkt, SCRAMBLE_LENGTH); > - mysql->scramble_buff[SCRAMBLE_LENGTH] = 0; This looks ok > } > > if (mysql && mysql->passwd[0]) > diff --git a/sql-common/client.c b/sql-common/client.c > index 6d030ce0a17..28b477c1e9a 100644 > --- a/sql-common/client.c > +++ b/sql-common/client.c > @@ -4178,10 +4178,6 @@ static int native_password_auth_client(MYSQL_PLUGIN_VIO *vio, MYSQL *mysql) > > if (pkt_len != SCRAMBLE_LENGTH + 1) > DBUG_RETURN(CR_SERVER_HANDSHAKE_ERR); > - > - /* save it in MYSQL */ > - memcpy(mysql->scramble, pkt, SCRAMBLE_LENGTH); > - mysql->scramble[SCRAMBLE_LENGTH] = 0; this isn't, you removed memcpy from the plugin, but didn't add it to mysql_real_connect in sql-common/client.c. > } > > if (mysql->passwd[0]) Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: b96afca238a: MDEV-16440 merge 5.7 P_S sysvars instrumentation and tables
by Sergei Golubchik 08 Sep '24

08 Sep '24

Hi, Nikita, Despite the email subject, it's a review of everything in bb-10.10-MDEV-16440, *except* 9c57bbfef92. That is 37e1ccbf678..c16b2429f02 without notifiable_work_zone.h I'll send a separate email about notifiable_work_zone.h On Sep 08, Nikita Malyavin wrote: > revision-id: b96afca238a (mariadb-10.6.1-472-gb96afca238a) > parent(s): 37e1ccbf678 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-06-19 14:52:30 +0300 > message: > > MDEV-16440 merge 5.7 P_S sysvars instrumentation and tables > diff --git a/mysql-test/suite/perfschema/t/variables_stress.test b/mysql-test/suite/perfschema/t/variables_stress.test > new file mode 100644 > index 00000000000..be3012bf8af > --- /dev/null > +++ b/mysql-test/suite/perfschema/t/variables_stress.test > @@ -0,0 +1,71 @@ stress tests generally don't belong in mtr and aren't reliable enough for the mtr. Also, you forgot to check in the result file for it. on the other hand, looking at the patch I got the impression that this stress test helped to find a lot of bugs. > +let $i = 64; > +enable_connect_log; > +while ($i) { > + echo i = $i; > + connect(con_$i, localhost, root); > +# set debug_sync = "tp_hanger wait_for banger"; > + send set debug_dbug = "+d,tp_wanger"; > + dec $i; > +} > +connection default; > + > +SELECT THREAD_ID, PROCESSLIST_ID FROM performance_schema.threads; > +set debug_sync = "now signal banger"; > +connect(banger_1, localhost, root); > +let $banger1= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > +connect(banger_2, localhost, root); > +let $banger2= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > +connect(banger_3, localhost, root); > +let $banger3= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > +connect(banger_4, localhost, root); > +let $banger4= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > +connect(banger_5, localhost, root); > +let $banger5= `SELECT THREAD_ID FROM performance_schema.threads WHERE PROCESSLIST_ID = CONNECTION_ID()`; > + > +send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + > +let $i = 2000; > +let $j=65; > +while ($i) { > + echo i = $i; > + connection banger_1; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + connection banger_2; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + connection banger_3; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + connection banger_4; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + connection banger_5; > + reap; > + send select count(*) from performance_schema.variables_by_thread where variable_name="time_zone"; > + dec $i; > + > + let $threads= `show status where variable_name = 'threads_connected'`; > + if ($threads < 63) { > + connect(con_$i, localhost, root); > + send set debug_dbug = "+d,tp_wanger"; > + inc $j; > + } > +} > +reap; > +select * from performance_schema.variables_by_thread where variable_name="time_zone"; > +echo $banger1; > +echo $banger2; > +echo $banger3; > +echo $banger4; > +echo $banger5; > +# eval select * from performance_schema.variables_by_thread where variable_name like "%id" and thread_id=197; > +connection default; > +# set global debug_dbug = ""; > +set debug_sync = reset; > + > +show status where variable_name = 'threads_connected'; > diff --git a/sql/sql_plugin.h b/sql/sql_plugin.h > index d4df8c6468f..3d02d1c7509 100644 > --- a/sql/sql_plugin.h > +++ b/sql/sql_plugin.h > @@ -186,6 +186,10 @@ extern SHOW_COMP_OPTION plugin_status(const char *name, size_t len, int type); > extern bool check_valid_path(const char *path, size_t length); > extern void plugin_mutex_init(); > > +template <class T> class Dynamic_array; you can simply include sql_array.h, can't you? > + > +void plugin_lock_by_sys_var_array(THD *thd, Dynamic_array<SHOW_VAR> *vars); > + > typedef my_bool (plugin_foreach_func)(THD *thd, > plugin_ref plugin, > void *arg); > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 0597b086b7b..ec86389d08f 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -1296,6 +1296,9 @@ dispatch_command_return do_command(THD *thd, bool blocking) > goto out; > } > > + if (unlikely(thd->apc_target.have_apc_requests())) > + thd->apc_target.process_apc_requests(); 1. why here? 2. there generally shouldn't be direct checks of apc requests, the code should do check_killed() often enough. So if you really need a check here (see question 1), add if (thd->check_killed())... > + > packet= (char*) net->read_pos; > /* > 'packet_length' contains length of data, as it was stored in packet > diff --git a/vio/viopipe.c b/vio/viopipe.c > index aeaad311b7e..5106c6d6513 100644 > --- a/vio/viopipe.c > +++ b/vio/viopipe.c > @@ -53,8 +53,9 @@ static size_t wait_overlapped_result(Vio *vio, int timeout) > } > else > { > + DWORD last_error= GetLastError(); > /* Error or timeout, cancel the pending I/O operation. */ > - CancelIo(vio->hPipe); > + CancelIoEx(vio->hPipe, &vio->overlapped); why, CancelIoEx resets last error, but CancelIo doesn't? > /* > If the wait timed out, set error code to indicate a > diff --git a/sql/threadpool_win.cc b/sql/threadpool_win.cc > index ed68e31c755..25844d1f3ef 100644 > --- a/sql/threadpool_win.cc > +++ b/sql/threadpool_win.cc > @@ -131,6 +131,11 @@ void TP_pool_win::resume(TP_connection* c) > SubmitThreadpoolWork(((TP_connection_win*)c)->work); > } > > +int TP_pool_win::wake(TP_connection *c) > +{ > + return 0; > +} How is that supposed to work? How do you wake up threadpool threads on Windows? > + > #define CHECK_ALLOC_ERROR(op) \ > do \ > { \ > diff --git a/mysys/my_thr_init.c b/mysys/my_thr_init.c > index cde34e5bdb9..89f190ee989 100644 > --- a/mysys/my_thr_init.c > +++ b/mysys/my_thr_init.c > @@ -432,9 +432,21 @@ const char *my_thread_name(void) > extern void **my_thread_var_dbug() > { > struct st_my_thread_var *tmp; > + int last_error; > if (!my_thread_global_init_done) > return NULL; > +#ifdef WIN32 > + last_error= GetLastError(); > +#else > + last_error= errno; > +#endif > + > tmp= my_thread_var; > +#ifdef WIN32 > + SetLastError(last_error); > +#else > + errno= last_error; > +#endif huh? my_thread_var can change errno? > return tmp && tmp->init ? &tmp->dbug : 0; > } > #endif /* DBUG_OFF */ > diff --git a/storage/perfschema/pfs_variable.h b/storage/perfschema/pfs_variable.h > index e59b02f2af8..f509e97048c 100644 > --- a/storage/perfschema/pfs_variable.h > +++ b/storage/perfschema/pfs_variable.h > @@ -212,7 +212,7 @@ class Find_THD_variable : public Find_THD_Impl > return false; > > /* Hold this lock to keep THD during materialization. */ > - mysql_mutex_lock(&thd->LOCK_thd_data); > + mysql_mutex_lock(&thd->LOCK_thd_kill); see commits 736a54f49c72 and c6c2a2b8d463 > return true; > } > void set_unsafe_thd(THD *unsafe_thd) { m_unsafe_thd= unsafe_thd; } > diff --git a/sql/threadpool_generic.h b/sql/threadpool_generic.h > index b7a35b7cbf0..e16ec4a4966 100644 > --- a/sql/threadpool_generic.h > +++ b/sql/threadpool_generic.h > @@ -88,9 +89,17 @@ struct TP_connection_generic :public TP_connection > ulonglong abs_wait_timeout; > ulonglong enqueue_time; > TP_file_handle fd; > + /** > + Designates whether fd is currently connected to the poll denoted by > + thread_group->pollfd. See also change_group. > + */ > bool bound_to_poll_descriptor; > int waiting; > bool fix_group; > +#ifndef WIN32 > + Notifiable_work_zone work_zone; > + bool leave_work_zone() final; > +#endif Do you need this #ifndef? WIN32 is *never* defined, it's _WIN32. and _WIN32 is never defined *here*, on Windows we use TP_connection_win. > #ifdef _WIN32 > win_aiosocket win_sock{}; > void init_vio(st_vio *vio) override > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index a0ba1d7e5a3..bfe49cddbf4 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1909,13 +1910,13 @@ void THD::awake_no_mutex(killed_state state_to_set) > } > > /* Broadcast a condition to kick the target if it is waiting on it. */ > -void THD::abort_current_cond_wait(bool force) > +void THD::abort_current_cond_wait(bool force, bool mark_abort) a bit illogical, these values aren't orthogonal. you want a 3-state enum, like ABORT_NONE, ABORT_CONNECTIONS_BUT_NOT_SYSTEM_THREADS, ABORT_ALL > { > mysql_mutex_assert_owner(&LOCK_thd_kill); > if (mysys_var) > { > mysql_mutex_lock(&mysys_var->mutex); > - if (!system_thread || force) // Don't abort locks > + if (mark_abort && (!system_thread || force)) // Don't abort locks > mysys_var->abort=1; > > /* > @@ -4373,6 +4374,14 @@ my_bool thd_net_is_killed(THD *thd) > return thd && thd->killed ? 1 : 0; > } > > +/* returns true if any APC was processed */ > +void thd_net_process_apc_requests(THD *thd) even though it's called from net_serv.cc, there's nothing about NET here, so calling it thd_net_process_apc_requests is misleading > +{ > + if (unlikely(thd->apc_target.have_apc_requests())) > + thd->apc_target.process_apc_requests(); you also call it from the threadpool_common and a couple of times you have these two lines verbatim in other places (e.g. sql_parse.cc) why couldn't you call thd->check_killed() instead? > + DEBUG_SYNC(thd, "net_after_apc"); > +} > + > > void thd_increment_bytes_received(void *thd, size_t length) > { > diff --git a/sql/my_apc.h b/sql/my_apc.h > index cc98e36bbe4..ae60bdc2763 100644 > --- a/sql/my_apc.h > +++ b/sql/my_apc.h > @@ -81,7 +83,7 @@ class Apc_target > */ > inline bool have_apc_requests() > { > - return MY_TEST(apc_calls); > + return MY_TEST(apc_calls.load(std::memory_order_acquire)); why? > } > > inline bool is_enabled() { return enabled; } > @@ -95,14 +97,19 @@ class Apc_target > virtual ~Apc_call() {} > }; > > + class Call_request; > /* Make a call in the target thread (see function definition for details) */ > - bool make_apc_call(THD *caller_thd, Apc_call *call, int timeout_sec, bool *timed_out); > + bool make_apc_call(THD *caller_thd, Apc_call *call, > + int timeout_sec, bool *timed_out); > + > + void enqueue_request(Call_request *request_buff, Apc_call *call); > + int wait_for_completion(THD *caller_thd, Call_request *request, > + int timeout_sec); You wrote > In pfs_variable.cc we will have to make an additional step between > enqueue_request and wait_for_completion. These two functions will be called > directly and therefore both should have a public interface and I don't think I quite agree with that. Your "additional step" seems to be abort_current_cond_wait() and notify_apc(). It's not an exotic additional step that only pfs_variable.cc needs, anyone, who makes an apc call needs the target thread to wake up and process it. It's something that should be part of apc code, not done only in pfs_variable.cc And this case you likely won't need enqueue_request and wait_for_completion as a public api > > #ifndef DBUG_OFF > int n_calls_processed; /* Number of calls served by this target */ > #endif > private: > - class Call_request; > > /* > Non-zero value means we're enabled. It's an int, not bool, because one can > diff --git a/sql/threadpool.h b/sql/threadpool.h > index 7737d056b4a..08e85d362e7 100644 > --- a/sql/threadpool.h > +++ b/sql/threadpool.h > @@ -112,6 +119,8 @@ struct TP_connection > > /* Read for the next client command (async) with specified timeout */ > virtual int start_io() = 0; > + IF_WIN(,virtual) bool leave_work_zone(){ return true; } > + IF_WIN(Notifiable_work_zone work_zone;,) // Dummy object. Ugh, this is unreadable. IF_WIN was supposed to be used in expressions. And why do you need virtual methods here, again? > > virtual void wait_begin(int type)= 0; > virtual void wait_end() = 0; > diff --git a/sql/threadpool_generic.cc b/sql/threadpool_generic.cc > index eb08441a4d5..f61dd90606b 100644 > --- a/sql/threadpool_generic.cc > +++ b/sql/threadpool_generic.cc > @@ -114,7 +114,8 @@ struct pool_timer_t > > static pool_timer_t pool_timer; > > -static void queue_put(thread_group_t *thread_group, TP_connection_generic *connection); > +IF_DBUG(,static) you can remove this IF_DBUG, I don't see you use queue_put outside of this file. > +void queue_put(thread_group_t *thread_group, TP_connection_generic *connection); > static void queue_put(thread_group_t *thread_group, native_event *ev, int cnt); > static int wake_thread(thread_group_t *thread_group,bool due_to_stall); > static int wake_or_create_thread(thread_group_t *thread_group, bool due_to_stall=false); > @@ -1485,8 +1520,41 @@ static int change_group(TP_connection_generic *c, > } > > > +#ifndef WIN32 > +bool TP_connection_generic::leave_work_zone() > +{ > + auto leave_state= work_zone.try_leave(); > + while (unlikely(leave_state == Notifiable_work_zone::SIGNAL)) > + { > + if (thd->apc_target.have_apc_requests()) > + thd->apc_target.process_apc_requests(); > + leave_state= work_zone.try_leave(); > + } > + return leave_state == Notifiable_work_zone::OWNER; > +} > +#endif > + > +#ifndef DBUG_OFF > +#include <random> > +static thread_local std::mt19937 mt_random{std::random_device()()}; > +static thread_local std::uniform_int_distribution<int> kill_dist{1, 100000}; > +void queue_put(thread_group_t *thread_group, TP_connection_generic *connection); > +void tp_send(TP_connection_type* c); > +#endif > + > int TP_connection_generic::start_io() > { > + if(DBUG_IF("tp_wanger")) > + { > + bool survive= kill_dist(mt_random) != 555; does that compile? you define kill_dist under #ifndef DBUG_OFF. You can fix this with, like #ifndef DBUG_OFF if (DBUG_IF("tp_wanger")) { static thread_local std::mt19937 ... etc } #endif > + if (survive) > + { > + state= TP_STATE_INTERRUPTED; > + tp_send(this); > + } > + return survive ? 0 : 1; > + } > + > /* > Usually, connection will stay in the same group for the entire > connection's life. However, we do allow group_count to > diff --git a/vio/viosocket.c b/vio/viosocket.c > index 002ff274b74..45c3d80a33d 100644 > --- a/vio/viosocket.c > +++ b/vio/viosocket.c > @@ -909,20 +916,53 @@ static my_bool socket_peek_read(Vio *vio, uint *bytes) > */ > > #ifndef _WIN32 > +#ifdef _GNU_SOURCE > +#define PFD_SIZE 1 > +static inline int vio_poll(struct pollfd pfd[], nfds_t nr, int timeout) > +{ > + struct timespec tm, *tm_arg= NULL; > + sigset_t signals; > + /* Convert the timeout, in milliseconds, to seconds and microseconds. */ > + if (timeout >= 0) > + { > + tm.tv_sec= timeout / 1000; > + tm.tv_nsec= (timeout % 1000) * 1000 * 1000; > + tm_arg= &tm; > + } > + sigemptyset(&signals); > + > + return ppoll(pfd, 1, tm_arg, &signals); > +} > +#else > +#define PFD_SIZE 2 > +#define vio_poll poll > +#endif why do you need that? How is your vio_poll() different from poll() ? > + > int vio_io_wait(Vio *vio, enum enum_vio_io_event event, int timeout) > { > int ret; > short revents __attribute__((unused)) = 0; > - struct pollfd pfd; > + const short revents_read = MY_POLL_SET_IN | MY_POLL_SET_ERR | POLLRDHUP; > + struct pollfd pfd[PFD_SIZE]; > my_socket sd= mysql_socket_getfd(vio->mysql_socket); > + int pfd_size; > MYSQL_SOCKET_WAIT_VARIABLES(locker, state) /* no ';' */ > DBUG_ENTER("vio_io_wait"); > DBUG_PRINT("enter", ("timeout: %d", timeout)); > > - memset(&pfd, 0, sizeof(pfd)); > + memset(pfd, 0, sizeof(pfd)); > > - pfd.fd= sd; > + pfd[0].fd= sd; > > +#ifndef _GNU_SOURCE > + my_socket self_pipe= threadlocal_get_self_pipe(); > + if (self_pipe) > + { > + pfd[1].fd= self_pipe; > + pfd[1].events= MY_POLL_SET_IN; > + } > +#endif > + pfd_size= PFD_SIZE; really? I'd expect it to be essentially pfd_size= self_pipe ? 2 : 1; > /* > Set the poll bitmask describing the type of events. > The error flags are only valid in the revents bitmask. > @@ -970,16 +1023,16 @@ int vio_io_wait(Vio *vio, enum enum_vio_io_event event, int timeout) > } > > #else > - > int vio_io_wait(Vio *vio, enum enum_vio_io_event event, int timeout) > { > int ret; > - struct timeval tm; > - my_socket fd= mysql_socket_getfd(vio->mysql_socket); > - fd_set readfds, writefds, exceptfds; > MYSQL_SOCKET_WAIT_VARIABLES(locker, state) /* no ';' */ > DBUG_ENTER("vio_io_wait"); > > + /* Not read , e.g connect or write - use select() based wait */ > + struct timeval tm; > + my_socket fd= mysql_socket_getfd(vio->mysql_socket); > + fd_set readfds, writefds, exceptfds; it's C, you cannot mix declarations and code > /* Convert the timeout, in milliseconds, to seconds and microseconds. */ > if (timeout >= 0) > { > @@ -1013,14 +1066,17 @@ int vio_io_wait(Vio *vio, enum enum_vio_io_event event, int timeout) > ret= select(0, &readfds, &writefds, &exceptfds, (timeout >= 0) ? &tm : NULL); > > END_SOCKET_WAIT(locker, timeout); > /* Set error code to indicate a timeout error. */ > if (ret == 0) > WSASetLastError(SOCKET_ETIMEDOUT); > > /* Error or timeout? */ > - if (ret <= 0) > + if (ret <= 0){ indentation > + DBUG_PRINT("vio", ("vio_io_wait: error return: %d. Last error: %d", > + ret, socket_errno)); > DBUG_RETURN(ret); > + } > > /* The requested I/O event is ready? */ > switch (event) > diff --git a/sql/sql_connect.cc b/sql/sql_connect.cc > index 9beb3cca8f8..1c8975cfb23 100644 > --- a/sql/sql_connect.cc > +++ b/sql/sql_connect.cc > @@ -23,7 +23,53 @@ > #include "mariadb.h" > #include "mysqld.h" > #include "sql_priv.h" > -#ifndef _WIN32 > +#ifdef _WIN32 > +#ifdef _WIN32_WINNT > +#undef _WIN32_WINNT > +#endif > + > +#define _WIN32_WINNT 0x0500 > +#include "windows.h" // QueueUserAPC2 > +#define WINDOWS_KERNEL32_DLLNAME_W "kernel32" > + > +// https://github.com/dotnet/runtime/pull/55649/files > + > +// These declarations are for a new special user-mode APC feature introduced in Windows. These are not yet available in Windows > +// SDK headers, so some names below are prefixed with "CLONE_" to avoid conflicts in the future. Once the prefixed declarations > +// become available in the Windows SDK headers, the prefixed declarations below can be removed in favor of the SDK ones. > + > +//enum CLONE_QUEUE_USER_APC_FLAGS > +//{ > +// CLONE_QUEUE_USER_APC_FLAGS_NONE = 0x0, > +// CLONE_QUEUE_USER_APC_FLAGS_SPECIAL_USER_APC = 0x1, > +// > +// CLONE_QUEUE_USER_APC_CALLBACK_DATA_CONTEXT = 0x10000 > +//}; > +//typedef BOOL (WINAPI *QueueUserAPC2Proc)(PAPCFUNC ApcRoutine, HANDLE Thread, ULONG_PTR Data, CLONE_QUEUE_USER_APC_FLAGS Flags); > +//void c() > +//{ > +// HMODULE hKernel32 = LoadLibraryExA(WINDOWS_KERNEL32_DLLNAME_W, NULL, LOAD_LIBRARY_SEARCH_SYSTEM32); > +// > +// // See if QueueUserAPC2 exists > +// QueueUserAPC2Proc pfnQueueUserAPC2Proc = (QueueUserAPC2Proc)GetProcAddress(hKernel32, "QueueUserAPC2"); > +// if (pfnQueueUserAPC2Proc == nullptr) > +// { > +// abort(); > +// } > +// > +// // See if QueueUserAPC2 supports the special user-mode APC with a callback that includes the interrupted CONTEXT. A special > +// // user-mode APC can interrupt a thread that is in user mode and not in a non-alertable wait. > +// if (!(*pfnQueueUserAPC2Proc)(EmptyApcCallback, GetCurrentThread(), 0, SpecialUserModeApcWithContextFlags)) > +// { > +// return; > +// } > +// > +// return pfnQueueUserAPC2Proc; > +//} > +// > +//static QueueUserAPC2Proc pfnQueueUserAPC2 = InitializeSpecialUserModeApc(); what's that big commented block for? and win-something defines/includes - still needed? > + > +#else > #include <netdb.h> // getservbyname, servent > #endif > #include "sql_audit.h" > @@ -1353,6 +1399,107 @@ bool thd_is_connection_alive(THD *thd) > return FALSE; > } > > +#if !defined(_WIN32) > +static void self_pipe_write(); > +#if !defined(_GNU_SOURCE) > +class Thread_apc_context; > +static thread_local Thread_apc_context *_THR_APC_CTX= NULL; may be it should be part of THD (rather, Apc_target) or mysys_var? that is, accessible as current_thd->...->self_pipe or my_thread_var->self_pipe the first one (in Apc_target) seems quite logical > +#endif > + > +class Thread_apc_context > +{ > +public: > +#ifndef _GNU_SOURCE > + my_socket self_pipe[2]{}; > +#endif > + > + bool setup_thread_apc() > + { > +#if defined(_GNU_SOURCE) > + struct sigaction act {}; > + act.sa_handler= [](int) -> void {self_pipe_write();}; why? it doesn't do anything, as far as I can see > + act.sa_flags= 0; > + int ret= sigaction(SIG_APC_NOTIFY, &act, NULL); > + DBUG_ASSERT(ret == 0); > + > + sigset_t signals; > + ret|= sigemptyset(&signals); > + DBUG_ASSERT(ret == 0); > + ret|= sigaddset(&signals, SIG_APC_NOTIFY); > + DBUG_ASSERT(ret == 0); > + > + ret|= pthread_sigmask(SIG_BLOCK, &signals, NULL); > + DBUG_ASSERT(ret == 0); > +#else > + // Self-pipe trick. Create a new pipe and store it thread-locally > + // It can be accessed from Vio later. See also vio_io_wait() > + // From the other end, it is accessed through a threadlocal > + // _THR_APC_CTX, from a SIG_APC_NOTIFY signal handler. > + int ret= pipe(self_pipe); > + > + struct sigaction act {}; > + act.sa_handler= [](int) -> void { self_pipe_write(); }; > + act.sa_flags= 0; > + ret|= sigaction(SIG_APC_NOTIFY, &act, NULL); why? you send a signal and convert it to a self-pipe write. why not to write to the pipe in the first place? like Apc_target::wake() which will send a signal or write to a pipe, as appropriate. And TP_pool::wake() will call thd->apc_target->wake(); > + return ret == 0; > +#endif > + return true; > + } > + bool inited; > + Thread_apc_context() > + { > + inited = setup_thread_apc(); > +#ifndef _GNU_SOURCE > + if (inited) > + _THR_APC_CTX= this; > +#endif > + } > +#ifndef _GNU_SOURCE > + ~Thread_apc_context() > + { > + _THR_APC_CTX= NULL; > + closesocket(self_pipe[0]); > + closesocket(self_pipe[1]); > + } > +#endif > +}; > + > +#ifdef _GNU_SOURCE > +static void self_pipe_write() > +{ > + // No self-pipe is actually used. Instead, ppoll is used to wake up on signal. > +} > +#else > + > +my_socket threadlocal_get_self_pipe() > +{ > + return _THR_APC_CTX ? _THR_APC_CTX->self_pipe[0] : 0; > +} > + > +static void self_pipe_write() > +{ > + DBUG_ASSERT(IF_WIN(0, pthread_self() == select_thread) || _THR_APC_CTX); > + if (!_THR_APC_CTX) > + return; // SIGUSR1 is used in thr_alarm, which can wake up main thread. > + char buf[4]{}; > + send(_THR_APC_CTX->self_pipe[1], buf, sizeof buf, 0); > +} > +#endif > +#endif /* !_WIN32 */ > + > +bool thread_per_connection_notify_apc(THD *thd) > +{ > +#ifdef WIN32 > + auto vio= thd->net.vio; > + HANDLE h= > + (vio->type == VIO_TYPE_NAMEDPIPE) ? vio->hPipe : (HANDLE)vio->mysql_socket.fd; > + return CancelIoEx(h, NULL); > +#else > + bool ret = pthread_kill(thd->mysys_var->pthread_self, SIG_APC_NOTIFY) == 0; > + DBUG_ASSERT(ret || errno == EAGAIN); > + return ret; > +#endif > +} 1. I'd prefer APC framework implementation to stay in my_apc.cc. not being spread over three (?) different files. 2. how is it diferent from ::wake() ? > > void do_handle_one_connection(CONNECT *connect, bool put_in_cache) > { > diff --git a/sql/threadpool_common.cc b/sql/threadpool_common.cc > index f72f46b3a6b..8f24bc9e4f7 100644 > --- a/sql/threadpool_common.cc > +++ b/sql/threadpool_common.cc > @@ -320,6 +346,10 @@ static void threadpool_remove_connection(THD *thd) > end_connection(thd); > close_connection(thd, 0); > unlink_thd(thd); > + // The rest of APC requests should be processed after unlinking. > + // This guarantees that no new APC requests can be added. > + // TODO: better notify the requestor with some KILLED state here. > + thd->apc_target.process_apc_requests(); not sure it guarantees much. One can find the thd before the unlink and add a new request after the unlink. or is it impossible? > PSI_CALL_delete_current_thread(); // before THD is destroyed > delete thd; > > diff --git a/sql/my_apc.cc b/sql/my_apc.cc > index 9777deb399a..a89d9c8bc19 100644 > --- a/sql/my_apc.cc > +++ b/sql/my_apc.cc > @@ -47,11 +47,12 @@ void Apc_target::init(mysql_mutex_t *target_mutex) > void Apc_target::enqueue_request(Call_request *qe) > { > mysql_mutex_assert_owner(LOCK_thd_kill_ptr); > - if (apc_calls) > + Call_request *old_apc_calls= apc_calls; eh, what? you modify apc_calls without a lock somewhere? > + if (old_apc_calls) > { > - Call_request *after= apc_calls->prev; > - qe->next= apc_calls; > - apc_calls->prev= qe; > + Call_request *after= old_apc_calls->prev; > + qe->next= old_apc_calls; > + old_apc_calls->prev= qe; > > qe->prev= after; > after->next= qe; > @@ -100,89 +109,118 @@ void init_show_explain_psi_keys(void) > if (PSI_server == NULL) > return; > > - PSI_server->register_cond("sql", show_explain_psi_conds, > - array_elements(show_explain_psi_conds)); > + PSI_server->register_cond("sql", apc_request_psi_conds, > + array_elements(apc_request_psi_conds)); > + PSI_server->register_mutex("sql", apc_request_psi_locks, > + array_elements(apc_request_psi_locks)); > } > #endif > > - > -/* > - Make an APC (Async Procedure Call) to another thread. > - > - @detail > - Make an APC call: schedule it for execution and wait until the target > - thread has executed it. > - > - - The caller is responsible for making sure he's not posting request > - to the thread he's calling this function from. > - > - - The caller must have locked target_mutex. The function will release it. > - > - @retval FALSE - Ok, the call has been made > - @retval TRUE - Call wasnt made (either the target is in disabled state or > - timeout occurred) > -*/ > - > -bool Apc_target::make_apc_call(THD *caller_thd, Apc_call *call, > - int timeout_sec, bool *timed_out) > +/** > + Wait gracefully until the request is completed. > + @retval 0 -- Success > + @retval 1 -- Timeout > + */ > +int Apc_target::wait_for_completion(THD *caller_thd, Call_request *apc_request, > + int timeout_sec) > { > - bool res= TRUE; > - *timed_out= FALSE; > - > - if (enabled) > - { > - /* Create and post the request */ > - Call_request apc_request; > - apc_request.call= call; > - apc_request.processed= FALSE; > - mysql_cond_init(key_show_explain_request_COND, &apc_request.COND_request, > - NULL); > - enqueue_request(&apc_request); > - apc_request.what="enqueued by make_apc_call"; > - > struct timespec abstime; > const int timeout= timeout_sec; > set_timespec(abstime, timeout); > > + DBUG_EXECUTE_IF("apc_timeout", set_timespec_nsec(abstime, 1000000);); > + int res = 1; > int wait_res= 0; > PSI_stage_info old_stage; > - caller_thd->ENTER_COND(&apc_request.COND_request, LOCK_thd_kill_ptr, > + > + mysql_mutex_unlock(LOCK_thd_kill_ptr); I don't understand how this works. After you release LOCK_thd_kill_ptr, the THD might be destroyed. How do you detect that? By the wait timing out? Who will destroy the Call_request? > + > + mysql_mutex_lock(&apc_request->LOCK_request); > + caller_thd->ENTER_COND(&apc_request->COND_request, &apc_request->LOCK_request, > &stage_show_explain, &old_stage); > /* todo: how about processing other errors here? */ > - while (!apc_request.processed && (wait_res != ETIMEDOUT)) > + while (!apc_request->processed && (wait_res != ETIMEDOUT)) > { > - /* We own LOCK_thd_kill_ptr */ > - wait_res= mysql_cond_timedwait(&apc_request.COND_request, > - LOCK_thd_kill_ptr, &abstime); > - // &apc_request.LOCK_request, &abstime); > + wait_res= mysql_cond_timedwait(&apc_request->COND_request, > + &apc_request->LOCK_request, &abstime); > if (caller_thd->killed) > break; > + > + if (caller_thd->apc_target.have_apc_requests()) > + { > + mysql_mutex_unlock(&apc_request->LOCK_request); > + caller_thd->apc_target.process_apc_requests(); > + mysql_mutex_lock(&apc_request->LOCK_request); > + } > } > > - if (!apc_request.processed) > + if (!apc_request->processed) > { > /* > The wait has timed out, or this thread was KILLed. > - Remove the request from the queue (ok to do because we own > - LOCK_thd_kill_ptr) > + We can't delete it from the queue, because LOCK_thd_kill_ptr is already > + released. It can't be reacquired because of the ordering with > + apc_request->LOCK_request. > + However, apc_request->processed is guarded by this lock. > + Set processed= TRUE and transfer the ownership to the processor thread. > + It should free the resources itself. > + apc_request cannot be referred after unlock anymore in this case. > */ > - apc_request.processed= TRUE; > - dequeue_request(&apc_request); > - *timed_out= TRUE; > - res= TRUE; > + apc_request->processed= TRUE; > + res= 1; > } > else > { > /* Request was successfully executed and dequeued by the target thread */ > - res= FALSE; > + res= 0; > } > - /* > - exit_cond() will call mysql_mutex_unlock(LOCK_thd_kill_ptr) for us: > - */ > + > + /* EXIT_COND() will call mysql_mutex_unlock(LOCK_request) for us */ > caller_thd->EXIT_COND(&old_stage); > > - /* Destroy all APC request data */ > - mysql_cond_destroy(&apc_request.COND_request); > + return res; > +} > + > +/** Create and post the request */ > +void Apc_target::enqueue_request(Call_request *request_buff, Apc_call *call) > +{ > + request_buff->call= call; > + request_buff->processed= FALSE; > + enqueue_request(request_buff); > + request_buff->what="enqueued by make_apc_call"; > +} > + > +/** > + Make an APC (Async Procedure Call) to another thread. > + > + @detail > + Make an APC call: schedule it for execution and wait until the target > + thread has executed it. > + > + - The caller is responsible for making sure he's not posting request > + to the thread he's calling this function from. > + > + - The caller must have locked target_mutex. The function will release it. > + > + @retval FALSE - Ok, the call has been made > + @retval TRUE - Call wasnt made (either the target is in disabled state or > + timeout occurred) > +*/ > + > +bool Apc_target::make_apc_call(THD *caller_thd, Apc_call *call, > + int timeout_sec, bool *timed_out) > +{ > + bool res= TRUE; > + *timed_out= FALSE; > + > + if (enabled) > + { > + Call_request *apc_request= new Call_request; > + enqueue_request(apc_request, call); > + res= wait_for_completion(caller_thd, apc_request, timeout_sec); > + *timed_out= res; > + if (!*timed_out) > + delete apc_request; > } > else > { > diff --git a/storage/perfschema/pfs_variable.cc b/storage/perfschema/pfs_variable.cc > index 17b7dfc200c..698ff1eca6c 100644 > --- a/storage/perfschema/pfs_variable.cc > +++ b/storage/perfschema/pfs_variable.cc > @@ -164,7 +167,11 @@ int PFS_system_variable_cache::do_materialize_global(void) > during materialization. > */ > if (!m_external_init) > + { > + mysql_mutex_lock(&LOCK_plugin); > init_show_var_array(OPT_GLOBAL, true); > + mysql_mutex_unlock(&LOCK_plugin); 1. why not move the locking inside init_show_var_array()? 2. init_show_var_array() locks LOCK_system_variables_hash - is that not enough? > + } > > /* Resolve the value for each SHOW_VAR in the array, add to cache. */ > for (SHOW_VAR *show_var= m_show_var_array.front(); > @@ -230,37 +236,26 @@ int PFS_system_variable_cache::do_materialize_all(THD *unsafe_thd) > m_materialized= false; > m_cache.clear(); > > - /* Block plugins from unloading. */ > - mysql_mutex_lock(&LOCK_plugin_delete); also please remove all references of LOCK_plugin_delete > - > /* > Build array of SHOW_VARs from system variable hash. Do this within > LOCK_plugin_delete to ensure that the hash table remains unchanged > while this thread is materialized. > */ > if (!m_external_init) > + { > + mysql_mutex_lock(&LOCK_plugin); > init_show_var_array(OPT_SESSION, false); > + mysql_mutex_unlock(&LOCK_plugin); > + } > > /* Get and lock a validated THD from the thread manager. */ > if ((m_safe_thd= get_THD(unsafe_thd)) != NULL) > { > DEBUG_SYNC(m_current_thd, "materialize_session_variable_array_THD_locked"); > - for (SHOW_VAR *show_var= m_show_var_array.front(); > - show_var->value && (show_var != m_show_var_array.end()); show_var++) > - { > - /* Resolve value, convert to text, add to cache. */ > - System_variable system_var(m_safe_thd, show_var, m_query_scope, false); > - m_cache.push(system_var); > - } > - > - /* Release lock taken in get_THD(). */ > - mysql_mutex_unlock(&m_safe_thd->LOCK_thd_data); > + ret= make_call(&PFS_system_variable_cache::refresh_vars, 1); > > m_materialized= true; > - ret= 0; > } > - > - mysql_mutex_unlock(&LOCK_plugin_delete); > return ret; > } > > @@ -312,6 +307,105 @@ void PFS_system_variable_cache::free_mem_root(void) > } > } > } > +class PFS_system_variable_cache_apc: public Apc_target::Apc_call > +{ > +public: > + typedef PFS_system_variable_cache::Request_func Request; > + PFS_system_variable_cache_apc(PFS_system_variable_cache *pfs, Request func, > + uint param) > + : m_pfs(pfs), m_func(func), m_param(param) {} > +private: > + PFS_system_variable_cache *m_pfs; > + Request m_func; > + uint m_param; > + > + void call_in_target_thread() override > + { > + call(m_pfs, m_func, m_param); > + } > +public: > + static void call(PFS_system_variable_cache *pfs, Request func, uint param) > + { > + THD *safe_thd= pfs->safe_thd(); > + > + DBUG_ASSERT(pfs->query_scope() == OPT_SESSION); > + > + mysql_mutex_lock(&LOCK_global_system_variables); > + if (!safe_thd->variables.dynamic_variables_ptr || > + global_system_variables.dynamic_variables_head > > + safe_thd->variables.dynamic_variables_head) > + { > + mysql_prlock_rdlock(&LOCK_system_variables_hash); > + sync_dynamic_session_variables(safe_thd, false); > + mysql_prlock_unlock(&LOCK_system_variables_hash); > + } > + mysql_mutex_unlock(&LOCK_global_system_variables); > + > + (pfs->*func)(param); > + } > +}; > + > +void PFS_system_variable_cache::refresh_vars(uint all) > +{ > + for (SHOW_VAR *show_var= m_show_var_array.front(); > + show_var->value && (show_var != m_show_var_array.end()); show_var++) > + { > + sys_var *value= (sys_var *)show_var->value; > + > + /* Match the system variable scope to the target scope. */ > + if (all || match_scope(value->scope())) > + { > + /* Resolve value, convert to text, add to cache. */ > + System_variable system_var(m_safe_thd, show_var, m_query_scope, false); > + m_cache.push(system_var); > + } > + } > +} > +void PFS_system_variable_cache::refresh_one_var(uint index) > +{ > + SHOW_VAR *show_var= &m_show_var_array.at(index); > + > + if (show_var && show_var->value && > + (show_var != m_show_var_array.end())) > + { > + sys_var *value= (sys_var *)show_var->value; > + > + /* Match the system variable scope to the target scope. */ > + if (match_scope(value->scope())) > + { > + /* Resolve value, convert to text, add to cache. */ > + System_variable system_var(m_safe_thd, show_var, m_query_scope, false); > + m_cache.push(system_var); > + } > + } > +} > + > + > +#define MAKE_CALL_MAX_RETRIES 3 unused > + > +int PFS_system_variable_cache::make_call(Request_func func, uint param) > +{ > + int ret= 0; > + THD *requestor_thd= m_current_thd; > + if (requestor_thd == m_safe_thd) > + { > + mysql_mutex_unlock(&m_safe_thd->LOCK_thd_kill); > + PFS_system_variable_cache_apc::call(this, func, param); > + } > + else > + { > + PFS_system_variable_cache_apc apc_call(this, func, param); > + auto *request= new Apc_target::Call_request; > + m_safe_thd->apc_target.enqueue_request(request, &apc_call); > + m_safe_thd->abort_current_cond_wait(false, false); > + m_safe_thd->scheduler->notify_apc(m_safe_thd); > + DEBUG_SYNC(requestor_thd, "apc_after_notify"); > + ret= m_safe_thd->apc_target.wait_for_completion(requestor_thd, request, 10); > + if (ret == 0) > + delete request; > + } > + return ret; > +} > > /** > Build a SESSION system variable cache for a pfs_thread. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: c46e7c5cb64: MDEV-12252 ROW data type for stored function return values
by Sergei Golubchik 05 Sep '24

05 Sep '24

Hi, Alexander, Looks good, thanks. Great commit comment too. Just a few questions and test suggestions - see below. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org On Aug 23, Alexander Barkov wrote: > revision-id: c46e7c5cb64 (mariadb-11.4.1-5-gc46e7c5cb64) > parent(s): ae6684d79f4 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-02-20 13:49:54 +0400 > message: > > MDEV-12252 ROW data type for stored function return values > diff --git a/mysql-test/main/sp-anchor-row-type-table.test b/mysql-test/main/sp-anchor-row-type-table.test > index 7123f9160db..2fbcfbffc1b 100644 > --- a/mysql-test/main/sp-anchor-row-type-table.test > +++ b/mysql-test/main/sp-anchor-row-type-table.test > @@ -939,3 +939,70 @@ DROP TABLE t1; > --echo # > --echo # End of 10.4 tests > --echo # > + > +--echo # > +--echo # Start of 11.3 tests > +--echo # > + > +--echo # > +--echo # MDEV-12252 ROW data type for stored function return values > +--echo # > + > +--error ER_PARSE_ERROR > +CREATE FUNCTION f1() RETURNS ROW TYPE OF step1.step2.step3 RETURN ROW(1,2); > + > +CREATE FUNCTION f1() RETURNS ROW TYPE OF t1 RETURN ROW(1,2); you don't resolve t1 at CREATE FUNCTION time. why? is it how TYPE OF works in DECLARE too? > +SHOW CREATE FUNCTION f1; > +--error ER_NO_SUCH_TABLE > +SELECT f1(); > +DROP FUNCTION f1; > + > +CREATE FUNCTION f1() RETURNS ROW TYPE OF test.t1 RETURN ROW(1,2); > +SHOW CREATE FUNCTION f1; > +--error ER_NO_SUCH_TABLE > +SELECT f1(); > +DROP FUNCTION f1; > + > + > +CREATE TABLE t1 (a INT, b VARCHAR(32)); > +CREATE FUNCTION f1() RETURNS ROW TYPE OF test.t1 RETURN (SELECT * FROM t1); > +SHOW CREATE FUNCTION f1; > +DELIMITER $$; > +CREATE PROCEDURE p1() > +BEGIN > + DECLARE r ROW TYPE OF t1 DEFAULT f1(); > + SELECT r.a, r.b; > +END; > +$$ > +DELIMITER ;$$ > + > +# Testing with no rows > +CALL p1(); > +--error ER_OPERAND_COLUMNS > +SELECT f1(); > +SELECT f1()=ROW(1,'b1') AS c; > +SELECT f1()=ROW(NULL,NULL) AS c; > + > +# Testing with one row > +INSERT INTO t1 VALUES (1,'b1'); > +CALL p1(); > +--error ER_OPERAND_COLUMNS > +SELECT f1(); > +SELECT f1()=ROW(1,'b1') AS c; > +SELECT f1()=ROW(1,'') AS c; > +SELECT f1()=ROW(2,'b1') AS c; > +SELECT f1()=ROW(1,NULL) AS c; > +SELECT f1()=ROW(NULL,'b1') AS c; do one more SHOW CREATE FUNCTION here > + > +# Testing with two rows > +INSERT INTO t1 VALUES (2,'b2'); > +--error ER_SUBQUERY_NO_1_ROW > +CALL p1(); > +--error ER_OPERAND_COLUMNS > +SELECT f1(); > +--error ER_SUBQUERY_NO_1_ROW > +SELECT f1()=ROW(1,'b1') AS c; > + > +DROP PROCEDURE p1; > +DROP FUNCTION f1; > +DROP TABLE t1; > diff --git a/mysql-test/main/sp-anchor-type.test b/mysql-test/main/sp-anchor-type.test > index 0e24ef900d8..fed53fd011a 100644 > --- a/mysql-test/main/sp-anchor-type.test > +++ b/mysql-test/main/sp-anchor-type.test > @@ -767,3 +767,78 @@ BEGIN NOT ATOMIC > END; > $$ > DELIMITER ;$$ > + > + > +--echo # > +--echo # Start of 11.3 tests > +--echo # > + > +--echo # > +--echo # MDEV-12252 ROW data type for stored function return values > +--echo # > + > +--error ER_PARSE_ERROR > +CREATE FUNCTION f1() RETURNS TYPE OF a RETURN 1; > + > +CREATE FUNCTION f1() RETURNS TYPE OF t1.a RETURN (SELECT min(a) FROM t1); > +SHOW CREATE FUNCTION f1; > +--error ER_NO_SUCH_TABLE > +SELECT f1(); > +CREATE TABLE t1 (b INT); > +--error ER_BAD_FIELD_ERROR > +SELECT f1(); > +DROP TABLE t1; > +DROP FUNCTION f1; > + > +CREATE DATABASE db1; > +DELIMITER $$; > +CREATE FUNCTION db1.f1() RETURNS TYPE OF db1.t1.a > +BEGIN > + RETURN (SELECT min(a) FROM t1); > +END; > +$$ > +DELIMITER ;$$ > +SHOW CREATE FUNCTION db1.f1; > +--error ER_NO_SUCH_TABLE > +SELECT db1.f1(); > +CREATE TABLE db1.t1 (b TIME); > +--error ER_BAD_FIELD_ERROR > +SELECT db1.f1(); > +DROP TABLE db1.t1; > +CREATE TABLE db1.t1 (a TIME); > +SELECT db1.f1(); > +INSERT INTO db1.t1 VALUES ('10:20:30'); > +SELECT db1.f1(); > +DROP TABLE db1.t1; > +DROP FUNCTION db1.f1; > +DROP DATABASE db1; > + > +CREATE TABLE t1 (a TIME); > +CREATE FUNCTION f1() RETURNS TYPE OF test.t1.a RETURN (SELECT min(a) FROM t1); > +SHOW CREATE FUNCTION f1; > +SELECT f1(); > +DROP FUNCTION f1; > +DROP TABLE t1; > + > +CREATE TABLE t1 (a TIME); > +DELIMITER $$; > +CREATE FUNCTION f1() RETURNS TYPE OF t1.a > +BEGIN > + RETURN (SELECT min(a) FROM t1); > +END; > +$$ > +DELIMITER ;$$ > +SHOW CREATE FUNCTION f1; > +SELECT f1(); > +CREATE TABLE t2 AS SELECT f1(); > +SHOW CREATE TABLE t2; > +SELECT * FROM t2; > +DROP TABLE t2; > +INSERT INTO t1 VALUES ('10:20:30'); > +SELECT f1(); > +CREATE TABLE t2 AS SELECT f1(); > +SHOW CREATE TABLE t2; > +SELECT * FROM t2; > +DROP TABLE t2; please, also try here DROP TABLE t1; CREATE TABLE t1 (a INT); INSERT INTO t1 VALUES (10); CREATE TABLE t2 AS SELECT f1(); the point is to change TYPE OF t1.a after the function has been successfully called, and thus is already in the sp cache. > +DROP FUNCTION f1; > +DROP TABLE t1; > diff --git a/mysql-test/suite/compat/oracle/t/sp-package.test b/mysql-test/suite/compat/oracle/t/sp-package.test > index aefede41c8b..7c231d736c5 100644 > --- a/mysql-test/suite/compat/oracle/t/sp-package.test > +++ b/mysql-test/suite/compat/oracle/t/sp-package.test > @@ -3109,3 +3109,79 @@ DROP TABLE t1; > DROP FUNCTION f1_deterministic; > DROP FUNCTION f2_not_deterministic; > DROP PACKAGE pkg1; > + > +--echo # > +--echo # MDEV-12252 ROW data type for stored function return values > +--echo # > + > +--echo # > +--echo # Testing fixed ROW type with package routines > +--echo # > + > +DELIMITER $$; > +CREATE PACKAGE pkg > +AS > + FUNCTION f1() RETURN ROW(a INT, b VARCHAR(32)); > + PROCEDURE p1(r ROW(a INT, b VARCHAR(32))); > + PROCEDURE p2(); > +END; > +$$ > +CREATE PACKAGE BODY pkg > +AS > + FUNCTION f1() RETURN ROW(a INT, b VARCHAR(32)) AS > + BEGIN > + RETURN ROW(1,'b1'); > + END; > + PROCEDURE p1(r ROW(a INT, b VARCHAR(32))) AS > + BEGIN > + SELECT r.a, r.b; > + END; > + PROCEDURE p2() AS > + BEGIN > + CALL p1(f1()); > + END; > +END; > +$$ > +DELIMITER ;$$ > +CALL pkg.p1(pkg.f1()); > +CALL pkg.p2; > +DROP PACKAGE pkg; > + > + > +--echo # > +--echo # Testing table%ROWTYPE with package routines > +--echo # now you can also add package tests outside of oracle mode > + > +CREATE TABLE t1 (a INT, b VARCHAR(32)); > +INSERT INTO t1 VALUES (1,'b1'); > +DELIMITER /; > +CREATE PACKAGE pkg > +AS > + FUNCTION f1 RETURN t1%ROWTYPE; > + PROCEDURE p1(r t1%ROWTYPE); > + PROCEDURE p2; > +END; > +/ > +CREATE PACKAGE BODY pkg > +AS > + FUNCTION f1 RETURN t1%ROWTYPE AS > + r t1%ROWTYPE; > + BEGIN > + SELECT * INTO r FROM t1; > + RETURN r; > + END; > + PROCEDURE p1(r t1%ROWTYPE) AS > + BEGIN > + SELECT r.a || ' ' || r.b; > + END; > + PROCEDURE p2 AS > + BEGIN > + p1(f1()); > + END; > +END; > +/ > +DELIMITER ;/ > +CALL pkg.p1(pkg.f1()); > +CALL pkg.p2; > +DROP PACKAGE pkg; > +DROP TABLE t1; > diff --git a/sql/field.h b/sql/field.h > index 7f1c243a180..7af069a9735 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -1373,6 +1374,20 @@ class Field: public Value_source > in str and restore it with set() if needed > */ > virtual void sql_type(String &str) const =0; > + void sql_type_for_sp_returns(String &str) const This is oddly specific. Strange that we don't need to print the field type with charset/collation anywhere else, but for sp returns we do. > + { > + sql_type(str); > + if (has_charset()) > + { > + str.append(STRING_WITH_LEN(" CHARSET ")); > + str.append(charset()->cs_name); > + if (Charset(charset()).can_have_collate_clause()) > + { > + str.append(STRING_WITH_LEN(" COLLATE ")); > + str.append(charset()->coll_name); > + } > + } > + } > virtual void sql_rpl_type(String *str) const { sql_type(*str); } > virtual uint size_of() const =0; // For new field > inline bool is_null(my_ptrdiff_t row_offset= 0) const > diff --git a/sql/field.cc b/sql/field.cc > index 35be045620d..23b050de221 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -2768,6 +2768,32 @@ bool Field_row::sp_prepare_and_store_item(THD *thd, Item **value) > } > > > +uint Field_row::cols() const > +{ > + // The table with ROW members must be already instantiated > + DBUG_ASSERT(m_table); > + return m_table->s->fields; > +} > + > + > +void Field_row::sql_type(String &res) const would be more logical to call this Field_row::sql_type_for_sp_returns() > +{ > + res.set_ascii(STRING_WITH_LEN("row(")); > + for (uint i= 0; i < m_table->s->fields; i++) > + { > + if (i > 0) > + res.append(','); > + Field *field= m_table->field[i]; > + res.append(field->field_name); > + res.append(' '); > + StringBuffer<64> col; > + field->sql_type_for_sp_returns(col); > + res.append(col.to_lex_cstring()); > + } > + res.append(')'); > +} > + > + > /**************************************************************************** > Functions for the Field_decimal class > This is an number stored as a pre-space (or pre-zero) string > diff --git a/sql/item.cc b/sql/item.cc > index 54ad511cb6b..0a125752979 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -3001,10 +3003,31 @@ Item_sp::init_result_field(THD *thd, uint max_length, uint maybe_null, > dummy_table->s->table_name= empty_clex_str; > dummy_table->maybe_null= maybe_null; > > + if (m_sp->m_return_field_def.is_column_type_ref() && > + m_sp->m_return_field_def.column_type_ref()-> > + resolve_type_ref(thd, &m_sp->m_return_field_def)) > + DBUG_RETURN(TRUE); Here you overwrite old m_return_field_def value which was a column_type_ref with its actual type, don't you? What will happen if you change the column's type after the sp was already executed once? > + > if (!(sp_result_field= m_sp->create_result_field(max_length, name, > dummy_table))) > DBUG_RETURN(TRUE); > > + /* > + In case of a ROW return type we need to create Item_field_row > + on top of Field_row, and remember it in sp_result_item_field_row. > + ROW members are later accessed using sp_result_item_field_row->addr(i), > + e.g. when copying the function return value to a local variable. > + For scalar return types no Item_field is needed around sp_result_field, > + as the value is fetched directly from sp_result_field, > + inside Item_func_sp::val_xxx() methods. Sorry, I don't understand that. Why cannot you use Field_row here? > + */ > + if (Field_row *field_row= dynamic_cast<Field_row*>(sp_result_field)) > + { > + if (!(sp_result_item_field_row= > + m_sp->m_return_field_def.make_item_field_row(thd, field_row))) > + DBUG_RETURN(true); > + } > + > if (sp_result_field->pack_length() > sizeof(result_buf)) > { > void *tmp; > diff --git a/sql/sp.cc b/sql/sp.cc > index ddaeeb8cb7a..7eab8304c31 100644 > --- a/sql/sp.cc > +++ b/sql/sp.cc > @@ -1091,18 +1136,19 @@ sp_returns_type(THD *thd, String &result, const sp_head *sp) > bzero((char*) &share, sizeof(share)); > table.in_use= thd; > table.s = &share; > - field= sp->create_result_field(0, 0, &table); > - field->sql_type(result); > + field= create_result_field(0, 0, &table); > > - if (field->has_charset()) > + if (m_return_field_def.is_row()) > { > - result.append(STRING_WITH_LEN(" CHARSET ")); > - result.append(field->charset()->cs_name); > - if (Charset(field->charset()).can_have_collate_clause()) > - { > - result.append(STRING_WITH_LEN(" COLLATE ")); > - result.append(field->charset()->coll_name); > - } > + Field_row *field_row= dynamic_cast<Field_row*>(field); > + if (!field_row->row_create_fields( > + thd, m_return_field_def.row_field_definitions())) > + field->sql_type(result); when is this used? you're creating a TABLE and Field's - only to print types and charsets. This might be tolerable, but only if it's not needed often > + } > + else > + { > + DBUG_ASSERT(m_return_field_def.type_handler()->is_scalar_type()); > + field->sql_type_for_sp_returns(result); > } > > delete field; > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 5977e309ae4..f11e4f89466 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -18347,7 +18338,50 @@ sp_parameters: > ; > > sf_returned_type_clause: > - RETURNS_SYM sf_return_type > + RETURNS_SYM > + { > + LEX *lex= Lex; you still do this lex=Lex? :) it became obsolete, like, 15 years ago. If not more. > + lex->init_last_field(&lex->sphead->m_return_field_def, > + &empty_clex_str); > + } > + sf_return_type > + ; > + > +sf_return_type: > + field_type > + { > + if (unlikely(Lex->sf_return_fill_definition($1))) > + MYSQL_YYABORT; > + } > + | ROW_SYM row_type_body > + { > + if (Lex->sf_return_fill_definition_row($2)) > + MYSQL_YYABORT; > + } > + | ROW_SYM TYPE_SYM OF_SYM ident > + { > + if (Lex->sf_return_fill_definition_rowtype_of( > + Qualified_column_ident(&$4))) > + MYSQL_YYABORT; > + } > + | ROW_SYM TYPE_SYM OF_SYM ident '.' ident > + { > + if (Lex->sf_return_fill_definition_rowtype_of( > + Qualified_column_ident(&$4, &$6))) > + MYSQL_YYABORT; > + } > + | TYPE_SYM OF_SYM ident '.' ident > + { > + if (Lex->sf_return_fill_definition_type_of( > + Qualified_column_ident(&$3, &$5))) > + MYSQL_YYABORT; > + } > + | TYPE_SYM OF_SYM ident '.' ident '.' ident > + { > + if (Lex->sf_return_fill_definition_type_of( > + Qualified_column_ident(thd, &$3, &$5, &$7))) > + MYSQL_YYABORT; > + } > ; > > package_implementation_item_declaration:

2 1

Clean up executable test result files:
by Otto Kekäläinen 04 Sep '24

04 Sep '24

Hi! In https://github.com/MariaDB/server/commit/9cb75f333fa2ae816909f95a4e19e6ddac… Alexander made some test result files executable (look at 100644 → 100755 in git diff). I sent twice a PR to fix this: - Oct 2023: https://github.com/MariaDB/server/pull/2779 - Feb 2024: https://github.com/MariaDB/server/pull/3042 Unfortunately, I can see that on the main branch this is executable again: mariadb-server main+ ± find mysql-test -executable -name *.test -or -executable -name *.result mysql-test/main/lowercase_table2.result Could the next person doing merges fix this tiny annoyance once and for all? Thanks :)

1 0

Re: [PATCH] MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction
by Kristian Nielsen 03 Sep '24

03 Sep '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> So please do a small refactoring of trx_group_commit_leader() and keep >> only ... > > and no need for refactoring really arises. Well, the refactoring is good, even without the MDEV-32014 patch. The function trx_group_commit_leader() is already too long and complicated, and does too much in a single function. And the first part (obtaining the queue) is cleanly separated from the second part (committing each entry in the queue in-order in binlog and participating engines). It's just nice that this improvement of the existing code also makes the MDEV-32014 patch integrate cleaner. > I noted earlier on this part on the PR page and LiBing seems to have > practically proven (by the most recent commit that you must've missed) that > conducting the rename inside the flush stage of > trx_group_commit_leader() is viable: > > https://github.com/MariaDB/server/pull/3402/commits/287511a703f7dff47939633… > > By that measure the flushing is expanded onto possibly multiple "implicitly" > rotating log files Hm, I don't like the change 287511a703f7dff47939633475edd436a542cf6f. In fact, that is one of the things I like the most about Libing Song's patch, that it so cleanly separates the commit-by-rotate logic from the complex group commit logic in queue_for_group_commit() and trx_group_commit_leader(), which is already too complex. The original patch without 287511a703f7dff47939633475edd436a542cf6f seems much cleaner. It is better to do as in the original patch, and treat commit-by-rotate like a normal binlog rotate, not try to participate that transaction in binlog group commit. We cannot share the fsync anyway, as the files are different. And it introduces a lot of complexities to suddenly have binlog rotations in the middle of group commit - what about the complex accounting around xid_count and binlog checkpoints? There will be a lot of corner cases eg. if _two_ commit-by-rotate happens in the same group commit, with lots of potential for subtle bugs that will occur only very rarely in production and be almost impossible to track down. I strongly prefer the original patch I reviewed (with my suggested changes to write_transaction_to_binlog_events() and trx_group_commit_leader()), where the commit-by-rotate logic is cleanly separated from the group commit logic and implemented mostly using the existing binlog rotate logic. - Kristian.

2 2

Re: [PATCH] MDEV-34857: Implement --slave-abort-blocking-timeout
by Michael Widenius 03 Sep '24

03 Sep '24

Review of MDEV-34857: Implement --slave-abort-blocking-timeout If a slave replicating an event has waited for more than @@slave_abort_blocking_timeout for a conflicting metadata lock held by a non-replication thread, the blocking query is killed to allow replication to proceed and not be blocked indefinitely by a user query. <cut> diff --git a/sql/mdl.cc b/sql/mdl.cc index faccd1c9476..9845718e165 100644 --- a/sql/mdl.cc +++ b/sql/mdl.cc <cut> @@ -2397,14 +2398,39 @@ MDL_context::acquire_lock(MDL_request *mdl_request, double lock_wait_timeout) find_deadlock(); - struct timespec abs_timeout, abs_shortwait; + struct timespec abs_timeout, abs_shortwait, abs_abort_blocking_timeout; + bool abort_blocking_enabled= false; + double abort_blocking_timeout= slave_abort_blocking_timeout; Add a comment here. Something like /* This code implements aborting of queries on the slaves that stops replication from continuing. */ + if (abort_blocking_timeout < lock_wait_timeout && + m_owner->get_thd()->rgi_slave) + { + set_timespec_nsec(abs_abort_blocking_timeout, + (ulonglong)(abort_blocking_timeout * 1000000000ULL)); + abort_blocking_enabled= true; + } set_timespec_nsec(abs_timeout, (ulonglong)(lock_wait_timeout * 1000000000ULL)); - set_timespec(abs_shortwait, 1); wait_status= MDL_wait::EMPTY; - while (cmp_timespec(abs_shortwait, abs_timeout) <= 0) + for (;;) { + bool abort_blocking= false; + set_timespec(abs_shortwait, 1); + if (abort_blocking_enabled && + cmp_timespec(abs_shortwait, abs_abort_blocking_timeout) >= 0) + { + /* + If a slave DDL has waited for --slave-abort-select-timeout, then notify + any blocking SELECT once before continuing to wait until the full + timeout. + */ + abs_shortwait= abs_abort_blocking_timeout; + abort_blocking= true; + abort_blocking_enabled= false; + } + if (cmp_timespec(abs_shortwait, abs_timeout) > 0) + break; Is the above right last compare right? Assuming that cmp_timespec(abs_shortwait, abs_abort_blocking_timeout) > 0 then the above will not break and we will not signal other threads to break even if the previous code expectrs us to abort blocking. (I think the cmp_timespec() return test should be the same in both cases) Shouldn't the code be: for (;;) { bool abort_blocking= false; set_timespec(abs_shortwait, 1); if (abort_blocking_enabled && cmp_timespec(abs_shortwait, abs_abort_blocking_timeout) >= 0) { /* If a slave DDL has waited for --slave-abort-select-timeout, then notify any blocking SELECT once before continuing to wait until the full timeout. */ abort_blocking= true; abort_blocking_enabled= false; } else if (cmp_timespec(abs_shortwait, abs_timeout) > 0) break; Or have I missed something? /* abs_timeout is far away. Wait a short while and notify locks. */ wait_status= m_wait.timed_wait(m_owner, &abs_shortwait, FALSE, mdl_request->key.get_wait_state_name()); @@ -2425,9 +2451,8 @@ MDL_context::acquire_lock(MDL_request *mdl_request, double lock_wait_timeout) mysql_prlock_wrlock(&lock->m_rwlock); if (lock->needs_notification(ticket)) - lock->notify_conflicting_locks(this); + lock->notify_conflicting_locks(this, abort_blocking); mysql_prlock_unlock(&lock->m_rwlock); - set_timespec(abs_shortwait, 1); I like the new code in that we are not setting abs_shortwait two times anymore. (Using a while loop in the is case was sub optimal. Ok to push after the above change or if you have a good argument why the current code is correct

1 0

Re: [PATCH] MDEV-32014 Rename binlog cache temporary file to binlog file for large transaction
by andrei.elkin＠pp.inet.fi 02 Sep '24

02 Sep '24

(Re-sending to the list) Howdy Kristian. > > The patch calls the trx_group_commit_leader() function. That is ackward, as > commit-by-rotate does not actually participate in group commit. But the > problem is with the existing function trx_group_commit_leader(), which does > too much in a single function. I noted earlier on this part on the PR page and LiBing seems to have practically proven (by the most recent commit that you must've missed) that conducting the rename inside the flush stage of trx_group_commit_leader() is viable: https://github.com/MariaDB/server/pull/3402/commits/287511a703f7dff47939633… By that measure the flushing is expanded onto possibly multiple "implicitly" rotating log files > > So please do a small refactoring of trx_group_commit_leader() and keep > only ... and no need for refactoring really arises. Could you please assess that approach, cheers, Andrei

1 0

Next LTS version in February 2025?
by Otto Kekäläinen 30 Aug '24

30 Aug '24

Hi! Traditionally there has been a new major MariaDB version in February of the year Debian has a new major version. What is the plan for February 2025? I was reading https://mariadb.org/about/#maintenance-policy and old mariadb.orb blog posts but none was explicit about this.

2 2

Re: fec3696fef6: MDEV-15751 CURRENT_TIMESTAMP should return a TIMESTAMP [WITH TIME ZONE?]
by Sergei Golubchik 29 Aug '24

29 Aug '24

Hi, Alexander, This has waited far too long :( > commit fec3696fef6 > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Tue Feb 14 13:27:46 2023 +0400 > > diff --git a/sql/sql_class.h b/sql/sql_class.h > index f0bf695a1c1..7378225eaa2 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3900,7 +3900,8 @@ class THD: public THD_count, /* this must be first */ > } > const Type_handler *type_handler_for_datetime() const; > bool timestamp_to_TIME(MYSQL_TIME *ltime, my_time_t ts, > - ulong sec_part, date_mode_t fuzzydate); > + ulong sec_part, date_mode_t fuzzydate, > + bool zero_timesteamp_means_zero_datetime); I don't understand it. NOW() cannot return a zero timestamp, how this feature can possibly need changes to zero timestamp treatment? > inline my_time_t query_start() { return start_time; } > inline ulong query_start_sec_part() > { used|= QUERY_START_SEC_PART_USED; return start_time_sec_part; } > diff --git a/sql/item_timefunc.h b/sql/item_timefunc.h > index a5f6d9307c6..1c463c8a332 100644 > --- a/sql/item_timefunc.h > +++ b/sql/item_timefunc.h > @@ -707,6 +707,49 @@ class Item_datetimefunc :public Item_func > }; > > > +class Item_timestampfunc: public Item_func > +{ > +protected: > + Datetime to_datetime(THD *thd) > + { > + return Timestamp_or_zero_datetime_native_null(thd, this).to_datetime(thd); how does it differ from Datetime(this) ? > + } > +public: > + Item_timestampfunc(THD *thd): Item_func(thd) {} > + Item_timestampfunc(THD *thd, Item *a): Item_func(thd, a) {} > + const Type_handler *type_handler() const override > + { return &type_handler_timestamp2; } > + bool get_date(THD *thd, MYSQL_TIME *ltime, date_mode_t fuzzydate) override > + { > + return null_value= to_datetime(thd).copy_to_mysql_time(ltime); > + } > + double val_real() override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_double(); may be `return null_value ? 0 : dt.to_double()` ? why to call to_double if it isn't a valid datetime? (same everywhere below) > + } > + longlong val_int() override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_longlong(); > + } > + my_decimal *val_decimal(my_decimal *to) override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_decimal(to); > + } > + String *val_str(String *to) override > + { > + Datetime dt= to_datetime(current_thd); > + null_value= !dt.is_valid_datetime(); > + return dt.to_string(to, decimals); > + } > +}; > + > + > /* Abstract CURTIME function. Children should define what time zone is used */ > > class Item_func_curtime :public Item_timefunc > @@ -837,20 +880,48 @@ class Item_func_now :public Item_datetimefunc > }; > > > -class Item_func_now_local :public Item_func_now > +/* > + This is a replacement for Item_func_now_local, > + returning TIMESTAMP instead of DATETIME. why did you rename it? why didn't you change Item_func_now_utc? > +*/ > +class Item_func_current_timestamp: public Item_timestampfunc > { > public: > - Item_func_now_local(THD *thd, uint dec): Item_func_now(thd, dec) {} > + Item_func_current_timestamp(THD *thd, uint dec) > + :Item_timestampfunc(thd) > + { decimals= dec; } why not Item_timestampfunc(thd, dec) ? > LEX_CSTRING func_name_cstring() const override > { > static LEX_CSTRING name= {STRING_WITH_LEN("current_timestamp") }; > return name; > } > - int save_in_field(Field *field, bool no_conversions) override; > - void store_now_in_TIME(THD *thd, MYSQL_TIME *now_time) override; > + void print(String *str, enum_query_type query_type) override > + { > + str->append(func_name_cstring()); > + str->append('('); > + if (decimals) > + str->append_ulonglong(decimals); > + str->append(')'); > + } > + bool fix_length_and_dec(THD *thd) override > + { > + if (check_decimal_scale_or_error(TIME_SECOND_PART_DIGITS)) > + return true; > + fix_attributes_datetime(decimals); > + return false; > + } > + bool val_native(THD *thd, Native *to) override; > + bool check_vcol_func_processor(void *arg) override > + { > + /* > + NOW is safe for replication as slaves will run with same time as > + master > + */ what does this replication comment do in check_vcol_func_processor? (I know it was in the old code too, but "old code did it" never stopped your cleanups before :) > + return mark_unsupported_function(func_name(), "()", arg, VCOL_TIME_FUNC); > + } > enum Functype functype() const override { return NOW_FUNC; } > Item *get_copy(THD *thd) override > - { return get_item_copy<Item_func_now_local>(thd, this); } > + { return get_item_copy<Item_func_current_timestamp>(thd, this); } > }; > > > diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc > index 52b9ae7a682..3e224e3e6f7 100644 > --- a/sql/item_timefunc.cc > +++ b/sql/item_timefunc.cc > @@ -1642,32 +1633,10 @@ void Item_func_now::print(String *str, enum_query_type query_type) > } > > > -int Item_func_now_local::save_in_field(Field *field, bool no_conversions) > +bool Item_func_current_timestamp::val_native(THD *thd, Native *to) > { > - if (field->type() == MYSQL_TYPE_TIMESTAMP) > - { > - THD *thd= field->get_thd(); > - my_time_t ts= thd->query_start(); > - ulong sec_part= decimals ? thd->query_start_sec_part() : 0; > - sec_part-= my_time_fraction_remainder(sec_part, decimals); > - field->set_notnull(); > - field->store_timestamp(ts, sec_part); > - return 0; > - } > - else > - return Item_datetimefunc::save_in_field(field, no_conversions); > -} > - > - > -/** > - Converts current time in my_time_t to MYSQL_TIME representation for local > - time zone. Defines time zone (local) used for whole NOW function. > -*/ > -void Item_func_now_local::store_now_in_TIME(THD *thd, MYSQL_TIME *now_time) > -{ > - thd->variables.time_zone->gmt_sec_to_TIME(now_time, thd->query_start()); > - set_sec_part(thd->query_start_sec_part(), now_time, this); > - thd->used|= THD::TIME_ZONE_USED; > + Timestamp ts(Timeval(thd->query_start(), thd->query_start_sec_part())); > + return null_value= ts.trunc(decimals).to_native(to, decimals); can this possibly fail? > } > > > @@ -2732,7 +2693,6 @@ String *Item_func_tochar::val_str(String* str) > > bool Item_func_from_unixtime::fix_length_and_dec(THD *thd) > { > - thd->used|= THD::TIME_ZONE_USED; would be nice to have a test case for this. Like direct insert into a timestamp column no longer uses timezone - e.g. timestamp within dst switch time is inserted correctly. and vice versa, when from_unixtime is converted to a string, like `insert ... values(concat(from_unixtime(...)))` then TIME_ZONE_USED is still set and show binlog events (or mysqlbinlog) should show that the timezone was correctly binlogged, as before. > tz= thd->variables.time_zone; > Type_std_attributes::set( > Type_temporal_attributes_not_fixed_dec(MAX_DATETIME_WIDTH, Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

MariaDB patches in Debian
by Otto Kekäläinen 27 Aug '24

27 Aug '24

Hi! Could perhaps Tuukka or Daniel review the patches currently carried in Debian if some of them would be applicable to be imported or re-implemented directly upstream so they can be dropped in Debian? https://udd.debian.org/patches.cgi?src=mariadb&version=1%3A11.4.3-1 Thanks

1 0

Re: [MariaDB commits] [PATCH 1/6] MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates
by andrei.elkin＠pp.inet.fi 23 Aug '24

23 Aug '24

Howdy Kristian. > Before doing mark_start_commit(), check that there is no pending deadlock > kill. If there is a pending kill, we won't commit (we will abort, roll back, > and retry). Then we should not mark the commit as started, since that could > potentially make the following GCO start too early, before we completed the > commit after the retry. > > This condition could trigger in some corner cases, where InnoDB would take > temporarily table/row locks that are released again immediately, not held > until the transaction commits. This happens with dict_stats updates and > possibly auto-increment locks. > > Such locks can be passed to thd_rpl_deadlock_check() and cause a deadlock > kill to be scheduled in the background. But since the blocking locks are > held only temporarily, they can be released before the background kill > happens. This way, the kill can be delayed until after mark_start_commit() > has been called. Thus we need to check the synchronous indication > rgi->killed_for_retry, not just the asynchroneous thd->killed. I think I understood what's going on, also thanks to a verbose Jira ticket description. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > --- > sql/rpl_parallel.cc | 20 ++++++++++++++++---- > sql/rpl_rli.cc | 17 +++++++++++++++++ > 2 files changed, 33 insertions(+), 4 deletions(-) > > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 1cfdf96ee3b..9c4222d7817 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -1450,11 +1450,23 @@ handle_rpl_parallel_thread(void *arg) > after mark_start_commit(), we have to unmark, which has at least a > theoretical possibility of leaving a window where it looks like all > transactions in a GCO have started committing, while in fact one > - will need to rollback and retry. This is not supposed to be possible > - (since there is a deadlock, at least one transaction should be > - blocked from reaching commit), but this seems a fragile ensurance, > - and there were historically a number of subtle bugs in this area. > + will need to rollback and retry. > + > + Normally this will not happen, since the kill is there to resolve a > + deadlock that is preventing at least one transaction from proceeding. > + One case it can happen is with InnoDB dict stats update, which can > + temporarily cause transactions to block each other, but locks are > + released immediately, they don't linger until commit. There could be > + other similar cases, there were historically a number of subtle bugs > + in this area. > + > + But once we start the commit, we can expect that no new lock > + conflicts will be introduced. So by handling any lingering deadlock > + kill at this point just before mark_start_commit(), we should be > + robust even towards spurious deadlock kills. > */ > + if (rgi->killed_for_retry != rpl_group_info::RETRY_KILL_NONE) > + wait_for_pending_deadlock_kill(thd, rgi); Assuming my understanding, please correct me if anything, I left a question on this block in https://github.com/MariaDB/server/commit/df0c36a354ffde2766ebed2615642c74b7… quoted further. ... this guard would not let a victim of an optimistic conflict proceed until it clears out itself it is such a victim. For instance in the binlog sequence like T_1, T_2, D_3 where T_2 is the victim, D_3 is a DDL, the victim T_2 won't anymore release the next gco member D_3 into its execution. But what if at this point T_2 is only going to be marked in rgi->killed_for_retry by T_1? That is it is apparently able, with this patch as well, having ...NONE killed status go through. T_1 would reach this very point later and when that is done before T_2 finds itself killed, T_1 would release D_3, and afterward T_2 would finally see itself KILLed into retry. Cheers, Andrei

2 4

Open PRs pending second approver (has already one)
by Otto Kekäläinen 23 Aug '24

23 Aug '24

Hi! There are couple PRs that have 1 approval, and are pending the second one. Would somebody have time to review them? In order of importance: https://github.com/MariaDB/server/pull/3401 "MDEV-34680 Asynchronous and Buffered Logging in MySQL Audit Plugin" https://github.com/MariaDB/server/pull/3406 (related but zero approvals: https://github.com/MariaDB/server/pull/3324 "MDEV-12182 Add client tcp port to audit plugin logs") https://github.com/MariaDB/server/pull/3208 "Refine OpenSSL test names and allow more complete OpenSSL version coverage" https://github.com/MariaDB/server/pull/3225 "MDEV-29955: add ${PCRE_LIBRARY_PATH} to link directories" Thanks, Otto

1 0

Re: 186aa192d1f: MDEV-34704 Quick mode produces the bug for mariadb client
by Sergei Golubchik 22 Aug '24

22 Aug '24

Hi, Sanja, ok to push, but see one comment below On Aug 22, Oleksandr Byelkin wrote: > revision-id: 186aa192d1f (mariadb-11.5.1-34-g186aa192d1f) > parent(s): 9e1923ca238 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2024-08-05 16:16:09 +0200 > message: > > MDEV-34704 Quick mode produces the bug for mariadb client > > --quick-max-column-width parameter added to limit field > width in --quick mode. > > diff --git a/mysql-test/main/client.test b/mysql-test/main/client.test > new file mode 100644 > index 00000000000..59e687ff8b9 > --- /dev/null > +++ b/mysql-test/main/client.test > @@ -0,0 +1,33 @@ > +--echo # > +--echo # MDEV-34704: Quick mode produces the bug for mariadb client > +--echo # > + > + > +create table t1 (aaaaaaaaa char (5), aaaaa char (10), a char (127), b char(1)); > +insert into t1 values ("X", "X", "X", "X"); > + to make the result file more readable, I'd suggest to add here, like --echo mysql --table --quick ... --echo mysql --table --quick --quick-max-column-width=0 ... etc before every corresponding --exec > +--exec echo "select * from test.t1;" | $MYSQL --table --quick 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=0 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=10 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=20 2>&1 > + > +insert into t1 values ("01234", "0123456789", "01234567890123456789", "1"); > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick 2>&1 > + > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=0 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=10 2>&1 > + > +--exec echo "select * from test.t1;" | $MYSQL --table --quick --quick-max-column-width=20 2>&1 > + > +drop table t1; > + > +--echo # > +--echo # End of 10.7 tests > +--echo # As Marko said, use here only one line --echo # End of 10.7 tests otherwise merge mixes it up with --echo # --echo # MDEV-34704: Quick mode produces the bug for mariadb client --echo # Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Suggestion: use 'main' as the development branch instead of an ever changing 11.x
by Otto Kekäläinen 22 Aug '24

22 Aug '24

Hi! Has the core developers had discussions about using branch name 'main' for development instead of switching to a new branch every 3 months? If all new features and other additions would always target the branch 'main' it would lower the barrier of entry for new contributors, and decrease the number of unnecessary rebases done by contributors. If the permanent development target was 'main', you would need to create new 11.x branches only at "freeze" when a new major version branch is cut for release. MariaDB is currently the only open source project I know that does *not* have a main/master/trunk branch and instead does these constant new branch names. I imagine this is not only annoying for contributors, but also the core developers who waste time asking contributors to rebase PRs every 3 months (see https://github.com/MariaDB/server/pull/2556 and https://github.com/MariaDB/server/pull/2671 as examples). - Otto

7 11

Re: [MariaDB commits] [PATCH 3/6] Fix sporadic test failure in rpl.rpl_create_drop_event
by Kristian Nielsen 22 Aug '24

22 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Kristian Nielsen via developers <developers(a)lists.mariadb.org> writes: >> The already-started instance of the event happily continues to run after >> being dropped. I wonder if that's a bug, or if the documentation is >> wrong? > > To this question, I vaguely remember discussions about this matter which > must've concuded in that it's not a bug when the scheduler races ahead > of DROP EVENT to launch a piece of job, like INSERT in this case. Aha, I see. Following up on this, I found this comment in the code Stops the scheduler (again). Waits for acknowledgement from the scheduler that it has stopped - synchronous stopping. Already running events will not be stopped. If the user needs them stopped manual intervention is needed. So then the behaviour seen in the test case is expected, and just fixing the test to handle the possible race should be correct. I also see in the code that the thread running the events starts by loading the event definition from the system tables (mysql.event ...), db_repository->load_named_event(). So there should be no memory issues even if the event is dropped while an instance of it is running. Thanks! - Kristian.

1 0

Re: [MariaDB commits] [PATCH 3/6] Fix sporadic test failure in rpl.rpl_create_drop_event
by Kristian Nielsen 21 Aug '24

21 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> My question is if it is ok and by design that the event can continue running >> after the scheduler is stopped and even after the event is dropped? That > Should not be ok. The docs https://mariadb.com/kb/en/drop-event/ > > are explicit > > Description > This statement drops the event named event_name. The event immediately ceases being active, and is deleted completely from the server. Thanks for the link, Andrei! Interesting, this doesn't seem to be how it works. This small test starts an event that runs for 3 seconds, but drops the event after 1 second: ----------------------------------------------------------------------- SET GLOBAL event_scheduler= OFF; CREATE TABLE t1(a INT); CREATE EVENT ev1 ON SCHEDULE EVERY 5 SECOND DO INSERT INTO t1 VALUES(SLEEP(3)); SET GLOBAL event_scheduler= ON; --sleep 1 SHOW FULL PROCESSLIST; SET GLOBAL event_scheduler= OFF; DROP EVENT ev1; SHOW FULL PROCESSLIST; --sleep 1 SHOW FULL PROCESSLIST; DROP TABLE t1; ----------------------------------------------------------------------- The already-started instance of the event happily continues to run after being dropped. I wonder if that's a bug, or if the documentation is wrong? And if it is a bug, if there are any use-after-free or similar memory issues with the event continuing to run after being dropped? Valgrind doesn't complain... ----------------------------------------------------------------------- SET GLOBAL event_scheduler= OFF; CREATE TABLE t1(a INT); CREATE EVENT ev1 ON SCHEDULE EVERY 5 SECOND DO INSERT INTO t1 VALUES(SLEEP(3)); Warnings: Warning 1105 Event scheduler is switched off, use SET GLOBAL event_scheduler=ON to enable it. SET GLOBAL event_scheduler= ON; SHOW FULL PROCESSLIST; Id User Host db Command Time State Info Progress 4 root localhost test Query 0 starting SHOW FULL PROCESSLIST 0.000 5 event_scheduler localhost NULL Daemon 1 Waiting for next activation NULL 0.000 6 root localhost test Connect 1 User sleep INSERT INTO t1 VALUES(SLEEP(3)) 0.000 SET GLOBAL event_scheduler= OFF; DROP EVENT ev1; SHOW FULL PROCESSLIST; Id User Host db Command Time State Info Progress 4 root localhost test Query 0 starting SHOW FULL PROCESSLIST 0.000 6 root localhost test Connect 1 User sleep INSERT INTO t1 VALUES(SLEEP(3)) 0.000 SHOW FULL PROCESSLIST; Id User Host db Command Time State Info Progress 4 root localhost test Query 0 starting SHOW FULL PROCESSLIST 0.000 6 root localhost test Connect 2 User sleep INSERT INTO t1 VALUES(SLEEP(3)) 0.000 DROP TABLE t1; ----------------------------------------------------------------------- - Kristian.

1 0

Re: [PATCH 1/6] MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates
by Kristian Nielsen 20 Aug '24

20 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Howdy Kristian. > I was going to review. Thanks Andrei! > Is the patch is any github branch? Looks so by the MDEV. It'd be good to > have it there for reviewing either. Yes, the entire patch series is in bb-10.5-knielsen: https://github.com/MariaDB/server/commits/bb-10.5-knielsen This particular issue is the commit: MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates - Kristian.

1 0

Re: [MariaDB commits] [PATCH 3/6] Fix sporadic test failure in rpl.rpl_create_drop_event
by Kristian Nielsen 20 Aug '24

20 Aug '24

Hi Bar, I want to ask you about this fix of a sporadic failure of the test rpl.rpl_create_drop_event, which is authored by you. Not so much the test case fix, which should be good, but I want to ask someone who knows the event scheduler code better than me to be sure I am not merely hiding a real bug with the test case fix. What happens in the test case when it failed is that a new run of an event is started, and this run can continue while the event scheduler is stopped and even after the event itself is dropped: CREATE OR REPLACE EVENT ev1 ON SCHEDULE EVERY 1 SECOND DO INSERT INTO t1 VALUES (11); ... SET GLOBAL event_scheduler=off; DROP EVENT ev1; My question is if it is ok and by design that the event can continue running after the scheduler is stopped and even after the event is dropped? That this doesn't cause any problems like accessing freed memory or such? Or if you do not know, if you know who else I could ask? Thanks, - Kristian. Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > Depending on timing, an extra event run could start just when the event > scheduler is shut down and delay running until after the table has been > dropped; this would cause the test to fail with a "table does not exist" > error in the log. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > --- > mysql-test/suite/rpl/t/rpl_create_drop_event.test | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/mysql-test/suite/rpl/t/rpl_create_drop_event.test b/mysql-test/suite/rpl/t/rpl_create_drop_event.test > index 96a7e82d6f7..79bb0ffec90 100644 > --- a/mysql-test/suite/rpl/t/rpl_create_drop_event.test > +++ b/mysql-test/suite/rpl/t/rpl_create_drop_event.test > @@ -14,6 +14,12 @@ SET GLOBAL event_scheduler=on; > let $wait_condition= SELECT count(*)>0 FROM t1; > --source include/wait_condition.inc > SET GLOBAL event_scheduler=off; > +# If the time rolls to the next whole second just at this point, a new event > +# run may be scheduled. Wait for this to disappear, otherwise we see occasional > +# test failures if the table gets dropped before the extra event run completes. > +# Expect 5 connections: default, master, master1, server_1, binlog dump thread > +--let $wait_condition= SELECT COUNT(*) = 5 FROM INFORMATION_SCHEMA.PROCESSLIST; > +--source include/wait_condition.inc > SELECT DISTINCT a FROM t1; > DELETE FROM t1;

1 0

Re: [PATCH 1/6] MDEV-34696: do_gco_wait() completes too early on InnoDB dict stats updates
by Kristian Nielsen 20 Aug '24

20 Aug '24

Hi Brandon, Andrei, Does one of you want to review this patch for 10.5? It fixes a real (and serious) issue in parallel replication that is seen as sporadic failure of rpl.rpl_start_alter_4 in Buildbot. This is yet another occurrence of the problem where mark_start_commit() is done too early, this code unfortunately is fragile and has had a number of problems in the past. But I am quite confident about the patch, calling wait_for_pending_deadlock_kill() should be very safe, and should be correct here. The added assertion helps detect the problem more reliably and could also help catch any other remaining issues around mark_start_commit (though hopefully there are no more left now). - Kristian. Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > Before doing mark_start_commit(), check that there is no pending deadlock > kill. If there is a pending kill, we won't commit (we will abort, roll back, > and retry). Then we should not mark the commit as started, since that could > potentially make the following GCO start too early, before we completed the > commit after the retry. > > This condition could trigger in some corner cases, where InnoDB would take > temporarily table/row locks that are released again immediately, not held > until the transaction commits. This happens with dict_stats updates and > possibly auto-increment locks. > > Such locks can be passed to thd_rpl_deadlock_check() and cause a deadlock > kill to be scheduled in the background. But since the blocking locks are > held only temporarily, they can be released before the background kill > happens. This way, the kill can be delayed until after mark_start_commit() > has been called. Thus we need to check the synchronous indication > rgi->killed_for_retry, not just the asynchroneous thd->killed. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > --- > sql/rpl_parallel.cc | 20 ++++++++++++++++---- > sql/rpl_rli.cc | 17 +++++++++++++++++ > 2 files changed, 33 insertions(+), 4 deletions(-) > > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 1cfdf96ee3b..9c4222d7817 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -1450,11 +1450,23 @@ handle_rpl_parallel_thread(void *arg) > after mark_start_commit(), we have to unmark, which has at least a > theoretical possibility of leaving a window where it looks like all > transactions in a GCO have started committing, while in fact one > - will need to rollback and retry. This is not supposed to be possible > - (since there is a deadlock, at least one transaction should be > - blocked from reaching commit), but this seems a fragile ensurance, > - and there were historically a number of subtle bugs in this area. > + will need to rollback and retry. > + > + Normally this will not happen, since the kill is there to resolve a > + deadlock that is preventing at least one transaction from proceeding. > + One case it can happen is with InnoDB dict stats update, which can > + temporarily cause transactions to block each other, but locks are > + released immediately, they don't linger until commit. There could be > + other similar cases, there were historically a number of subtle bugs > + in this area. > + > + But once we start the commit, we can expect that no new lock > + conflicts will be introduced. So by handling any lingering deadlock > + kill at this point just before mark_start_commit(), we should be > + robust even towards spurious deadlock kills. > */ > + if (rgi->killed_for_retry != rpl_group_info::RETRY_KILL_NONE) > + wait_for_pending_deadlock_kill(thd, rgi); > if (!thd->killed) > { > DEBUG_SYNC(thd, "rpl_parallel_before_mark_start_commit"); > diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc > index 9d4e09a362c..8284b07f7ff 100644 > --- a/sql/rpl_rli.cc > +++ b/sql/rpl_rli.cc > @@ -2518,6 +2518,23 @@ rpl_group_info::unmark_start_commit() > > e= this->parallel_entry; > mysql_mutex_lock(&e->LOCK_parallel_entry); > + /* > + Assert that we have not already wrongly completed this GCO and signalled > + the next one to start, only to now unmark and make the signal invalid. > + This is to catch problems like MDEV-34696. > + > + The error inject rpl_parallel_simulate_temp_err_xid is used to test this > + precise situation, that we handle it gracefully if it somehow occurs in a > + release build. So disable the assert in this case. > + */ > +#ifndef DBUG_OFF > + bool allow_unmark_after_complete= false; > + DBUG_EXECUTE_IF("rpl_parallel_simulate_temp_err_xid", > + allow_unmark_after_complete= true;); > + DBUG_ASSERT(!gco->next_gco || > + gco->next_gco->wait_count > e->count_committing_event_groups || > + allow_unmark_after_complete); > +#endif > --e->count_committing_event_groups; > mysql_mutex_unlock(&e->LOCK_parallel_entry); > }

1 0

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 15 Aug '24

15 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> This is not the way to communicate back from engine to server. Instead, >> simply return a different error code than HA_ERR_LOCK_WAIT_TIMEOUT from >> InnoDB that you can check for here. You can maybe use one of the existing >> HA_ERR_* from include/my_base.h, or add a new one if necessary. > However, adding a new error is clearly more intrusive. The new error is completely handled inside the code you add for signalling the condition inside InnoDB locking code and handling it in replication row event apply. It will not be visible to users. So it is not intrusive to add a new one. Just be careful to not pick a number that conflicts with another entry in a later branch, if you do this in a GA release. Alternatively, you can use the existing HA_ERR_ROW_NOT_VISIBLE error, which is used in a somewhat similar way in storage/maria/ma_blockrec.c and nowhere else. But I would just add a new error for clarity. > And at the same time I thought a some sort of runtime state the slave > applier had always missed. So the intent of this struct is future-minded > as well. This is not the point. Even if the rgi->exec_flags already existed, it would still be wrong to use it for this. Here is the code where you are handling the condition: > + if (error == HA_ERR_LOCK_WAIT_TIMEOUT && !is_index_unique && > + (rgi->exec_flags & (1 << rpl_group_info::HIT_BUSY_INDEX))) > + { > + rgi->exec_flags &= ~(1 << rpl_group_info::HIT_BUSY_INDEX); > + skip_first_compare= true; So you _already_ are using a specific error code to signal the condition from InnoDB to replication, the error HA_ERR_LOCK_WAIT_TIMEOUT. But this error can also be thrown in other situations, so you need to add these extra conditions to distinguish. So essentially, you are using the wrong error code to signal the condition. Use a correct error code that uniquely identifies the condition that occured, and then you can avoid these extra conditions and simplify the code. > I would be fine with either method. Good, then please change it to return a proper error code. The only reason that you are using HA_ERR_LOCK_WAIT_TIMEOUT is that then you could do the quick change inside InnoDB to set its timeout value to 0, and this happens to then propagate the HA_ERR_LOCK_WAIT_TIMEOUT up to the callers without waiting. But that's not an appropriate way to design new code and APIs. Yes, it is frustrating when conceptually simple changes need extra effort, due to the need to work in an existing old and complex code base like MariaDB. But that is just the way it is, it is the same for all of us. - Kristian.

1 0

Can we get MariaDB GitHub CI to consistently be green?
by Otto Kekäläinen 10 Aug '24

10 Aug '24

Hi! Thanks to everyone who has worked on polishing the CI tests and integrations. Looking at e.g. 10.11 branch[1] I see that commit ccb7a1e[2] has a green checkmark next to it and all 15 CI jobs passed. I just wanted to check if everyone developing the MariaDB Server is committed to getting the CI to be consistently green? This means that GitHub rules need to be a bit more strict, not allowing any failing tests jobs at all, and developers and managers need to agree to stop adding new commits on any release branch if it can't be done without a fully passing CI run. Thanks to Daniel's review on latest failures we know these are at the moment recurring: * MDEV-25614 Galera test failure on GCF-354 included in MDEV-33073 always green buildbot * MDEV-33785 aarch64 macos encryption.create_or_replace * MDEV-33601 galera_3nodes.galera_safe_to_bootstrap test failing * MDEV-33786 galera_3nodes.galera_vote_rejoin_mysqldump mysql_shutdown failed at line 85: I see two approaches to get to consistently green CI: 1) Stop all development and focus on just fixing these, don't continue until CI is fully green, and once it is fully green make the GitHub branch protection settings one notch stricter to not allow any new commits unless the CI is fully green so it never regresses again. 2) Disable these tests and make the rules in GitHub branch protection one notch stricter right away, and not allow any new commits unless the CI is fully green ensuring no new recurring failures are introduced. - Otto [1] https://github.com/MariaDB/server/commits/10.11 [2] https://github.com/MariaDB/server/commit/ccb7a1e9a15e6a47aba97f9bdbfab2e4bf…

7 12

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 01 Aug '24

01 Aug '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Indeed a combination of the engine locking protocol and the slave retry > applies correctly to IDEMPOTENT mode. >> The optimistic parallel replication should work correctly with >> IDEMPOTENT, > Indeed. Ok good to know. It's definitely a tricky issue, but apparently no known bugs, at least to the two of us. As you saw, I did have to try a test case to feel confident it works ;-) - Kristian.

1 0

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 31 Jul '24

31 Jul '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: >> You need to make sure that --slave_exec_mode=IDEMPOTENT is not used. >> Otherwise you can get silent data corruption. > > IDEMPOTENT is always risky. Here it would add up another bit, in the > parallel mode. What do you mean "IDEMPOTENT is always risky"? As far as I know, if the slave would replicate without error in STRICT, it will replicate exactly the same way in IDEMPOTENT. Only if there is an error in STRICT mode will the IDEMPOTENT mode do something different (ignore the error in some cases). So the risk is that we might fail to detect that the slave is diverged from the master, right? Note that diverged slave can be undetected even in STRICT mode in many cases. > Btw should IDEMPOTENT be disallowed in the parallel mode actually? No, why? IDEMPOTENT should work just as documented in all parallel replication modes. Or do you know of any bug with IDEMPOTENT and optimistic parallel replication that I don't know of? If so do you have a test case? The optimistic parallel replication should work correctly with IDEMPOTENT, since the second query that's ignoring the error should still be taking InnoDB row locks, which will conflict as normal with the first transaction and cause the second to be retried, this time without (ignored) error. I just tried with the below testcase, it seems to work. Only when you fiddle with the InnoDB locking like in your patch and ignore certain locks, do you get problems, and then you can end up with a later transaction completely ignoring a lock that is temporarily held by an earlier XA PREPARE (even though the corresponding XA COMMIT is also earlier). And this will result in incorrect replication and diverged slave, I think (I did not test). This is not "risky", this is incorrect behaviour and a serious bug. So it seems you will need to maybe give an error when replicating any XA transaction if slave_exec_mode=IDEMPOTENT? Or handle it some other way to avoid silently replicating incorrectly. As you know, my opinion is that this is just yet another symptom of the real issue of MDEV-32020, but I know you're going to ignore that. So I've given up caring about what happens to XA replication, as long as it doesn't negatively affect non-XA stuff. > I just don't see any its usefullness except creating a chaotic state. Why do you think it creates a "chaotic state"? - Kristian. ----------------------------------------------------------------------- --source include/have_innodb.inc --source include/have_binlog_format_row.inc --source include/master-slave.inc --connection master CALL mtr.add_suppression("Can't find record in 't1'"); ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; CREATE TABLE t2 (a INT PRIMARY KEY) ENGINE=InnoDB; BEGIN; INSERT INTO t1 VALUES (1, 1); INSERT INTO t1 VALUES (2, 1); INSERT INTO t1 VALUES (3, 1); INSERT INTO t1 VALUES (4, 1); INSERT INTO t1 VALUES (5, 1); COMMIT; --sync_slave_with_master --source include/stop_slave.inc SET @old_exec_mode= @@GLOBAL.slave_exec_mode; SET GLOBAL slave_exec_mode=IDEMPOTENT; SET @old_parallel_mode= @@GLOBAL.slave_parallel_mode; SET GLOBAL slave_parallel_mode= optimistic; SET @old_threads= @@GLOBAL.slave_parallel_threads; SET GLOBAL slave_parallel_threads= 8; --source include/start_slave.inc # Block certain queries on the slave temporarily. BEGIN; INSERT INTO t2 VALUES (1), (2), (3); --connection master UPDATE t1 SET b=10 WHERE a=1; BEGIN; INSERT INTO t2 VALUES (1); # A DELETE to make room for a later INSERT, will be delayed a bit on slave. DELETE FROM t1 WHERE a=2; COMMIT; UPDATE t1 SET b=20 WHERE a=3; BEGIN; INSERT INTO t2 VALUES (2); # An INSERT to be deleted again shortly, will be delayed a bit on slave. INSERT INTO t1 VALUES (6,2); COMMIT; UPDATE t1 SET b=30 WHERE a=4; # This INSERT cannot run until after the earlier DELETE. INSERT INTO t1 VALUES (2, 1); # This DELETE cannot run until after the earlier INSERT. DELETE FROM t1 WHERE a=6; SELECT * FROM t1 ORDER BY a; --save_master_pos --connection slave --sleep 1 ROLLBACK; --sync_with_master SELECT * FROM t1 ORDER BY a; --source include/stop_slave.inc SET GLOBAL slave_exec_mode= @old_exec_mode; SET GLOBAL slave_parallel_mode= @old_parallel_mode; SET GLOBAL slave_parallel_threads= @old_threads; --source include/start_slave.inc --connection master DROP TABLE t1, t2; --source include/rpl_end.inc

1 0

Re: 1287c901: TLS/SSL changes (major rework)
by Sergei Golubchik 31 Jul '24

31 Jul '24

Hi, Georg, On Jul 31, Georg Richter wrote: > revision-id: 1287c901 (v3.4.0-5-g1287c901) > parent(s): 5386f1a3 > author: Georg Richter > committer: Georg Richter > timestamp: 2024-07-16 13:12:26 +0200 > message: > > TLS/SSL changes (major rework) > diff --git a/include/ma_common.h b/include/ma_common.h > index dfa96621..dc900b0d 100644 > --- a/include/ma_common.h > +++ b/include/ma_common.h > @@ -131,15 +131,9 @@ typedef struct st_mariadb_field_extension > MARIADB_CONST_STRING metadata[MARIADB_FIELD_ATTR_LAST+1]; /* 10.5 */ > } MA_FIELD_EXTENSION; > > -#if defined(HAVE_SCHANNEL) || defined(HAVE_GNUTLS) > -#define reset_tls_self_signed_error(mysql) \ > - do { \ > - free((char*)mysql->net.tls_self_signed_error); \ > - mysql->net.tls_self_signed_error= 0; \ > - } while(0) > -#else > -#define reset_tls_self_signed_error(mysql) \ > - do { \ > - mysql->net.tls_self_signed_error= 0; \ > +#ifdef HAVE_TLS > +#define reset_tls_error(mysql) \ > + do { \ > + mysql->net.tls_verify_status= 0; \ you've lost the original error message when you've changed char* pointer to the message to my_bool flag. > } while(0) > #endif > diff --git a/include/ma_tls.h b/include/ma_tls.h > index 23f53560..444ea4aa 100644 > --- a/include/ma_tls.h > +++ b/include/ma_tls.h > @@ -134,6 +141,7 @@ const char *ma_tls_get_cipher(MARIADB_TLS *ssl); > hash_type hash_type as defined in ma_hash.h > fp buffer for fingerprint > fp_len buffer length > + my_bool verify_period I don't see any 'my bool verify_period' in the list of arguments > > Returns: > actual size of finger print > @@ -150,7 +158,7 @@ unsigned int ma_tls_get_finger_print(MARIADB_TLS *ctls, uint hash_type, char *fp > int ma_tls_get_protocol_version(MARIADB_TLS *ctls); > const char *ma_pvio_tls_get_protocol_version(MARIADB_TLS *ctls); > int ma_pvio_tls_get_protocol_version_id(MARIADB_TLS *ctls); > -unsigned int ma_tls_get_peer_cert_info(MARIADB_TLS *ctls); > +unsigned int ma_tls_get_peer_cert_info(MARIADB_TLS *ctls, unsigned int size); is this part of the public API? > void ma_tls_set_connection(MYSQL *mysql); > > /* Function prototypes */ > diff --git a/include/mysql.h b/include/mysql.h > index 6c84a462..ba92527d 100644 > --- a/include/mysql.h > +++ b/include/mysql.h > @@ -448,6 +449,16 @@ typedef struct st_mysql_time > #define MYSQL_WAIT_EXCEPT 4 > #define MYSQL_WAIT_TIMEOUT 8 > > +#define MARIADB_TLS_VERIFY_OK 0 > +#define MARIADB_TLS_VERIFY_TRUST 1 > +#define MARIADB_TLS_VERIFY_HOST 2 we've agreed to swap two previous lines > +#define MARIADB_TLS_VERIFY_PERIOD 4 > +#define MARIADB_TLS_VERIFY_FINGERPRINT 8 is that right? you want to allow fingerprints even if the cert is expired? > +#define MARIADB_TLS_VERIFY_REVOKED 16 > +#define MARIADB_TLS_VERIFY_UNKNOWN 32 > +#define MARIADB_TLS_VERIFY_ERROR 128 /* last */ > + > + > typedef struct character_set > { > unsigned int number; /* character set number */ > diff --git a/libmariadb/ma_tls.c b/libmariadb/ma_tls.c > index 1bda7dcf..f6ea6661 100644 > --- a/libmariadb/ma_tls.c > +++ b/libmariadb/ma_tls.c > @@ -103,9 +103,54 @@ my_bool ma_pvio_tls_close(MARIADB_TLS *ctls) > return ma_tls_close(ctls); > } > > -int ma_pvio_tls_verify_server_cert(MARIADB_TLS *ctls) > +int ma_pvio_tls_verify_server_cert(MARIADB_TLS *ctls, unsigned int flags) > { > - return ma_tls_verify_server_cert(ctls); > + MYSQL *mysql; > + int rc; > + > + if (!ctls || !ctls->pvio || !ctls->pvio->mysql) > + return 0; > + > + mysql= ctls->pvio->mysql; > + > + /* Skip peer certificate verification */ > + if (ctls->pvio->mysql->options.extension->tls_allow_invalid_server_cert) > + { > + return 0; > + } > + > + rc= ma_tls_verify_server_cert(ctls, flags); > + > + /* Set error messages */ > + if (!mysql->net.last_errno) > + { > + if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_PERIOD) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Certificate not yet valid or expired"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_FINGERPRINT) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Fingerprint validation of peer certificate failed"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_REVOKED) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Certificate revoked"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_HOST) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Hostname verification failed"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_UNKNOWN) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Peer certificate verification failed"); > + else if (mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_TRUST) > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Peer certificate is not trusted"); let's try to preserve the error as set by the ssl library here > + } > + > + return rc; > } > > const char *ma_pvio_tls_cipher(MARIADB_TLS *ctls) > diff --git a/plugins/auth/my_auth.c b/plugins/auth/my_auth.c > index faea968c..e7429fae 100644 > --- a/plugins/auth/my_auth.c > +++ b/plugins/auth/my_auth.c > @@ -382,14 +419,30 @@ static int send_client_reply_packet(MCPVIO_EXT *mpvio, > errno); > goto error; > } > + mysql->net.tls_verify_status = 0; > if (ma_pvio_start_ssl(mysql->net.pvio)) > goto error; > - if (mysql->net.tls_self_signed_error && > - (!mysql->passwd || !mysql->passwd[0] || !hashing(mpvio->plugin))) > - { > - /* cannot use auth to validate the cert */ > - set_error_from_tls_self_signed_error(mysql); > - goto error; > + > + verify_flags= MARIADB_TLS_VERIFY_PERIOD | MARIADB_TLS_VERIFY_REVOKED; > + if (have_fingerprint(mysql)) > + { > + verify_flags|= MARIADB_TLS_VERIFY_FINGERPRINT; > + } else { > + verify_flags|= MARIADB_TLS_VERIFY_TRUST | MARIADB_TLS_VERIFY_HOST; > + } > + > + if (ma_pvio_tls_verify_server_cert(mysql->net.pvio->ctls, verify_flags) > MARIADB_TLS_VERIFY_OK) > + { > + if (mysql->net.tls_verify_status > MARIADB_TLS_VERIFY_TRUST || > + (mysql->options.ssl_ca || mysql->options.ssl_capath)) > + goto error; > + > + if (is_local_connection(mysql->net.pvio)) I'd say this should be earlier. Like if (!is_local_connection(mysql->net.pvio)) if (have_fingerprint(mysql)) { verify_flags|= MARIADB_TLS_VERIFY_FINGERPRINT; } else { verify_flags|= MARIADB_TLS_VERIFY_TRUST | MARIADB_TLS_VERIFY_HOST; } > + { > + CLEAR_CLIENT_ERROR(mysql); > + } > + else if (!password_and_hashing(mysql, mpvio->plugin)) > + goto error; > } > } > #endif /* HAVE_TLS */ > @@ -769,8 +822,14 @@ retry: > auth_plugin= &dummy_fallback_client_plugin; > > /* can we use this plugin with this tls server cert ? */ > - if (mysql->net.tls_self_signed_error && !hashing(auth_plugin)) > - return set_error_from_tls_self_signed_error(mysql); > + if ((mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_TRUST) && > + !password_and_hashing(mysql, auth_plugin)) > + { > + my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > + ER(CR_SSL_CONNECTION_ERROR), > + "Certificate verification failure: The certificate is NOT trusted."); already commented about using library error message > + return 1; > + } > goto retry; > } > /* > @@ -782,7 +841,9 @@ retry: > if (ma_read_ok_packet(mysql, mysql->net.read_pos + 1, pkt_length)) > return -1; > > - if (!mysql->net.tls_self_signed_error) > + if (!mysql->net.tls_verify_status || > + ((mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_TRUST) && > + is_local_connection(mysql->net.pvio))) Not needed, see above > return 0; > > assert(mysql->options.use_ssl); > diff --git a/libmariadb/secure/openssl.c b/libmariadb/secure/openssl.c > index 8231a244..b5b573a9 100644 > --- a/libmariadb/secure/openssl.c > +++ b/libmariadb/secure/openssl.c > @@ -459,24 +461,38 @@ error: > return NULL; > } > > -unsigned int ma_tls_get_peer_cert_info(MARIADB_TLS *ctls) > +unsigned int ma_tls_get_peer_cert_info(MARIADB_TLS *ctls, uint hash_size) > { > X509 *cert; > + unsigned int hash_alg; > SSL *ssl; > + char fp[129]; > > + switch (hash_size) { > + case 0: > + case 256: > + hash_alg= MA_HASH_SHA256; > + break; > + case 384: > + hash_alg= MA_HASH_SHA384; > + break; > + case 512: > + hash_alg= MA_HASH_SHA512; > + break; > + default: > + return 1; > + } > + > if (!ctls || !ctls->ssl) > return 1; > > - /* Did we already read peer cert information ? */ > - if (ctls->cert_info.version) > - return 0; > - > ssl= (SSL *)ctls->ssl; > > /* Store peer certificate information */ > + if (!ctls->cert_info.version) > + { when can it be already set? > if ((cert= SSL_get_peer_certificate(ssl))) > { > - char fp[33]; > #if OPENSSL_VERSION_NUMBER >= 0x10101000L > const ASN1_TIME *not_before= X509_get0_notBefore(cert), > *not_after= X509_get0_notAfter(cert); > @@ -714,9 +714,40 @@ my_bool ma_tls_close(MARIADB_TLS *ctls) > return rc; > } > > -int ma_tls_verify_server_cert(MARIADB_TLS *ctls) > +/** Check for possible errors, and store the result in net.tls_verify_status. > + verification will happen after handshake by ma_tls_verify_server_cert(). > + To retrieve all errors, this callback function returns always true. > + (By default OpenSSL stops verification after first error > +*/ > +static int ma_verification_callback(int preverify_ok __attribute__((unused)), X509_STORE_CTX *ctx) > { > - X509 *cert; > + SSL *ssl; > + > + int x509_err= X509_STORE_CTX_get_error(ctx); > + > + if ((ssl = X509_STORE_CTX_get_ex_data(ctx, SSL_get_ex_data_X509_STORE_CTX_idx()))) > + { > + MYSQL *mysql= (MYSQL *)SSL_get_app_data(ssl); > + > + if ((x509_err == X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT || > + x509_err == X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN)) > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_TRUST; > + else if (x509_err == X509_V_ERR_CERT_REVOKED) > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_REVOKED; > + else if (x509_err == X509_V_ERR_CERT_NOT_YET_VALID || > + x509_err == X509_V_ERR_CERT_HAS_EXPIRED) > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_PERIOD; > + else if (x509_err != X509_V_OK) > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_UNKNOWN; and here you need to store the error message in mysql->net. for example, else if (x509_err == X509_V_ERR_CERT_NOT_YET_VALID || x509_err == X509_V_ERR_CERT_HAS_EXPIRED) { if (mysql->net.tls_verify_status < MARIADB_TLS_VERIFY_PERIOD) mysql->net.tls_verify_error= X509_verify_cert_error_string(x509_err); mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_PERIOD; } or else if (x509_err == X509_V_ERR_CERT_NOT_YET_VALID || x509_err == X509_V_ERR_CERT_HAS_EXPIRED) { if (mysql->net.tls_verify_status < MARIADB_TLS_VERIFY_PERIOD) my_set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, ER(CR_SSL_CONNECTION_ERROR), X509_verify_cert_error_string(x509_err)); mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_PERIOD; } > + } > + > + /* continue verification */ > + return 1; > +} > + > +int ma_tls_verify_server_cert(MARIADB_TLS *ctls, unsigned int verify_flags) > +{ > + X509 *cert= NULL; > MYSQL *mysql; > SSL *ssl; > MARIADB_PVIO *pvio; > @@ -734,52 +765,97 @@ int ma_tls_verify_server_cert(MARIADB_TLS *ctls) > mysql= (MYSQL *)SSL_get_app_data(ssl); > pvio= mysql->net.pvio; > > + if (verify_flags & MARIADB_TLS_VERIFY_FINGERPRINT) > + { > + if (ma_pvio_tls_check_fp(ctls, mysql->options.extension->tls_fp, mysql->options.extension->tls_fp_list)) > + { > + mysql->net.tls_verify_status |= MARIADB_TLS_VERIFY_FINGERPRINT; > + return 1; > + } > + > + /* if certificates are valid and no revocation error occured, > + we can return */ > + if (!(mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_PERIOD) && > + !(mysql->net.tls_verify_status & MARIADB_TLS_VERIFY_REVOKED)) you can check this before trying the fingerprint > + { > + mysql->net.tls_verify_status= MARIADB_TLS_VERIFY_OK; > + return 0; > + } > + } > + > + if (mysql->net.tls_verify_status & verify_flags) > + { > + return 1; > + } > + > + if (verify_flags & MARIADB_TLS_VERIFY_HOST) > + { > if (!mysql->host) > { > pvio->set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > ER(CR_SSL_CONNECTION_ERROR), "Invalid (empty) hostname"); > - return 1; > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > + return MARIADB_TLS_VERIFY_ERROR; > } > > if (!(cert= SSL_get_peer_certificate(ssl))) > { > pvio->set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > ER(CR_SSL_CONNECTION_ERROR), "Unable to get server certificate"); > - return 1; > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; not sure it's MARIADB_TLS_VERIFY_HOST, may be MARIADB_TLS_VERIFY_UNKNOWN ? > + return MARIADB_TLS_VERIFY_ERROR; > } > + > #ifdef HAVE_OPENSSL_CHECK_HOST > if (X509_check_host(cert, mysql->host, strlen(mysql->host), 0, 0) != 1 > && X509_check_ip_asc(cert, mysql->host, 0) != 1) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > #else > x509sn= X509_get_subject_name(cert); > > if ((cn_pos= X509_NAME_get_index_by_NID(x509sn, NID_commonName, -1)) < 0) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > > if (!(cn_entry= X509_NAME_get_entry(x509sn, cn_pos))) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > > if (!(cn_asn1 = X509_NAME_ENTRY_get_data(cn_entry))) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > > cn_str = (char *)ASN1_STRING_data(cn_asn1); > > /* Make sure there is no embedded \0 in the CN */ > if ((size_t)ASN1_STRING_length(cn_asn1) != strlen(cn_str)) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > > if (strcmp(cn_str, mysql->host)) > + { > + mysql->net.tls_verify_status|= MARIADB_TLS_VERIFY_HOST; > goto error; > + } > #endif > X509_free(cert); > - > + } > return 0; > error: > + if (cert) > X509_free(cert); > > - pvio->set_error(mysql, CR_SSL_CONNECTION_ERROR, SQLSTATE_UNKNOWN, > - ER(CR_SSL_CONNECTION_ERROR), "Validation of SSL server certificate failed"); > return 1; > } > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 31 Jul '24

31 Jul '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> Also, are you aware that when you skip a record like this, you may be >> skipping the very record that needs to be changed? > Yet luckily this is covered by the BASE (of the patch) logics. > After(needless) scan the rest of duplicate key record set the third group > of events would have to retry, to success. (This is optimizable, but I You need to make sure that --slave_exec_mode=IDEMPOTENT is not used. Otherwise you can get silent data corruption. - Kristian.

1 0

Re: [PATCH] MDEV-34481 optimize away waiting for owned by prepared xa non-unique index record
by Kristian Nielsen 31 Jul '24

31 Jul '24

> From: Andrei <andrei.elkin(a)mariadb.com> > > This work partly implements a ROW binlog format MDEV-33455 part > that is makes non-unique-index-only table XA replication safe in RBR. > Existence of at least one non-null unique index has always guaranteed > safety (no hang to error). Hi Andrei, A couple early comments on your MDEV-34481 patch, about things that affect non-XA parts of the code and will need to be changed: > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 6419a58fbe4..b2321cfd865 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5444,20 +5449,33 @@ thd_need_wait_reports(const MYSQL_THD thd) > transaction. > */ > extern "C" int > -thd_rpl_deadlock_check(MYSQL_THD thd, MYSQL_THD other_thd) > +thd_rpl_deadlock_check(MYSQL_THD thd, MYSQL_THD other_thd, > + bool is_other_prepared, bool is_index_unique, > + uint *flagged) > + > + /* > + Return to the caller the fact of the non-unique index wait lock > + conflicts with one of a prepared state transaction. > + */ > + if (!is_index_unique && rgi && rgi->is_row_event_execution()) { > + if (is_other_prepared && !other_thd) > + { > + rgi->exec_flags |= 1 << rpl_group_info::HIT_BUSY_INDEX; > + (*flagged)++; > + } > + } > + No, thd_rpl_deadlock_check() has a specific purpose related to handling deadlock between active transactions. Don't add this XA-specific and unrelated logic to this function. Instead, add a separate function that InnoDB can call in the specific case you need. That function then only needs the parameters `thd` and `flagged`, the others are not needed: extern "C" bool thd_xa_skip_lock_check(MYSQL_THD thd, uint *flagged) { return (thd->rgi_slave && thd->rgi_slave->us_row_event_execution()); } Then in InnoDB, call something like this: if (lock->trx->state == TRX_STATE_PREPARED && (!lock->index->is_unique() || lock->index->n_nullable)) count_non_unique_index_hit++; About the HIT_BUSY_INDEX, see here: > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index f3d01b7d9ed..166295ed263 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -8149,11 +8154,20 @@ int Rows_log_event::find_row(rpl_group_info *rgi) > HA_READ_KEY_EXACT)))) > { > DBUG_PRINT("info",("no record matching the key found in the table")); > - if (error == HA_ERR_KEY_NOT_FOUND) > - error= row_not_found_error(rgi); > - table->file->print_error(error, MYF(0)); > - table->file->ha_index_end(); > - goto end; > + if (error == HA_ERR_LOCK_WAIT_TIMEOUT && !is_index_unique && > + (rgi->exec_flags & (1 << rpl_group_info::HIT_BUSY_INDEX))) > + { > + rgi->exec_flags &= ~(1 << rpl_group_info::HIT_BUSY_INDEX); > + skip_first_compare= true; > + } Here, you are effectively making a decision in the storage engine to skip this record which is locked by an XA PREPAREd transaction. But you are communicating this back to the server in a very convoluted way, by introducing the rgi->exec_flags and setting HIT_BUSY_INDEX. This is not the way to communicate back from engine to server. Instead, simply return a different error code than HA_ERR_LOCK_WAIT_TIMEOUT from InnoDB that you can check for here. You can maybe use one of the existing HA_ERR_* from include/my_base.h, or add a new one if necessary. if (error == HA_ERR_<something>) skip_first_compare= true; > diff --git a/sql/rpl_rli.h b/sql/rpl_rli.h > index 3f24481839b..a9ec44b58cb 100644 > --- a/sql/rpl_rli.h > +++ b/sql/rpl_rli.h > @@ -833,6 +833,15 @@ struct rpl_group_info > RETRY_KILL_KILLED > }; > uchar killed_for_retry; > + enum enum_exec_flags { > + HIT_BUSY_INDEX= 0 > + }; > + /* > + Being executed replication event's context. It could contain > + notification from engine such as attempts to lock already acquired > + index record. > + */ > + uint32 exec_flags; ... and then you don't need to add this. Also, are you aware that when you skip a record like this, you may be skipping the very record that needs to be changed? For example a case like this, with three transactions in order: XA START 'T1'; UPDATE my_table SET value=10 WHERE key=2 XA END 'T1'; XA PREPARE 'T1'; XA COMMIT 'T1'; XA START 'T2'; UPDATE my_table SET value=20 WHERE key=2; Transaction 'T2' may run speculatively in optimistic parallel replication after prepare of T1, but before commit of T1. Then it will skip the record key=2, IIUC. I didn't see any explanation of this case in the patch commit message or comments. Also I don't see this case tested in the test case, that needs to be added I believe. - Kristian.

1 0

Re: [MariaDB commits] [PATCH] MDEV-33856 New definition for Seconds_Behind_Master
by Kristian Nielsen 25 Jul '24

25 Jul '24

Hi Monty, Here is my review of the MDEV-33856 patch: > commit cc516f4d9913d732065c7abef983df42e5a8bab3 (origin/bb-11.6-MDEV-33856-seconds_behind_master) > Author: Monty <monty(a)mariadb.org> > Date: Tue May 14 23:47:59 2024 +0300 > > MDEV-33856 New definition for Seconds_Behind_Master If I understand the patch correctly, in introduces a new `Master_slave_time_diff` value as an alternative to the traditional Seconds_behind_master, with a clearer semantics. And does not change how Seconds_behind_master works. I think this is fine, and can help to preserve backwards compatibility. Just maybe mention this in the commit message, and/or change the bug title to "Alternative definition for Seconds_Behind_Master". > diff --git a/mysql-test/suite/perfschema/t/relaylog.test b/mysql-test/suite/perfschema/t/relaylog.test > index 12fc96a8b27..16f72efae98 100644 > --- a/mysql-test/suite/perfschema/t/relaylog.test > +++ b/mysql-test/suite/perfschema/t/relaylog.test > @@ -43,7 +43,7 @@ select > if (sum_number_of_bytes_read > 0, "MANY", "NONE") as SUM_NUMBER_OF_BYTES_READ, > if (sum_number_of_bytes_write > 0, "MANY", "NONE") as SUM_NUMBER_OF_BYTES_WRITE > from performance_schema.file_summary_by_instance > - where file_name like "%master-%" order by file_name; > + where file_name like "%master-%" and file_name not like "%sql/share%" order by file_name; > > select * from performance_schema.file_summary_by_instance > where file_name like "%slave-%" order by file_name; > @@ -96,7 +96,7 @@ sync_slave_with_master; > -- echo "============ Performance schema on slave ============" > > select * from performance_schema.file_summary_by_instance > - where file_name like "%master-%" order by file_name; > + where file_name like "%master-%" and file_name not like "%sql/share%" order by file_name; > > select > substring(file_name, locate("slave-", file_name)) as FILE_NAME, What is the reason for these changes (adding match on "%sql/share%" ? I don't understand the relation to the subject of the patch. > diff --git a/mysql-test/suite/rpl/t/master_last_event_time.inc b/mysql-test/suite/rpl/t/master_last_event_time.inc > new file mode 100644 > index 00000000000..0790d557a7b > --- /dev/null > +++ b/mysql-test/suite/rpl/t/master_last_event_time.inc > > +--connection server_3 > +--sleep 1 > +--source include/start_slave.inc > +--source include/sync_with_master_gtid.inc What is the purpose of --sleep 1 here? Usually it's wrong to have sleeps in the tests. Sometimes it is correct, but then there should be a comment explaining why a sleep is necessary (and sufficient) here. This sleep looks wrong, since there's a proper wait immediately after for the slave to sync up. > diff --git a/mysql-test/suite/rpl/t/master_last_event_time_stmt.test b/mysql-test/suite/rpl/t/master_last_event_time_stmt.test > new file mode 100644 > index 00000000000..fe0f16a3db3 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/master_last_event_time_stmt.test > @@ -0,0 +1,84 @@ > +# > +# Statement specific tests for master_last_event_time > +# > +--source include/have_binlog_format_statement.inc > > +# Slave will replicate the above with a row event which will be very fast > +# compared to the master event. I don't understand this comment. Why will the slave replicate using a row event, when the test case sets binlog_format=statement? > diff --git a/sql/log.cc b/sql/log.cc > index 1cc0fa77f08..63d9c756911 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -1898,10 +1898,20 @@ static inline int > binlog_commit_flush_xid_caches(THD *thd, binlog_cache_mngr *cache_mngr, > bool all, my_xid xid) > Xid_log_event end_evt(thd, xid, TRUE); > - return (binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); > + if (!thd->slave_thread && ! thd->user_time.val) > + { > + /* > + Ensure that on the master the event time is the time of commit, > + not the start of statement time. > + */ > + my_hrtime_t hrtime= my_hrtime(); > + end_evt.when= hrtime_to_my_time(hrtime); > + } > + DBUG_RETURN(binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); Ok, so correct me if I'm wrong, but I think the purpose here is: You want to have the time when the transaction ended (in case of long-running query). But for auto-committed query, there is no explicit COMMIT query, so the thd->start_time is the start of the transaction. So here you ensure that the XID_EVENT will have the time at start of commit in either case. If this is correct, probably could use a comment explaining this. Some comments: 1. It is better to check for thd->rgi_slave being non-NULL than for thd->slave_thread. The thd->rgi_slave is associated with the replication of some event or transaction. The thd->slave_thread just identifies the thread/connection. (This is mostly style/code clarity, as in current code the two values would track each other.) 2. Why the condition on thd->user_time.val? What is the situation where we are not replicating an event, but user_time is set and we do not want to put the current time in the XID_EVENT? 3. This handles an XID end event, which is for InnoDB transactions. But MyISAM/Aria use a different end event, a COMMIT query, and there are are few other cases as well. Don't you want consistently to have the end of the query as the time in all end events? Then this code should probably instead be in binlog_flush_cache() or perhaps write_transaction_to_binlog()? 4. Even given (3), there are still cases (eg. DDL) where there is no end event, and thus there is no event containing the time for the end of the operation on the master. Won't this cause some inconsistency in how the slave determines and displays its timings with this patch? If not, why not? > for (; has_xid && !entry.need_unlog && ha_info; ha_info= ha_info->next()) > { > if (ha_info->is_started() && ha_info->ht() != binlog_hton && > !ha_info->ht()->commit_checkpoint_request) > + { > entry.need_unlog= true; > + break; > + } > } Agree, this code always irritated me ;-) You could also move the condition (!entry.need_unlog && ha_info) out of the loop in an outer if () to not have to check it on every loop iteration (just an observation, entirely up to you, it's not something your patch introduced). > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 3802f76efe0..706a4903eb0 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -1104,10 +1116,12 @@ Log_event* Log_event::read_log_event(const uchar *buf, uint event_len, > switch(event_type) { > case QUERY_EVENT: > ev= new Query_log_event(buf, event_len, fdle, QUERY_EVENT); > + set_orig_exec_time_in_thd(((Query_log_event*) ev)->exec_time); > break; > case QUERY_COMPRESSED_EVENT: > ev= new Query_compressed_log_event(buf, event_len, fdle, > - QUERY_COMPRESSED_EVENT); > + QUERY_COMPRESSED_EVENT); > + set_orig_exec_time_in_thd(((Query_compressed_log_event*) ev)->exec_time); > @@ -964,6 +964,18 @@ Log_event* Log_event::read_log_event(IO_CACHE* file, > DBUG_RETURN(res); > } > > +/* > + Update thd->orig_exec_time > +*/ > + > +inline void set_orig_exec_time_in_thd(my_time_t exec_time) > +{ > +#if !defined(MYSQL_CLIENT) && defined(HAVE_REPLICATION) > + THD *thd= current_thd; > + if (likely(thd)) > + thd->orig_exec_time= exec_time; > +#endif > +} Using current_thd doesn't seem right here, and in fact it's not, as seen by this part of the patch where thd->orig_exec_time has to be set again for parallel replication: > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index 51ca3b183ec..d279761767a 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -60,7 +60,10 @@ rpt_handle_event(rpl_parallel_thread::queued_event *qev, > rgi->future_event_relay_log_pos= qev->future_event_relay_log_pos; > strcpy(rgi->future_event_master_log_name, qev->future_event_master_log_name); > if (event_can_update_last_master_timestamp(ev)) > + { > rgi->last_master_timestamp= ev->when + ev->exec_time; > + thd->orig_exec_time= ev->exec_time; > + } So let's not set thd->orig_exec_time from Log_event::read_log_event(), it's redundant (for parallel replication), and logically setting it should be done when applying the event, not when reading the event. Instead, set the thd->orig_exec_time in exec_relay_log_event(), so we retain similarity with how replication is done in the parallel and non-parallel case. There is already a similar if (event_can_update_last_master_timestamp(ev)) in exec_relay_log_event(). (Also see comments below about moving this from THD to rpl_group_info to not increase THD just for this very specific case). > + > +/* > + Get the time when the event had been executed on the master. > + This works for both query events and load data events. > +*/ > + > +#if Q_EXEC_TIME_OFFSET != L_EXEC_TIME_OFFSET > +#error "Q_EXEC_TIME_OFFSET is not same as L_EXEC_TIME_OFFSET" > +#endif > + > +time_t query_event_get_time(const uchar *buf, > + const Format_description_log_event > + *description_event) > +{ > + time_t when= uint4korr(buf); > + buf+= description_event->common_header_len; > + return when + uint4korr(buf + Q_EXEC_TIME_OFFSET); Suggestion: Add an assert here: DBUG_ASSERT(LOG_EVENT_IS_QUERY(buf[EVENT_TYPE_OFFSET]) || LOG_EVENT_IS_LOAD_DATA(buf[EVENT_TYPE_OFFSET])); just to avoid someone mistakenly adding code later to call this for a wrong event type. > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 2d671139a6c..05c9c4e8012 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3168,6 +3168,7 @@ class THD: public THD_count, /* this must be first */ > ulonglong start_utime, utime_after_lock, utime_after_query; > /* This can be used by handlers to send signals to the SQL level */ > ulonglong replication_flags; > + my_time_t orig_exec_time; // Exec time for last read binlog event > @@ -1348,8 +1346,14 @@ Query_log_event::Query_log_event(THD* thd_arg, const char* query_arg, > memset(&host, 0, sizeof(host)); > error_code= errcode; > > - end_time= my_time(0); > - exec_time = (ulong) (end_time - thd_arg->start_time); > + /* > + For slave threads, remember the original master exec time. > + This is needed to be able to calculate the master commit time. > + */ > + exec_time= ((thd->slave_thread) ? > + thd->orig_exec_time : > + (my_time(0) - thd_arg->start_time)); > + I think we can here avoid increasing the THD size, which is sad for a special case like this. Instead put the orig_exec_time into THD::rgi_slave, extending the struct rpl_group_info. Then the assignment above will be to rgi->orig_exec_time instead of thd->orig_exec_time, and the check here will be for thd->rgi_slave instead of thd->slave_thread. The rgi is the appropriate place for orig_exec_time, as it is a property of the transaction/event group being replicated. @@ -3171,146 +2856,170 @@ static const LEX_CSTRING msg_ignored= { STRING_WITH_LEN("Ignored") }; #endif > -static bool send_show_master_info_data(THD *thd, Master_info *mi, bool full, > - String *gtid_pos) > +inline void store_string_or_null(Field **field, const char *str) > +{ > > - mysql_mutex_unlock(&mi->run_lock); > +void store_master_info(THD *thd, Master_info *mi, TABLE *table, > > - protocol->store_string_or_null(mi->host, &my_charset_bin); > - protocol->store_string_or_null(mi->user, &my_charset_bin); > - protocol->store((uint32) mi->port); > - protocol->store((uint32) mi->connect_retry); > - protocol->store(mi->master_log_name, strlen(mi->master_log_name), > + store_string_or_null(field++, mi->host); > + store_string_or_null(field++, mi->user); > + (*field++)->store((uint32) mi->port); > + (*field++)->store((uint32) mi->connect_retry); > + (*field++)->store(mi->master_log_name, strlen(mi->master_log_name), So this patch refactors the code for SHOW SLAVE STATUS to use a shared code path with information_schema tables? That sounds like a good change. But for the future, please do such refactoring as a separate patch, you can see my patch series of pre-computing binlog checksums (MDEV-31273) for an example. Doing it together makes it very hard to review the actual code changes and see what is refactoring and what is new logic for master_slave_time_diff. > + if (unlikely(!rli->slave_timestamp) && Log_event::is_group_event(typ)) > + { > + /* > + First event for this slave. Assume that all the slave was up to date > + with the master just before the current event. > + */ > + rli->slave_timestamp= (time_t) ev->when + (time_t) ev->exec_time-1; I think the comment is meant to explain why there is the "-1" / minus-one in this expression, but it's not really clear exactly what is meant, so perhaps elaborate it a bit. Does it mean that master_slave_time_diff will start out as "1 second" just after the slave restarts? One way or the other, I do not have a strong opinion. But I want to give a general remark, from my experience with the troubles over the years with Seconds_Behind_Master. The problem with Seconds_Behind_Master was always that it didn't quite do what someone expected in some particular case, and then a new special case was added to try to accomodate. And after a number of iterations of this, we ended up with a semantics for Seconds_Behind_Master that no-one _really_ understands, and which still doesn't quite always do what people expect (which is in any case impossible if different people have different expectations). This "-1" here is clearly on the same road towards adding special cases. So my strong advice would be that, going forward, be _extremely_ conservative about modifying the semantics of master_slave_time_diff. The one good thing about the new status is that it has a (hopefully) clear semantics that can be documented (now including this "-1") and completely understood by users. Let's stick to that no matter who comes and wants just another small tweak, unless such tweaks actually make the semantics and documentation _simpler_. > + /* > + replay_log.description_event_for_exec can be null if the slave thread > + is getting killed > + */ > + if (LOG_EVENT_IS_QUERY((Log_event_type) buf[EVENT_TYPE_OFFSET]) || > + LOG_EVENT_IS_LOAD_DATA((Log_event_type) buf[EVENT_TYPE_OFFSET])) > + { > + time_t exec_time= query_event_get_time(buf, rli->relay_log. > + description_event_for_queue); > + set_if_bigger(rli->newest_master_timestamp, exec_time); > + } This confused me, because in Query_log_event, the exec_time is the duration of the query (a time interval), not an absolute time. So please use the name "end_time" for the variable instead of "exec_time", and consider renaming query_event_get_time() to query_event_get_end_time(). > > +static int make_slave_status_old_format(THD *thd, ST_SCHEMA_TABLE *schema_table) > +{ > + ST_FIELD_INFO *field_info= schema_table->fields_info; > + Name_resolution_context *context= &thd->lex->first_select_lex()->context; > + DBUG_ASSERT(thd->lex->sql_command == SQLCOM_SHOW_SLAVE_STAT); > + bool all_slaves= thd->lex->mi.show_all_slaves; > + ulonglong used_fields= ~0ULL; > + > + if (!all_slaves) > + { > + /* Remove 2 first fields and all fields above and including field 55 */ > + used_fields&= ~((1ULL << 0) | (1ULL << 1)); > + used_fields&= ((1ULL << 56)-1); Should there be some constant used for the magic numbers 55/56? And/or an assertion that this value is correct? Also, is the intention here that we will from here on never add more fields to legacy SHOW SLAVE STATUS? If so, there should probably be a comment about this; also see the test case mysql-test/suite/rpl/t/rpl_show_slave_status.test which has comments that might need updating. - Kristian

2 2

Moving forward from default-mysql-* packages in Debian
by Otto Kekäläinen 19 Jul '24

19 Jul '24

Hi! tl;dr: Anyone here interested in trying to push various open source projects in Debian and the upstreams to use mariadb-server directly instead of via default-mysql-server dependency? ## Situation Back in 2016 Debian introduced the virtual packages: - default-mysql-server - default-mysql-client - default-libmysqlclient-dev In Debian these have since pointed to MariaDB, and in Ubuntu they point to MySQL. Not much has happened since. There are only a handful packages in Debian that have declared an explicit dependency on MariaDB directly. Most packages continue to depend on default-mysql-* instead of MariaDB directly. As MariaDB and MySQL diverge, those packages will eventually end up depending on MySQL only unless they transition slowly to depend on MariaDB directly. It didn't happen organically, so it would need a push. Anyone here interested in taking up such a task? It requires talking to all the upstreams to officially support/adopt MariaDB going forward. ## Background - https://lists.debian.org/debian-devel-announce/2016/09/msg00000.html - https://lists.debian.org/debian-devel/2016/11/msg00832.html - https://wiki.debian.org/Teams/MySQL/default-mysql-server - https://tracker.debian.org/pkg/mysql-defaults ## Package lists # apt rdepends default-mysql-client default-mysql-client Reverse Depends: |Depends: audiolink |Depends: zoneminder |Depends: wordpress Recommends: tango-db |Depends: sqitch |Recommends: rsyslog-mysql |Depends: roundcube-mysql |Depends: rt5-db-mysql |Depends: rt4-db-mysql |Depends: redmine-mysql |Recommends: prelude-manager |Depends: postfixadmin |PreDepends: openstack-compute-node Depends: opendnssec-enforcer-mysql |Depends: ocsinventory-reports |Recommends: oar-server-mysql |Recommends: mysql-workbench |Suggests: mysql-sandbox Suggests: munin-plugins-core Suggests: motion Depends: libreoffice-canzeley-client Depends: lcmaps-plugins-jobrep-admin |Depends: kamailio-mysql-modules |Recommends: icinga2-ido-mysql Suggests: education-main-server |Depends: dbconfig-mysql |Suggests: cedar-backup3 |Depends: bacula-director-mysql |Depends: automysqlbackup # apt rdepends default-mysql-server default-mysql-server Reverse Depends: |Depends: python3-testing.mysqld (>= 1.0.2) |Depends: zoph |Recommends: zoneminder |Recommends: zabbix-server-mysql |Recommends: zabbix-proxy-mysql |Suggests: wordpress Suggests: trojan |Recommends: sympa |Recommends: spip |Suggests: sogo |Suggests: rsyslog-mysql Suggests: roundcube-mysql |Suggests: rt5-db-mysql |Suggests: rt4-db-mysql |Suggests: redmine-mysql |Suggests: adminer |Suggests: python3-mysqldb |Suggests: pwman3 |Suggests: prometheus-mysqld-exporter Recommends: postfix-gld |Suggests: phpmyadmin Suggests: pdns-backend-mysql Suggests: orthanc-mysql |Depends: openstack-cloud-services |Depends: openstack-cluster-installer |Suggests: ocsinventory-server |Suggests: oar-server-mysql |Recommends: mysqltuner |Suggests: mysql-sandbox |Recommends: mediawiki |Suggests: mailman3 |Suggests: mailman3-web Suggests: libreoffice-canzeley-client |Suggests: libreoffice-sdbc-mysql Suggests: inspircd Suggests: icingadb |Suggests: icinga2-ido-mysql Suggests: goval-dictionary Suggests: golang-github-ctdk-goiardi-dev Suggests: goiardi Suggests: digitemp |Depends: diaspora-common |Suggests: dbconfig-mysql Suggests: collectd-core |Recommends: cacti |Recommends: bacula-director-mysql |Recommends: audiolink # apt rdepends default-libmysqlclient-dev default-libmysqlclient-dev Reverse Depends: Depends: libboinc-app-dev Depends: libvtk9-dev Depends: urweb Depends: librust-mysqlclient-sys-dev Suggests: qtbase5-gles-dev Suggests: qtbase5-dev |Depends: postgresql-16-mysql-fdw Depends: libpoco-dev Recommends: newlisp |Suggests: mysql-sandbox Depends: libmysql-ocaml-dev Depends: libmysql++-dev Depends: libmailutils-dev Depends: kannel-dev Depends: libhoel-dev Suggests: libglpk40 Depends: libgdal-dev Suggests: fp-units-db-3.2.2 Suggests: fp-units-win-db-3.2.2 Depends: cl-sql-mysql

1 0

Re: [MariaDB commits] [PATCH] MDEV-15393 part 2. Fixes to 'gtid_slave_pos duplicate key errors after mysqldump restore' MDEV-34615 mysqldump wipes off pre-existing gtid slave state
by Kristian Nielsen 18 Jul '24

18 Jul '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> The most common case is the provisioning of a slave with a complete dump >> from a master. In this case, it is required that the GTID position is set >> correctly across all domains. Merging the existing and new GTID position as >> attempted in this patch will break the provisioning of the slave. > > 'Will break' but merely for the technical (your pp1,2 to the concat) > reasons. I don't see what else can be a problem of MDEV-34615 part. No. What I mean is, consider some slave server that has an existing gtid_slave_pos='20-2-1000' say, domain_id=20. And then this server is for some reason re-provisioned (ie. all data overwritten) with a new mysqldump, say from a master with GTID pos '10-1-2000'. Now the correct GTID position matching the new dump is SET GLOBAL gtid_slave_pos='10-1-2000'; If you make the MDEV-34615 idea default, then the position for the new slave will be multi-domain '20-2-1000,10-1-2000', which is not correct and could cause problems later. Basically, only the user knows if the old @@gtid_slave_pos is meaningful and should be kept, or if it should be overwritten, mysqldump cannot reliably determine this. And changing the default will end up breaking things in some cases that are working now. The merging of GTID positions can be useful in the case where we are setting up a multi-source slave C by importing dumps from two master A and B, say. In this case the first import needs to overwrite any old gtid_slave_pos, and the second import should merge. Thus the need for the user to specify. I hope this explains the problem. - Kristian.

1 0

Re: [MariaDB commits] [PATCH] MDEV-15393 part 2. Fixes to 'gtid_slave_pos duplicate key errors after mysqldump restore' MDEV-34615 mysqldump wipes off pre-existing gtid slave state
by Kristian Nielsen 17 Jul '24

17 Jul '24

> commit 7a50135f3f908380e2dfb45e3f95c9fb10c01d62 > Author: Andrei <andrei.elkin(a)mariadb.com> > Date: Mon Jul 15 17:50:37 2024 +0300 > > MDEV-15393 part 2. Fixes to 'gtid_slave_pos duplicate key errors after mysqldump restore' > MDEV-34615 mysqldump wipes off pre-existing gtid slave state Hi Andrei, here my review of the patch, with a particular important issue to not break existing use cases and backwards compatibility, as well as some general issues/comments. > When mysqldump run on mysql system database it generates inserts sql commands > into mysql.gtid_slave_pos. After running the backup script those Here and below, I suggest better wording for text that is hard to understand or using wrong/unusual grammar. "When mysqldump is run to dump the `mysql` system database, it generates INSERT statements into the table mysql.gtid_slave_pos" > inserts did not produce an expected gtid state on slave. "did not produce the exspected gtid state" > In particular the maximum of mysql.gtid_slave_pos.sub_id did not make > into > rpl_global_gtid_slave_state.last_sub_id > a memory object that the table persists. I think you mean here "an in-memory object that is supposed to match the current state of the table". > And that was regardless of whether --gtid option in or out. "whether the --gtid option was specified by the user or not". > The fixes ensure the insert block of the dump script is followed > with a "summing-up" SET @global.gtid_slave_pos assignment. > To address MDEV-34615 the assignment makes sure a possibly > pre-existing gtid state (e.g reflecting another domain) remains as a > sub-state upon the dump installation. This change is not appropriate, as it breaks backwards compatibility and breaks a common use case. Certainly not in a GA release such as 10.5! The most common case is the provisioning of a slave with a complete dump from a master. In this case, it is required that the GTID position is set correctly across all domains. Merging the existing and new GTID position as attempted in this patch will break the provisioning of the slave. The existing supported way of not overwriting the slave's old @@GLOBAL.gtid_slave_pos is to use --master-data=2 --gtid. This will output the GTID position in the dump commented out, and the user can themselves (manually or through a script of their own) merge the positions as required for their setup and application. If you think it is important to have a facility in mysqldump to do merging automatically, then you should implement this as a new option. Such new option would need to go to the next development version (11.6 or 11.7 I suppose?). It could be --master-data = 5 or 6 for example (idea being bit 2 means "merge old and new position), or maybe some other way that doesn't require "magic" constants. I am commenting below on some problems with the implementation of the merging of old and new position. But note that these comments apply to a different patch that implement this as a new option. These changes must not be made to the existing default behaviour. > diff --git a/client/mysqldump.c b/client/mysqldump.c > index 1c9d92d6e09..0be890e2528 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -5985,8 +5985,13 @@ static int dump_selected_tables(char *db, char **table_names, int tables) > } /* dump_selected_tables */ > > > +const char fmt_gtid_pos[] = "%sSET GLOBAL gtid_slave_pos=concat('%s', " > + "if(length(@@global.gtid_slave_pos) = 0, '', ','), " > + "@@global.gtid_slave_pos);\n"; Two problems with this: 1. This will result in a syntax error if the new position is empty and the old position is not (eg. SET GLOBAL gtid_slave_pos=',0-1-2'). 2. Unless I'm missing something obvious, this doesn't actually work to merge two different GTIDs with the same domain! It produces an error ER_DUPLICATE_GTID_DOMAIN. It seems there is a lack of coverage in the associated test case, if the actual merging of domains isn't even tested? SELECT @@GLOBAL.gtid_slave_pos; @@GLOBAL.gtid_slave_pos 0-1-2 SET GLOBAL gtid_slave_pos= CONCAT('0-1-10', ',', @@GLOBAL.gtid_slave_pos); rpl.tmp2 'mix' [ fail ] mysqltest: At line 12: query 'SET GLOBAL gtid_slave_pos= CONCAT('0-1-10', ',', @@GLOBAL.gtid_slave_pos)' failed: ER_DUPLICATE_GTID_DOMAIN (1943): GTID 0-1-2 and 0-1-10 conflict (duplicate domain id 0) > @@ -6054,8 +6059,11 @@ static int do_show_master_status(MYSQL *mysql_con, int consistent_binlog_pos, > fprintf(md_result_file, > "%sCHANGE MASTER TO MASTER_USE_GTID=slave_pos;\n", > comment_prefix); > - fprintf(md_result_file, > - "%sSET GLOBAL gtid_slave_pos='%s';\n", > + /* > + Defer print of SET gtid_slave_pos until after its placeholder > + table is guaranteed to have been dumped. > + */ > + sprintf(set_gtid_pos, fmt_gtid_pos, set_gtid_pos is a fixed size buffer, if I understand the patch correctly. This seems to mean that you have a buffer overflow here if the GTID position has too many domains? Note that it is not appropriate to simply use snprintf() instead, as this will merely truncate the output and give silent data corruption (or in this case probably a strange syntax error). Instead at least a proper error should be given (better to support arbitrary GTID position of course, but that might be a separate task if this is already an existing limitation of mysqldump, I did not check). > @@ -6181,10 +6189,8 @@ static int do_show_slave_status(MYSQL *mysql_con, int use_gtid, > mysql_free_result(slave); > return 1; > } > - print_comment(md_result_file, 0, > - "-- GTID position to start replication:\n"); > - fprintf(md_result_file, "%sSET GLOBAL gtid_slave_pos='%s';\n", > - gtid_comment_prefix, gtid_pos); > + /* similarly to do_show_master_status */ > + sprintf(set_gtid_pos, fmt_gtid_pos, gtid_comment_prefix, gtid_pos); I assume this is also a potential buffer overflow, didn't check closely. > @@ -6913,6 +6932,11 @@ int main(int argc, char **argv) > int exit_code; > int consistent_binlog_pos= 0; > int have_mariadb_gtid= 0; > + /* > + to hold SET @@global.gtid_slave_pos which is deferred to print > + until the function epilogue. > + */ > + char set_gtid_pos[3 + sizeof(fmt_gtid_pos) + MAX_GTID_LENGTH]= {0}; - Kristian.

1 0

'_' in numeric literals [MDEV-33228]
by Rohan Gupta 16 Jul '24

16 Jul '24

I was going through this issue https://jira.mariadb.org/browse/MDEV-34228, and noticed that during parsing we check if '_' is present in a token , if it does it is parsed as an identifier. My question here is to follow a SQL2023 standard of '_' in numeric literals, how would one go distinguishing the difference between numeric literals and identifiers ? mariadb documentation says that a identifier cannot start with a number until its in backticks `{identifier}`. Rohan Gupta

2 1

Re: d78376da69d: MDEV-34504 PURGE BINARY LOGS not working anymore
by Sergei Golubchik 10 Jul '24

10 Jul '24

Hi, Monty, On Jul 09, Michael Widenius wrote: > revision-id: d78376da69d (mariadb-11.4.2-11-gd78376da69d) > parent(s): 4ffeca643d5 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-07-09 11:01:29 +0300 > message: > > MDEV-34504 PURGE BINARY LOGS not working anymore > > PURGE BINARY LOGS did not always purge binary logs. This commit fixes > some of the issues and adds notifications if a binary log cannot be > purged. > > User visible changes: > - 'PURGE BINARY LOG TO log_name' and 'PURGE BINARY LOGS BEFORE date' > worked differently. 'TO' ignored 'slave_connections_needed_for_purge' > while 'BEFORE' did not. Now both versions ignores the > 'slave_connections_needed_for_purge variable'. > - 'PURGE BINARY LOG..' commands now returns 'note' if a binary log cannot > be deleted like > Note 1375 Binary log 'master-bin.000004' is not purged because it is > the current active binlog > - Automatic binary log purges, based on date or size, will write a > note to the error log if a binary log matching the size or date > cannot yet be deleted. > - If 'slave_connections_needed_for_purge' is set from a config or > command line, it is set to 0 if Galera is enabled and 1 otherwise > (old default). This ensures that automatic binary log purge works > with Galera as before the addition of > 'slave_connections_needed_for_purge'. > If the variable is changed to 0, a warning will be printed to the error > log. > > Code changes: > - Added THD argument to several purge_logs related functions that needed > THD. > - Added 'interactive' options to purge_logs functions. This allowed > me to remove testing of sql_command == SQLCOM_PURGE. it's not really _ineractive_ but ok > - Changed purge_logs_before_date() to first check if log is applicable > before calling can_purge_logs(). This ensures we do not get a > notification for logs that does not match the remove criteria. > - MYSQL_BIN_LOG::can_purge_log() will write notifications to the user > or error log if a log cannot yet be removed. > - log_in_use() will return reason why a binary log cannot be removed. > - Moved checking of binlog_format for Galera to be after Galera is > initialized (The old check never worked). If Galera is enabled > we now change the binlog_format to ROW, with a warning, instead of > aborting the server. If this change happens, the binlog_format variable > will be marked with AUTO or FORCED, for information_schema.system_variables, > and a warning will be printed to the error log. What's "FORCE"? "AUTO" means that the value was automatically changed by the server, "FORCE" means the same, it looks redundant. And it's not the "source of the value", better to revert that part. > - Print also a warning if FLASHBACK changes the binlog_format to ROW. > Before this was done silently. > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 2b12001de9e..604eaadcc92 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -5847,7 +5850,28 @@ int mysqld_main(int argc, char **argv) > #ifdef WITH_WSREP > wsrep_set_wsrep_on(nullptr); > if (WSREP_ON && wsrep_check_opts()) unireg_abort(1); > -#endif > + > + if (!opt_bootstrap && WSREP_PROVIDER_EXISTS && WSREP_ON) > + { > + if (global_system_variables.binlog_format != BINLOG_FORMAT_ROW) > + { > + sql_print_warning("Binlog_format changed to \"ROW\" because of Galera"); should be sql_print_information() > + global_system_variables.binlog_format= BINLOG_FORMAT_ROW; > + mark_binlog_format_used(binlog_format_used); > + } > + binlog_format_used= 1; > + if (!slave_connections_needed_for_purge_option_used) > + { > + slave_connections_needed_for_purge= > + internal_slave_connections_needed_for_purge= 0; > + mark_slave_connections_needed_for_purge_as_auto(); you don't need that, you can use SYSVAR_AUTOSIZE(internal_slave_connections_needed_for_purge, 0); > + sql_print_information( > + "slave_connections_needed_for_purge changed to 0 because " > + "of Galera. Change it to 1 or higher if this Galera node " > + "is also Master in a normal replication setup"); > + } > + } > +#endif /* WITH_WSREP */ > > #ifdef _WIN32 > /* > @@ -8219,7 +8243,9 @@ mysqld_get_one_option(const struct my_option *opt, const char *argument, > ((enum_slave_parallel_mode)opt_slave_parallel_mode); > break; > } > - > + case (int) OPT_SLAVE_CONNECTIONS_NEEDED_FOR_PURGE: > + slave_connections_needed_for_purge_option_used= 1; you don't need that, you can use if (!IS_SYSVAR_AUTOSIZE(&internal_slave_connections_needed_for_purge)) > + break; > case (int)OPT_BINLOG_IGNORE_DB: > { > binlog_filter->add_ignore_db(argument); > @@ -8714,18 +8740,14 @@ static int get_options(int *argc_ptr, char ***argv_ptr) > opt_bin_log= opt_bin_log_used= 1; > > /* Force format to row */ > + if (global_system_variables.binlog_format != BINLOG_FORMAT_ROW) > + { > + sql_print_warning("Binlog_format changed to \"ROW\" because of " > + "flashback"); sql_print_information > + global_system_variables.binlog_format= BINLOG_FORMAT_ROW; > + mark_binlog_format_used(binlog_format_used); > + } > binlog_format_used= 1; > - global_system_variables.binlog_format= BINLOG_FORMAT_ROW; > - } > - > - if (!opt_bootstrap && WSREP_PROVIDER_EXISTS && WSREP_ON && > - global_system_variables.binlog_format != BINLOG_FORMAT_ROW) > - { > - > - WSREP_ERROR ("Only binlog_format = 'ROW' is currently supported. " > - "Configured value: '%s'. Please adjust your configuration.", > - binlog_format_names[global_system_variables.binlog_format]); > - return 1; > } Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

GitHub account owners - please +1 these MariaDB mention PRs
by Otto Kekäläinen 10 Jul '24

10 Jul '24

Hi! I assume everyone on this mailing list has a GitHub account. Could you please +1 these PRs that add a mention of MariaDB in the documentation of various projects that currently only mention MySQL? Open each link below and press the thumbs up icon in the lower left corner of the PR description: https://github.com/bookshelf/bookshelf/pull/2129 https://github.com/ponyorm/pony/pull/708 https://github.com/metabase/metabase/issues/40325 https://github.com/SeaQL/seaql.github.io/pull/116 I am asking this because there are a lot of database tools and libraries out there that work with both MySQL and MariaDB, but they only mention MySQL, which is a pity, because the users of those tools might actually switch from MariaDB to MySQL simply out of confusion/fear that the tool supports only MySQL, which is almost never true. One of those rare cases is Javascript/Node.js Drizzle ORM which needs changes to support MariaDB. There is already a PR for that, you could +1 it as well: https://github.com/drizzle-team/drizzle-orm/pull/1692 The Rust Diesel ORM has a similar situation going on: https://github.com/diesel-rs/diesel/issues/1882 https://github.com/diesel-rs/diesel/pull/3964 However most projects have absolutely no MySQL-specific things and work with MariaDB, and just needed the docs updated, like was done in these cases: https://github.com/mysql-net/MySqlConnector/pull/1460 https://github.com/rails/rails/pull/51330 https://github.com/prisma/docs/pull/5706 https://github.com/gregstoll/gallery2/pull/168 https://github.com/coleifer/peewee/pull/2858

3 5

MariaDB 11.4 build consumers too much disk space - solved by not shipping the embedded server in Debian anymore
by Otto Kekäläinen 10 Jul '24

10 Jul '24

Hi all! While preparing 11.4.2 for upload to Debian[1] I noticed that builders fail on lack of disk space[2]. Below are some listings of build artifacts sorted by size. I solved this for now by stopping[3] to ship the embedded server in Debian, I don't think it had many users anyway. Leaving this here for visibility/comments though. [1] https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/88 [2] https://salsa.debian.org/otto/mariadb-server/-/jobs/5793989 [3] https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/88/di… ± du -shc * | sort -hr 12G total 7,8G builddir 3,7G debian 234M mysql-test 95M storage 30M sql 17M strings 5,1M plugin ± find debian/tmp/ -type f -exec du -b {} \; | sort -n | tail -n 25 8938376 debian/tmp/usr/bin/myisampack 8956744 debian/tmp/usr/bin/mariadb-test 9441288 debian/tmp/usr/bin/myisamchk 9661400 debian/tmp/usr/bin/mariadb-client-test 9725048 debian/tmp/usr/bin/mariadb-binlog 9894072 debian/tmp/usr/lib/mysql/plugin/ha_spider.so 10277896 debian/tmp/usr/lib/mysql/plugin/ha_connect.so 10622344 debian/tmp/usr/bin/aria_s3_copy 11875712 debian/tmp/usr/bin/aria_dump_log 12048792 debian/tmp/usr/bin/aria_ftdump 12079744 debian/tmp/usr/bin/aria_pack 12814096 debian/tmp/usr/bin/aria_read_log 12943168 debian/tmp/usr/bin/aria_chk 25659064 debian/tmp/usr/lib/mysql/plugin/ha_mroonga.so 138399064 debian/tmp/usr/bin/sst_dump 143275720 debian/tmp/usr/lib/mysql/plugin/ha_rocksdb.so 145093496 debian/tmp/usr/bin/mariadb-ldb 214442664 debian/tmp/usr/bin/test-connect-t 214822248 debian/tmp/usr/bin/mariadb-embedded 214849432 debian/tmp/usr/lib/x86_64-linux-gnu/libmariadbd.so.19 215011504 debian/tmp/usr/bin/mariadb-test-embedded 216148632 debian/tmp/usr/bin/mariadb-client-test-embedded 248523904 debian/tmp/usr/sbin/mariadbd 256329048 debian/tmp/usr/bin/mariadb-backup 622299760 debian/tmp/usr/lib/x86_64-linux-gnu/libmariadbd.a ± find builddir -type f -exec du -b {} \; | sort -n | tail -n 25 18278408 builddir/storage/rocksdb/librocksdb_tools.a 25227146 builddir/sql/libwsrep.a 25298746 builddir/sql/libwsrep_provider.a 25614608 builddir/storage/mroonga/vendor/groonga/lib/libgroonga.a 25659064 builddir/storage/mroonga/ha_mroonga.so 92889078 builddir/storage/perfschema/libperfschema_embedded.a 97604058 builddir/storage/perfschema/libperfschema.a 125637396 builddir/storage/innobase/libinnobase_embedded.a 131904864 builddir/storage/innobase/libinnobase.a 138399064 builddir/storage/rocksdb/sst_dump 143275720 builddir/storage/rocksdb/ha_rocksdb.so 145093496 builddir/storage/rocksdb/mariadb-ldb 189426944 builddir/unittest/sql/my_json_writer-t 189648448 builddir/unittest/sql/explain_filename-t 214442664 builddir/unittest/embedded/test-connect-t 214822248 builddir/libmysqld/examples/mariadb-embedded 214849432 builddir/libmysqld/libmariadbd.so.19 215011504 builddir/libmysqld/examples/mariadb-test-embedded 216148632 builddir/libmysqld/examples/mariadb-client-test-embedded 248523904 builddir/sql/mariadbd 256329048 builddir/extra/mariabackup/mariadb-backup 337460022 builddir/libmysqld/libsql_embedded.a 380429390 builddir/sql/libsql.a 511486028 builddir/storage/rocksdb/librocksdblib.a 622299760 builddir/libmysqld/libmariadbd.a

1 2

Re: [MariaDB commits] [PATCH] MDEV-34504 PURGE BINARY LOGS not working anymore
by Kristian Nielsen 09 Jul '24

09 Jul '24

Hi Monty, Here is my review of the patch for MDEV-34504. Looks good, just a bunch of small suggestions: > From: Monty <monty(a)mariadb.org> > > PURGE BINARY LOGS did not always purge binary logs. This commit fixes > some of the issues and adds notifications if a binary log cannot be > purged. > diff --git a/mysql-test/suite/binlog/r/binlog_index.result b/mysql-test/suite/binlog/r/binlog_index.result > index 9dfda71f9a7..2d2363a7fec 100644 > --- a/mysql-test/suite/binlog/r/binlog_index.result > +++ b/mysql-test/suite/binlog/r/binlog_index.result > @@ -30,6 +30,7 @@ flush logs; > flush logs; > *** must be a warning master-bin.000001 was not found *** > Warnings: > +Note 1375 Binary log 'master-bin.000004' is not purged because it is the current active binlog Wording: "it is the active binlog" or "it is the currently active binlog". > +Warnings: > +Note 1375 Binary log 'master-bin.000004' is not purged because it is in use by an active XID transaction A slightly better explanation is: "because it may be needed for crash recovery". This more directly explains to the user why the file cannot be removed yet (and what happens if the user removes the file anyway through the file system). It's true that this is currently related mostly to xa transactions, but the details are more complex. It's not really "active" transactions (the transactions are already committed at this point, it's the InnoDB redo log that may not yet be synced to disk). And there may be other reasons in the future (I think maybe binlog rotation already can be a reason, too). > --- a/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test > +++ b/mysql-test/suite/binlog/t/binlog_flush_binlogs_delete_domain.test > @@ -91,6 +91,8 @@ while ($domain_cnt) > FLUSH BINARY LOGS; > --let $purge_to_binlog= query_get_value(SHOW MASTER STATUS, File, 1) > --eval PURGE BINARY LOGS TO '$purge_to_binlog' > +--replace_column 2 # > +SHOW BINARY LOGS; Better use --source include/show_binary_logs.inc here (for consistency with other tests, it does the same thing). > diff --git a/sql/log.cc b/sql/log.cc > index c27c4f3353b..1384fb0b3e7 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -4791,8 +4791,8 @@ int MYSQL_BIN_LOG::purge_first_log(Relay_log_info* rli, bool included) > > @@ -4902,7 +4903,6 @@ int MYSQL_BIN_LOG::purge_logs(const char *to_log, > log_info.log_file_name); > goto err; > } > - > if (find_next_log(&log_info, 0) || exit_loop) > break; > } Accidental deletion of empty line. > @@ -5428,18 +5435,23 @@ int MYSQL_BIN_LOG::real_purge_logs_by_size(ulonglong binlog_pos) > -MYSQL_BIN_LOG::can_purge_log(const char *log_file_name_arg) > +MYSQL_BIN_LOG::can_purge_log(THD *thd, const char *log_file_name_arg, > + bool interactive) > { > - THD *thd= current_thd; // May be NULL at startup > @@ -5464,8 +5479,7 @@ MYSQL_BIN_LOG::can_purge_log(const char *log_file_name_arg) > purge_sending_new_binlog_file= sending_new_binlog_file; > } > if ((res= log_in_use(log_file_name_arg, > - (is_relay_log || > - (thd && thd->lex->sql_command == SQLCOM_PURGE)) ? > + (is_relay_log || interactive) ? > 0 : slave_connections_needed_for_purge))) It's great that you avoid using current_thd here. In fact, with this change, thd is no longer used at all in can_purge_log(), so you don't need to pass it in as an argument any longer. (I'm guessing this is a left-over from an earlier version of the patch, before you introduced the `interactive` argument, which I also think is very good). So please remove the `thd` argument of can_purge_log() ... > @@ -5338,6 +5339,7 @@ int MYSQL_BIN_LOG::real_purge_logs_by_size(ulonglong binlog_pos) > MY_STAT stat_area; > char to_log[FN_REFLEN]; > ulonglong found_space= 0; > + THD *thd= current_thd; ... and then you can also remove this call of current_thd. > @@ -5473,9 +5487,39 @@ MYSQL_BIN_LOG::can_purge_log(const char *log_file_name_arg) > waiting_for_slave_to_change_binlog= 1; > strmake(purge_binlog_name, log_file_name_arg, > sizeof(purge_binlog_name)-1); > + if (res == 1) > + reason= "it is in use by a slave thread"; > + else > + reason= "less than 'slave_connections_needed_for_purge' slaves has " > + "processed it"; > + goto error; Grammar: s/has/have/: "less than 'slave_connections_needed_for_purge' slaves have processed it" > @@ -5847,7 +5850,28 @@ int mysqld_main(int argc, char **argv) > #ifdef WITH_WSREP > wsrep_set_wsrep_on(nullptr); > if (WSREP_ON && wsrep_check_opts()) unireg_abort(1); > -#endif > + > + if (!opt_bootstrap && WSREP_PROVIDER_EXISTS && WSREP_ON) > + { > + if (global_system_variables.binlog_format != BINLOG_FORMAT_ROW) > + { > + sql_print_warning("Binlog_format changed to \"ROW\" because of Galera"); > + global_system_variables.binlog_format= BINLOG_FORMAT_ROW; > + mark_binlog_format_used(binlog_format_used); > + } This change seems unrelated to MDEV-34504. Not critical I think, but for the future it's best to put such changes in a separate commit. (The change itself is good, I think.) > @@ -1303,13 +1312,19 @@ Sys_slave_connections_needed_for_purge( > "slave_connections_needed_for_purge", > "Minimum number of connected slaves required for automatic binary " > "log purge with max_binlog_total_size, binlog_expire_logs_seconds " > - "or binlog_expire_logs_days.", > + "or binlog_expire_logs_days. Default is 0 when Galera is enabled and 1 " > + "otherwise.", Since Galera is #ifdef, I guess this should be something like: "or binlog_expire_logs_days. " #ifdef WSREP "Default is 0 when Galera is enabled and 1 otherwise.", #else "Defaults to 1.", #endif > diff --git a/sql/sys_vars.h b/sql/sys_vars.h > new file mode 100644 > index 00000000000..8f4eac38cd0 > --- /dev/null > +++ b/sql/sys_vars.h > @@ -0,0 +1,17 @@ > +/* Copyright (c) 2024, MariaDB Corporation. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. Hm. It's not copyrighted by MariaDB Corporation solely, there are other copyright holders. If you want to have a copyright line before the GPL header, you could maybe say "Copyright (c) 2024, MariaDB Corporation and others", or "Parts of this file Copyright (c) 2024, MariaDB Corporation". Or just omit it. - Kristian.

1 0

Re: [MariaDB commits] [PATCH] MDEV-33487 Feature non-blocking binlog
by Kristian Nielsen 08 Jul '24

08 Jul '24

Here is my review of the MDEV-33487 / PR#3087 patch. First, some general remarks. This patch addresses a specific bottleneck when binlogging huge transactions (eg. multi-gigabyte table updates logged in ROW mode). The root cause is a fundamental limitation in the current binlog format that requires all transactions to be binlogged as a single event group, using consecutive bytes in a single binlog file. Thus, while writing the large transaction to the binlog during commit, all other transaction commits will have to wait before they can commit. Eventually, we should move towards an enhanced binlog format that doesn't have this limitation, so that transactions can be partially binlogged during execution, avoiding this bottlenect. There are a number of other problems caused by this limitation that are not addressed by this patch. Changing the binlog format is a complex task not easily done, so this patch should be seen as a temporary solution to one specific bottleneck, until if/when this underlying limitation can hopefully be removed. Below my detailed review of the code in the patch. This patch touches core code of the binlog group commit mechanism, which is complex code with a number of tricky issues to consider, and very important to keep correct and maintainable. I have tried to give concrete suggestions on ways to write the code, as well as some necessary extra tests. I think if MariaDB PLC wants this patch to be merged, they need to assign a core developer to take responsibility for it to ensure that any problems that turn up later will be handled in a reasonable time frame (eg. prioritise time to work with the author to debug and get fixes reviewed and merged). > From: poempeng <poempeng(a)tencent.com> > > During the commit stage of a large transaction, it may cost too much > time to write binlog cache and block all subsequent transactions for > a long time. One possible solution of this issue is to write the binlog > of the large transaction to a temporary file and then rename it to the > next new binlog file. It's not really the cost of writing (which is similar with and without the patch), the problem is doing costly I/O under the global mutex LOCK_log that blocks other commits. I suggest making that clear from the commit message, for example: Very large transactions (eg. gigabytes) can stall other transactions for a long time because the data is written while holding LOCK_log, which blocks other commits from binlogging. One possible solution to this issue is to write a large transaction to its own binlog file outside of holding LOCK_log, and then forcing a binlog rotate after obtaining LOCK_log, simply renaming the new binlog file in place. > diff --git a/sql/log.cc b/sql/log.cc > index 460cefea47b..9de1c6118cd 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -74,6 +74,9 @@ > #include <utility> // pair > #endif > > +#include <atomic> > +#include <chrono> These should not be needed after changes mentioned below. > @@ -3727,7 +3730,10 @@ bool MYSQL_BIN_LOG::open(const char *log_name, > enum cache_type io_cache_type_arg, > ulong max_size_arg, > bool null_created_arg, > - bool need_mutex) > + bool need_mutex, > + const char *file_to_rename, > + my_off_t file_size, > + group_commit_entry *entry) > + if (file_to_rename) > + { > + if (write_gtid_and_skip_event(file_size, entry)) > + goto err; > + } > + I don't think you need to write these here, inside MYSQL_BIN_LOG::open(). Instead you can do it in the caller after open() returns, right? Then you can avoid the extra arguments file_size and entry, and the conditional here. > @@ -6826,7 +6850,6 @@ MYSQL_BIN_LOG::write_gtid_event(THD *thd, bool standalone, > thd->variables.server_id= global_system_variables.server_id; > } > #endif > - Avoid unrelated changes in the diff like this one (to simplify later merges and history search). > @@ -7849,11 +7872,18 @@ int Event_log::write_cache_raw(THD *thd, IO_CACHE *cache) > - mysql_mutex_assert_owner(&LOCK_log); > + > + IO_CACHE *out_file= f; > + if (likely(f == nullptr)) > + { > + mysql_mutex_assert_owner(&LOCK_log); > + out_file= get_log_file(); > + } Don't add another conditional here. Instead just pass in the IO_CACHE to use as a parameter in each call site, eg. make the `f` parameter mandatory, not optional. > @@ -8581,7 +8612,23 @@ MYSQL_BIN_LOG::queue_for_group_commit(group_commit_entry *orig_entry) > bool > MYSQL_BIN_LOG::write_transaction_to_binlog_events(group_commit_entry *entry) > { > - int is_leader= queue_for_group_commit(entry); > + int is_leader; > + if (unlikely(entry->cache_mngr->trx_cache.get_byte_position() >= > + non_blocking_binlog_threshold) || > + DBUG_IF("non_blocking_binlog_ignore_cache_size")) > + { > + if (!can_use_non_blocking_binlog(entry) || > + write_non_blocking_binlog(entry)) > + goto original_commit_path; > + /* thread using non-blocking binlog is treated as a single group */ > + is_leader= 1; > + entry->non_blocking_log= true; > + entry->next= nullptr; > + goto group_commit_leader; > + } > +original_commit_path: > + > + is_leader= queue_for_group_commit(entry); Ok, so when we force a binlog rotation due to binlog_non_blocking_threshold, we do this by itself, not as part of a group commit. I think that is sensible. The way this gets integrated in the existing code is not very nice, with all these extra goto's and conditionals on entry->non_blocking_log. Can we instead make the decision on binlog_non_blocking_threshold in the function MYSQL_BIN_LOG::write_transaction_to_binlog()? And then call into MYSQL_BIN_LOG::trx_group_commit_leader() directly, skipping the group commit code with `is_leader` in write_transaction_to_binlog_events(). That should greatly simplify the logic. Probably the trx_group_commit_leader() needs to be refactored a bit, split in two, where the first part is dealing with grabbing the group commit queue, while only the second part is called in the binlog_non_blocking_threshold case, to do the actual binlogging. You might also consider using a separate function for the binlogging in the binlog_non_blocking_threshold case, extracting pieces from trx_group_commit_leader() into smaller shared functions; it depends on what gives the cleanest and simplest code, and is best decided as part of the refactoring of trx_group_commit_leader(). Another thing related to is this: How does the code ensure that commit ordering (on the slave) is preserved? This is the wait_for_prior_commit() mechanism, and is needed to ensure on the slave that the GTIDs are binlogged in the right order. This is normally handled by queue_for_group_commit(), which is skipped in the binlog_non_blocking_threshold case. It was not clear to me from the code how this gets handled. This probably also needs a test case to test that binlog order will be correct on the slave when multiple transactions commit in parallel (eg. optimistic parallel replication), and several of them use the binlog_non_blocking_threshold case. > @@ -12459,6 +12519,7 @@ mysql_bin_log_commit_pos(THD *thd, ulonglong *out_pos, const char **out_file) > } > #endif /* INNODB_COMPATIBILITY_HOOKS */ > > +mysql_rwlock_t binlog_checksum_rwlock; > > static void > binlog_checksum_update(MYSQL_THD thd, struct st_mysql_sys_var *var, > @@ -12468,6 +12529,7 @@ binlog_checksum_update(MYSQL_THD thd, struct st_mysql_sys_var *var, > bool check_purge= false; > ulong UNINIT_VAR(prev_binlog_id); > > + mysql_rwlock_wrlock(&binlog_checksum_rwlock); > mysql_mutex_lock(mysql_bin_log.get_log_lock()); > if(mysql_bin_log.is_open()) > { Ok, so you introduce a new lock to protect changing binlog_checksum_options. This is otherwise protected by LOCK_log, but we don't want to be holding that while writing the bin transaction, obviously. But did you check all places where binlog_checksum_options is accessed? Your patch doesn't use the new binlog_checksum_rwlock anywhere except in binlog_checksum_update(). There might be some places that currently use LOCK_log to protect the access, and could instead use the new binlog_checksum_rwlock (reducing contention on LOCK_log). Or did you check already, and there was no such opportunity? > +bool MYSQL_BIN_LOG::can_use_non_blocking_binlog(group_commit_entry *entry) > +{ > + DBUG_ASSERT(entry->cache_mngr->trx_cache.get_byte_position() >= > + non_blocking_binlog_threshold || > + DBUG_IF("non_blocking_binlog_ignore_cache_size")); > + THD *thd= entry->thd; > + if (unlikely(!is_open()) || encrypt_binlog || > + !entry->cache_mngr->stmt_cache.empty() || > + entry->cache_mngr->trx_cache.has_incident() || thd->slave_thread || > + thd->wait_for_commit_ptr) Oh. So you disable this on the slave. Is that really necessary? Ok, on the slave, in many cases the following transactions in any case are blocked from committing before the large one, so I see that this could be acceptable. It does seem a very special-purpose use case though. So if this is kept disabled for slave threads, it needs to be very clearly documented, and there should be a comment here as well in the code about why this is done. > +std::string MYSQL_BIN_LOG::generate_random_file_name() "generate_tmp_binlog_file_name" ? > + if (unlikely(!binlog_dir_inited.load(std::memory_order_acquire))) > + { > + char dev[FN_REFLEN]; > + size_t dev_length; > + mysql_mutex_lock(&LOCK_log); > + if (!binlog_dir_inited.load(std::memory_order_relaxed) && name != nullptr) This is not necessary. Just initialize what you need when the binlog is initialized, just like other binlog initialization, avoiding the check for binlog_dir_inited. See for example init_server_components() in sql/mysqld.cc > + std::string temp_file_name; Normally we don't use std::string in the server source, as it makes it harder to control the memory allocation and/or to avoid dynamic memory allocation. In this case we would normally use a fixed-size buffer of length FN_REFLEN, or the class String from sql/sql_string.h with some fixed stack-allocated initial buffer for the common case of not too long name, to avoid a dynamic memory allocation. > + temp_file_name.reserve(FN_REFLEN); > + auto now_in_sys= std::chrono::system_clock::now().time_since_epoch(); > + auto now_in_ms= > + std::chrono::duration_cast<std::chrono::milliseconds>(now_in_sys) > + .count(); You don't need to add this millisecond timestamp, do you? The file name will already be unique from temp_bin_counter? > + auto count= temp_bin_counter.fetch_add(1); This by default uses std::memory_order_seq_cst, which is overly restrictive. You can just use class Atomic_counter from include/my_counter.h to get a simple atomic counter with std::memory_order_relaxed semantics. > + temp_file_name.append(binlog_dir); > + temp_file_name.append("_temp_bin_"); > + temp_file_name.append(std::to_string(now_in_ms)); > + temp_file_name.push_back('_'); > + temp_file_name.append(std::to_string(count)); > +template <typename F= std::function<void()>> class Scoped_guard > +{ > +public: > + Scoped_guard(F f); > + ~Scoped_guard(); > + > +private: > + F func; > +}; Hm. Yes, this is nice, but it is not good if each new feature implements its own specific class like this. So either this should go somewhere shared where it can be used in the future by other code (did you check that there isn't already something similar that can be used, in the MariaDB source code or in the C++ standard library?), preferably as a separate commit. Or alternatively stay with the existing old-fascioned style for error handling used elsewhere in sql/log.cc. > + size_t skip_event_len= non_blocking_binlog_reserved_size - skip_event_start; > + skip_event_buf= > + (uchar *) my_malloc(PSI_INSTRUMENT_ME, skip_event_len, MYF(MY_ZEROFILL)); > + generate_skip_event(skip_event_buf, skip_event_len, skip_event_start, > + entry->thd); > + if (my_b_write(&log_file, skip_event_buf, skip_event_len) != 0) > + return true; Do we really need to generate this skip event? It seems quite hacky to have a random dummy event in the middle of the binlog. The size you need to reserve should be possible to pre-calculate: - Format_description_log_event - Binlog_checkpoint_event - Gtid_list_log_event - Gtid_log_event The only issue should be the Gtid_list_log_event. This can grow in size, but only if a new GTID domain id or server id gets added during the writing of the temporary file. This is very unlikely to happen, and the code can check for if this happens and in this case fall back to writing the data to the binlog file normally under LOCK_log. This way, we can avoid this strange skip event. > + if (temp_file_name.size() >= FN_REFLEN) > + { > + sql_print_warning("The name of temp file '%s' is too long!", > + temp_file_name.c_str()); > + return true; This will go in the server log for the user/DBA to wonder about. So add a bit of context to the message so that it is clear that it is about the temporary file used for binlog_non_blocking_threshold. > + thd->push_internal_handler(&dummy_error_handler); Hm. Why do you need to do this? If this is really needed, then it needs a good comment explaning why. But I suspect you might not need this at all. Is it to avoid assertions when an error is handled on the file operations? If you don't want mysql_file_open() and so to set an error with my_error() (because you want to handle the error yourself somehow), then just don't pass MY_WME as a flag to the function. > + if (unlikely(generate_new_name(new_name, name, 0))) > + abort(); > + new_name_ptr= new_name; > + /* > + We log the whole file name for log file as the user may decide > + to change base names at some point. > + */ > + Rotate_log_event r(new_name + dirname_length(new_name), 0, > + LOG_EVENT_OFFSET, 0); > + if (unlikely(write_event(&r, alg))) > + abort(); > + if (unlikely(flush_io_cache(&log_file) != 0)) > + abort(); No, don't crash the server like this. Errors during writing binlog are indeed very tricky. But it shouldn't crash the server. In the worst case, we may be left with a corrup binlog, but the error should be handled and reported to the user/DBA, similar to how it is done in other parts of the code. How does this code work with respect to the binlog checkpoint mechanism? The binlog checkpoint mechanism is a way to ensure that sufficient binlog files are kept to ensure correct crash recovery, in an asynchroneous way that does not require syncing the InnoDB redo log at binlog rotation. It uses the struct xid_count_per_binlog to keep track of which binlogs are active, and uses this to write binlog checkpoint events as appropriate. There are some tricky concurrency issues around this mechanism, so it is important to consider carefully how this will be handled in different cases by the code. For example, if the binlog checkpoint gets delayed and spans multiple binlog files including one or more generated by this patch; or what happens if RESET MASTER is run in parallel, etc. I don't see any changes related to xid_count_per_binlog in the patch. Is this because this is all handled correctly by MYSQL_BIN_LOG::open()? I would like to see a comment somewhere explaining how this is working. Also I think there needs to be some test cases testing this, eg. delayed binlog checkpointing when several binlog_non_blocking_threshold binlog rotations are running in parallel. > diff --git a/sql/log.h b/sql/log.h > index fc5209d1922..af2b56da802 100644 > --- a/sql/log.h > +++ b/sql/log.h > @@ -19,6 +19,7 @@ > +private: > + // reserved size for format_description_log_event, gtid_list_log_event ... > + static const my_off_t non_blocking_binlog_reserved_size= 10 * 1024 * 1024; Oh, 10 *megabyte* default for the dummy event? That's surely excessive? But hopefully we can avoid the dummy event completely as suggested above. > +static Sys_var_ulonglong Sys_var_non_blocking_binlog_threshold( > + "non_blocking_binlog_threshold", > + "If the binlog size of a transaction exceeds this value, we write it to a " > + "new temporary file and rename it to the next binlog file.", > + GLOBAL_VAR(non_blocking_binlog_threshold), CMD_LINE(OPT_ARG), > + VALID_RANGE(256 * 1024 * 1024, ULONGLONG_MAX), DEFAULT(ULONGLONG_MAX), > + BLOCK_SIZE(1)); Maybe name it so that it starts with "binlog_", eg. binlog_non_blocking_threshold? I think the VALID_RANGE should be extended, there is no need to forbid users to set it low for testing purposes etc. (even though I agree normally a large value would be strongly recommended). Suggestion for better description: If the binlog size in bytes of a transaction exceeds this value, force a binlog rotation and write the transaction to its own binlog file. This can reduce stalls of other `commits when binlogging very large transactions, at the cost of creating extra binlog files smaller than the configured --max-binlog-size. - Kristian.

1 0

Question regarding underscore in Numeric Literals
by Rohan Gupta 08 Jul '24

08 Jul '24

https://jira.mariadb.org/browse/MDEV-34228 I am trying my hands at this issue, which is a new feature introduced in SQL2023 and have been unable to determine if the underscore should only be part of queries, or it should be part of the users query's outputs. Also I have been trying to fiddle, around the parser in sql_lex.cc. Any pointers or guidance on how to approach this? Rohan Gupta

2 1

How to debug build/project?
by Rohan Gupta 05 Jul '24

05 Jul '24

I followed https://mariadb.org/get-involved/getting-started-for-developers/get-code-bu… and am able to run the server and mariadb client. Right now i am trying to debug the project after building and running it but havent been successful. I am trying the issue https://jira.mariadb.org/browse/MDEV-34228 In SQL 2023 we have the use of underscore with numeric literals. Any pointers on how to approach this will be appreciated. Been trying to fiddle around numericliteral.h file. Thank You

3 2

PRs with one approval pending to get the second one
by Otto Kekäläinen 01 Jul '24

01 Jul '24

Hi! Here are a couple of PRs that have already one approval, but have been pending to get the second approval for some weeks. Could any devs here spare some review time and add their approval so the PRs can be merged? (or shout out if they object them being merged) https://github.com/MariaDB/server/pull/3301 https://github.com/MariaDB/server/pull/3261 https://github.com/MariaDB/server/pull/3245 https://github.com/MariaDB/server/pull/3244 https://github.com/MariaDB/server/pull/3225 https://github.com/MariaDB/server/pull/2961

1 0

UDFs and character sets
by Jaco Kroon 25 Jun '24

25 Jun '24

Hi, It's a known issue that strings to and from UDF functions are a pain (https://dev.mysql.com/blog-archive/a-tale-of-udfs-with-character-sets/, amongst others). Most recently I ran into an issue trying to create a UDF to assist with IPv6 manipulations (in it's current, not working as ex form at https://github.com/jkroonza/uls-mysql/blob/master/src/uls_inet6.c) The two UDF functions there is intended to find the "network" (first) and "broadcast" (last) address of a range, with the intention to add functions to perform other manipulations (like adding some offset to an IPv6, or even add two IPv6's together, eg, to find the "next" network for 2c0f:f720::/40 you can add 0:0:100:: to 2c0f:f720::). The referenced works great when you get values from inet6 column to pass to the function (it gets transformed into textual presentation before being passed to the UDF) eg: MariaDB [test]> select uls_inet6_network_address('::ffff:192.168.1.123', 120); +--------------------------------------------------------+ | uls_inet6_network_address('::ffff:192.168.1.123', 120) | +--------------------------------------------------------+ | ::ffff:192.168.1.0 | +--------------------------------------------------------+ 1 row in set (0.001 sec) However, given this: MariaDB [test]> create table t(t inet6 not null, primary key(t)); Query OK, 0 rows affected (0.024 sec) MariaDB [test]> insert into t values('::ffff:192.168.1.15'); Query OK, 1 row affected (0.014 sec) MariaDB [test]> select * from t where uls_inet6_network_address('::ffff:192.168.1.123', 120) < t; Empty set, 1 warning (0.009 sec) MariaDB [test]> show warnings; +---------+------+---------------------------------------------+ | Level | Code | Message | +---------+------+---------------------------------------------+ | Warning | 1292 | Incorrect inet6 value: '::ffff:192.168.1.0' | +---------+------+---------------------------------------------+ 1 row in set (0.000 sec) I started to dig, to find this: MariaDB [test]> select charset(uls_inet6_network_address('::ffff:192.168.1.123', 120)); +-----------------------------------------------------------------+ | charset(uls_inet6_network_address('::ffff:192.168.1.123', 120)) | +-----------------------------------------------------------------+ | binary | +-----------------------------------------------------------------+ 1 row in set (0.000 sec) Then I played a bit and found that if I can rig it to the returned string has 16 characters I don't get an error; eg: MariaDB [test]> select * from t where uls_inet6_network_address('::ff:f:ffff:ffff', 120) > t; +---------------------+ | t | +---------------------+ | ::ffff:192.168.1.15 | +---------------------+ 1 row in set (0.001 sec) Based on this I'm inferring that: 1. Contextually the difference between BINARY() and CHAR() types is the character set (ie, binary is implemented as a kind of characters set for strings, or perhaps strings are binaries with a specific character set). 2. The inet6 storage column (which I now realize is simply BINARY(16), with a transform applied to transform strings to and from the BINARY() format, I think the technical term in the code is "fixed binary storage storage"). 3. When string (character) data gets sent *to* an inet6 column a "character set conversion" is performed, and as is the case with UDFs, it's already BINARY, so no conversion is actually performed, and since the BINARY length is now *not* 16 bytes, we get the above warning, and thus comparisons are as if with a NULL value, and always false. I've been contemplating possible "solutions" to the problem without breaking backwards compatibility, and it's tricky. The below aims at a simple way of specifying the return "character set" of a string return in more detail. And possibly even looking at the character sets for parameters. MySQL have started down a path (as per the tale above) whereby it's using some other mechanism to give indications about character sets. I don't like the particular mechanism, but it could work, looks like it does only character set though, in which case for my example above, I could just force both input and output to "latin1" so that MySQL/MariaDB is forced to convert between that and INET6 again. I think we can do better. I'm not convinced their change will permit backwards compatibility - ie, any UDF that uses those functions will require them and cannot opt to operate in a degraded mode. Not sure that's even a desirable thing to have though ... so perhaps copying what they're doing here is the way to go. My ideas: Option 1. Simply permit specifying a more precise string type during function creation. For example, instead of: MariaDB [uls]> CREATE FUNCTION uls_inet6_last_address RETURNS STRING SONAME 'uls_inet6.so'; Do this: MariaDB [uls]> CREATE FUNCTION uls_inet6_last_address RETURNS INET6 SONAME 'uls_inet6.so'; And this is permitted because INET6 is a specialization of BINARY, which is really what MySQL passes to/from UDF functions anyway. The downside is this will only work for returns. This would, work for my use case, since from INET6 => UDF the decoder is called anyway, so I do get the text presentation on function entry, and now I can now I just need to return the binary form of INET6. If sent back to the client it gets converted by the server to This eliminates at least *one* of the two pointless conversions to/from binary format. Option 2. A mechanism to pass the character set/binary type upon entry and exit. What I'm thinking here is a non-backwards compatible change, potentially. Or use the *extension pointer somehow. I'm still having some trouble navigating the codebase to figure out exactly how the void *extension pointers in UDF_ARGS and UDF_INIT structs are used. But I'm thinking they can be (ab)used to somehow pass an additional structure, which may be how 3 is achieved perhaps? To be more precise, the current UDF_ARGS looks like: 47 typedef struct UDF_ARGS { 48 unsigned int arg_count; /**< Number of arguments */ 49 enum Item_result *arg_type; /**< Pointer to item_results */ 50 char **args; /**< Pointer to argument */ 51 unsigned long *lengths; /**< Length of string arguments */ 52 char *maybe_null; /**< Set to 1 for all maybe_null args */ 53 char **attributes; /**< Pointer to attribute name */ 54 unsigned long *attribute_lengths; /**< Length of attribute arguments */ 55 void *extension; 56 } UDF_ARGS; I can't find any source files that references both the UDF_ARGS type and references this extension field. As such I'm hoping I can safely assume that in current implementations this will always be NULL. If so it becomes possible to turn this into a pointer-sized version field, or struct size field, ie something like: union { void *extension; size_t argsize; /* one alternative */ long api_version; /* another alternative */ }; for argsize this way it becomes possible to check that this field is >= sizeof(UDF_ARGS) against which the UDF was compiled. This *feels* flakey, but should be fine. If a user wants more fine-grained compile backwards compat, then the user can check that all fields (further than those listed) are positioned inside the structure. Another option is simply to have extension be a pointer to some UDF_ARGS_EXPANDED struct, which really is a pointer back to UDF_ARGS which is really something like: typedef struct UDF2_ARGS { ... everything from above, except void* extension becomes: void *UDF2_ARGS extension; /* this points to self */ size_t argsize; /* or long api_version */ bool udf2_supported; /* set to false by the server, if the UDF supports this, set to true */ const char ** string_arg_types; /* in _init this will be passed as the raw type, eg, INET6 if the input is an INET6 field, can be set in _init to have the server cast the type, ie, perform some conversion */ ... additional fields; } UDF2_ARGS; So in my example, if udf2_supported comes back as false after _init, then current behaviour is maintained, and INET6 will be converted to string, and return strings are assumed to be BINARY (current behaviour). If however, udf2_supported, then my code could set string_args_types[0] = "inet6"; - which the engine knows is a BINARY(16) type, and use the INET6 conversion code to convert to and from BINARY(16) as needed. Option 3. Some other more formal UDF API versioning/api definition/scheme. So call this UDFv2, and make this extensible so that features can be "negotiated", or that the UDF itself can even "fail to load" in case where it requires support from the running server that's not provided. The idea here would be that a symbol like "mariadb_udf_init" is exported, passing it some structure that defines the capabilities of the server it's running on, the module can then decline to load, or proceed, but limit itself based on functionality available kind of thing. I'm guessing option 2 is also a way of achieving this. And may be sufficient. I do however like the idea of having a {func}_load and {func}_unload call (some of our PBKDF functions would also benefit from this in that we can load the openssl algorithms once at startup instead of every time on _init. There may be other complications though, but still, this could be one suggestion for v2. I'm happy to write a more formal specification based off of the informal specification, and I'm inclined to attempt option 2 above. I just honestly have no idea what's all REALLY involved, and where to find the various bits and pieces of code. So I guess the initial steps would be: 1. Add a _load function which is invoked at *load* time, passing server and version information. Not sure what else could be useful, intension is to initialise the UDF for use as needed. UDFs that want to maintain backwards compatible would need to track if this was indeed called or not, and if not, act accordingly, or if it can't be backwards compatible without this, error out in _init. Suggestions as to parameters to pass would be great, I suggest we use a struct again with a size field as it's first field to ensure that we can add extra fields at a later stage, safely. Eg: struct UDF_LOADINFO { size_t load_size; const char* server; struct { uint16_t major, minor, patch; /* 10, 6, 17 - for example what I've got runing currently */ const char* extra; /* MariaDB-log, the stuff after the dash from SELECT version(); */ } version; } UDF_LOADINFO; And so we will call "bool (*func_load)(const UDF_LOADINFO*)" IFF it exist. 2. Add an _unload function "void (*func_unload)()", purpose of which is to clean up after _load. Note: It's possible to use __attribute__((constructor)) and destructor functions, but my experience is that they're messy, and if you have multiple functions in a UDF you can't know which ones you're being loaded for. 3. Expand for UDF2_ARGS as explained in option 2 above, as well as UDF2_INIT. 4. Ability to flag functions as *deterministic* (same return for same arguments) rather than merely const or not const. This way SQL can optimize calls in the same manner as for stored functions. I'm not going to dig into that, but happy to add the flag. 5. Document and flag this as UDFv2 as and when it hits formal release. Kind regards, Jaco

2 7

Re: 98ebe0a3afc: MDEV-19123 Change default charset from latin1 to utf8mb4
by Sergei Golubchik 17 Jun '24

17 Jun '24

Hi, Alexander, No comments about actual code changes, only about tests. Please, see below On Jun 12, Alexander Barkov wrote: > revision-id: 98ebe0a3afc (mariadb-11.5.1-12-g98ebe0a3afc) > parent(s): 186a30de58b > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-06-11 14:17:11 +0400 > message: > > MDEV-19123 Change default charset from latin1 to utf8mb4 > > Changing the default server character set from latin1 to utf8mb4. > diff --git a/mysql-test/main/column_compression.test b/mysql-test/main/column_compression.test > --- a/mysql-test/main/column_compression.test > +++ b/mysql-test/main/column_compression.test > @@ -9,20 +9,20 @@ let $typec= BLOB COMPRESSED; > let $typeu= BLOB; > --source column_compression.inc > > -let $typec= TEXT COMPRESSED; > -let $typeu= TEXT; > +let $typec= TEXT COMPRESSED CHARACTER SET latin1; > +let $typeu= TEXT CHARACTER SET latin1; why? > --source column_compression.inc > > let $typec= VARBINARY(10000) COMPRESSED; > let $typeu= VARBINARY(10000); > --source column_compression.inc > > -let $typec= VARCHAR(10000) COMPRESSED; > -let $typeu= VARCHAR(10000); > +let $typec= VARCHAR(10000) COMPRESSED CHARACTER SET latin1; > +let $typeu= VARCHAR(10000) CHARACTER SET latin1; > --source column_compression.inc > > let $typec= TEXT COMPRESSED CHARSET ucs2; > -let $typeu= TEXT; > +let $typeu= TEXT CHARACTER SET latin1; > --source column_compression.inc > > SET column_compression_zlib_wrap=DEFAULT; > diff --git a/mysql-test/main/create.result b/mysql-test/main/create.result > --- a/mysql-test/main/create.result > +++ b/mysql-test/main/create.result > @@ -506,9 +506,9 @@ FROM t1; > SHOW CREATE TABLE t2; > Table Create Table > t2 CREATE TABLE `t2` ( > - `ifnull(c_tinytext, CAST('yet another binary data' AS BINARY))` tinyblob DEFAULT NULL, > - `ifnull(c_text, CAST('yet another binary data' AS BINARY))` blob DEFAULT NULL, > - `ifnull(c_mediumtext, CAST('yet another binary data' AS BINARY))` mediumblob DEFAULT NULL, > + `ifnull(c_tinytext, CAST('yet another binary data' AS BINARY))` blob DEFAULT NULL, > + `ifnull(c_text, CAST('yet another binary data' AS BINARY))` mediumblob DEFAULT NULL, > + `ifnull(c_mediumtext, CAST('yet another binary data' AS BINARY))` longblob DEFAULT NULL, looks wrong UPD: there were more changes like this below, I didn't comment on them > `ifnull(c_longtext, CAST('yet another binary data' AS BINARY))` longblob DEFAULT NULL > ) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_uca1400_ai_ci > DROP TABLE t2; > diff --git a/mysql-test/main/ctype_utf8_def_upgrade.result b/mysql-test/main/ctype_utf8_def_upgrade.result > --- a/mysql-test/main/ctype_utf8_def_upgrade.result > +++ b/mysql-test/main/ctype_utf8_def_upgrade.result > @@ -61,23 +61,23 @@ t1 CREATE TABLE `t1` ( > PRIMARY KEY (`Host`,`Db`) > ) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_uca1400_ai_ci COMMENT='Host privileges; Merged with database privileges' > DROP TABLE t1; > -SET @@character_set_database=DEFAULT; > +SET @@collation_database=DEFAULT; why? > # Now do the same, but doing 'ALTER DATABASE' to create the db.opt file, > # instead of setting variables directly. > # Emulate a pre-4.1 database without db.opt > SHOW CREATE DATABASE db1; > Database Create Database > -db1 CREATE DATABASE `db1` /*!40100 DEFAULT CHARACTER SET utf8mb3 COLLATE utf8mb3_uca1400_ai_ci */ > +db1 CREATE DATABASE `db1` /*!40100 DEFAULT CHARACTER SET utf8mb3 COLLATE utf8mb3_general_ci */ > USE db1; > -SELECT @@character_set_database, 'taken from defaults' AS comment; > -@@character_set_database comment > -utf8mb3 taken from defaults > +SELECT @@collation_database, 'taken from defaults' AS comment; > +@@collation_database comment > +utf8mb3_general_ci taken from defaults > USE test; > ALTER DATABASE db1 DEFAULT CHARACTER SET latin1; > USE db1; > -SELECT @@character_set_database, 'taken from db.opt' AS comment; > -@@character_set_database comment > -latin1 taken from db.opt > +SELECT @@collation_database, 'taken from db.opt' AS comment; > +@@collation_database comment > +latin1_swedish_ci taken from db.opt > SELECT COUNT(*) FROM t1; > ERROR HY000: Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You can try REPAIR TABLE ... USE_FRM possibly followed by ALTER TABLE ... FORCE or dump and restore the table to fix this" from storage engine MyISAM > REPAIR TABLE t1 USE_FRM; > diff --git a/mysql-test/main/endspace.test b/mysql-test/main/endspace.test > --- a/mysql-test/main/endspace.test > +++ b/mysql-test/main/endspace.test > @@ -13,7 +13,7 @@ drop table if exists t1; > # Test MyISAM tables. > # > > -create table t1 (text1 varchar(32) not NULL, KEY key1 (text1)); > +create table t1 (text1 varchar(32) not NULL, KEY key1 (text1)) charset=latin1; why is that? > insert into t1 values ('teststring'), ('nothing'), ('teststring\t'); > check table t1; > select * from t1 ignore key (key1) where text1='teststring' or > diff --git a/mysql-test/main/func_compress.test b/mysql-test/main/func_compress.test > --- a/mysql-test/main/func_compress.test > +++ b/mysql-test/main/func_compress.test > @@ -1,5 +1,9 @@ > -- source include/have_compress.inc > -- source include/have_normal_zlib.inc > + > +--source include/test_db_charset_latin1.inc same as in column_compression.test ? > + > + > # > # Test for compress and uncompress functions: > # > diff --git a/mysql-test/main/long_unique.result b/mysql-test/main/long_unique.result > --- a/mysql-test/main/long_unique.result > +++ b/mysql-test/main/long_unique.result > @@ -41,7 +41,7 @@ Ignored NO > > MyISAM file: DATADIR/test/t1 > Record format: Packed > -Character set: latin1_swedish_ci (8) > +Character set: ? (0) huh? > Data records: 10 Deleted blocks: 0 > Recordlength: 12 > > diff --git a/mysql-test/main/mysqlbinlog_row_compressed.result b/mysql-test/main/mysqlbinlog_row_compressed.result > --- a/mysql-test/main/mysqlbinlog_row_compressed.result > +++ b/mysql-test/main/mysqlbinlog_row_compressed.result > @@ -40,9 +40,10 @@ SET @@session.sql_mode=#/*!*/; > SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/; > /*!\C latin1 *//*!*/; > SET @@session.character_set_client=latin1,@@session.collation_connection=8,@@session.collation_server=#/*!*/; > +SET @@session.character_set_collations='utf8mb3=utf8mb3_uca1400_ai_ci,ucs2=ucs2_uca1400_ai_ci,utf8mb4=utf8mb4_uca1400_ai_ci,utf16=utf16_uca1400_ai_ci,utf32=utf32_uca1400_ai_ci'/*!*/; why did this appear? > SET @@session.lc_time_names=0/*!*/; > SET @@session.collation_database=DEFAULT/*!*/; > -CREATE TABLE t1 (pk INT PRIMARY KEY, f1 INT, f2 INT, f3 TINYINT, f4 MEDIUMINT, f5 BIGINT, f6 INT, f7 INT, f8 char(1)) > +CREATE TABLE t1 (pk INT PRIMARY KEY, f1 INT, f2 INT, f3 TINYINT, f4 MEDIUMINT, f5 BIGINT, f6 INT, f7 INT, f8 char(1)) CHARSET=latin1 > /*!*/; > # at <pos> > #<date> server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 ddl thread_id=TID > diff --git a/mysql-test/main/sp.result b/mysql-test/main/sp.result > --- a/mysql-test/main/sp.result > +++ b/mysql-test/main/sp.result > @@ -6800,6 +6800,8 @@ DROP FUNCTION IF EXISTS f2; > # > > CREATE FUNCTION f1() RETURNS VARCHAR(65525) RETURN 'Hello'; > +Warnings: > +Note 1246 Converting column '' from VARCHAR to TEXT this could use latin1, I suppose > > CREATE FUNCTION f2() RETURNS TINYINT RETURN 1; > > diff --git a/mysql-test/suite/binlog_encryption/rpl_special_charset.opt b/mysql-test/suite/binlog_encryption/rpl_special_charset.opt > --- a/mysql-test/suite/binlog_encryption/rpl_special_charset.opt > +++ b/mysql-test/suite/binlog_encryption/rpl_special_charset.opt > @@ -1 +1 @@ > ---character-set-server=utf16 > +--character-set-server=utf16 --collation-server=utf16_general_ci why? > diff --git a/mysql-test/suite/federated/assisted_discovery.test b/mysql-test/suite/federated/assisted_discovery.test > --- a/mysql-test/suite/federated/assisted_discovery.test > +++ b/mysql-test/suite/federated/assisted_discovery.test > @@ -38,7 +38,7 @@ create table t1 ( > d varchar(4096) not null, > primary key (a), > key (b,c,d(255)) > -); > +) CHARSET=latin1; why? > show create table t1; > > connection master; > diff --git a/mysql-test/suite/funcs_1/r/is_columns_innodb.result b/mysql-test/suite/funcs_1/r/is_columns_innodb.result > --- a/mysql-test/suite/funcs_1/r/is_columns_innodb.result > +++ b/mysql-test/suite/funcs_1/r/is_columns_innodb.result > @@ -739,6 +740,9 @@ WHERE table_schema LIKE 'test%' > AND CHARACTER_OCTET_LENGTH / CHARACTER_MAXIMUM_LENGTH <> 1 > ORDER BY CHARACTER_SET_NAME, COLLATION_NAME, COL_CML; > COL_CML DATA_TYPE CHARACTER_SET_NAME COLLATION_NAME > +4.0000 char utf8mb4 utf8mb4_uca1400_ai_ci > +4.0000 enum utf8mb4 utf8mb4_uca1400_ai_ci > +4.0000 set utf8mb4 utf8mb4_uca1400_ai_ci I suppose this changed the intention of the test > Warnings: > Warning 1365 Division by 0 > Warning 1365 Division by 0 > diff --git a/mysql-test/suite/gcol/r/innodb_virtual_index.result b/mysql-test/suite/gcol/r/innodb_virtual_index.result > --- a/mysql-test/suite/gcol/r/innodb_virtual_index.result > +++ b/mysql-test/suite/gcol/r/innodb_virtual_index.result > @@ -114,7 +114,7 @@ KEY `vbidxcol` (`vbidxcol`), > KEY `a_2` (`a`,`vbidxcol`), > KEY `vbidxcol_2` (`vbidxcol`), > FULLTEXT KEY `ftsic` (`c`,`b`) > -) ENGINE=InnoDB; > +) ENGINE=InnoDB CHARSET=latin1; why latin1 everywhere in this file? > Warnings: > Note 1831 Duplicate index `vbidxcol_2`. This is deprecated and will be disallowed in a future release > ALTER TABLE ibstd_08 ADD COLUMN nc07006 BIGINT AUTO_INCREMENT NOT NULL , ADD KEY auto_nc07006(nc07006); > diff --git a/mysql-test/suite/innodb/r/online_table_rebuild.result b/mysql-test/suite/innodb/r/online_table_rebuild.result > --- a/mysql-test/suite/innodb/r/online_table_rebuild.result > +++ b/mysql-test/suite/innodb/r/online_table_rebuild.result > @@ -10,7 +10,7 @@ INSERT INTO t1 VALUES(2, repeat('b', 100), repeat('a', 100)); > COMMIT; > SET DEBUG_SYNC="now SIGNAL dml_commit"; > connection default; > -ERROR 23000: Duplicate entry 'bbbbbbbbbb' for key 'f2' > +ERROR 23000: Duplicate entry 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb' for key 'f2' why? > connection default; > SET DEBUG_SYNC="inplace_after_index_build SIGNAL dml_start WAIT_FOR dml_commit"; > ALTER TABLE t1 ADD PRIMARY KEY(f1); > diff --git a/mysql-test/suite/innodb/t/alter_primary_key.test b/mysql-test/suite/innodb/t/alter_primary_key.test > --- a/mysql-test/suite/innodb/t/alter_primary_key.test > +++ b/mysql-test/suite/innodb/t/alter_primary_key.test > @@ -1,6 +1,9 @@ > --source innodb_default_row_format.inc > --source include/have_debug.inc > --source include/have_debug_sync.inc > +--disable_query_log > +--source include/test_db_charset_latin1.inc > +--enable_query_log you don't disable query log in other tests, why here? > > --echo # > --echo # MDEV-23244 ALTER TABLE…ADD PRIMARY KEY fails to flag > diff --git a/mysql-test/suite/perfschema/t/short_option_1-master.opt b/mysql-test/suite/perfschema/t/short_option_1-master.opt > --- a/mysql-test/suite/perfschema/t/short_option_1-master.opt > +++ b/mysql-test/suite/perfschema/t/short_option_1-master.opt > @@ -1 +1 @@ > --a -Cutf8 -W1 > +-a -Cutf8 -W1 --collation-server=utf8_general_ci was it necessary? > diff --git a/mysql-test/suite/rpl/include/rpl_charset.inc b/mysql-test/suite/rpl/include/rpl_charset.inc > --- a/mysql-test/suite/rpl/include/rpl_charset.inc > +++ b/mysql-test/suite/rpl/include/rpl_charset.inc > @@ -145,4 +145,11 @@ sync_slave_with_master; > --echo # > > > +connection master; > +SET GLOBAL collation_server=utf8mb4_uca1400_ai_ci; > +SET SESSION collation_server=utf8mb4_uca1400_ai_ci; > +connection slave; > +SET GLOBAL collation_server=utf8mb4_uca1400_ai_ci; > +SET SESSION collation_server=utf8mb4_uca1400_ai_ci; why? > + > --source include/rpl_end.inc > diff --git a/mysql-test/suite/sys_vars/inc/secure_timestamp_func.inc b/mysql-test/suite/sys_vars/inc/secure_timestamp_func.inc > --- a/mysql-test/suite/sys_vars/inc/secure_timestamp_func.inc > +++ b/mysql-test/suite/sys_vars/inc/secure_timestamp_func.inc > @@ -58,7 +58,7 @@ set time_zone='+00:00'; > set timestamp=1234567890.101112; > select @@timestamp, now(6); > > -create table t1 (b varchar(20), a timestamp(6) default current_timestamp(6)); > +create table t1 (b varchar(20), a timestamp(6) default current_timestamp(6)) charset=latin1; why? charset shouldn't affect secure_timestamp functionality (same in all other secure_timestamp tests) > insert t1 (b) values ('replicated'); > sync_slave_with_master; > create trigger t1rbr before insert on t1 for each row set new.a=now(6); > diff --git a/storage/connect/mysql-test/connect/r/bson.result b/storage/connect/mysql-test/connect/r/bson.result > --- a/storage/connect/mysql-test/connect/r/bson.result > +++ b/storage/connect/mysql-test/connect/r/bson.result > @@ -321,7 +321,7 @@ WHO CHAR(12), > WEEK INT(2) JPATH='$.WEEK[2].NUMBER', > WHAT CHAR(32) JPATH='$.WEEK[2].EXPENSE[*].WHAT', > AMOUNT DOUBLE(8,2) JPATH='$.WEEK[2].EXPENSE[*].AMOUNT') > -ENGINE=CONNECT TABLE_TYPE=BSON FILE_NAME='expense.json'; > +ENGINE=CONNECT CHARSET=latin1 TABLE_TYPE=BSON FILE_NAME='expense.json'; is it necessary? are there any connect/bson tests for utf8mb4? > SELECT * FROM t4; > WHO WEEK WHAT AMOUNT > Joe 5 Beer 14.00 > diff --git a/storage/connect/mysql-test/connect/r/endian.result b/storage/connect/mysql-test/connect/r/endian.result > --- a/storage/connect/mysql-test/connect/r/endian.result > +++ b/storage/connect/mysql-test/connect/r/endian.result > @@ -10,7 +10,7 @@ birth DATE NOT NULL FIELD_FORMAT='L', > id CHAR(5) NOT NULL FIELD_FORMAT='L2', > salary DOUBLE(9,2) NOT NULL DEFAULT 0.00 FIELD_FORMAT='LF', > dept INT(4) NOT NULL FIELD_FORMAT='L2' > -) ENGINE=CONNECT TABLE_TYPE=BIN BLOCK_SIZE=5 FILE_NAME='Testbal.dat'; > +) ENGINE=CONNECT CHARSET=latin1 TABLE_TYPE=BIN BLOCK_SIZE=5 FILE_NAME='Testbal.dat'; or any connect utf8mb4 tests whatsoever? > SELECT * FROM t1; > fig name birth id salary dept > 5500 ARCHIBALD 1980-01-25 3789 4380.50 318 > diff --git a/storage/rocksdb/mysql-test/rocksdb/r/checksum_table_live.result b/storage/rocksdb/mysql-test/rocksdb/r/checksum_table_live.result > --- a/storage/rocksdb/mysql-test/rocksdb/r/checksum_table_live.result > +++ b/storage/rocksdb/mysql-test/rocksdb/r/checksum_table_live.result > @@ -1,7 +1,7 @@ > DROP TABLE IF EXISTS t1,t2; > -CREATE TABLE t1 (a INT PRIMARY KEY, b CHAR(8)) ENGINE=rocksdb CHECKSUM=1; > +CREATE TABLE t1 (a INT PRIMARY KEY, b CHAR(8)) ENGINE=rocksdb CHECKSUM=1 CHARSET=latin1; again. was it necessary? are there any rocksdb tests with utf8mb4? > INSERT INTO t1 (a,b) VALUES (1,'a'),(2,'b'); > -CREATE TABLE t2 (a INT PRIMARY KEY, b CHAR(8)) ENGINE=rocksdb CHECKSUM=1; > +CREATE TABLE t2 (a INT PRIMARY KEY, b CHAR(8)) ENGINE=rocksdb CHECKSUM=1 CHARSET=latin1; > CHECKSUM TABLE t1; > Table Checksum > test.t1 4259194219 Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

Review of MDEV-30073 Wrong result on 2nd execution of PS for query
by Michael Widenius 12 Jun '24

12 Jun '24

Review of MDEV-30073 Wrong result on 2nd execution of PS for query with NOT EXISTS Bye the way, nice to see that this patch close so many different MDEVs! Please add a list of all the MDEV's closed by this patch in the commit message! MDEV-23828: query with EXISTS over subquery using mergeable derived table / view executed in PS/SP for the second time MDEV-30396: 2nd execution of query with EXISTS subquery over view / DT MDEV-31175: 2nd execution of SELECT with mergeable view / DT and aggregate function MDEV-33081: 2-nd execution of query with forced materialization of a mergeable view because of too many base tables MDEV-31269: 2nd execution of query with HAVING containing IN predicand over subquery with EXISTS in WHERE clause MDEV-31281: 2nd execution of query with IN subquery whose left operand is aggregate function or aggregate function MDEV-31178: 2 execution of query with conversion from IN predicate to IN subquery applied > index 446154e517b..d474a39e228 100644 > --- a/mysql-test/main/func_group.test > +++ b/mysql-test/main/func_group.test > @@ -1268,12 +1268,14 @@ set @@optimizer_switch='materialization=on,in_to_exists=off,semijoin=off'; > --echo # 1) Test that subquery materialization is setup for query with > --echo # premature optimize() exit due to "Impossible WHERE" > --echo # > +--disable_ps2_protocol > SELECT MIN(t2.pk) > FROM t2 JOIN t1 ON t1.pk=t2.pk > WHERE 'j' > HAVING ('m') IN ( > SELECT v > FROM t2); > +--enable_ps2_protocol Why this change? Does this produce wrong results after your patch? At least I cannot see anything in the query that would disallow double execution. If double execution does not work anymore for this one, please add a note about this in the commit message or in the test case. > diff --git a/mysql-test/main/mysqltest_ps.test b/mysql-test/main/mysqltest_ps.test > index c5a332f691f..853ce8bd38e 100644 > --- a/mysql-test/main/mysqltest_ps.test > +++ b/mysql-test/main/mysqltest_ps.test > @@ -32,3 +32,4 @@ select 1 + "2 a"; > create table t (a int primary key, b blob default ''); > select a, (2*a) AS a from t group by a; > drop table t; > + > Remove the change to the above. No need to touch this test case in this commit > diff --git a/mysql-test/main/order_by.test b/mysql-test/main/order_by.test > index c1e0c318283..7ca9a8f8def 100644 > --- a/mysql-test/main/order_by.test > +++ b/mysql-test/main/order_by.test > @@ -90,7 +90,9 @@ drop table t1; > create table t1 (i int); > insert into t1 values(1),(2),(1),(2),(1),(2),(3); > select distinct i from t1; > +--disable_ps2_protocol > select distinct i from t1 order by rand(5); > +--enable_ps2_protocol > Why disable ps2_protocol here? > diff --git a/mysql-test/main/type_date.test b/mysql-test/main/type_date.test > index 3566e427d50..5f17a893d53 100644 > --- a/mysql-test/main/type_date.test > +++ b/mysql-test/main/type_date.test > @@ -601,7 +601,7 @@ SHOW CREATE TABLE t2; > DROP TABLE t2; > DROP VIEW v1; > DROP TABLE t1; > - > +# --enable_ps2_protocol The above looks wrong. Probably leftover from some testing. Please revert the change to this file > diff --git a/mysql-test/main/type_newdecimal.test b/mysql-test/main/type_newdecimal.test > index 366199e8aa8..7223b7b44f0 100644 > --- a/mysql-test/main/type_newdecimal.test > +++ b/mysql-test/main/type_newdecimal.test > @@ -1929,7 +1929,9 @@ drop table t1; > SELECT AVG(DISTINCT 0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001); > --enable_view_protocol > CREATE TABLE t1 AS SELECT NULL AS v1; > +--disable_ps2_protocol > SELECT 1 FROM t1 GROUP BY v1 ORDER BY AVG ( from_unixtime ( '' ) ) ; > +--enable_ps2_protocol > DROP TABLE t1; Why the change? > diff --git a/mysql-test/main/win.test b/mysql-test/main/win.test > index cba6b0fd7af..00c79425fc0 100644 > --- a/mysql-test/main/win.test > +++ b/mysql-test/main/win.test > @@ -2770,6 +2770,7 @@ drop table t1; > CREATE TABLE t1 ( a char(25), b text); > INSERT INTO t1 VALUES ('foo','bar'); > > +--disable_ps2_protocol > SELECT > SUM(b) OVER (PARTITION BY a), > ROW_NUMBER() OVER (PARTITION BY b) > @@ -2777,6 +2778,7 @@ FROM t1 > GROUP BY > LEFT((SYSDATE()), 'foo') > WITH ROLLUP; > +--enable_ps2_protocol > drop table t1; Why ? > diff --git a/mysql-test/suite/federated/federatedx_create_handlers.test b/mysql-test/suite/federated/federatedx_create_handlers.test > index 8e504590282..b2884397333 100644 > --- a/mysql-test/suite/federated/federatedx_create_handlers.test > +++ b/mysql-test/suite/federated/federatedx_create_handlers.test > @@ -94,6 +94,7 @@ DEFAULT CHARSET=latin1; > INSERT INTO federated.t3 VALUES > ('yyy'), ('www'), ('yyy'), ('xxx'), ('www'), ('yyy'), ('www'); > > +--sorted_result > SELECT * > FROM federated.t3, (SELECT * FROM federated.t1 WHERE id > 3) t > WHERE federated.t3.name=t.name; > I assume this has nothing to do with the patch, but just a random failure you fixed? If that is the case, please add a comment about this file change in the commit message or move this change to another commit. > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -3027,7 +3027,11 @@ Item* Item_ref::build_clone(THD *thd) > unlikely(!(copy->ref= (Item**) alloc_root(thd->mem_root, > sizeof(Item*)))) || > unlikely(!(*copy->ref= (* ref)->build_clone(thd)))) > + { > + if (copy) > + copy->ref= 0; > return 0; > + } > return copy; > } Ok code, but I would have prefered this a bit more simple approach: Item* Item_ref::build_clone(THD *thd) { Item_ref *copy= (Item_ref *) get_copy(thd); if (likely(copy)) { if (unlikely(!(copy->ref= (Item**) alloc_root(thd->mem_root, sizeof(Item*)))) || unlikely(!(*copy->ref= (* ref)->build_clone(thd)))) { copy->ref= 0; return 0; } } return copy; } > @@ -5823,22 +5827,48 @@ Item_field::fix_outer_field(THD *thd, Field **from_field, Item **reference) > select->group_list.elements && > (place == SELECT_LIST || place == IN_HAVING)) > { > - Item_outer_ref *rf; > /* > - If an outer field is resolved in a grouping select then it > - is replaced for an Item_outer_ref object. Otherwise an > - Item_field object is used. > - The new Item_outer_ref object is saved in the inner_refs_list of > - the outer select. Here it is only created. It can be fixed only > - after the original field has been fixed and this is done in the > - fix_inner_refs() function. > - */ > - ; > - if (!(rf= new (thd->mem_root) Item_outer_ref(thd, context, this))) > - return -1; > - thd->change_item_tree(reference, rf); > - select->inner_refs_list.push_back(rf, thd->mem_root); > - rf->in_sum_func= thd->lex->in_sum_func; > + This Item_field represents a reference to a column in some > + outer select. Wrap this Item_field item into an Item_outer_ref > + object if we are at the first execution of the query or at > + processing the prepare command for it. Make this wrapping > + permanent for the first query execution so that it could be used > + for next executions of the query. > + Add the wrapper item to the list st_select_lex::inner_refs_list > + of the select against which this Item_field has been resolved. > + */ > + bool is_first_execution= thd->is_first_query_execution(); > + if (is_first_execution || > + thd->stmt_arena->is_stmt_prepare()) The above is the same as: ((stmt_arena->is_conventional() || stmt_arena->state != STMT_EXECUTED) && !stmt_arena->is_stmt_prepare()) || stmt_arena->is_stmt_prepare()) => ((stmt_arena->state == STMT_CONVENTIONAL_EXECUTION || stmt_arena->state != STMT_EXECUTED) && stmt_arena->state != STMT_INITIALIZED) || stmt_arena->state == STMT_INITIALIZED)) => (stmt_arena->state != Query_arena::STMT_EXECUTED && stmt_arena->state != Query_arena::STMT_INITIALIZED) || stmt_arena->state == Query_arena::STMT_INITIALIZED => stmt_arena->state != Query_arena::STMT_EXECUTED Is it not better to use the above as it is shorter and easier to understand as there are less hidden abstractions? Is STMT_EXECUTED safe to use? We don't set state to STMT_EXECUTE in case of an error during execute. If the error is temporary (depending on data or out of space during execution) then state will be left in STMT_INITIALIZED. Don't we have a problem in this case for the next execution? In 11.5 we can trivially simulate an error in almost any query by setting the tmp disk quota to a very low value. Could be fixed by adding a new state STMT_OPTIMIZED that tells us that all optimizations rewrites has been done and then we could check for STMT_OPTIMIZED || STMT_EXECUTED or force a re-prepare if the previous execution returned an error (may be can set state to STMT_ERROR in this case)? Another options is for a new prepare in the case the last query ended with an error. > + /* > + If an outer field is resolved in a grouping select then it > + is replaced for an Item_outer_ref object. Otherwise an > + Item_field object is used. > + The new Item_outer_ref object is saved in the inner_refs_list of > + the outer select. Here it is only created. It can be fixed only > + after the original field has been fixed and this is done in the > + fix_inner_refs() function. > + */ > + if (!(rf= new (thd->mem_root) Item_outer_ref(thd, context, this))) > + rc= -1; > + select->inner_refs_list.push_back(rf, thd->mem_root); > + if (is_first_execution) > + *reference= rf; > + else > + thd->change_item_tree(reference, rf); > + if (arena) > + thd->restore_active_arena(arena, &backup); > + if (rc) > + return rc; > + rf->in_sum_func= thd->lex->in_sum_func; > + } > } I don't like that we use 'rf' when we know it is 0. We can fix that with the following code change that also removes the 'rc' variable (shorter): { Query_arena *arena=0, backup; Item_outer_ref *rf; bool is_first_execution; if ((is_first_execution= thd->is_first_query_execution()) && !thd->is_conventional()) arena= thd->activate_stmt_arena_if_needed(&backup); /* If an outer field is resolved in a grouping select then it is replaced for an Item_outer_ref object. Otherwise an Item_field object is used. The new Item_outer_ref object is saved in the inner_refs_list of the outer select. Here it is only created. It can be fixed only after the original field has been fixed and this is done in the fix_inner_refs() function. */ if (!(rf= new (thd->mem_root) Item_outer_ref(thd, context, this))) { select->inner_refs_list.push_back(rf, thd->mem_root); if (is_first_execution) *reference= rf; else thd->change_item_tree(reference, rf); } if (arena) thd->restore_active_arena(arena, &backup); if (!rf) // Memory allocation failed return -1 rf->in_sum_func= thd->lex->in_sum_func; } Now code has exactly the same amount of jump instructions as your code, but it is shorter, avoids the usage of 'ref' and does not need 'rc'. Note that in comparing to the original code, we are creating an arena for the states: STMT_INITIALIZED, STMT_INITIALIZED_FOR_SP= 1, STMT_PREPARED= 2, STMT_ERROR= -1 This is probably ok, just wanted to ensure you are aware of this. Would it be enough/safer to just test for SMT_PREPARED to see if an arena should be used? (Just wondering) > /* > A reference is resolved to a nest level that's outer or the same as > @@ -5938,7 +5968,7 @@ Item_field::fix_outer_field(THD *thd, Field **from_field, Item **reference) > else if (ref != not_found_item) > { > Item *save; > - Item_ref *rf; > + Item_ref *rf= NULL;; remove extra ; > /* Should have been checked in resolve_ref_in_select_and_group(). */ > DBUG_ASSERT(*ref && (*ref)->is_fixed()); > @@ -5950,35 +5980,64 @@ Item_field::fix_outer_field(THD *thd, Field **from_field, Item **reference) <cut> > + This Item_field represents a reference to a column in some outer select. > + Wrap this Item_field item into an object of the Item_ref class or a class > + derived from Item_ref we are at the first execution of the query or at > + processing the prepare command for it. Make this wrapping permanent for > + the first query execution so that it could be used for next executions > + of the query. The type of the wrapper depends on the place where this > + item field occurs or on type of the query. If it is in the the having clause > + we use a wrapper of the Item_ref type. Otherwise we use a wrapper either > + of Item_direct_ref type or of Item_outer_ref. The latter is used for > + grouping queries. Such a wrapper is always added to the list inner_refs_list > + for the select against which the outer reference has been resolved. > */ > + bool is_first_execution= thd->is_first_query_execution(); > + if (is_first_execution || > + thd->stmt_arena->is_stmt_prepare()) > + { Replace with if (thd->stmt_arena->state != Query_arena::STMT_EXECUTED) { bool is_first_execution= thd->is_first_query_execution(); > + int rc= 0; > + Query_arena *arena, backup; > + arena= 0; > + if (is_first_execution) > + arena= thd->activate_stmt_arena_if_needed(&backup); > + rf= (place == IN_HAVING ? > + new (thd->mem_root) > + Item_ref(thd, context, ref, table_name, > + field_name, alias_name_used) : > + (!select->group_list.elements ? > + new (thd->mem_root) > + Item_direct_ref(thd, context, ref, table_name, > + field_name, alias_name_used) : > + new (thd->mem_root) > + Item_outer_ref(thd, context, ref, table_name, > + field_name, alias_name_used))); > + *ref= save; > + if (!rf) > + rc= -1; > + if (!rc && place != IN_HAVING && select->group_list.elements) > + { > + outer_context->select_lex->inner_refs_list.push_back( > + (Item_outer_ref*)rf, thd->mem_root); > + ((Item_outer_ref*)rf)->in_sum_func= thd->lex->in_sum_func; > + } > + if (is_first_execution) > + *reference= rf; > + else > + thd->change_item_tree(reference, rf); > + if (arena) > + thd->restore_active_arena(arena, &backup); > + if (rc) > + return rc; Pleace do same change as in previous case by using an 'if' instead of setting rc. > We can not "move" aggregate function in the place where > @@ -6003,22 +6062,46 @@ Item_field::fix_outer_field(THD *thd, Field **from_field, Item **reference) > this, (Item_ident*)*reference, false); > if (last_checked_context->select_lex->having_fix_field) > { > + This Item_field represents a reference to a column in having clause > + of some outer select. Wrap this Item_field item into an Item_ref object > + if we are at the first execution of the query or at processing the > + prepare command for it. Make this wrapping permanent for the first query > + execution so that it could be used for next executions of the query. > */ > - DBUG_ASSERT(!rf->fixed); // Assured by Item_ref() > - if (rf->fix_fields(thd, reference) || rf->check_cols(1)) > - return -1; > + Item_ref *rf; > + bool is_first_execution= thd->is_first_query_execution(); > + if (is_first_execution || > + thd->stmt_arena->is_stmt_prepare()) Fix test, as in other cases. > + { > + int rc= 0; > + Query_arena *arena, backup; > + arena= 0; > + if (is_first_execution) > + arena= thd->activate_stmt_arena_if_needed(&backup); > + rf= new (thd->mem_root) Item_ref(thd, context, > + (*from_field)->table->s->db, > + Lex_cstring_strlen((*from_field)-> > + table->alias.c_ptr()), > + field_name); > + if (!rf) > + rc= -1; > + if (is_first_execution) > + *reference= rf; > + else > + thd->change_item_tree(reference, rf); > + if (arena) > + thd->restore_active_arena(arena, &backup); > + if (rc) > + return rc; > + /* > + rf is Item_ref => never substitute other items (in this case) > + during fix_fields() => we can use rf after fix_fields() > + */ > + DBUG_ASSERT(!rf->fixed); // Assured by Item_ref() > + if (rf->fix_fields(thd, reference) || rf->check_cols(1)) > + return -1; > + } same here. Change if (!rf) rc= -1; ... if (arena) thd->restore_active_arena(arena, &backup); to if (rc) { .... } if (arena) thd->restore_active_arena(arena, &backup); > @@ -8321,6 +8404,9 @@ bool Item_ref::fix_fields(THD *thd, Item **reference) > goto error; > } > + if ((*ref)->fix_fields_if_needed(thd, reference)) > + goto error; > + > set_properties(); Add a comment when this is needed (ie, when reference <> 0 and it is not fixed. I tried to come up with a simple comment but failed :( I assume this is at least true for the prepare phase of a prepared statement. > > if ((*ref)->check_cols(1)) > @@ -9282,8 +9368,21 @@ bool Item_direct_view_ref::fix_fields(THD *thd, Item **reference) > bitmap_set_bit(fld->table->read_set, fld->field_index); > } > } > - else if ((*ref)->fix_fields_if_needed(thd, ref)) > - return TRUE; > + else > + { > + if (thd->is_noninitial_query_execution()) replace (thd->is_noninitial_query_execution()) with thd->is_stmt_executed() See later comments in definition of is_noninitial_query_execution() > + { > + bool rc= false; > + bool save_wrapper= thd->lex->current_select->no_wrap_view_item; > + thd->lex->current_select->no_wrap_view_item= TRUE; > + rc= (*ref)->fix_fields_if_needed(thd, ref); > + thd->lex->current_select->no_wrap_view_item= save_wrapper; > + if (rc) > + return TRUE; > + } > + else if ((*ref)->fix_fields_if_needed(thd, ref)) > + return TRUE; > + } Do we need no_wrap_view_item anymore (we probably do, but it is not obvious)? The original code that introduced no_wrap_view_item used it to test things in sql_base.cc but that code does not exist anymore. The only usage we have of it now is in item.cc: if (!thd->lex->current_select->no_wrap_view_item && thd->lex->in_sum_func && thd->lex->in_sum_func->nest_level == select->nest_level) set_if_bigger(thd->lex->in_sum_func->max_arg_level, select->nest_level); There is no code comment what the above code is supposed to do. Could you PLEASE add a comment about what is the purpose o the above code, especially why the nest_level is not applicable to no_wrap_view_items ? I have verified that we need !thd->lex->current_select->no_wrap_view_item here. We get crashes in main.derived_view if remove the test. > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index f8959cfd2bf..cc9c81608ca 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -4588,7 +4588,9 @@ bool Item_func_in::value_list_convert_const_to_int(THD *thd) > */ > if (arg[0]->type() != Item::NULL_ITEM && > !convert_const_to_int(thd, field_item, &arg[0])) > - all_converted= false; > + all_converted= false; > + else > + block_transform_into_subq= true; > } if (all_converted) m_comparator.set_handler(&type_handler_slonglong); The above looks strange. You are setting block_transform_into_subq to true in the case of IN (1,1) but not setting it for ("abb",NULL); I assume the idea is that if if convert_const_to_int() changes anything, then we block conversion of the IN to sub queries as our code cannot convert back to use the original code in this case? (as the translation to use subqueries is permanent). Here is a test cases the sets block_transform_into_subq to true: create table t1 (a bigint); insert into t1 select seq from seq_1_to_100; select benchmark(1,1); select * from t1 where a in ('1','2','3'); drop table t1; Is there not a better way to do this ? - Make all translations of strings to integers permantently in the statement arena - Keep the derived table around for next execution (which would be a nice speedup). If nothing can be done to improve things, we should ae least move setting block_transform_into_subq to the end of the function and fix that we only block the transformation for prepared statements: > if (all_converted) > m_comparator.set_handler(&type_handler_slonglong); -> if (all_converted) { m_comparator.set_handler(&type_handler_slonglong); if (thd->is_first_query_execution()) block_transform_into_subq= 1; } ---- > diff --git a/sql/item_cmpfunc.h b/sql/item_cmpfunc.h > index 465625f69bf..1404fa30d6b 100644 > --- a/sql/item_cmpfunc.h > +++ b/sql/item_cmpfunc.h > @@ -2401,6 +2401,7 @@ class Item_func_in :public Item_func_opt_neg, > { > return check_argument_types_like_args0(); > } > + bool block_transform_into_subq; Please add a comment about the purpose of this variable, when it should be used and how it is used. As far as I can see, this set in the case where an IN() requires a item conversion in which case we have to block it for prepared statements as we cannot roll it back. > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index ff84e6b7bf9..4dcd1f3ae9d 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -104,6 +104,16 @@ void Item_subselect::init(st_select_lex *select_lex, > unit->item= this; > engine->change_result(this, result, FALSE); > } > + else if ((unit->item->substype() == ALL_SUBS || > + unit->item->substype() == ANY_SUBS) && > + (((Item_in_subselect*) unit->item)-> > + test_set_strategy(SUBS_MAXMIN_INJECTED) || > + ((Item_in_subselect*) unit->item)-> > + test_set_strategy(SUBS_MAXMIN_ENGINE))) > + { > + unit->item= this; > + engine->change_result(this, result, FALSE); > + } > else > { > /* Better to combine the test with the previous test as the code inside the {} is identical? Note that I understand the above code to 100%, so adding a comment what the purpose of this code would be great. Also change the function 'test_set_strategy()' to 'test_choosen_strategy(SUBS_MAXMIN_INJECTED | SUBS_MAXMIN_INJECTED)' This will make the code shorter and easier to read. Later in the review there is a new definition for test_choosen_strategy(). This is very similar to test_strategy(), but also ensure that the strategy is chosen. > @@ -200,15 +211,6 @@ void Item_in_subselect::cleanup() > > void Item_allany_subselect::cleanup() > { > - /* > - The MAX/MIN transformation through injection is reverted through the > - change_item_tree() mechanism. Revert the select_lex object of the > - query to its initial state. > - */ > - for (SELECT_LEX *sl= unit->first_select(); > - sl; sl= sl->next_select()) > - if (test_set_strategy(SUBS_MAXMIN_INJECTED)) > - sl->with_sum_func= false; > Item_in_subselect::cleanup(); > } I assume the above is now removed becasue MAX/MIN optimization changes are now permanent in Item_allany_subselect::transform_into_max_min(JOIN *join). Please add an explanation for the above change to the commit message and also add a note that thanks to this change we don't need to cleanup for MAX/MIN optimization. > @@ -497,6 +505,9 @@ void Item_subselect::recalc_used_tables(st_select_lex *new_parent, > Ref_to_outside *upper; > DBUG_ENTER("recalc_used_tables"); > > + if (!unit->thd->is_first_query_execution()) > + DBUG_VOID_RETURN; With prepared statements, the optimizer plan can change for each query. How come we don't need to recalc_used_tables() just because we have done one execution before? > @@ -2143,11 +2155,11 @@ bool Item_allany_subselect::transform_into_max_min(JOIN *join) > } > if (upper_item) > upper_item->set_sum_test(item); > - thd->change_item_tree(&select_lex->ref_pointer_array[0], item); > + select_lex->ref_pointer_array[0]= item; > { > List_iterator<Item> it(select_lex->item_list); > it++; > - thd->change_item_tree(it.ref(), item); > + it.replace(item); > } Looks like we are now are doing the MAX/MIN transformation permanently. > +uint st_select_lex_unit::ub_eq_for_exists2_in() > +{ > + if (derived && derived->is_materialized_derived()) > + return 0; > + st_select_lex *sl= first_select(); > + if (sl->next_select()) > + return 0; > + uint n= sl->select_n_eq; > + for (st_select_lex_unit *unit= sl->first_inner_unit(); > + unit; > + unit= unit->next_unit()) > + { > + n+= unit->ub_eq_for_exists2_in(); > + } > + return n; > +} Please use a more clear function name. 'ub' is not clear. Please also add a function comment to explain the purpose if this function as I do not understand the purpose of it. The function is is used here: bool Item_exists_subselect::fix_fields(THD *thd, Item **ref) { DBUG_ENTER("Item_exists_subselect::fix_fields"); st_select_lex *sel= unit->first_select(); sel->select_n_reserved= unit->ub_eq_for_exists2_in(); But it's purpose is not that clear or what it trying to calculate. For example, if you have a query: SELECT (a=1) + (b=2) + (c=3) select_n_eq will be 3. In case of SELECT (a = 1 or b = 1) or (a=2 or b=4) select_n_eq will be 4 I don't see how either value can be of any use for the optimizer. Why not instead calculate the number of = operations in the top level Item_comp_and() ? You could have the counter in Item_comp_and of how many '=' comparisons there are inside it. This would give you an exact number of = that may be needed, compared to the current code. Note find_inner_outer_equalities() is only appending items from the top and level or alternatively 0 or 1 element for other items. > +bool Item_singlerow_subselect::test_set_strategy(uchar strategy_arg) > +{ > + DBUG_ASSERT(strategy_arg == SUBS_MAXMIN_INJECTED || > + strategy_arg == SUBS_MAXMIN_ENGINE); > + return ((strategy & SUBS_STRATEGY_CHOSEN) && > + (strategy & ~SUBS_STRATEGY_CHOSEN) == strategy_arg); > +} Change to the following so that we can check all bits at once, like in test_strategy(). bool Item_singlerow_subselect::test_chosen_strategy(uint strategy_arg) { DBUG_ASSERT(strategy_arg && !(strategy_arg & ~(SUBS_MAXMIN_INJECTED | SUBS_MAXMIN_ENGINE))); return ((strategy & SUBS_STRATEGY_CHOSEN) && (strategy & strategy_arg); } > diff --git a/sql/item_subselect.h b/sql/item_subselect.h > index d3572e1abe7..9e318e74bab 100644 > --- a/sql/item_subselect.h > +++ b/sql/item_subselect.h > @@ -302,6 +302,7 @@ class Item_singlerow_subselect :public Item_subselect > { > protected: > Item_cache *value, **row; > + uchar strategy; Change to uint16 to ensure that we can add more bits later without having to change a lot of code (thanks to alignment, the storage required does not change if we use uint16) > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index 4cf403c1618..e1a32185ad1 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -93,10 +93,15 @@ bool Item_sum::init_sum_func_check(THD *thd) > thd->lex->in_sum_func= this; > nest_level= thd->lex->current_select->nest_level; > ref_by= 0; > + if (!thd->is_noninitial_query_execution() || > + (thd->active_stmt_arena_to_use()->state == > + Query_arena::STMT_SP_QUERY_ARGUMENTS)) > + { > aggr_level= -1; > aggr_sel= NULL; > max_arg_level= -1; > max_sum_func_level= -1; > + } Please fix the indentation. Please add a comment for the above test. Here is a suggestion: /* Item_sum objects are persistent and does not have to be updated for pre-executed prepared statements (stmt_arena->state == EXECUTED) except when setting variables for stored procedures (state == Query_arena::STMT_SP_QUERY_ARGUMENTS)) */ > outer_fields.empty(); > return FALSE; > } > @@ -409,7 +414,7 @@ bool Item_sum::register_sum_func(THD *thd, Item **ref) > sl= sl->master_unit()->outer_select() ) > sl->master_unit()->item->get_with_sum_func_cache()->set_with_sum_func(); > } > - if (aggr_sel) > + if (aggr_sel && aggr_sel != thd->lex->current_select ) > thd->lex->current_select->mark_as_dependent(thd, aggr_sel, NULL); Remove extra space before && and before ) Please add comment under which circumstances this is true: aggr_sel != thd->lex->current_select I would like to understand why this was not needed before. > +++ b/sql/opt_subselect.cc > @@ -962,8 +962,13 @@ bool JOIN::transform_max_min_subquery() > if (!subselect || (subselect->substype() != Item_subselect::ALL_SUBS && > subselect->substype() != Item_subselect::ANY_SUBS)) > DBUG_RETURN(0); > - DBUG_RETURN(((Item_allany_subselect *) subselect)-> > - transform_into_max_min(this)); > + bool rc= false; > + Query_arena *arena, backup; > + arena= thd->activate_stmt_arena_if_needed(&backup); > + rc= (((Item_allany_subselect *) subselect)->transform_into_max_min(this)); > + if (arena) > + thd->restore_active_arena(arena, &backup); > + DBUG_RETURN(rc); > } Please update the comment for this function to make it clear that the changes to use min/max are now permanent. > > @@ -1889,7 +1894,7 @@ static bool convert_subq_to_sj(JOIN *parent_join, Item_in_subselect *subq_pred) > with thd->change_item_tree > */ > Item_func_eq *item_eq= > - new (thd->mem_root) Item_func_eq(thd, left_exp_orig, > + new (thd->mem_root) Item_func_eq(thd, left_exp, > subq_lex->ref_pointer_array[0]); > if (!item_eq) > goto restore_tl_and_exit; I assume this was a bug in the old code? I checked the history and this line has changed back and between left_exp_orig and left_exp over time. The documentation for left_expr_orig is: /* Important for PS/SP: left_expr_orig is the item that left_expr originally pointed at. That item is allocated on the statement arena, while left_expr could later be changed to something on the execution arena. */ Item *left_expr_orig; As an experiment I added left_exp_orig back and did run the test that you had changed in your commit with valgrind. All test passed. Do you have any test that shows that left_exp is the right value here or do you have any other proof why left_exp is right ? > @@ -6535,6 +6540,10 @@ bool JOIN::choose_subquery_plan(table_map join_tables) > if (is_in_subquery()) > { > in_subs= unit->item->get_IN_subquery(); > + if (in_subs->test_set_strategy(SUBS_MAXMIN_INJECTED)) > + return false; > + if (in_subs->test_set_strategy(SUBS_MAXMIN_ENGINE)) > + return false; > if (in_subs->create_in_to_exists_cond(this)) > return true; You can replace the above two changed lines with: if (in_subs->test_chosen_strategy(SUBS_MAXMIN_INJECTED | SUBS_MAXMIN_INJECTED)) > diff --git a/sql/sql_array.h b/sql/sql_array.h > index b6de1b18d78..b9fdef0623c 100644 > --- a/sql/sql_array.h > +++ b/sql/sql_array.h > @@ -39,7 +39,10 @@ template <typename Element_type> class Bounds_checked_array > > Bounds_checked_array(Element_type *el, size_t size_arg) > : m_array(el), m_size(size_arg) > - {} > + { > + if (!m_size) > + m_array= NULL; > + } Strange that one would call it with m_size() with el != 0 and m_size == 0. Sounds like a bug in the caller. I would change the above to be a DBUG_ASSERT instead. > void reset() { m_array= NULL; m_size= 0; } > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index f1fd0053a82..a6cdd218c44 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -5783,10 +5783,7 @@ find_field_in_view(THD *thd, TABLE_LIST *table_list, > { > if (!my_strcasecmp(system_charset_info, field_it.name()->str, name)) > { > - // in PS use own arena or data will be freed after prepare > - if (register_tree_change && > - thd->stmt_arena->is_stmt_prepare_or_first_stmt_execute()) > - arena= thd->activate_stmt_arena_if_needed(&backup); > + arena= thd->activate_stmt_arena_if_needed(&backup); Why this change? Will it not use statement area for a lot of cases where it did not before? like for state == STMT_EXECUTED ? - This was discussed on slack, but we did not have time to reach a conclusion. > @@ -5805,11 +5802,23 @@ find_field_in_view(THD *thd, TABLE_LIST *table_list, > the replacing item. > */ > if (*ref && !(*ref)->is_autogenerated_name()) > + { > + arena=0; > + if (thd->is_first_query_execution()) > + arena= thd->activate_stmt_arena_if_needed(&backup); > item->set_name(thd, (*ref)->name); > + if (arena) > + thd->restore_active_arena(arena, &backup); Item::set_name() does never allocate any memory, so we don't need arena here. (I checked all code on the sql level). > + } > + if (item != *ref) > + { > + if (register_tree_change && > + !thd->is_first_query_execution()) The above is same as: register_tree_change && !(stmp_arena->state != STMT_INITIALIZED && stmt_arena->state != Query_arena::STMT_EXECUTED) => register_tree_change && (stmp_arena->state == STMT_INITIALIZED || stmt_arena->state == STMT_EXECUTED)) Is the above correct and the intention of this cde? > + thd->change_item_tree(ref, item); > + else > + *ref= item; > + } You can remove the else before *ref. It is faster to update *ref than handle the jump. I am still a bit concerned that we set *ref=item when register_tree_change != 0 as our code comments says that if register_tree_change is set, then *ref will be removed before next execution and any tries to acces it can result in a core dump. > @@ -7639,10 +7648,14 @@ bool setup_fields(THD *thd, Ref_ptr_array ref_pointer_array, > TODO: remove it when (if) we made one list for allfields and > ref_pointer_array > */ > - if (!ref_pointer_array.is_null()) > + if (thd->is_first_query_execution() || > + thd->stmt_arena->is_stmt_prepare()) The new test is same as (see earlier examples): thd->stmt_arena->state != Query_arena::STMT_EXECUTED Please replace the test with the above or replace it with is_stmt_exceuted() <cut> > /* > @@ -7682,17 +7695,6 @@ bool setup_fields(THD *thd, Ref_ptr_array ref_pointer_array, > ref[0]= item; > ref.pop_front(); > } > - /* > - split_sum_func() must be called for Window Function items, see > - Item_window_func::split_sum_func. > - */ > - if (sum_func_list && > - ((item->with_sum_func() && item->type() != Item::SUM_FUNC_ITEM) || > - item->with_window_func)) > - { > - item->split_sum_func(thd, ref_pointer_array, *sum_func_list, > - SPLIT_SUM_SELECT); > - } I assume this is moved part of this from setup_fields to JOIN::prepare(). Any particular reason why this move was required ? Just curious. Still, it would be good to explain this in the commit message. (One reason is that it will allow us to remove sum_func_list from the arguments to setup_fields) I don't see any big problem with the code as it is JOIN::prepare() that is calling both setup_fields() and split_sum_func(). It will be a bit slower as we have to traverse the field_list twice, but that is not a high cost. Note that if we go with the above code change, you should also remove sum_func_list from the arguments as the above code was the only place it was used. Also, please add back the comment to the moved code: /* split_sum_func() must be called for Window Function items, see Item_window_func::split_sum_func. */ > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 7913f1a40d2..19f1ec9728e 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -3959,12 +3962,16 @@ void Query_arena::free_items() > Item *next; > DBUG_ENTER("Query_arena::free_items"); > /* This works because items are allocated on THD::mem_root */ > + for (next= free_list; next; next= next->next) > + { > + next->cleanup(); > + } Why not call cleanup in the following loop instead of looping over the elements twice (as we did before)? Is there some cases when we cannot do cleanup as part of delete_self()? If yes, please add comment in the code why we have to traverse the list twice as otherwise it is very likely some developer will try to optimize the first loop away while looking at the code. If this is the case, is this not a problem also for free_items() ? > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 7e5d9ac96e3..2f05b2dbd03 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -2569,6 +2569,20 @@ class THD: public THD_count, /* this must be first */ > return static_cast<PSI_thread*>(my_atomic_loadptr((void*volatile*)&m_psi)); > } > > + inline bool is_first_query_execution() > + { > + return (stmt_arena->is_conventional() || > + stmt_arena->state != Query_arena::STMT_EXECUTED) && > + !stmt_arena->is_stmt_prepare(); (stmt_arena->is_conventional() || stmt_arena->state != Query_arena::STMT_EXECUTED) && !stmt_arena->is_stmt_prepare() => (stmt_arena->state == STMT_CONVENTIONAL_EXECUTION || stmt_arena->state != STMT_EXECUTED) && stmp_arena->state != STMT_INITIALIZED) => stmt_arena->state != STMT_INITIALIZED && stmt_arena->state != STMT_EXECUTED Simpler and easier to understand. > + } > + > + inline bool is_noninitial_query_execution() > + { > + return !stmt_arena->is_conventional() && > + stmt_arena->state == Query_arena::STMT_EXECUTED && > + !stmt_arena->is_stmt_prepare() ; Please add comment to explain this function and when to use it. (name of is_first_query_execution() is clear, but this one is not) Anyway: !stmt_arena->is_conventional() && stmt_arena->state == Query_arena::STMT_EXECUTED && !stmt_arena->is_stmt_prepare() ; => stmt_arena->state != STMT_CONVENTIONAL_EXECUTION && stmt_arena->state == STMT_EXECUTED && stmt_arena->state != STMT_INITIALIZED => stmt_arena->state == STMT_EXECUTED Better to renamed it to is_stmt_executed() and move it to where we define is_stmt_prepare() in class THD and class Query_arena > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index b669925a263..d7b7e98a6e4 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -3601,6 +3607,8 @@ ulong st_select_lex::get_table_join_options() > > uint st_select_lex::get_cardinality_of_ref_ptrs_slice(uint order_group_num_arg) > { > + if (card_of_ref_ptrs_slice) > + return card_of_ref_ptrs_slice; I assume this is just a cache and the code should work also without it? If yes, please add a comment about that. > if (!((options & SELECT_DISTINCT) && !group_list.elements)) > hidden_bit_fields= 0; > > @@ -3620,20 +3628,23 @@ uint st_select_lex::get_cardinality_of_ref_ptrs_slice(uint order_group_num_arg) > order_group_num * 2 + > hidden_bit_fields + > fields_in_window_functions; > + card_of_ref_ptrs_slice= n; > return n; > } Does caching card_of_ref_ptrs_slice really help ? (The function is quite trivial) We call the function in one place, setup_ref_array() It is called once per query (For selectr in JOIN::prepare()) I think having the variable and testing and updating clearing it for each normal query will cost us more than not having it. This assumes we have more normal queries than we have prepared statements. I would suggest we remove the variable. > -bool st_select_lex::setup_ref_array(THD *thd, uint order_group_num) > +static > +bool setup_ref_ptrs_array(THD *thd, Ref_ptr_array *ref_ptrs, uint n_elems) > { > + if (!ref_ptrs->is_null()) > + { > + if (ref_ptrs->size() >= n_elems) > + return false; > + } > Item **array= static_cast<Item**>( > thd->active_stmt_arena_to_use()->alloc(sizeof(Item*) * n_elems)); > if (likely(array != NULL)) > + *ref_ptrs= Ref_ptr_array(array, n_elems); > return array == NULL; > } I assume the change is that the new code will alloocate a new array if the old array was not big enough. That is good. Howevever it would be good to know when this can happen. For ref_pointer_array this should be impossible as we cache the value and it cannot change for two iterations. Is this reallocate needed for the save_ref_ptrs array? Please add a comment to explain in which case the old array was not big enough It would also be good to have the code for setup_ref_ptr_array, setup_ref_array and save_ref_ptrs_after_persistent_rewrites and save_ref_ptrs_if_needed close to each other as these are very similar and very related. > + > +bool st_select_lex::setup_ref_array(THD *thd, uint order_group_num) > +{ > + /* > + We have to create array in prepared statement memory if it is a > + prepared statement > + */ > + uint slice_card= !thd->is_noninitial_query_execution() ? > + get_cardinality_of_ref_ptrs_slice(order_group_num) : > + card_of_ref_ptrs_slice; > + uint n_elems= slice_card * 5; > + DBUG_ASSERT(n_elems % 5 == 0); You can remove the DBUG_ASSERT. It is obvious that this is always correct (for this code). > + return setup_ref_ptrs_array(thd, &ref_pointer_array, n_elems); > +} With the cache, why don't have to test for !thd->is_noninitial_query_execution(). This should be good enough: uint slice_card= get_cardinality_of_ref_ptrs_slice(order_group_num); With the cache, it will return at once after the first call. As the cache is updated on first run, the answer should always be the same is in your original code. > +bool st_select_lex::save_ref_ptrs_after_persistent_rewrites(THD *thd) > +{ > + uint n_elems= card_of_ref_ptrs_slice; > + if (setup_ref_ptrs_array(thd, &save_ref_ptrs, n_elems)) > + return true; > + if (!ref_pointer_array.is_null()) > + join->copy_ref_ptr_array(save_ref_ptrs, join->ref_ptr_array_slice(0)); > + return false; > +} Please add to function start: /* Ensure that setup_ref_array has been called */ DBUG_ASSERT(card_of_ref_ptrs_slice); It looks like all calls to setup_ref_ptrs_array() are using the same value for all future calls. If this is the case, why did you change setup_ref_ptrs_array() to check if number of elements has grown? If this is not needed, remove the check and add an assert() to ensure that the elements does not grow over time. > +bool st_select_lex::save_ref_ptrs_if_needed(THD *thd) > +{ > + if (thd->is_first_query_execution()) > + { > + if (save_ref_ptrs_after_persistent_rewrites(thd)) > + return true; > + } > + return false; > +} The above if is true in case of stmt_arena->state != STMT_INITIALIZED && stmt_arena->state != STMT_EXECUTED We call save_ref_prts_if_needed in JOIN::prepare() and subselect_single_select_engine::prepare(), mysql_select() and st_select_lex_unit::prepare_join I assume this is ok. However, I did find some inconsistently in how this functioin is called. JOIN::prepare() calls it on errors before returning (for most cases but not all). subselect_single_select_engine::prepare() & mysql_select() calls it if join->prepare() returns error, but not otherwise. st_select_lex_unit::prepare_join() calls it always after prepare. I would like to know when we MUST call save_ref_prts_if_needed(). Do we have to call it in case JOIN::prepare returns 0 ? It would be easier if things would be done this way: - JOIN::prepare() would always call it in case of error if there is any need to call it. - We would not have to call it in subselect_single_select_engine::prepare() or mysql_select() - We would remove the call in st_select_lex_unit::prepare_join() or alternative call it only if join->prepare() returns 0. (Depending on what is correct here) > void st_select_lex_unit::print(String *str, enum_query_type query_type) > { > if (with_clause) > @@ -4958,7 +5007,7 @@ bool st_select_lex::optimize_unflattened_subqueries(bool const_only) > } > if ((res= inner_join->optimize())) > return TRUE; > - if (!inner_join->cleaned) > + if (inner_join->thd->is_first_query_execution()) > sl->update_used_tables(); > sl->update_correlated_cache(); > is_correlated_unit|= sl->is_correlated; Fix intendation. Why is cleaned not good enough here? cleaned means that the join structure cannot be used anymore. Why not do: if (!inner_join->cleaned && inner_join->thd->is_first_query_execution()) Should be safer. > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index 3ab50d4aaa8..59029057a23 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -1045,6 +1045,8 @@ class st_select_lex_unit: public st_select_lex_node { > > bool can_be_merged(); > > + uint ub_eq_for_exists2_in(); > + > friend class st_select_lex; > }; > > @@ -1176,6 +1178,7 @@ class st_select_lex: public st_select_lex_node > uint curr_tvc_name; > /* true <=> select has been created a TVC wrapper */ > bool is_tvc_wrapper; > + uint fields_added_by_fix_inner_refs; You can remove the above variable as it is not used anywhere. It is set at: sql_select.cc:1499: select_lex->fields_added_by_fix_inner_refs= sql_lex.cc:3119: fields_added_by_fix_inner_refs= 0; but not used anywhere. > /* > Needed to correctly generate 'PRIMARY' or 'SIMPLE' for select_type column > of EXPLAIN > @@ -1201,6 +1204,8 @@ class st_select_lex: public st_select_lex_node > > /// Array of pointers to top elements of all_fields list > Ref_ptr_array ref_pointer_array; > + Ref_ptr_array save_ref_ptrs; > + uint card_of_ref_ptrs_slice; Please add uint variables together with other uint variables. Otherwise we may loose 4 byte for each variable because of alignment. > > /* > number of items in select_list and HAVING clause used to get number > @@ -1220,6 +1225,7 @@ class st_select_lex: public st_select_lex_node > uint order_group_num; > /* reserved for exists 2 in */ > uint select_n_reserved; > + uint select_n_eq; Please add a descripton for the above variable. (Name does not tell me what it is used for). > /* > it counts the number of bit fields in the SELECT list. These are used when DISTINCT is > converted to a GROUP BY involving BIT fields. > @@ -1295,6 +1301,8 @@ class st_select_lex: public st_select_lex_node > case of an error during prepare the PS is not created. > */ > uint8 changed_elements; // see TOUCHED_SEL_* > + uint8 save_uncacheable; > + uint8 save_master_uncacheable; The above two variables can be removed as these are only assigned to but never used for anything. > /* TODO: add foloowing first_* to bitmap above */ > bool first_natural_join_processing; > bool first_cond_optimization; > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 5db53b6f2b8..83569207a0c 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -4956,11 +4956,24 @@ mysql_execute_command(THD *thd) > if ((res= multi_delete_precheck(thd, all_tables))) > break; > > + if (thd->stmt_arena->is_stmt_prepare()) > + { > + if (add_item_to_list(thd, new (thd->mem_root) Item_null(thd))) > + goto error; > + } > + else if (thd->is_first_query_execution()) > + { The above tests for stmt_arena->state != STMT_INITIALIZED && stmt_arena->state != STMT_EXECUTED While .. != STMT_EXECUTED would be enough as it is tested on if above. No big deal, just and observarion > + if (select_lex->item_list.elements != 0) > + select_lex->item_list.empty(); > + bool rc= false; You don't have to set it to false here. > + Query_arena *arena= 0, backup; > + arena= thd->activate_stmt_arena_if_needed(&backup); > + rc= add_item_to_list(thd, new (thd->stmt_arena->mem_root) Item_null(thd)); > + if (arena) > + thd->restore_active_arena(arena, &backup); > + if (rc) > + goto error; > + } > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index a7b84bbfe3b..27eb466b47c 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -551,6 +551,19 @@ fix_inner_refs(THD *thd, List<Item> &all_fields, SELECT_LEX *select, > the found_in_group_by field of the references from the list. > */ > List_iterator_fast <Item_outer_ref> ref_it(select->inner_refs_list); > + > + if (thd->is_noninitial_query_execution()) > + { > + while ((ref= ref_it++)) > + { > + if (ref->found_in_select_list) > + continue; > + int el= all_fields.elements; > + all_fields.push_front(ref_pointer_array[el], thd->mem_root); > + } > + return false; > + } Please move the comment before this loop after the loop as it refers to the next loop. Also please add a short comment why the purpose of this loop is. Will this not cause the ref_pointer_array to grow for each prepared statement call (after the first one)? Looking at the other parts of your patch, I thought that the size and content of ref_pointer_array[] will be fixed after the first call. (or saved in save_ref_ptrs). Adding something to ref_pointer_array in case of STMT_EXECUTED sounds like it is going against the above principles. > + bool is_first_execution= thd->is_first_query_execution(); > + if (is_first_execution || > + thd->stmt_arena->is_stmt_prepare()) The above can be replaced with if (thd->stmt_area->state != STMT_EXECUTED) It a bit confusing that we have defined: inline bool is_stmt_execute() const { return state == STMT_PREPARED || state == STMT_EXECUTED; } as it would be been logical to have inline bool is_stmt_execute() const { return state == STMT_EXECUTED; } Adding a function is_stmt_executed() would easily be confused with is_stmt_execute(). Maybe we should rename is_stmt_execute to is_stmt_preared_or_executed() and add is_stmt_executed() ? > @@ -1336,18 +1367,13 @@ JOIN::prepare(TABLE_LIST *tables_init, COND *conds_init, uint og_num, > DBUG_RETURN(-1); > } > > - if (thd->lex->current_select->first_cond_optimization) > - { > - if ( conds && ! thd->lex->current_select->merged_into) > - select_lex->select_n_reserved= conds->exists2in_reserved_items(); > - else > - select_lex->select_n_reserved= 0; > - } I asume this is now what is calcualated in Item_exists_subselect::fix_fields() and not needed here anymore? > @@ -1463,6 +1489,38 @@ JOIN::prepare(TABLE_LIST *tables_init, COND *conds_init, uint og_num, > if (res) > DBUG_RETURN(res); > > + if (thd->is_first_query_execution() || > + thd->lex->is_ps_or_view_context_analysis()) > + { > + if (fix_inner_refs(thd, all_fields, select_lex, ref_ptrs)) > + DBUG_RETURN(-1); Remove the above outside the if as the code is identical also for the else part. > + if (thd->is_first_query_execution() && > + all_fields.elements > select_lex->item_list.elements) > + select_lex->fields_added_by_fix_inner_refs= > + select_lex->inner_refs_list.elements; You can remove the above 4 lines as fields_added_by_fix_inner_refs is not used anywhere. > + } > + else > + { > + if (fix_inner_refs(thd, all_fields, select_lex, ref_ptrs)) > + DBUG_RETURN(-1); > + select_lex->uncacheable= select_lex->save_uncacheable; > + select_lex->master_unit()->uncacheable= select_lex->save_master_uncacheable; you can remove the above as neither variable is used. > + } > + The above means you can replace the above query block with: if (fix_inner_refs(thd, all_fields, select_lex, ref_ptrs)) DBUG_RETURN(-1); > @@ -1624,6 +1681,7 @@ JOIN::prepare(TABLE_LIST *tables_init, COND *conds_init, uint og_num, > err: > delete procedure; /* purecov: inspected */ > procedure= 0; > + select_lex->save_ref_ptrs_if_needed(thd); > DBUG_RETURN(-1); /* purecov: inspected */ > } ok. Please check if there is any other case in JOIN::prepare where we return <> 0 where we should call save_ref_ptrs_if_needed(thd). By the way, I am not sure why we would save_ref_ptrs_if_needed() at all in case of errors. The state will in this case not be set to STMT_EXECUTED and thus the state should not be used by the next queryt. Please add a comment why we need the result in case of an error. > // Update used tables after all handling derived table procedures > - select_lex->update_used_tables(); > + if (select_lex->first_cond_optimization) > + select_lex->update_used_tables(); Why first_cond_optimization() instead of state != STMT_EXECUTED ? > - if (transform_max_min_subquery()) > + if (select_lex->first_cond_optimization && transform_max_min_subquery()) > DBUG_RETURN(1); /* purecov: inspected */ > > if (select_lex->first_cond_optimization) Please combine the above two ifs > @@ -2042,6 +2100,11 @@ JOIN::optimize_inner() <cut> > + select_lex->save_uncacheable= select_lex->uncacheable; > + select_lex->save_master_uncacheable= select_lex->master_unit()->uncacheable; Remove as not used. > @@ -4832,6 +4895,7 @@ mysql_select(THD *thd, TABLE_LIST *tables, List<Item> &fields, COND *conds, > if ((err= join->prepare(tables, conds, og_num, order, false, group, > having, proc_param, select_lex, unit))) > { > + select_lex->save_ref_ptrs_if_needed(thd); Can probably be removed. See previous comment > @@ -4858,6 +4922,7 @@ mysql_select(THD *thd, TABLE_LIST *tables, List<Item> &fields, COND *conds, > if ((err= join->prepare(tables, conds, og_num, order, false, group, having, > proc_param, select_lex, unit))) > { > + select_lex->save_ref_ptrs_if_needed(thd); > goto err; Can probably be removed. See previous comment > @@ -25172,14 +25253,33 @@ find_order_in_list(THD *thd, Ref_ptr_array ref_pointer_array, <cut> > - from_field= find_field_in_tables(thd, (Item_ident*) order_item, tables, > - NULL, &view_ref, IGNORE_ERRORS, FALSE, > - FALSE); > + if (!thd->is_noninitial_query_execution()) > + { > + from_field= find_field_in_tables(thd, (Item_ident*) order_item, tables, > + NULL, &view_ref, IGNORE_ERRORS, FALSE, > + FALSE); > + order->view_ref= view_ref; > + order->from_field= from_field; > + } > + else > + { > + if (order->view_ref) > + { Fix indentation > + view_ref= order->view_ref; > + from_field= order->from_field; > + } > + else > + { Fix indentation > + from_field= find_field_in_tables(thd, (Item_ident*) order_item, > + tables, NULL, &view_ref, > + IGNORE_ERRORS, FALSE, FALSE); > + } > + } > if (!from_field) > from_field= (Field*) not_found_field; > } > @@ -25232,6 +25332,12 @@ find_order_in_list(THD *thd, Ref_ptr_array ref_pointer_array, > if (found_item != not_found_item) > { > order->item= &ref_pointer_array[all_fields.elements-1-counter]; > + if (!thd->is_first_query_execution() && > + !thd->lex->is_ps_or_view_context_analysis()) > + { > + if ((*order->item)->fix_fields_if_needed_for_order_by(thd, order->item)) > + return TRUE; > + } The above code is new. Why do need this when we did not need it before? Please add a comment for the block above. > order->in_field_list= 0; > return FALSE; > } > diff --git a/sql/sql_union.cc b/sql/sql_union.cc > index c3c4198439a..5fb31ccbe46 100644 > --- a/sql/sql_union.cc > +++ b/sql/sql_union.cc > @@ -1114,6 +1114,8 @@ bool st_select_lex_unit::prepare_join(THD *thd_arg, SELECT_LEX *sl, > thd_arg->lex->proc_list.first), > sl, this); > > + sl->save_ref_ptrs_if_needed(thd); You may be able to remove the above, look at other comments. > @@ -2784,6 +2786,8 @@ bool st_select_lex::cleanup() > > if (join) > { > + if (join->thd->stmt_arena->is_stmt_prepare()) > + inner_refs_list.empty(); > List_iterator<TABLE_LIST> ti(leaf_tables); > TABLE_LIST *tbl; > while ((tbl= ti++)) > @@ -2813,7 +2817,6 @@ bool st_select_lex::cleanup() > continue; > error= (bool) ((uint) error | (uint) lex_unit->cleanup()); > } > - inner_refs_list.empty(); > exclude_from_table_unique_test= FALSE; > hidden_bit_fields= 0; > DBUG_RETURN(error); The above patch looks a bit strange. We are deleting the join in this function. Isn't the inner_refs_list depending on the join struct in any way? One effect of the above is that we are resetting inner_refs_list only later when excecuting the next query in st_select_lex::init_select() for anything else than STMT_INITIALIZED. I tried reverting the fix and I did get errors. I assume this is because we now only update inner_refs_list once and permantently during first execution? It would be good to add some kind of comment in the commit message about this change. Why only is_stmt_prepare and not also for STMT_CONVENTIONAL_EXECUTION ? Regards, Monty

1 0

Re: 175adef14ef: MDEV-32537 due to Linux, restrict thread name 15 characters, also in PS.
by Sergei Golubchik 11 Jun '24

11 Jun '24

Hi, Vladislav, On Jun 08, Vladislav Vaintroub wrote: > revision-id: 175adef14ef (mariadb-11.0.1-288-g175adef14ef) > parent(s): 4ffb583d72f > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2024-01-25 09:57:52 +0100 > message: > > MDEV-32537 due to Linux, restrict thread name 15 characters, also in PS. > > Rename some threads to workaround this restrictions, > e.g "rpl_parallel_thread"->"rpl_parallel", > "slave_background" -> "slave_bg" etc. I know I suggested that, but I'm getting a second thought now, sorry. This would break compatibility with performance schema tools, whatever they might be. I don't know if there are any, though, so may be it's not a concern. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: 4ffb583d72f: MDEV-32537 Validate my_thread_set_name parameter against performance schema names.
by Sergei Golubchik 11 Jun '24

11 Jun '24

Hi, Vladislav, On Jun 08, Vladislav Vaintroub wrote: > revision-id: 4ffb583d72f (mariadb-11.0.1-287-g4ffb583d72f) > parent(s): 7d863d3388d > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2024-01-20 11:26:21 +0100 > message: > > MDEV-32537 Validate my_thread_set_name parameter against performance schema names. > > In order to be consistent with PSI force thread name to be the same as > last part of thread_class_name e.g thread with PSI name > "thread/sql/main" must be called "main" . Due to Linux restriction on > thread name length ( 15 chars), full name can't be used. Does it have to be that way? I thought that mysql_thread_create() could automatically name the thread, why would we need to do it manually? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: MariaDB topic on GitHub
by Ian Gilfillan 11 Jun '24

11 Jun '24

Hi! Thanks for the suggestions. We'll take a look on our side at adding topics and improving the repo metadata. On Tue, Mar 19, 2024 at 8:55 AM Ian Gilfillan <ian(a)mariadb.org> wrote: > Hi! > > Thanks for the suggestions. We'll take a look on our side at adding topics > and improving the repo metadata. > > On Tue, Mar 19, 2024 at 5:20 AM Otto Kekäläinen <otto(a)kekalainen.net> > wrote: > >> Hi! >> >> I was browsing the GitHub topic 'mariadb' at >> https://github.com/topics/mariadb and to my surprise MariaDB itself >> isn't in that list at all. This seems to be due to >> https://github.com/MariaDB/server not having any topics defined at >> all. >> >> I recommend you add in the GitHub settings of >> https://github.com/MariaDB/server at least the topic 'mariadb', and >> while at it, also add link to mariadb.org. >> >> More ideas on how to improve the look and features of the GitHub page >> can be found by browsing projects like https://github.com/pulsar-edit >> and https://github.com/prisma which have all their GitHub account >> metadata polished. >> >> - Otto >> >

3 8

Re: 7d863d3388d: MDEV-32537 Name threads to improve debugging experience and diagnostics.
by Sergei Golubchik 08 Jun '24

08 Jun '24

Hi, Vladislav, I'd prefer you use my_thread_set_name() everywhere, even in linux- or windows-only code. It's not performance critical, and it'd be much easier to grep to thread names if they all are set by the same function. Otherwise ok. On Jun 08, Vladislav Vaintroub wrote: > revision-id: 7d863d3388d (mariadb-11.0.1-286-g7d863d3388d) > parent(s): 8bf9f218556 > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2024-01-18 20:35:54 +0100 > message: > > MDEV-32537 Name threads to improve debugging experience and diagnostics. > > Use SetThreadDescription/pthread_setname_np to give threads a name. > > diff --git a/mysys/my_thr_init.c b/mysys/my_thr_init.c > index 2e8decd7d06..4271935472e 100644 > --- a/mysys/my_thr_init.c > +++ b/mysys/my_thr_init.c > @@ -192,6 +192,35 @@ my_bool my_thread_global_init(void) > return 0; > } > > +#ifdef _WIN32 > +#define MAX_THREAD_NAME 256 > +#elif defined(__linux__) > +#define MAX_THREAD_NAME 16 > +#elif defined(__FreeBSD__) || defined(__OpenBSD__) > +#include <pthread_np.h> > +#endif > + > +void my_thread_set_name(const char *name) > +{ > +#ifdef _WIN32 > + wchar_t wname[MAX_THREAD_NAME]; > + wname[0]= 0; > + MultiByteToWideChar(CP_UTF8, 0, name, -1, wname, MAX_THREAD_NAME); > + SetThreadDescription(GetCurrentThread(), wname); > +#elif defined __linux__ > + char shortname[MAX_THREAD_NAME]; > + snprintf(shortname, MAX_THREAD_NAME, "%s", name); > + pthread_setname_np(pthread_self(), shortname); > +#elif defined __NetBSD__ > + pthread_setname_np(pthread_self(), "%s", (void *) name); > +#elif defined __FreeBSD__ || defined __OpenBSD__ > + pthread_set_name_np(pthread_self(), name); > +#elif defined __APPLE__ > + pthread_setname_np(name); > +#else > + (void) name; > +#endif > +} > > /** > End the mysys thread system. Called when ending the last thread > diff --git a/tpool/aio_linux.cc b/tpool/aio_linux.cc > index 507c6b9264f..f39b1e9b457 100644 > --- a/tpool/aio_linux.cc > +++ b/tpool/aio_linux.cc > @@ -93,6 +94,7 @@ class aio_linux final : public aio > > static void getevent_thread_routine(aio_linux *aio) > { > + pthread_setname_np(pthread_self(), "my_getevents"); > /* > We collect events in small batches to hopefully reduce the > number of system calls. > diff --git a/tpool/aio_win.cc b/tpool/aio_win.cc > index b44f705bd1e..930bab88777 100644 > --- a/tpool/aio_win.cc > +++ b/tpool/aio_win.cc > @@ -92,6 +92,7 @@ class tpool_generic_win_aio : public aio > > static void aio_completion_thread_proc(tpool_generic_win_aio* aio) > { > + SetThreadDescription(GetCurrentThread(), L"aio_completion_thread_proc"); > aio->completion_thread_work(); > } > > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: [PATCH] MDEV-21322: Report slave progress to the master
by Kristian Nielsen 04 Jun '24

04 Jun '24

Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > From: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > > This patch extends the command `SHOW REPLICA HOSTS` with three > columns: Hi Brandon, please find my review of MDEV-21322 below. > This patch extends the command `SHOW REPLICA HOSTS` with three > columns: > > 1) Gtid_State_Sent. This represents that latest GTIDs sent to the > replica in each domain. It will always be populated, regardless of > the semi-sync status (i.e. asynchronous connections will still > update this column with the latest GTID state sent to the replica). > > 2) Gtid_State_Ack. For semi-synchronous connections (only), this > column represents the last GTID in each domain that the replica has > acknowledged. Use "Pos" instead of "State", eg. Gtid_Pos_Sent and Gtid_Pos_Ack. To be consistent with eg. gtid_binlog_pos, which has last GTID per domain_id; vs. gtid_binlog_state, which has GTID per every (domain_id, server_id) combination. > @@ -2272,6 +2292,30 @@ send_event_to_slave(binlog_send_info *info, Log_event_type event_type, > return "Failed to run hook 'after_send_event'"; > } > > + if (info->thd->slave_info) > + { > + strncpy(info->thd->slave_info->gtid_state_sent.log_file, > + info->log_file_name + info->dirlen, > + strlen(info->log_file_name) - info->dirlen); > + info->thd->slave_info->gtid_state_sent.log_pos= pos; > + } Isn't there missing synchronisation here against concurrent access from another thread? I guess we need LOCK_thd_data around the strncpy(). Maybe we can avoid taking LOCK_thd_data except in the uncommon case where the gile name changes. It would be best to avoid a mutex lock/unlock for _every_ event sent to a slave. The position doesn't need locking I think, you can use an atomic std::memory_order_relaxed mostly for documentation if you like. > @@ -125,8 +114,11 @@ int THD::register_slave(uchar *packet, size_t packet_length) > if (check_access(this, PRIV_COM_REGISTER_SLAVE, any_db.str, NULL,NULL,0,0)) > return 1; > if (!(si= (Slave_info*)my_malloc(key_memory_SLAVE_INFO, sizeof(Slave_info), > - MYF(MY_WME)))) > + MYF(MY_WME|MY_ZEROFILL)))) > return 1; > + memset(si->gtid_state_sent.log_file, '\0', FN_REFLEN); > + memset(si->gtid_state_ack.log_file, '\0', FN_REFLEN); I think it's good to add the MY_ZEROFILL here, but then the two memset() are redundant and can be removed. > @@ -1235,6 +1251,14 @@ int Repl_semi_sync_master::update_sync_header(THD* thd, unsigned char *packet, > *need_sync= sync; > > l_end: > + if (is_on()) > + { > + thd->slave_info->sync_status= > + sync ? thd->slave_info->sync_status= > + Slave_info::SYNC_STATE_SEMI_SYNC_ACTIVE > + : thd->slave_info->sync_status= > + Slave_info::SYNC_STATE_SEMI_SYNC_STALE; > + } There's something wierd here, double assignment of thd->slave_info->sync_status, probably you just meant: thd->slave_info->sync_status= sync ? Slave_info::SYNC_STATE_SEMI_SYNC_ACTIVE : Slave_info::SYNC_STATE_SEMI_SYNC_STALE; More importantly, update_sync_header() is called just _before_ writing the event on the socket to the connected slave. Is that really the right place to do so? Shouldn't the status be updated only _after_ the event has been sent? Or does it not matter, because the actual TCP sending is asynchroneous anyway? In that case, should add a comment explaining why updating the status here is ok. > --- a/mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result > +++ b/mysql-test/suite/rpl/r/rpl_mixed_ddl_dml.result > @@ -11,8 +11,8 @@ n > 2002 > connection master; > show slave hosts; > -Server_id Host Port Master_id > -2 127.0.0.1 9999 1 > +Server_id Host Port Master_id Gtid_State_Sent Gtid_State_Ack Sync_Status > +2 127.0.0.1 9999 1 0-1-2 Asynchronous Just a remark that this output could be non-deterministic unless the test guarantees that the state will be stable at this point. So please check that the test will produce stable output; or if you already did so, then great! > +--echo # state, and Gtid_State_Sent/Ack should start, and only Gtid_State_Sent Typo, s/should start/should start empty/ ? > +--connection server_2 > +--source include/stop_slave.inc > +set global rpl_semi_sync_slave_enabled = 1; > +--source include/start_slave.inc > + > +--connection server_1 > +let $status_var= Rpl_semi_sync_master_clients; > +let $status_var_value= 1; > +source include/wait_for_status_var.inc; > + > +--replace_result $SLAVE_MYPORT SLAVE_PORT $SERVER_MYPORT_3 SLAVE2_PORT $DEFAULT_MASTER_PORT DEFAULT_PORT > +SHOW REPLICA HOSTS; Hm, won't this output be non-deterministic? From a quick look in the code, Rpl_semi_sync_master_clients is incremented from the start of mysql_binlog_send(), but the state is only set from update_sync_header() when the first event is sent to the slave. So won't there be a (small) window where the state has not yet switched to ACTIVE when this SHOW SLAVE HOSTS runs? So better wait for the state to become ACTIVE instead. Or for the expected value in Gtid_Pos_Sent, where that is non-empty. Also, better use SHOW SLAVE HOSTS here, for consistency with rest of test suite. > +--let $nextval= `SELECT max(a)+1 from t1` > +--eval insert into t1 values ($nextval) Is there a reason here to use separate --let and --eval? And not just: INSERT INTO t1 SELECT max(a)+1 FROM t1; > +--echo # Ensuring master gtid_binlog_pos matches Gtid_State_Ack > +--let $gtid_ack= query_get_value(show slave hosts, Gtid_State_Ack, 1) > +if (`SELECT strcmp("$master_gtid","$gtid_ack") != 0`) > +{ > + --echo # Master gtid_binlog_pos: $master_gtid > + --echo # Gtid_State_Ack: $gtid_ack > + --die Master's gtid_binlog_pos should match Gtid_State_Ack, but doesn't Again, sync_with_master_gtid.inc on the slave will not guarantee that the master saw the ack yet, even though it will be a rare race when it doesn't happen. So you'd need to wait for the expected value here with wait_condition.inc or somilar instead, I think. Probably the same in subsequent parts of the test case. > +--echo # > +--echo # 21322.5: When connecting a new slave (server_id 3) which initially has > +--echo # semi-sync disabled, SHOW SLAVE HOSTS on the master should show its > +--echo # Sync_Status as asynchronous (while server_id 2 is still semi-sync > +--echo # active). Sorry, it's a very long and complex test case, I didn't have the time today to carefully read and check every detail in it after this point. Please carefully check all the cases for possible races like described above. Replication tests are notorious for having non-deterministic failures (as I'm sure you've experienced first-hand), and it's important to try to avoid that as much as possible. A general remark on the test, the output of Gtid_Pos_Sent and Gtid_Pos_Ack are full GTID positions (multiple domain_id). But there is no test of having more than one domain in the output. I think you need to extend the test to test at least one case where there are multiple domains. And also need to test when the GTID state in the binlog has multiple server_id (though maybe the test case already does this, I didn't check myself). - Kristian.

2 3

Re: eba212af92d: MDEV-9101 Limit size of created disk temporary files and tables
by Sergei Golubchik 08 May '24

08 May '24

Hi, Monty, On Apr 18, Michael Widenius wrote: > commit eba212af92d > Author: Michael Widenius <monty(a)mariadb.org> > Date: Thu Mar 14 17:59:00 2024 +0100 > > diff --git a/sql/json_table.cc b/sql/json_table.cc > index 6713e796eb7..da5ab0c450a 100644 > --- a/sql/json_table.cc > +++ b/sql/json_table.cc > @@ -734,7 +734,8 @@ bool Create_json_table::finalize(THD *thd, TABLE *table, > > table->db_stat= HA_OPEN_KEYFILE; > if (unlikely(table->file->ha_open(table, table->s->path.str, O_RDWR, > - HA_OPEN_TMP_TABLE | HA_OPEN_INTERNAL_TABLE))) > + HA_OPEN_TMP_TABLE | HA_OPEN_INTERNAL_TABLE | > + HA_OPEN_SIZE_TRACKING))) shouldn't HA_OPEN_INTERNAL_TABLE always imply HA_OPEN_SIZE_TRACKING automatically? > DBUG_RETURN(true); > > table->set_created(); > diff --git a/storage/maria/ma_statrec.c b/storage/maria/ma_statrec.c > index d8a8b0a05d7..ff0c6c7a617 100644 > --- a/storage/maria/ma_statrec.c > +++ b/storage/maria/ma_statrec.c > @@ -45,6 +45,13 @@ my_bool _ma_write_static_record(MARIA_HA *info, const uchar *record) > my_errno=HA_ERR_RECORD_FILE_FULL; > return(2); > } > + if (info->s->tracked && are there temporary files that you don't want to track? are there non-temporary files that you want to track? > + _ma_update_tmp_file_size(&info->s->track_data, > + MY_ALIGN(info->s->state.state.data_file_length+ > + info->s->base.pack_reclength, > + MARIA_TRACK_INCREMENT_SIZE))) > + return(2); > + > if (info->opt_flag & WRITE_CACHE_USED) > { /* Cash in use */ > if (my_b_write(&info->rec_cache, record, > diff --git a/storage/maria/ma_page.c b/storage/maria/ma_page.c > index 5881456a69a..0877c966d1c 100644 > --- a/storage/maria/ma_page.c > +++ b/storage/maria/ma_page.c > @@ -417,6 +417,13 @@ my_off_t _ma_new(register MARIA_HA *info, int level, > DBUG_RETURN(HA_OFFSET_ERROR); > } > share->state.state.key_file_length+= block_size; > + if (info->s->tracked && > + _ma_update_tmp_file_size(&share->track_index, > + share->state.state.key_file_length)) shouldn't you do it before share->state.state.key_file_length is increased? Currently it seems that you're failing the operation, but the file will stay increased same pattern everywhere else > + { > + mysql_mutex_unlock(&share->intern_lock); > + DBUG_RETURN(HA_OFFSET_ERROR); > + } > /* Following is for not transactional tables */ > info->state->key_file_length= share->state.state.key_file_length; > mysql_mutex_unlock(&share->intern_lock); > diff --git a/storage/maria/ma_delete_all.c b/storage/maria/ma_delete_all.c > index f355d0da3e8..ed2087d3091 100644 > --- a/storage/maria/ma_delete_all.c > +++ b/storage/maria/ma_delete_all.c > @@ -132,9 +132,16 @@ int maria_delete_all_rows(MARIA_HA *info) > if (_ma_flush_table_files(info, MARIA_FLUSH_DATA|MARIA_FLUSH_INDEX, > FLUSH_IGNORE_CHANGED, FLUSH_IGNORE_CHANGED) || > mysql_file_chsize(info->dfile.file, 0, 0, MYF(MY_WME)) || > - mysql_file_chsize(share->kfile.file, share->base.keystart, 0, MYF(MY_WME))) > + mysql_file_chsize(share->kfile.file, share->base.keystart, 0, > + MYF(MY_WME))) should be > 0 here, shouldn't it? Both for kfile and dfile. > goto err; > > + if (info->s->tracked) > + { > + _ma_update_tmp_file_size(&info->s->track_data, 0); > + _ma_update_tmp_file_size(&info->s->track_index, share->base.keystart); but this shouldn't update the size if mysql_file_chsize returned -1 > + } > + > if (_ma_initialize_data_file(share, info->dfile.file)) > goto err; > > diff --git a/sql/uniques.cc b/sql/uniques.cc > index 36725e80a6b..44d93bb0f88 100644 > --- a/sql/uniques.cc > +++ b/sql/uniques.cc > @@ -720,7 +720,7 @@ bool Unique::merge(TABLE *table, uchar *buff, size_t buff_size, > /* Open cached file for table records if it isn't open */ > if (! my_b_inited(outfile) && > open_cached_file(outfile, mysql_tmpdir, TEMP_PREFIX, DISK_CHUNK_SIZE, > - MYF(MY_WME))) > + MYF(MY_WME | MY_TRACK | MY_TRACK_WITH_LIMIT))) as far as I can see, open_cached_file is only used for temporary files. it should apply MY_TRACK | MY_TRACK_WITH_LIMIT internally, it'll be less error-prone > return 1; > > bzero((char*) &sort_param,sizeof(sort_param)); > diff --git a/storage/maria/ma_write.c b/storage/maria/ma_write.c > index 9a6859bfd4d..7404db84067 100644 > --- a/storage/maria/ma_write.c > +++ b/storage/maria/ma_write.c > @@ -386,6 +387,11 @@ int maria_write(MARIA_HA *info, const uchar *record) > } > } > } > + else if (my_errno == HA_ERR_LOCAL_TMP_SPACE_FULL || > + my_errno == HA_ERR_GLOBAL_TMP_SPACE_FULL) > + { > + filepos= HA_OFFSET_ERROR; /* Avoid write_record_abort() */ why avoid? HA_ERR_LOCAL_TMP_SPACE_FULL is like EQUOT or ENOSPC. why should be masked as HA_OFFSET_ERROR, and not be a fatal error? > + } > else > fatal_error= 1; > > diff --git a/storage/maria/ma_close.c b/storage/maria/ma_close.c > index 7441e29a97b..47935d61b88 100644 > --- a/storage/maria/ma_close.c > +++ b/storage/maria/ma_close.c > @@ -14,7 +14,7 @@ > along with this program; if not, write to the Free Software > Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA */ > > -/* close a isam-database */ > +/* close an Aria table */ at last :) it's what 27 years old? > /* > TODO: > We need to have a separate mutex on the closed file to allow other threads > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index e0c7cef1a9e..4959c7833a1 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -2922,15 +2922,16 @@ bool compute_window_func(THD *thd, > if (unlikely(thd->is_error() || thd->is_killed())) > break; > > - > /* Return to current row after notifying cursors for each window > function. */ > - tbl->file->ha_rnd_pos(tbl->record[0], rowid_buf); > + if (tbl->file->ha_rnd_pos(tbl->record[0], rowid_buf)) > + return true; how can it fail? > } > > /* We now have computed values for each window function. They can now > be saved in the current row. */ > - save_window_function_values(window_functions, tbl, rowid_buf); > + if (save_window_function_values(window_functions, tbl, rowid_buf)) > + return true; > > rownum++; > } > diff --git a/mysql-test/main/status.test b/mysql-test/main/status.test > index acf29f35b9e..b6be046a9ac 100644 > --- a/mysql-test/main/status.test > +++ b/mysql-test/main/status.test > @@ -244,9 +244,9 @@ let $rnd_next = `show global status like 'handler_read_rnd_next'`; > let $tmp_table = `show global status like 'Created_tmp_tables'`; > show status like 'com_show_status'; > show status like 'hand%write%'; > -show status like '%tmp%'; > +show status like '%_tmp%'; Doesn't look like a very meaningful change. Did you mean '%\_tmp%' ? > show status like 'hand%write%'; > -show status like '%tmp%'; > +show status like '%_tmp%'; > show status like 'com_show_status'; > let $rnd_next2 = `show global status like 'handler_read_rnd_next'`; > let $tmp_table2 = `show global status like 'Created_tmp_tables'`; > diff --git a/mysys/mf_cache.c b/mysys/mf_cache.c > index 2fec59f4e70..712187f6181 100644 > --- a/mysys/mf_cache.c > +++ b/mysys/mf_cache.c > @@ -43,7 +43,7 @@ my_bool open_cached_file(IO_CACHE *cache, const char* dir, const char *prefix, > cache->file_name=0; > cache->buffer=0; /* Mark that not open */ > if (!init_io_cache(cache, -1, cache_size, WRITE_CACHE, 0L, 0, > - MYF(cache_myflags | MY_NABP))) > + MYF(cache_myflags | MY_NABP | MY_TRACK))) Why not MY_TRACK_WITH_LIMIT ? > { > DBUG_RETURN(0); > } > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 6f2797b6b39..324f7679e8f 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -22275,7 +22275,8 @@ bool open_tmp_table(TABLE *table) > int error; > if (unlikely((error= table->file->ha_open(table, table->s->path.str, O_RDWR, > HA_OPEN_TMP_TABLE | > - HA_OPEN_INTERNAL_TABLE)))) > + HA_OPEN_INTERNAL_TABLE | > + HA_OPEN_SIZE_TRACKING)))) shouldn't HA_OPEN_TMP_TABLE or HA_OPEN_INTERNAL_TABLE imply HA_OPEN_SIZE_TRACKING? > { > table->file->print_error(error, MYF(0)); /* purecov: inspected */ > table->db_stat= 0; > diff --git a/sql/handler.cc b/sql/handler.cc > index fec473d97bb..a05b7476724 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -443,9 +444,12 @@ int ha_init_errors(void) > /* Zerofill it to avoid uninitialized gaps. */ remove the comment, if you no longer zerofill > if (! (handler_errmsgs= (const char**) my_malloc(key_memory_handler_errmsgs, > HA_ERR_ERRORS * sizeof(char*), > - MYF(MY_WME | MY_ZEROFILL)))) > + MYF(MY_WME)))) > return 1; > > + /* Copy default handler error messages */ > + memcpy(handler_errmsgs, handler_error_messages, HA_ERR_ERRORS * sizeof(char*)); > + > /* Set the dedicated error messages. */ > SETMSG(HA_ERR_KEY_NOT_FOUND, ER_DEFAULT(ER_KEY_NOT_FOUND)); > SETMSG(HA_ERR_FOUND_DUPP_KEY, ER_DEFAULT(ER_DUP_KEY)); > diff --git a/storage/maria/maria_def.h b/storage/maria/maria_def.h > index 705562eb795..b256466f1f0 100644 > --- a/storage/maria/maria_def.h > +++ b/storage/maria/maria_def.h > @@ -778,6 +785,14 @@ typedef struct st_maria_share > myf write_flag; > enum data_file_type data_file_type; > enum pagecache_page_type page_type; /* value depending transactional */ > + > + /* > + tracked will cause lost bytes (not aligned) but this is ok as it is always > + used with tmp_file_tracking if set I don't understand your explanation why "this is ok" > + */ > + my_bool tracked; /* Tracked table (always internal) */ > + struct tmp_file_tracking track_data,track_index; > + > /** > if Checkpoint looking at table; protected by close_lock or THR_LOCK_maria > */ > diff --git a/include/my_sys.h b/include/my_sys.h > index af509c7df45..0df12963ab9 100644 > --- a/include/my_sys.h > +++ b/include/my_sys.h > @@ -178,6 +180,17 @@ uchar *my_large_malloc(size_t *size, myf my_flags); > void my_large_free(void *ptr, size_t size); > void my_large_page_truncate(size_t *size); > > +/* Tracking tmp file usage */ > + > +struct tmp_file_tracking > +{ > + ulonglong last_position; I had to read how "last_position" is used to realize that it's in fact previous file_size value. Why not to call it previous_file_size? Or recorded_file_size ? Or old_file_size? it's not a "position" > + ulonglong file_size; > +}; > + > +typedef int (*TMPFILE_SIZE_CB)(struct tmp_file_tracking *track, int no_error); > +extern TMPFILE_SIZE_CB update_tmp_file_size; > + > #ifdef _WIN32 > extern BOOL my_obtain_privilege(LPCSTR lpPrivilege); > #endif > diff --git a/sql/sql_class.h b/sql/sql_class.h > index c1efb49f6da..42eee9913ec 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1070,12 +1074,22 @@ typedef struct system_status_var > */ > > #define last_system_status_var questions > -#define last_cleared_system_status_var local_memory_used > + > +/* Parameters to set_status_var_init() */ > + > +#define STATUS_OFFSET(A) offsetof(STATUS_VAR,A) > +/* Clear as part of flush */ > +#define clear_up_to_tmp_space_used STATUS_OFFSET(tmp_space_used) > +/* Clear as part of startup */ > +#define clear_up_to_memory_used STATUS_OFFSET(local_memory_used) > +/* Full initialization. Note that global_memory_used is updated early! */ > +#define clear_up_to_global_memory_used STATUS_OFFSET(global_memory_used) > +#define last_restored_status_var clear_up_to_tmp_space_used those are bad names. and the fact that you need to rename these defines and change all code that uses them - this kind of shows they're not good. when there will be a new variable, clear_up_to_tmp_space_used can become clear_up_to_new_variable and everything will need to be changed again. better use names that don't depend on individual variables. last_restored_status_var and last_cleared_system_status_var are good. Try to split variables logically. First there are additive variables - global value is the sum of all local values. They're cleared, they're added up tp global, etc. Then there are non-additive, not cleared, not added - like local_memory_used and tmp_space_used, for example. And global_memory_used is special because it's initialized early, right? so, may be defines based on these properties would be easier to understand. because now I'm totally at loss when to use clear_up_to_tmp_space_used, when to use clear_up_to_memory_used, when clear_up_to_global_memory_used and when last_restored_status_var. > + > > /** Number of contiguous global status variables */ > -constexpr int COUNT_GLOBAL_STATUS_VARS= int(offsetof(STATUS_VAR, > - last_system_status_var) / > - sizeof(ulong)) + 1; > +constexpr int COUNT_GLOBAL_STATUS_VARS= > + int(STATUS_OFFSET(last_system_status_var) /sizeof(ulong)) + 1; > > /* > Global status variables > diff --git a/sql/filesort.cc b/sql/filesort.cc > index abcd4127a0b..df4508b635a 100644 > --- a/sql/filesort.cc > +++ b/sql/filesort.cc > @@ -1082,6 +1078,11 @@ static ha_rows find_all_keys(THD *thd, Sort_param *param, SQL_SELECT *select, > DBUG_RETURN(num_records); > > err: > + if (!quick_select) > + { > + (void) file->extra(HA_EXTRA_NO_CACHE); > + file->ha_rnd_end(); > + } why? > sort_form->column_bitmaps_set(save_read_set, save_write_set); > DBUG_RETURN(HA_POS_ERROR); > } /* find_all_keys */ > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index dc4a955aa79..4727b7b5413 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -1947,6 +1950,10 @@ static void mysqld_exit(int exit_code) > if (exit_code == 0 || opt_endinfo) > SAFEMALLOC_REPORT_MEMORY(0); > } > + if (global_tmp_space_used) > + fprintf(stderr, "Warning: tmp_space not freed: %llu\n", > + (ulonglong) global_tmp_space_used); This is confusing. almost all (if not all) temp space is automatically freed by the OS when the process exits, so this warning will only scare the user about lost space when in fact no space is lost, it's only some space was not accounted for inside MariaDB. I suggest either make this warning debug-only or rephrase it like "Note: %llu tmp_space was lost and will be freed automatically when the process exits" > + > DBUG_LEAVE; > #ifdef _WIN32 > my_report_svc_status(SERVICE_STOPPED, exit_code, 0); > @@ -7571,6 +7636,7 @@ SHOW_VAR status_vars[]= { > {"Tc_log_page_size", (char*) &tc_log_page_size, SHOW_LONG_NOFLUSH}, > {"Tc_log_page_waits", (char*) &tc_log_page_waits, SHOW_LONG}, > #endif > + {"tmp_space_used", (char*) offsetof(STATUS_VAR, tmp_space_used), SHOW_LONGLONG_STATUS}, first letter - upper case > #ifdef HAVE_POOL_OF_THREADS > {"Threadpool_idle_threads", (char *) &show_threadpool_idle_threads, SHOW_SIMPLE_FUNC}, > {"Threadpool_threads", (char *) &show_threadpool_threads, SHOW_SIMPLE_FUNC}, > diff --git a/sql/log.cc b/sql/log.cc > index 460cefea47b..a9bc3758dd5 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2388,6 +2384,8 @@ bool Event_log::check_write_error(THD *thd) > case ER_TRANS_CACHE_FULL: > case ER_STMT_CACHE_FULL: > case ER_ERROR_ON_WRITE: > + case EE_LOCAL_TMP_SPACE_FULL: > + case EE_GLOBAL_TMP_SPACE_FULL: not a good idea to mix EE_ OS-like errors and ER_ SQL layer errors > case ER_BINLOG_LOGGING_IMPOSSIBLE: > checked= TRUE; > break; > @@ -6650,7 +6651,8 @@ Event_log::flush_and_set_pending_rows_event(THD *thd, Rows_log_event* event, > { > set_write_error(thd, is_transactional); > if (check_cache_error(thd, cache_data) && > - stmt_has_updated_non_trans_table(thd)) > + (stmt_has_updated_non_trans_table(thd) || > + !is_transactional)) why? > cache_data->set_incident(); > delete pending; > cache_data->set_pending(NULL); > @@ -7868,8 +7877,16 @@ int Event_log::write_cache(THD *thd, binlog_cache_data *cache_data) > DBUG_RETURN(res ? ER_ERROR_ON_WRITE : 0); > } > > - if (reinit_io_cache(cache, READ_CACHE, 0, 0, 0)) > + /* > + Allow flush of transaction logs to temporary go over the tmp space limit > + as we do not want the commit to fail > + */ > + cache->myflags&= ~MY_TRACK_WITH_LIMIT; > + res= reinit_io_cache(cache, READ_CACHE, 0, 0, 0); > + cache->myflags|= MY_TRACK_WITH_LIMIT; you also remove MY_TRACK_WITH_LIMIT inside reinit_io_cache(), so this is redundant. Or it's redundant inside reinit_io_cache(), either way, no point in doing it twice > + if (res) > DBUG_RETURN(ER_ERROR_ON_WRITE); > + > /* Amount of remaining bytes in the IO_CACHE read buffer. */ > size_t log_file_pos; > uchar header_buf[LOG_EVENT_HEADER_LEN]; > diff --git a/mysys/mf_iocache.c b/mysys/mf_iocache.c > index 4ee1331bdb3..6214057f9e3 100644 > --- a/mysys/mf_iocache.c > +++ b/mysys/mf_iocache.c > @@ -73,6 +74,50 @@ int (*_my_b_encr_read)(IO_CACHE *info,uchar *Buffer,size_t Count)= 0; > int (*_my_b_encr_write)(IO_CACHE *info,const uchar *Buffer,size_t Count)= 0; > > > +static inline my_bool tmp_file_track(IO_CACHE *info, ulonglong file_size) > +{ > + if ((info->myflags & (MY_TRACK | MY_TRACK_WITH_LIMIT)) && > + update_tmp_file_size) > + { > + if (info->tracking.file_size < file_size) > + { > + int error; > + info->tracking.file_size= file_size; > + if ((error= update_tmp_file_size(&info->tracking, > + !(info->myflags & > + MY_TRACK_WITH_LIMIT)))) > + { > + if (info->myflags & MY_WME) > + my_error(my_errno, MYF(0)); > + info->error= -1; > + return 1; > + } > + } > + } > + return 0; > +} > + > +my_bool io_cache_tmp_file_track(IO_CACHE *info, ulonglong file_size) > +{ > + return tmp_file_track(info, file_size); > +} > + > + > +void end_tracking_io_cache(IO_CACHE *info) what does "end tracking IO_CACHE" mean? I read it as "end tracking of this IO_CACHE, it is no longer tracked after this function call" - but it's not what it is doing. so what how do you read this function name? > +{ > + if ((info->myflags & (MY_TRACK | MY_TRACK_WITH_LIMIT)) && > + info->tracking.file_size) > + { > + info->tracking.file_size= 0; > + update_tmp_file_size(&info->tracking, 1); > + } > +} > + > +void truncate_io_cache(IO_CACHE *info) > +{ > + if (my_chsize(info->file, 0, 0, MYF(MY_WME)) <= 0) > + end_tracking_io_cache(info); not quite. If my_chsize() returns -1, you should not do info->tracking.file_size= 0, because file size wasn't actually changed. > +} > > static void > init_functions(IO_CACHE* info) Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 0d6dc52f43d: MDEV-30046 Refactor write_record and fix idempotent replication
by Sergei Golubchik 04 May '24

04 May '24

Hi, Nikita, On Apr 29, Nikita Malyavin wrote: > revision-id: 0d6dc52f43d (mariadb-10.5.24-175-g0d6dc52f43d) > parent(s): 652e9ee405e > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-04-27 21:08:21 +0200 > message: > > MDEV-30046 Refactor write_record and fix idempotent replication > > Idempotent write_row works same as REPLACE: if there is a duplicating > record in the table, then it will be deleted and re-inserted, with the > same update optimization. > > The code in Rows:log_event::write_row was basically copy-pasted from > write_record. > > What's done: > REPLACE operation was unified across replication and sql. It is now > representred as a Write_record class, that holds the whole state, and allows > re-using some resources in between the row writes. > > Replace, IODKU and single insert implementations are split across different > methods, reluting in a much cleaner code. > > The entry point is preserved as a single Write_record::write_record() call. > The implementation to call is chosen on the constructor stage. > > This allowed several optimizations to be done: > 1. The table key list is not iterated for every row. We find last unique key in > the order of checking once and preserve it across the rows. See last_uniq_key(). > 2. ib_handler::referenced_by_foreign_key acquires a global lock. This call was > done per row as well. Not all the table config that allows optimized replace is > folded into a single boolean field can_optimize. All the fields to check are > even stored in a single register on a 64-bit platform. > 3. DUP_REPLACE and DUP_UPDATE cases now have one less level of indirection > 4. modified_non_trans_tables is checked and set only when it's really needed. > 5. Obsolete bitmap manipulations are removed. > > Also: > * Unify replace initialization step across implementations: > add prepare_for_replace and finalize_replace > * alloca is removed in favor of mem_root allocation. This memory is reused > across the rows. > * An rpl-related callback is added to the replace branch, meaning that an extra > check is made per row replace even for the common case. It can be avoided with > templates if considered a problem. > > diff --git a/mysql-test/main/long_unique_bugs_replication.test b/mysql-test/main/long_unique_bugs_replication.test > index 9c44d13e6a5..f2225cde101 100644 > --- a/mysql-test/main/long_unique_bugs_replication.test > +++ b/mysql-test/main/long_unique_bugs_replication.test > + > +--echo # > +--echo # End of 10.4 tests > +--echo # ... > --echo # > --echo # End of 10.4 tests > --echo # may be 10.5? > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 38e8b062f18..b5810eeaf7b 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -5705,10 +5708,10 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > &m_cols_ai : &m_cols); > bitmap_intersect(table->write_set, after_image); > > - this->slave_exec_mode= slave_exec_mode_options; // fix the mode > - > + COPY_INFO copy_info; > + Write_record write_record; > // Do event specific preparations > - error= do_before_row_operations(rli); > + error= do_before_row_operations(rgi, &copy_info, &write_record); You create write_record on the stack and save it inside this->m_write_record 1. why not to have it permanently a part of Rows_log_event ? 2. if you'll keep it on the stack, please, reset m_write_record=0 before returning from ::do_apply_event(). And may be even do m_write_record= &write_record here, not in ::do_before_row_operations() it'll be a bit easier to see how a local variable is stored and then freed and that it doesn't actually leave the scope. > > /* > Bug#56662 Assertion failed: next_insert_id == 0, file handler.cc > @@ -7132,6 +7148,40 @@ Write_rows_log_event::do_before_row_operations(const Slave_reporting_capability > m_table->mark_auto_increment_column(true); > } > > + if (slave_exec_mode == SLAVE_EXEC_MODE_IDEMPOTENT && > + (m_table->file->ha_table_flags() & HA_DUPLICATE_POS || > + m_table->s->long_unique_table)) > + error= m_table->file->ha_rnd_init_with_error(0); I'd put this inside if (slave_exec_mode == SLAVE_EXEC_MODE_IDEMPOTENT) about ~40 lines above the place where you put it now. > + > + if (!error) > + { > + bzero(copy_info, sizeof *copy_info); > + copy_info->handle_duplicates= > + slave_exec_mode == SLAVE_EXEC_MODE_IDEMPOTENT ? > + DUP_REPLACE : DUP_ERROR; simply copy_info->handle_duplicates= thd->lex->duplicates; > + copy_info->table_list= m_table->pos_in_table_list; > + > + int (*callback)(void *, void*)= NULL; > + if (!get_flags(COMPLETE_ROWS_F)) > + { > + /* > + If row is incomplete we will use the record found to fill > + missing columns. > + */ > + callback= [](void *e, void* r)->int { > + auto rgi= static_cast<rpl_group_info*>(r); > + auto event= static_cast<Write_rows_log_event*>(e); > + return event->incomplete_record_callback(rgi); > + }; > + } > + new (write_record) Write_record(thd, m_table, copy_info, > + m_vers_from_plain && > + m_table->versioned(VERS_TIMESTAMP), > + m_table->triggers && do_invoke_trigger(), > + NULL, callback, this, rgi); please, use write_record->init(...) here. And in all similar places below, I won't comment on them separately. > + m_write_record= write_record; > + } > + > return error; > } > > @@ -7173,7 +7223,15 @@ Write_rows_log_event::do_after_row_operations(const Slave_reporting_capability * > { > m_table->file->print_error(local_error, MYF(0)); > } > - return error? error : local_error; > + int rnd_error= 0; > + if (m_table->file->inited) > + { > + DBUG_ASSERT(slave_exec_mode == SLAVE_EXEC_MODE_IDEMPOTENT); > + DBUG_ASSERT(m_table->file->ha_table_flags() & HA_DUPLICATE_POS || > + m_table->s->long_unique_table); DBUG_ASSERT(m_table->file->inited == handler::RND); > + rnd_error= m_table->file->ha_rnd_end(); > + } > + return error? error : local_error ? local_error : rnd_error; > } > > bool Rows_log_event::process_triggers(trg_event_type event, > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index ff71361493a..e410efda121 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -677,6 +678,30 @@ Field **TABLE::field_to_fill() > return triggers && triggers->nullable_fields() ? triggers->nullable_fields() : field; > } > > +int prepare_for_replace(TABLE *table, enum_duplicates handle_duplicates, > + bool ignore) > +{ > + bool create_lookup_handler= handle_duplicates != DUP_ERROR; why do you initialize create_lookup_handler here and then immediately repeat it two lines below? :) less confusing would be bool create_lookup_handler= handle_duplicates != DUP_ERROR || ignore; if (create_lookup_handler) { > + if (ignore || handle_duplicates != DUP_ERROR) > + { > + create_lookup_handler= true; > + table->file->extra(HA_EXTRA_IGNORE_DUP_KEY); > + if (table->file->ha_table_flags() & HA_DUPLICATE_POS || > + table->s->long_unique_table) > + { > + if (table->file->ha_rnd_init_with_error(false)) > + return 1; > + } > + } > + return table->file->prepare_for_insert(create_lookup_handler); > +} > + > +int finalize_replace(TABLE *table, enum_duplicates handle_duplicates, > + bool ignore) > +{ DBUG_ASSERT(m_table->file->inited != handler::INDEX); > + return table->file->inited ? table->file->ha_rnd_end() : 0; > +} > + > > /** > INSERT statement implementation > @@ -1728,31 +1745,74 @@ int mysql_prepare_insert(THD *thd, TABLE_LIST *table_list, > } > > > /* Check if there is more uniq keys after field */ this comment isn't correct anymore, is it? on the other hand, the function is small and the name is clear, I think you can simply delete the comment, if you'd like. > +static ushort get_last_unique_key(TABLE *table, KEY *key_info) > +{ > + for (uint key= table->s->keys; key; --key) > + if (key_info[key-1].flags & HA_NOSAME) > + return key-1; > + return MAX_KEY; > +} > > +/** > + An update optimization can be done in REPLACE only when certain criterial met. > + In particular, if the "error key" was the last one to check. If not, there can > + be other keys that can fail the uniqueness check after we will delete the row. > + > + The problem arises with the order of the key checking. It is divided in two > + parts: > + 1. Higher-level handler::ha_write_row goes first. It checks for long unique > + and period overlaps (the latter is not supported by REPLACE yet). better "(when the latter will be supported by REPLACE)" this way the comment will be less wrong and misleading when we'll finally implement this support and but forget to update the comment :) > + 2. Engine usual unique index check. > + However, the order of the keys in TABLE_SHARE::key_info is: > + * first, engine unique keys > + * then long unique, which is not seen by engine > + WITHOUT OVERLAPS key has no particular order > + see sort_keys in sql_table.cc > + > + So we need to wrap this order. > + @return last unique key to check for duplication, or MAX_KEY if none. > + */ > -static int last_uniq_key(TABLE *table, const KEY *key, uint keynr) > +ushort Write_record::get_last_unique_key() const > { > + if (info->handle_duplicates != DUP_REPLACE) > + return MAX_KEY; // Not used. > + if (!table->s->key_info) > + return MAX_KEY; > /* > When an underlying storage engine informs that the unique key > conflicts are not reported in the ascending order by setting > the HA_DUPLICATE_KEY_NOT_IN_ORDER flag, we cannot rely on this > information to determine the last key conflict. > > The information about the last key conflict will be used to > do a replace of the new row on the conflicting row, rather > than doing a delete (of old row) + insert (of new row). > > Hence check for this flag and disable replacing the last row > - by returning 0 always. Returning 0 will result in doing > + by returning MAX_KEY always. Returning MAX_KEY will result in doing > a delete + insert always. > */ > if (table->file->ha_table_flags() & HA_DUPLICATE_KEY_NOT_IN_ORDER) > - return 0; > + return MAX_KEY; > > - while (++keynr < table->s->keys) > - if (key[keynr].flags & HA_NOSAME) > - return 0; > - return 1; > + /* > + Note, that TABLE_SHARE and TABLE see long uniques differently: > + - TABLE_SHARE sees as HA_KEY_ALG_LONG_HASH and HA_NOSAME > + - TABLE sees as usual non-unique indexes > + */ > + return > + table->key_info[0].flags & HA_NOSAME ? > + /* > + We have an in-engine unique, which should be checked last. > + Go through the TABLE list, which knows nothing about long uniques. > + */ > + ::get_last_unique_key(table, table->key_info) : > + /* > + We can only have long uniques. > + Go through the TABLE_SHARE list, where long uniques can be found. > + */ > + ::get_last_unique_key(table, table->s->key_info); > } > > > @@ -1786,449 +1846,511 @@ int vers_insert_history_row(TABLE *table) > } > > /* > - Write a record to table with optional deleting of conflicting records, > - invoke proper triggers if needed. > - > - SYNOPSIS > - write_record() > - thd - thread context > - table - table to which record should be written > - info - COPY_INFO structure describing handling of duplicates > - and which is used for counting number of records inserted > - and deleted. > - sink - result sink for the RETURNING clause > - > - NOTE > - Once this record will be written to table after insert trigger will > - be invoked. If instead of inserting new record we will update old one > - then both on update triggers will work instead. Similarly both on > - delete triggers will be invoked if we will delete conflicting records. > - > - Sets thd->transaction.stmt.modified_non_trans_table to TRUE if table which > - is updated didn't have transactions. > - > - RETURN VALUE > - 0 - success > - non-0 - error > - > -int write_record(THD *thd, TABLE *table, COPY_INFO *info, select_result *sink) > -{ > - int error, trg_error= 0; > - char *key=0; > - MY_BITMAP *save_read_set, *save_write_set; > - ulonglong prev_insert_id= table->file->next_insert_id; > - ulonglong insert_id_for_cur_row= 0; > - ulonglong prev_insert_id_for_cur_row= 0; > - DBUG_ENTER("write_record"); > + Retrieve the old (duplicated) row into record[1] to be able to > + either update or delete the offending record. We either: > + > + - use ha_rnd_pos() with a row-id (available as dup_ref) to the > + offending row, if that is possible (MyISAM and Blackhole, also long unique > + and application-time periods), or else > + > + - use ha_index_read_idx_map() with the key that is duplicated, to > + retrieve the offending row. > */ > +int Write_record::locate_dup_record() > +{ > + handler *h= table->file; > + int error= 0; > + if (h->ha_table_flags() & HA_DUPLICATE_POS || h->lookup_errkey != (uint)-1) > + { > + DBUG_PRINT("info", ("Locating offending record using rnd_pos()")); > + > + error= h->ha_rnd_pos(table->record[1], h->dup_ref); > + if (unlikely(error)) > + { > + DBUG_PRINT("info", ("rnd_pos() returns error %d",error)); > + h->print_error(error, MYF(0)); > + } > + } > + else > + { > + DBUG_PRINT("info", > + ("Locating offending record using ha_index_read_idx_map")); > > + if (h->lookup_handler) > + h= h->lookup_handler; Why? > + > + error= h->extra(HA_EXTRA_FLUSH_CACHE); > + if (unlikely(error)) > + { > + DBUG_PRINT("info",("Error when setting HA_EXTRA_FLUSH_CACHE")); > + return my_errno; > + } > > - info->records++; > - save_read_set= table->read_set; > - save_write_set= table->write_set; > + if (unlikely(key == NULL)) > + { > + key= static_cast<uchar*>(thd->alloc(table->s->max_unique_length)); > + if (key == NULL) > + { > + DBUG_PRINT("info",("Can't allocate key buffer")); > + return ENOMEM; > + } > + } > > - DBUG_EXECUTE_IF("rpl_write_record_small_sleep_gtid_100_200", > + key_copy(key, table->record[0], table->key_info + key_nr, 0); > + > + error= h->ha_index_read_idx_map(table->record[1], key_nr, key, > + HA_WHOLE_KEY, HA_READ_KEY_EXACT); why did you revert commit 5e345281e3599c79 here? > + if (unlikely(error)) > { > - if (thd->rgi_slave && (thd->rgi_slave->current_gtid.seq_no == 100 || > - thd->rgi_slave->current_gtid.seq_no == 200)) > - my_sleep(20000); > - }); > - if (info->handle_duplicates == DUP_REPLACE || > - info->handle_duplicates == DUP_UPDATE) > + DBUG_PRINT("info", ("index_read_idx() returns %d", error)); > + h->print_error(error, MYF(0)); > + } > + } > + > + if (unlikely(error)) > + return error; > + > + if (table->vfield) > + { > + /* > + We have not yet called update_virtual_fields() > + in handler methods for the just read row in record[1]. > + */ > + table->move_fields(table->field, table->record[1], table->record[0]); > + error= table->update_virtual_fields(table->file, VCOL_UPDATE_FOR_REPLACE); > + table->move_fields(table->field, table->record[0], table->record[1]); > + } > + return error; > +} > + > +/** > + REPLACE is defined as either INSERT or DELETE + INSERT. If > + possible, we can replace it with an UPDATE, but that will not > + work on InnoDB if FOREIGN KEY checks are necessary. > + > + Suppose that we got the duplicate key to be a key that is not > + the last unique key for the table and we perform an update: > + then there might be another key for which the unique check will > + fail, so we're better off just deleting the row and trying to insert again. > + > + Additionally we don't use ha_update_row in following cases: > + * when triggers should be invoked, as it may spoil the intermedite NEW_ROW > + representation > + * when we have referenced keys, as there can be different ON DELETE/ON UPDATE > + actions > +*/ > +int Write_record::replace_row(ha_rows *inserted, ha_rows *deleted) > +{ > + int error; > + while (unlikely(error=table->file->ha_write_row(table->record[0]))) > { > - while (unlikely(error=table->file->ha_write_row(table->record[0]))) > + error= prepare_handle_duplicate(error); > + if (unlikely(error)) > + return restore_on_error(); > + > + if (incomplete_records_cb && (error= incomplete_records_cb(arg1, arg2))) This callback is strange. I'd rather inherit Write_record_rpl from Write_record and put rpl-specific logic there. Also you could avoid a switch on INSERT/REPLACE/ODKU if you'd use inheritance instead. But the switch is less strange than the callback :) > + return restore_on_error(); > + > + if (can_optimize && key_nr == last_unique_key) > { > - uint key_nr; > - /* > - If we do more than one iteration of this loop, from the second one the > - row will have an explicit value in the autoinc field, which was set at > - the first call of handler::update_auto_increment(). So we must save > - the autogenerated value to avoid thd->insert_id_for_cur_row to become > - 0. > - */ > - if (table->file->insert_id_for_cur_row > 0) > - insert_id_for_cur_row= table->file->insert_id_for_cur_row; > - else > - table->file->insert_id_for_cur_row= insert_id_for_cur_row; > - bool is_duplicate_key_error; > - if (table->file->is_fatal_error(error, HA_CHECK_ALL)) > - goto err; > - is_duplicate_key_error= > - table->file->is_fatal_error(error, HA_CHECK_ALL & ~HA_CHECK_DUP); > - if (!is_duplicate_key_error) > - { > - /* > - We come here when we had an ignorable error which is not a duplicate > - key error. In this we ignore error if ignore flag is set, otherwise > - report error as usual. We will not do any duplicate key processing. > - */ > - if (info->ignore) > - { > - table->file->print_error(error, MYF(ME_WARNING)); > - goto after_trg_or_ignored_err; /* Ignoring a not fatal error */ > - } > - goto err; > - } > - if (unlikely((int) (key_nr = table->file->get_dup_key(error)) < 0)) > - { > - error= HA_ERR_FOUND_DUPP_KEY; /* Database can't find key */ > - goto err; > - } > - DEBUG_SYNC(thd, "write_row_replace"); > + DBUG_PRINT("info", ("Updating row using ha_update_row()")); > + if (table->versioned(VERS_TRX_ID)) > + table->vers_start_field()->store(0, false); > > - /* Read all columns for the row we are going to replace */ > - table->use_all_columns(); > - /* > - Don't allow REPLACE to replace a row when a auto_increment column > - was used. This ensures that we don't get a problem when the > - whole range of the key has been used. > - */ > - if (info->handle_duplicates == DUP_REPLACE && table->next_number_field && > - key_nr == table->s->next_number_index && insert_id_for_cur_row > 0) > - goto err; > - if (table->file->has_dup_ref()) > + error= table->file->ha_update_row(table->record[1], table->record[0]); > + > + if (likely(!error)) > + ++*deleted; > + else if (error != HA_ERR_RECORD_IS_THE_SAME) > + return on_ha_error(error); > + > + if (versioned) > { > - DBUG_ASSERT(table->file->inited == handler::RND); > - if (table->file->ha_rnd_pos(table->record[1],table->file->dup_ref)) > - goto err; > + store_record(table, record[2]); > + error= vers_insert_history_row(table); > + restore_record(table, record[2]); > + if (unlikely(error)) > + return on_ha_error(error); > } > + > + break; > + } > + else > + { > + DBUG_PRINT("info", ("Deleting offending row and trying to write" > + " new one again")); > + > + auto *trg = table->triggers; > + if (use_triggers && trg->process_triggers(table->in_use, TRG_EVENT_DELETE, > + TRG_ACTION_BEFORE, TRUE)) > + return restore_on_error(); > + if (!versioned) > + error = table->file->ha_delete_row(table->record[1]); > else > { > - if (table->file->extra(HA_EXTRA_FLUSH_CACHE)) /* Not needed with NISAM */ > - { > - error=my_errno; > - goto err; > - } > - > - if (!key) > - { > - if (!(key=(char*) my_safe_alloca(table->s->max_unique_length))) > - { > - error=ENOMEM; > - goto err; > - } > - } > - key_copy((uchar*) key,table->record[0],table->key_info+key_nr,0); > - key_part_map keypart_map= (1 << table->key_info[key_nr].user_defined_key_parts) - 1; > - if ((error= (table->file->ha_index_read_idx_map(table->record[1], > - key_nr, (uchar*) key, > - keypart_map, > - HA_READ_KEY_EXACT)))) > - goto err; > + store_record(table, record[2]); > + restore_record(table, record[1]); > + table->vers_update_end(); > + error = table->file->ha_update_row(table->record[1], table->record[0]); > + restore_record(table, record[2]); > } > - if (table->vfield) > + > + if (unlikely(error)) > + return on_ha_error(error); > + ++*deleted; > + > + if (use_triggers) > { > - /* > - We have not yet called update_virtual_fields(VOL_UPDATE_FOR_READ) > - in handler methods for the just read row in record[1]. > - */ > - table->move_fields(table->field, table->record[1], table->record[0]); > - int verr = table->update_virtual_fields(table->file, VCOL_UPDATE_FOR_REPLACE); > - table->move_fields(table->field, table->record[0], table->record[1]); > - if (verr) > - goto err; > + error= trg->process_triggers(table->in_use, TRG_EVENT_DELETE, > + TRG_ACTION_AFTER, TRUE); > + if (unlikely(error)) > + return restore_on_error(); > } > - if (info->handle_duplicates == DUP_UPDATE) > - { > - int res= 0; > - /* > - We don't check for other UNIQUE keys - the first row > - that matches, is updated. If update causes a conflict again, > - an error is returned > - */ > - DBUG_ASSERT(table->insert_values != NULL); > - store_record(table,insert_values); > - restore_record(table,record[1]); > - table->reset_default_fields(); > + } > + } > > - /* > - in INSERT ... ON DUPLICATE KEY UPDATE the set of modified fields can > - change per row. Thus, we have to do reset_default_fields() per row. > - Twice (before insert and before update). > - */ > - DBUG_ASSERT(info->update_fields->elements == > - info->update_values->elements); > - if (fill_record_n_invoke_before_triggers(thd, table, > - *info->update_fields, > - *info->update_values, > - info->ignore, > - TRG_EVENT_UPDATE)) > - goto before_trg_err; > - > - bool different_records= (!records_are_comparable(table) || > - compare_record(table)); > - /* > - Default fields must be updated before checking view updateability. > - This branch of INSERT is executed only when a UNIQUE key was violated > - with the ON DUPLICATE KEY UPDATE option. In this case the INSERT > - operation is transformed to an UPDATE, and the default fields must > - be updated as if this is an UPDATE. > - */ > - if (different_records && table->default_field) > - table->evaluate_update_default_function(); > - > - /* CHECK OPTION for VIEW ... ON DUPLICATE KEY UPDATE ... */ > - res= info->table_list->view_check_option(table->in_use, info->ignore); > - if (res == VIEW_CHECK_SKIP) > - goto after_trg_or_ignored_err; > - if (res == VIEW_CHECK_ERROR) > - goto before_trg_err; > - > - table->file->restore_auto_increment(prev_insert_id); > - info->touched++; > - if (different_records) > - { > - if (unlikely(error=table->file->ha_update_row(table->record[1], > - table->record[0])) && > - error != HA_ERR_RECORD_IS_THE_SAME) > - { > - if (info->ignore && > - !table->file->is_fatal_error(error, HA_CHECK_ALL)) > - { > - if (!(thd->variables.old_behavior & > - OLD_MODE_NO_DUP_KEY_WARNINGS_WITH_IGNORE)) > - table->file->print_error(error, MYF(ME_WARNING)); > - goto after_trg_or_ignored_err; > - } > - goto err; > - } > + /* > + If more than one iteration of the above loop is done, from > + the second one the row being inserted will have an explicit > + value in the autoinc field, which was set at the first call of > + handler::update_auto_increment(). This value is saved to avoid > + thd->insert_id_for_cur_row becoming 0. Use this saved autoinc > + value. > + */ > + if (table->file->insert_id_for_cur_row == 0) > + table->file->insert_id_for_cur_row = insert_id_for_cur_row; > > - if (error != HA_ERR_RECORD_IS_THE_SAME) > - { > - info->updated++; > - if (table->versioned() && > - table->vers_check_update(*info->update_fields)) > - { > - if (table->versioned(VERS_TIMESTAMP)) > - { > - store_record(table, record[2]); > - if ((error= vers_insert_history_row(table))) > - { > - info->last_errno= error; > - table->file->print_error(error, MYF(0)); > - trg_error= 1; > - restore_record(table, record[2]); > - goto after_trg_or_ignored_err; > - } > - restore_record(table, record[2]); > - } > - info->copied++; > - } > - } > - else > - error= 0; > - /* > - If ON DUP KEY UPDATE updates a row instead of inserting > - one, it's like a regular UPDATE statement: it should not > - affect the value of a next SELECT LAST_INSERT_ID() or > - mysql_insert_id(). Except if LAST_INSERT_ID(#) was in the > - INSERT query, which is handled separately by > - THD::arg_of_last_insert_id_function. > - */ > - prev_insert_id_for_cur_row= table->file->insert_id_for_cur_row; > - insert_id_for_cur_row= table->file->insert_id_for_cur_row= 0; > - trg_error= (table->triggers && > - table->triggers->process_triggers(thd, TRG_EVENT_UPDATE, > - TRG_ACTION_AFTER, TRUE)); > - info->copied++; > - } > + return after_insert(inserted); > +} > > - /* > - Only update next_insert_id if the AUTO_INCREMENT value was explicitly > - updated, so we don't update next_insert_id with the value from the > - row being updated. Otherwise reset next_insert_id to what it was > - before the duplicate key error, since that value is unused. > - */ > - if (table->next_number_field_updated) > - { > - DBUG_ASSERT(table->next_number_field != NULL); > > - table->file->adjust_next_insert_id_after_explicit_value(table->next_number_field->val_int()); > - } > - else if (prev_insert_id_for_cur_row) > - { > - table->file->restore_auto_increment(prev_insert_id_for_cur_row); > - } > - goto ok; > +int Write_record::insert_on_duplicate_update(ha_rows *inserted, > + ha_rows *updated) > +{ > + int res= table->file->ha_write_row(table->record[0]); > + if (likely(!res)) > + return after_insert(inserted); > + > + res=prepare_handle_duplicate(res); > + if (unlikely(res)) > + return restore_on_error(); > + > + ulonglong prev_insert_id_for_cur_row= 0; > + /* > + We don't check for other UNIQUE keys - the first row > + that matches, is updated. If update causes a conflict again, > + an error is returned > + */ > + DBUG_ASSERT(table->insert_values != NULL); > + store_record(table,insert_values); > + restore_record(table,record[1]); > + table->reset_default_fields(); > + > + /* > + in INSERT ... ON DUPLICATE KEY UPDATE the set of modified fields can > + change per row. Thus, we have to do reset_default_fields() per row. > + Twice (before insert and before update). > + */ > + DBUG_ASSERT(info->update_fields->elements == > + info->update_values->elements); > + if (fill_record_n_invoke_before_triggers(thd, table, > + *info->update_fields, > + *info->update_values, > + info->ignore, > + TRG_EVENT_UPDATE)) > + return restore_on_error(); > + > + bool different_records= (!records_are_comparable(table) || > + compare_record(table)); > + /* > + Default fields must be updated before checking view updateability. > + This branch of INSERT is executed only when a UNIQUE key was violated > + with the ON DUPLICATE KEY UPDATE option. In this case the INSERT > + operation is transformed to an UPDATE, and the default fields must > + be updated as if this is an UPDATE. > + */ > + if (different_records && table->default_field) > + table->evaluate_update_default_function(); > + > + /* CHECK OPTION for VIEW ... ON DUPLICATE KEY UPDATE ... */ > + res= info->table_list->view_check_option(table->in_use, info->ignore); > + if (unlikely(res)) > + { > + if (res == VIEW_CHECK_ERROR) > + return restore_on_error(); > + DBUG_ASSERT(res == VIEW_CHECK_SKIP); > + return 0; > + } > + > + table->file->restore_auto_increment(prev_insert_id); > + info->touched++; > + if (different_records) > + { > + int error= table->file->ha_update_row(table->record[1], table->record[0]); > + > + if (unlikely(error) && error != HA_ERR_RECORD_IS_THE_SAME) > + { > + if (info->ignore && !table->file->is_fatal_error(error, HA_CHECK_ALL)) why not is_fatal_error(error) ? > + { > + if (!(thd->variables.old_behavior & > + OLD_MODE_NO_DUP_KEY_WARNINGS_WITH_IGNORE)) > + table->file->print_error(error, MYF(ME_WARNING)); > + return 0; > } > - else /* DUP_REPLACE */ > + return on_ha_error(error); > + } > + > + if (error != HA_ERR_RECORD_IS_THE_SAME) > + { > + ++*updated; > + if (table->versioned() && > + table->vers_check_update(*info->update_fields)) > { > - /* > - The manual defines the REPLACE semantics that it is either > - an INSERT or DELETE(s) + INSERT; FOREIGN KEY checks in > - InnoDB do not function in the defined way if we allow MySQL > - to convert the latter operation internally to an UPDATE. > - We also should not perform this conversion if we have > - timestamp field with ON UPDATE which is different from DEFAULT. > - Another case when conversion should not be performed is when > - we have ON DELETE trigger on table so user may notice that > - we cheat here. Note that it is ok to do such conversion for > - tables which have ON UPDATE but have no ON DELETE triggers, > - we just should not expose this fact to users by invoking > - ON UPDATE triggers. > - > - Note, TABLE_SHARE and TABLE see long uniques differently: > - - TABLE_SHARE sees as HA_KEY_ALG_LONG_HASH and HA_NOSAME > - - TABLE sees as usual non-unique indexes > - */ > - bool is_long_unique= table->s->key_info && > - table->s->key_info[key_nr].algorithm == > - HA_KEY_ALG_LONG_HASH; > - if ((is_long_unique ? > - /* > - We have a long unique. Test that there are no in-engine > - uniques and the current long unique is the last long unique. > - */ > - !(table->key_info[0].flags & HA_NOSAME) && > - last_uniq_key(table, table->s->key_info, key_nr) : > - /* > - We have a normal key - not a long unique. > - Test is the current normal key is unique and > - it is the last normal unique. > - */ > - last_uniq_key(table, table->key_info, key_nr)) && > - !table->file->referenced_by_foreign_key() && > - (!table->triggers || !table->triggers->has_delete_triggers())) > - { > - if (table->versioned(VERS_TRX_ID)) > - { > - bitmap_set_bit(table->write_set, table->vers_start_field()->field_index); > - table->file->column_bitmaps_signal(); > - table->vers_start_field()->store(0, false); > - } > - if (unlikely(error= table->file->ha_update_row(table->record[1], > - table->record[0])) && > - error != HA_ERR_RECORD_IS_THE_SAME) > - goto err; > - if (likely(!error)) > - { > - info->deleted++; > - if (!table->file->has_transactions()) > - thd->transaction->stmt.modified_non_trans_table= TRUE; > - if (table->versioned(VERS_TIMESTAMP)) > - { > - store_record(table, record[2]); > - error= vers_insert_history_row(table); > - restore_record(table, record[2]); > - if (unlikely(error)) > - goto err; > - } > - } > - else > - error= 0; // error was HA_ERR_RECORD_IS_THE_SAME > - /* > - Since we pretend that we have done insert we should call > - its after triggers. > - */ > - goto after_trg_n_copied_inc; > - } > - else > + if (versioned) > { > - if (table->triggers && > - table->triggers->process_triggers(thd, TRG_EVENT_DELETE, > - TRG_ACTION_BEFORE, TRUE)) > - goto before_trg_err; > - > - if (!table->versioned(VERS_TIMESTAMP)) > - error= table->file->ha_delete_row(table->record[1]); > - else > - { > - store_record(table, record[2]); > - restore_record(table, record[1]); > - table->vers_update_end(); > - error= table->file->ha_update_row(table->record[1], > - table->record[0]); > - restore_record(table, record[2]); > - } > + store_record(table, record[2]); > + error= vers_insert_history_row(table); > + restore_record(table, record[2]); > if (unlikely(error)) > - goto err; > - if (!table->versioned(VERS_TIMESTAMP)) > - info->deleted++; > - else > - info->updated++; > - if (!table->file->has_transactions_and_rollback()) > - thd->transaction->stmt.modified_non_trans_table= TRUE; > - if (table->triggers && > - table->triggers->process_triggers(thd, TRG_EVENT_DELETE, > - TRG_ACTION_AFTER, TRUE)) > - { > - trg_error= 1; > - goto after_trg_or_ignored_err; > - } > - /* Let us attempt do write_row() once more */ > + return on_ha_error(error); > } > + ++*inserted; > } > } > - > - /* > - If more than one iteration of the above while loop is done, from > - the second one the row being inserted will have an explicit > - value in the autoinc field, which was set at the first call of > - handler::update_auto_increment(). This value is saved to avoid > - thd->insert_id_for_cur_row becoming 0. Use this saved autoinc > - value. > - */ > - if (table->file->insert_id_for_cur_row == 0) > - table->file->insert_id_for_cur_row= insert_id_for_cur_row; > - > /* > - Restore column maps if they where replaced during an duplicate key > - problem. > + If ON DUP KEY UPDATE updates a row instead of inserting > + one, it's like a regular UPDATE statement: it should not > + affect the value of a next SELECT LAST_INSERT_ID() or > + mysql_insert_id(). Except if LAST_INSERT_ID(#) was in the > + INSERT query, which is handled separately by > + THD::arg_of_last_insert_id_function. > */ > - if (table->read_set != save_read_set || > - table->write_set != save_write_set) > - table->column_bitmaps_set(save_read_set, save_write_set); > + prev_insert_id_for_cur_row= table->file->insert_id_for_cur_row; > + insert_id_for_cur_row= table->file->insert_id_for_cur_row= 0; > + > + ++*inserted; // Conforms the older behavior; > + > + if (use_triggers > + && table->triggers->process_triggers(thd, TRG_EVENT_UPDATE, > + TRG_ACTION_AFTER, TRUE)) > + return restore_on_error(); > } > - else if (unlikely((error=table->file->ha_write_row(table->record[0])))) > + > + /* > + Only update next_insert_id if the AUTO_INCREMENT value was explicitly > + updated, so we don't update next_insert_id with the value from the > + row being updated. Otherwise reset next_insert_id to what it was > + before the duplicate key error, since that value is unused. > + */ > + if (table->next_number_field_updated) > + { > + DBUG_ASSERT(table->next_number_field != NULL); > + > + table->file->adjust_next_insert_id_after_explicit_value( > + table->next_number_field->val_int()); > + } > + else if (prev_insert_id_for_cur_row) > + { > + table->file->restore_auto_increment(prev_insert_id_for_cur_row); > + } > + > + return send_data(); > +} > + > +bool Write_record::is_fatal_error(int error) > +{ > + return error == HA_ERR_LOCK_DEADLOCK || > + error == HA_ERR_LOCK_WAIT_TIMEOUT || These errors were only in rpl code, not anywhere in sql_insert.cc. I don't think it's wrong to check for them everywhere consistently, but does it change the behavior? Can you create a test case for INSERT before your changes that comes to thd->is_fatal_error() with HA_ERR_LOCK_DEADLOCK or HA_ERR_LOCK_WAIT_TIMEOUT? What will be the behavior before and after your change? > + table->file->is_fatal_error(error, HA_CHECK_ALL); > +} > + > +int Write_record::single_insert(ha_rows *inserted) > +{ > + DBUG_EXECUTE_IF("rpl_write_record_small_sleep_gtid_100_200", > + { > + uint64 seq= thd->rgi_slave ? thd->rgi_slave->current_gtid.seq_no : 0; > + if (seq == 100 || seq == 200) > + my_sleep(20000); > + }); > + > + int error= table->file->ha_write_row(table->record[0]); > + if (unlikely(error)) > { > DEBUG_SYNC(thd, "write_row_noreplace"); > - if (!info->ignore || > - table->file->is_fatal_error(error, HA_CHECK_ALL)) > - goto err; > + if (!info->ignore || is_fatal_error(error)) > + return on_ha_error(error); > if (!(thd->variables.old_behavior & > OLD_MODE_NO_DUP_KEY_WARNINGS_WITH_IGNORE)) > table->file->print_error(error, MYF(ME_WARNING)); > table->file->restore_auto_increment(prev_insert_id); > - goto after_trg_or_ignored_err; > + return 0; > } > + return after_insert(inserted); > +} > > -after_trg_n_copied_inc: > - info->copied++; > - thd->record_first_successful_insert_id_in_cur_stmt(table->file->insert_id_for_cur_row); > - trg_error= (table->triggers && > - table->triggers->process_triggers(thd, TRG_EVENT_INSERT, > - TRG_ACTION_AFTER, TRUE)); > +/** > + Write a record to table with optional deleting of conflicting records, > + invoke proper triggers if needed. > + > + @note > + Once this record will be written to table after insert trigger will > + be invoked. If instead of inserting new record we will update old one > + then both on update triggers will work instead. Similarly both on > + delete triggers will be invoked if we will delete conflicting records. > > -ok: > + Sets thd->transaction.stmt.modified_non_trans_table to TRUE if table which > + is updated didn't have transactions. > + > + @return > + 0 - success > + non-0 - error > +*/ > +int Write_record::write_record() > +{ > + ha_rows inserted= 0, updated= 0; > + prev_insert_id= table->file->next_insert_id; > + info->records++; > + ignored_error= false; > + int res= 0; > + switch(info->handle_duplicates) > + { > + case DUP_ERROR: > + res= single_insert(&inserted); > + break; > + case DUP_UPDATE: > + res= insert_on_duplicate_update(&inserted, &updated); > + info->updated+= updated; > + break; > + case DUP_REPLACE: > + res= replace_row(&inserted, &updated); > + if (versioned) > + info->updated+= updated; > + else > + info->deleted+= updated; > + } > + > + info->copied+= inserted; > + if (inserted || updated) > + notify_non_trans_table_modified(); > + return res; > +} > + > + > +int Write_record::prepare_handle_duplicate(int error) > +{ > /* > - We send the row after writing it to the table so that the > - correct values are sent to the client. Otherwise it won't show > - autoinc values (generated inside the handler::ha_write()) and > - values updated in ON DUPLICATE KEY UPDATE. > + If we do more than one iteration of the replace loop, from the second one the > + row will have an explicit value in the autoinc field, which was set at > + the first call of handler::update_auto_increment(). So we must save > + the autogenerated value to avoid thd->insert_id_for_cur_row to become > + 0. > */ > - if (sink && sink->send_data(thd->lex->returning()->item_list) < 0) > - trg_error= 1; > + if (table->file->insert_id_for_cur_row > 0) > + insert_id_for_cur_row= table->file->insert_id_for_cur_row; > + else > + table->file->insert_id_for_cur_row= insert_id_for_cur_row; > + > + if (is_fatal_error(error)) > + return on_ha_error(error); > > -after_trg_or_ignored_err: > - if (key) > - my_safe_afree(key,table->s->max_unique_length); > + bool is_duplicate_key_error= > + table->file->is_fatal_error(error, HA_CHECK_ALL & ~HA_CHECK_DUP); > + if (!is_duplicate_key_error) > + { > + /* > + We come here when we had an ignorable error which is not a duplicate > + key error. In this we ignore error if ignore flag is set, otherwise > + report error as usual. We will not do any duplicate key processing. > + */ > + if (info->ignore) > + { > + table->file->print_error(error, MYF(ME_WARNING)); > + ignored_error= true; > + return 1; /* Ignoring a not fatal error */ > + } > + return on_ha_error(error); > + } > + > + key_nr = table->file->get_dup_key(error); > + > + if (unlikely(key_nr == std::numeric_limits<decltype(key_nr)>::max())) Well, the handler returns (uint)-1, please compare with it, instead of inventing numerous other ways of archieving the same. And why did you decide to use ushort for the key number? We pretty much never use ushort, key number is uint everywhere. I agree that ushort is enough size-wise, but let's rather be consistent. > + return on_ha_error(HA_ERR_FOUND_DUPP_KEY); /* Database can't find key */ > + > + DEBUG_SYNC(thd, "write_row_replace"); > + > + /* Read all columns for the row we are going to replace */ > + table->use_all_columns(); > + /* > + Don't allow REPLACE to replace a row when an auto_increment column > + was used. This ensures that we don't get a problem when the > + whole range of the key has been used. > + */ > + if (info->handle_duplicates == DUP_REPLACE && table->next_number_field && > + key_nr == table->s->next_number_index && insert_id_for_cur_row > 0) > + return on_ha_error(error); > + > + error= locate_dup_record(); > + if (error) > + return on_ha_error(error); > + > + return 0; > +} > + > +inline > +void Write_record::notify_non_trans_table_modified() > +{ > if (!table->file->has_transactions_and_rollback()) > thd->transaction->stmt.modified_non_trans_table= TRUE; ok, as you like. I'm personally not a great fan of one-liners that are used only once. > - DBUG_RETURN(trg_error); > +} > > -err: > +inline > +int Write_record::on_ha_error(int error) > +{ > info->last_errno= error; > table->file->print_error(error,MYF(0)); > - > -before_trg_err: > + DBUG_PRINT("info", ("Returning error %d", error)); > + return restore_on_error(); > +} > + > +inline > +int Write_record::restore_on_error() > +{ > table->file->restore_auto_increment(prev_insert_id); > - if (key) > - my_safe_afree(key, table->s->max_unique_length); > - table->column_bitmaps_set(save_read_set, save_write_set); > - DBUG_RETURN(1); > + return ignored_error ? 0 : 1; > +} > + > +inline > +int Write_record::after_insert(ha_rows *inserted) > +{ > + ++*inserted; > + thd->record_first_successful_insert_id_in_cur_stmt(table->file->insert_id_for_cur_row); > + return after_ins_trg() || send_data(); > +} > + > +inline > +int Write_record::after_ins_trg() > +{ > + int res= 0; > + if (use_triggers) > + res= table->triggers->process_triggers(thd, TRG_EVENT_INSERT, > + TRG_ACTION_AFTER, TRUE); > + return res; > +} > + > +inline > +int Write_record::send_data() > +{ > + /* > + We send the row after writing it to the table so that the > + correct values are sent to the client. Otherwise it won't show > + autoinc values (generated inside the handler::ha_write()) and > + values updated in ON DUPLICATE KEY UPDATE. > + */ > + return sink ? sink->send_data(thd->lex->returning()->item_list) < 0 : 0; > } > > > + > /****************************************************************************** > Check that there aren't any null_fields > ******************************************************************************/ > @@ -4816,18 +4934,8 @@ select_create::prepare(List<Item> &_values, SELECT_LEX_UNIT *u) > > restore_record(table,s->default_values); // Get empty record > thd->cuted_fields=0; > - bool create_lookup_handler= info.handle_duplicates != DUP_ERROR; > - if (info.ignore || info.handle_duplicates != DUP_ERROR) > - { > - create_lookup_handler= true; > - table->file->extra(HA_EXTRA_IGNORE_DUP_KEY); > - if (table->file->ha_table_flags() & HA_DUPLICATE_POS) > - { > - if (table->file->ha_rnd_init_with_error(0)) > - DBUG_RETURN(1); > - } > - } > - table->file->prepare_for_insert(create_lookup_handler); > + if (prepare_for_replace(table, info.handle_duplicates, info.ignore)) ok. strange that select_create::prepare doesn't call select_insert::prepare > + DBUG_RETURN(1); > if (info.handle_duplicates == DUP_REPLACE && > (!table->triggers || !table->triggers->has_delete_triggers())) > table->file->extra(HA_EXTRA_WRITE_CAN_REPLACE); > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 5db53b6f2b8..7ce4850f534 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -4814,7 +4815,8 @@ mysql_execute_command(THD *thd) > &lex->value_list, > lex->duplicates, > lex->ignore, > - result))) > + result, > + &write))) Why do you need Write_record here on the stack? Wouldn't it be more logical to have it in select_insert ? > { > if (lex->analyze_stmt) > ((select_result_interceptor*)sel_result)->disable_my_ok_calls(); Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 652e9ee405e: MDEV-30046 wrong row targeted with "insert ... on duplicate" and "replace"
by Sergei Golubchik 29 Apr '24

29 Apr '24

Hi, Nikita, Good fix. Ok to push, thanks On Apr 29, Nikita Malyavin wrote: > revision-id: 652e9ee405e (mariadb-10.5.24-174-g652e9ee405e) > parent(s): 3d417476256 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-04-27 21:08:21 +0200 > message: > > MDEV-30046 wrong row targeted with "insert ... on duplicate" and "replace" > > When HA_DUPLICATE_POS is not supported, the row to replace was navigated by > ha_index_read_idx_map, which uses only hash to navigate. > > Suchwise, given a hash collision it may choose an incorrect row. > > handler::position would be correct and very convenient to use here. > The code is updated to set dup_ref by a handler independently of engine > capabilities, when extra lookup is made (for long unique or something else, > for example WITHOUT OVERLAPS) > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 1

SOUNDEX sources
by lucyfrost＠tutanota.com 25 Apr '24

25 Apr '24

I'm developing a project which uses soundex matching (alongside other options more sophisticated than that). This is custom software with a mariad back end. MariaDB soundex built-in produces codes of apparently arbitary length; the usual is 4 characters. I've come to the conclusion that the best way forward is to mimic Maria's soundex in my code. I cannot find a description of the algorithm Maria uses though: the only source code I can find is in mf_soundex.c, and that is explicitly limited to 4 characters. But I have installed triggers which generate codes such as A4152636125262 Where is the soundex function found? Anyone know?

3 4

3 PRs pending second approver
by Otto Kekäläinen 25 Apr '24

25 Apr '24

Hi all with permission to review / add approval, This is ready to be merged into 11.5 and already has approval from Andrew, just needs one more approval: https://github.com/MariaDB/server/pull/2671 This has also one approval from Andrew, pending a second to be merged: https://github.com/MariaDB/server/pull/2169 Likewise, approved by Andrew, pending second: https://github.com/MariaDB/server/pull/2516 Additionally, this has been open for since January without any reviews: https://github.com/MariaDB/server/pull/2985 This was updated a month ago based on feedback, but hasn't gotten a re-review since: https://github.com/MariaDB/server/pull/3114 Pending re-review: https://github.com/MariaDB/server/pull/2556 - Otto

3 2

Re: [MariaDB commits] [PATCH] MDEV-33602: Sporadic test failure in rpl.rpl_gtid_stop_start
by Kristian Nielsen 25 Apr '24

25 Apr '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >>> --- a/sql/rpl_rli.cc >>> +++ b/sql/rpl_rli.cc >>> @@ -1030,7 +1030,7 @@ void Relay_log_info::inc_group_relay_log_pos(ulonglong log_pos, >>> rgi->last_master_timestamp > last_master_timestamp) >>> last_master_timestamp= rgi->last_master_timestamp; >>> } >>> - else >>> + else if (!mi->using_gtid || seen_gtid_log_list_event) >>> { >>> /* Non-parallel case. */ >>> group_relay_log_pos= event_relay_log_pos; >> >> Agree, this looks fine. >> The dump thread should always be sending the gtid_list event. And yes, the >> log_pos of the events received until then are not useful to set for the >> slave. >> >> Right, so this seems a good approach for a fix. >> >> Maybe I should push the workaround to 10.5 to help the effort to remove >> sporadic failures, and then file a separate bug about the underlying problem >> and your proposed solution? > > Sounds reasonable. Thank you! Filed as https://jira.mariadb.org/browse/MDEV-33996 - Kristian.

1 0

Re: [MariaDB commits] [PATCH] MDEV-33602: Sporadic test failure in rpl.rpl_gtid_stop_start
by Kristian Nielsen 24 Apr '24

24 Apr '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: >> 1. I have not so far been able to reproduce the failure, so this patch is >> assumed to fix the sporadic failure, but I have not been able to verify that >> it does. > > I verified your analysis to confirm. Thanks for checking! >> But on the other hand this will not solve the fundamental issue, that until the >> GTID mode connect is completed, we will in the general case not have a valid >> corresponding old-style position. So maybe the current behaviour is >> ok? > Sure but it looks fixes are not overly difficult. > > As it's Gtid_list_log_event::log_pos that makes the file:pos (old-style coordinates) > state be valid why won't be keep the coordinates intact until the event > has been arrived/processed? Say that situation is remembered in `RLI::seen_gtid_log_list_event`. > Then for instance in the serial case the fixes would look like > > --- a/sql/rpl_rli.cc > +++ b/sql/rpl_rli.cc > @@ -1030,7 +1030,7 @@ void Relay_log_info::inc_group_relay_log_pos(ulonglong log_pos, > rgi->last_master_timestamp > last_master_timestamp) > last_master_timestamp= rgi->last_master_timestamp; > } > - else > + else if (!mi->using_gtid || seen_gtid_log_list_event) > { > /* Non-parallel case. */ > group_relay_log_pos= event_relay_log_pos; Agree, this looks fine. The dump thread should always be sending the gtid_list event. And yes, the log_pos of the events received until then are not useful to set for the slave. Right, so this seems a good approach for a fix. Maybe I should push the workaround to 10.5 to help the effort to remove sporadic failures, and then file a separate bug about the underlying problem and your proposed solution? Thanks, - Kristian.

1 0

Re: [MariaDB commits] [PATCH] MDEV-33602: Sporadic test failure in rpl.rpl_gtid_stop_start
by Kristian Nielsen 23 Apr '24

23 Apr '24

Hi Andrei, Brandon, What do you think of this patch to fix sporadic failure of rpl.rpl_gtid_stop_start in buildbot, MDEV-33602? The test fails relatively frequent, so would be good to get it fixed. Two issues: 1. I have not so far been able to reproduce the failure, so this patch is assumed to fix the sporadic failure, but I have not been able to verify that it does. 2. The question is if this will be just hiding a real bug? When the GTID-mode slave connects, it receives a few events from the start of the binlog (Gtid_list, Binlog_checkpoint, etc), and these briefly update the old-style position to a wrong value until the fake Gtid_list is received with the correct position. One might argue that this should be avoided. But on the other hand this will not solve the fundamental issue, that until the GTID mode connect is completed, we will in the general case not have a valid corresponding old-style position. So maybe the current behaviour is ok? - Kristian. Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > The test could fail with a duplicate key error because switching to non-GTID > mode could start at the wrong old-style position. The position could be > wrong when the previous GTID connect was stopped before receiving the fake > GTID list event which gives the old-style position corresponding to the GTID > connected position. > > Work-around by injecting an extra event and syncing the slave before > switching to non-GTID mode. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > --- > .../suite/rpl/r/rpl_gtid_stop_start.result | 4 ++++ > mysql-test/suite/rpl/t/rpl_gtid_stop_start.test | 16 ++++++++++++++++ > 2 files changed, 20 insertions(+) > > diff --git a/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result b/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result > index ae0050c353a..e8633cd45bb 100644 > --- a/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result > +++ b/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result > @@ -111,6 +111,10 @@ a > 6 > 7 > *** MDEV-4486: Allow to start old-style replication even if mysql.gtid_slave_pos is unavailable > +connection server_1; > +INSERT INTO t1 VALUES (8); > +DELETE FROM t1 WHERE a=8; > +connection server_2; > connection server_2; > include/stop_slave.inc > CHANGE MASTER TO master_use_gtid= no; > diff --git a/mysql-test/suite/rpl/t/rpl_gtid_stop_start.test b/mysql-test/suite/rpl/t/rpl_gtid_stop_start.test > index b5ff294908b..032ebb77d1e 100644 > --- a/mysql-test/suite/rpl/t/rpl_gtid_stop_start.test > +++ b/mysql-test/suite/rpl/t/rpl_gtid_stop_start.test > @@ -173,6 +173,22 @@ SELECT * FROM t1 ORDER BY a; > > --echo *** MDEV-4486: Allow to start old-style replication even if mysql.gtid_slave_pos is unavailable > > +# In GTID mode, the old-style replication position is also updated. But during > +# GTID connect, the old-style position is not known until receiving the fake > +# GTID list event, which contains the required position value. If we happened > +# to stop the slave above before this fake GTID list event, the test could fail > +# with duplicate key errors due to switching to non-GTID mode at a wrong > +# position too far back in the binlog. > +# > +# Work-around this by injecting an extra dummt event and syncing the slave to > +# it, ensuring the old-style position will be updated. > +--connection server_1 > +INSERT INTO t1 VALUES (8); > +DELETE FROM t1 WHERE a=8; > +--save_master_pos > +--connection server_2 > +--sync_with_master > + > --connection server_2 > --source include/stop_slave.inc > CHANGE MASTER TO master_use_gtid= no;

1 0

MDEV-27512, --slave-skip-errors, and optimistic parallel replication
by Kristian Nielsen 18 Apr '24

18 Apr '24

Hi Brandon, I only now got to check on MDEV-27512. This bug seems to be about using --slave_skip_errors=ALL with optimistic parallel replication. I saw your patch for the bug. While the patch by itself is perhaps ok, I think it's only fixing a minor thing, and not addressing the real problem, let me explain my thinking: Now, --slave-skip-errors is obviously a dangerous option, but presumably the use case is to make the slave continue at all costs in case an unexpected error occurs, rather than stop the slave and break replication. But optimistic parallel replication by design can introduce many different transient errors due to conflicts, that are then handled by rolling back and retrying the offending transactions. These errors are *expected*, and they do *not* cause slave to stop or replication to break. However, it appears that --slave_skip_errors also affects such transient errors due to optimistic parallel replication, and will cause any such transaction to be silently ignored! This must be very wrong, it will cause massive replication divergence. It seems to me this is the real bug here. When an error is encountered during optimistic apply of a transaction (is_parallel_retry_error() returns true, eg. rgi->speculation == SPECULATE_OPTIMISTIC, then this error should _not_ be subject to --slave-skip-errors. The transaction should be rolled back as normal, and wait_for_prior_commit() done. Then after setting rgi->speculation = SPECULATE_WAIT and retrying, if we still get the error, --slave-skip-errors can apply. I put together a quick test that seems to show this behaviour, included below. This tests replicates correctly without replication stopping with error. But running it with --mysqld=--slave-skip-errors=all, it replicates incorrectly, skipping lots of transactions. The test is somewhat contrieved, but I think it shows the real problem, that --slave-skip-errors can randomly cause transactions to be skipped or not depending on if optimistic parallel replication triggers a matching transient error or not. So in summary, it looks like there is a real problem here, that optimistic parallel replication is not working correctly with --slave-skip-errors, transient errors incorrectly causes conflicts to skip transactions rather than retrying them. This will cause replication to diverge even when no real errors occur. And the patch for MDEV-27512 doesn't seem to address this issue. So I think MDEV-27512 should be re-opened, what do you think? - Kristian. ----------------------------------------------------------------------- --source include/have_innodb.inc --source include/have_binlog_format_row.inc --source include/master-slave.inc --connection master ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; INSERT INTO t1 VALUES (1,NULL), (2,2), (3,NULL), (4,4), (5, NULL), (6, 6); --sync_slave_with_master --source include/stop_slave.inc CHANGE MASTER TO master_use_gtid=slave_pos; SET @old_timeout= @@GLOBAL.innodb_lock_wait_timeout; SET GLOBAL innodb_lock_wait_timeout= 5; SET @old_parallel= @@GLOBAL.slave_parallel_threads; SET @old_mode= @@GLOBAL.slave_parallel_mode; SET GLOBAL slave_parallel_mode= aggressive; SET GLOBAL slave_parallel_threads= 20; --connection master UPDATE t1 SET b=b+1 WHERE a=6; --disable_query_log let $i= 0; while ($i < 40) { eval UPDATE t1 SET b=b+1 WHERE a=2; inc $i; } --enable_query_log SELECT * FROM t1 ORDER BY a; --save_master_pos --connection slave1 # Block first worker, and recursively pause all following workers that get # temporary errors before they can retry. BEGIN; SELECT * FROM t1 WHERE a=6 FOR UPDATE; --connection slave # Cause initial row not found error. SET STATEMENT sql_log_bin=0 FOR UPDATE t1 SET a=7 WHERE a=2; --source include/start_slave.inc --sleep 2 # Now following workers should be waiting for prior commit before retrying. # Remove the row not found error. SET STATEMENT sql_log_bin=0 FOR UPDATE t1 SET a=2 WHERE a=7; --connection slave1 ROLLBACK; --connection slave --sync_with_master SELECT * FROM t1 ORDER BY a; # Cleanup --connection slave --source include/stop_slave.inc SET GLOBAL innodb_lock_wait_timeout= @old_timeout; SET GLOBAL slave_parallel_threads= @old_parallel; SET GLOBAL slave_parallel_mode= @old_mode; --source include/start_slave.inc --connection default DROP TABLE t1; --source include/rpl_end.inc

2 2

Feedback plugin status?
by Otto Kekäläinen 16 Apr '24

16 Apr '24

Hi! I was about to merge a long overdue MR in Debian to enable the Feedback plugin (https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/16) However I noticed all the links at https://mariadb.org/about/#usage-statistics are broken. The wiki page at https://mariadb.com/kb/en/feedback-plugin/ says that the plugin is 'currently not working'. What does this mean? What is the path to get it working again? Is there a need for somebody to rewrite the stats server part? Thanks, - Otto

3 4

MariaDB/MySQL wire protocol ownership and MariaDB Connector C source code hosting?
by Otto Kekäläinen 13 Apr '24

13 Apr '24

Hi! The MariaDB wire protocol is an interplay between the MariaDB Server and the MariaDB Connector C. However, only the server is in the MariaDB Foundation's GitHub account at https://github.com/mariadb/server, and the Connector C is in the MariaDB PLC account at https://github.com/mariadb-corporation/mariadb-connector-c. Is there perhaps some plan to hand over the Connector C to the MariaDB Foundation? The reason I am asking is that the connector does not seem equally "true open source" as the server is, making changes that require coordination in both server and connector (e.g. https://github.com/mariadb-corporation/mariadb-connector-c/pull/227) kind of out-of-scope for the MariaDB Foundation, which seems counterproductive. If/when wire protocol changes might be made and changes to both server and libmariadb coordinated in an open "working group" like suggested in https://github.com/MariaDB/server/pull/2684, it might be very challenging for the Foundation or the open source community to drive it if the PLC staff member maintaining the connector does not participate. Also, seems the PR reviews at https://github.com/mariadb-corporation/mariadb-connector-c/pulls have been stalled since 2022. MariaDB Foundation staff are funded to facilitate reviews, but can't really be *responsible* for reviewing them as long as it is not a Foundation repository and review policies are not owned by the Foundation. It would just make a lot of sense if such a central piece of the MariaDB Server would be in the Foundation's GitHub account. The MariaDB Connector C / libmaraidb is already published and distributed by the MariadB Foundation, so moving the source code hosting over shouldn't big such a big thing, right? - Otto

3 2

7 PRs pending second approver - can somebody do it?
by Otto Kekäläinen 11 Apr '24

11 Apr '24

Hi Sergei, Andrew or anyone else who wants to help, These PRs have already been approved by one reviewer 2-8 weeks ago and are pending the second reviewer to approve/merge. These indicate that Sergei's review is requested, but I guess anybody could be the second reviewer: https://github.com/MariaDB/server/pull/2169 https://github.com/MariaDB/server/pull/2671 The indicate that Andrewi's review is requested, but I guess anybody could be the second reviewer: https://github.com/MariaDB/server/pull/3100 https://github.com/MariaDB/server/pull/3120 https://github.com/MariaDB/server/pull/3125 https://github.com/MariaDB/server/pull/3142 https://github.com/MariaDB/server/pull/3154 I am sending this to the mailing list as well just to raise general awareness that while Andrew and Daniel are putting in a lot of effort to review "external" contributions, it is not enough and even approved PRs get stalled. Other core contributors should perhaps get into the habit of reviewing/approving PRs as well. If there was a rule in github.com/MariaDB/server that prevents any commits from being merged unless it has two approvals, it would both help improve the code quality AND also force core contributors to spend more time adding approvals on GitHub and thus automatically increase the likelihood that there are no one-approval PRs around for weeks and weeks without getting the second approval. - Otto

3 2

MariaDB supports DBdeployer and MySqlJS, but they don't support MariaDB?
by Otto Kekäläinen 09 Apr '24

09 Apr '24

Hi! While digging into the CI situation I noticed that https://buildbot.mariadb.org/ advertises: "We run other ecosystem tests for: * DBdeployer * MySqlJS" However, when I browse https://github.com/mysqljs/mysql and https://github.com/datacharmer/dbdeployer there are zero mentions of MariaDB. Just wondering is anybody working to get them to mention MariaDB in their docs? If they are tested as part of the MariaDB CI, surely they should reciprocate and at least tell in their README that MariaDB is also supported. Maybe somebody from the MariaDB Foundation could send a PR to update their README, so they don't need to do anything else than just accept it. This was done for example in https://github.com/mysql-net/MySqlConnector/pull/1460/files by Daniel Lenski, and now that project mentions both MySQL and MariaDB. - Otto

3 2

Safe use of sprint/printf/fprintf in MariaDB code base
by Otto Kekäläinen 08 Apr '24

08 Apr '24

Hi all, Reading about the xz-utils backdoor authors submission of converting safe_fprintf() to fprintf() in libarchive[1] presumably in order to introduce intentional vulnerability reminded me that the MariaDB code base still has a plenty unsafe sprint/printf/fprintf use that can easily be found with scanners such as Flawfinder[2] and cppcheck[3]. There are currently 6 merge requests open by two authors (CC'd) to fix some of these issues[4]. Could we please have some more attention on these by the core contributors? If core contributors are not happy with the submissions, could you perhaps write your own safe functions (there are already some in m_string.h[5]) like many other projects seem to have (also libarchive had[6]) and then ask all contributors to use them consistently? Use of specific memory safe functions could also be mandated via the coding standards[7]. [1] https://github.com/libarchive/libarchive/pull/1609 [2] https://github.com/MariaDB/server/blob/11.5/.gitlab-ci.yml#L461-L489 [3] https://github.com/MariaDB/server/blob/11.5/.gitlab-ci.yml#L522-L554 [4] https://github.com/MariaDB/server/pulls?q=is%3Apr+is%3Aopen+sprintf+ [5] https://github.com/MariaDB/server/blob/11.5/include/m_string.h [6] https://github.com/libarchive/libarchive/blob/6110e9c82d8ba830c3440f36b9904… [7] https://github.com/MariaDB/server/blob/11.5/CODING_STANDARDS.md

2 3

question about Revision #f24038b851
by ShawnYan 04 Apr '24

04 Apr '24

hi, there, when I check release notes, https://mariadb.com/kb/en/mariadb-10419-changelog/ <https://mariadb.com/kb/en/mariadb-10419-changelog/ > I notice this commit record, Revision #f24038b851 2021-03-07 14:06:01 +0100 mark Aria allocations for temp tables as MY_THREAD_SPECIFIC but, it can not find in the jira system, can't learn more detail/background about this commit, could anyone help with it, may we know which task refer to this commit? https://github.com/MariaDB/server/commit/f24038b851 <https://github.com/MariaDB/server/commit/f24038b851 > Thank you. hi, there, when I check release notes, https://mariadb.com/kb/en/mariadb-10419-changelog/ <https://mariadb.com/kb/en/mariadb-10419-changelog/ > I notice this commit record, Revision #f24038b851 2021-03-07 14:06:01 +0100 mark Aria allocations for temp tables as MY_THREAD_SPECIFIC but, it can not find in the jira system, can't learn more detail about this commit, could anyone help with it, may we know which task refer to this commit? https://github.com/MariaDB/server/commit/f24038b851 <https://github.com/MariaDB/server/commit/f24038b851 > Thank you.

2 1

Re: [PATCH 0/2] Suggestion for smaller fix to XA parallel replication performance, MDEV-31949
by Kristian Nielsen 19 Mar '24

19 Mar '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > In normal cases, like that of not so many duplicate xids, the size > of maybe_active_xid must remain constrained because of "continuous" > relocation of elements from the tail to replace garbage (normally found > in head indexes). > Nevertheless as the array can be dynamically re-allocated to reflect some > some spike in the load, I suggest in > > origin/review__knielsen_xa_sched_minimal_fix > > to take care of that. I think your suggestion in review__knielsen_xa_sched_minimal_fix is fine. I'm not sure exactly how large the array can grow in pathological cases. As you wrote, in most normal cases we would expect it to not grow that much. But shrinking it if necessary surely cannot hurt. Thanks, - Kristian.

1 0

MariaDB topic on GitHub
by Otto Kekäläinen 19 Mar '24

19 Mar '24

Hi! I was browsing the GitHub topic 'mariadb' at https://github.com/topics/mariadb and to my surprise MariaDB itself isn't in that list at all. This seems to be due to https://github.com/MariaDB/server not having any topics defined at all. I recommend you add in the GitHub settings of https://github.com/MariaDB/server at least the topic 'mariadb', and while at it, also add link to mariadb.org. More ideas on how to improve the look and features of the GitHub page can be found by browsing projects like https://github.com/pulsar-edit and https://github.com/prisma which have all their GitHub account metadata polished. - Otto

1 0

Re: [PATCH] MDEV-33551: Semi-sync Wait Point AFTER_COMMIT Slow on Workloads with Heavy Concurrency
by Kristian Nielsen 18 Mar '24

18 Mar '24

> From: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > > When using semi-sync replication with > rpl_semi_sync_master_wait_point=AFTER_COMMIT, the performance of the > primary can significantly reduce compared to AFTER_SYNC's > performance for workloads with many concurrent users executing > transactions. This is because all connections on the primary share Hi Brandon, Here is my second round review of the MDEV-33551 patch. I think these comments of mine from first round still need to be addressed: > > A bit of care must be taken about lifetime of Tranx_node and THD > > respectively (in case one goes away but not the other). But the whole > > > +void Repl_semi_sync_master::await_all_slave_replies() > > > + wait_result= cond_timewait(front.thd, &abstime); > > > > I'm wondering, is it safe here to wait on front.thd->COND_wakeup_ready? > > Isn't there a possibility that this might go away during the wait? I tried a simple testcase (attached) which disconnects the session thread while await_all_slave_replies() is running during shutdown. As I expected, I could see the COND_wakeup_ready of the session thread being de-allocated while await_all_slave_replies() is still waking up from that condition. That cannot be correct. Similarly, it seems clearly possible that a session thread is killed and exits while its entry is still present in the m_active_tranxs list, which will similarly lead to signalling a de-allocated COND_wakeup_ready. When a thread aborts the wait before the entry is removed, I think it should simply look up its entry (can use the m_trx_htb hash table, or even just iterate the list) and set its THD to NULL (or remove the entry, whatever is simpler). For the await_all_slave_replies() problem, I think the whole wait-at-shutdown is much too complex as it is now. The whole is_thd_awaiting_semisync_ack(), n_threads_awaiting_ack, special-case not killing threads waiting for semi-sync-ack, and now introducing unmark_thd_awaiting_ack. And it still has the bug waiting on a COND_wakeup_ready of another THD that can disappear during the wait. I think all of that should just go away, the wait in await_all_slave_replies() should be done much simpler with a dedicated condition variable to wait for. Eg. await_all_slave_replies() just does a mysql_cond_wait(COND_binlog_send) under LOCK_binlog until the last binlog position is ack'ed, and this condition is then signalled from report_reply_binlog() (either always, or await_all_slave_replies() can set a flag to request signal). Or have the shutdown thread enqueue its THD at the end of the m_active_tranxs list. Let's avoid all this complex interaction between SHUTDOWN WAIT FOR ALL SLAVES and session threads. BTW, COND_binlog_send is currently unused in your patch. Below detailed comments on individual parts of the patch: > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -1750,18 +1750,12 @@ static void close_connections(void) > - if (shutdown_wait_for_slaves && repl_semisync_master.get_master_enabled()) > + if (shutdown_wait_for_slaves && repl_semisync_master.get_master_enabled() && > + repl_semisync_master.sync_get_master_wait_sessions()) Do you need to check sync_get_master_wait_sessions() > 0 here? Why not wait unconditionally for the last binlog position sent? > --- a/sql/semisync_master.cc > +++ b/sql/semisync_master.cc > @@ -68,6 +68,14 @@ static ulonglong timespec_to_usec(const struct timespec *ts) > return (ulonglong) ts->tv_sec * TIME_MILLION + ts->tv_nsec / TIME_THOUSAND; > } > > +int signal_waiting_transaction(THD *waiting_thd, const char *binlog_file, > + my_off_t binlog_pos) > +{ > + if (waiting_thd->is_awaiting_semisync_ack) > + mysql_cond_signal(&waiting_thd->COND_wakeup_ready); The check on is_awaiting_semi_sync_ack seems wrong here. Normally, the THD will be waiting, otherwise it would not be in the list in the first place, so faster with an occasional redundant signal than a conditional always. And if the THD is not waiting, it's not safe to access it anyway. Instead, as mentioned above, have a check for thd != NULL here, and let a non-waiting thread clear the THD in its entry before it releases the LOCK_binlog. > -int Repl_semi_sync_master::cond_timewait(struct timespec *wait_time) > +int Repl_semi_sync_master::cond_timewait(THD *thd, struct timespec *_wait_time) > + if (_wait_time == NULL) > + { > + create_timeout(&gen_wait_time, NULL); > + real_wait_time= &gen_wait_time; > + } > + else > + { > + real_wait_time= _wait_time; > + } Don't pass in booleans that are always hard-coded true or false in the caller. Instead create two different functions, one that takes a time_spec, and one that does not (the latter creating it and calling the former). > @@ -533,7 +559,8 @@ void Repl_semi_sync_master::remove_slave() > Signal transactions waiting in commit_trx() that they do not have to > wait anymore. > */ > - cond_broadcast(); > + m_active_tranxs->clear_active_tranx_nodes(NULL, NULL, > + signal_waiting_transaction); This does not compile on my machine (the second NULL is passed to my_off_t position), is that a last-minute change that was never tested? Or NULL is defined differently on your machine, so it doesn't fail to compile? BTW, question about the old code: Why didn't the old code clear the m_active_tranxs list? Was it a bug? Or was it cleared elsewhere before (and if so where)? > -void Repl_semi_sync_master::await_slave_reply() > +void Repl_semi_sync_master::await_all_slave_replies() > + The lock is taken per iteration rather than over the whole loop to allow > + more opportunity for the user threads to handle their ACKs. That sounds like a mis-understanding. The lock is always released by mysql_cond_timedwait() in each iteration of the loop, no need to release/take it again for just a couple instructions. > + while (TRUE) > + { > + lock(); > + if (m_active_tranxs->is_empty() || !get_master_enabled() || !is_on()) > + { > + unlock(); > + break; > + } > + wait_result= m_active_tranxs->for_front( > + [](THD *thd, const char *binlog_file, my_off_t binlog_pos) -> int { > + return (thd->is_awaiting_semisync_ack) > + ? repl_semisync_master.cond_timewait(thd, NULL) > + : 0; > + }); As I wrote at the top, it is not safe to wait on the COND_wakeup_ready of another THD, since that THD might go away during the wait. Instead, let's do a specific wait for the end position here, either on COND_binlog or by enqueing an extra THD at the end. (And btw, in your code, why wait for the _front_ of the list and not the _rear_?) > diff --git a/sql/semisync_master.h b/sql/semisync_master.h > index 99f46869354..1eb690b572b 100644 > --- a/sql/semisync_master.h > +++ b/sql/semisync_master.h > > + * The pre_delete_hook parameter is a function pointer that will be invoked > + * for each Active_tranx node, in order, from m_trx_front to log_file_name, > + * e.g. to signal their wakeup condition. Repl_semi_sync_binlog::LOCK_binlog > + * is held while this is invoked. "from m_trx_front to log_file_name", I don't understand? (Typo s/log_file_name/m_trx_rear/ ?) > +/* > + Element in Repl_semi_sync_master::wait_queue to preserve the state of a > + transaction waiting for an ACK. > +*/ > +typedef struct _semisync_wait_trx { > + const char *binlog_name; > + my_off_t binlog_pos; > + THD *thd; > +} semisync_wait_trx_t; > + This is not longer used (should be removed). > * Input: (the transaction events' ending binlog position) > * trx_wait_binlog_name - (IN) ending position's file name > * trx_wait_binlog_pos - (IN) ending position's file offset > + * unmark_thd_awaiting_ack - (IN) Whether or not to unmark > + * thd->is_awaiting_semisync_ack after receiving > + * an ACK from the replica. If using wait_point > + * AFTER_SYNC with binlog_group_commit, only the > + * last transaction written to binlog in the > + * group should negate the boolean, because the > + * same thread (i.e. leader thread) will wait for > + * all transaction ACKs. Negating it too early > + * would break SHUTDOWN WAIT FOR ALL SLAVES, as > + * that is the condition tested to bypass killing > + * threads in phase 1. In all other situations, > + * the boolean should be negated immediately. That seems unnecessarily convoluted. I think the real problem here is that write_tranx_in_binlog() sets the is_awaiting_semi_sync_ack flag on the wrong THD, on the waiter_thd instead of the trx_thd. But once we fix the await_all_slave_replies(), do we even need the is_awaiting_semi_sync_ack flag any more? If so, why? If not, then this point becomes moot anyway. - Kristian. --source include/have_innodb.inc --source include/have_debug.inc --source include/have_binlog_format_mixed.inc --source include/master-slave.inc --connection master CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; INSERT INTO t1 VALUES (1,0); --sync_slave_with_master source include/stop_slave.inc; SET @old_semi_enabled= @@GLOBAL.rpl_semi_sync_slave_enabled; SET GLOBAL rpl_semi_sync_slave_enabled = 1; --source include/start_slave.inc --connection master SET GLOBAL rpl_semi_sync_master_enabled= 1; SET GLOBAL rpl_semi_sync_master_timeout= 5000; let $status_var= Rpl_semi_sync_master_clients; let $status_var_value= 1; source include/wait_for_status_var.inc; show status like 'Rpl_semi_sync_master_status'; show status like 'Rpl_semi_sync_master_clients'; --connection slave SET @old_dbug= @@GLOBAL.debug_dbug; SET GLOBAL debug_dbug="+d,simulate_delay_semisync_slave_reply"; --connect(con1, localhost, root,,) --connect(con2, localhost, root,,) --connection con1 send INSERT INTO t1 VALUES (2,0); --sleep 0.3 --disconnect con1 --connection default --write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect wait EOF #let $status_var= Rpl_semi_sync_master_wait_sessions; #let $status_var_value= 1; #source include/wait_for_status_var.inc; --sleep 0.1 --connection con2 SHUTDOWN WAIT FOR ALL SLAVES; --disconnect con2 --connection default --source include/wait_until_disconnected.inc --connection master --source include/wait_until_disconnected.inc --connection server_1 --source include/wait_until_disconnected.inc --connection default --append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect restart EOF --enable_reconnect --source include/wait_until_connected_again.inc --connection master --enable_reconnect --source include/wait_until_connected_again.inc --connection server_1 --enable_reconnect --source include/wait_until_connected_again.inc --save_master_pos --connection slave SET GLOBAL debug_dbug= @old_dbug; SET GLOBAL rpl_semi_sync_slave_enabled= @old_semi_enabled; --source include/stop_slave.inc --source include/start_slave.inc --sync_with_master --connection default DROP TABLE t1; --source include/rpl_end.inc

2 3

Re: e6cacca0b39: MDEV-23729 MDEV-32218 INFORMATION_SCHEMA table for user data
by Sergei Golubchik 15 Mar '24

15 Mar '24

Hi, Nikita, no test for FLUSH PRIVILEGES no test for a user with manually expired password doesn't compile in embedded hostname 'neo' is replaced, but not everywhere I'll try to fix it myself, but no guarantees On Mar 15, Nikita Malyavin wrote: > revision-id: e6cacca0b39 (mariadb-11.4.1-10-ge6cacca0b39) > parent(s): 929c2e06aae > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-03-15 15:32:45 +0100 > message: > > MDEV-23729 MDEV-32218 INFORMATION_SCHEMA table for user data > > * A new table INFORMATION_SCHEMA.USERS is introduced. > * It stores auxiliary user data > * An unprivileged user can access their own data, and that is the main > difference with what mysql.global_priv provides > * The fields are currently: USER, PASSWORD_ERRORS, PASSWORD_EXPIRATION_TIME > * If password_errors is ignored for the user, PASSWORD_ERRORS is NULL > * PASSWORD_EXPIRATION_TIME is a timestamp with exact point in time, calculated > from password_last_changed and password_lifetime (i.e. days) stored for the user > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 14450a5a610..a87b04f3d39 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc ... > +int fill_users_schema_table(THD *thd, TABLE_LIST *tables, COND *cond) > +{ > + int res= 0; > +#ifndef NO_EMBEDDED_ACCESS_CHECKS > + bool see_whole_table= check_access(thd, SELECT_ACL, "mysql", NULL, NULL, > + true, true) == 0; > + TABLE *table= tables->table; > + > + if (!see_whole_table) > + { > + mysql_mutex_lock(&acl_cache->lock); > + ACL_USER *cur_user= find_user_exact(thd->security_ctx->priv_host, > + thd->security_ctx->priv_user); > + if (!cur_user) > + { > + mysql_mutex_unlock(&acl_cache->lock); > + my_error(ER_INVALID_CURRENT_USER, MYF(0)); Hmm. Questionable. I'd just return an empty result set here, I_S tables normally don't throw errors. But this is an exceptional situation and I cannot say that an error is wrong here, let's keep it your way. > + return 1; > + } > + > + res= fill_users_schema_record(thd, table, cur_user); > + mysql_mutex_unlock(&acl_cache->lock); > + return res; > + } > + > + mysql_mutex_lock(&acl_cache->lock); > + for (size_t i= 0; res == 0 && i < acl_users.elements; i++) > + { > + ACL_USER *user= dynamic_element(&acl_users, i, ACL_USER*); > + res= fill_users_schema_record(thd, table, user); > + } > + mysql_mutex_unlock(&acl_cache->lock); > +#endif > + return res; > +} > + Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: [PATCH 0/2] Suggestion for smaller fix to XA parallel replication performance, MDEV-31949
by Kristian Nielsen 13 Mar '24

13 Mar '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > which makes a worker with dependent pieces of job be *effectively* placed at the > tail of the available worker ('fifo') queue. Indeed. And this BTW is why a simple sliding window of size 2*workers is not enough, because in principle a worker can sit in the fifo for many transactions without being scheduled, if there are many dependency restrictions. I coded the thing with "generations" as a quick way to handle this and have a way to clean out old dependency information. This could of course be improved in many ways. Maybe remember the two last XID for every worker and have a hash table mapping each XID to its worker, or something. But this is just a PoC. > No you are not. Thanks for the catch! To excuse myself it was supposed > to be a poc that I committed hastedly. Yes yes, no problem, I understand, it's good to do poc. I was just momentarily confused, but the intent is clear, thanks. - Kristian.

1 0

Re: a473db9e4fe: MDEV-25829 Change default collation to utf8mb4_1400_ai_ci
by Sergei Golubchik 13 Mar '24

13 Mar '24

Hi, Alexander, On Mar 08, Alexander Barkov wrote: > commit a473db9e4fe > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Thu Nov 2 14:16:09 2023 +0400 > > MDEV-25829 Change default collation to utf8mb4_1400_ai_ci "utf8mb4_uca1400_ai_ci". It's still not too late to rename the MDEV and change commits to match. I'm not reviewing InnoDB changes, I hope Marko did > > Step#3 The main patch > > diff --git a/mysql-test/include/ctype_utf8mb4.inc b/mysql-test/include/ctype_utf8mb4.inc > index 436b0f2782f..73011cef81d 100644 > --- a/mysql-test/include/ctype_utf8mb4.inc > +++ b/mysql-test/include/ctype_utf8mb4.inc > @@ -58,11 +58,11 @@ select CONVERT(_koi8r' > # "a\0" < "a" > # "a\0" < "a " > > -SELECT 'a' = 'a '; > -SELECT 'a\0' < 'a'; > -SELECT 'a\0' < 'a '; > -SELECT 'a\t' < 'a'; > -SELECT 'a\t' < 'a '; > +SELECT 'a' = 'a ' collate utf8mb4_general_ci; > +SELECT 'a\0' < 'a' collate utf8mb4_general_ci; > +SELECT 'a\0' < 'a ' collate utf8mb4_general_ci; > +SELECT 'a\t' < 'a' collate utf8mb4_general_ci; > +SELECT 'a\t' < 'a ' collate utf8mb4_general_ci; 1. what will happen in utf8mb4_uca1400_ai_ci? 2. if you specifically want this file to test utf8mb4_general_ci, you can do SET NAMES, cannot you? Once for the whole file. Or set character_set_collations > > # > # The same for binary collation > diff --git a/mysql-test/main/ctype_euckr.result b/mysql-test/main/ctype_euckr.result > index 9e030f9cc6d..8de90f89a26 100644 > --- a/mysql-test/main/ctype_euckr.result > +++ b/mysql-test/main/ctype_euckr.result > @@ -2244,7 +2244,7 @@ FE80 > DELETE FROM t2 WHERE a='?'; > ALTER TABLE t2 ADD u VARCHAR(1) CHARACTER SET utf8, ADD a2 VARCHAR(1) CHARACTER SET euckr; > UPDATE IGNORE t2 SET u=a, a2=u; > -SELECT s as unassigned_code FROM t2 WHERE u='?'; > +SELECT s as unassigned_code FROM t2 WHERE u=binary'?'; why? > unassigned_code > A2E8 > A2E9 > diff --git a/mysql-test/main/ctype_like_range.result b/mysql-test/main/ctype_like_range.result > index e5e5e61126b..fa694b0c250 100644 > --- a/mysql-test/main/ctype_like_range.result > +++ b/mysql-test/main/ctype_like_range.result > @@ -284,7 +284,7 @@ id name val > 32 mn 63616161616161616161616161616161 > 32 mx 63616161616161616161616161616161 > 32 sp -------------------------------- > -ALTER TABLE t1 MODIFY a VARCHAR(32) CHARACTER SET utf8; > +ALTER TABLE t1 MODIFY a VARCHAR(32) CHARACTER SET utf8 COLLATE utf8_general_ci; do you have tests for ctype_like_range with utf8mb3_uca1400_ai_ci ? > INSERT INTO t1 (a) VALUES (_ucs2 0x0425),(_ucs2 0x045F); > INSERT INTO t1 (a) VALUES (_ucs2 0x2525),(_ucs2 0x5F5F); > SELECT * FROM v1; > diff --git a/mysql-test/main/ctype_utf8.test b/mysql-test/main/ctype_utf8.test > index a875fe51f3a..2f56d10bccf 100644 > --- a/mysql-test/main/ctype_utf8.test > +++ b/mysql-test/main/ctype_utf8.test > @@ -24,7 +24,7 @@ drop database if exists mysqltest; > --disable_warnings > drop table if exists t1,t2; > --enable_warnings > -set names utf8; > +set names utf8 collate utf8_general_ci; why not to change the test to work with utf8mb3_uca1400_ai_ci? > > select left(_utf8 0xD0B0D0B1D0B2,1); > select right(_utf8 0xD0B0D0B2D0B2,1); > diff --git a/mysql-test/main/dyncol.test b/mysql-test/main/dyncol.test > index 493e9b3842d..ba302861e03 100644 > --- a/mysql-test/main/dyncol.test > +++ b/mysql-test/main/dyncol.test > @@ -10,10 +10,10 @@ > --echo # column create > --echo # > select hex(COLUMN_CREATE(1, NULL AS char character set utf8)); > -select hex(COLUMN_CREATE(1, "afaf" AS char character set utf8)); > -select hex(COLUMN_CREATE(1, 1212 AS char character set utf8)); > -select hex(COLUMN_CREATE(1, 12.12 AS char character set utf8)); > -select hex(COLUMN_CREATE(1, 99999999999999999999999999999 AS char character set utf8)); > +select hex(COLUMN_CREATE(1, "afaf" AS char character set utf8 collate utf8_general_ci)); > +select hex(COLUMN_CREATE(1, 1212 AS char character set utf8 collate utf8_general_ci)); > +select hex(COLUMN_CREATE(1, 12.12 AS char character set utf8 collate utf8_general_ci)); > +select hex(COLUMN_CREATE(1, 99999999999999999999999999999 AS char character set utf8 collate utf8_general_ci)); why? > select hex(COLUMN_CREATE(1, NULL AS unsigned int)); > select hex(COLUMN_CREATE(1, 1212 AS unsigned int)); > select hex(COLUMN_CREATE(1, 7 AS unsigned int)); > diff --git a/mysql-test/main/func_json.result b/mysql-test/main/func_json.result > index ab35822a6ce..fce2caa2842 100644 > --- a/mysql-test/main/func_json.result > +++ b/mysql-test/main/func_json.result > @@ -414,6 +414,13 @@ select json_object('foo', json_unquote(json_object('bar', c)),'qux', c) as fld f > fld > {"foo": "{\"bar\": \"abc\"}", "qux": "abc"} > {"foo": "{\"bar\": \"def\"}", "qux": "def"} > +create table t2 as select json_object('foo', json_unquote(json_object('bar', c)),'qux', c) as fld from t1 limit 0; > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `fld` varchar(39) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci > +drop table t2; why do you test here? > drop table t1; > select json_object("a", json_object("b", "abcd")); > json_object("a", json_object("b", "abcd")) > diff --git a/mysql-test/main/func_regexp_pcre.test b/mysql-test/main/func_regexp_pcre.test > index 77f5af6e0ff..7cf9e8007de 100644 > --- a/mysql-test/main/func_regexp_pcre.test > +++ b/mysql-test/main/func_regexp_pcre.test > @@ -23,17 +23,17 @@ SELECT 'Ã ' RLIKE '\\x{00C0}' COLLATE utf8_bin; > SELECT 'Ã' RLIKE '\\x{00C0}' COLLATE utf8_bin; > > # Checking how (?i) and (?-i) affect case sensitivity > -CREATE TABLE t1 (s VARCHAR(10) CHARACTER SET utf8); > +CREATE TABLE t1 (s VARCHAR(10) CHARACTER SET utf8 COLLATE utf8_general_ci); why? > INSERT INTO t1 VALUES ('a'),('A'); > -CREATE TABLE t2 (p VARCHAR(10) CHARACTER SET utf8); > +CREATE TABLE t2 (p VARCHAR(10) CHARACTER SET utf8 COLLATE utf8_general_ci); > INSERT INTO t2 VALUES ('a'),('(?i)a'),('(?-i)a'),('A'),('(?i)A'),('(?-i)A'); > SELECT s,p,s RLIKE p, s COLLATE utf8_bin RLIKE p FROM t1,t2 ORDER BY BINARY s, BINARY p; > DROP TABLE t1,t2; > > > # Checking Unicode character classes > -CREATE TABLE t1 (ch VARCHAR(22)) CHARACTER SET utf8; > -CREATE TABLE t2 (class VARCHAR(32)) CHARACTER SET utf8; > +CREATE TABLE t1 (ch VARCHAR(22)) CHARACTER SET utf8 COLLATE utf8_general_ci; > +CREATE TABLE t2 (class VARCHAR(32)) CHARACTER SET utf8 COLLATE utf8_general_ci; and here > INSERT INTO t1 VALUES ('Ð¯'),('Î£'),('A'),('Ã'); > INSERT INTO t1 VALUES ('Ñ'),('Ï'),('a'),('Ã '); > INSERT INTO t1 VALUES ('ã'),('ê°·'),('à¶´'); > diff --git a/mysql-test/main/mrr_icp_extra.result b/mysql-test/main/mrr_icp_extra.result > index 8f6ee88acc6..d7cde103288 100644 > --- a/mysql-test/main/mrr_icp_extra.result > +++ b/mysql-test/main/mrr_icp_extra.result > @@ -754,7 +754,7 @@ t1 CREATE TABLE `t1` ( > KEY `t` (`t`(5)) > ) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_swedish_ci > drop table t1; > -create table t1 (v char(10) character set utf8); > +create table t1 (v char(10) character set utf8 collate utf8_general_ci); why here? > show create table t1; > Table Create Table > t1 CREATE TABLE `t1` ( > diff --git a/mysql-test/main/myisam.test b/mysql-test/main/myisam.test > index 1a20f97a54f..4a283737122 100644 > --- a/mysql-test/main/myisam.test > +++ b/mysql-test/main/myisam.test > @@ -1039,7 +1039,7 @@ create table t1 (v varchar(65536)); > show create table t1; > drop table t1; > set statement sql_mode = 'NO_ENGINE_SUBSTITUTION' for > -create table t1 (v varchar(65530) character set utf8); > +create table t1 (v varchar(65530) character set utf8 collate utf8_general_ci); why? collation doesn't matter here, does it? > show create table t1; > drop table t1; > > diff --git a/mysql-test/main/order_by.result b/mysql-test/main/order_by.result > index 274f29e34dc..85059fb3370 100644 > --- a/mysql-test/main/order_by.result > +++ b/mysql-test/main/order_by.result > @@ -4142,7 +4142,7 @@ drop table t1; > # > # MDEV-21922: Allow packing addon fields even if they don't honour max_length_for_sort_data > # > -create table t1 (a varchar(200) character set utf8, b int); > +create table t1 (a varchar(200) character set utf8 collate utf8_general_ci, b int); why? I think I see too many forced utf8_general_ci in tests and it's difficult to believe that the collation actually matters in all these cases. > insert into t1 select seq, seq from seq_1_to_10; > select * from t1 order by a; > a b > diff --git a/sql/item_jsonfunc.cc b/sql/item_jsonfunc.cc > index 508ea9f644e..5731ad1f04b 100644 > --- a/sql/item_jsonfunc.cc > +++ b/sql/item_jsonfunc.cc > @@ -842,7 +842,7 @@ String *Item_func_json_quote::val_str(String *str) > bool Item_func_json_unquote::fix_length_and_dec(THD *thd) > { > collation.set(&my_charset_utf8mb3_general_ci, > - DERIVATION_COERCIBLE, MY_REPERTOIRE_ASCII); > + DERIVATION_CAST, MY_REPERTOIRE_ASCII); why? I guess that's why you have json_unquote test above. > max_length= args[0]->max_length; > set_maybe_null(); > return FALSE; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: b657bc75f90: MDEV-25829 Change default collation to utf8mb4_1400_ai_ci
by Sergei Golubchik 13 Mar '24

13 Mar '24

Hi, Alexander, On Mar 08, Alexander Barkov wrote: > revision-id: b657bc75f90 (mariadb-11.0.1-262-gb657bc75f90) > parent(s): 8e5f41b3453 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-11-15 10:58:03 +0400 > message: > > MDEV-25829 Change default collation to utf8mb4_1400_ai_ci > > Step#2 - Adding a new collation derivation level for CAST and CONVERT. > > Now character string cast functions: > - CAST(string_expr AS CHAR) > - CONVERT(expr USING charset_name) > > have a new collation derivation level between: > > - string literals > - utf8 metadata functions, e.g. user() and database() > > Before the change these cast functions had collation derivation equal > to table columns, which caused more illegal mix of collation conflicts. > > Note, binary string cast functions: > - CAST(string_expr AS BINARY) > - CONVERT(expr USING binary) > did not change their collation derivation. Why not? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 8e5f41b3453: MDEV-25829 Change default collation to utf8mb4_1400_ai_ci
by Sergei Golubchik 13 Mar '24

13 Mar '24

Hi, Alexander, On Mar 08, Alexander Barkov wrote: > revision-id: 8e5f41b3453 (mariadb-11.0.1-261-g8e5f41b3453) > parent(s): 9e457cbe501 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-11-15 09:37:26 +0400 > message: > > MDEV-25829 Change default collation to utf8mb4_1400_ai_ci > > Step#1 - Changing collation derivation for string user variables > from EXPLICIT to COERCIBLE. from IMPLICIT, oherwise ok to push. and I still think user variable could remember the derivation. It's definitely not a part of MDEV-25829, but if you agree that it should, let's create an MDEV for that. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: [PATCH 0/2] Suggestion for smaller fix to XA parallel replication performance, MDEV-31949
by Kristian Nielsen 12 Mar '24

12 Mar '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > I reviewed your branch to agree with the idea of xid conflicts > handling and its implementation. > > I however could not understand the need of the refactoring part. > In order to track the xid dependency couple functions and and > rpl_parallel_entry::maybe_active_xid > a sort of a sliding window - that the 2nd commit of your branch > introduced must be sufficient to my analysis. > Of course it needs to hold worker indexes of active xid:s. > > This observation led me to create a review branch > > origin/review__knielsen_xa_sched_minimal_fix > > [where origin git@github.com:MariaDB/server.git] > which is a somewhat light elaboration over the 2nd commit of your branch. The reason for the refactoring part is to allow to distribute the transactions as evenly as possible over the worker threads, given the scheduling constraints imposed by XA dependencies. Let's say we have 5 worker threads, and transactions T1..T15. If there are no scheduling constraints, we can schedule evenly like this: W1: T1 T6 T11 W2: T2 T7 T12 W3: T3 T8 T13 W4: T4 T9 T14 W5: T5 T10 T15 Now suppose we need to schedule T6 on the same worker as T4, and T8 on the same worker as T3. The most even way to distribute the transactions is now this: W1: T1 T7 T12 W2: T2 T9 T14 W3: T3 T8 T13 W4: T4 T6 T11 W5: T5 T10 T15 You see, we try to always have 5 consecutive transactions T_i, T_{i+1}, ..., T+{i+4} scheduled on 5 different worker threads, except when this is not possible due to dependency restrictions. But this most-even scheduling requires to change the scheduling order of threads from the original sequential 1,2,3,4,5. You see, the last transactions T11..T15 are scheduled on worker threads in order: W4, W1, W3, W2, W5. The refactoring patch introduces thread_sched_fifo to hold the current scheduling order of worker threads. Whenever we have to schedule out-of-order due to a scheduling dependency, we put the scheduled worker at the end of this fifo, to preserve even scheduling for the next N transactions. In the original scheduling code, we never scheduled workers out-of-order, so the scheduling order was always cyclic W1, W2, W3, W4, W5, W1, ..., and a simple incrementing counter was sufficient. But this is no longer sufficient when out-of-order scheduling occurs. In the example, if we use a simple counter and try to schedule on worker ((i+1) mod N) after worker (i), then we get the following uneven scheduling: W1: T1 T11 W2: T2 T12 W3: T3 T8 T13 W4: T4 T6 T9 T14 W5: T5 T7 T10 T15 Because T6 and T8 have particular scheduling requirements, they cause the scheduling to skip workers. The result is that W4 and W5 get too many transactions, while W1 and W2 get too few, and parallelism is reduced. I hope this explain the reasoning. > Could you please have a look at it? If I understand your patch correctly, it schedules using a simple incrementing counter when there are no dependency requirements, and so would suffer from the uneven scheduling in the above example. It also looks like there's a mistake in the code: > + if ((idx= check_xa_xid_dependency(&gtid_ev->xid)) < (uint32) -1) > + { > + /* > + A previously scheduled event group with the same XID might still be > + active in a worker, so schedule this event group in the same worker > + to avoid a conflict. > + */ > + } > + else > + { > + /* Record this XID now active. */ > + xid_active_generation *a= > + (xid_active_generation *)alloc_dynamic(&maybe_active_xid); > + if (!a) > + return NULL; > + ++idx; > + if (idx >= rpl_thread_max) > + idx= 0; If check_xa_xid_dependency() returns "no dependency", then idx is set to (uint32)-1 and the else {...} branch is executed. This does idx++, which increments the (uint32)-1 to 0. So it looks to me that _all_ transactions are scheduled on worker 0, or am I missing something? This is a simple mistake, it's an easy fix to not assign idx when check_xa_xid_dependency() returns "no dependency". But the code would still suffer from uneven scheduling as in the example, IIUC. So there's a good reason to introduce the thread_sched_fifo as done in my refactoring patch. This though still leaves the limitation in my minimal patch that XA PREPARE xid1 cannot group commit with XA COMMIT xid1. The impact of this will depend on how many transactions on average appear between an XA PREPARE and its associated XA COMMIT, as well as on how expensive fsync() is relative to the cost of each transaction. - Kristian.

1 0

Re: 9e5d4dfc49b: MDEV-23729 MDEV-32218 INFORMATION_SCHEMA table for user login data
by Sergei Golubchik 11 Mar '24

11 Mar '24

Hi, Nikita, On Mar 10, Nikita Malyavin wrote: > revision-id: 9e5d4dfc49b (mariadb-11.4.1-10-g9e5d4dfc49b) > parent(s): 929c2e06aae > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-02-29 17:24:27 +0100 > message: > > MDEV-23729 MDEV-32218 INFORMATION_SCHEMA table for user login data > > * A new table INFORMATION_SCHEMA.LOGON is introduced. There were many cases where we lack an INFORMATION_SCHEMA with all user accounts. Forcing users to select from mysql.user is a shame. Let's use the chance and create a table USERS. It doesn't need more columns now than what you've created, but we'll likely create more in the future. > * Upon idea, it stores auxiliary user data related to login/security/resources > * An unprivileged user can access their own data, and that is the main > difference with what mysql.global_priv provides exactly! > * The fields are currently: USER, WRONG_PASSWORD_ATTEMPTS, EXPIRATION_TIME I wanted to write about splitting USER into USER and HOST, but indeed it's just USER everywhere else in INFORMATION_SCHEMA (e.g. DEFINER and GRANTEE columns). And functions like USER() and CURRENT_USER() don't split either. So, you're right, let's consistently use user@host everywhere. > diff --git a/mysql-test/main/information_schema_stats.result b/mysql-test/main/information_schema_stats.result > index 352bcbab823..e38788872fa 100644 > --- a/mysql-test/main/information_schema_stats.result > +++ b/mysql-test/main/information_schema_stats.result ... > +connect(localhost,naughty_user,wrong_passwd,test,16000,/home/nik/mariadb/bld/mysql-test/var/tmp/mysqld.1.sock); here and below - you forgot to replace the path with MASTER_MYSOCK > diff --git a/mysql-test/main/information_schema_stats.test b/mysql-test/main/information_schema_stats.test > index fd5171c3fb4..dbcabe45965 100644 > --- a/mysql-test/main/information_schema_stats.test > +++ b/mysql-test/main/information_schema_stats.test > @@ -47,3 +47,69 @@ select * from information_schema.index_statistics where table_schema='test' and > select * from information_schema.table_statistics where table_schema='test' and table_name='just_a_test'; > set global userstat=@save_userstat; > --enable_ps2_protocol > + > +--echo # > +--echo # MDEV-23729 INFORMATION_SCHEMA Table info. about user locked due to > +--echo # max_password_errors > +--echo # > +--echo # MDEV-32218 message to notify end-user N-days prior the password get > +--echo # expired > +--echo # I don't see how you test "info about user locked due to max_password_errors". This is the reason for implementing this new table, it has to be tested. At least add select * from information_schema.logon where wrong_password_attemps >= @max_password_errors; and show it it's empty at first and then not empty. > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 14450a5a610..4d199bf0e61 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -12956,6 +12956,87 @@ int fill_schema_column_privileges(THD *thd, TABLE_LIST *tables, COND *cond) > #endif > } > > +namespace Show > +{ > + ST_FIELD_INFO users_fields_info[] = > + { > + Column("USER", Userhost(), NOT_NULL), > + Column("WRONG_PASSWORD_ATTEMPTS", SLonglong(), NULLABLE), @@max_password_errors is documented as "If there is more than this number of failed connect attempts due to invalid password, user will be blocked from further connections until FLUSH_PRIVILEGES" Let's use call incorrect password consistently everywhere. It could be "bad password", "wrong password", "incorrect password", "invalid password", but it should be the same everywhere. I personally think that "invalid" implies some kind if validity check, e.g. not shorter than 10 characters, not equal to the username, etc and a password that doesn't satisfy these validity rules is "invalid". Trying to login with an incorrect password is a different kind of error, a password can be valid but wrong. May be, ask Ian and/or Daniel about it? And then don't forget to update @@max_password_errors help text to match. Also, you don't have a test case for FLUSH PRIVILEGES. > + Column("EXPIRATION_TIME", SLonglong(), NULLABLE), PASSWORD_EXPIRATION_TIME, the user account itself does not expire. > + CEnd() > + }; > +}; > + > +static bool ignore_max_password_errors(const ACL_USER *acl_user); > + > +static int fill_logon_schema_record(THD *thd, TABLE * table, ACL_USER *user) > +{ > + ulonglong lifetime= user->password_lifetime < 0 > + ? default_password_lifetime > + : user->password_lifetime; > + > + bool ignore_password_errors= ignore_max_password_errors(user); why? I think it's still useful to show how many wrong password attempts were there for an account even if it doesn't get blocked. > + bool ignore_expiration_date= lifetime == 0; > + > + /* Skip user if nothing to show */ > + if (ignore_password_errors && ignore_expiration_date) > + return 0; > + > + Grantee_str grantee(user->user.str, safe_str(user->host.hostname)); > + table->field[0]->store(grantee, strlen(grantee), system_charset_info); > + if (ignore_password_errors) > + { > + table->field[1]->set_null(); > + } > + else > + { > + table->field[1]->set_notnull(); > + table->field[1]->store(user->password_errors); > + } > + if (ignore_expiration_date) > + { > + table->field[2]->set_null(); > + } > + else > + { > + table->field[2]->set_notnull(); > + table->field[2]->store(user->password_last_changed > + + user->password_lifetime * 3600 * 24, true); I think it'd be more generally useful to show password_last_changed and expiration period separately. > + } > + > + return schema_table_store_record(thd, table); > +} > + > +int fill_logon_schema_table(THD *thd, TABLE_LIST *tables, COND *cond) > +{ > + int res= 0; > +#ifndef NO_EMBEDDED_ACCESS_CHECKS > + bool see_whole_table= check_global_access(thd, PROCESS_ACL, true) == 0; I don't think PROCESS_ACL is a very logical choice here. And there's nothing better as far as I can see. May be let's just do if (check_access(thd, SELECT_ACL, "mysql", NULL, NULL, 1, 1)) ? There are many checks like that in e.g. sql_parse.cc > + > + TABLE *table= tables->table; > + > + if (!see_whole_table) > + { > + mysql_mutex_lock(&acl_cache->lock); > + ACL_USER *cur_user= find_user_exact(thd->security_ctx->priv_host, > + thd->security_ctx->priv_user); 1. cur_user can be NULL if someone dropped it while there was an active connection for this user 2. add a test for it > + > + res= fill_logon_schema_record(thd, table, cur_user); > + mysql_mutex_unlock(&acl_cache->lock); > + return res; > + } > + > + mysql_mutex_lock(&acl_cache->lock); > + for (size_t i= 0; res == 0 && i < acl_users.elements; i++) > + { > + ACL_USER *user= dynamic_element(&acl_users, i, ACL_USER*); > + res= fill_logon_schema_record(thd, table, user); > + } > + mysql_mutex_unlock(&acl_cache->lock); > +#endif > + return res; > +} > + Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Backup MariaDB Databases for Suprema BioStar 2 Door Access System
by Turritopsis Dohrnii Teo En Ming 09 Mar '24

09 Mar '24

Subject: Backup MariaDB Databases for Suprema BioStar 2 Door Access System Good day from Singapore, On 7 March 2024 Thursday, when I was installing new self-signed SSL certificate for the door access system for a law firm in Singapore, I notice that Suprema BioStar 2 also uses the open source MariaDB database server. If I did not remember wrongly, there are 3 databases. However, the entire platform is a Windows environment, not Linux. How can I backup all the databases in the MariaDB database server? Thank you. Regards, Mr. Turritopsis Dohrnii Teo En Ming Targeted Individual in Singapore Blogs: https://tdtemcerts.blogspot.com https://tdtemcerts.wordpress.com GIMP also stands for Government-Induced Medical Problems.

1 0

Re: be66e975ecf: MDEV-31340 Remove MY_COLLATION_HANDLER::strcasecmp()
by Sergei Golubchik 09 Mar '24

09 Mar '24

Hi, Alexander, Ok to push, thanks for the great refactoring! I've looked at the diff of this and previous commit excluding InnoDB. On Mar 08, Alexander Barkov wrote: > revision-id: be66e975ecf (mariadb-11.0.1-282-gbe66e975ecf) > parent(s): c0c1c80346b > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-01-19 11:04:02 +0400 > message: > > MDEV-31340 Remove MY_COLLATION_HANDLER::strcasecmp() > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

64-bit time_t support in MariaDB?
by Otto Kekäläinen 07 Mar '24

07 Mar '24

Hello! Does MariaDB already fully support 64-bit time_t (the year 2038 problem)? MariaDB 10.11 was today uploaded to Debian experimental without any 64-bit time_t support changes, just a library name change[1] to test that it works with the Debian 64-bit time_t tooling as Debian is currently undergoing a transition to it[2]. So far MariaDB built successfully and passed the main MTR suite on 11 architectures[3]. Monty mentioned that some work is in progress, and indeed I found his commits in two branches [4,5] but looking at the commits and their link to buildbot it seems they are not passing the CI. I also found one MDEV[6] to make TIMESTAMP use the whole 32-bit unsigned range. Are there any other efforts in progress? Does MTR main suite include tests for 64-bit time support? Should MTR automatically currently fail if toolchain time is 64-bit? [1] https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/063109a306a01… [2] https://lists.debian.org/debian-devel-announce/2024/02/msg00000.html [3] https://buildd.debian.org/status/package.php?p=mariadb&suite=experimental [4] https://github.com/MariaDB/server/commits/bb-11.4-timestamp/ [5] https://github.com/MariaDB/server/commits/bb-11.4-monty/ [6] https://jira.mariadb.org/browse/MDEV-32188

2 6

Re: [PATCH 2/2] MDEV-33551: Semi-sync Wait Point AFTER_COMMIT Slow on Workloads with Heavy Concurrency
by Kristian Nielsen 06 Mar '24

06 Mar '24

> commit c51636f254703602f6f6e2e4a260e607e737b9c1 (origin/10.6-MDEV-33551) > Author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > Date: Mon Feb 26 07:01:47 2024 -0700 > > MDEV-33551: Semi-sync Wait Point AFTER_COMMIT Slow on Workloads with Heavy Concurrency First some overall comments. The approach of waking up each thread on its own condition seems good to me. Monty had a different suggestion, which is to have only a single ack sent per group commit; this could also be a possibility, but it seems orthogonal to your fix (ie. both could make sense). Monty also had the idea to move the AFTER_COMMIT sync point into the group commit code, though that changes the semantics of AFTER_COMMIT somewhat, especially with --innodb-flush-log-at-trx-commit=3 (or non-InnoDB storage engine). So in this review I'll focus on keeping your overall approach the same. Your patch introduces a new wait_queue with the list of waiting threads. But there is already a very similar queue, m_active_tranxs. It looks to me as if the m_active_tranx could easily be extended so it can be used for this, and avoid introducing another queue. Your wait_queue is pushed in write_tranx_in_binlog() which already does m_active_tranxs->insert_tranx_node(), and it is popped in report_reply_binlog() which also does m_active_tranxs->clear_active_tranx_nodes(). So add some info to struct Tranx_node, maybe a THD * is all that is needed? A bit of care must be taken about lifetime of Tranx_node and THD respectively (in case one goes away but not the other). But the whole m_active_tranxs is already protected by a lock, and there is a hash for looking up specific position, so I think a THD that wants to abort its wait can simply clear the THD * from its Tranx_node or something like that to ensure the semisync ack will not try to wake up a THD that no longer exists. You'll need to see how this will affect the code for shutdown (await_all_slave_replies()) where it needs to wait for all acks (IIUC). But I hope we can be pragmatic here and find something simple that works. Another general thing related to this: > 3) Repl_semi_sync_master::commit_trx() no longer loops to await > its specific ACK. It waits once, and will either fail from > timeout, or receive its ACK. We still need the loop to wait. From `man pthread_cond_wait`: "When using condition variables there is always a Boolean predicate in‐ volving shared variables associated with each condition wait that is true if the thread should proceed. Spurious wakeups from the pthread_cond_timedwait() or pthread_cond_wait() functions may occur. Since the return from pthread_cond_timedwait() or pthread_cond_wait() does not imply anything about the value of this predicate, the predi‐ cate should be re-evaluated upon such return." This affects the code in the patch in some places, detailed comments below. > 2) The time when thd::is_awaiting_semi_sync_ack is set is moved > to at binlogging time, to ensure transactions which have been > binlogged and queued up to await an ACK are not killed, > and are still waited on. I don't understand this. Why shouldn't a session waiting for ACK be killable? On the contrary, such a wait can take potentially a long time, seems important that it can be killed? I don't fully understand the existing code for this, maybe the patch is correct and I'm just looking for an explanation why it is correct like this. Or maybe this is a limitation of the existing code, and your patch doesn't change it (so having the ability to kill a waiting thread could be added in a different patch)? > diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_cond_var_per_thd.test b/mysql-test/suite/rpl/t/rpl_semi_sync_cond_var_per_thd.test > index f8fa0a99d9c..ebe15a4ca32 100644 > --- a/mysql-test/suite/rpl/t/rpl_semi_sync_cond_var_per_thd.test > +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_cond_var_per_thd.test > @@ -22,6 +22,7 @@ > --source include/master-slave.inc > > --connection master > +call mtr.add_suppression("Got an error reading communication packets"); Is this suppression needed? If so, why? > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index e224871795e..b315edc091c 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -1750,18 +1750,12 @@ static void close_connections(void) > /* > If we are waiting on any ACKs, delay killing the thread until either an ACK > is received or the timeout is hit. > - while (waiting_threads-- > 0) > - repl_semisync_master.await_slave_reply(); > + repl_semisync_master.await_all_slave_replies(); Just curious here, why did you change this code? Was it just because now there is no general COND_binlog_send to wait on? Or was it fix a bug not directly related to the main issue? I'm asking because if it makes things simpler, I think it's fine to keep the COND_binlog_send and signal it in addition to the individual thread-specific condition (in fact your patch doesn't seem to remove it though it looks as if it's no longer used). On the other hand, this old code in close_connections() that simply calls await_slave_reply() N times (N = sync_get_master_wait_sessions()) looks quite wrong to me, looping for a specific number of times makes no sense. So fixing it is good in any case. > diff --git a/sql/semisync_master.cc b/sql/semisync_master.cc > index 0eaf0f0e0e2..f421e924a61 100644 > --- a/sql/semisync_master.cc > +++ b/sql/semisync_master.cc > @@ -502,16 +502,33 @@ void Repl_semi_sync_master::unlock() > > void Repl_semi_sync_master::cond_broadcast() > { > - mysql_cond_broadcast(&COND_binlog_send); > + while (!wait_queue.empty()) > + { > + semisync_wait_trx_t next_waiter= wait_queue.front(); Maybe this function should now have a better name than cond_broadcast() (wakeup_ready_tranx() or something?). But this code will probably change in any case if the wait_queue is folded into m_active_tranxs. > -int Repl_semi_sync_master::cond_timewait(struct timespec *wait_time) > +int Repl_semi_sync_master::cond_timewait(THD *thd, struct timespec *wait_time) I think this function now makes little sense, clearer just to call directly mysql_cond_wait(thd->COND_wakeup_ready) at the caller, what do you think? > @@ -695,12 +712,36 @@ int Repl_semi_sync_master::report_reply_binlog(uint32 server_id, > + lock(); > + while (!wait_queue.empty()) > + { > + semisync_wait_trx_t next_waiter= wait_queue.front(); > > - cond_broadcast(); > + cmp= Active_tranx::compare(m_reply_file_name, m_reply_file_pos, > + next_waiter.binlog_name, next_waiter.binlog_pos); > + if (cmp >= 0) > + { > + DBUG_PRINT("semisync", ("%s: signal thread %llu.", > + "Repl_semi_sync_master::report_reply_binlog", > + next_waiter.thd->thread_id)); > + mysql_cond_signal(&next_waiter.thd->COND_wakeup_ready); > + wait_queue.pop(); This code is very similar to Active_tranx::clear_active_tranx_nodes(), so hopefully the two can be integrated and wait_queue becomes unnecessary. > @@ -779,7 +821,20 @@ int Repl_semi_sync_master::report_binlog_update(THD* thd, const char *log_file, > strcpy(log_info->log_file, log_file + dirname_length(log_file)); > log_info->log_pos = log_pos; > > - return write_tranx_in_binlog(log_info->log_file, log_pos); > + /* > + THD arg depends on wait point mode. If after storage engine commit, the > + individual connection threads will perform the wait for semi-sync ACKt, > + thd is the thread of the user connection thread. > + If it is after binlog sync, the binlog leader thread will perform the > + semi-sync waits on behalf of the grouped transaction (which at this > + point, we (current_thd) are the leader). If using binlog_group_commit, > + thd is the thread of the user connection thread. > + */ > + thd_to_cond_wait= ((wait_point() == SEMI_SYNC_MASTER_WAIT_POINT_AFTER_STORAGE_COMMIT) > + ? thd > + : current_thd); This condition is complex. It's good that you added the detailed comment explaining what's going on, but I think it would be better to avoid it completely. Why not just pass in the value of thd_to_cond_wait explicitly from the caller? As an aside, we have this code in trx_group_commit_leader(): for (current= queue; current != NULL; current= current->next) { current->error= repl_semisync_master.wait_after_sync(current->cache_mngr-> last_commit_pos_file, current->cache_mngr-> last_commit_pos_offset); It seems silly to wait for every binlog position to be acked her one by one, why not just wait for the last one? repl_semisync_master.wait_after_sync(last_in_queue->cache_mngr->last_commit_pos_file, ...); But while related, it's perhaps not directly part of the problem you're addressing in your patch, so up to you if you want to include this or leave it for a possible later patch. > @@ -852,12 +907,11 @@ int Repl_semi_sync_master::commit_trx(const char* trx_wait_binlog_name, > - if (!get_master_enabled() || !is_on()) > + if (!get_master_enabled() || !is_on() || thd_killed(thd)) > goto l_end; > - while (is_on() && !thd_killed(thd)) > - { > - /* We have to check these again as things may have changed */ > - if (!rpl_semi_sync_master_clients && !rpl_semi_sync_master_wait_no_slave) > - { > - aborted= 1; > - break; > - } As described earlier, because of the (at least theoretical) possibility of spurious wakeup, we should keep the while loop here and the checks. > @@ -1253,6 +1294,8 @@ int Repl_semi_sync_master::write_tranx_in_binlog(const char* log_file_name, > } > else > { > + wait_queue.push({log_file_name, log_file_pos, thd}); Hopefully this can be instead folded into m_active_tranxs->insert_tranx_node() which is called just a bit below. > -void Repl_semi_sync_master::await_slave_reply() > +void Repl_semi_sync_master::await_all_slave_replies() > + while (TRUE) > + { > + lock(); > + if (wait_queue.empty() || !get_master_enabled() || !is_on()) > + { > + unlock(); > + break; > + } > + front= wait_queue.front(); > create_timeout(&abstime, NULL); > - cond_timewait(&abstime); > - > -end: > + wait_result= cond_timewait(front.thd, &abstime); I'm wondering, is it safe here to wait on front.thd->COND_wakeup_ready? Isn't there a possibility that this might go away during the wait? A simple way to avoid this could be just to keep the original global COND_binlog_send and wait for that here (and signal both that and thd->COND_wakeup_ready when waking up). IIUC, this code will only run in a single thread during shutdown, so probably fine to have some extra unnecessary wakeups here. > diff --git a/sql/semisync_master.h b/sql/semisync_master.h > index 99f46869354..6de0333d4ce 100644 > --- a/sql/semisync_master.h > +++ b/sql/semisync_master.h > @@ -21,6 +21,7 @@ > +/* > + Element in Repl_semi_sync_master::wait_queue to preserve the state of a > + transaction waiting for an ACK. > +*/ > +typedef struct _semisync_wait_trx { > + const char *binlog_name; > + my_off_t binlog_pos; > + THD *thd; > +} semisync_wait_trx_t; > + std::queue<semisync_wait_trx_t> wait_queue; Let's see if we can avoid this, and use the existing m_active_tranxs queue instead. diff --git a/sql/semisync_master.cc b/sql/semisync_master.cc index 8cc721e5737..0eaf0f0e0e2 100644 --- a/sql/semisync_master.cc +++ b/sql/semisync_master.cc @@ -979,6 +979,14 @@ int Repl_semi_sync_master::commit_trx(const char* trx_wait_binlog_name, { rpl_semi_sync_master_trx_wait_num++; rpl_semi_sync_master_trx_wait_time += wait_time; + /* + Assert we have either recieved our ACK; or have timed out and are + awoken in an off state. + */ + DBUG_ASSERT(!get_master_enabled() || !is_on() || thd->is_killed() || + 0 <= Active_tranx::compare( + m_reply_file_name, m_reply_file_pos, + trx_wait_binlog_name, trx_wait_binlog_pos)); } } } I think it's a bit much to crash the server here (even if only debug build), because of the possibility of spurious wakeups. It's good to have the test case though, maybe put the assertion inside DBUG_EXECUTE_IF()? I'd make it just log an error in the error log which the test case could flag (to avoid crashing), but that's a matter of style I guess and up to you. - Kristian.

2 2

Re: 95f2a07d323: MDEV-33582 Add more warnings to be able to better diagnose network issues
by Sergei Golubchik 05 Mar '24

05 Mar '24

Hi, Michael, On Mar 04, Michael Widenius wrote: > revision-id: 95f2a07d323 (mariadb-10.6.17-30-g95f2a07d323) > parent(s): 33dcf8155b4 > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-03-04 14:52:03 +0200 > message: > > MDEV-33582 Add more warnings to be able to better diagnose network issues > > Warnings are added to net_server.cc when > global_system_variables.log_warnings >= 4. > > When the above condition holds then: > - All communication errors from net_serv.cc is also written to the > general log. "to the error log" > - In case of a of not being able to read or write a packet, a more detailed "of a of" ? > error is given. > > Other things: > - vio_close() is now first marking the socket closed before closing it. > The reason for this is to ensure that the connection that gets a read > error can check if the reason was that the socket was closed. > - Added option to NET to avoid calling my_error() when there is no data > to be read on socket. This is only used in semi sync master to speed > up handling zero size packages that seams to happen from time to time. "seems" I don't understand why it happens and whether it's normal or not, and if normal, why it's normal only for semisync and not normal otherwise. > - Add a new state to vio to be able to detect if vio is acive, shutdown or > closed. > - Testing of the new warnings is done in rpl_get_lock.test > - Suppress some of the new warnings in mtr to allow one to run some of > the tests with -mysqld=--log-warnings=4. All test in the 'rpl' suite > can now be run with this option. > - Ensure that global.log_warnings are restored at test end in a way > that allows one to use mtr --mysqld=--log-warnings=4. > > diff --git a/mysql-test/mariadb-test-run.pl b/mysql-test/mariadb-test-run.pl > index 4b3d323760d..eabfe1171cc 100755 > --- a/mysql-test/mariadb-test-run.pl > +++ b/mysql-test/mariadb-test-run.pl > @@ -4486,6 +4486,11 @@ sub extract_warning_lines ($$) { > qr/Slave I\/0: Master command COM_BINLOG_DUMP failed/, > qr/Error reading packet/, > qr/Lost connection to MariaDB server at 'reading initial communication packet'/, > + qr/Could not read packet:.* state: [2-3] /, > + qr/Could not read packet:.* errno: 104 /, > + qr/Could not write packet:.* errno: 32 /, > + qr/Could not write packet:.* errno: 104 /, > + qr/Connection was killed/, why are these errors "normal" and can be ignored? > qr/Failed on request_dump/, > qr/Slave: Can't drop database.* database doesn't exist/, > qr/Slave: Operation DROP USER failed for 'create_rout_db'/, > diff --git a/mysql-test/suite/rpl/include/rpl_extra_col_master.test b/mysql-test/suite/rpl/include/rpl_extra_col_master.test > index 02645a577f5..512118f91cd 100644 > --- a/mysql-test/suite/rpl/include/rpl_extra_col_master.test > +++ b/mysql-test/suite/rpl/include/rpl_extra_col_master.test > @@ -59,6 +59,10 @@ > #VARCHAR(M) > # > > +--disable_query_log > +call mtr.add_suppression("Could not read packet:.* errno: 11"); > +--enable_query_log error 11? "Resource temporarily unavailable"? What does it mean in this context? > + > --let $_saved_conn= $CURRENT_CONNECTION > > let $binformat = `SHOW VARIABLES LIKE '%binlog_format%'`; > diff --git a/sql/semisync_master_ack_receiver.cc b/sql/semisync_master_ack_receiver.cc > index 9320c7fc3e6..1bb462ef934 100644 > --- a/sql/semisync_master_ack_receiver.cc > +++ b/sql/semisync_master_ack_receiver.cc > @@ -256,6 +256,12 @@ void Ack_receiver::run() > thd->security_ctx->skip_grants(); > thd->set_command(COM_DAEMON); > init_net(&net, net_buff, REPLY_MESSAGE_MAX_LENGTH); > + /* > + Don't generate error message in case of no data on socket. Will speed > + up my_net_read() slightly in case of NET_READ_INTERRUPTED. > + */ > + net.no_errors_for_no_data= 1; do you mean, it's for performance reasons? > + Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: ea9a05f0821: This commit is about improving repair of tables
by Sergei Golubchik 04 Mar '24

04 Mar '24

Hi, Michael, First, instead of situationally fixing repair code here and there, let's take a step back and see how repair works. I've walked the whole REPAIR code line by line, including all storage engines, and that's a simplified description: * if USE_FRM was specified * < FRM_VER_TRUE_VARCHAR(≈2005) && has varchar -> "too old" * if not MyISAM or Aria or CSV -> silently ignore USE_FRM * rename/truncate/rename * if ALGORITHM=COPY is needed (and not FORCE/USE_FRM), use it * otherwise, per engine: * MyISAM run repair, QUICK or EXTENDED, or QUICK then EXTENDED * Aria, same as MyISAM * S3 -> "Table '%-.192s' is read only" * CSV crashes * Archive, perform OPTIMIZE * all other engines * if needs upgrade: use ALTER ALGORITHM=COPY * otherwise "The storage engine for the table doesn't support REPAIR" * if ALGORITHM=COPY is needed but wasn't used -> "Operation failed" I think it's too complex to explain and not user friendly. Note that MyISAM internally can try one repair method and if that fails fall back to another one. So does Aria. Why cannot the server do the same? So, I'd suggest the following logic instead: * if USE_FRM - do USE_FRM stuff (or an warning) * if marked corrupted or EXTENDED or doesn't need upgrade - run repair * if repair isn't supported = ALTER IGNORE unless needs upgrade * if needs upgrade - run upgrade = ALTER IGNORE COPY * if fails because of corruption - go to step 2 if repair wasn't run yet here USE_FRM is handled as before (except that it adds a warning if USE_FRM isn't supported by the engine). After that the server runs engine's repair if the table is marked corrupted or if EXTENDED was used (so no need for a new FORCE syntax). There is no error "engine does not support REPAIR", the server falls back to ALTER automatically in that case. If the data need to be upgraded and ALTER fails, the server tries repair and then ALTER again - just like MyISAM and Aria try different repair methods. Also, note that ALTER should always use IGNORE to delete duplicates in unique after collation of hash changes - that's what repair in MyISAM/Aria does. Anyway, some detailed comments to your patch are inline, please, see below. On Feb 21, Michael Widenius wrote: > commit ea9a05f0821 > Author: Michael Widenius <monty(a)mariadb.org> > Date: Mon Jan 29 11:52:44 2024 +0200 > > This commit is about improving repair of tables now as there is an MDEV, it should be here in the subject line > > The 3 ways to repair a table are: > 1) ALTER TABLE table_name FORCE" (not other options). > As an alias we allow: "ALTER TABLE table_name ENGINE=original_engine" > 2) "REPAIR TABLE" (without FORCE) > 3) "OPTIMIZE TABLE" > > All of the above commands will optimize row space usage (which means that > space will be needed to hold a temporary copy of the table) and > re-genereate all indexes. They will also try to replicate the original > table definition as exact as possible. > > For ALTER TABLE and "REPAIR TABLE without FORCE", the following will hold: > If the table is from an older MariaDB version and data conversion is > needed (for example for old type HASH columns, MySQL JSON type or new > TIMESTAMP format) "ALTER TABLE table_name FORCE, algorithm=COPY" will be > used. > > The differences between the algoritms are > 1) Will use the fastest algorithm the engine supports to do a full repair > of the table (except if data conversions are is needed). > 2) Will use the storage engine internal REPAIR facility (MyISAM, Aria). > If the engine does not support REPAIR then > "ALTER TABLE FORCE, ALGORITHM=COPY" will be used. > If there was data incompatibilites (which means that FORCE was used) > then there will be a warning after REPAIR that ALTER TABLE FORCE is > still needed. > The reason for this is that REPAIR may be able to go around data > errors (wrong incompatible data, crashed or unreadable sectors) that > ALTER TABLE cannot do. > 3) Will use the storage engine internal OPTIMIZE. If engine does not > support optimize, then "ALTER TABLE FORCE" is used. > > The above will ensure that ALTER TABLE FORCE is able to > correct almost any errors in the row or index data. In case of > corrupted blocks then REPAIR possibile followed by ALTER TABLE is needed. > This is important as mariadb-upgrade executes ALTER TABLE table_name > FORCE for any table that must be re-created. > > Bugs fixed with InnoDB tables: > - No error for INNODB_DEFAULT_ROW_FORMAT=COMPACT even if row length > would be too wide. (Independent of innodb_strict_mode). > - Tables using symlinks will be symlinked after any of the above commands > (Independent of the setting of --symbolic-links) > > If one specifies an algorithm together with ALTER TABLE FORCE, things > will work as before (except if data conversion is required as then > the COPY algoritm is enforced). > > ALTER TABLE .. OPTIMIZE ALL PARTITIONS will work as before. > > Other things: > - FORCE argument added to REPAIR to allow one to first run internal > repair to fix damaged blocks and then follow it with ALTER TABLE. > - REPAIR will not update frm_version if ha_check_for_upgrade() finds > that table is still incompatible with current version. In this case the > REPAIR will end with an error. doesn't look like "the REPAIR will end with an error" > - Default error messages length for %M increased from 128 to 256 to not > cut information from REPAIR. > - Documented HA_ADMIN_XX variables related to repair. > - Added HA_ADMIN_NEEDS_DATA_CONVERSION to signal that we have to > do data conversions when converting the table (and thus ALTER TABLE > copy algorithm is needed). > > diff --git a/include/my_handler_errors.h b/include/my_handler_errors.h > index df414888907..06dde5b3f74 100644 > --- a/include/my_handler_errors.h > +++ b/include/my_handler_errors.h > @@ -101,7 +101,7 @@ static const char *handler_error_messages[]= > "Operation was interrupted by end user (probably kill command?)", > "Disk full", > /* 190 */ > - "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You may have retry or dump and restore the table to fix this", > + "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You can try REPAIR TABLE ... USE_FRM possible followed by ALTER TABLE ... FORCE or dump and restore the table to fix this", 1. typo. "possibly" 2. please, don't recommend USE_FRM, it's a last resort lossy method, designed only for the case when MYI file is lost or corrupted. 3. REPAIR USE_FRM is likey broken for CSV, but I couldn't test it, because REPAIR crashes on CSV tables with or without USE_FRM. > "Too many words in a FTS phrase or proximity search", > "Table encrypted but decryption failed. This could be because correct encryption management plugin is not loaded, used encryption key is not available or encryption method does not match.", > "Foreign key cascade delete/update exceeds max depth", > diff --git a/mysql-test/main/ctype_utf8_def_upgrade.result b/mysql-test/main/ctype_utf8_def_upgrade.result > index d30f8670536..e7c21b76058 100644 > --- a/mysql-test/main/ctype_utf8_def_upgrade.result > +++ b/mysql-test/main/ctype_utf8_def_upgrade.result > @@ -10,24 +10,39 @@ SELECT @@character_set_database; > latin1 > SET @@character_set_database="latin1"; > SELECT COUNT(*) FROM t1; > -ERROR HY000: Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You may have retry " from storage engine MyISAM > +ERROR HY000: Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You can try REPAIR TABLE ... USE_FRM possible followed by ALTER TABLE ... FORCE or dump and restore the table to fix this" from storage engine MyISAM > +ALTER TABLE t1 FORCE; > +ERROR HY000: Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You can try REPAIR TABLE ... USE_FRM possible followed by ALTER TABLE ... FORCE or dump and restore the table to fix this" from storage engine MyISAM > CHECK TABLE t1; > Table Op Msg_type Msg_text > -test.t1 check Error Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You may have retry " from storage engine MyISAM > +test.t1 check Error Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You can try REPAIR TABLE ... USE_FRM possible followed by ALTER TABLE ... FORCE or dump and restore the table to fix this" from storage engine MyISAM > test.t1 check error Corrupt > REPAIR TABLE t1; > Table Op Msg_type Msg_text > -test.t1 repair Error Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You may have retry " from storage engine MyISAM > +test.t1 repair Error Got error 190 "Incompatible key or row definition between the MariaDB .frm file and the information in the storage engine. You can try REPAIR TABLE ... USE_FRM possible followed by ALTER TABLE ... FORCE or dump and restore the table to fix this" from storage engine MyISAM > +test.t1 repair error Corrupt > +REPAIR TABLE t1 FORCE; > +Table Op Msg_type Msg_text > +test.t1 repair Warning Can't open table > test.t1 repair error Corrupt > REPAIR TABLE t1 USE_FRM; > Table Op Msg_type Msg_text > -test.t1 repair status OK > +test.t1 repair note Table data recovered > +t1 repair error Table rebuild required. Please do "ALTER TABLE `test.t1` FORCE" or dump/reload to fix it! better to avoid exclamation marks in the error messages! > +test.t1 repair status Operation failed > SELECT COUNT(*) FROM t1; > COUNT(*) > 0 > CHECK TABLE t1; > Table Op Msg_type Msg_text > +test.t1 check error Table rebuild required. Please do "ALTER TABLE `t1` FORCE" or dump/reload to fix it! > +ALTER TABLE t1 FORCE; > +CHECK TABLE t1; > +Table Op Msg_type Msg_text > test.t1 check status OK > +SELECT COUNT(*) FROM t1; > +COUNT(*) > +0 > SHOW CREATE TABLE t1; > Table Create Table > t1 CREATE TABLE `t1` ( > diff --git a/sql/handler.h b/sql/handler.h > index fe272d841d9..f429cdacc6f 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -2249,6 +2267,8 @@ struct Table_scope_and_contents_source_pod_st // For trivial members > enum_stats_auto_recalc stats_auto_recalc; > bool varchar; ///< 1 if table has a VARCHAR > bool sequence; // If SEQUENCE=1 was used > + /* True if we are using OPTIMIZE TABLE, REPAIR TABLE or ALTER TABLE FORCE */ it'd be good to clarify here /* this is used to suppress certain warnings that are only used for CREATE/ALTER TABLE */ > + bool recreate_identical_table; > > List<Virtual_column_info> *check_constraint_list; > > diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc > index fcbd8a55c15..d4eabd7c34a 100644 > --- a/sql/sql_admin.cc > +++ b/sql/sql_admin.cc > @@ -48,7 +48,8 @@ const LEX_CSTRING msg_optimize= { STRING_WITH_LEN("optimize") }; > /* Prepare, run and cleanup for mysql_recreate_table() */ > > static bool admin_recreate_table(THD *thd, TABLE_LIST *table_list, > - Recreate_info *recreate_info) > + Recreate_info *recreate_info, > + bool partition_admin, bool table_copy) you don't need to pass partition_admin as an argument, if it's always thd->lex->alter_info.partition_flags & ALTER_PARTITION_ADMIN > { > bool result_code; > DBUG_ENTER("admin_recreate_table"); > @@ -859,20 +867,29 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > /* purecov: end */ > } > > - if (operator_func == &handler::ha_repair && > - !(check_opt->sql_flags & TT_USEFRM)) > + if (operator_func == &handler::ha_repair) > { > handler *file= table->table->file; > int check_old_types= file->check_old_types(); > int check_for_upgrade= file->ha_check_for_upgrade(check_opt); why check_old_types is not part of ha_check_for_upgrade? they're always used together, as far as I can see. > + require_data_conversion= > + (check_old_types == HA_ADMIN_NEEDS_DATA_CONVERSION || > + check_for_upgrade == HA_ADMIN_NEEDS_DATA_CONVERSION); > + require_alter_table= > + (check_old_types == HA_ADMIN_NEEDS_ALTER || > + check_for_upgrade == HA_ADMIN_NEEDS_ALTER); > > - if (check_old_types == HA_ADMIN_NEEDS_ALTER || > - check_for_upgrade == HA_ADMIN_NEEDS_ALTER) > + if (!(check_opt->sql_flags & (TT_USEFRM | TT_FORCE))) > + { > + if (require_data_conversion || require_alter_table) > { > /* We use extra_open_options to be able to open crashed tables */ > thd->open_options|= extra_open_options; > - result_code= admin_recreate_table(thd, table, &recreate_info) ? > + result_code= admin_recreate_table(thd, table, &recreate_info, > + (lex->alter_info.partition_flags & > + ALTER_PARTITION_ADMIN), 1) ? > HA_ADMIN_FAILED : HA_ADMIN_OK; > + recreate_used= 1; > thd->open_options&= ~extra_open_options; > goto send_result; > } > if (check_old_types || check_for_upgrade) > { > /* If repair is not implemented for the engine, run ALTER TABLE */ > need_repair_or_alter= 1; > } > } > - > + } > result_code= compl_result_code= HA_ADMIN_OK; > > if (operator_func == &handler::ha_analyze) > @@ -1229,7 +1277,10 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > *save_next_global= table->next_global; > table->next_local= table->next_global= 0; > > - result_code= admin_recreate_table(thd, table, &recreate_info); > + result_code= admin_recreate_table(thd, table, &recreate_info, > + (lex->alter_info.partition_flags & > + ALTER_PARTITION_ADMIN), 0); > + recreate_used= 1; that's a bit too late, you check recreate_used earlier > trans_commit_stmt(thd); > trans_commit(thd); > close_thread_tables(thd); > @@ -1641,7 +1696,8 @@ bool Sql_cmd_repair_table::execute(THD *thd) > WSREP_TO_ISOLATION_BEGIN_WRTCHK(NULL, NULL, first_table); > res= mysql_admin_table(thd, first_table, &m_lex->check_opt, &msg_repair, > TL_WRITE, 1, > - MY_TEST(m_lex->check_opt.sql_flags & TT_USEFRM), > + MY_TEST(m_lex->check_opt.sql_flags & > + (TT_USEFRM | TT_FORCE)), better rename repair_table_use_frm argument of mysql_admin_table and open_only_one_table, because it's not only TT_USEFRM anymore. a good name could be no_errors_from_open > HA_OPEN_FOR_REPAIR, &prepare_for_repair, > &handler::ha_repair, &view_repair, true); > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 55319a2242f..7d5a968070a 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -10868,19 +10869,56 @@ do_continue:; > } > #endif /* WITH_WSREP */ > > + /* > + ALTER TABLE ... ENGINE to the same engine is a common way to > + request table rebuild. Set ALTER_RECREATE flag to force table > + rebuild. > + */ > + if (new_db_type == old_db_type) > + { > + if (create_info->used_fields & HA_CREATE_USED_ENGINE) > + alter_info->flags|= ALTER_RECREATE; > + > + /* > + Check if we are using ALTER TABLE FORCE without any other options > + (except ENGINE == current_engine). > + In this case we will try to recreate an identical to the original. > + This code is also used with REPAIR and OPTIMIZE. > + */ > + if (alter_info->flags == ALTER_RECREATE && > + ((create_info->used_fields & ~HA_CREATE_USED_ENGINE) == 0)) > + create_info->recreate_identical_table= 1; > + } > + > /* > We can use only copy algorithm if one of the following is true: > + - If the table is from an old MariaDB version and requires data > + modifications. In this case we ignore --alter-algorithm as eh, no, you cannot ignore it, if the user specified incompatible set of options, ALTER TABLE has to fail with an error, saying ALGORITHM=INPLACE (for example) is impossible here. The server cannot just silently ignore an explicit user command. > + as we cannot use any other algorithm than COPY (using other > + algorithms could open up the problem that the table .frm version > + is updated without data transformations and the table would be > + corrupted without any way for MariaDB to notice this during > + check/upgrade). > + This logic ensurses that ALTER TABLE ... FORCE (no other > + options) will always be be able to repair a table structure and > + convert data from any old format. > - In-place is impossible for given operation. > - Changes to partitioning which were not handled by fast_alter_part_table() > needs to be handled using table copying algorithm unless the engine > supports auto-partitioning as such engines can do some changes > using in-place API. > */ > + check_opt.init(); > + > + require_copy_algorithm= (table->file->ha_check_for_upgrade(&check_opt) == > + HA_ADMIN_NEEDS_DATA_CONVERSION); > + if (require_copy_algorithm || > - if (is_inplace_alter_impossible(table, create_info, alter_info) > - || IF_PARTITIONING((partition_changed && > + is_inplace_alter_impossible(table, create_info, alter_info) || > + IF_PARTITIONING((partition_changed && > !(old_db_type->partition_flags() & HA_USE_AUTO_PARTITION)), 0)) > { > - if (alter_info->algorithm_is_nocopy(thd)) > + if (alter_info->algorithm_is_nocopy(thd) && > + !(require_copy_algorithm && alter_info->algorithm_not_specified())) > { > my_error(ER_ALTER_OPERATION_NOT_SUPPORTED, MYF(0), > alter_info->algorithm_clause(thd), "ALGORITHM=COPY"); > @@ -12451,10 +12482,14 @@ bool mysql_recreate_table(THD *thd, TABLE_LIST *table_list, > create_info.init(); > create_info.row_type=ROW_TYPE_NOT_USED; > create_info.alter_info= &alter_info; > + create_info.recreate_identical_table= 1; > /* Force alter table to recreate table */ > alter_info.flags= (ALTER_CHANGE_COLUMN | ALTER_RECREATE); > + alter_info.partition_flags= thd->lex->alter_info.partition_flags; > + if (partition_admin) > + alter_info.partition_flags= ALTER_PARTITION_ADMIN; when is this needed? I couldn't find a single code path where alter_info.partition_flags is not ALTER_PARTITION_ADMIN at this point. Neither code analysys nor mtr have found anything. You can safely remove this if() and add DBUG_ASSERT(alter_info.partition_flags & ALTER_PARTITION_ADMIN); > > - if (table_copy) > + if (table_copy && !partition_admin) > alter_info.set_requested_algorithm( > Alter_info::ALTER_TABLE_ALGORITHM_COPY); > > diff --git a/sql/table.cc b/sql/table.cc > index f84eeda297f..b09e9b11f17 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1846,6 +1846,8 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > */ > if (share->frm_version == FRM_VER_TRUE_VARCHAR -1 && frm_image[33] == 5) > share->frm_version= FRM_VER_TRUE_VARCHAR; > + if (share->frm_version <= FRM_VER_TRUE_VARCHAR) > + share->keep_original_mysql_version= 1; this is wrong. see prepare_frm_header(), frm_version is *always* FRM_VER_TRUE_VARCHAR if there are no vcols. you've basically just disabled update_frm_version() for almost everyone. > > new_field_pack_flag= frm_image[27]; > new_frm_ver= (frm_image[2] - FRM_VER); Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

RISC-V CI for MariaDB
by ali.tariq＠10xengineers.ai 04 Mar '24

04 Mar '24

Hi all, I have seen the status of MariaDB in the RISC-V software ecosystem sheet (https://docs.google.com/spreadsheets/d/10rWj7QlzCiXwTZ778vex5_nRLP9gLrNC8Pw…) as "SUPPORTED" but not yet "FULLY SUPPORTED" or "Optimized" for RISC-V architecture, which means optimization for RISC-V is still in progress. I want to bump a CI platform called Cloud-V (https://cloud-v.co/) (created in collaboration with the RISC-V Labs subgroup) which provides CI for RISC-V with emulated and physical RISC-V computes. You can enroll for access at this link (https://cloud-v.co/contactus)

1 0

Storing binlig in InnoDB/engine
by Kristian Nielsen 28 Feb '24

28 Feb '24

Hi Marko, I looked a bit more at the idea that we have discussed a couple times, of storing the binlog in an InnoDB tablespace to avoid the need for two-phase commit between binlog and InnoDB and save one (or potentially both) fsync()s during a commit. I managed to get some InnoDB code written that is able to create a tablespace and write to it, patch below or on github: https://github.com/MariaDB/server/commits/knielsen_binlog_in_engine https://github.com/MariaDB/server/commit/95958c3842ebf0f7e358d6c3f51b887bd9… This is far from a complete patch, just an exercise for me on how InnoDB tablespaces and mini-transactions work in detail. But it turned out to be useful for me to get started and find a few questions to ask. My general approach to writing is to use fsp_page_create() for the first write to a page, and then buf_page_get_gen() for subsequent writes. But maybe this should be refined for the actual implementation. I'm thinking if perhaps fsp_page_create() does too much, you mentioned earlier that some parts of the page header could be simplified/omitted. And maybe there is a way to pin the current page in the buffer pool so buf_page_get_gen() is not needed for every write? I'm currently passing RW_SX_LATCH to buf_page_get_gen() (otherwise I got an assertion when writing). I'm not sure though how these latches work, or if binlog writing would need such latches; maybe it makes more sense to have a simple mutex protecting page access? Another assertion was fixed by doing mtr.set_named_space() before writing. Again, I'm not sure what this does exactly or if it's appropriate? I tried in this patch to reserve 2 "special" tablespace ids for the binlog tablespaces. Idea would be to cycle between them, keeping at most the two last tablespaces active. But do the tablespace IDs appear in the redo log and used for recovery? In that case, I assume that all binlog tablespaces written since the last InnoDB checkpoint will need a unique tablespace ID? So maybe 2 is too few. I was thinking maybe the binlog could allocate new tablespace IDs as necessary, but re-use them after each InnoDB checkpoint. This would avoid wasting ids and eventually hitting the 2**32 limit. - Kristian. commit 95958c3842ebf0f7e358d6c3f51b887bd9948845 (HEAD -> binlog_in_inno, origin/knielsen_binlog_in_engine) Author: Kristian Nielsen <knielsen(a)knielsen-hq.org> Date: Sun Feb 25 17:41:50 2024 +0100 Binlog in Engine: Very first sketch, able to create and write an InnoDB tablespace Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> diff --git a/mysql-test/suite/binlog/t/binlog_in_engine.test b/mysql-test/suite/binlog/t/binlog_in_engine.test new file mode 100644 index 00000000000..947139c9bcc --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_in_engine.test @@ -0,0 +1,11 @@ +--source include/have_innodb.inc +--source include/have_binlog_format_mixed.inc + +CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1); +BEGIN; +INSERT INTO t1 VALUES (2); +INSERT INTO t1 VALUES (3); +COMMIT; +SELECT * FROM t1 ORDER BY a; +DROP TABLE t1; diff --git a/storage/innobase/buf/buf0flu.cc b/storage/innobase/buf/buf0flu.cc index bd43429eb5d..09d8fdaff39 100644 --- a/storage/innobase/buf/buf0flu.cc +++ b/storage/innobase/buf/buf0flu.cc @@ -1626,7 +1626,8 @@ static ulint buf_flush_list(ulint max_n= ULINT_UNDEFINED, bool buf_flush_list_space(fil_space_t *space, ulint *n_flushed) { const auto space_id= space->id; - ut_ad(space_id <= SRV_SPACE_ID_UPPER_BOUND); + ut_ad(space_id <= SRV_SPACE_ID_UPPER_BOUND || + space_id == SRV_SPACE_ID_BINLOG0 || space_id == SRV_SPACE_ID_BINLOG1); bool may_have_skipped= false; ulint max_n_flush= srv_io_capacity; diff --git a/storage/innobase/fil/fil0fil.cc b/storage/innobase/fil/fil0fil.cc index 0ce54df6574..e83225a4883 100644 --- a/storage/innobase/fil/fil0fil.cc +++ b/storage/innobase/fil/fil0fil.cc @@ -184,7 +184,7 @@ it is an absolute path. */ const char* fil_path_to_mysql_datadir; /** Common InnoDB file extensions */ -const char* dot_ext[] = { "", ".ibd", ".isl", ".cfg" }; +const char* dot_ext[] = { "", ".ibd", ".isl", ".cfg", ".ibb" }; /** Number of pending tablespace flushes */ Atomic_counter<ulint> fil_n_pending_tablespace_flushes; @@ -1044,6 +1044,9 @@ fil_space_t *fil_space_t::create(uint32_t id, uint32_t flags, if (UNIV_LIKELY(id <= fil_system.max_assigned_id)) { break; } + if (id == SRV_SPACE_ID_BINLOG0 || id == SRV_SPACE_ID_BINLOG1) { + break; + } if (UNIV_UNLIKELY(srv_operation == SRV_OPERATION_BACKUP)) { break; } @@ -1603,9 +1606,10 @@ inline void mtr_t::log_file_op(mfile_type_t type, uint32_t space_id, ut_ad(!(byte(type) & 15)); /* fil_name_parse() requires that there be at least one path - separator and that the file path end with ".ibd". */ + separator and that the file path end with ".ibd" or "ibb". */ ut_ad(strchr(path, '/')); - ut_ad(!strcmp(&path[strlen(path) - strlen(DOT_IBD)], DOT_IBD)); + ut_ad(!strcmp(&path[strlen(path) - strlen(DOT_IBD)], DOT_IBD) || + !strcmp(&path[strlen(path) - strlen(DOT_IBB)], DOT_IBB)); m_modifications= true; if (!is_logged()) diff --git a/storage/innobase/fsp/fsp0fsp.cc b/storage/innobase/fsp/fsp0fsp.cc index 787bda53895..f1bb42b3a15 100644 --- a/storage/innobase/fsp/fsp0fsp.cc +++ b/storage/innobase/fsp/fsp0fsp.cc @@ -3763,3 +3763,129 @@ void fsp_shrink_temp_space() mtr.commit(); sql_print_information("InnoDB: Temporary tablespace truncated successfully"); } + + + +fil_space_t* binlog_space; +buf_block_t *binlog_cur_block; +uint32_t binlog_cur_page_no; +uint32_t binlog_cur_page_offset; + +/** Create a binlog tablespace file +@param[in] name file name +@return DB_SUCCESS or error code */ +dberr_t fsp_binlog_tablespace_create(const char* name) +{ + pfs_os_file_t fh; + bool ret; + + uint32_t size= (1<<20) >> srv_page_size_shift /* ToDo --max-binlog-size */; + if(srv_read_only_mode) + return DB_ERROR; + + os_file_create_subdirs_if_needed(name); + + /* ToDo: Do we need here an mtr.log_file_op(FILE_CREATE) like in fil_ibd_create(()? */ + fh = os_file_create( + innodb_data_file_key, + name, + OS_FILE_CREATE | OS_FILE_ON_ERROR_NO_EXIT, + OS_FILE_AIO, OS_DATA_FILE, srv_read_only_mode, &ret); + + if (!ret) { + os_file_close(fh); + return DB_ERROR; + } + + /* ToDo: Enryption? */ + fil_encryption_t mode= FIL_ENCRYPTION_OFF; + fil_space_crypt_t* crypt_data= nullptr; + + /* We created the binlog file and now write it full of zeros */ + if (!os_file_set_size(name, fh, + os_offset_t{size} << srv_page_size_shift)) { + ib::error() << "Unable to allocate " << name; + os_file_close(fh); + os_file_delete(innodb_data_file_key, name); + return DB_ERROR; + } + + mysql_mutex_lock(&fil_system.mutex); + uint32_t space_id= SRV_SPACE_ID_BINLOG0; + if (!(binlog_space= fil_space_t::create(space_id, + ( FSP_FLAGS_FCRC32_MASK_MARKER | + FSP_FLAGS_FCRC32_PAGE_SSIZE()), + FIL_TYPE_TABLESPACE, crypt_data, + mode, true))) { + mysql_mutex_unlock(&fil_system.mutex); + return DB_ERROR; + } + + fil_node_t* node = binlog_space->add(name, fh, size, false, true); + IF_WIN(node->find_metadata(), node->find_metadata(fh, true)); + mysql_mutex_unlock(&fil_system.mutex); + + binlog_cur_page_no= 0; + binlog_cur_page_offset= FIL_PAGE_DATA; + return DB_SUCCESS; +} + +void fsp_binlog_write_start(uint32_t page_no, + const uchar *data, uint32_t len, mtr_t *mtr) +{ + buf_block_t *block= fsp_page_create(binlog_space, page_no, mtr); + mtr->memcpy<mtr_t::MAYBE_NOP>(*block, FIL_PAGE_DATA + block->page.frame, + data, len); + binlog_cur_block= block; +} + +void fsp_binlog_write_offset(uint32_t page_no, uint32_t offset, + const uchar *data, uint32_t len, mtr_t *mtr) +{ + dberr_t err; + /* ToDo: Is RW_SX_LATCH appropriate here? */ + buf_block_t *block= buf_page_get_gen(page_id_t{binlog_space->id, page_no}, + 0, RW_SX_LATCH, binlog_cur_block, + BUF_GET, mtr, &err); + ut_a(err == DB_SUCCESS); + mtr->memcpy<mtr_t::MAYBE_NOP>(*block, + offset + block->page.frame, + data, len); +} + +void fsp_binlog_append(const uchar *data, uint32_t len, mtr_t *mtr) +{ + ut_ad(binlog_cur_page_offset <= srv_page_size - FIL_PAGE_DATA_END); + uint32_t remain= ((uint32_t)srv_page_size - FIL_PAGE_DATA_END) - + binlog_cur_page_offset; + // ToDo: Some kind of mutex to protect binlog access. + while (len > 0) { + if (remain < 4) { + binlog_cur_page_offset= FIL_PAGE_DATA; + remain= ((uint32_t)srv_page_size - FIL_PAGE_DATA_END) - + binlog_cur_page_offset; + ++binlog_cur_page_no; + } + uint32_t this_len= std::min<uint32_t>(len, remain); + if (binlog_cur_page_offset == FIL_PAGE_DATA) + fsp_binlog_write_start(binlog_cur_page_no, data, this_len, mtr); + else + fsp_binlog_write_offset(binlog_cur_page_no, binlog_cur_page_offset, + data, this_len, mtr); + len-= this_len; + data+= this_len; + binlog_cur_page_offset+= this_len; + } +} + + +void fsp_binlog_test(const uchar *data, uint32_t len) +{ + mtr_t mtr; + mtr.start(); + if (!binlog_space) + fsp_binlog_tablespace_create("./binlog-000000.ibb"); + mtr.set_named_space(binlog_space); + fsp_binlog_append(data, len, &mtr); + mtr.commit(); +} diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index 93127bb1c3a..df2bd07d2dc 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -4481,6 +4481,10 @@ innobase_commit( if (commit_trx || (!thd_test_options(thd, OPTION_NOT_AUTOCOMMIT | OPTION_BEGIN))) { + /* ToDo: This is just a random very initial test of writing + something into a binlog tablespace. */ + if (!opt_bootstrap) + fsp_binlog_test((const uchar *)"Hulubulu!!?!", 12); /* Run the fast part of commit if we did not already. */ if (!trx->active_commit_ordered) { innobase_commit_ordered_2(trx, thd); diff --git a/storage/innobase/include/fil0fil.h b/storage/innobase/include/fil0fil.h index f3660eff7c6..17b35f2f892 100644 --- a/storage/innobase/include/fil0fil.h +++ b/storage/innobase/include/fil0fil.h @@ -1129,10 +1129,12 @@ enum ib_extention { NO_EXT = 0, IBD = 1, ISL = 2, - CFG = 3 + CFG = 3, + IBB = 4 }; extern const char* dot_ext[]; #define DOT_IBD dot_ext[IBD] +#define DOT_IBB dot_ext[IBB] #define DOT_ISL dot_ext[ISL] #define DOT_CFG dot_ext[CFG] diff --git a/storage/innobase/include/fsp0fsp.h b/storage/innobase/include/fsp0fsp.h index ddc45e53fe6..26a45518ba2 100644 --- a/storage/innobase/include/fsp0fsp.h +++ b/storage/innobase/include/fsp0fsp.h @@ -579,6 +579,8 @@ void fsp_system_tablespace_truncate(); /** Truncate the temporary tablespace */ void fsp_shrink_temp_space(); +extern void fsp_binlog_test(const uchar *data, uint32_t len); + #ifndef UNIV_DEBUG # define fsp_init_file_page(space, block, mtr) fsp_init_file_page(block, mtr) #endif diff --git a/storage/innobase/include/fsp0types.h b/storage/innobase/include/fsp0types.h index 757ead55d03..e3d45796190 100644 --- a/storage/innobase/include/fsp0types.h +++ b/storage/innobase/include/fsp0types.h @@ -27,8 +27,12 @@ Created May 26, 2009 Vasil Dimov #pragma once #include "ut0byte.h" -/** All persistent tablespaces have a smaller fil_space_t::id than this. */ +/** All persistent tablespaces (except binlog tablespaces) have a smaller +fil_space_t::id than this. */ constexpr uint32_t SRV_SPACE_ID_UPPER_BOUND= 0xFFFFFFF0U; +/** Binlog tablespaces. */ +constexpr uint32_t SRV_SPACE_ID_BINLOG0 = SRV_SPACE_ID_UPPER_BOUND + 1; +constexpr uint32_t SRV_SPACE_ID_BINLOG1 = SRV_SPACE_ID_UPPER_BOUND + 2; /** The fil_space_t::id of the innodb_temporary tablespace. */ constexpr uint32_t SRV_TMP_SPACE_ID= 0xFFFFFFFEU; diff --git a/storage/innobase/include/mtr0log.h b/storage/innobase/include/mtr0log.h index e2419309764..86f6e3794f6 100644 --- a/storage/innobase/include/mtr0log.h +++ b/storage/innobase/include/mtr0log.h @@ -25,7 +25,7 @@ Mini-transaction log record encoding and decoding #include "mtr0mtr.h" /** The smallest invalid page identifier for persistent tablespaces */ -constexpr page_id_t end_page_id{SRV_SPACE_ID_UPPER_BOUND, 0}; +constexpr page_id_t end_page_id{SRV_SPACE_ID_BINLOG1, 0}; /** The minimum 2-byte integer (0b10xxxxxx xxxxxxxx) */ constexpr uint32_t MIN_2BYTE= 1 << 7; diff --git a/storage/innobase/log/log0recv.cc b/storage/innobase/log/log0recv.cc index ef31a4d00c1..310acb73071 100644 --- a/storage/innobase/log/log0recv.cc +++ b/storage/innobase/log/log0recv.cc @@ -2973,7 +2973,8 @@ recv_sys_t::parse_mtr_result recv_sys_t::parse(source &l, bool if_exists) if (is_predefined_tablespace(space_id)) goto file_rec_error; - if (fnend - fn < 4 || memcmp(fnend - 4, DOT_IBD, 4)) + if (fnend - fn < 4 || + (memcmp(fnend - 4, DOT_IBD, 4) && memcmp(fnend - 4, DOT_IBB, 4))) goto file_rec_error; if (UNIV_UNLIKELY(!recv_needed_recovery && srv_read_only_mode))

2 5

[PATCH 0/2] Suggestion for smaller fix to XA parallel replication performance, MDEV-31949
by Kristian Nielsen 28 Feb '24

28 Feb '24

As was discussed recently off-list, there is a desire to find a minimal fix to the performance issues in 10.6 with parallel replication of user XA due to non-optimal scheduling (MDEV-31949). How about something like this two-patch series? It _only_ modifies the scheduling of event groups to worker threads, and only for XA transactions, and is quite a bit simpler than the other proposed patches for MDEV-31949. The current 10.6 code for parallel replication of user XA needs to avoid applying in parallel two different event groups that have the same XA XID. Typically the XA PREPARE and corresponding XA COMMIT/ROLLBACK, but could also be another transaction that uses a duplicate XID. The method currently used in 10.6 takes a hash of the XID and assigns the transaction to worker (hash MOD N) amongst the N workers. This is sub-optimal, since there is a high chance that amonst N transactions, several of them will map to the same worker. Thus less than N transactions will be able to replicate in parallel and group commit together. The second patch in the series improves this by keeping track of recently active XIDs and which worker they were scheduled on. A subsequent event group with the same XID is scheduled on the same worker. If the XID is different, the event group is scheduled normally on the next free worker in round-robin fashion. This permits most XA PREPARE to be scheduled without dependency restrictions, same as normal non-XA transactions. Only if the user submits multiple XA transactions close together using the same duplicate XID will there be dependency restrictions. The XA COMMIT will normally be scheduled on the same worker as the XA PREPARE, unless the two events are far apart in the replication stream. This is mostly unavoidable in the current XA replication design, since the XA PREPARE and XA COMMIT of a single transaction cannot group-commit together. The first patch is a refactor/preparation patch. It rewrites the scheduling to use an explicit FIFO for scheduling the threads instead of a simple `i := (i+1) % N` cyclic counter. This allows to combine explicit scheduling of some transactions with round-robin scheduling of the rest in the second patch. Patches also available on github: https://github.com/MariaDB/server/commits/knielsen_xa_sched_minimal_fix https://github.com/MariaDB/server/commit/66d6cce96f831b638812490844d7542317… https://github.com/MariaDB/server/commit/3a32fb546c111e4627cad7c66bcc089bc1… This is just a quick proof-of-concept I did today, but it seems to work, and is small and simple and suitable for targeted fixing and cherry-picking. Brandon and/or Andrei, maybe you could try your benchmarks with this patch and see if it also solves the performance issues you were seeing? Or if there's something else required for XA that I'm missing? - Kristian. Kristian Nielsen (2): Refactor parallel replication round-robin scheduling to use explicit FIFO More precise dependency tracking of XA XID in parallel replication sql/rpl_parallel.cc | 158 ++++++++++++++++++++++++++++++++++++++------ sql/rpl_parallel.h | 48 +++++++++++++- 2 files changed, 181 insertions(+), 25 deletions(-) -- 2.30.2

1 3

Fwd: 2342aaf46eb: MDEV-30973 Implement debug logging for slave_transaction_retries
by Aleksey Midenkov 28 Feb '24

28 Feb '24

Hi Sergei, the commits are in bb-11.4-midenok-MDEV-30973 New changes are in 4 top commits. There is also a small CMake change "Columnstore empty submodule fix" which I would push ASAP. On Tue, Nov 7, 2023 at 8:35 PM Sergei Golubchik <serg(a)mariadb.org> wrote: > > Hi, Aleksey, > > On Nov 07, Aleksey Midenkov wrote: > > revision-id: 2342aaf46eb (mariadb-11.0.1-256-g2342aaf46eb) > > parent(s): 80bc5df1c80 > > author: Aleksey Midenkov > > committer: Aleksey Midenkov > > timestamp: 2023-10-10 13:11:19 +0300 > > message: > > > > MDEV-30973 Implement debug logging for slave_transaction_retries > > > > Configuration options: > > > > log_slave_retries > > log_slave_retries_max > > 1. > a commit comment of >100 lines that's a bit too much, I think. > > 2. > comments in MDEV-30973 say that one should be able to have everything in > one file, namely in the error log. I don't see how it's possible in your > commit. Fixed. > > 3. > one of the reasons for not simply writing into erorr log was > "you get what are relatively large entries in there which can make it > hard/annoying to also look for the normal, one-line warning and error > output". But as far as I can see your slave retry entries are not large, > but normal one-line messages. I'd say they can fit into the error log > just fine. "large entries" here is not fully correct. "large number of entries" is more correct. > > I've still commented below in the assumption that the separate log > implementation will stay. But I honestly don't see a reason for it > anymore. > > > diff --git a/mysql-test/suite/rpl/t/slave_deadlock.test b/mysql-test/suite/rpl/t/slave_deadlock.test > > new file mode 100644 > > index 00000000000..9fcda7c9b71 > > --- /dev/null > > +++ b/mysql-test/suite/rpl/t/slave_deadlock.test > > @@ -0,0 +1,194 @@ > > +--source include/have_debug.inc > > +--source include/have_innodb.inc > > +--source include/have_binlog_format_statement.inc > > + > > +# On --repeat test infinitely increments thread_id and finally fails --sync_with_master at log_pos 1754: > > +# > > +# 231009 20:58:15 server id 1 end_log_pos 1754 CRC32 0x7e5aecd1 Query thread_id=561 exec_time=0 error_code=0 xid=0 > > I'm sorry, I don't understand that. Could you elaborate please? I don't know, I've just shown what I observed. Some replication bug? I tried to retest and couldn't reproduce. Removed. > > > +# > > +# force_restart seems to fix this > > + > > +--source include/force_restart.inc > > + > > +# Note that test requires big enough --debug-sync-timeout unfair scheduling at high load. > > Sorry, cannot parse that. Could you elaborate please? I missed *due to* unfair scheduling at high load. That is: thread scheduling latency may reach up to 1 hour. It is executed on VM and actually VM is not always good for concurrency testing (this is quite an omission in our CI system). > > > +# slave_deadlock.opt sets this to 1 hour > > + > > +--let $rpl_topology=1->2 > > +--source include/rpl_init.inc > > + > > +--echo # > > +--echo # MDEV-30973 Enhance output of innodb deadlocks (SAMU-124, SAMU-131) > > +--echo # > > +--connection server_2 > > +SET @old_dbug= @@global.debug_dbug; > > +set @old_parallel_threads= @@global.slave_parallel_threads; > > +set @old_domain_parallel_threads= @@global.slave_domain_parallel_threads; > > +set @old_parallel_mode= @@global.slave_parallel_mode; > > +set @old_slave_transaction_retries= @@global.slave_transaction_retries; > > +set @old_innodb_lock_wait_timeout= @@global.innodb_lock_wait_timeout; > > +set @old_log_slave_retries= @@global.log_slave_retries; > > +set @old_log_slave_retries_file= @@global.log_slave_retries_file; > > +set @old_log_slave_retries_max= @@global.log_slave_retries_max; > > All this makes little sense, it's simpler to put all settings into the .opt > file > > generally it's better to do your way, to have less > server restarts, but you have an .opt file already, you can as well > use it. > > except if you decide to remove the .opt file - then nevermind, the > above it good. I'm taking it for future. > > > +--source include/stop_slave.inc > > +change master to master_use_gtid= slave_pos; > > +set global slave_parallel_threads= 10; > > +set global slave_domain_parallel_threads= 1; > > +set global slave_parallel_mode= optimistic; > > +set global slave_transaction_retries= 4; > > +set global innodb_lock_wait_timeout= 1; > ... > > diff --git a/mysql-test/suite/versioning/r/rpl_row.result b/mysql-test/suite/versioning/r/rpl_row.result > > index c161f68d0bd..6343fbecdec 100644 > > --- a/mysql-test/suite/versioning/r/rpl_row.result > > +++ b/mysql-test/suite/versioning/r/rpl_row.result > > @@ -37,12 +37,7 @@ select *, unix_timestamp(row_start) as row_start, unix_timestamp(row_end) as row > > x y row_start row_end > > 1 1 12345.000000 2147483647.999999 > > set timestamp= default; > > -### INSERT INTO `test`.`t1` > > -### SET > > -### @1=1 > > -### @2=1 > > -### @3=12345.000000 > > -### @4=2147483647.999999 > > why? That was a bug in grep.inc (fixed). I guess this commit was temporary which I forgot to deal with. > > > +/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/; > > connection slave; > > select * from t1; > > x y > > diff --git a/sql/log.cc b/sql/log.cc > > index 877315c60de..78e874fda5b 100644 > > --- a/sql/log.cc > > +++ b/sql/log.cc > > @@ -1188,9 +1188,17 @@ bool Log_to_file_event_handler:: > > log_error(enum loglevel level, const char *format, > > va_list args) > > { > > - return vprint_msg_to_log(level, format, args); > > + return vprint_msg_to_log(stderr, level, format, args); > > } > > > > +#ifdef HAVE_REPLICATION > > +bool Log_to_file_event_handler::log_slave_retry(const char *format, va_list args) > > +{ > > + DBUG_ASSERT(slave_retries_file.get()); > > + return vprint_msg_to_log(slave_retries_file.get(), NO_LEVEL, format, args); > > I don't think you should use vprint_msg_to_log and add a NO_LEVEL hack. > vprint_msg_to_log also prints into the windows event log, into > wsrep json log, and it locks LOCK_error_log. You don't need any of that, > just create your own function, it's just one fprintf() anyway. I still think formatting log message in single function is a good idea. Any other above deficiencies were fixed. > > > +} > > +#endif > > + > > void Log_to_file_event_handler::init_pthread_objects() > > { > > mysql_log.init_pthread_objects(); > > @@ -9752,13 +9787,15 @@ static void print_buffer_to_file(enum loglevel level, const char *buffer, > > } > > } > > > > +#ifndef EMBEDDED_LIBRARY > > mysql_mutex_lock(&LOCK_error_log); > > +#endif > > > > skr= my_time(0); > > localtime_r(&skr, &tm_tmp); > > start=&tm_tmp; > > > > - fprintf(stderr, "%d-%02d-%02d %2d:%02d:%02d %lu [%s] %.*s%.*s\n", > > + fprintf(file, "%d-%02d-%02d %2d:%02d:%02d [T%lu] %s%.*s%.*s\n", > > eh... no. don't change the format of existing log files, please. This change is because of a serious deficiency in logging format: you cannot grep by thread number (with simple expression) and grepping by thread number is important. I understand this may cause incompatibilities but I hope the drawbacks are less than the advantage. What if we keep it? > > > start->tm_year + 1900, > > start->tm_mon+1, > > start->tm_mday, > > @@ -9834,6 +9872,22 @@ void sql_print_error(const char *format, ...) > > } > > > > > > +#ifdef HAVE_REPLICATION > > +void sql_print_error2(const char *format, ...) > > s/sql_print_error2/print_slave_error/ ? Fixed (slave_print_error). > > > +{ > > + va_list args; > > + DBUG_ENTER("sql_print_error"); > > + > > + va_start(args, format); > > + error_log_print(ERROR_LEVEL, format, args); > > + slave_retries_print(format, args); > > + va_end(args); > > + > > + DBUG_VOID_RETURN; > > +} > > +#endif > > + > > + > > void sql_print_warning(const char *format, ...) > > { > > va_list args; > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > > index 8cc3ad51ee9..4f6b2273597 100644 > > --- a/sql/mysqld.cc > > +++ b/sql/mysqld.cc > > @@ -765,6 +765,90 @@ char *opt_relay_logname = 0, *opt_relaylog_index_name=0; > > char *opt_logname, *opt_slow_logname, *opt_bin_logname; > > char *opt_binlog_index_name=0; > > > > +#ifdef HAVE_REPLICATION > > +char default_slave_retries_path[FN_REFLEN]; > > +bool opt_slave_retries= false; > > +char *opt_slave_retries_path= NULL; > > +uint opt_slave_retries_max_log= 0; > > +#endif > > +Slave_retries_file slave_retries_file; > > + > > +bool Slave_retries_file::open() > > +{ > > +#ifdef HAVE_REPLICATION > > + DBUG_ASSERT(opt_slave_retries); > > + DBUG_ASSERT(opt_slave_retries_path); > > + file= fopen(opt_slave_retries_path, "a"); > > my_fopen or my_open > > and anyway, why a new class, why not to use MYSQL_LOG like general and slow > logs do? I rewrote the code via MYSQL_LOG. > > > + if (!file) > > + { > > + sql_print_error("Open of log_slave_retries '%s' failed: %s (%d)", > > + opt_slave_retries_path, strerror(errno), errno); > > This is inconsistent, the server doesn't print this for other log files. > > I've just tried with general log and got > > 2023-11-06 17:58:19 0 [ERROR] mysqld: File '/tmp' not found (Errcode: 21 "Is a directory") > 2023-11-06 17:58:19 0 [ERROR] Could not use /tmp for logging (error 21). Turning logging off for the whole duration of the MariaDB server process. To turn it on again: fix the cause, shutdown the MariaDB server and restart it. > > Please, do the same Looks like wrong behavior: you supplied wrong arguments and the server was started anyway! What is the justification for this? It is easy to miss: if you don't read the error log, you don't even know something is wrong. And if you got lucky to notice that, the server is already up and running (and loaded with clients). Anyway since my logging now works like general log it now works the same. > > > + return true; > > + } > > + else > > + { > > + slave_retries_print("%s (mariadbd %s) starting as process %lu ...", > > + my_progname, server_version, (ulong) getpid()); > > Any reason, why it shouldn't be the same as in the error log? > > sql_print_information("Starting MariaDB %s source revision %s as process %lu", > server_version, SOURCE_REVISION, (ulong) getpid()); > I think it would be nice to add my_progname into the message. Made it now like in the original code. > > + } > > +#endif > > + return false; > > +} > > + > > +void Slave_retries_file::close() > > +{ > > +#ifdef HAVE_REPLICATION > > + if (file) > > + { > > + acquire(); > > + fclose(file); > > + release(); > > + } > > + file= NULL; > > +#endif > > +} > > + > > +void Slave_retries_file::init() > > +{ > > +#ifdef HAVE_REPLICATION > > + mysql_mutex_init(key_LOCK_global_system_variables, &mutex, MY_MUTEX_INIT_FAST); > > +#endif > > +} > > + > > +void Slave_retries_file::destroy() > > +{ > > +#ifdef HAVE_REPLICATION > > + mysql_mutex_destroy(&mutex); > > +#endif > > +} > > + > > +bool Slave_retries_file::acquire() > > +{ > > +#ifdef HAVE_REPLICATION > > + if (!file) > > + return false; > > + mysql_mutex_lock(&mutex); > > + return true; > > +#else > > + return false; > > +#endif > > +} > > + > > +void Slave_retries_file::release() > > +{ > > +#ifdef HAVE_REPLICATION > > + mysql_mutex_unlock(&mutex); > > +#endif > > +} > > + > > +void Slave_retries_file::flush() > > +{ > > + if (acquire()) > > + { > > + fflush(file); > > + release(); > > + } > > +} > > + > > /* Static variables */ > > > > my_bool opt_stack_trace; > > @@ -5055,6 +5145,35 @@ static int init_server_components() > > "this server. However this will be ignored as the " > > "--log-bin option is not defined."); > > } > > + > > + slave_retries_file.init(); > > + if (!log_error_file_ptr[0]) > > + fn_format(default_slave_retries_path, pidfile_name, mysql_data_home, > > + "-retries.err", MY_REPLACE_EXT); > > + else > > + fn_format(default_slave_retries_path, log_error_file_ptr, mysql_data_home, > > + "-retries.err", MY_REPLACE_EXT); > > that's not how all other log names work, > grep mysqld.cc for opt_slow_logname Now it works like other logs. > > > + if (opt_slave_retries_path && !opt_abort) > > + { > > + if (!opt_slave_retries_path[0]) > > + { > > + if (!default_slave_retries_path[0]) > > + { > > + /* fn_format failed */ > > + opt_slave_retries_path= NULL; > > + opt_slave_retries= false; > > + } > > + else > > + { > > + opt_slave_retries_path= my_strdup(key_memory_Sys_var_charptr_value, > > + default_slave_retries_path, MYF(0)); > > + } > > + } > > + > > + if (opt_slave_retries && > > + slave_retries_file.open()) > > + unireg_abort(1); > > + } > > #endif > > > > if (opt_bin_log) > > diff --git a/sql/mysqld.h b/sql/mysqld.h > > index 69cfc0cad56..5f1561ca10d 100644 > > --- a/sql/mysqld.h > > +++ b/sql/mysqld.h > > @@ -822,6 +848,8 @@ enum options_mysqld > > OPT_KEY_CACHE_CHANGED_BLOCKS_HASH_SIZE, > > OPT_LOG_BASENAME, > > OPT_LOG_ERROR, > > + OPT_SLAVE_RETRIES, > > + OPT_SLAVE_RETRIES_LOG, > > why do you need them? Removed. > > > OPT_LOWER_CASE_TABLE_NAMES, > > OPT_PLUGIN_LOAD, > > OPT_PLUGIN_LOAD_ADD, > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > > index d742bc6c1c3..c53ca0f0097 100644 > > --- a/sql/sql_show.cc > > +++ b/sql/sql_show.cc > > @@ -10662,6 +10663,15 @@ char *thd_get_error_context_description(THD *thd, char *buffer, > > /* Don't wait if LOCK_thd_data is used as this could cause a deadlock */ > > if (!mysql_mutex_trylock(&thd->LOCK_thd_data)) > > { > > + if (thd->rgi_slave) > > is rgi_slave and its content protected by thd->LOCK_thd_data ? AFAICS they are per-thread and don't require protection. > > > + { > > + rpl_group_info *rgi= thd->rgi_slave; > > + len= my_snprintf(header, sizeof(header), > > + ", GTID %u-%u-%llu", rgi->current_gtid.domain_id, > > + rgi->current_gtid.server_id, rgi->current_gtid.seq_no); > > + str.append(header, len); > > + } > > + > > if (const char *info= thread_state_info(thd)) > > { > > str.append(' '); > > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > > index ebb64b340b0..11c321f06dc 100644 > > --- a/sql/sys_vars.cc > > +++ b/sql/sys_vars.cc > > @@ -5862,6 +5862,61 @@ Sys_slave_trans_retry_interval( > > "slave_transaction_retry_errors", > > GLOBAL_VAR(slave_trans_retry_interval), CMD_LINE(REQUIRED_ARG), > > VALID_RANGE(0, 3600), DEFAULT(0), BLOCK_SIZE(1)); > > + > > +static bool update_slave_retries_log(sys_var *self, THD* thd, enum_var_type type) > > +{ > > + if (opt_slave_retries) > > + { > > + if (!slave_retries_file.get() && slave_retries_file.open()) > > + { > > + my_error(ER_CANT_OPEN_FILE, MYF(0), opt_slave_retries_path, errno); > > + opt_slave_retries= false; > > + return true; > > + } > > + } > > + else > > + slave_retries_file.close(); > > + return false; > > +} > > + > > +static Sys_var_mybool Sys_slave_retries_log( > > + "log_slave_retries", "Log transaction retries", > > + GLOBAL_VAR(opt_slave_retries), > > + CMD_LINE(OPT_ARG, OPT_SLAVE_RETRIES), DEFAULT(false), > > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0), ON_UPDATE(update_slave_retries_log)); > > + > > + > > +static bool check_slave_retries_file(sys_var *self, THD *thd, set_var *var) > > +{ > > + if (opt_slave_retries) > > + { > > + my_error(ER_NOT_SUPPORTED_YET, MYF(0), "modifying while log_slave_retries is ON"); > > as always, please don't hard-code error messages. > you can use > > my_error(ER_OPTION_PREVENTS_STATEMENT, MYF(0), "log_slave_retries=ON"); Fixed. > > > + return true; > > + } > > + return false; > > +} > > + > > +static bool update_slave_retries_file(sys_var *self, THD* thd, enum_var_type type) > > +{ > > + if (!opt_slave_retries_path) > > + opt_slave_retries_path= my_strdup(key_memory_Sys_var_charptr_value, > > + default_slave_retries_path, MYF(0)); > > sys_var.cc can handle that for you, see PREALLOCATED > in e.g. opt_slow_logname Yes. > > also, you don't reopen the log when its name changes, how comes? It reopens. > > > + return false; > > +} > > + > > +static Sys_var_charptr Sys_slave_retries_log_path( > > + "log_slave_retries_file", "Log file path for transaction retries", > > + GLOBAL_VAR(opt_slave_retries_path), > > + CMD_LINE(REQUIRED_ARG, OPT_SLAVE_RETRIES_LOG), DEFAULT(NULL), > > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(check_slave_retries_file), > > + ON_UPDATE(update_slave_retries_file)); > > + > > + > > +static Sys_var_uint Sys_slave_retries_log_max( > > + "log_slave_retries_max", "Log max N transaction retries " > > + "(default 0 means infinity)", > > just "(0 means infinity)". Default is printed separately > Fixed. > > + GLOBAL_VAR(opt_slave_retries_max_log), CMD_LINE(REQUIRED_ARG), > > + VALID_RANGE(0, UINT_MAX), DEFAULT(0), BLOCK_SIZE(1)); > > #endif > > > > static bool check_locale(sys_var *self, THD *thd, set_var *var) > > diff --git a/storage/innobase/lock/lock0lock.cc b/storage/innobase/lock/lock0lock.cc > > index 0e9cef669d4..8a8eeaf39f6 100644 > > --- a/storage/innobase/lock/lock0lock.cc > > +++ b/storage/innobase/lock/lock0lock.cc > > @@ -49,6 +49,7 @@ Created 5/7/1996 Heikki Tuuri > > #include <debug_sync.h> > > > > #include <set> > > +#include <stdio.h> > > I presume Marko has reviewed (or will review) InnoDB changes I didn't receive his explicit confirmation. > > > > > #ifdef WITH_WSREP > > #include <mysql/service_wsrep.h> > > diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result > > index 77d6d10b3e6..274c034d455 100644 > > --- a/mysql-test/main/mysqld--help.result > > +++ b/mysql-test/main/mysqld--help.result > > @@ -1673,6 +1678,9 @@ log-isam myisam.log > > log-output FILE > > log-queries-not-using-indexes FALSE > > log-short-format FALSE > > +log-slave-retries FALSE > > +log-slave-retries-file (No default value) > > this is another sign that log-slave-retries-file doesn't behave like > other log filename variables. > > > +log-slave-retries-max 0 > > log-slave-updates FALSE > > log-slow-admin-statements TRUE > > log-slow-disabled-statements sp > > Regards, > Sergei > Chief Architect, MariaDB Server > and security(a)mariadb.org Thanks! -- @midenok

1 0

Re: f22662c36b1: MDEV-33442 REPAIR TABLE corrupts UUIDs
by Sergei Golubchik 26 Feb '24

26 Feb '24

Hi, Alexander, On Feb 24, Alexander Barkov wrote: > revision-id: f22662c36b1 (mariadb-10.11.7-6-gf22662c36b1) > parent(s): 656f8867720 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-02-15 18:11:03 +0400 > message: > > MDEV-33442 REPAIR TABLE corrupts UUIDs Looks quite good. A question: you removed a hard-coded check for MYSQL_TYPE_NEWDECIMAL, but I don't see that you added Type_handler_olddecimal::type_handler_for_implicit_upgrade() A suggestion, rename Column_definition_implicit_upgrade(). You've changed its semantics, it's good to make sure that the code using old Column_definition_implicit_upgrade() (added e.g. in 10.6 and merged up) won't compile. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: 7ff9e91a7d2: MDEV-32975 Default charset doesn't work with PHP MySQLi extension
by Sergei Golubchik 25 Feb '24

25 Feb '24

Hi, Alexander, the fix is good, thanks. a couple of notes about comments, see below. ok to push after fixing them On Feb 25, Alexander Barkov wrote: > revision-id: 7ff9e91a7d2 (mariadb-10.4.32-179-g7ff9e91a7d2) > parent(s): 1070575a890 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-01-26 13:12:03 +0400 > message: > > MDEV-32975 Default charset doesn't work with PHP MySQLi extension > > When sending the server default collation ID to the client > in the handshake packet, translate a 2-byte collation ID > to the ID of the default collation for the character set. > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 1949a0c02b6..5103ff02105 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -12831,7 +12831,32 @@ static bool send_server_handshake_packet(MPVIO_EXT *mpvio, > > int2store(end, thd->client_capabilities); > /* write server characteristics: up to 16 bytes allowed */ may be move this comment to be directly above end[2]=... line, as before? Otherwise they're separated by another comment and an if() block, so the comment looks out of place > - end[2]= (char) default_charset_info->number; > + > + /* > + Send the default server collation ID. > + Note, since MySQL-4.1 it's actually the > + client responsibility to send its character set to the server, > + so this value means nothing and is ignored by most connectors. > + But some rare connectors (e.g. PHP) still read it. where did you get it from? protocol description doesn't say that the charset field in the handshake packet is obsolete and means nothing. better to remove this claim. If you want to obsolete this field, don't do it in an obscure comment in the code, but change the protocol description in KB. > + */ > + CHARSET_INFO *handshake_cs= default_charset_info; > + if (handshake_cs->number > 0xFF) > + { > + /* > + A workaround for a 2-byte collation ID: translate it into > + the ID of the primary collation of this character set. > + */ > + CHARSET_INFO *cs= get_charset_by_csname(handshake_cs->csname, > + MY_CS_PRIMARY, MYF(MY_WME)); > + /* > + cs should not normally be NULL, however it may be possible > + with a dynamic character set incorrectly defined in Index.xml. > + For safety let's fallback to latin1 in case cs is NULL. > + */ > + handshake_cs= cs ? cs : &my_charset_latin1; > + } > + end[2]= (char) handshake_cs->number; > + > int2store(end+3, mpvio->auth_info.thd->server_status); > int2store(end+5, thd->client_capabilities >> 16); > end[7]= data_len; > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 367c87a34f8: Optimize performance of my_bitmap
by Sergei Golubchik 23 Feb '24

23 Feb '24

Hi, Monty, On Feb 23, Michael Widenius wrote: > commit 367c87a34f8 > Author: Michael Widenius <monty(a)mariadb.org> > Date: Sun Feb 18 17:30:01 2024 +0200 > > diff --git a/include/my_bitmap.h b/include/my_bitmap.h > --- a/include/my_bitmap.h > +++ b/include/my_bitmap.h > @@ -81,14 +79,15 @@ extern void bitmap_union(MY_BITMAP *map, const MY_BITMAP *map2); > extern void bitmap_xor(MY_BITMAP *map, const MY_BITMAP *map2); > extern void bitmap_invert(MY_BITMAP *map); > extern void bitmap_copy(MY_BITMAP *map, const MY_BITMAP *map2); > +extern void bitmap_copy_data(MY_BITMAP *map, const uchar *ptr, uint bits); Why? In all cases where you use it, the source is a bitmap too. you can use a function that copies MY_BITMAP to MY_BITMAP. > > extern uint bitmap_lock_set_next(MY_BITMAP *map); > extern void bitmap_lock_clear_bit(MY_BITMAP *map, uint bitmap_bit); > /* Fast, not thread safe, bitmap functions */ > -#define bitmap_buffer_size(bits) (((bits)+31)/32)*4 > +#define bitmap_buffer_size(bits) (((bits)+63)/64)*8 may be MY_ALIGN(bits, sizeof(my_bitmap_map)*8)*sizeof(my_bitmap_map) ? and the same below? less error-prone or, even better #define my_bitmap_map_bytes sizeof(my_bitmap_map) #define my_bitmap_map_bits (my_bitmap_map_bytes*8) #define bitmap_buffer_size(bits) MY_ALIGN(bits,my_bitmap_map_bits)*my_bitmap_map_bytes > #define no_bytes_in_map(map) (((map)->n_bits + 7)/8) > -#define no_words_in_map(map) (((map)->n_bits + 31)/32) > -#define bytes_word_aligned(bytes) (4*((bytes + 3)/4)) > +#define no_words_in_map(map) (((map)->n_bits + 63)/64) > +#define bytes_word_aligned(bytes) (8*((bytes + 7)/8)) > /* The following functions must be compatible with create_last_word_mask()! */ > static inline void > bitmap_set_bit(MY_BITMAP *map,uint bit) > diff --git a/mysys/my_bitmap.c b/mysys/my_bitmap.c > --- a/mysys/my_bitmap.c > +++ b/mysys/my_bitmap.c > @@ -31,95 +42,82 @@ > New version written and test program added and some changes to the interface > was made by Mikael Ronstrom 2005, with assistance of Tomas Ulin and Mats > Kindahl. > + Updated to 64 bits and use my_find_first_bit() to speed up > + bitmap_get_next_set() by Monty in 2024 > */ > > #include "mysys_priv.h" > #include <my_bitmap.h> > #include <m_string.h> > #include <my_bit.h> > +#include <my_byteorder.h> > + > + > +/* Defines to check bitmaps */ > + > +#define DBUG_ASSERT_BITMAP(M) \ > + DBUG_ASSERT((M)->bitmap); \ > + DBUG_ASSERT((M)->n_bits > 0); \ > + DBUG_ASSERT((M)->last_word_ptr == (M)->bitmap + no_words_in_map(M)-1); \ > + DBUG_ASSERT((*(M)->last_word_ptr & (M)->last_word_mask) == 0); > + > +#define DBUG_ASSERT_BITMAP_AND_BIT(M,B) \ > + DBUG_ASSERT_BITMAP(M); \ > + DBUG_ASSERT((B) < (M)->n_bits); > + > +#define DBUG_ASSERT_DIFFERENT_BITMAPS(M,N) \ > + DBUG_ASSERT_BITMAP(M); \ > + DBUG_ASSERT_BITMAP(N); > + > +#define DBUG_ASSERT_IDENTICAL_BITMAPS(M,N) \ > + DBUG_ASSERT_BITMAP(M); \ > + DBUG_ASSERT_BITMAP(N); \ > + DBUG_ASSERT((M)->n_bits == (N)->n_bits); > > /* > - Create a mask with the upper 'unused' bits set and the lower 'used' > - bits clear. The bits within each byte is stored in big-endian order. > + Create a mask for the usable bits on the LAST my_bitmap_map position for > + a bitmap with 'bits' number of bits. > + > + The lowest 'bits' bits are set to zero and the rest bits are set to 1. > + For (bits & 63) == 0 , 0 is returned as in this case all bits in > + the my_bitmap_position are significant > + > + This code assumes the underlying storage is a 64 bit ulonglong. doesn't have to be if you'll use my_bitmap_map_bits > */ > > -static inline uchar invers_last_byte_mask(uint bits) > -{ > - return last_byte_mask(bits) ^ 255; > -} > +static inline my_bitmap_map last_word_mask(uint bits) > +{ > + uint bits_in_last_map= (bits & 63); > + return bits_in_last_map ? ~((1ULL << bits_in_last_map)-1) : 0ULL; you sure a condition is faster than a linear code of few bit operations? > +} > > > +static inline my_bitmap_map first_word_mask(uint bits) > +{ > + uint bits_in_last_map= (bits & 63); > + return ~((1ULL << bits_in_last_map)-1); and no condition here. strange. and shouldn't one of these two functions be inverted? supposedly one masks all bits before the first and the other - all bits after the last. > +} > -void create_last_word_mask(MY_BITMAP *map) > -{ > - unsigned char const mask= invers_last_byte_mask(map->n_bits); > - > - /* > - The first bytes are to be set to zero since they represent real bits > - in the bitvector. The last bytes are set to 0xFF since they represent > - bytes not used by the bitvector. Finally the last byte contains bits > - as set by the mask above. > - */ > - unsigned char *ptr= (unsigned char*)&map->last_word_mask; > - > - map->last_word_ptr= map->bitmap + no_words_in_map(map)-1; > - switch (no_bytes_in_map(map) & 3) { > - case 1: > - map->last_word_mask= ~0U; > - ptr[0]= mask; > - return; > - case 2: > - map->last_word_mask= ~0U; > - ptr[0]= 0; > - ptr[1]= mask; > - return; > - case 3: > - map->last_word_mask= 0U; > - ptr[2]= mask; > - ptr[3]= 0xFFU; > - return; > - case 0: > - map->last_word_mask= 0U; > - ptr[3]= mask; > - return; > - } > -} > > > -static inline my_bitmap_map last_word_mask(uint bit) > -{ > - my_bitmap_map last_word_mask; > - uint n_bits= bit + 1; > - unsigned char const mask= invers_last_byte_mask(n_bits); > - > - /* > - The first bytes are to be set to zero since they represent real bits > - in the bitvector. The last bytes are set to 0xFF since they represent > - bytes not used by the bitvector. Finally the last byte contains bits > - as set by the mask above. > - */ > - unsigned char *ptr= (unsigned char*)&last_word_mask; > - > - switch ((n_bits + 7)/8 & 3) { > - case 1: > - last_word_mask= ~0U; > - ptr[0]= mask; > - break; > - case 2: > - last_word_mask= ~0U; > - ptr[0]= 0; > - ptr[1]= mask; > - break; > - case 3: > - last_word_mask= 0U; > - ptr[2]= mask; > - ptr[3]= 0xFFU; > - break; > - case 0: > - last_word_mask= 0U; > - ptr[3]= mask; > - break; > - } > - return last_word_mask; > -} > +/* > + Update the bitmap's last_word_ptr and last_word_mask > + Also ensure that the last world is all zero to make it > + easy to find the next set bit. > + > + Note that if n_bits is 0, then last_word_ptr will point to > + bitmap (safetly). The bitmap will not be usable for almost any operation. ether "safety" or "safely" > +*/ > + > +void create_last_word_mask(MY_BITMAP *map) > +{ > + my_bitmap_map mask= last_word_mask(map->n_bits); > + map->last_word_mask= mask; > + map->last_word_ptr= map->bitmap + MY_MAX(no_words_in_map(map),1) -1; > + if (map->n_bits > 0) > + { > + *map->last_word_ptr&= ~mask; /* Set not used bits to 0 */ > + DBUG_ASSERT_BITMAP(map); > + } > +} > > > @@ -326,37 +312,45 @@ void bitmap_set_prefix(MY_BITMAP *map, uint prefix_size) > } > if ((d= no_bytes_in_map(map)-prefix_bytes)) > memset(m, 0, d); > + DBUG_ASSERT_BITMAP(map); > } > > > +/** > + Check if bitmap is a bitmap of prefix bits set in the beginning > + > + @param map bitmap > + @param prefix_size number of bits that should be set. 0 is allowed. > + > + @return 1 Yes, prefix bits where set or prefix_size == 0. > + @return 0 No > +*/ > + > my_bool bitmap_is_prefix(const MY_BITMAP *map, uint prefix_size) > { > uint prefix_mask= last_byte_mask(prefix_size); > uchar *m= (uchar*) map->bitmap; > uchar *end_prefix= m+(prefix_size-1)/8; > uchar *end; > - DBUG_ASSERT(m); > - DBUG_ASSERT(prefix_size <= map->n_bits); > > /* Empty prefix is always true */ > if (!prefix_size) > return 1; > > + DBUG_ASSERT_BITMAP_AND_BIT(map, prefix_size-1); > + > while (m < end_prefix) > if (*m++ != 0xff) > return 0; > > - end= ((uchar*) map->bitmap) + no_bytes_in_map(map) - 1; > - if (m == end) > - return ((*m & last_byte_mask(map->n_bits)) == prefix_mask); > - > + end= ((uchar*) map->last_word_ptr) + sizeof(*map->last_word_ptr)-1; good, here you use sizeof(), not just 7. > if (*m != prefix_mask) > return 0; > > - while (++m < end) > + while (++m <= end) > if (*m != 0) > return 0; > - return ((*m & last_byte_mask(map->n_bits)) == 0); > + return 1; > } > > > @@ -447,50 +442,51 @@ void bitmap_intersect(MY_BITMAP *map, const MY_BITMAP *map2) > > /* > Check if there is some bit index between start_bit and end_bit, such that > - this is bit is set for all bitmaps in bitmap_list. > + this is atleast on bit that set for all bitmaps in bitmap_list. "at least" and "one" and "that is set" ? > > SYNOPSIS > bitmap_exists_intersection() > bitmpap_array [in] a set of MY_BITMAPs > - bitmap_count [in] number of elements in bitmpap_array > + bitmap_count [in] number of elements in bitmap_array > start_bit [in] beginning (inclusive) of the range of bits to search > end_bit [in] end (inclusive) of the range of bits to search, must be > no bigger than the bits of the shortest bitmap. > > - NOTES > - This function assumes that for at least one of the bitmaps in bitmap_array all > - bits outside the range [start_bit, end_bit] are 0. As a result is not > - necessary to take care of the bits outside the range [start_bit, end_bit]. > - > RETURN > TRUE if an intersecion exists > FALSE no intersection > */ > > -my_bool bitmap_exists_intersection(const MY_BITMAP **bitmap_array, > +my_bool bitmap_exists_intersection(MY_BITMAP **bitmap_array, > uint bitmap_count, > uint start_bit, uint end_bit) > { > uint i, j, start_idx, end_idx; > - my_bitmap_map cur_res; > + my_bitmap_map cur_res, first_map; > > DBUG_ASSERT(bitmap_count); > DBUG_ASSERT(end_bit >= start_bit); > for (j= 0; j < bitmap_count; j++) > - DBUG_ASSERT(end_bit < bitmap_array[j]->n_bits); > + { > + DBUG_ASSERT_BITMAP_AND_BIT(bitmap_array[j], end_bit); > + } > > start_idx= start_bit/8/sizeof(my_bitmap_map); > end_idx= end_bit/8/sizeof(my_bitmap_map); > > + first_map= first_word_mask(start_bit); > + cur_res= first_map; > for (i= start_idx; i < end_idx; i++) > { > - cur_res= ~0; > for (j= 0; cur_res && j < bitmap_count; j++) > cur_res &= bitmap_array[j]->bitmap[i]; > if (cur_res) > return TRUE; > + cur_res= ~(my_bitmap_map) 0; > } > - cur_res= ~last_word_mask(end_bit); > + cur_res= ~last_word_mask(end_bit+1); > + if (start_idx == end_idx) > + cur_res&= first_map; > for (j= 0; cur_res && j < bitmap_count; j++) > cur_res &= bitmap_array[j]->bitmap[end_idx]; > return cur_res != 0; > @@ -501,21 +497,26 @@ my_bool bitmap_exists_intersection(const MY_BITMAP **bitmap_array, > > my_bool bitmap_union_is_set_all(const MY_BITMAP *map1, const MY_BITMAP *map2) > { > - my_bitmap_map *m1= map1->bitmap, *m2= map2->bitmap, *end; > + my_bitmap_map *m1= map1->bitmap, *m2= map2->bitmap, *end= map1->last_word_ptr; > + DBUG_ASSERT_IDENTICAL_BITMAPS(map1,map2); > > - DBUG_ASSERT(map1->bitmap); > - DBUG_ASSERT(map2->bitmap); > - DBUG_ASSERT(map1->n_bits==map2->n_bits); > - end= map1->last_word_ptr; > while ( m1 < end) > - if ((*m1++ | *m2++) != 0xFFFFFFFF) > + if ((*m1++ | *m2++) != 0xFFFFFFFFFFFFFFFFULL) better ~(my_bitmap_map)0 > return FALSE; > /* here both maps have the same number of bits - see assert above */ > - return ((*m1 | *m2 | map1->last_word_mask) != 0xFFFFFFFF); > + return ((*m1 | *m2 | map1->last_word_mask) != 0xFFFFFFFFFFFFFFFFULL); > } > > > > +#ifdef NOT_USED > + > +/* > + If we re-introduce this one one, we should change the interface to > + use_from_bit and not from_byte. We should also add testing for this > + in bitmap-t.c > +*/ just remove it, it's easier to write it from scratch again when needed then to undertand and fix this old dead code > + > /* > Set/clear all bits above a bit. > > @@ -603,45 +601,51 @@ uint bitmap_bits_set(const MY_BITMAP *map) > my_bitmap_map *data_ptr= map->bitmap; > my_bitmap_map *end= map->last_word_ptr; > uint res= 0; > - DBUG_ASSERT(map->bitmap); > + DBUG_ASSERT_BITMAP(map); > > - for (; data_ptr < end; data_ptr++) > - res+= my_count_bits_uint32(*data_ptr); > + for (; data_ptr <= end; data_ptr++) > + res+= my_count_bits(*data_ptr); > > - /*Reset last bits to zero*/ > - res+= my_count_bits_uint32(*map->last_word_ptr & ~map->last_word_mask); > return res; > } > > void bitmap_copy(MY_BITMAP *map, const MY_BITMAP *map2) > { > - my_bitmap_map *to= map->bitmap, *from= map2->bitmap, *end; > - > - DBUG_ASSERT(map->bitmap); > - DBUG_ASSERT(map2->bitmap); > - DBUG_ASSERT(map->n_bits == map2->n_bits); > - end= map->last_word_ptr; > + my_bitmap_map *to= map->bitmap, *from= map2->bitmap, *end= map->last_word_ptr; > + DBUG_ASSERT_IDENTICAL_BITMAPS(map,map2); > > while (to <= end) > *to++ = *from++; > } > > > +/* > + Copy data into the bitmap from a byte array > +*/ > + > +void bitmap_copy_data(MY_BITMAP *map, const uchar *ptr, uint bits) not needed, as far as I understand > +{ > + my_bitmap_map *to= map->bitmap; > + uint length= (bits + 7)/8; > + uint map_length= no_bytes_in_map(map); > + DBUG_ASSERT_BITMAP_AND_BIT(map, bits-1); > + > + memcpy(to, ptr, length); > + if (map_length != length) > + bzero(to + length, map_length - length); > + *map->last_word_ptr&= ~map->last_word_mask; > +} > + > + > uint bitmap_get_first_set(const MY_BITMAP *map) > { > - uint i; > my_bitmap_map *data_ptr= map->bitmap, *end= map->last_word_ptr; > + DBUG_ASSERT_BITMAP(map); > > - DBUG_ASSERT(map->bitmap); > - > - for (i=0; data_ptr < end; data_ptr++, i++) > + for (uint i=0; data_ptr <= end; data_ptr++, i++) > if (*data_ptr) > - goto found; > - if (!(*data_ptr & ~map->last_word_mask)) > - return MY_BIT_NONE; > - > -found: > - return get_first_set(*data_ptr, i); > + return my_find_first_bit(*data_ptr) + i * sizeof(my_bitmap_map)*8; > + return MY_BIT_NONE; > } > > > @@ -703,35 +696,18 @@ uint bitmap_get_next_set(const MY_BITMAP *map, uint bitmap_bit) > > uint bitmap_get_first(const MY_BITMAP *map) this should be named bitmap_get_first_clear(). because it's always is_bit_set/is_bit_clear, is_set_all/is_clear_all, test_and_set/test_and_clear, set_bit/clear_bit, etc. there is bitmap_get_first_set, so this one must be bitmap_get_first_clear > { > - uchar *byte_ptr; > - uint i,j,k; > - my_bitmap_map *data_ptr, *end= map->last_word_ptr; > - > - DBUG_ASSERT(map->bitmap); > - data_ptr= map->bitmap; > - *map->last_word_ptr|= map->last_word_mask; > + uint i; > + my_bitmap_map *data_ptr= map->bitmap, *end= map->last_word_ptr; > + DBUG_ASSERT_BITMAP(map); > > - for (i=0; data_ptr < end; data_ptr++, i++) > - if (*data_ptr != 0xFFFFFFFF) > + for (i= 0; data_ptr < end; data_ptr++, i++) > + if (*data_ptr != 0xFFFFFFFFFFFFFFFFULL) > goto found; > - if ((*data_ptr | map->last_word_mask) == 0xFFFFFFFF) > + if ((*data_ptr | map->last_word_mask) == 0xFFFFFFFFFFFFFFFFULL) > return MY_BIT_NONE; > - > found: > - byte_ptr= (uchar*)data_ptr; > - for (j=0; ; j++, byte_ptr++) > - { > - if (*byte_ptr != 0xFF) > - { > - for (k=0; ; k++) > - { > - if (!(*byte_ptr & (1 << k))) > - return (i*32) + (j*8) + k; > - } > - } > - } > - DBUG_ASSERT(0); > - return MY_BIT_NONE; /* Impossible */ > + /* find first zero bit by reverting all bits and find first bit */ > + return my_find_first_bit(~*data_ptr) + i * sizeof(my_bitmap_map)*8; > } > > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 264aef71a37: Have ha_partition ignore HA_EXTRA..CHILDREN extra() calls if no myisamrg
by Sergei Golubchik 23 Feb '24

23 Feb '24

Hi, Michael, On Feb 23, Michael Widenius wrote: > revision-id: 264aef71a37 (mariadb-10.6.17-15-g264aef71a37) > parent(s): 042c3fc432b > author: Michael Widenius > committer: Michael Widenius > timestamp: 2024-02-22 09:20:51 +0200 > message: > > Have ha_partition ignore HA_EXTRA..CHILDREN extra() calls if no myisamrg > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index e174ef8d219..f2b3ab3aff9 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -3030,6 +3031,10 @@ bool ha_partition::create_handlers(MEM_ROOT *mem_root) > DBUG_PRINT("info", ("InnoDB")); > m_innodb= TRUE; > } > + else if (ha_legacy_type(hton0) == DB_TYPE_MRG_MYISAM) > + { > + m_myisammrg= TRUE; > + } > DBUG_RETURN(FALSE); > } > > @@ -9419,9 +9424,14 @@ int ha_partition::extra(enum ha_extra_function operation) > } > /* Category 9) Operations only used by MERGE */ > case HA_EXTRA_ADD_CHILDREN_LIST: > + if (!m_myisammrg) > + DBUG_RETURN(0); > DBUG_RETURN(loop_partitions(extra_cb, &operation)); > case HA_EXTRA_ATTACH_CHILDREN: > { > + if (!m_myisammrg) > + DBUG_RETURN(0); > + > int result; This is, of course, quite horrible. Checking the legacy type, doing something for a specify underlying engine. But I don't see a better way of doing it, short of changing the storage engine API. What if we redefine ::extra() call to return HA_ERR_WRONG_COMMAND for unsupported extra operations? The caller will not abort on it, but it'll allow the caller to know whether the operation was executed or ignored. This will allow partitioning handler to skip all extra() methods that are not implemented in the underlying engine. For all engines and all extra operations. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: Second part of review of XA patches in bb-10.6-MDEV-31949
by Kristian Nielsen 17 Feb '24

17 Feb '24

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > > I think it's better to define a proper API for this. > > It feels an overkill in the current case. "Define a proper API" means that you think through how this should work for different storage engines and TC_LOG implementations. And then you document those thoughts. I don't understand how doing this can be overkill, you are re-defining how durability of external XA transactions are coordinated between the server layer and the storage engines. >> And vice versa, for every xid >> X_COMPLETE found, roll it back if X is found prepared in InnoDB, > > Actually >> else commit it. > safer must be to commit both X_COMPLETE and X. Generally even if > X_COMPLETE and X commits in this order, X can race to be durable before > crash. Right, and "commit X" would mean "XA ROLLBACK X" if the "complete" operation is a rollback. I didn't follow why rolling back the X_COMPLETE would be a problem, but it's not important for now; I can see how committing should work as well. >> One reason I'm asking is I'm wondering how this will affect the scheduling >> of XA in parallel replication? Like, in case of duplicate XID, the X_PREPARE >> or X_COMPLETE could conflict between two transactions, so maybe the >> wakeup_subsequent_commits can only be done after the XA COMMIT of the >> X_PREPARE and X_COMMIT? Which I think would require changes to the current >> patch. > > Hmm, right now in the branch (see Sliding_window etc) for the duplicate > XID case, like > P_1,C_2,P_3,C_4 > P - XA-PREPARE, C - ... commit) P_3 can start only when C_2 wakes it up. Yes. So my point is: In this case, the C_2 cannot do wakeup_subsequent_commits inside the group commit along with run_commit_ordered(), that would be too early. C_2 must first complete the extra internal XA transaction on the mysql.gtid_slave_pos table before it can wakeup P_3. Or is this not a problem? > My bad I left out a mention of the plan, to treat - call it > XAC_0 -> XAP_1 > dependency - by the optimistic retry. That's in order to minimize the Aha. So if we optimistically apply the XA PREPARE, and we get an error ER_XAER_DUPID that the XID already exists as prepared, then we would do a retry like other temporary errors, is that what you mean? Is this implemented in the current patch? Maybe it is automatically handled by convert_kill_to_deadlock_error() ? If this XAC_0 -> XAP_1 conflict can be handled by normal optimistic rollback/retry, then I agree there is no need to handle the dependency explicitly in the sliding window. That was not clear to me from the code. >> This I don't understand at all. >> How can an XA PREPARE/COMMIT/ROLLBACK be marked as "ddl" ? > > That's a binlogging whim. E.g a temp table "DDL" involved into > transaction. I was not sure I could quickly eliminate the flagging, so > decided to live with it. Aha, right, CREATE TEMPORARY TABLE can be part of a transaction and thus DDL (in statement-based binlogging). I missed that. > But I could not get the meaning of > > ... handler.cc to avoid special cases for whether the binlog is enabled > The burden of checking whether the binlog is enabled, e.g for taking a > TC_LOG->log_and_order() path is on the programmer. If binlog is globally enabled (--log-bin option), then tc_log is an object of class MYSQL_BIN_LOG. Else it is class TC_LOG_MMAP (or TC_LOG_DUMMY). So a virtual function can be used instead of a conditional on opt_bin_log. But binlogging can be disabled on a transaction-by-transaction basis by setting SQL_LOG_BIN, is that the case here? Then yes, it seems an explicit condition will be needed. - Kristian.

1 0

[Andrei Elkin] Re: Third/last part of review of XA patches in bb-10.6-MDEV-31949
by andrei.elkin＠pp.inet.fi 12 Feb '24

12 Feb '24

Kristian, Using the chance to forwarding the reply that did not make to the developers list, let me comment on binlog_commit() that I might hastily rate as a big subject. > Hi Andrei, > > Here is the last part of my review of the changes in branch > bb-10.6-MDEV-31949: > >> --- a/sql/log_event_server.cc >> +++ b/sql/log_event_server.cc >> @@ -3324,6 +3324,11 @@ Gtid_log_event::Gtid_log_event(THD *thd_arg, uint64 seq_no_arg, >> flags2|= thd->lex->sql_command == SQLCOM_XA_PREPARE ? >> FL_PREPARED_XA : FL_COMPLETED_XA; >> xid.set(xid_state.get_xid()); >> + >> + // multi-engine external completion is unaware of the prepared engine #. >> + extra_engines= thd->transaction->all.ha_list ? >> + ha_count_rw_2pc(thd_arg, >> + thd_arg->in_multi_stmt_transaction_mode()) - 1 : 0; > > I cannot understand the comment. > I assume it's about that an XA COMMIT of a detached xid (different > session > that the XA PREPARE, or replication slave) - right? Correct. > > But what is the consequences of not knowing extra_engines in this > case? > Is the extra_engines used for anything than the optional truncate of > the > binlog after master crash? In the multi-engine case when the # of prepared in Engine is greater than in the value in GTID event then the trx is partially committed, so is decided for to be completed. > >> @@ -4175,7 +4173,8 @@ int XA_prepare_log_event::do_commit() >> thd->lex->xid= &xid; >> if (!one_phase) >> { >> - if ((res= thd->wait_for_prior_commit())) >> + if (thd->is_current_stmt_binlog_disabled() && >> + (res= thd->wait_for_prior_commit())) > > Why?000 > Is it to allow XA_PREPARE_LOG_EVENT to group commit with other event > groups? > That makes sense, but then shouldn't the wait_for_prior_commit() be > somewhere in handler.cc (or maybe xa.cc)? > Look at Xid_log_event::do_commit(), it doesn't have any > wait_for_prior_commit(), that is in ha_commit_one_phase(). Right. > (The wait-for-prior-commit facility is designed as a server-level > feature, > not a replication-level, it could in principle be used outside of > replication, even though it not currently (IIRC).) I think I understood what you mean which is follow the ha_commit_one_phase() pattern. [todo] > >> diff --git a/sql/sql_array.h b/sql/sql_array.h >> index 8610e971016..94c6d86ac05 100644 >> --- a/sql/sql_array.h >> +++ b/sql/sql_array.h >> @@ -172,6 +172,8 @@ template <class Elem> class Dynamic_array >> >> size_t size() const { return array.elements; } >> >> + size_t max_size() const { return array.max_element; } >> + > > Not that it hurts that much, but I think you can avoid adding this. Left over of a cleanup from the previous version. [todo] > >> --- a/sql/log.cc >> +++ b/sql/log.cc >> @@ -85,6 +85,7 @@ const char *log_bin_index= 0; >> const char *log_bin_basename= 0; >> >> MYSQL_BIN_LOG mysql_bin_log(&sync_binlog_period); >> +XID_cache_element *xid_cache_search(THD *thd, XID *xid); > > This duplicates the same declaration in sql/xa.h > So probably this can be removed now? [todo] > >> @@ -2186,16 +2202,13 @@ int binlog_commit(THD *thd, bool all, bool >> ro_1pc) >> } >> >> if (cache_mngr->trx_cache.empty() && >> - (thd->transaction->xid_state.get_state_code() != XA_PREPARED || >> - !(thd->ha_data[binlog_hton->slot].ha_info[1].is_started() && >> - thd->ha_data[binlog_hton->slot].ha_info[1].is_trx_read_write()))) >> + (!(is_preparing_xa(thd) || is_prepared_xa(thd)) || >> + !is_binlog_read_write(thd))) > > This binlog_commit function started out as the "commit" method for the > binlog handlerton. But that is not how it is used anymore, now it is > called > directly from several different places, including some places specific > to > external user XA. > > A large part of the function has become complex conditions like this > that > are basically impossible for the developer (or reviewer) to understand > exactly why this set of conditions is the correct one. Again, the > function > is trying to guess what the caller wants, depending on the state of > various > thd properties etc. > > Instead, the function should be split into multiple functions. One > that is > called to handle XA PREPARE; one for binlog_commit_by_xid(), etc. Then > each > version should be able to greatly reduce the number of conditions > needed. > > And if any logic is shared between the different versions, this can > then be > put in smaller functions that are called from each version. [my initial response was: Your proposal leads to a significant refactoring, which I add only one hunk. I'd really consider that as a separate exercise I think I've understood. Sure, binlog_prepare() that logs XA-PREPARE can call a common function with binlog_commit() that originally intended to log only normal trx:s. I can look into that. [todo] >> >> @@ -11050,10 +11124,30 @@ bool >> Recovery_context::decide_or_assess(xid_recovery_member *member, int >> round, >> } >> else if (member->in_engine_prepare < last_gtid_engines) >> { >> - DBUG_ASSERT(member->in_engine_prepare > 0); >> + DBUG_ASSERT(member->in_engine_prepare > 0 || >> + /* XA event over xid that was completed in engines */ >> + (member->xid == 0 && >> + member->xa_binlog_state >= xid_recovery_member::XA_PREPARE)); >> + if (member->in_engine_prepare > 0 && >> + member->xa_binlog_state == xid_recovery_member::XA_PREPARE) >> + { > > The function decide_or_assess() is hard to understand. > I can see that it's called in 3 cases: XID_LOG_EVENT, XA PREPARE, XA > COMMIT/ROLLBACK. > So again here, I think this should be split into 3 different > functions, one > for each case (or maybe two, one for internal XID and one for external > XA, > whatever makes the more sense). > > I don't understand the comment "XA event over xid that was completed > in > engines". But I remember seeing something in log_event_server.cc that > sets > extra_engines to 0 in the external user XA case. > > So is that the case here, that last_gtid_engines==0 marks this is user > XA? > That would be much clearer if the logic for internal and external XA > was not > mixed in the same function. > >> + /* >> + the first of the two options below can occur when round 1 turns to 2 >> + so event sequences across binlog files are like the following >> + g_current(B_0, xid), ... g_last(B^*, xid), >> + where `g`:s are xa event groups associated with the same member via >> + common `xid`. As elsewhere B_0 is the 2nd round's first and, >> + B^* the 1st round binlogs. >> + */ >> if (reset_truncate_coord(member->binlog_coord < last_gtid_coord ? >> member->binlog_coord.second : pos)) > > I'm trying to understand what is going on here. I think this is refering to > the way binlog files are scanned? > First we scan the most recent binlog file. At the end of it, we may find > that the latest binlog checkpoint is in an earlier file, then we continue to > scan those older files. So for example we may scan in order: > > master-bin.000004, master-bin.000002, master-bin.000003 > > This means that we may see the XA PREPARE and XA COMMIT out-of-order as we > go from the first scan to the remaining - "round 1 turns to 2". > > But I don't understand that this is "the first of the two options below": > > member->binlog_coord < last_gtid_coord ? member->binlog_coord.second : pos > > If last_gtid_coord is in master-bin.000002, but member->binlog_coord is in > master-bin.000004 ("round 1 turns to 2"), then I would think we would have > > member->binlog_coord > last_gtid_coord > > So it would be the *second* of the options below? What am I missing here? > > Also, I would suggest a simplification here. After we do the initial scan in > round 1, if we find that we need to scan more files, then reset all the > state and hashes. And then do a full scan of all necessary binlog files > in-order. In my example, that would be: > > master-bin.000004 > <reset state> > master-bin.000002 > master-bin.000003 > master-bin.000004 > > This way, the logic becomes simpler, as we don't need logic to handle seeing > events out-of-order. > > This will mean that occasionally we will need to scan one more binlog file, > but I think that is a good trade-off. The recovery code is notoriously > difficult to test and get correct, because it's an unusual occurence to need > recovery in the first place, and very hard to reproduce any problems. And > robust and correct recovery is the whole point of having XA in the first > palce. So making the code as simple as possible is crucial. > > The extra cost is small: > > - For one, recovery is a rare event > - Re-scanning only costs CPU, the file is already read from storage into OS > buffers. > - Normally, the binlog checkpoint completes shortly after rotation, so the > last binlog file that must be rescanned will be short. > > So simply redo the scan from the correct point when a "round 2" is needed, > to avoid any logic to handle seeing events out-of-order during recovery. > > >> @@ -11182,7 +11311,9 @@ void Recovery_context::process_gtid(int round, Gtid_log_event *gev, >> last_gtid_no2pc= false; >> last_gtid_standalone= >> (gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false; >> - if (do_truncate && last_gtid_standalone) >> + /* a standalone XA-COMPLETE is truncation-safe */ >> + if (do_truncate && last_gtid_standalone >> + && !(gev->flags2 & Gtid_log_event::FL_COMPLETED_XA)) >> update_binlog_unsafe_coord_if_needed(linfo); > > I don't understand the comment here. > What is "a standalone XA-COMPLETE" ? > What does it mean to be "truncation-safe"? > Please reword the comment to be comprehensible. > >> + /* >> + the user XA events are decided similary to the normal transaction, >> + difference is just in the timing which is Gtid event read now. >> + */ > > I don't understand this comment. Please reformulate to make it > understandable. What does normal transactions do, and what does user > XA do? > How does the timing differ, and why? > >> + sql_print_warning("GTID %u-%u-%s XA transaction state % " >> + "in binlog file:%s pos:%llu is inconsistent " >> + "with the former same state; " >> + "consider filtered binary loggig.", >> + ctx.last_gtid.domain_id, ctx.last_gtid.server_id, >> + buf, is_xac ? "'COMPLETE'" : "'PREPARE'", >> + linfo->log_file_name, ctx.last_gtid_coord.second); > > Typo: s/loggig/logging/ > > Also, the phrase "the former same state" is not understandable (at > least to > me), what does it mean? Please reformulate. > I also have to be short now in replying to the rest, which I'll expand on tomorrow morning. Now just this. >> @@ -909,14 +910,16 @@ ha_commit_checkpoint_request(void *cookie, void (*pre_hook)(void *)) >> there's no need to rollback here as all transactions must >> be rolled back already >> */ >> -void ha_close_connection(THD* thd) >> +void ha_close_connection(THD* thd, bool skip_binlog) > > I tried to understand what the purpose of this change is. > ha_close_connection() is now called from two new places related to > replication of XA PREPARE. And in one place with this mysterious > skip_binlog=true condition. > > My guess is that this is to detach the XA PREPARE from the current > THD? The reason is described in 6cbb63f592 commit name II.1. ``` OTHER CHANGES, to ha_close_connection() are done to keep the binlog hton connection alive for the end of XA-prepare's binlogging on the parallel slave. Other than the binlog hton:s are disconnected from the slave transaction handler in TC_LOG::run_commit_ordered() which makes the prepared transaction engine objects available to binlog-commit-order followers. ``` In other words XAP_k in run_commit_ordered() releases with ha_close_connection(thd, true) all resources that the following XAC_k+i need. The latter does not need XAP_k to release binlog which it still need while being is this stack of TC_LOG::run_commit_ordered MYSQL_BIN_LOG::trx_group_commit_leader MYSQL_BIN_LOG::write_transaction_to_binlog_events MYSQL_BIN_LOG::write_transaction_to_binlog binlog_flush_cache binlog_commit_flush_xa_prepare binlog_commit binlog_prepare prepare_or_error ha_prepare > So > that it becomes visible in XA RECOVER to other threads, and (maybe > more > important) to make the THD available to run another transaction? One > of the > things InnoDB does in close_connection is: trx_disconnect_prepared(). > > But I think the purpose of the close_connection handlerton call is > different And you must have meant 'ha_close_connection', right? > - it is to free things at the end of a client connection? Well, with the ha- prefix, it's to disconnect THD from all transaction branches/htons. > So it can do > more > than detach the current external XA - which is presumably the reason > for the > need to hack it to not call the specific binlog_close_connection()? > > So this shouldn't hijack the close_connection to mean > "detach_xa_trx". It > looks like we need an explicit handlerton call for detaching an xa > transaction, that can then be used instead of this hack in > ha_close_connection(). I need to digest that in the morning. Generally I don't see a problem to conduct hton release flexibly. > >> -static my_bool xacommit_handlerton(THD *unused1, plugin_ref plugin, >> - void *arg) >> +my_bool xacommit_handlerton(THD *unused1, plugin_ref plugin, void *arg) >> { >> handlerton *hton= plugin_hton(plugin); >> - if (hton->recover) >> + if (hton->recover && hton != binlog_hton) >> { >> hton->commit_by_xid(hton, ((struct xahton_st *)arg)->xid); > >> -int ha_commit_or_rollback_by_xid(XID *xid, bool commit) >> +int ha_commit_or_rollback_by_xid(XID *xid, bool commit, THD *thd) >> { > >> + if (!skip_binlog) >> + { >> + // when binlog is ON start from its transaction branch >> if (commit) >> binlog_commit_by_xid(binlog_hton, xid); >> else >> binlog_rollback_by_xid(binlog_hton, xid); > > I think the purpose if this is to ensure that commit/rollback of XA > calls > into the binlog code first (to binlog before doing it in the engines). > > But as I think Serg also remarked already, this should not be > needed. The > binlog is the (internal) transaction coordinator when it is enabled, > we have > a dedicated interface for that, class TC_LOG. This is used elsewhere > in > handler.cc to avoid special cases for whether the binlog is enabled or > not. > > So it looks like there needs to be extra virtual functions in TC_LOG > for > the XA commit/rollback, and these can then be called on the tc_log > object > and do what is required. I need to process that tomorrow. > Then some logic in handler.cc can be omitted/simplified. > > ---- > > I think with this, I got through all of the code changes in branch > bb-10.6-MDEV-31949 as of commit > 1bdc9bc8077c79e55cb4dbfc7cc47a193d17d54f > from January 20, 2024. So this concludes the current review. I have > not > looked at the test cases in this round of review. Thanks again! Cheers, Andrei

1 1

Re2: Starting on review of XA patches in bb-10.6-MDEV-31949
by andrei.elkin＠pp.inet.fi 12 Feb '24

12 Feb '24

Salve Kristian, this is the 2nd follow-up replay to your 1st review, to cover few deferred notes in particular. I mark with [x] any addressed [todo] note. > Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > >> Hi Andrei, > So you're in this context > > 1 innobase_commit_by_xid( > 2 /*===================*/ > 3 handlerton* hton, > 4 XID* xid) /*!< in: X/Open XA transaction > identification */ > 5 { > 6 DBUG_ASSERT(hton == innodb_hton_ptr); > 7 > 8 DBUG_EXECUTE_IF("innobase_xa_fail", > 9 return XAER_RMFAIL;); > 10 > 11 if (high_level_read_only) { > 12 return(XAER_RMFAIL); > 13 } > 14 > 15 if (trx_t* trx = trx_get_trx_by_xid(xid)) { > 16 THD *thd = current_thd; > 17 if (thd) > 18 thd_binlog_pos(thd, &trx->mysql_log_file_name, > 19 &trx->mysql_log_offset); > 20 if (trx->mysql_log_offset > 0) > 21 trx->flush_log_later = true; > > and actually considering `trx->mysql_log_offset` as the return value? > ... I could not grasp your way of thinking in this place. > Could you please expand on? In another mail yesterday concerning with this part I suggested to replace if (trx->mysql_log_offset > 0) with if (thd_binlog_format(thd) != UNSPEC) > > >> >> It would be much better if this could be done similar to how normal >> commits >> are done. We have the commit_ordered handlerton which does the fast >> part of >> commit-to-memory. And then the slow part which does the rest in commit >> handlerton. > > Notice that the *slow* part of the XA commit is the same as in the > normal > case: > > trx_commit_complete_for_mysql > *innobase_commit* > commit_one_phase_2 > ha_commit_one_phase > > The fast parts of the two - the XA and normal - could be unified - into > > innobase_commit_ordered() > > but I did not think (had time for) towards that. > >> >> So we could have commit_by_xid_ordered() and commit_by_xid(). The >> problem is >> that commit_by_xid_ordered() needs to release the xid, so it becomes >> harder >> to get hold of the transaction (which the commit handlerton just finds >> from >> the thd). Maybe commit_by_xid_ordered() would need to return some trx >> handle. >> >> What part of the commit is skipped by this in InnoDB? Is it _only_ >> flushing >> the log up to the LSN of the commit record, or are there more things? > > > So the xa's fast innobase_commit_by_xid() skips a "slow" > trx_commit_complete_for_mysql() > and the same does the normal's innobase_commit_ordered(). > >> >> How is another storage engine supposed to handle this? This also needs >> to be >> documented somewhere, eg. in comments on the handlerton calls. >> >> This also means that the --innodb-flush-log-at-trx-commit=3 is not >> supported >> for XA COMMIT when it's done from a different session that the XA >> PREPARE. >> For example, with --innodb-flush-log-at-trx-commit=3 but >> --sync-binlog=0, >> normal transactions are still durable in InnoDB. But external XA >> transactions will not be. This could be solved if the commit_by_xid() >> was >> split into a fast and slow part similar to normal commit. > > [todo] Let me process that carefully later. [x] My 1st reply already said, just missed to underscore that, that the XA also the slow and fast commit paths, therefore iflatc=3 is good. The past paths are commit_one_phase_2 and innobase_commit_by_xid functionally equivalent. One would vote to unify them, and I'd accept that. > >> >> Probably you'll say that this works for now, and perhaps that's true. >> But at >> least it really _really_ needs some detailed comments explaining what >> is >> going on, and what should be kept in mind if modifying/improving this >> code >> in the future, otherwise it will be impossible for the next developer >> to >> understand. > > [todo] Agreed. > >> >> >>> diff --git a/sql/log.cc b/sql/log.cc >>> index e54d4087d46..7e3dbeaef29 100644 >>> --- a/sql/log.cc >>> +++ b/sql/log.cc >> >>> @@ -9365,18 +9381,84 @@ TC_LOG::run_commit_ordered(THD *thd, bool >>> all) >>> all ? thd->transaction->all.ha_list : >>> thd->transaction->stmt.ha_list; >>> >>> mysql_mutex_assert_owner(&LOCK_commit_ordered); >>> - for (; ha_info; ha_info= ha_info->next()) >>> + >>> + if (!ha_info) >> >> Your patch puts a lot of XA/binlog logic into run_commit_ordered(). >> But >> that's not appropriate. The run_commit_ordered() function has the >> purpose >> solely to call the commit_ordered() handlerton in each engine, and >> it's also >> called from TC_LOG_MMAP (that's why it's TC_LOG::, not >> MYSQL_BIN_LOG::). >> >> Instead make a new function that handles the part of binlog commit >> that >> needs to be done under LOCK_commit_ordered. Be sure to comment it that >> this >> runs under this highly contended mutex, and needs to do as little work >> as >> possible. Agreed. [todo] >> >> This function can then call run_commit_ordered(). It can also be used >> to >> share a few bits of code that are now duplicated between the >> opt_optimize_thread_scheduling and !opt_optimize_thread_scheduling >> cases, >> like: >> >> ++num_commits; >> run_commit_ordered(...) >> entry->thd->wakeup_subsequent_commits() >> if (entry->queued_by_other) { ... } >> >> (if it makes sense to do so in the actual code, only if makes things >> simpler, that's easier to decide when working with the actual code). Will see. [todo] >> >> There sesems to be 3 new cases: >> >> - XA PREPARE >> - XA COMMIT from same session as XA PREPARE >> - XA COMMIT of detached XA transaction >> >> The logic to decide this was already done earlier, when the event >> group was >> queued for group commit, eg. to decide which end_event to use. For >> example >> there is no need to call xid_cache_search() here, that can be done >> earlier >> (and it's also wrong to call my_error() here, as we're in the wrong >> thread >> context). Indeed. The error's thread context was overlooked. Thanks for catching it. >> >> So the logic about what to do should be placed higher up in the call >> stack, >> ideally where we already know, eg. binlog_commit_by_xid(). And errors >> such >> as ER_XAER_NOTA can be thrown already there, and the decision on what >> to do >> put consisely as a flag into the struct group_commit_entry, so the >> logic >> during binlog group commit becomes as simple as possible. >> >> (It might even make sense to simply store a function pointer to the >> method >> that should be called to handle this; normal, attached XA COMMIT, >> detached >> XA COMMIT, XA PREPARE. But again it's easier when actually working >> with the >> code to decide if this is simpler than a couple flags or case on enum >> value). > > > [todo] Let me process the above suggestion block a bit later. [x] To the need of xid_cache_search() etc, let me combine this part to reply to the 2nd review mail's notes around a Dummy XID though ... (read on) > >> >>> + if (thd->rgi_slave && thd->rgi_slave->is_parallel_exec) >>> + { >>> + DBUG_ASSERT(thd->transaction->xid_state.is_dummy_XA()); >>> + >>> + auto xs= xid_cache_search(thd, xid); >>> + if(!xs) >>> + { >>> + my_error(ER_XAER_NOTA, MYF(0)); >>> + return; >>> + } >>> + else >>> + { >>> + thd->transaction->xid_state.xid_cache_element= xs; >>> + } >>> + DBUG_ASSERT(thd->transaction->xid_state.is_recovered()); >>> + } >>> + >>> plugin_foreach(NULL, commit ? xacommit_handlerton : >>> xarollback_handlerton, >>> MYSQL_STORAGE_ENGINE_PLUGIN, &xaop); >>> xid_cache_delete(thd); >> >> Why this check for thd->rgi_slave->is_parallel_exec here deep in >> binlog >> group commit? That seems very wrong, and again no comment whatsoever >> explaining what is going on, so again I have to try to guess... >> >> I'm guessing that it's something with calling xid_cache_delete() which >> needs >> to find thd->transaction->xid_state.xid_cache_element ? > > Right. > >> >> But this could be done much better in a higher-level part of the code >> that's >> specific to xa and (parallel) replication, not here deep down in >> binlog >> group commit code. >> >> And why should this be needed only for parallel replication, not for >> non-parellel replication or non-replication SQL? I don't understand. > > > It's a kind of optimization for the slave side, in that `xid` of XAP > (Prepare) gets released for a XAC (Commit) the soonest. > ... the reason's said here. >> >> >> A couple remarks about the tests I pushed to >> knielsen_mdev31949_review: >> > >> >> I think the failures in rpl.rpl_recovery_xa I linked are from >> duplicate XID >> in XA PREPARE ; XA COMMIT ; XA PREPARE. There comes an ambiguity >> between >> whether the transaction prepared in the engine at recovery is from the >> first >> XA PREPARE (so it should be committed) or the second (so it should be >> rolled >> back). This happens when the second XA PREPARE is in the engine but >> not >> binlogged. >> >> One way to fix could actually be to use the binlog position stored >> inside >> InnoDB. That's a neat reminder on the Innodb durably committed binlog position! Indeed with the duplicate xid case there exists this trouble of identification and the Engine is able and must come to rescue. >> If the XA COMMIT is before this binlog position, then we know the >> first transaction became durable in InnoDB, so we know we have to roll >> back >> the current prepared transaction in the engine. Otherwise we should >> commit >> it. I see the recovery code already uses binlog positions to decide >> the fate >> of pending XA transactions in the engines, but I did not yet fully >> understand what is going on there. >> >> Another way could be to binlog an extra event before the second XA >> PREPARE >> with the duplicate XID. Well, this does not cover the case of prior to crash XA PREPARE only made to Engine. The above consulting with Innodb resolves the ambiguity generally. >> This event would mean "This xid is now durable >> committed in the engine", so we would know not to commit any prepared >> trx in >> the engine with the same xid. This event would only be needed in case >> of XA >> PREPARE with a duplicate xid. This though requires the prior XA COMMIT >> to >> fsync inside InnoDB first, I think. > > Thanks for this piece of analysis! > [todo] I am processing this a bit later. [x] > >> >> >> Another test failure in rpl.slave_provision_xa I think happens because >> the >> XA PREPARE does not update the binlog position inside innodb. >> Maybe the way >> to solve this problem is to have the slave be idempotent, and silently >> ignore any duplicate XA PREPARE if the xid is already prepared? Indeed, the fact that XAP does not update the innodb's binlog position must cause few concerns. I still need to have a close look. [todo] > > [todo] I am processing this a bit later. [x] > >> >> >> About the "ToDo" of using some Xid_list_log_event. I think this is a >> way >> during recovery to get the state of the XA PREPARE/COMMIT's in earlier >> binlog files before the last binlog checkpoint, if I understood things >> correctly? > > True. > >> >> Another way to do this is to delay the binlog checkpoints until all XA >> PREPARE in a binlog file have been XA COMMIT'ted (or XA ROLLBACK'ed), >> so >> that they will all be scanned during recovery. I actually implemented >> this >> in my branch knielsen_mdev32020. The advantage of this is that we may >> anyway >> need this to provide the XA PREPARE events to a slave set up from a >> logical >> backup (like mysqldump), to avoid purging XA PREPARE events too early. >> And >> then it's a bit simpler to simply scan an extra file rather than both >> scan >> and read Xid_list_log_event. But I think Xid_list_log_event could work >> fine >> as well, just mentioning it for you to consider. > > Xlle can't be generally avoided so we have to take conservatively > first. > To add up to a separate thread discussion over this problematics, I've grown more inclined with the first implementation basing on your idea to defer Binlog-CheckPoint. That might work for a majority of XA users, Xid list to be added later if time permits or will be convincingly :-) requested. Now I am turning to account the rest - 2,3 reviews you sent that only glanced through - to reply within few hours. Cheers, Andrei

2 2

Re: 64-bit time_t support in MariaDB?
by Otto Kekäläinen 12 Feb '24

12 Feb '24

> it's not related to time, it's just an assert that's verified at compile > time, that is, during compilation. Like static_assert in C++. > > I just need to find 32-bit arm somewhere. Unfortunately I don't have any with SSH access to give. Seems the reported of MDEV-33429 has armhf hardware and is helping out. Once you have a patch, I can push it to Launchpad builders (or you can do it yourself, anyone with an Launchpad can push code to their own PPA). Related, Cfarm is a project that gives free accounts to open source developers in case you want to have access to play around with multiple architectures: https://portal.cfarm.net/machines/list/ Unfortunately they don't seem to have armhf/armel.

2 1

Re2: thd_binlog_pos / Re: Starting on review of XA patches in bb-10.6-MDEV-31949
by andrei.elkin＠pp.inet.fi 08 Feb '24

08 Feb '24

>> So you're in this context >> >> 1 innobase_commit_by_xid( > >> 18 thd_binlog_pos(thd, &trx->mysql_log_file_name, >> 19 &trx->mysql_log_offset); >> 20 if (trx->mysql_log_offset > 0) >> 21 trx->flush_log_later = true; >> >> and actually considering `trx->mysql_log_offset` as the return value? >> Could you please expand on? > > My point was just, that what we want to do here is for the (internal) > transaction coordinator to tell the storage engine that the engine does > not > need to fsync() for the commit to be durable, another fsync was already > done > to ensure this. This is what we in the normal commit path do by calling > commit_ordered(), as documented sql/handler.h. > > The thd_binlog_pos() is completely unrelated, it's a way for the > storage... I see your point. My patch's block of if (trx_t* trx = trx_get_trx_by_xid(xid)) { + THD *thd = current_thd; + if (thd) + thd_binlog_pos(thd, &trx->mysql_log_file_name, + &trx->mysql_log_offset); + if (trx->mysql_log_offset > 0) + trx->flush_log_later = true; + /* use cases are: disconnected xa, slave xa, recovery */ innobase_commit_low(trx); + trx->mysql_log_file_name = NULL; should not indeed rely on anything that thd_binlog_pos() computes. I think I wanted to spare --skip-log-bin setups from the flush_log_later policy. I see now that alternatively thd_binlog_format(thd) could be used which will always return BINLOG_FORMAT_UNSPEC when that's the case (or @@skip_log_bin=1). You think it's better to covert the block to use that?

1 0

Starting on review of XA patches in bb-10.6-MDEV-31949
by Kristian Nielsen 07 Feb '24

07 Feb '24

Hi Andrei, It's not an easy review of the XA patches in bb-10.6-MDEV-31949. One thing that would have helped is a clear design document that lists all the different concerns and points that need to be considered around recovery, backup, replication, possible conflicts, etc. This could then be referenced by the reviewer to see if a given concern was considered, and if so what the resolution is. But I'll try as best I can with what we have. As I'm trying to understand the different issues, I found making test cases to check my understanding helped. I have pushed some to the branch knielsen_mdev31949_review . They seem to show some problems in the current patch: mysql-test/suite/rpl/t/rpl_recovery_xa.test - A trx is left in "XA PREPARED" state after crash recovery, even though the binlogged event group is incomplete (it should have been rolled back). - A trx is committed after crash recovery, even though it is not binlogged (it should have been rolled back). - An incomplete XA event group is output by mysqlbinlog with a "ROLLBACK" at the end which gives an error (should probably be XA ROLLBACK). mysql-test/suite/rpl/t/rpl_recovery_xa2.test - An XA PREPAREd transaction is lost after recovery when it was binlogged in an earlier binlog file. (I think this might be the issue that's referenced as a ToDo with Xid_list_log_event in patch?) mysql-test/suite/mariabackup/slave_provision_xa.test - An XA PREPARE does not update the binlog position inside InnoDB. So a slave provisioned with mariabackup from this starting position can fail with duplicate apply of XA PREPARE. Some other initial remarks follow. Since there's a desire to progress with this quickly, I thought it would be useful to send these early, so you can start looking/discussions before I complete building an understanding of the whole design and code picture. > diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc > index 17e512bf34e..a4cdc72e18c 100644 > --- a/storage/innobase/handler/ha_innodb.cc > +++ b/storage/innobase/handler/ha_innodb.cc > @@ -17144,8 +17144,16 @@ innobase_commit_by_xid( > } > > if (trx_t* trx = trx_get_trx_by_xid(xid)) { > + THD *thd = current_thd; > + if (thd) > + thd_binlog_pos(thd, &trx->mysql_log_file_name, > + &trx->mysql_log_offset); I guess this is necessary to get the correct binlog position stored inside innodb to match the binlogged XA COMMIT event, right? But will it work if called during crash recovery? During crash recovery, we could actually update the position, but only if this is trx is the last event group in the old binlog. Maybe that's better handled elsewhere (there are currently other cases where the binlog position in InnoDB can be wrong immediately after restart until the first new transaction is committed). > + if (trx->mysql_log_offset > 0) > + trx->flush_log_later = true; But I'm guessing this is the motivation. If called from within binlog group commit, you want to disable fsync in InnoDB. But if called from recovery, or when binlog is disabled, this condition will be false, and fsync will be done. If so, that's a _really_ obfuscated way to do it :-/ It surely needs a very clear comment explaining what is going on. And it is very fragile. What if thd_binlog_pos() is changed at some point, to slightly change the conditions under which it returns 0 or not? That could silently break this code. It would be much better if this could be done similar to how normal commits are done. We have the commit_ordered handlerton which does the fast part of commit-to-memory. And then the slow part which does the rest in commit handlerton. So we could have commit_by_xid_ordered() and commit_by_xid(). The problem is that commit_by_xid_ordered() needs to release the xid, so it becomes harder to get hold of the transaction (which the commit handlerton just finds from the thd). Maybe commit_by_xid_ordered() would need to return some trx handle. What part of the commit is skipped by this in InnoDB? Is it _only_ flushing the log up to the LSN of the commit record, or are there more things? How is another storage engine supposed to handle this? This also needs to be documented somewhere, eg. in comments on the handlerton calls. This also means that the --innodb-flush-log-at-trx-commit=3 is not supported for XA COMMIT when it's done from a different session that the XA PREPARE. For example, with --innodb-flush-log-at-trx-commit=3 but --sync-binlog=0, normal transactions are still durable in InnoDB. But external XA transactions will not be. This could be solved if the commit_by_xid() was split into a fast and slow part similar to normal commit. Probably you'll say that this works for now, and perhaps that's true. But at least it really _really_ needs some detailed comments explaining what is going on, and what should be kept in mind if modifying/improving this code in the future, otherwise it will be impossible for the next developer to understand. > diff --git a/sql/log.cc b/sql/log.cc > index e54d4087d46..7e3dbeaef29 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -9365,18 +9381,84 @@ TC_LOG::run_commit_ordered(THD *thd, bool all) > all ? thd->transaction->all.ha_list : thd->transaction->stmt.ha_list; > > mysql_mutex_assert_owner(&LOCK_commit_ordered); > - for (; ha_info; ha_info= ha_info->next()) > + > + if (!ha_info) Your patch puts a lot of XA/binlog logic into run_commit_ordered(). But that's not appropriate. The run_commit_ordered() function has the purpose solely to call the commit_ordered() handlerton in each engine, and it's also called from TC_LOG_MMAP (that's why it's TC_LOG::, not MYSQL_BIN_LOG::). Instead make a new function that handles the part of binlog commit that needs to be done under LOCK_commit_ordered. Be sure to comment it that this runs under this highly contended mutex, and needs to do as little work as possible. This function can then call run_commit_ordered(). It can also be used to share a few bits of code that are now duplicated between the opt_optimize_thread_scheduling and !opt_optimize_thread_scheduling cases, like: ++num_commits; run_commit_ordered(...) entry->thd->wakeup_subsequent_commits() if (entry->queued_by_other) { ... } (if it makes sense to do so in the actual code, only if makes things simpler, that's easier to decide when working with the actual code). There sesems to be 3 new cases: - XA PREPARE - XA COMMIT from same session as XA PREPARE - XA COMMIT of detached XA transaction The logic to decide this was already done earlier, when the event group was queued for group commit, eg. to decide which end_event to use. For example there is no need to call xid_cache_search() here, that can be done earlier (and it's also wrong to call my_error() here, as we're in the wrong thread context). So the logic about what to do should be placed higher up in the call stack, ideally where we already know, eg. binlog_commit_by_xid(). And errors such as ER_XAER_NOTA can be thrown already there, and the decision on what to do put consisely as a flag into the struct group_commit_entry, so the logic during binlog group commit becomes as simple as possible. (It might even make sense to simply store a function pointer to the method that should be called to handle this; normal, attached XA COMMIT, detached XA COMMIT, XA PREPARE. But again it's easier when actually working with the code to decide if this is simpler than a couple flags or case on enum value). > + if (thd->rgi_slave && thd->rgi_slave->is_parallel_exec) > + { > + DBUG_ASSERT(thd->transaction->xid_state.is_dummy_XA()); > + > + auto xs= xid_cache_search(thd, xid); > + if(!xs) > + { > + my_error(ER_XAER_NOTA, MYF(0)); > + return; > + } > + else > + { > + thd->transaction->xid_state.xid_cache_element= xs; > + } > + DBUG_ASSERT(thd->transaction->xid_state.is_recovered()); > + } > + > plugin_foreach(NULL, commit ? xacommit_handlerton : xarollback_handlerton, > MYSQL_STORAGE_ENGINE_PLUGIN, &xaop); > xid_cache_delete(thd); Why this check for thd->rgi_slave->is_parallel_exec here deep in binlog group commit? That seems very wrong, and again no comment whatsoever explaining what is going on, so again I have to try to guess... I'm guessing that it's something with calling xid_cache_delete() which needs to find thd->transaction->xid_state.xid_cache_element ? But this could be done much better in a higher-level part of the code that's specific to xa and (parallel) replication, not here deep down in binlog group commit code. And why should this be needed only for parallel replication, not for non-parellel replication or non-replication SQL? I don't understand. A couple remarks about the tests I pushed to knielsen_mdev31949_review: I think the failures in rpl.rpl_recovery_xa I linked are from duplicate XID in XA PREPARE ; XA COMMIT ; XA PREPARE. There comes an ambiguity between whether the transaction prepared in the engine at recovery is from the first XA PREPARE (so it should be committed) or the second (so it should be rolled back). This happens when the second XA PREPARE is in the engine but not binlogged. One way to fix could actually be to use the binlog position stored inside InnoDB. If the XA COMMIT is before this binlog position, then we know the first transaction became durable in InnoDB, so we know we have to roll back the current prepared transaction in the engine. Otherwise we should commit it. I see the recovery code already uses binlog positions to decide the fate of pending XA transactions in the engines, but I did not yet fully understand what is going on there. Another way could be to binlog an extra event before the second XA PREPARE with the duplicate XID. This event would mean "This xid is now durable committed in the engine", so we would know not to commit any prepared trx in the engine with the same xid. This event would only be needed in case of XA PREPARE with a duplicate xid. This though requires the prior XA COMMIT to fsync inside InnoDB first, I think. Another test failure in rpl.slave_provision_xa I think happens because the XA PREPARE does not update the binlog position inside innodb. Maybe the way to solve this problem is to have the slave be idempotent, and silently ignore any duplicate XA PREPARE if the xid is already prepared? About the "ToDo" of using some Xid_list_log_event. I think this is a way during recovery to get the state of the XA PREPARE/COMMIT's in earlier binlog files before the last binlog checkpoint, if I understood things correctly? Another way to do this is to delay the binlog checkpoints until all XA PREPARE in a binlog file have been XA COMMIT'ted (or XA ROLLBACK'ed), so that they will all be scanned during recovery. I actually implemented this in my branch knielsen_mdev32020. The advantage of this is that we may anyway need this to provide the XA PREPARE events to a slave set up from a logical backup (like mysqldump), to avoid purging XA PREPARE events too early. And then it's a bit simpler to simply scan an extra file rather than both scan and read Xid_list_log_event. But I think Xid_list_log_event could work fine as well, just mentioning it for you to consider. That's it for now, I will continue trying to understand what is going on and complete the review. - Kristian.

1 2

Re: Xlle question
by andrei.elkin＠pp.inet.fi 06 Feb '24

06 Feb '24

>> Xlle can't be generally avoided so we have to take conservatively >> first. > > Why it cannot be avoided? What is it needed for? The main purpose is to discern at recovery a prepared XA (XAP) that was never binlogged from one that was. Your case >>> Another way to do this is to delay the binlog checkpoints until all >>> XA >>> PREPARE in a binlog file have been XA COMMIT'ted assumes that XAC must always come in "normal" time. But of course, being a part of the user's decision, it can be late significantly up until a purge from binary logs that contain the incl the total RESET MASTER. In the latter case its eventual post RM XA-COMMIT would be bin-logged with the full awareness.

1 0

Fwd/Re: Starting on review of XA patches in bb-10.6-MDEV-31949
by andrei.elkin＠pp.inet.fi 05 Feb '24

05 Feb '24

From: Andrei Elkin <andrei.elkin(a)mariadb.com> Subject: Re: Starting on review of XA patches in bb-10.6-MDEV-31949 To: Kristian Nielsen <knielsen(a)knielsen-hq.org> Cc: developers(a)lists.mariadb.org, andrei elkin <andrei.elkin(a)mariadb.com>, Michael Widenius <michael.widenius(a)gmail.com> Date: Mon, 05 Feb 2024 14:11:01 +0200 (36 minutes, 34 seconds ago) Organization: MariaDB Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > Hi Andrei, > > It's not an easy review of the XA patches in bb-10.6-MDEV-31949. > > One thing that would have helped is a clear design document that lists > all > the different concerns and points that need to be considered around > recovery, backup, replication, possible conflicts, etc. This could then > be > referenced by the reviewer to see if a given concern was considered, > and if > so what the resolution is. I am sorry for discomfort the lack of the above caused. Unfortunately the IV recovery part has consumed more time than was expected (which is at the final stage of my local testing). > But I'll try as best I can with what we > have. If you'll need any online discussion to compensate for that I'd be glad to do it. > > As I'm trying to understand the different issues, I found making test > cases > to check my understanding helped. I have pushed some to the branch > knielsen_mdev31949_review. Thank you for streamlining it this way! > They seem to show some problems in the current > patch: > > mysql-test/suite/rpl/t/rpl_recovery_xa.test > - A trx is left in "XA PREPARED" state after crash recovery, even > though > the binlogged event group is incomplete (it should have been > rolled > back). > - A trx is committed after crash recovery, even though it is not > binlogged (it should have been rolled back). > - An incomplete XA event group is output by mysqlbinlog with a > "ROLLBACK" > at the end which gives an error (should probably be XA ROLLBACK). > > mysql-test/suite/rpl/t/rpl_recovery_xa2.test > - An XA PREPAREd transaction is lost after recovery when it was > binlogged > in an earlier binlog file. (I think this might be the issue that's > referenced as a ToDo with Xid_list_log_event in patch?) > > mysql-test/suite/mariabackup/slave_provision_xa.test > - An XA PREPARE does not update the binlog position inside InnoDB. > So a > slave provisioned with mariabackup from this starting position can > fail > with duplicate apply of XA PREPARE. > > Some other initial remarks follow. Since there's a desire to progress > with > this quickly, I thought it would be useful to send these early, so you > can > start looking/discussions before I complete building an understanding > of the > whole design and code picture. Super! > > >> diff --git a/storage/innobase/handler/ha_innodb.cc >> b/storage/innobase/handler/ha_innodb.cc >> index 17e512bf34e..a4cdc72e18c 100644 >> --- a/storage/innobase/handler/ha_innodb.cc >> +++ b/storage/innobase/handler/ha_innodb.cc >> @@ -17144,8 +17144,16 @@ innobase_commit_by_xid( >> } >> >> if (trx_t* trx = trx_get_trx_by_xid(xid)) { >> + THD *thd = current_thd; >> + if (thd) >> + thd_binlog_pos(thd, &trx->mysql_log_file_name, >> + &trx->mysql_log_offset); > > I guess this is necessary to get the correct binlog position stored > inside > innodb to match the binlogged XA COMMIT event, right? Correct. > But will it work if called during crash recovery? Right. The unpushed part IV makes sure of that. > > During crash recovery, we could actually update the position, but only > if > this is trx is the last event group in the old binlog. Maybe that's > better > handled elsewhere (there are currently other cases where the binlog > position > in InnoDB can be wrong immediately after restart until the first new > transaction is committed). > >> + if (trx->mysql_log_offset > 0) >> + trx->flush_log_later = true; > > But I'm guessing this is the motivation. If called from within binlog > group > commit, you want to disable fsync in InnoDB. But if called from > recovery, or > when binlog is disabled, this condition will be false, and fsync will > be > done. Indeed. > > If so, that's a _really_ obfuscated way to do it :-/ I remember it was a very quick decision for myself to miss this sort of reflection. [todo] I'll add comments of course. > It surely needs a very clear comment explaining what is going on. > And it is very fragile. What if thd_binlog_pos() is changed at some > point, > to slightly change the conditions under which it returns 0 or not? That > could silently break this code. So you're in this context 1 innobase_commit_by_xid( 2 /*===================*/ 3 handlerton* hton, 4 XID* xid) /*!< in: X/Open XA transaction identification */ 5 { 6 DBUG_ASSERT(hton == innodb_hton_ptr); 7 8 DBUG_EXECUTE_IF("innobase_xa_fail", 9 return XAER_RMFAIL;); 10 11 if (high_level_read_only) { 12 return(XAER_RMFAIL); 13 } 14 15 if (trx_t* trx = trx_get_trx_by_xid(xid)) { 16 THD *thd = current_thd; 17 if (thd) 18 thd_binlog_pos(thd, &trx->mysql_log_file_name, 19 &trx->mysql_log_offset); 20 if (trx->mysql_log_offset > 0) 21 trx->flush_log_later = true; and actually considering `trx->mysql_log_offset` as the return value? ... I could not grasp your way of thinking in this place. Could you please expand on? > > It would be much better if this could be done similar to how normal > commits > are done. We have the commit_ordered handlerton which does the fast > part of > commit-to-memory. And then the slow part which does the rest in commit > handlerton. Notice that the *slow* part of the XA commit is the same as in the normal case: trx_commit_complete_for_mysql *innobase_commit* commit_one_phase_2 ha_commit_one_phase The fast parts of the two - the XA and normal - could be unified - into innobase_commit_ordered() but I did not think (had time for) towards that. > > So we could have commit_by_xid_ordered() and commit_by_xid(). The > problem is > that commit_by_xid_ordered() needs to release the xid, so it becomes > harder > to get hold of the transaction (which the commit handlerton just finds > from > the thd). Maybe commit_by_xid_ordered() would need to return some trx > handle. > > What part of the commit is skipped by this in InnoDB? Is it _only_ > flushing > the log up to the LSN of the commit record, or are there more things? So the xa's fast innobase_commit_by_xid() skips a "slow" trx_commit_complete_for_mysql() and the same does the normal's innobase_commit_ordered(). > > How is another storage engine supposed to handle this? This also needs > to be > documented somewhere, eg. in comments on the handlerton calls. > > This also means that the --innodb-flush-log-at-trx-commit=3 is not > supported > for XA COMMIT when it's done from a different session that the XA > PREPARE. > For example, with --innodb-flush-log-at-trx-commit=3 but > --sync-binlog=0, > normal transactions are still durable in InnoDB. But external XA > transactions will not be. This could be solved if the commit_by_xid() > was > split into a fast and slow part similar to normal commit. [todo] Let me process that carefully later. > > Probably you'll say that this works for now, and perhaps that's true. > But at > least it really _really_ needs some detailed comments explaining what > is > going on, and what should be kept in mind if modifying/improving this > code > in the future, otherwise it will be impossible for the next developer > to > understand. [todo] Agreed. > > >> diff --git a/sql/log.cc b/sql/log.cc >> index e54d4087d46..7e3dbeaef29 100644 >> --- a/sql/log.cc >> +++ b/sql/log.cc > >> @@ -9365,18 +9381,84 @@ TC_LOG::run_commit_ordered(THD *thd, bool all) >> all ? thd->transaction->all.ha_list : >> thd->transaction->stmt.ha_list; >> >> mysql_mutex_assert_owner(&LOCK_commit_ordered); >> - for (; ha_info; ha_info= ha_info->next()) >> + >> + if (!ha_info) > > Your patch puts a lot of XA/binlog logic into run_commit_ordered(). But > that's not appropriate. The run_commit_ordered() function has the > purpose > solely to call the commit_ordered() handlerton in each engine, and it's > also > called from TC_LOG_MMAP (that's why it's TC_LOG::, not > MYSQL_BIN_LOG::). > > Instead make a new function that handles the part of binlog commit that > needs to be done under LOCK_commit_ordered. Be sure to comment it that > this > runs under this highly contended mutex, and needs to do as little work > as > possible. > > This function can then call run_commit_ordered(). It can also be used > to > share a few bits of code that are now duplicated between the > opt_optimize_thread_scheduling and !opt_optimize_thread_scheduling > cases, > like: > > ++num_commits; > run_commit_ordered(...) > entry->thd->wakeup_subsequent_commits() > if (entry->queued_by_other) { ... } > > (if it makes sense to do so in the actual code, only if makes things > simpler, that's easier to decide when working with the actual code). > > There sesems to be 3 new cases: > > - XA PREPARE > - XA COMMIT from same session as XA PREPARE > - XA COMMIT of detached XA transaction > > The logic to decide this was already done earlier, when the event group > was > queued for group commit, eg. to decide which end_event to use. For > example > there is no need to call xid_cache_search() here, that can be done > earlier > (and it's also wrong to call my_error() here, as we're in the wrong > thread > context). > > So the logic about what to do should be placed higher up in the call > stack, > ideally where we already know, eg. binlog_commit_by_xid(). And errors > such > as ER_XAER_NOTA can be thrown already there, and the decision on what > to do > put consisely as a flag into the struct group_commit_entry, so the > logic > during binlog group commit becomes as simple as possible. > > (It might even make sense to simply store a function pointer to the > method > that should be called to handle this; normal, attached XA COMMIT, > detached > XA COMMIT, XA PREPARE. But again it's easier when actually working with > the > code to decide if this is simpler than a couple flags or case on enum > value). [todo] Let me process the above suggestion block a bit later. > >> + if (thd->rgi_slave && thd->rgi_slave->is_parallel_exec) >> + { >> + DBUG_ASSERT(thd->transaction->xid_state.is_dummy_XA()); >> + >> + auto xs= xid_cache_search(thd, xid); >> + if(!xs) >> + { >> + my_error(ER_XAER_NOTA, MYF(0)); >> + return; >> + } >> + else >> + { >> + thd->transaction->xid_state.xid_cache_element= xs; >> + } >> + DBUG_ASSERT(thd->transaction->xid_state.is_recovered()); >> + } >> + >> plugin_foreach(NULL, commit ? xacommit_handlerton : >> xarollback_handlerton, >> MYSQL_STORAGE_ENGINE_PLUGIN, &xaop); >> xid_cache_delete(thd); > > Why this check for thd->rgi_slave->is_parallel_exec here deep in binlog > group commit? That seems very wrong, and again no comment whatsoever > explaining what is going on, so again I have to try to guess... > > I'm guessing that it's something with calling xid_cache_delete() which > needs > to find thd->transaction->xid_state.xid_cache_element ? Right. > > But this could be done much better in a higher-level part of the code > that's > specific to xa and (parallel) replication, not here deep down in binlog > group commit code. > > And why should this be needed only for parallel replication, not for > non-parellel replication or non-replication SQL? I don't understand. It's a kind of optimization for the slave side, in that `xid` of XAP (Prepare) gets released for a XAC (Commit) the soonest. > > > A couple remarks about the tests I pushed to knielsen_mdev31949_review: > > > I think the failures in rpl.rpl_recovery_xa I linked are from duplicate > XID > in XA PREPARE ; XA COMMIT ; XA PREPARE. There comes an ambiguity > between > whether the transaction prepared in the engine at recovery is from the > first > XA PREPARE (so it should be committed) or the second (so it should be > rolled > back). This happens when the second XA PREPARE is in the engine but not > binlogged. > > One way to fix could actually be to use the binlog position stored > inside > InnoDB. If the XA COMMIT is before this binlog position, then we know > the > first transaction became durable in InnoDB, so we know we have to roll > back > the current prepared transaction in the engine. Otherwise we should > commit > it. I see the recovery code already uses binlog positions to decide the > fate > of pending XA transactions in the engines, but I did not yet fully > understand what is going on there. > > Another way could be to binlog an extra event before the second XA > PREPARE > with the duplicate XID. This event would mean "This xid is now durable > committed in the engine", so we would know not to commit any prepared > trx in > the engine with the same xid. This event would only be needed in case > of XA > PREPARE with a duplicate xid. This though requires the prior XA COMMIT > to > fsync inside InnoDB first, I think. Thanks for this piece of analysis! [todo] I am processing this a bit later. > > > Another test failure in rpl.slave_provision_xa I think happens because > the > XA PREPARE does not update the binlog position inside innodb. Maybe the > way > to solve this problem is to have the slave be idempotent, and silently > ignore any duplicate XA PREPARE if the xid is already prepared? [todo] I am processing this a bit later. > > > About the "ToDo" of using some Xid_list_log_event. I think this is a > way > during recovery to get the state of the XA PREPARE/COMMIT's in earlier > binlog files before the last binlog checkpoint, if I understood things > correctly? True. > > Another way to do this is to delay the binlog checkpoints until all XA > PREPARE in a binlog file have been XA COMMIT'ted (or XA ROLLBACK'ed), > so > that they will all be scanned during recovery. I actually implemented > this > in my branch knielsen_mdev32020. The advantage of this is that we may > anyway > need this to provide the XA PREPARE events to a slave set up from a > logical > backup (like mysqldump), to avoid purging XA PREPARE events too early. > And > then it's a bit simpler to simply scan an extra file rather than both > scan > and read Xid_list_log_event. But I think Xid_list_log_event could work > fine > as well, just mentioning it for you to consider. Xlle can't be generally avoided so we have to take conservatively first. > > > That's it for now, I will continue trying to understand what is going > on and > complete the review. Thank you for so prompt piece of feedback! I'll return with a reply within a work day to [todo] tagged. Cheers, Andrei ----------

2 1

Review of MDEV-7850: Extend GTID Binlog Events with Thread Id
by Kristian Nielsen 31 Jan '24

31 Jan '24

> commit c37b2087b4abe576f1b0391c8d379dba6299dcb5 (origin/bb-11.4-MDEV-7850) > Author: Brandon Nesterenko <brandon.nesterenko(a)mariadb.com> > Date: Mon Jul 10 18:53:19 2023 +0300 > > MDEV-7850: Extend GTID Binlog Events with Thread Id > > This patch augments Gtid_log_event with the user thread-id. > In particular that compensates for the loss of this info in > Rows_log_events. Here's my review of this patch (which was already pushed, but there are some misunderstandings that should be fixed; Monty asked me about it as he had been told that I had reviewed the patch, but that is not the case (until now)): > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 41ddc038594..f167284ad1d 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -1901,11 +1904,12 @@ Query_log_event::begin_event(String *packet, ulong ev_offset, > /* > Currently we only need to replace GTID event. > The length of GTID differs depending on whether it contains commit id. > + And/or thread id. > */ > - DBUG_ASSERT(data_len == LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN || > - data_len == LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2); > - if (data_len != LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN && > - data_len != LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2) > + DBUG_ASSERT(data_len >= LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN && > + data_len <= LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2 + 9); > + if (data_len < LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN || > + data_len > LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2 + 9) This change of the asserts/checks doesn't make any sense. They were there originally, because we are hand-crafting a BEGIN query event, and there were only two cases of the length, so two fixed-sized events were crafted by two different code paths. And the asserts are there to safe-guard if the length would change. But now this check was already wrong, as more optional info has been added to the GTID event without updating the code for Query_log_event::begin_event() (XID, flags_extra, extra_engines). But the new code in this patch is generic and can handle any length GTID event, so these checks no longer make any sense, and should just be removed (rather than attempt to correct them). > @@ -1926,9 +1930,19 @@ Query_log_event::begin_event(String *packet, ulong ev_offset, > } > else > { > - DBUG_ASSERT(data_len == LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2); > - /* Put in an empty time_zone_str to take up the extra 2 bytes. */ > - int2store(q + Q_STATUS_VARS_LEN_OFFSET, 2); > + DBUG_ASSERT(data_len <= LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 11); This assert should not be there. The length can be larger than 11, and the code below works fine for arbitrary length. > + DBUG_ASSERT(data_len >= LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2); > + > + /* Put in an empty time_zone_str to take up the extra 2 plus the number of > + dummy_bytes. */ > + size_t dummy_bytes= > + data_len - (LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN + 2); > + > + int2store(q + Q_STATUS_VARS_LEN_OFFSET, dummy_bytes + 2); > + for (size_t i= 0; i < dummy_bytes; i++) > + q[Q_DATA_OFFSET + i]= Q_DUMMY; > + q+= dummy_bytes; > + > q[Q_DATA_OFFSET]= Q_TIME_ZONE_CODE; > q[Q_DATA_OFFSET+1]= 0; /* Zero length for empty time_zone_str */ > q[Q_DATA_OFFSET+2]= 0; /* Zero terminator for empty db */ This new code with Q_DUMMY is generic and independent of the length of the GTID event. That's good! But it needs a comment explaining (that the Q_DUMMY is unknown to the old slave, which will make the old slave just skip the rest of the user vars, which are just there to fill up the extra bytes). And now the code that adds the dummy empty time_zone_str is redundant, and should just be removed, as should the code for the case of data_len == LOG_EVENT_HEADER_LEN + GTID_HEADER_LEN. Since now any extra bytes in the GTID can just be replaced by Q_DUMMY. This way, the code will be fixed to work for all the possible GTID lengths, and be shorter and simpler (and a lot less confusing). > @@ -1584,6 +1584,9 @@ Query_log_event::Query_log_event(const uchar *buf, uint event_len, > sa_seq_no = uint8korr(pos); > pos+= 8; > } > + } > + case Q_DUMMY: > + { > break; > } > default: > /* That's why you must write status vars in growing order of code */ > DBUG_PRINT("info",("Query_log_event has unknown status vars (first has code: %u), skipping the rest of them", (uint) *(pos-1))); > pos= (const uchar*) end; // Break loop There's no point in having this code. Q_DUMMY is supposed to be higher than any known status var type, so it will be ignored anyway in the `default` case. And besides, server that knows about Q_DUMMY can never receive it in the first place. (But if this is really wanted, then don't delete the `break` statement in the prior switch case, that's likely to introduce a bug the next time a case is added. And assign Q_DUMMY the next free value, not the value 255). > diff --git a/sql/log_event.h b/sql/log_event.h > index 473506fafed..d324642ad5e 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -324,6 +324,7 @@ class String; > > #define Q_HRNOW 128 > #define Q_XID 129 > +#define Q_DUMMY 255 This needs a good comment, explaining how Q_DUMMY is a code that will be unknown to any server version, and can be used to make the slave skip this (and any following) user vars. diff --git a/mysql-test/suite/rpl/t/rpl_gtid_thread_id.test b/mysql-test/suite/rpl/t/rpl_gtid_thread_id.test new file mode 100644 index 00000000000..b8ed1865da9 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_gtid_thread_id.test @@ -0,0 +1,125 @@ +# +# Verify that GTID log events are written into the binary log along with the +# id of the thread which executed the transaction. On the primary, this is the <snip> Disclaimer: I did not review this testcase. - Kristian.

2 3

Slow or hanging parallel replication in 10.6
by Kristian Nielsen 30 Jan '24

30 Jan '24

As we are discussing possible problems with XA parallel replication being slow or hanging in 10.6, I just want to remind of these recently fixed bugs: MDEV-31482: Lock wait timeout with INSERT-SELECT, autoinc, and statement-based replication This can cause slow or hanging parallel replication, as it hangs for --innodb-lock-wait-timeout MDEV-32096 Parallel replication lags because innobase_kill_query() may fail to interrupt a lock wait This can cause show or hanging parallel replication MDEV-31655: Parallel replication deadlock victim preference code errorneously removed This can cause parallel replication to be slow or fail with "too many retries" These bugs were fixed in 10.6.16. These bugs are not related to XA. But the symptoms of these bugs are that optimistic parallel replication occasionally is slow, hangs, or breaks under high load. So it is important not to mistakenly assume that these symptoms are necessarily caused by XA, even though the workload that has problems might be using external XA transactions. In versions prior to 10.6.16, these kinds of problems are likely to be caused by the above known bugs. (I got this list from a quick scan through the 10.6 commit history, there may be one or two more like them I missed. But the conclusion is the same, using optimistic parallel replication under high load in 10.6 requires using at least 10.6.16 to work). - Kristian.

1 0

PR#2779: Fix merge commit 5ea5291: No test file or result files should be executable
by Otto Kekäläinen 28 Jan '24

28 Jan '24

Hi Sanja! Did you notice my PR https://github.com/MariaDB/server/pull/2779? I tagged you about a minor bug in you previous merge. I was hoping you would notice it and review that PR, but Daniel Black jumped on it. I am asking now because it seems the exact same thing happened for mysql-test/main/lowercase_table2.result in your commit fecd78b8378 from November 8th, so I thought I'd send an extra email now to ensure you are aware of this happening. - Otto

2 3

Suggestions for the problems around replication of XA in 10.5
by Kristian Nielsen 27 Jan '24

27 Jan '24

Hi Monty, As promised, here my thoughts around the issues with the implementation of replication of external XA as implemented since MariaDB 10.5. I see at least two architectural issues with the current implementation. One is that it splits transactions in two separate GTIDs in the binlog (XA PREPARE and XA COMMIT). This breaks with the fundamental principle that replication applies transactions one after the other in strict sequence, and the replication position/GTID is the single transaction last replicated. This for example means that a mysqldump backup can no longer be used to provision a slave, since any XA PREPAREd event at the time of the dump will be missing; a testcase rpl_xa_provision.test in MDEV-32020 demonstrates this. Another architectural issue is that each XA PREPARE keeps row locks around on every slave until commit (or rollback). This means replication will break if _any_ transaction replicated after the XA PREPARE gets blocked on a lock. This can easily happen; surely in many ways in statement-based replication, and even in row-based replication without primary key as demonstrated by testcase rpl_mdev32020.test in MDEV-32020. There are other problems; for example the update of mysql.gtid_slave_pos cannot be committed crash-safe together with the transaction for XA PREPARE (since the transaction is not committed). I believe the root of the problem is architectural: external XA should be replicated only after they commit on the master. Trying to fix individual problems one by one will not address the root problem and will lead to ever increasing complexity without ever being fully successful. The current implementation appears to only address a very specific and rare use-case, where enhanced semi-synchronous replication is used with row-based binlogging to try to fail-over to a slave and preserve any external XA that was in PREPAREd state on the master before the failover. Mixed-mode replication, provisioning slaves with mysqldump, slaves not intended for failover, etc., seem to be not considered and basically broken since 10.5. Here is my idea for a design that solves most of these problems. At XA PREPARE, we can still write the events to the binlog, but without a GTID, and we do not replicate it to slaves by default. Then at XA COMMIT we binlog a commit transaction that can be replicated normally to the slaves without problems. If necessary, the events for XA COMMIT can be read from the PREPARE earlier in the binlog, eg. after server crash/restart. We already have the binlog checkpoint mechanism to ensure that required binlog files are preserved until no longer needed for transaction recovery. This way we make external XA preserved across server restart, and all normal replication features continue to work - mysqldump, mixed-mode, etc. Nice and simple. Then optionally we can support the specific usecase of being able to recover external XA PREPAREd transactions on a slave after failover. When enabled, the slave can receive the XA PREPARE events and binlog them itself, without applying. Then as part of failover, those XA PREPARE in the binlog that are still pending can be applied, leaving them in PREPAREd state on the new master. This way, _only_ the few transactions that need to be failed-over need special handling, the majority can still just replicate normally. There are different refinements and optimizations that can be added on top of this. But the point is that this is a simple implementation that is robust, correct, and crash-safe from the start, without needing to add complexity and fixes on top. I've done some initial proof-of-concept code for this, and continue to work on it on the branch knielsen_mdev32020 on github. - Kristian.

3 6

Presentation and Interest in "SIMD for SQL Articulations and Capabilities" Undertaking
by Rayan Chowdhury 25 Jan '24

25 Jan '24

Dear MariaDB Developers, I trust this email thinks that you are well. I am Rayan Chowdhury, and I'm an understudy from India as of now seeking Computer Science in NIIT University. I'm excited about the chance to contribute to MariaDB after discovering the "SIMD for SQL Expressions and Functions" project recently. I have a strong fascination with data set frameworks and streamlining, and the objectives of this undertaking adjust impeccably with my goals. Adding vectorized assessment for SQL capabilities and articulations in the MariaDB ColumnStore motor is both captivating and testing, and I'm anxious to be a piece of the work to further develop execution around here. I've started my journey by signing up for the developers' mailing list, and I'm excited to get involved in the MariaDB community. Prior to jumping into commitments, I needed to connect with express my advantage in the undertaking and look for direction on how I can get everything rolling. I'm anticipating adding to MariaDB and gaining from the accomplished designers locally. Much thanks to you for your time, and I'm amped up for the chance of cooperating on this venture. Best respects, Rayan Chowdhury CSE Student, NIIT University

2 1

Re: 1a3974f534f: MDEV-33299 Assertion `(tm->tv_usec % (int) log_10_int[6 - dec]) == 0' failed in void my_timestamp_to_binary(const timeval*, uchar*, uint)
by Sergei Golubchik 24 Jan '24

24 Jan '24

Hi, Alexander, On Jan 24, Alexander Barkov wrote: > revision-id: 1a3974f534f (mariadb-11.3.1-12-g1a3974f534f) > parent(s): 83a79ba33b0 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-01-23 18:04:31 +0400 > message: > > MDEV-33299 Assertion `(tm->tv_usec % (int) log_10_int[6 - dec]) == 0' failed in void my_timestamp_to_binary(const timeval*, uchar*, uint) > > diff --git a/sql/opt_rewrite_date_cmp.cc b/sql/opt_rewrite_date_cmp.cc > index 5a6410414da..8ff0ab3847e 100644 > --- a/sql/opt_rewrite_date_cmp.cc > +++ b/sql/opt_rewrite_date_cmp.cc > @@ -288,10 +288,13 @@ Item *Date_cmp_func_rewriter::create_end_bound() > const_arg_ts.hour= 23; > const_arg_ts.minute= TIME_MAX_MINUTE; > const_arg_ts.second= TIME_MAX_SECOND; > - const_arg_ts.second_part= TIME_MAX_SECOND_PART; > + const_arg_ts.second_part= TIME_MAX_SECOND_PART - > + my_time_fraction_remainder(TIME_MAX_SECOND_PART, > + field_ref->decimals); > if (check_datetime_range(&const_arg_ts)) > return nullptr; > res= new (thd->mem_root) Item_datetime(thd); > + res->decimals= field_ref->decimals; would it work you if always set res->decimals=6 here (and below)? looks like a simpler way to make the item consistent. > res->set(&const_arg_ts); > break; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: af51be1106b: MDEV-32837 long unique does not work like unique key when using replace
by Sergei Golubchik 24 Jan '24

24 Jan '24

Hi, Alexander, Generally, ok to push. One request for a change and one question below: On Jan 23, Alexander Barkov wrote: > revision-id: af51be1106b (mariadb-10.5.23-57-gaf51be1106b) > parent(s): c9902a20b3a > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-01-17 15:51:54 +0400 > message: > > MDEV-32837 long unique does not work like unique key when using replace > > write_record() when performing REPLACE has an optimization: > - if the unique violation happened in the last unique key, then do UPDATE > - otherwise, do DELETE+INSERT > > This patch skips this optimization in case if a non-versioned table > has hash unique (i.e. UNIQUE..USING HASH) constraints. > > Rationale: > > hash unique constraints are checked before regular unique indexes, > so even if a hash unique is the last unique constraint in the > table->key_info[] array - it guarantees nothing - there still > can be regular (non-hash) contraints that violate as well. > So just go the non-optmized path with DELETE followed > by INSERT on the next loop iteration. Don't forget to update the commit comment to match the final fix. > diff --git a/mysql-test/main/long_unique_bugs_no_protocol.test b/mysql-test/main/long_unique_bugs_no_protocol.test > new file mode 100644 > index 00000000000..a6f1f8e57e8 > --- /dev/null > +++ b/mysql-test/main/long_unique_bugs_no_protocol.test > @@ -0,0 +1,61 @@ > +--source include/no_protocol.inc why no protocol? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 0c0721cb2f7: MDEV-31616 Problems with a stored function EMPTY() on upgrade to 10.6.
by Sergei Golubchik 22 Jan '24

22 Jan '24

Hi, Alexey, On Dec 27, Alexey Botchkov wrote: > revision-id: 0c0721cb2f7 (mariadb-10.6.16-45-g0c0721cb2f7) > parent(s): aff5ed3988a > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2023-12-19 00:00:48 +0400 > message: > > MDEV-31616 Problems with a stored function EMPTY() on upgrade to 10.6. > > The IDENT_sys doesn't include keywords, so the fucntion with the typo: fucntion > keyword name can be created, but cannot be called. > Moving some keywords to keyword_funcname set so the functions with these > names are allowed. The current logic in the code is: * built-in functions use IDENT_sys (no keywords allowed) in function_call_generic * functions with keyword names are explicitly listed in function_call_conflict. * stored function invocations use IDENT_sys via function_call_generic but * stored function declaration uses sp_name (that is keyword_ident) rule so, it'd make sense to introduce IDENT_funcname and use it instead of IDENT_sys in function_call_generic. That was a good idea. But also, I expect it to make function_call_conflict redundant, because all names from there should be in IDENT_funcname. And the patch doesn't solve the issue that many keywords from IDENT_sys are not in IDENT_funcname. So it's still possible to create functions that cannot be invoked. Note, that simply changing sp_name to use IDENT_funcname is not correct, because stored procedure names have different limitations. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: 02b85f1cf9b: MDEV-31340 Remove MY_COLLATION_HANDLER::strcasecmp()
by Sergei Golubchik 19 Jan '24

19 Jan '24

Hi, Alexander, Phew. It was a big patch. Mostly mechanical changes, admittedly, but still a lot to read. Generally I quite liked it, great work! Shorter way to write an identifier comparison, type safety that enforces namespaces and correct collations. Nice. But there are, of course, comments and questions, please, see below. On Dec 10, Alexander Barkov wrote: > commit 02b85f1cf9b > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Wed Apr 26 15:27:01 2023 +0400 > > MDEV-31340 Remove MY_COLLATION_HANDLER::strcasecmp() > > - Removing the virtual function strnncoll() from MY_COLLATION_HANDLER > > - Adding a wrapper function CHARSET_INFO::streq(), to compare > two strings for equality. For now it calls strnncoll() internally. > In the future it will turn into a virtual function. why would you want it as a virtual function? What is collation-dependent there? > > - Adding new accent sensitive case insensitive collations: > - utf8mb4_general1400_as_ci > - utf8mb3_general1400_as_ci > They implements accent sensitive case insensitive comparison. > The weight of a character is equal to the code point of its > upper case variant. These collation use Unicode-14.0.0 casefolding data. > > The result of > my_charset_utf8mb3_general1400_as_ci.strcoll() > is very close to the former > my_charset_utf8mb3_general_ci.strcasecmp() > > There is only a difference in a couple dozen rare characters, because: > - the switch from "tolower" to "toupper" comparison, to make > utf8mb3_general1400_as_ci closer to utf8mb3_general_ci > - the switch from Unicode-3.0.0 to Unicode-14.0.0 > This difference should be tolarable. See the list of affected > characters in the MDEV description. > > Note, utf8mb4_general1400_as_ci correctly handles non-BMP characters! > Unlike utf8mb4_general_ci, it does not treat all BMP characters > as equal. > > - Adding classes representing names of the file based database objects: > > Lex_ident_db > Lex_ident_table > Lex_ident_trigger > > Their comparison collation depends on the underlying > file system case sensitivity and on --lower-case-table-names > and can be either my_charset_bin or my_charset_utf8mb3_general1400_as_ci. > > - Adding classes representing names of other database objects, > whose names have case insensitive comparison style, > using my_charset_utf8mb3_general1400_as_ci: > > Lex_ident_column > Lex_ident_sys_var > Lex_ident_user_var > Lex_ident_sp_var I don't think Lex_ident_sp_var is a separate domain, it's the same as Lex_ident_column. Both logically: select a from t1 where b > c (here you don't know what's a column and what's an sp var) and also in the code, in fill_spvar_definition() you yourself cast from Lex_ident_sp_var to Lex_ident_column, which kind of defeats the purpose of having two different types and emphasizes that it's actually just one and the same. > Lex_ident_ps > Lex_ident_i_s_table > Lex_ident_window > Lex_ident_func > Lex_ident_partition > Lex_ident_with_element > Lex_ident_rpl_filter > Lex_ident_master_info > Lex_ident_host > Lex_ident_locale > Lex_ident_plugin > Lex_ident_engine > Lex_ident_server > Lex_ident_savepoint > Lex_ident_table_option_name > Lex_ident_table_option_value option value is a string, not an ident > Lex_ident_charset > > - All the mentioned Lex_ident_xxx classes implement a method streq(): > > if (ident1.streq(ident2)) > do_equal(); > > This method works as a wrapper for CHARSET_INFO::streq(). > > - Changing a lot of "LEX_CSTRING name" to "Lex_ident_xxx name" > in class members and in function/method parameters. > > - Replacing all calls like > system_charset_info->coll->strcasecmp(ident1, ident2) > to > ident1.streq(ident2) > > - Taking advantage of the c++11 user defined literal operator > for Lex_ident_xxx data type. See lex_ident.h. > Use example: > > const Lex_ident_column primary_key_name= "PRIMARY"_Lex_ident_column; > > is now a shorter version of: > > const Lex_ident_column primary_key_name= > Lex_ident_column({STRING_WITH_LEN("PRIMARY")}); Of all the things in your patch this will be the first thing that Monty will remove when he'll stumble upon it. Could you instead implement a constructor that understands const Lex_ident_column primary_key_name= {STRING_WITH_LEN("PRIMARY")}; > diff --git a/include/m_ctype.h b/include/m_ctype.h > index 9d13989d1fe..e5c5145b256 100644 > --- a/include/m_ctype.h > +++ b/include/m_ctype.h > @@ -997,6 +1006,18 @@ struct charset_info_st > return state & MY_CS_COMPILED; > } > > + /* > + There will be a separate virtual function streq() in > + MY_COLLATION_HANDLER eventually. > + */ > + my_bool streq(const LEX_CSTRING a, const LEX_CSTRING b) const > + { > + DBUG_ASSERT(is_valid_comparison_operand(a)); > + DBUG_ASSERT(is_valid_comparison_operand(b)); > + return 0 == (coll->strnncoll)(this, (const uchar *) a.str, a.length, > + (const uchar *) b.str, b.length, FALSE); as you have also added strnncoll helper, you can write the above as return 0 == strnncoll(a, b); also, perhaps you should move these asserts into strnncoll helper? > + } > + > int strnncoll(const uchar *a, size_t alen, > const uchar *b, size_t blen, my_bool b_is_prefix= FALSE) const > { > diff --git a/mysql-test/main/ctype_utf8mb4_general1400_as_ci_casefold.result b/mysql-test/main/ctype_utf8mb4_general1400_as_ci_casefold.result > new file mode 100644 > index 00000000000..c6ccf662d16 > --- /dev/null > +++ b/mysql-test/main/ctype_utf8mb4_general1400_as_ci_casefold.result > @@ -0,0 +1,1446 @@ > +# > +# Start of 11.4 tests > +# > +# > +# MDEV-31340 Remove MY_COLLATION_HANDLER::strcasecmp() > +# > +SET NAMES utf8mb4 COLLATE utf8mb4_general1400_as_ci; > +EXECUTE IMMEDIATE SFORMAT(' > +CREATE VIEW v_bmp AS > +SELECT > + seq AS codepoint, > + LPAD(HEX(seq),4,''0'') AS codepoint_hex4, > + CONVERT(CHAR(seq USING utf32) USING {}) COLLATE {} AS c > +FROM > + seq_0_to_65535', @@character_set_connection, @@collation_connection); > +SELECT COLLATION(c) FROM v_bmp LIMIT 1; > +COLLATION(c) > +utf8mb4_general1400_as_ci > +SELECT > +codepoint_hex4, > +HEX(CAST(LOWER(c) AS CHAR CHARACTER SET ucs2)), > +HEX(CAST(UPPER(c) AS CHAR CHARACTER SET ucs2)) > +FROM v_bmp > +WHERE BINARY(c)<>BINARY(LOWER(c)) OR BINARY(c)<>BINARY(UPPER(c)); > +codepoint_hex4 HEX(CAST(LOWER(c) AS CHAR CHARACTER SET ucs2)) HEX(CAST(UPPER(c) AS CHAR CHARACTER SET ucs2)) what is the point of this table? detect changes when going to the next unicode version? > +0041 0061 0041 ... > +# > +# End of 11.4 tests > +# > diff --git a/mysql-test/main/ctype_utf8mb4_general1400_as_ci_ws.result b/mysql-test/main/ctype_utf8mb4_general1400_as_ci_ws.result > new file mode 100644 > index 00000000000..49f43777feb > --- /dev/null > +++ b/mysql-test/main/ctype_utf8mb4_general1400_as_ci_ws.result > diff --git a/mysql-test/main/ctype_utf8mb4_general1400_as_ci_ws.test b/mysql-test/main/ctype_utf8mb4_general1400_as_ci_ws.test > new file mode 100644 > index 00000000000..b5eb369ae9a > --- /dev/null > +++ b/mysql-test/main/ctype_utf8mb4_general1400_as_ci_ws.test > @@ -0,0 +1,39 @@ > +--source include/have_utf32.inc > +--source include/have_sequence.inc > + > +--echo # > +--echo # Start of 11.1 tests > +--echo # > + > +--echo # > +--echo # Collation utfmb4_general1400_as_ci > +--echo # > + > +SET NAMES utf8mb4 COLLATE utf8mb4_general1400_as_ci; > + > +EXECUTE IMMEDIATE SFORMAT(' > +CREATE VIEW v_bmp AS > +SELECT > + seq AS codepoint, > + LPAD(HEX(seq),6,''0'') AS codepoint_hex6, > + CONVERT(CHAR(seq USING utf32) USING {}) COLLATE {} AS c > +FROM > + seq_0_to_1114111', @@character_set_connection, @@collation_connection); > + > +SELECT COLLATION(c) FROM v_bmp LIMIT 1; > + > +SELECT > + SUM(codepoint_hex6=HEX(LOWER(c))) AS count_bmp_weight_is_lower, > + SUM(codepoint_hex6<>HEX(LOWER(c))) AS count_bmp_weight_is_not_lower > +FROM v_bmp; > + > +SELECT codepoint_hex6,HEX(WEIGHT_STRING(c)) > +FROM v_bmp > +WHERE codepoint_hex6<>HEX(WEIGHT_STRING(c)); > + > +DROP VIEW v_bmp; > + > + > +--echo # > +--echo # End of 11.1 tests 11.4 ? > +--echo # > diff --git a/mysql-test/suite/compat/oracle/r/lower_case_table_names.result b/mysql-test/suite/compat/oracle/r/lower_case_table_names.result > new file mode 100644 > index 00000000000..2518be31402 > --- /dev/null > +++ b/mysql-test/suite/compat/oracle/r/lower_case_table_names.result > @@ -0,0 +1,9 @@ > +SELECT @@lower_case_table_names; > +@@lower_case_table_names > +1 > +# > +# TODO: Report this problem as MDEV what is the problem? isn't the comparison accent-sensitive? > +# > +SET NAMES utf8; > +CREATE TABLE t1 (a öracle_schema.date); > +ERROR HY000: Unknown data type: 'öracle_schema.date' > diff --git a/mysql-test/suite/unit/disabled.def b/mysql-test/suite/unit/disabled.def > new file mode 100644 > index 00000000000..3a68bea32d0 > --- /dev/null > +++ b/mysql-test/suite/unit/disabled.def > @@ -0,0 +1,12 @@ > +############################################################################## > +# > +# List the test cases that are to be disabled temporarily. > +# > +# Separate the test case name and the comment with ':'. > +# > +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> > +# > +# Do not use any TAB characters for whitespace. > +# > +############################################################################## > +conc_charset : TODO: client side ID for utf8mb4_tolower_ci 1. There is no utf8mb4_tolower_ci 2. what do you mean "client side ID"? > diff --git a/sql/ddl_log.cc b/sql/ddl_log.cc > index 61c8f50d1e3..fdfb605a57f 100644 > --- a/sql/ddl_log.cc > +++ b/sql/ddl_log.cc > @@ -1146,22 +1146,23 @@ static void execute_rename_table(DDL_LOG_ENTRY *ddl_log_entry, handler *file, > static void rename_triggers(THD *thd, DDL_LOG_ENTRY *ddl_log_entry, > bool swap_tables) > { > - LEX_CSTRING to_table, from_table, to_db, from_db, from_converted_name; > + Lex_ident_table to_table, from_table, from_converted_name; > + Lex_ident_db to_db, from_db; > char to_path[FN_REFLEN+1], from_path[FN_REFLEN+1], conv_path[FN_REFLEN+1]; > > if (!swap_tables) > { > - from_db= ddl_log_entry->db; > - from_table= ddl_log_entry->name; > - to_db= ddl_log_entry->from_db; > - to_table= ddl_log_entry->from_name; > + from_db= Lex_ident_db(ddl_log_entry->db); this would look better with implicit casting, that is, if you'd have an assignment operator from LEX_CSTRING > + from_table= Lex_ident_table(ddl_log_entry->name); > + to_db= Lex_ident_db(ddl_log_entry->from_db); > + to_table= Lex_ident_table(ddl_log_entry->from_name); > } > else > { > - from_db= ddl_log_entry->from_db; > - from_table= ddl_log_entry->from_name; > - to_db= ddl_log_entry->db; > - to_table= ddl_log_entry->extra_name; > + from_db= Lex_ident_db(ddl_log_entry->from_db); > + from_table= Lex_ident_table(ddl_log_entry->from_name); > + to_db= Lex_ident_db(ddl_log_entry->db); > + to_table= Lex_ident_table(ddl_log_entry->extra_name); > } > > build_filename_and_delete_tmp_file(from_path, sizeof(from_path), > @@ -1211,11 +1212,11 @@ static void rename_triggers(THD *thd, DDL_LOG_ENTRY *ddl_log_entry, > > (void) Table_triggers_list::prepare_for_rename(thd, > &trigger_param, > - &from_db, > - &from_table, > - &from_converted_name, > - &to_db, > - &to_table); > + from_db, > + from_table, > + from_converted_name, > + to_db, > + to_table); You rather inconsistently replace pointers with const references in some methods, like above, but not in others, like below or in the next hunk. > (void) Table_triggers_list::change_table_name(thd, > &trigger_param, > &from_db, > @@ -1707,9 +1708,8 @@ static int ddl_log_execute_action(THD *thd, MEM_ROOT *mem_root, > > switch (ddl_log_entry->phase) { > case DDL_DROP_DB_PHASE_INIT: > - drop_database_objects(thd, &path, &db, > - !my_strcasecmp(system_charset_info, > - MYSQL_SCHEMA_NAME.str, db.str)); > + // QQ: was a bug here? This comment makes little sense in the code, as one cannot see what *was* here before. Just drop it. > + drop_database_objects(thd, &path, &db, MYSQL_SCHEMA_NAME.streq(db)); > > strxnmov(to_path, sizeof(to_path)-1, path.str, MY_DB_OPT_FILE, NullS); > mysql_file_delete_with_symlink(key_file_misc, to_path, "", MYF(0)); > diff --git a/sql/debug_sync.cc b/sql/debug_sync.cc > index 362463a7f22..4704691615a 100644 > --- a/sql/debug_sync.cc > +++ b/sql/debug_sync.cc > @@ -858,6 +858,13 @@ static st_debug_sync_action *debug_sync_get_action(THD *thd, > } > > > +class Debug_token: public Lex_ident_ci > +{ > +public: > + using Lex_ident_ci::Lex_ident_ci; > +}; I wonder if you can create those types with less boilerplate will typedef work? will using work? if not, may be, like template<> Lex_ident_templ: Lex_ident_ci { ... } Lex_ident_templ<> Debug_token; > + > + > /** > Set a debug sync action. > > @@ -909,7 +916,8 @@ static bool debug_sync_set_action(THD *thd, st_debug_sync_action *action) > } > else > { > - const char *dsp_name= action->sync_point.c_ptr(); > + const Debug_token dsp_name(action->sync_point.c_ptr(), > + action->sync_point.length()); or dsp_name(action->sync_point->to_lex_cstring()) ? > #ifdef DBUG_TRACE > DBUG_EXECUTE("debug_sync", { > /* Functions as DBUG_PRINT args can change keyword and line nr. */ > diff --git a/sql/grant.cc b/sql/grant.cc > index 1ba197bc1d9..0ebf47431bc 100644 > --- a/sql/grant.cc > +++ b/sql/grant.cc > @@ -31,8 +31,8 @@ bool Grant_privilege::add_column_privilege(THD *thd, > class LEX_COLUMN *point; > while ((point=iter++)) > { > - if (!my_strcasecmp(system_charset_info, > - point->column.c_ptr(), new_str->c_ptr())) > + const Lex_ident_column tmp(point->column.c_ptr(), point->column.length()); That feels like an overkill, creating Lex_ident_column just to compare two strings. What is it compiled to at -O3? > + if (tmp.streq(new_str->to_lex_cstring())) > break; > } > m_column_privilege_total|= which_grant; > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index a615d26b9a9..fc840c21191 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -118,7 +118,7 @@ int ha_partition::notify_tabledef_changed(LEX_CSTRING *db, > LEX_CSTRING table_name; > const char *table_name_ptr; > if (create_partition_name(from_buff, sizeof(from_buff), > - from_path, name_buffer_ptr, > + from_path, Lex_cstring_strlen(name_buffer_ptr), Don't, please. You _always_ use Lex_cstring_strlen, in _all_ invocations of create_partition_name(). Just pass a char* down as before and do strlen inside, if you need it. > NORMAL_PART_NAME, FALSE)) > res=1; > table_name_ptr= from_buff + dirname_length(from_buff); > diff --git a/sql/handler.cc b/sql/handler.cc > index a977054d0da..7b9d11d00a2 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -7531,7 +7528,8 @@ int handler::check_duplicate_long_entry_key(const uchar *new_rec, uint key_no) > else > { > Item_func_left *fnc= static_cast<Item_func_left *>(arguments[j]); > - DBUG_ASSERT(!my_strcasecmp(system_charset_info, "left", fnc->func_name())); > + DBUG_ASSERT(Lex_ident_func(fnc->func_name_cstring()). eh, why func_name_cstring() doesn't return Lex_ident_func? > + streq("left"_Lex_cstring)); > DBUG_ASSERT(fnc->arguments()[0]->type() == Item::FIELD_ITEM); > t_field= static_cast<Item_field *>(fnc->arguments()[0])->field; > uint length= (uint)fnc->arguments()[1]->val_int(); > diff --git a/sql/item.cc b/sql/item.cc > index 5c4f1704ebe..5689756eca9 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -3463,13 +3463,11 @@ bool Item_field::eq(const Item *item, bool binary_cmp) const > (In cases where we would choose wrong we would have to generate a > ER_NON_UNIQ_ERROR). > */ > - return (!lex_string_cmp(system_charset_info, &item_field->name, > - &field_name) && > - (!item_field->table_name.str || !table_name.str || > - (!my_strcasecmp(table_alias_charset, item_field->table_name.str, > - table_name.str) && > - (!item_field->db_name.str || !db_name.str || > - (item_field->db_name.str && !strcmp(item_field->db_name.str, > + return (item_field->name.streq(field_name) && > + (!item_field->table_name.str || !table_name.str || > + (item_field->table_name.streq(table_name) && > + (!item_field->db_name.str || !db_name.str || > + (item_field->db_name.str && !strcmp(item_field->db_name.str, > db_name.str)))))); I presume, # db_name should also be compared with streq > } > > diff --git a/sql/item_create.cc b/sql/item_create.cc > index 4f0df8f732d..501db19c1b8 100644 > --- a/sql/item_create.cc > +++ b/sql/item_create.cc > @@ -2638,7 +2639,8 @@ Create_qfunc::create_func(THD *thd, const LEX_CSTRING *name, > if (thd->lex->copy_db_to(&db)) > return NULL; > > - return create_with_db(thd, &db, name, false, item_list); Again, can LEX_CSTRING be implicitly cast to the correct Lex_ident variant? > + return create_with_db(thd, Lex_ident_db(db), Lex_ident_func(*name), > + false, item_list); > } > > > diff --git a/sql/item_windowfunc.cc b/sql/item_windowfunc.cc > index 5438cade27a..f0a5737bdf7 100644 > --- a/sql/item_windowfunc.cc > +++ b/sql/item_windowfunc.cc > @@ -28,7 +28,7 @@ Item_window_func::resolve_window_name(THD *thd) > return false; > } > DBUG_ASSERT(window_name != NULL && window_spec == NULL); > - const char *ref_name= window_name->str; > + const LEX_CSTRING &ref_name= *window_name; Why LEX_CSTRING ? > > /* !TODO: Add the code to resolve ref_name in outer queries */ > /* > diff --git a/sql/opt_trace.cc b/sql/opt_trace.cc > index 1008690ccaa..dbb2708f487 100644 > --- a/sql/opt_trace.cc > +++ b/sql/opt_trace.cc > @@ -43,7 +43,7 @@ bool list_has_optimizer_trace_table(const TABLE_LIST *tbl) > for (; tbl; tbl= tbl->next_global) > { > if (tbl->schema_table && > - 0 == strcmp(tbl->schema_table->table_name, I_S_table_name)) > + 0 == strcmp(tbl->schema_table->table_name.str, I_S_table_name)) Why not streq? > return true; > } > return false; > @@ -190,9 +190,8 @@ void opt_trace_disable_if_no_security_context_access(THD *thd) > if (!(thd->main_security_ctx.check_access(GLOBAL_ACLS & ~GRANT_ACL)) && > (0 != strcmp(thd->main_security_ctx.priv_user, > thd->security_context()->priv_user) || > - 0 != my_strcasecmp(system_charset_info, > - thd->main_security_ctx.priv_host, > - thd->security_context()->priv_host))) > + !Lex_ident_host(Lex_cstring_strlen(thd->main_security_ctx.priv_host)). > + streq(Lex_cstring_strlen(thd->security_context()->priv_host)))) Doesn't really look like an improvement :) May be you can get rid of intermediate Lex_cstring_strlen()'s ? > trace->missing_privilege(); > } > > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index 1b65de6e3c0..7b76c6d65b1 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -1403,8 +1397,8 @@ bool partition_info::check_partition_info(THD *thd, handlerton **eng_type, > num_parts_not_set++; > part_elem->engine_type= default_engine_type; > } > - if (check_table_name(part_elem->partition_name, > - strlen(part_elem->partition_name), FALSE)) > + if (check_table_name(part_elem->partition_name.str, > + part_elem->partition_name.length, FALSE)) better change check_table_name to accept a LEX_CSTRING or Lex_ident, it'll simplify almost all (except one) invocations of it > { > my_error(ER_WRONG_PARTITION_NAME, MYF(0)); > goto end; > @@ -2604,8 +2596,8 @@ bool partition_info::has_same_partitioning(partition_info *new_part_info) > part_state must be PART_NORMAL! > */ > if (!part_elem || !new_part_elem || > - strcmp(part_elem->partition_name, > - new_part_elem->partition_name) || > + strcmp(part_elem->partition_name.str, > + new_part_elem->partition_name.str) || why strcmp? > part_elem->part_state != PART_NORMAL || > new_part_elem->part_state != PART_NORMAL || > part_elem->max_value != new_part_elem->max_value || > @@ -2666,8 +2658,8 @@ bool partition_info::has_same_partitioning(partition_info *new_part_info) > sub_part_elem->engine_type != new_sub_part_elem->engine_type) > DBUG_RETURN(false); > > - if (strcmp(sub_part_elem->partition_name, > - new_sub_part_elem->partition_name) || > + if (strcmp(sub_part_elem->partition_name.str, > + new_sub_part_elem->partition_name.str) || same here > sub_part_elem->part_state != PART_NORMAL || > new_sub_part_elem->part_state != PART_NORMAL || > sub_part_elem->part_min_rows != > diff --git a/sql/partition_info.h b/sql/partition_info.h > index 3a8c3a37524..92e8906274f 100644 > --- a/sql/partition_info.h > +++ b/sql/partition_info.h > @@ -526,11 +527,13 @@ void partition_info::vers_update_el_ids() > } > > > -inline > -bool make_partition_name(char *move_ptr, uint i) > +static inline > +Lex_ident_partition make_partition_name_ls(char *move_ptr, uint i) it's not _ls really, is it? why not to keep the old name? > { > int res= snprintf(move_ptr, MAX_PART_NAME_SIZE + 1, "p%u", i); > - return res < 0 || res > MAX_PART_NAME_SIZE; > + return res < 0 || res > MAX_PART_NAME_SIZE ? > + Lex_ident_partition() : > + Lex_ident_partition(move_ptr, (size_t) res); > } > > > @@ -549,15 +552,16 @@ uint partition_info::next_part_no(uint new_parts) const > for (uint cur_part= 0; cur_part < new_parts; ++cur_part, ++suffix) > { > uint32 cur_suffix= suffix; > - if (make_partition_name(part_name, suffix)) > + Lex_ident_partition part_name_ls(make_partition_name_ls(part_name, suffix)); > + if (!part_name_ls.str) > return 0; > partition_element *el; > it.rewind(); > while ((el= it++)) > { > - if (0 == my_strcasecmp(&my_charset_latin1, el->partition_name, part_name)) > + if (el->partition_name.streq(part_name_ls)) > { > - if (make_partition_name(part_name, ++suffix)) > + if (!(part_name_ls= make_partition_name_ls(part_name, ++suffix)).str) you don't seem to use part_name_ls below > return 0; > it.rewind(); > } > diff --git a/sql/set_var.h b/sql/set_var.h > index aed4955ef62..8f7e3609f5f 100644 > --- a/sql/set_var.h > +++ b/sql/set_var.h > @@ -383,7 +383,7 @@ class set_var_default_role: public set_var_base > { > LEX_USER *user, *real_user; > LEX_CSTRING role; > - const char *real_role; > + LEX_CSTRING real_role; Not Lex_ident_user ? > public: > set_var_default_role(LEX_USER *user_arg, LEX_CSTRING role_arg) : > user(user_arg), role(role_arg) {} > diff --git a/sql/slave.cc b/sql/slave.cc > index 0a15cc55146..daeb05f423e 100644 > --- a/sql/slave.cc > +++ b/sql/slave.cc > @@ -3403,8 +3403,9 @@ static bool send_show_master_info_data(THD *thd, Master_info *mi, bool full, > static int cmp_mi_by_name(const Master_info **arg1, > const Master_info **arg2) > { > - return my_strcasecmp(system_charset_info, (*arg1)->connection_name.str, > - (*arg2)->connection_name.str); > + return Lex_ident_master_info::charset_info()->strnncoll( > + (*arg1)->connection_name, > + (*arg2)->connection_name); Why not (*arg1)->connection_name.streq() ? > } > > > diff --git a/sql/sp_pcontext.cc b/sql/sp_pcontext.cc > index 848d1f0c655..761f535832d 100644 > --- a/sql/sp_pcontext.cc > +++ b/sql/sp_pcontext.cc > @@ -139,8 +139,7 @@ sp_pcontext *sp_pcontext::push_context(THD *thd, sp_pcontext::enum_scope scope) > > bool cmp_labels(sp_label *a, sp_label *b) > { > - return (lex_string_cmp(system_charset_info, &a->name, &b->name) == 0 && > - a->type == b->type); > + return a->name.streq(b->name) && a->type == b->type; put type comparison first, please > } > > sp_pcontext *sp_pcontext::pop_context() > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 09af5523c76..dd36504ed56 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -567,14 +560,17 @@ static uchar* acl_role_get_key(ACL_ROLE *entry, size_t *length, > > struct ROLE_GRANT_PAIR : public Sql_alloc > { > - char *u_uname; > - char *u_hname; > - char *r_uname; > + LEX_CSTRING u_uname; > + LEX_CSTRING u_hname; > + LEX_CSTRING r_uname; Lex_ident_user ? (and in many places below) > LEX_STRING hashkey; > bool with_admin; > > - bool init(MEM_ROOT *mem, const char *username, const char *hostname, > - const char *rolename, bool with_admin_option); > + bool init(MEM_ROOT *mem, > + const LEX_CSTRING &username, > + const LEX_CSTRING &hostname, > + const LEX_CSTRING &rolename, > + bool with_admin_option); > }; > > static uchar* acl_role_map_get_key(ROLE_GRANT_PAIR *entry, size_t *length, > @@ -2731,7 +2733,7 @@ static bool acl_load(THD *thd, const Grant_tables& tables) > char *db_name; > db.user=safe_str(get_field(&acl_memroot, db_table.user())); > const char *hostname= get_field(&acl_memroot, db_table.host()); > - if (!hostname && find_acl_role(db.user, true)) > + if (!hostname && find_acl_role(Lex_cstring_strlen(db.user), true)) Perhaps it'd be better to change user and db to be Lex_ident_* in ACL_DB? > hostname= ""; > update_hostname(&db.host, hostname); > db.db= db_name= get_field(&acl_memroot, db_table.db()); > @@ -2822,14 +2824,29 @@ static bool acl_load(THD *thd, const Grant_tables& tables) > init_alloc_root(key_memory_acl_mem, &temp_root, ACL_ALLOC_BLOCK_SIZE, 0, MYF(0)); > while (!(read_record_info.read_record())) > { > - char *hostname= safe_str(get_field(&temp_root, roles_mapping_table.host())); > - char *username= safe_str(get_field(&temp_root, roles_mapping_table.user())); > - char *rolename= safe_str(get_field(&temp_root, roles_mapping_table.role())); > + StringBuffer<MAX_FIELD_WIDTH> host_buff, user_buff, role_buff; > + /* > + find_user_by_name() called recursively later in this code block There's no find_user_by_name() and acl_find_user_by_name does not seem to be called recursively. What did you mean here? also, why you didn't change acl_find_user_by_name to take a Lex_ident? > + needs a 0-terminated string. So does sql_print_error(). > + As val_lex_cstring() does not guarantee a 0-terminated string, > + let's alloc 0-terminated copies of val_lex_cstring() results on > + the memory root. you've already added Field::val_lex_string_strmake() in August, why not to use it here? > + */ > + const Lex_cstring hostname= safe_lexcstrdup_root(&temp_root, > + roles_mapping_table.host()-> > + val_lex_cstring(&host_buff)); > + const Lex_cstring username= safe_lexcstrdup_root(&temp_root, > + roles_mapping_table.user()-> > + val_lex_cstring(&user_buff)); > + const Lex_cstring rolename= safe_lexcstrdup_root(&temp_root, > + roles_mapping_table.role()-> > + val_lex_cstring(&role_buff)); > + > bool with_grant_option= get_YN_as_bool(roles_mapping_table.admin_option()); > > if (add_role_user_mapping(username, hostname, rolename)) { > sql_print_error("Invalid roles_mapping table entry user:'%s@%s', rolename:'%s'", > - username, hostname, rolename); > + username.str, hostname.str, rolename.str); > continue; > } > > @@ -4443,28 +4484,31 @@ static ACL_USER * find_user_wild(const char *host, const char *user, const char > /* > Find a role with the specified name > */ > -static ACL_ROLE *find_acl_role(const char *role, bool allow_public) > +static ACL_ROLE *find_acl_role(const LEX_CSTRING &role, bool allow_public) > { > - size_t length= strlen(role); > DBUG_ENTER("find_acl_role"); > - DBUG_PRINT("enter",("role: '%s'", role)); > + DBUG_PRINT("enter",("role: '%s'", role.str)); > DBUG_PRINT("info", ("Hash elements: %ld", acl_roles.records)); > > mysql_mutex_assert_owner(&acl_cache->lock); > > - if (!length || (!allow_public && strcasecmp(role, public_name.str) == 0)) > + if (!role.length || > + (!allow_public && > + my_charset_utf8mb3_general1400_as_ci.streq(role, public_name))) Lex_ident_user would've been nice here too > DBUG_RETURN(NULL); > > - ACL_ROLE *r= (ACL_ROLE *)my_hash_search(&acl_roles, (uchar *)role, length); > + ACL_ROLE *r= (ACL_ROLE *)my_hash_search(&acl_roles, (uchar *)role.str, > + role.length); > DBUG_RETURN(r); > } > > /* > Finds a grantee - something that privileges or roles can be granted to. > */ > -static ACL_USER_BASE *find_acl_user_base(const char *user, const char *host) > +static ACL_USER_BASE *find_acl_user_base(const LEX_CSTRING &user, > + const LEX_CSTRING &host) > { > - if (*host) > + if (*host.str) > return find_user_exact(host, user); > > return find_acl_role(user, true); > @@ -5657,8 +5696,8 @@ static GRANT_NAME *name_hash_search(HASH *name_hash, > { > if (!grant_name->host.hostname || > (host && > - !my_strcasecmp(system_charset_info, host, > - grant_name->host.hostname)) || > + Lex_ident_host(Lex_cstring_strlen(host)). Still don't like these Lex_ident_something(Lex_cstring_strlen(xxx)) Let's get rid of them > + streq(Lex_cstring_strlen(grant_name->host.hostname))) || > (ip && !strcmp(ip, grant_name->host.hostname))) > return grant_name; > } > @@ -7373,8 +7414,9 @@ int mysql_table_grant(THD *thd, TABLE_LIST *table_list, > } > } > if (Str->is_role()) > - propagate_role_grants(find_acl_role(Str->user.str, true), > - PRIVS_TO_MERGE::TABLE_COLUMN, db_name, table_name); > + propagate_role_grants(find_acl_role(Str->user, true), > + PRIVS_TO_MERGE::TABLE_COLUMN, > + db_name.str, table_name.str); may be pass Lex_ident_* down into propagate_role_grants too? > } > > thd->mem_root= old_root; > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 9d2415d3a4d..d7d7ebee024 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -233,33 +233,41 @@ uint get_table_def_key(const TABLE_LIST *table_list, const char **key) > # Pointer to list of names of open tables. > */ > > -struct list_open_tables_arg > +class list_open_tables_arg > { > +public: > THD *thd; > - const char *db; > + const Lex_ident_db db; > const char *wild; > TABLE_LIST table_list; > OPEN_TABLE_LIST **start_list, *open_list; > + > + list_open_tables_arg(THD *thd_arg, const LEX_CSTRING &db_arg, > + const char *wild_arg) > + :thd(thd_arg), db(db_arg), wild(wild_arg), > + start_list(&open_list), open_list(0) > + { > + bzero((char*) &table_list, sizeof(table_list)); > + } > }; > > > static my_bool list_open_tables_callback(TDC_element *element, > list_open_tables_arg *arg) > { > - const char *db= (char*) element->m_key; > - size_t db_length= strlen(db); > - const char *table_name= db + db_length + 1; > + const Lex_ident_db > + db= Lex_ident_db(Lex_cstring_strlen((const char*) element->m_key)); > + const char *table_name= db.str + db.length + 1; > > - if (arg->db && my_strcasecmp(system_charset_info, arg->db, db)) > + // QQ: a bug was here? same as above, "was" > + if (arg->db.str && !arg->db.streq(db)) > return FALSE; > if (arg->wild && wild_compare(table_name, arg->wild, 0)) > return FALSE; > > /* Check if user has SELECT privilege for any column in the table */ > - arg->table_list.db.str= db; > - arg->table_list.db.length= db_length; > - arg->table_list.table_name.str= table_name; > - arg->table_list.table_name.length= strlen(table_name); > + arg->table_list.db= db; > + arg->table_list.table_name= Lex_cstring_strlen(table_name); > arg->table_list.grant.privilege= NO_ACL; > > if (check_table_access(arg->thd, SELECT_ACL, &arg->table_list, TRUE, 1, TRUE)) > @@ -1906,7 +1904,7 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > DBUG_ASSERT(!table_list->table); > > /* an open table operation needs a lot of the stack space */ > - if (check_stack_overrun(thd, STACK_MIN_SIZE_FOR_OPEN, (uchar *)&alias)) > + if (check_stack_overrun(thd, STACK_MIN_SIZE_FOR_OPEN, (uchar *)&alias.str)) better pass `key` above. doesn't matter much but looks a bit less weird > DBUG_RETURN(TRUE); > > if (!(flags & MYSQL_OPEN_IGNORE_KILLED) && thd->killed) > @@ -1954,7 +1952,10 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > if (table->s->table_cache_key.length == key_length && > !memcmp(table->s->table_cache_key.str, key, key_length)) > { > - if (!my_strcasecmp(system_charset_info, table->alias.c_ptr(), alias) && > + // QQ: Why table_alias_charset is not used for comparison? Did you try to correct this? > + if (my_charset_utf8mb3_general1400_as_ci.streq( > + table->alias.to_lex_cstring(), > + alias) && > table->query_id != thd->query_id && /* skip tables already used */ > (thd->locked_tables_mode == LTM_LOCK_TABLES || > table->query_id == 0)) > @@ -6353,26 +6355,25 @@ find_field_in_natural_join(THD *thd, TABLE_LIST *table_ref, const char *name, si > */ > > Field * > -find_field_in_table(THD *thd, TABLE *table, const char *name, size_t length, > +find_field_in_table(THD *thd, TABLE *table, const Lex_ident_column &name, > bool allow_rowid, field_index_t *cached_field_index_ptr) > { > Field *field; > field_index_t cached_field_index= *cached_field_index_ptr; > DBUG_ENTER("find_field_in_table"); > DBUG_PRINT("enter", ("table: '%s', field name: '%s'", table->alias.c_ptr(), > - name)); > + name.str)); > > /* We assume here that table->field < NO_CACHED_FIELD_INDEX = UINT_MAX */ > if (cached_field_index < table->s->fields && > - !my_strcasecmp(system_charset_info, > - table->field[cached_field_index]->field_name.str, name)) > + table->field[cached_field_index]->field_name.streq(name)) > { > field= table->field[cached_field_index]; > DEBUG_SYNC(thd, "table_field_cached"); > } > else > { > - LEX_CSTRING fname= {name, length}; > + LEX_CSTRING fname= name; Do you need that? You can use `name` directly below > field= table->find_field_by_name(&fname); > } > > @@ -6999,9 +7000,9 @@ find_item_in_list(Item *find, List<Item> &items, uint *counter, > List_iterator<Item> li(items); > uint n_items= limit == 0 ? items.elements : limit; > Item **found=0, **found_unaliased= 0, *item; > - const char *db_name=0; > + const LEX_CSTRING *db_name=0; > const LEX_CSTRING *field_name= 0; > - const char *table_name=0; > + const LEX_CSTRING *table_name=0; Why not Lex_ident's ? > bool found_unaliased_non_uniq= 0; > /* > true if the item that we search for is a valid name reference > @@ -7059,12 +7060,11 @@ find_item_in_list(Item *find, List<Item> &items, uint *counter, > item is not fix_field()'ed yet. > */ > if (item_field->field_name.str && item_field->table_name.str && > - !lex_string_cmp(system_charset_info, &item_field->field_name, > - field_name) && > - !my_strcasecmp(table_alias_charset, item_field->table_name.str, > - table_name) && > - (!db_name || (item_field->db_name.str && > - !strcmp(item_field->db_name.str, db_name)))) > + item_field->field_name.streq(*field_name) && > + item_field->table_name.streq(*table_name) && > + (!db_name || !db_name->str || > + (item_field->db_name.str && > + !strcmp(item_field->db_name.str, db_name->str)))) why strcmp? > { > if (found_unaliased) > { > @@ -7135,11 +7132,10 @@ find_item_in_list(Item *find, List<Item> &items, uint *counter, > } > } > } > - else if (!table_name) > + else if (!table_name || !table_name->str) > { > if (is_ref_by_name && find->name.str && item->name.str && > - find->name.length == item->name.length && > - !lex_string_cmp(system_charset_info, &item->name, &find->name)) > + item->name.streq(find->name)) Do you have tests that will fail if streq will compare lengths? > { > found= li.ref(); > *counter= i; > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index a2414de645b..1b36cd9416f 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -855,10 +853,13 @@ THD::THD(my_thread_id id, bool is_wsrep_applier) > profiling.set_thd(this); > #endif > user_connect=(USER_CONN *)0; > - my_hash_init(key_memory_user_var_entry, &user_vars, system_charset_info, > + my_hash_init(key_memory_user_var_entry, &user_vars, > + Lex_ident_user_var::charset_info(), > USER_VARS_HASH_SIZE, 0, 0, (my_hash_get_key) get_var_key, > (my_hash_free_key) free_user_var, HASH_THREAD_SPECIFIC); > - my_hash_init(PSI_INSTRUMENT_ME, &sequences, system_charset_info, > + my_hash_init(PSI_INSTRUMENT_ME, &sequences, > + // QQ: why case insensitive? looks like a bug, can you do a test case for it, please? > + &my_charset_utf8mb3_general1400_as_ci, > SEQUENCES_HASH_SIZE, 0, 0, (my_hash_get_key) > get_sequence_last_key, (my_hash_free_key) free_sequence_last, > HASH_THREAD_SPECIFIC); > @@ -4011,7 +4015,7 @@ Table_ident::to_ident_db_internal_with_error(Query_arena *arena) const > if (is_derived_table()) > { > DBUG_ASSERT(db.str == empty_c_string && db.length == 0); > - return Lex_ident_db(empty_c_string, 0); > + return Lex_ident_db(empty_c_string, (size_t) 0); why? > } > // Normal table or JSON table > return arena->to_ident_db_internal_with_error(db); > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 214118d8431..5685240ceaf 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -433,13 +435,14 @@ class Alter_rename_key : public Sql_alloc > class Alter_index_ignorability: public Sql_alloc > { > public: > - Alter_index_ignorability(const char *name, bool is_ignored, bool if_exists) : > + Alter_index_ignorability(const LEX_CSTRING &name, > + bool is_ignored, bool if_exists) : > m_name(name), m_is_ignored(is_ignored), m_if_exists(if_exists) > { > - assert(name != NULL); > + assert(name.str != NULL); DBUG_ASSERT > } > > - const char *name() const { return m_name; } > + const Lex_ident_column &name() const { return m_name; } > bool if_exists() const { return m_if_exists; } > > /* The ignorability after the operation is performed. */ > @@ -7989,42 +7995,24 @@ class Identifier_chain2 > class Database_qualified_name > { > public: > - LEX_CSTRING m_db; > - LEX_CSTRING m_name; > - Database_qualified_name(const LEX_CSTRING *db, const LEX_CSTRING *name) > - :m_db(*db), m_name(*name) > + Lex_ident_db m_db; > + Lex_cstring m_name; // no comparison semantics > + Database_qualified_name() > { } > - Database_qualified_name(const LEX_CSTRING &db, const LEX_CSTRING &name) > + Database_qualified_name(const Lex_ident_db &db, const LEX_CSTRING &name) > :m_db(db), m_name(name) > { } > - Database_qualified_name(const char *db, size_t db_length, > - const char *name, size_t name_length) > - { > - m_db.str= db; > - m_db.length= db_length; > - m_name.str= name; > - m_name.length= name_length; > - } > - > - bool eq(const Database_qualified_name *other) const > + bool eq_routine_name(const Database_qualified_name *other) const eq_func_name, because the comparison semantics comes from Lex_ident_func > { > - CHARSET_INFO *cs= lower_case_table_names ? > - &my_charset_utf8mb3_general_ci : > - &my_charset_utf8mb3_bin; > - return > - m_db.length == other->m_db.length && > - m_name.length == other->m_name.length && > - !cs->strnncoll(m_db.str, m_db.length, > - other->m_db.str, other->m_db.length) && > - !cs->strnncoll(m_name.str, m_name.length, > - other->m_name.str, other->m_name.length); > + return m_db.streq(other->m_db) && no need to take lower_case_table_names into account when comparing db names here? > + Lex_ident_func(m_name).streq(other->m_name); > } > /* > Make copies of "db" and "name" on the memory root in internal format: > - Lower-case "db" if lower-case-table-names==1. > - Preserve "name" as is. > */ > - bool copy_sp_name_internal(MEM_ROOT *mem_root, const LEX_CSTRING &db, > + bool copy_sp_name_internal(MEM_ROOT *mem_root, const Lex_ident_db &db, > const LEX_CSTRING &name); > > // Export db and name as a qualified name string: 'db.name' > diff --git a/sql/sql_connect.cc b/sql/sql_connect.cc > index c5619b28f69..4e10f498e3c 100644 > --- a/sql/sql_connect.cc > +++ b/sql/sql_connect.cc > @@ -509,7 +514,9 @@ extern "C" void free_table_stats(TABLE_STATS* table_stats) > > void init_global_table_stats(void) > { > - my_hash_init(PSI_INSTRUMENT_ME, &global_table_stats, system_charset_info, > + my_hash_init(PSI_INSTRUMENT_ME, &global_table_stats, > + // QQ: why not Lex_ident_table::charset_info() ? just change it > + &my_charset_utf8mb3_general1400_as_ci, > max_connections, 0, 0, (my_hash_get_key) get_key_table_stats, > (my_hash_free_key) free_table_stats, 0); > } > diff --git a/sql/sql_db.cc b/sql/sql_db.cc > index 08441d9f067..51635bdf0ce 100644 > --- a/sql/sql_db.cc > +++ b/sql/sql_db.cc > @@ -98,12 +98,10 @@ typedef struct my_dbopt_st > */ > > static inline bool > -cmp_db_names(LEX_CSTRING *db1_name, const LEX_CSTRING *db2_name) > +cmp_db_names(const Lex_ident_db &db1_name, const Lex_ident_db &db2_name) > { > - return (db1_name->length == db2_name->length && > - (db1_name->length == 0 || can it be used with empty names, really? > - my_strcasecmp(table_alias_charset, > - db1_name->str, db2_name->str) == 0)); > + return (db1_name.length == 0 && db2_name.length == 0) || > + db1_name.streq(db2_name); > } > > #ifdef HAVE_PSI_INTERFACE > @@ -1087,8 +1085,7 @@ mysql_rm_db_internal(THD *thd, const Lex_ident_db &db, bool if_exists, > Disable drop of enabled log tables, must be done before name locking. > This check is only needed if we are dropping the "mysql" database. > */ > - if ((rm_mysql_schema= > - (my_strcasecmp(system_charset_info, MYSQL_SCHEMA_NAME.str, db.str) == 0))) > + if ((rm_mysql_schema= MYSQL_SCHEMA_NAME.streq(db))) // QQ: was a bug here? again, "was" > { > for (table= tables; table; table= table->next_local) > if (check_if_log_table(table, TRUE, "DROP")) > diff --git a/sql/sql_handler.cc b/sql/sql_handler.cc > index 406f67ff955..def1fc44392 100644 > --- a/sql/sql_handler.cc > +++ b/sql/sql_handler.cc > @@ -1080,11 +1079,14 @@ static SQL_HANDLER *mysql_ha_find_match(THD *thd, TABLE_LIST *tables) > { > if (tables->is_anonymous_derived_table()) > continue; > + /* > + TODO: for some reasons the code user case-insensitive comparison. > + Shouldn't it use Lex_ident_db/Lex_ident_table instead, > + to take into account the filesystem case sensitivity? > + */ must be a bug. try to create a test case for it. I think it could be something like (untested) create table t1 ...; create table T1; handler t1 open; handler t1 read first; handler t1 read next; drop table T1; handler t1 read next; > if ((! tables->db.str[0] || > - ! my_strcasecmp(&my_charset_latin1, hash_tables->db.str, > - tables->get_db_name())) && > - ! my_strcasecmp(&my_charset_latin1, hash_tables->table_name.str, > - tables->get_table_name())) > + Lex_ident_ci(hash_tables->db).streq(tables->get_db_name())) && > + Lex_ident_ci(hash_tables->table_name).streq(tables->get_table_name())) > { > /* Link into hash_tables list */ > hash_tables->next= head; > diff --git a/sql/sql_i_s.h b/sql/sql_i_s.h > index 263031ae2c9..127b35f6ab2 100644 > --- a/sql/sql_i_s.h > +++ b/sql/sql_i_s.h > @@ -329,7 +325,7 @@ typedef class Item COND; > > typedef struct st_schema_table > { > - const char *table_name; > + LEX_CSTRING table_name; why LEX_CSTRING? > ST_FIELD_INFO *fields_info; > /* for FLUSH table_name */ > int (*reset_table) (); > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index a84fd29affc..b74ecb51a69 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -9370,8 +9359,8 @@ bool LEX::create_package_finalize(THD *thd, > { > if (name2 && > (name2->m_explicit_name != name->m_explicit_name || > - strcmp(name2->m_db.str, name->m_db.str) || > - !Sp_handler::eq_routine_name(name2->m_name, name->m_name))) > + // QQ: was a bug here? "was" may be it was intentional, looking for an exact match only? > + !name2->eq_routine_name(name))) > { > bool exp= name2->m_explicit_name || name->m_explicit_name; > my_error(ER_END_IDENTIFIER_DOES_NOT_MATCH, MYF(0), > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 665d2143872..1430aa473af 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -1131,8 +1131,8 @@ static bool wsrep_tables_accessible_when_detached(const TABLE_LIST *tables) > { > for (const TABLE_LIST *table= tables; table; table= table->next_global) I'd personally use const TABLE_LIST *t= tables; ... it's a 3-line loop > { > - LEX_CSTRING db= table->db, tn= table->table_name; > - if (get_table_category(&db, &tn) < TABLE_CATEGORY_INFORMATION) > + if (get_table_category(table->db, > + table->table_name) < TABLE_CATEGORY_INFORMATION) > return false; > } > return tables != NULL; > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index 958c52fd95c..c8d64ea37ff 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -8055,13 +8050,9 @@ void make_used_partitions_str(MEM_ROOT *alloc, > if (parts_str->length()) > parts_str->append(','); > uint index= parts_str->length(); > - parts_str->append(head_pe->partition_name, > - strlen(head_pe->partition_name), > - system_charset_info); > + parts_str->append(head_pe->partition_name, system_charset_info); > parts_str->append('_'); > - parts_str->append(pe->partition_name, > - strlen(pe->partition_name), > - system_charset_info); > + parts_str->append(pe->partition_name, system_charset_info); better pe->partition_name->charset_info() (here and above and below) > used_partitions_list.append_str(alloc, parts_str->ptr() + index); > } > partition_id++; > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 8bab244c709..f335226b9a2 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -374,16 +374,16 @@ bool check_valid_path(const char *path, size_t len) > > static void fix_dl_name(MEM_ROOT *root, LEX_CSTRING *dl) > { > - const size_t so_ext_len= sizeof(SO_EXT) - 1; > - if (dl->length < so_ext_len || > - my_strcasecmp(&my_charset_latin1, dl->str + dl->length - so_ext_len, > - SO_EXT)) > + const Lex_ident_plugin so_ext(SO_EXT, sizeof(SO_EXT) - 1); STRING_WITH_LEN > + if (dl->length < so_ext.length || > + !so_ext.streq(Lex_cstring(dl->str + dl->length - so_ext.length, > + so_ext.length))) > { > - char *s= (char*)alloc_root(root, dl->length + so_ext_len + 1); > + char *s= (char*)alloc_root(root, dl->length + so_ext.length + 1); > memcpy(s, dl->str, dl->length); > strcpy(s + dl->length, SO_EXT); > dl->str= s; > - dl->length+= so_ext_len; > + dl->length+= so_ext.length; > } > } > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 1fcafd6ec08..7548ff1ec57 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -7182,10 +7170,12 @@ static int get_schema_views_record(THD *thd, TABLE_LIST *tables, > Security_context *sctx= thd->security_ctx; > if (!tables->allowed_show) > { > - if (!my_strcasecmp(system_charset_info, tables->definer.user.str, > - sctx->priv_user) && > - !my_strcasecmp(system_charset_info, tables->definer.host.str, > - sctx->priv_host)) > + // QQ: why case insensitive comparision? Same as above? Try a test case and fix if a bug? it's likely create user foo; create user FOO; create view ... definer=foo v1 ... connect(as FOO); select * from information schema.views; > + if (my_charset_utf8mb3_general1400_as_ci.streq( > + tables->definer.user, > + Lex_cstring_strlen(sctx->priv_user)) && > + Lex_ident_host(tables->definer.host). > + streq(Lex_cstring_strlen(sctx->priv_host))) > tables->allowed_show= TRUE; > #ifndef NO_EMBEDDED_ACCESS_CHECKS > else > @@ -7893,8 +7883,7 @@ static int get_schema_partitions_record(THD *thd, TABLE_LIST *tables, > > while ((part_elem= part_it++)) > { > - table->field[3]->store(part_elem->partition_name, > - strlen(part_elem->partition_name), cs); > + table->field[3]->store(part_elem->partition_name, cs); may be part_elem->partition_name->charset_info() ? > table->field[3]->set_notnull(); > /* PARTITION_ORDINAL_POSITION */ > table->field[5]->store((longlong) ++part_pos, TRUE); > diff --git a/sql/sql_trigger.cc b/sql/sql_trigger.cc > index 5304a535ad0..dcba31d06c9 100644 > --- a/sql/sql_trigger.cc > +++ b/sql/sql_trigger.cc > @@ -470,7 +470,7 @@ bool mysql_create_or_drop_trigger(THD *thd, TABLE_LIST *tables, bool create) > /* > We don't allow creating triggers on tables in the 'mysql' schema > */ > - if (create && lex_string_eq(&tables->db, STRING_WITH_LEN("mysql"))) > + if (create && tables->db.streq(MYSQL_SCHEMA_NAME)) // QQ: was a bug here? "was" > { > my_error(ER_NO_TRIGGERS_ON_SYSTEM_SCHEMA, MYF(0)); > DBUG_RETURN(TRUE); > diff --git a/sql/sql_trigger.h b/sql/sql_trigger.h > index 493349afb9d..5648f8f15ff 100644 > --- a/sql/sql_trigger.h > +++ b/sql/sql_trigger.h > @@ -74,8 +74,10 @@ struct st_trg_execution_order > /** > Trigger name referenced in the FOLLOWS/PRECEDES clause of the > CREATE TRIGGER statement. > + Cannot be Lex_ident_trigger, > + as this structure is used in %union in sql_yacc.yy Thanks. I already wrote above "why it's not simply a Lex_ident_trigger. That explains it, deleted my previous comment :) > */ > - LEX_CSTRING anchor_trigger_name; > + LEX_CSTRING anchor_trigger_name; // Used in sql_yacc %union > }; > > > diff --git a/sql/sql_truncate.cc b/sql/sql_truncate.cc > index beeee9da72f..e3efc3122df 100644 > --- a/sql/sql_truncate.cc > +++ b/sql/sql_truncate.cc > @@ -147,14 +147,12 @@ fk_truncate_illegal_if_parent(THD *thd, TABLE *table) > /* Loop over the set of foreign keys for which this table is a parent. */ > while ((fk_info= it++)) > { > - if (lex_string_cmp(system_charset_info, fk_info->referenced_db, > - &table->s->db) || > - lex_string_cmp(system_charset_info, fk_info->referenced_table, > - &table->s->table_name) || > - lex_string_cmp(system_charset_info, fk_info->foreign_db, > - &table->s->db) || > - lex_string_cmp(system_charset_info, fk_info->foreign_table, > - &table->s->table_name)) > + // QQ: why db and table don't use table_alias_charset for comparison? Same as above? Try a test case and fix if a bug? test looks fairly straightforward here too > + if (!Lex_ident_ci(*fk_info->referenced_db).streq(table->s->db) || > + !Lex_ident_ci(*fk_info->referenced_table). > + streq(table->s->table_name) || > + !Lex_ident_ci(*fk_info->foreign_db).streq(table->s->db) || > + !Lex_ident_ci(*fk_info->foreign_table).streq(table->s->table_name)) > break; > } > > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 10588b54fa5..751373b4571 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1347,12 +1347,9 @@ bool mysql_make_view(THD *thd, TABLE_SHARE *share, TABLE_LIST *table, > precedent; > precedent= precedent->referencing_view) > { > - if (precedent->view_name.length == table->table_name.length && > - precedent->view_db.length == table->db.length && > - my_strcasecmp(system_charset_info, > - precedent->view_name.str, table->table_name.str) == 0 && > - my_strcasecmp(system_charset_info, > - precedent->view_db.str, table->db.str) == 0) > + // QQ: was a bug here? "was" > + if (precedent->view_name.streq(table->table_name) && > + precedent->view_db.streq(table->db)) > { > my_error(ER_VIEW_RECURSIVE, MYF(0), > top_view->view_db.str, top_view->view_name.str); > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index fcfe490c035..d01c195cb8a 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -15676,7 +15680,8 @@ user_maybe_role: > It's OK to use in-place lowercase as long as > the character set is utf8. > */ > - my_casedn_str(system_charset_info, (char*) $$->host.str); > + $$->host.length= my_casedn_str(system_charset_info, > + (char*) $$->host.str); I thought my_casedn_str() changes were in the different commit? > } > else > { > diff --git a/sql/structs.h b/sql/structs.h > index 75bdc9e276b..d349ab72970 100644 > --- a/sql/structs.h > +++ b/sql/structs.h > @@ -321,11 +322,25 @@ typedef struct user_conn { > uint conn_per_hour, updates, questions; > /* Maximum amount of resources which account is allowed to consume. */ > USER_RESOURCES user_resources; > + > + /* > + The CHARSET_INFO used for hashes to compare the entire 'user\0hash' key. > + QQ: perhaps the user should be compared case sensitively, > + while host host should be comparad case insensitively. correct > + */ > + static CHARSET_INFO *user_host_key_charset_info_for_hash() > + { > + return &my_charset_utf8mb3_general1400_as_ci; > + } > } USER_CONN; > > typedef struct st_user_stats > { > char user[MY_MAX(USERNAME_LENGTH, LIST_PROCESS_HOST_LEN) + 1]; > + static CHARSET_INFO *user_key_charset_info_for_hash() > + { > + return &my_charset_utf8mb3_general1400_as_ci; > + } > // Account name the user is mapped to when this is a user from mapped_user. > // Otherwise, the same value as user. > char priv_user[MY_MAX(USERNAME_LENGTH, LIST_PROCESS_HOST_LEN) + 1]; > diff --git a/sql/table.cc b/sql/table.cc > index 8bc16753e68..d9bc5fabb4f 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -88,23 +88,34 @@ struct extra2_fields > static Virtual_column_info * unpack_vcol_info_from_frm(THD *, > TABLE *, String *, Virtual_column_info **, bool *); > > +/* > + Lex_ident_db does not have operator""_Lex_ident_db, > + because its constructor contains DBUG_SLOW_ASSERT > + and therefore it's not a constexpr constructor. > + So let's initialize a number of Lex_ident_db constants > + using operator""_Lex_cstring. > +*/ > + > /* INFORMATION_SCHEMA name */ > -LEX_CSTRING INFORMATION_SCHEMA_NAME= {STRING_WITH_LEN("information_schema")}; > +Lex_ident_db > + INFORMATION_SCHEMA_NAME= Lex_ident_db("information_schema"_Lex_cstring); don't think so. information_schema is case insensitive, even if fs is case sensitivity > > /* PERFORMANCE_SCHEMA name */ > -LEX_CSTRING PERFORMANCE_SCHEMA_DB_NAME= {STRING_WITH_LEN("performance_schema")}; > +Lex_ident_db > + PERFORMANCE_SCHEMA_DB_NAME= Lex_ident_db("performance_schema"_Lex_cstring); same > > /* MYSQL_SCHEMA name */ > -LEX_CSTRING MYSQL_SCHEMA_NAME= {STRING_WITH_LEN("mysql")}; > +Lex_ident_db > + MYSQL_SCHEMA_NAME= Lex_ident_db("mysql"_Lex_cstring); this uses fs case sensitivity, correct > > /* GENERAL_LOG name */ > -LEX_CSTRING GENERAL_LOG_NAME= {STRING_WITH_LEN("general_log")}; > +Lex_ident_table GENERAL_LOG_NAME= "general_log"_Lex_ident_table; > > /* SLOW_LOG name */ > -LEX_CSTRING SLOW_LOG_NAME= {STRING_WITH_LEN("slow_log")}; > +Lex_ident_table SLOW_LOG_NAME= "slow_log"_Lex_ident_table; > > -LEX_CSTRING TRANSACTION_REG_NAME= {STRING_WITH_LEN("transaction_registry")}; > -LEX_CSTRING MYSQL_PROC_NAME= {STRING_WITH_LEN("proc")}; > +Lex_ident_table TRANSACTION_REG_NAME= "transaction_registry"_Lex_ident_table; > +Lex_ident_table MYSQL_PROC_NAME= "proc"_Lex_ident_table; > > /* > Keyword added as a prefix when parsing the defining expression for a > @@ -280,42 +291,38 @@ const char *fn_frm_ext(const char *name) > } > > > -TABLE_CATEGORY get_table_category(const LEX_CSTRING *db, > - const LEX_CSTRING *name) > +TABLE_CATEGORY get_table_category(const Lex_ident_db &db, > + const Lex_ident_table &name) > { > - DBUG_ASSERT(db != NULL); > - DBUG_ASSERT(name != NULL); > - > #ifdef WITH_WSREP > - if (db->str && > - my_strcasecmp(system_charset_info, db->str, WSREP_SCHEMA) == 0) > + if (db.str && db.streq(MYSQL_SCHEMA_NAME)) //QQ: was a bug here? "was" > { > - if ((my_strcasecmp(system_charset_info, name->str, WSREP_STREAMING_TABLE) == 0 || > - my_strcasecmp(system_charset_info, name->str, WSREP_CLUSTER_TABLE) == 0 || > - my_strcasecmp(system_charset_info, name->str, WSREP_MEMBERS_TABLE) == 0)) > + if (name.streq(Lex_ident_table{STRING_WITH_LEN(WSREP_STREAMING_TABLE)}) || > + name.streq(Lex_ident_table{STRING_WITH_LEN(WSREP_CLUSTER_TABLE)}) || > + name.streq(Lex_ident_table{STRING_WITH_LEN(WSREP_MEMBERS_TABLE)})) > { > return TABLE_CATEGORY_INFORMATION; > } > } > #endif /* WITH_WSREP */ > - if (is_infoschema_db(db)) > + if (is_infoschema_db(&db)) > return TABLE_CATEGORY_INFORMATION; > > - if (is_perfschema_db(db)) > + if (is_perfschema_db(&db)) > return TABLE_CATEGORY_PERFORMANCE; > > - if (lex_string_eq(&MYSQL_SCHEMA_NAME, db)) > + if (db.streq(MYSQL_SCHEMA_NAME)) // QQ: was a bug here? "was" > { > - if (is_system_table_name(name->str, name->length)) > + if (is_system_table_name(name.str, name.length)) > return TABLE_CATEGORY_SYSTEM; > > - if (lex_string_eq(&GENERAL_LOG_NAME, name)) > + if (name.streq(GENERAL_LOG_NAME)) // QQ: was bug here? "was" > return TABLE_CATEGORY_LOG; > > - if (lex_string_eq(&SLOW_LOG_NAME, name)) > + if (name.streq(SLOW_LOG_NAME)) // QQ: was a bug here? "was" > return TABLE_CATEGORY_LOG; > > - if (lex_string_eq(&TRANSACTION_REG_NAME, name)) > + if (name.streq(TRANSACTION_REG_NAME)) // QQ: was a bug here? "was" > return TABLE_CATEGORY_LOG; > } > > diff --git a/sql/table.h b/sql/table.h > index 35be3bd4b2d..fe65025478d 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -3402,27 +3409,27 @@ static inline int set_zone(int nr,int min_zone,int max_zone) > } > > /* performance schema */ > -extern LEX_CSTRING PERFORMANCE_SCHEMA_DB_NAME; > +extern Lex_ident_db PERFORMANCE_SCHEMA_DB_NAME; > > -extern LEX_CSTRING GENERAL_LOG_NAME; > -extern LEX_CSTRING SLOW_LOG_NAME; > -extern LEX_CSTRING TRANSACTION_REG_NAME; > +extern Lex_ident_table GENERAL_LOG_NAME; > +extern Lex_ident_table SLOW_LOG_NAME; > +extern Lex_ident_table TRANSACTION_REG_NAME; > > /* information schema */ > -extern LEX_CSTRING INFORMATION_SCHEMA_NAME; > -extern LEX_CSTRING MYSQL_SCHEMA_NAME; > +extern Lex_ident_db INFORMATION_SCHEMA_NAME; > +extern Lex_ident_db MYSQL_SCHEMA_NAME; > > /* table names */ > -extern LEX_CSTRING MYSQL_PROC_NAME; > +extern Lex_ident_table MYSQL_PROC_NAME; > > inline bool is_infoschema_db(const LEX_CSTRING *name) > { > - return lex_string_eq(&INFORMATION_SCHEMA_NAME, name); > + return lex_string_eq(&INFORMATION_SCHEMA_NAME, name); // QQ: use Lex_ident_ci ? yes > } > > inline bool is_perfschema_db(const LEX_CSTRING *name) > { > - return lex_string_eq(&PERFORMANCE_SCHEMA_DB_NAME, name); > + return lex_string_eq(&PERFORMANCE_SCHEMA_DB_NAME, name); // QQ: use Lex_ident_ci? yes > } > > inline void mark_as_null_row(TABLE *table) > diff --git a/storage/blackhole/ha_blackhole.cc b/storage/blackhole/ha_blackhole.cc > index 343f3c70286..ce48c95f837 100644 > --- a/storage/blackhole/ha_blackhole.cc > +++ b/storage/blackhole/ha_blackhole.cc > @@ -416,7 +416,9 @@ static int blackhole_init(void *p) > mysql_mutex_init(bh_key_mutex_blackhole, > &blackhole_mutex, MY_MUTEX_INIT_FAST); > (void) my_hash_init(PSI_INSTRUMENT_ME, &blackhole_open_tables, > - system_charset_info, 32, 0, 0, > + // QQ: was a bug here? "was" > + Lex_ident_table::charset_info(), > + 32, 0, 0, > (my_hash_get_key) blackhole_get_key, > (my_hash_free_key) blackhole_free_key, 0); > > diff --git a/storage/csv/ha_tina.cc b/storage/csv/ha_tina.cc > index e705ff0e7c0..9fc0bcf604b 100644 > --- a/storage/csv/ha_tina.cc > +++ b/storage/csv/ha_tina.cc > @@ -184,7 +184,9 @@ static int tina_init_func(void *p) > tina_hton= (handlerton *)p; > mysql_mutex_init(csv_key_mutex_tina, &tina_mutex, MY_MUTEX_INIT_FAST); > (void) my_hash_init(csv_key_memory_tina_share, &tina_open_tables, > - system_charset_info, 32, 0, 0, (my_hash_get_key) > + // QQ: was a bug here? "was" > + table_alias_charset, > + 32, 0, 0, (my_hash_get_key) > tina_get_key, 0, 0); > tina_hton->db_type= DB_TYPE_CSV_DB; > tina_hton->create= tina_create_handler; > diff --git a/storage/innobase/dict/dict0dict.cc b/storage/innobase/dict/dict0dict.cc > index 5545727b015..231bcc1a132 100644 > --- a/storage/innobase/dict/dict0dict.cc > +++ b/storage/innobase/dict/dict0dict.cc > @@ -284,27 +284,26 @@ otherwise table->n_def */ > ulint > dict_table_has_column( > const dict_table_t* table, > - const char* col_name, > + const LEX_CSTRING &col_name, looks ok to me, but to avoid surprises Marko should review it too > ulint col_nr) > { > ulint col_max = table->n_def; > > ut_ad(table); > - ut_ad(col_name); > + ut_ad(col_name.str); > ut_ad(table->magic_n == DICT_TABLE_MAGIC_N); > > if (col_nr < col_max > - && innobase_strcasecmp( > - col_name, dict_table_get_col_name(table, col_nr)) == 0) { > + && Lex_ident_column(col_name). > + streq(dict_table_get_col_name_ls(table, col_nr))) { > return(col_nr); > } > > /** The order of column may changed, check it with other columns */ > for (ulint i = 0; i < col_max; i++) { > if (i != col_nr > - && innobase_strcasecmp( > - col_name, dict_table_get_col_name(table, i)) == 0) { > - > + && Lex_ident_column(col_name). > + streq(dict_table_get_col_name_ls(table, i))) { > return(i); > } > } > diff --git a/storage/mroonga/ha_mroonga.cpp b/storage/mroonga/ha_mroonga.cpp > index 44fd09e74ed..71784dad475 100644 > --- a/storage/mroonga/ha_mroonga.cpp > +++ b/storage/mroonga/ha_mroonga.cpp > @@ -1936,7 +1936,10 @@ static int mrn_init(void *p) > MY_MUTEX_INIT_FAST) != 0)) { > goto err_allocated_thds_mutex_init; > } > - if (mrn_my_hash_init(&mrn_allocated_thds, system_charset_info, 32, 0, 0, > + if (mrn_my_hash_init(&mrn_allocated_thds, > + // QQ: was a bug here? "was" > + &my_charset_bin, > + 32, 0, 0, > mrn_allocated_thds_get_key, 0, 0)) { > goto error_allocated_thds_hash_init; > } > @@ -1945,7 +1948,10 @@ static int mrn_init(void *p) > MY_MUTEX_INIT_FAST) != 0)) { > goto err_allocated_open_tables_mutex_init; > } > - if (mrn_my_hash_init(&mrn_open_tables, system_charset_info, 32, 0, 0, > + if (mrn_my_hash_init(&mrn_open_tables, > + // QQ: was a bug here? "was" > + Lex_ident_table::charset_info(), > + 32, 0, 0, > mrn_open_tables_get_key, 0, 0)) { > goto error_allocated_open_tables_hash_init; > } > @@ -1954,7 +1960,10 @@ static int mrn_init(void *p) > MY_MUTEX_INIT_FAST) != 0)) { > goto error_allocated_long_term_share_mutex_init; > } > - if (mrn_my_hash_init(&mrn_long_term_share, system_charset_info, 32, 0, 0, > + if (mrn_my_hash_init(&mrn_long_term_share, > + // QQ: was a bug here? "was" > + Lex_ident_table::charset_info(), > + 32, 0, 0, > mrn_long_term_share_get_key, 0, 0)) { > goto error_allocated_long_term_share_hash_init; > } > diff --git a/storage/perfschema/pfs_engine_table.h b/storage/perfschema/pfs_engine_table.h > index 1ef1e0282de..b4f4c9f0acc 100644 > --- a/storage/perfschema/pfs_engine_table.h > +++ b/storage/perfschema/pfs_engine_table.h > @@ -25,6 +25,22 @@ > > #include "table.h" > #include "sql_acl.h" > + > +/* > + The performance schema is implemented as a storage engine, in memory. > + The current storage engine interface exposed by the server, > + and in particular handlerton::discover, uses 'FRM' files to describe a > + table structure, which are later stored on disk, by the server, > + in ha_create_table_from_engine(). > + Because the table metadata is stored on disk, the table naming rules > + used by the performance schema then have to comply with the constraints > + imposed by the disk storage, and in particular with lower_case_table_names. > +*/ > + > +using PFS_ident_db= Lex_ident_db; > +using PFS_ident_table = Lex_ident_table; No, table metadata for performance schema are never stored on disk. let's use case insensitive comparison here, if possible. > + > + > /** > @file storage/perfschema/pfs_engine_table.h > Performance schema tables (declarations). Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 5

Re: 9d6de9212d5: MDEV-29095 REGEXP_REPLACE treats empty strings different than REPLACE in ORACLE mode
by Sergei Golubchik 16 Jan '24

16 Jan '24

Hi, Alexander, On Jan 10, Alexander Barkov wrote: > revision-id: 9d6de9212d5 (mariadb-10.4.32-45-g9d6de9212d5) > parent(s): bdfd93d30c1 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-11-24 13:23:56 +0400 > message: > > MDEV-29095 REGEXP_REPLACE treats empty strings different than REPLACE in ORACLE mode > > Turning REGEXP_REPLACE into two schema-qualified functions: > - mariadb_schema.regexp_replace() > - oracle_schema.regexp_replace() > > Fixing oracle_schema.regexp_replace(subj,pattern,replacement) to treat > NULL in "replacement" as an empty string. > > Adding new classes implementing oracle_schema.regexp_replace(): > - Item_func_regexp_replace_oracle > - Create_func_regexp_replace_oracle > > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 92d5e196da4..b688ba97d3e 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -1385,20 +1385,22 @@ bool Item_func_regexp_replace::append_replacement(String *str, > } > > > -String *Item_func_regexp_replace::val_str(String *str) > +String *Item_func_regexp_replace::val_str_internal(String *str, > + bool null_to_empty) > { > DBUG_ASSERT(fixed == 1); > char buff0[MAX_FIELD_WIDTH]; > char buff2[MAX_FIELD_WIDTH]; > String tmp0(buff0,sizeof(buff0),&my_charset_bin); > String tmp2(buff2,sizeof(buff2),&my_charset_bin); > - String *source= args[0]->val_str(&tmp0); > - String *replace= args[2]->val_str(&tmp2); > + String *source, *replace; > LEX_CSTRING src, rpl; > int startoffset= 0; > > - if ((null_value= (args[0]->null_value || args[2]->null_value || > - re.recompile(args[1])))) > + if ((null_value= > + (!(source= args[0]->val_str(&tmp0)) || > + !(replace= args[2]->val_str_null_to_empty(&tmp2, null_to_empty)) || why are you fixing it differently from Item_func_replace? If you think this approach is better then, please, change Item_func_replace to use it too, in a followup commit. > + re.recompile(args[1])))) > return (String *) 0; > > if (!(source= re.convert_if_needed(source, &re.subject_converter)) || > diff --git a/sql/item_strfunc.h b/sql/item_strfunc.h > index 826c7f784af..e014cc5fcbc 100644 > --- a/sql/item_strfunc.h > +++ b/sql/item_strfunc.h > @@ -412,11 +420,34 @@ class Item_func_regexp_replace :public Item_str_func > re.cleanup(); > DBUG_VOID_RETURN; > } > - String *val_str(String *str); > + String *val_str(String *str) > + { > + return val_str_internal(str, false); > + } > bool fix_fields(THD *thd, Item **ref); > bool fix_length_and_dec(); > const char *func_name() const { return "regexp_replace"; } > - Item *get_copy(THD *thd) { return 0;} > + Item *get_copy(THD *thd) { return 0;} // QQ: why? you could've checked the history in git. it was commit 343bcb152f8 Author: Igor Babaev <igor(a)askmonty.org> Date: Sun Nov 5 22:52:41 2017 -0800 Fixed mdev-14237 Server crash on query with regexp_substr It's better to prohibit pushdown of conditions that involve regexp_substr() and regexp_replace() into materialized derived tables / views until proper implementations of the get_copy() virtual method are provided for those functions. and I suspect the reason is Regexp_processor_pcre inside regexp Items. > +}; > + > + > +class Item_func_regexp_replace_oracle: public Item_func_regexp_replace > +{ > +public: > + Item_func_regexp_replace_oracle(THD *thd, Item *a, Item *b, Item *c) > + :Item_func_regexp_replace(thd, a, b, c) > + {} > + const Schema *schema() const { return &oracle_schema_ref; } > + bool fix_length_and_dec() > + { > + bool rc= Item_func_regexp_replace::fix_length_and_dec(); > + maybe_null= true; // Empty result is converted to NULL > + return rc; > + } > + String *val_str(String *str) > + { > + return val_str_internal(str, true); > + } > }; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 22e2f082400: MDEV-25370 Update for portion changes autoincrement key in bi-temp table
by Sergei Golubchik 12 Jan '24

12 Jan '24

Hi, Nikita, On Jan 06, Nikita Malyavin wrote: > revision-id: 22e2f082400 (mariadb-10.5.23-39-g22e2f082400) > parent(s): e472b682e06 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-12-20 02:46:09 +0100 > message: > > MDEV-25370 Update for portion changes autoincrement key in bi-temp table > > According to the standard, the autoincrement column (i.e. *identity > column*) should be advanced each insert implicitly made by > UPDATE/DELETE ... FOR PORTION. When you refer to the standard, specify the exact reference so that someone could verify it. E.g. as I've commented in MDEV, "SQL:2016, Part 2, 15.13 Effect of replacing rows in base tables, paragraph 10) b) i)" Also, the standard doesn't have autoincrement, it'd be better to say "According to the standard, the identity column (which autoincrement is a functional analog of) should be advanced...." > This is very unconvenient use in several notable cases. Concider a > WITHOUT OVERLAPS key with an autoinc column: > id int auto_increment, unique(id, p without overlaps) > > An update or delete with FOR PORTION creates a sense that id will > remain unchanged in such case. > > This commit makes an exception the autoinc columns that are part of > WITHOUT OVERLAPS keys and infers the following rule for the newly > inserted rows: Better to say here not that we're making an exception and violating the standard (because we are not), but that in the standard IDENTITY uses its own independent sequence generator, which knows nothing about periods. While autoincrement always requires an index (that is, our "identity" column is a pair {autoincrement, index}, not autoincrement alone), and the index can be declared WITHOUT OVERLAPS, so our "identity" column is period-aware. This is an extension of the standard, and we can freely define how it behaves in the cases not covered by the standard, that is in the WITHOUT OVERLAPS case. > ===== > Let V_j a set of values for currently updating/deleting row. > Let VI_j a set of values to be inserted. > * Case: > (a) if V_i is an identity column that is also a part of a unique constraint > that includes <without overlap specification>, then let VI_i be V_i > (b) otherwise, let VI_i be the value deduced as described by the rules of > ISO/IEC 9075-2, clauses 15.7 Effect of deleting rows from base tables > and 15.13 Effect of replacing rows in base tables. > ===== That's confusing, I'm sure that anyone who doesn't remember the corresponding part of the sql standard will think it's a direct quote from it. You've imitated the style rather convincingly :) It's also rather unnecessary, we don't have identity columns, so nobody cares how they should behave in this case. Perhaps it'd be better if you simply write that if the key, that defines autoincrement values, is declared WITHOUT OVERLAPS, then new rows created by UPDATE/DELETE FOR PORTION OF will use old autoincrement values, as they belong to non-overlapping ranges. But INSERT will generate a new autoincrement value, it will not try to find the smallest value in the overlapping ranges. > Note that this also infers that if an identity column is not a part of > any unique constraint that includes <without overlap specification>, then > its value will be DEFAULT and, that is, auto-incremented. > > Apart from WITHOUT OVERLAPS there is also another notable case, referred > by the reporter - a unique key that has an autoincrement column and a field > from the period specification: > id int auto_increment, unique(id, s), period for p(s, e) > > for this case, no exception is made, and the autoincrementing rules will be > proceeded accordung to the standard (i.e. the value will be advanced on > implicit inserts). > > diff --git a/mysql-test/suite/period/r/overlaps.result b/mysql-test/suite/period/r/overlaps.result > --- a/mysql-test/suite/period/r/overlaps.result > +++ b/mysql-test/suite/period/r/overlaps.result > @@ -449,3 +449,86 @@ VALUES > ('2000-01-01 00:00:00.000000', '2001-01-01 00:00:00.000000', 'abc'); > ERROR 23000: Duplicate entry 'abc-2001-01-01 00:00:00.000000-2000-01-01 00:00:00.000000' for key 'index_name' > DROP TABLE t1; > +create or replace table cars(id int auto_increment, > +price int, s date, e date, > +period for p(s,e), > +primary key(id, p without overlaps)); > +insert into cars(price, s, e) values (1000, '2018-01-01', '2020-01-01'); > +select * from cars; > +id price s e > +1 1000 2018-01-01 2020-01-01 > +update cars for portion of p from '2019-01-01' to '2019-12-01' set price= 1100; > +select * from cars; > +id price s e > +1 1000 2018-01-01 2019-01-01 > +1 1000 2019-12-01 2020-01-01 > +1 1100 2019-01-01 2019-12-01 > +delete from cars for portion of p from '2019-12-10' to '2019-12-20'; > +select * from cars; > +id price s e > +1 1000 2018-01-01 2019-01-01 > +1 1000 2019-12-01 2019-12-10 > +1 1000 2019-12-20 2020-01-01 > +1 1100 2019-01-01 2019-12-01 ok > +# AUTO_INCREMENT field is separate from WITHOUT OVERLAPS > +create or replace table cars(id int primary key auto_increment, > +car_id int, > +price int, s date, e date, > +period for p(s,e), > +unique(car_id, p without overlaps)); > +insert cars(car_id, price, s, e) values (1, 1000, '2018-01-01', '2020-01-01'); > +select * from cars; > +id car_id price s e > +1 1 1000 2018-01-01 2020-01-01 > +update cars for portion of p from '2019-01-01' to '2019-12-01' set price= 1100; > +select * from cars; > +id car_id price s e > +1 1 1100 2019-01-01 2019-12-01 > +2 1 1000 2018-01-01 2019-01-01 > +3 1 1000 2019-12-01 2020-01-01 > +delete from cars for portion of p from '2019-12-10' to '2019-12-20'; > +select * from cars; > +id car_id price s e > +1 1 1100 2019-01-01 2019-12-01 > +2 1 1000 2018-01-01 2019-01-01 > +4 1 1000 2019-12-01 2019-12-10 > +5 1 1000 2019-12-20 2020-01-01 right > +# AUTO_INCREMENT field is both standalone and in WITHOUT OVERLAPS > +create or replace table cars(id int primary key auto_increment, > +price int, s date, e date, > +period for p(s,e), > +unique(id, p without overlaps)); > +insert cars(price, s, e) values (1000, '2018-01-01', '2020-01-01'); > +select * from cars; > +id price s e > +1 1000 2018-01-01 2020-01-01 > +update cars for portion of p from '2019-01-01' to '2019-12-01' set price= 1100; > +ERROR 23000: Duplicate entry '1' for key 'PRIMARY' uhm, I don't know. I'd expect that here PRIMARY would define autoincrement values. > +truncate cars; > +insert cars(price, s, e) values (1000, '2018-01-01', '2020-01-01'); > +delete from cars for portion of p from '2019-12-10' to '2019-12-20'; > +ERROR 23000: Duplicate entry '1' for key 'PRIMARY' > +# AUTO_INCREMENT field is non-unique not sure here. I think here unique should define autoincrement values. That is, in both cases, the most restrictive index should define. > +create or replace table cars(id int auto_increment, key(id), > +car_id int, > +price int, s date, e date, > +period for p(s,e), > +unique(car_id, p without overlaps)); > +insert cars(car_id, price, s, e) values (1, 1000, '2018-01-01', '2020-01-01'); > +select * from cars; > +id car_id price s e > +1 1 1000 2018-01-01 2020-01-01 > +update cars for portion of p from '2019-01-01' to '2019-12-01' set price= 1100; > +select * from cars; > +id car_id price s e > +1 1 1100 2019-01-01 2019-12-01 > +2 1 1000 2018-01-01 2019-01-01 > +3 1 1000 2019-12-01 2020-01-01 > +delete from cars for portion of p from '2019-12-10' to '2019-12-20'; > +select * from cars; > +id car_id price s e > +1 1 1100 2019-01-01 2019-12-01 > +2 1 1000 2018-01-01 2019-01-01 > +4 1 1000 2019-12-01 2019-12-10 > +5 1 1000 2019-12-20 2020-01-01 please also add test for INSERT. Just add a second insert like insert cars(price, s, e) values (1000, '2021-01-01', '2022-01-01'); to show that id will *not* be 1 here, even though 1 would not conflict with anything in the specified date range. Simply to have the expected behavior recorded as a test result. > +drop table cars; > diff --git a/sql/table.cc b/sql/table.cc > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -8917,6 +8916,21 @@ int TABLE::update_generated_fields() > return res; > } > > +void TABLE::period_prepare_autoinc() > +{ > + if (!found_next_number_field) > + return; > + > + key_map::Iterator it(found_next_number_field->part_of_key_not_clustered); > + for (uint i; (i= it++) != key_map::Iterator::BITMAP_END;) > + { > + KEY &key= key_info[i]; > + if (key.without_overlaps) > + return; > + } No, I don't think you need to check *all* indexes that the field is part of. I believe you should only check the "main" index, the one that defines autoincrement field behavior. That is only if (key_info[s->next_number_index].without_overlaps) return; > + next_number_field= found_next_number_field; > +} > + > int TABLE::period_make_insert(Item *src, Field *dst) > { > THD *thd= in_use; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 10

Re: 6835d2ce268: MDEV-28649 json_array(false) in the definition of view return 0 instead of "true".
by Sergei Golubchik 12 Jan '24

12 Jan '24

Hi, Alexey, On Jan 12, Alexey Botchkov wrote: > revision-id: 6835d2ce268 (mariadb-10.4.32-116-g6835d2ce268) > parent(s): fbe604d8839 > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2023-12-15 00:56:20 +0400 > message: > > MDEV-28649 json_array(false) in the definition of view return 0 instead of "true". > > The 'temporary-only Field_bool introduced'; Why did you need a new Field and to change Type_handler_bool from long to tiny? On the first glance it seems that a dedicated Item_bool::print() should've been enough. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 424ee6eac78: MDEV-32837 long unique does not work like unique key when using replace
by Sergei Golubchik 11 Jan '24

11 Jan '24

Hi, Alexander, On Jan 11, Alexander Barkov wrote: > revision-id: 424ee6eac78 (mariadb-10.5.23-57-g424ee6eac78) > parent(s): c9902a20b3a > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2024-01-10 15:35:25 +0400 > message: > > MDEV-32837 long unique does not work like unique key when using replace > > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index ec9e379768f..edc2be207f3 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -2065,7 +2065,8 @@ int write_record(THD *thd, TABLE *table, COPY_INFO *info, select_result *sink) > we just should not expose this fact to users by invoking > ON UPDATE triggers. > */ > - if (last_uniq_key(table,key_nr) && > + if ((!table->s->long_unique_table || table->versioned()) && > + last_uniq_key(table,key_nr) && > !table->file->referenced_by_foreign_key() && > (!table->triggers || !table->triggers->has_delete_triggers())) > { 1. why table->versioned() ? 2. use show status in your test for handler_update/delete to see what REPLACE actually did 3. technically, you can keep the optimization if the long unique is the last *and there are no in-engine uniques*, this can be checked with && !(table->key_info->flags & HA_NOSAME) 4. but here's the problem that last_uniq_key() doesn't see long uniques, so it'll treat every one of them as the last. So it should be something like bool is_long_unique= table->s->key_info[key_nr].flags & HA_UNIQUE_HASH; if ((is_long_unique ? last_uniq_key(table, table->s->key_info, key_nr) && !(table->key_info->flags & HA_NOSAME) : last_uniq_key(table, table->key_info, key_nr)) && !table->file->referenced_by_foreign_key() && (!table->triggers || !table->triggers->has_delete_triggers())) I suspect this will correctly use the UPDATE optimization in all cases (only long unique / mix of uniques)x(last / not last key conflicting) Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 014167880d2: MDEV-32559 ensure spider alter table init queries are executed after ddl recovery
by Sergei Golubchik 10 Jan '24

10 Jan '24

Hi, Yuchen, On Jan 05, Yuchen Pei wrote: > revision-id: 014167880d2 (mariadb-10.10.4-52-g014167880d2) > parent(s): 21822625aa6 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-11-08 13:10:38 +1100 > message: > > MDEV-32559 ensure spider alter table init queries are executed after ddl recovery > > If spider was initialised in init_server_components(), i.e. with > --plugin-load-add, then all alter table init queries are to be > executed in the signal_ddl_recovery_done() handlerton callback. Since > this is part of the spider plugin init, we update the callback > signature to return an int, and deinit the plugin when it fails the > callback. > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index a3971a00a59..a922562d043 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -5159,6 +5159,14 @@ static int init_server_components() > if (!opt_abort && ddl_log_initialize()) > unireg_abort(1); > > + /* Plugin initialisation done here should defer any ALTER TABLE > + queries to after the ddl recovery is done, in the > + signal_ddl_recovery_done() callback called by > + ha_signal_ddl_recovery_done(). As such, between the plugin_init() > + call and the ha_signal_ddl_recovery_done() call below only things > + related to prepare for recovery should be done and nothing else, and > + definitely not anything assuming that all plugins have been > + initialised. */ Better put this comment in plugin.h near the (*init)(void*) declaration. This is part of the (*init)() callback documentation it should be there, not in some random .cc file that plugin authors shouldn't be required to look at. > if (plugin_init(&remaining_argc, remaining_argv, > (opt_noacl ? PLUGIN_INIT_SKIP_PLUGIN_TABLE : 0) | > (opt_abort ? PLUGIN_INIT_SKIP_INITIALIZATION : 0))) > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 3576730723e..85a8f14a6da 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -2553,9 +2554,15 @@ bool plugin_foreach_with_mask(THD *thd, plugin_foreach_func *func, > > for (idx= 0; idx < total; idx++) > { > - /* It will stop iterating on first engine error when "func" returns TRUE */ > + /* It will stop iterating on first engine error or mark DELETED if > + reap_on_fail is true when "func" returns TRUE. */ > if ((res= func(thd, plugins[idx], arg))) > + { > + if (reap_on_fail) > + plugin_ref_to_int(plugins[idx])->state= PLUGIN_IS_DELETED; may be you don't need a new argument for plugin_foreach_with_mask, and can put this logic into the `func` callback? That is into static my_bool signal_ddl_recovery_done(THD *, plugin_ref plugin, void *) { handlerton *hton= plugin_hton(plugin); if (hton->signal_ddl_recovery_done) if ((hton->signal_ddl_recovery_done)(hton)) plugin_ref_to_int(plugin)->state= PLUGIN_IS_DELETED; return 0; } or something like that. > + else > break; > + } > } > > plugin_unlock_list(0, plugins, total); > diff --git a/storage/spider/spd_table.cc b/storage/spider/spd_table.cc > index a9782a8a822..e50942d449f 100644 > --- a/storage/spider/spd_table.cc > +++ b/storage/spider/spd_table.cc > @@ -6376,15 +6376,13 @@ bool spider_init_system_tables() > DBUG_RETURN(TRUE); > } > > - const int size= sizeof(spider_init_queries) / sizeof(spider_init_queries[0]); > - for (int i= 0; i < size; i++) > + for (uint i= 0; i < size; i++) > { > - const LEX_STRING *query= &spider_init_queries[i]; > + const LEX_STRING *query= &queries[i]; I don't see much of a point, why not to run *all* queries from spider_ddl_recovery_done() ? > if (mysql_real_query(mysql, query->str, query->length)) > { > - fprintf(stderr, > - "[ERROR] SPIDER plugin initialization failed at '%s' by '%s'\n", > - query->str, mysql_error(mysql)); > + sql_print_error("SPIDER: plugin initialization failed at '%s' by '%s'\n", > + query->str, mysql_error(mysql)); > > mysql_close(mysql); > DBUG_RETURN(TRUE); > @@ -6667,10 +6665,20 @@ int spider_db_init( > spider_udf_table_mon_list_hash[roop_count].array.size_of_element); > } > > - if (spider_init_system_tables()) > - { > + if (spider_init_system_tables( > + spider_init_queries, > + sizeof(spider_init_queries) / sizeof(spider_init_queries[0]))) > + goto error_system_table_creation; > + > + /* If plugin_init() has already been called, it implies this > + function is called after the ddl recovery is done, thus we can > + safely execute ALTER TABLE init queries. Otherwise it is deferred to > + the spider_ddl_recovery_done() callback. */ > + if (plugins_are_initialized && > + spider_init_system_tables(spider_init_alter_table_queries, > + sizeof(spider_init_alter_table_queries) / > + sizeof(spider_init_alter_table_queries[0]))) Don't do that, instead, please, take the "ddl_recovery_done" change from the commit 0930eb86cb00edb2a - it makes the storage engine API simpler, the engine doesn't need to implement two code paths and try to detect whether it's loaded at run-time or at startup. > goto error_system_table_creation; > - } > > if (!(spider_table_sts_threads = (SPIDER_THREAD *) > spider_bulk_malloc(NULL, 256, MYF(MY_WME | MY_ZEROFILL), > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: fixes for your commits to make buildbot green
by Kristian Nielsen 09 Jan '24

09 Jan '24

Sergei Golubchik <serg(a)mariadb.org> writes: > as you know, I've cherry-picked all your 11.4 features into one branch > to assemble a preview release 11.4.0 Thanks, Serg! > This has required some fixes to make buildbot greener. Thanks. I think buildbot was not active on 11.4 branches when I tried earlier, but now they seem to run. Was something changed in buildbot around 11.4 branches recently? > There's nothing specific you need to do right now. But eventually you'll > rebase on top of the latest 11.4. Somewhere around that time, please > take a look at bb-11.4-release or bb-11.4-serg branch, see commits > related to your MDEV's and consider them. > a40dcb3da6a MDEV-4991 don't use ulong for sysvars > 723ebcb5df7 MDEV-4991 ASAN bug > f3d2d4609c3 MDEV-4991 compilation failure on fulltest2 > bd295d0919b MDEV-4991 update forgotten test results These are now applied (with a few modifications) and rebased on latest 11.4, in bb-11.4-knielsen-mdev4991 > I suggest you squash them with your commits, no need to keep indivial Sure, will do before final merge to 11.4. When is the deadline for pushing into the 11.4 release? - Kristian.

2 1

Re: [PATCH] MDEV-33031 Assertion failure upon reading from performance
by Yuchen Pei 09 Jan '24

09 Jan '24

Hi Sergei, Regarding 3be269542f4d18eaee0ad8fbeffa55708557879f, I have one comment (see below), otherwise ok to push. > [... 62 lines elided] > diff --git a/storage/spider/spd_param.cc b/storage/spider/spd_param.cc > index 2da262cd2bc..308857d153a 100644 > --- a/storage/spider/spd_param.cc > +++ b/storage/spider/spd_param.cc > @@ -116,16 +116,20 @@ extern volatile ulonglong spider_mon_table_cache_version_req; > MYSQL_SYSVAR_NAME(param_name).def_val; \ > } > +extern handlerton *spider_hton_ptr; > static int spider_trx_status_var(THD *thd, SHOW_VAR *var, char *buff, > ulonglong SPIDER_TRX::*counter) > { > - int error_num = 0; > - SPIDER_TRX *trx; > DBUG_ENTER("spider_direct_update"); > var->type = SHOW_LONGLONG; > - if ((trx = spider_get_trx(thd, TRUE, &error_num))) > - var->value = (char *) &(trx->*counter); > - DBUG_RETURN(error_num); > + var->value= buff; > + if (thd != current_thd) > + mysql_mutex_lock(&thd->LOCK_thd_data); > + SPIDER_TRX *trx = (SPIDER_TRX*)thd_get_ha_data(thd, spider_hton_ptr); > + *(ulonglong*)buff= trx ? trx->*counter : 0; > + if (thd != current_thd) > + mysql_mutex_unlock(&thd->LOCK_thd_data); > + DBUG_RETURN(0); If it always returns 0, how about we change the signature of this function to return void? > } Best, Yuchen

2 1

Re: [PATCH] cleanup: spider status variables
by Yuchen Pei 09 Jan '24

09 Jan '24

Hi Sergei, > reduce code duplication > --- > storage/spider/spd_param.cc | 49 ++++++++++++------------------------- > 1 file changed, 16 insertions(+), 33 deletions(-) b7bcbda853947a5be8307eda7026ca1ff567a885 looks ok to push. > [... 93 lines elided] Best, Yuchen

1 0

Re: dc50afee924: MDEV-32501 KEY_PERIOD_USAGE reveals information to unprivileged user
by Sergei Golubchik 08 Jan '24

08 Jan '24

Hi, Nikita, ok to push under condition that you do the change as below, if it's applicable. On Jan 05, Nikita Malyavin wrote: > revision-id: dc50afee924 (mariadb-11.3.1-13-gdc50afee924) > parent(s): 080d8c3bd8f > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2024-01-05 00:07:06 +0100 > message: > > MDEV-32501 KEY_PERIOD_USAGE reveals information to unprivileged user > > Restrict access to KEY_PERIOD_USAGE: show the constraint record iff any > non-select privilege on any table column is granted. > > Also drop the unprivileged user in the end of test and add merge anchor. > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 95a9d3647b0..e5516466e20 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -7798,6 +7798,12 @@ int get_schema_key_period_usage_record(THD *thd, TABLE_LIST *tables, > if (!period_name) > return 0; > > +#ifndef NO_EMBEDDED_ACCESS_CHECKS > + check_grant(thd, TABLE_ACLS, tables, 1, 1, 1); > + if ((tables->grant.all_privilege() & ~SELECT_ACL & TABLE_ACLS) == NO_ACL) > + return 0; > +#endif Compare with get_schema_key_column_usage_record() above your get_schema_key_period_usage_record(). Perhaps you can also check `thd->col_access` and avoid check_grant()? > bool err= false; > for (uint k= 0; !err && k < keys_total; k++) > { > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: c50eed1e904: MDEV-32645 CAST(AS UNSIGNED) fails with --view-protocol
by Sergei Golubchik 06 Jan '24

06 Jan '24

Hi, Alexander, Just one comment below On Jan 06, Alexander Barkov wrote: > revision-id: c50eed1e904 (mariadb-10.4.30-318-gc50eed1e904) > parent(s): 28cdbab1637 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-11-13 11:18:16 +0400 > message: > > MDEV-32645 CAST(AS UNSIGNED) fails with --view-protocol > > Item_float::neg() did not preserve the "presentation" from "this". > So > CAST(-1e0 AS UNSIGNED) -- cast from double to unsigned > changes its meaning to: > CAST(-1 AS UNSIGNED) -- cast signed to undigned > > Fixing Item_float::neg() to construct the new value for > Item_float::presentation as minus sign followed by its old value: > '1e0' -> '-1e0' > > diff --git a/sql/item.cc b/sql/item.cc > index 259c4bf8f4d..b88f78fbb59 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -6896,7 +6896,17 @@ Item *Item_float::neg(THD *thd) > else if (value < 0 && max_length) > max_length--; > value= -value; > - presentation= 0; > + if (presentation) > + { may be, still if (*presentation == '-') presentation++ else > + size_t presentation_length= strlen(presentation); > + if (char *tmp= (char*) thd->alloc(presentation_length + 2)) > + { > + tmp[0]= '-'; > + // Copy with the trailing '\0' > + memcpy(tmp + 1, presentation, presentation_length + 1); > + presentation= tmp; > + } > + } > name= null_clex_str; > return this; > } > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 9e7c80f8006: MDEV-32803 Assertion `total == 0' failed in Event_log::write_cache_raw
by Sergei Golubchik 05 Jan '24

05 Jan '24

Hi, Nikita, First, I'd suggest this change: --- a/sql/online_alter.cc +++ b/sql/online_alter.cc @@ -257,8 +257,7 @@ int online_alter_end_trans(Online_alter_cache_list { my_error(ER_ERROR_ON_WRITE, MYF(ME_ERROR_LOG), binlog->get_name(), errno); - cleanup_cache_list(cache_list, is_ending_transaction); - DBUG_RETURN(error); + break; } } and also I'd suggest to move if (ending_trans) from cleanup_cache_list() to the caller (there's only one anyway). It'll make the code clearer, in particular your statement below that "the cache ... wasn't destroyed". Better do these changes in a separate commit, they aren't part of the bug fix. The fix itself is ok to push. On Jan 05, Nikita Malyavin wrote: > revision-id: 9e7c80f8006 (mariadb-11.2.1-27-g9e7c80f8006) > parent(s): 0c5176b7744 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-11-17 23:40:10 +0100 > message: > > MDEV-32803 Assertion `total == 0' failed in Event_log::write_cache_raw > > A second DML in a transaction for a table of non-rollbackable engine > leads to a cache corruption, because the cache wasn't reset after a > statement end, but also wasn't destroyed. > > This patch resets the cache for a reuse by subsequent statements in > current transaction. > > diff --git a/sql/online_alter.cc b/sql/online_alter.cc > index e0afe602ca8..5f418d8ec29 100644 > --- a/sql/online_alter.cc > +++ b/sql/online_alter.cc > @@ -152,6 +152,7 @@ int online_alter_log_row(TABLE* table, const uchar *before_record, > if (!table->online_alter_cache) > { > table->online_alter_cache= get_cache_data(thd, table); > + DBUG_ASSERT(table->online_alter_cache->cache_log.type == WRITE_CACHE); > trans_register_ha(thd, false, online_alter_hton, 0); > if (thd->in_multi_stmt_transaction_mode()) > trans_register_ha(thd, true, online_alter_hton, 0); > @@ -237,6 +238,8 @@ int online_alter_end_trans(Online_alter_cache_list &cache_list, THD *thd, > mysql_mutex_lock(binlog->get_log_lock()); > error= binlog->write_cache_raw(thd, &cache.cache_log); > mysql_mutex_unlock(binlog->get_log_lock()); > + if (!is_ending_transaction) > + cache.reset(); > } > } > else if (!commit) // rollback > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 1c9edc72fc2: MDEV-32830 I. refactor XA binlogging for better integration with BGC/replication/recovery
by Sergei Golubchik 05 Jan '24

05 Jan '24

Hi, Andrei, On Jan 05, Andrei wrote: > commit 1c9edc72fc2 > Author: Andrei <andrei.elkin(a)mariadb.com> > Date: Wed Nov 1 20:32:45 2023 +0200 > > MDEV-32830 I. refactor XA binlogging for better integration with BGC/replication/recovery > > This commit is the part I in the series of three that addresses MDEV-31949. > > This part improves upon MDEV-742 design of the XA binlogging. > When binlog is ON, to execute XA-prepare first in engines to write > the xa-prepare replication event last. > MYSQL_BIN_LOG::run_ordered_commit() is made to execute XA commit and > rollback in engines, as well as > it completes with the XA transaction state, so that its > XID has become available for next use, e.g by another slave parallel > worker, that e.g could be waiting for its order to commit a being > prepared XID transaction. Is it intentional that line length in the paragraph above varies randomly from line to line? And I don't understand why MYSQL_BIN_LOG::run_ordered_commit() has to do the complete XA commit in all engines? it's not its job. And it won't make XID available sooner, it'll only make XID available in a different function. > Notable changes: > > sql/handler.cc > - ha_prepare() now is ensured to execute binlog_hton::prepare() as > the last branch to prepare; recovery concerns are addressed > separately (not by MDEV-21469, but rather along the plan of bug#76233) > - read-only cases are handled more uniformly with extending > has_binlog_hton() that registers the binlog branch at ha_prepare() > when necessary > - ha_rollback_trans() executes binlog_hton::rollback() as first > branch, as a part of the 76233 planned recovery; of <what> planned recovery? > - ditto to the external completion of XA via ha_commit_or_rollback_by_xid(). > sql/log.cc > - binlog_commit,rollback() simplified/clarified on the condition of an empty > statement/trx > - MYSQL_BIN_LOG::run_commit_ordered() is extended to cover > XA-{commit, rollback} respective actions in Engines > - XID unlogging by XA is converted to employ the standard > binlog-checkpoint mechanism (cache_mngr->using_xa= TRUE) > sql/xa.cc > - trans_xa_commit,rollback() are refactored to invoke a common function > for external (connection) completion > - adapting the above two and slave_applier_reset_xa_trans() to > possible (normal to slave) XID release, inside run_ordered_commit(). > > diff --git a/sql/handler.cc b/sql/handler.cc > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1472,6 +1472,48 @@ static int prepare_or_error(handlerton *ht, THD *thd, bool all) > return err; > } > > +/** > + Search in @c ha_info transaction list branches for binlog. > + When the 2nd argument is not NULL and the modified engine number > + is non-zero the binlog branch is activated. > + > + @param ha_info The first node in the transaction descriptor list. > + @param thd The client thread that executes the transaction. > + > + @return true when the binlog branch is registered or set up for transaction, > + false otherwise. > +*/ > +static bool has_binlog_hton(Ha_trx_info *ha_info, THD* thd= NULL) > +{ > + bool rc; > + uint rw_count; > + > + for (rc= false, rw_count= 0; ha_info && !rc; ha_info= ha_info->next()) > + { > + rc= ha_info->ht() == binlog_hton; > + rw_count += (ha_info->is_trx_read_write()); > + } > + > + if (!rc && thd && !thd->is_current_stmt_binlog_disabled() && > + unlikely(rw_count > 0)) > + { > + /* > + In case an engine is modified but binlog is not part of xa-prepared > + transaction (thd is non-NULL) set out binlogging of an empty xa-prepare. > + */ when can we land here? non-trans (or non-XA) table modified? > + DBUG_ASSERT(thd->transaction->xid_state.get_state_code() == XA_IDLE); > + > + THD_TRANS *ptr_trans= &thd->transaction->all; > + thd->ha_data[binlog_hton->slot].ha_info[1].register_ha(ptr_trans, > + binlog_hton); > + thd->ha_data[binlog_hton->slot].ha_info[1].set_trx_read_write(); > + thd->binlog_setup_trx_data(); > + > + rc= true; > + } > + > + return rc; > +} > > /** > @retval > @@ -1488,9 +1530,16 @@ int ha_prepare(THD *thd) > > if (ha_info) > { > + int err; > + bool has_binlog= has_binlog_hton(ha_info, thd); > + > for (; ha_info; ha_info= ha_info->next()) > { > handlerton *ht= ha_info->ht(); > + > + if (ht == binlog_hton) > + continue; Does binlog need a handlerton anymore? It was an easy way to get it registered for participating in 2pc. But if you always filter it out, then there's no need to add it to ha_info list in the first place, is there? > + > if (ht->prepare) > { > if (unlikely(prepare_or_error(ht, thd, all))) > diff --git a/sql/log.cc b/sql/log.cc > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -1895,6 +1896,11 @@ binlog_rollback_flush_trx_cache(THD *thd, bool all, > if (thd->transaction->xid_state.get_state_code() == XA_PREPARED) > buflen= serialize_with_xid(thd->transaction->xid_state.get_xid(), > buf, query, q_len); > + cache_mngr->using_xa= TRUE; > +#ifndef DBUG_OFF > + if (thd->killed != NOT_KILLED) > + thd->lex->sql_command= SQLCOM_XA_ROLLBACK; // for run_commit_ord asserts 1. don't abbrev, commits are for others not for you 2. what other values can sql_command have here? > +#endif > } > Query_log_event end_evt(thd, buf, buflen, TRUE, TRUE, TRUE, 0); > > @@ -2186,16 +2200,13 @@ int binlog_commit(THD *thd, bool all, bool ro_1pc) > } > > if (cache_mngr->trx_cache.empty() && > - (thd->transaction->xid_state.get_state_code() != XA_PREPARED || > - !(thd->ha_data[binlog_hton->slot].ha_info[1].is_started() && > - thd->ha_data[binlog_hton->slot].ha_info[1].is_trx_read_write()))) > + (!(is_preparing_xa(thd) || is_prepared_xa(thd)) || > + !is_binlog_read_write(thd))) > { > /* > - This is an empty transaction commit (both the regular and xa), > - or such transaction xa-prepare or > - either one's statement having no effect on the transactional cache > - as any prior to it. > - The empty xa-prepare sinks in only when binlog is read-only. > + This is an empty transaction/statement commit (both the regular and xa), > + or the xa transaction is running xa-prepare or xa-commit while > + its binlog branch is not read-write. weird. how and why do you get into binlog_commit on XA PREPARE? > */ > cache_mngr->reset(false, true); > THD_STAGE_INFO(thd, org_stage); > @@ -7752,7 +7763,7 @@ MYSQL_BIN_LOG::write_transaction_to_binlog(THD *thd, > entry.all= all; > entry.using_stmt_cache= using_stmt_cache; > entry.using_trx_cache= using_trx_cache; > - entry.need_unlog= is_preparing_xa(thd); > + entry.need_unlog= 0; is `need_unlog` always 0 now? > ha_info= all ? thd->transaction->all.ha_list : thd->transaction->stmt.ha_list; > entry.ro_1pc= is_ro_1pc; > entry.end_event= end_ev; > @@ -9365,18 +9381,102 @@ TC_LOG::run_commit_ordered(THD *thd, bool all) > all ? thd->transaction->all.ha_list : thd->transaction->stmt.ha_list; > > mysql_mutex_assert_owner(&LOCK_commit_ordered); > + > + if (!ha_info) > + { > + // slave or master's xa-"complete" external what does it mean? > + DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_COMMIT || > + thd->lex->sql_command == SQLCOM_XA_ROLLBACK); > + > + XID *xid= thd->lex->xid; > + bool commit= thd->lex->sql_command == SQLCOM_XA_COMMIT; > + struct xahton_st xaop= { xid, 1 }; > + > + plugin_foreach(NULL, commit ? xacommit_handlerton : xarollback_handlerton, > + MYSQL_STORAGE_ENGINE_PLUGIN, &xaop); is it for commit/rollback by xid? > + xid_cache_delete(thd); > + > + DBUG_ASSERT(!thd->transaction->xid_state.is_explicit_XA()); > + > + return; > + } > + > + // else > + if (is_preparing_xa(thd)) > + { > + for (; ha_info; ha_info= ha_info->next()) > + { > + DBUG_ASSERT(all); > + DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_PREPARE); > + DBUG_ASSERT(thd->transaction->xid_state.is_explicit_XA()); > + > + handlerton *ht= ha_info->ht(); > + if (!ht->prepare) > + { > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_GET_ERRNO, ER_THD(thd, ER_GET_ERRNO), > + HA_ERR_WRONG_COMMAND, > + ha_resolve_storage_engine_name(ht)); > + continue; > + } > + else if (ht != binlog_hton) > + { > + /* leader only */ > + //if (thd->binlog_setup_trx_data()->last_commit_pos_offset == 0) > + // ht->prepare(ht, thd, all); wait, what? why do you need a loop over all engines if you don't do anything there? > + } > + } > + if (thd->rgi_slave && thd->rgi_slave->is_parallel_exec) > + { > + thd->transaction->xid_state.set_state_code(XA_PREPARED); > + thd->transaction->xid_state.set_cache_element_to_recovered(); > + thd->transaction->xid_state.xid_cache_element= 0; when are you here? > + } > + } > + else if (thd->transaction->xid_state.is_explicit_XA()) > + { > + for (; ha_info; ha_info= ha_info->next()) > + { > + DBUG_ASSERT(all); > + DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_COMMIT || > + thd->lex->sql_command == SQLCOM_XA_ROLLBACK); > + DBUG_ASSERT(is_prepared_xa(thd) || > + ((thd->lex->sql_command == SQLCOM_XA_ROLLBACK && > + thd->transaction->xid_state.get_state_code() == XA_IDLE) || > + thd->lex->xa_opt == XA_ONE_PHASE)); > + DBUG_ASSERT(!thd->transaction->xid_state.is_recovered()); > + > + handlerton *ht= ha_info->ht(); > + if (thd->lex->sql_command == SQLCOM_XA_COMMIT) > + { > + if (ht->commit_ordered) > + ht->commit_ordered(ht, thd, all); > + } > + else if (ht != binlog_hton) why not? > + { > + ht->rollback(ht, thd, all); > + } > + } > + xid_cache_delete(thd); // trx is completed now > + > + DBUG_ASSERT(!thd->transaction->xid_state.is_explicit_XA()); > + } > + else > + { when are you here? > for (; ha_info; ha_info= ha_info->next()) > { > handlerton *ht= ha_info->ht(); > if (!ht->commit_ordered) > continue; > ht->commit_ordered(ht, thd, all); > DBUG_EXECUTE_IF("enable_log_write_upto_crash", > { > DBUG_SET_INITIAL("+d,crash_after_log_write_upto"); > sleep(1000); > }); > DEBUG_SYNC(thd, "commit_after_run_commit_ordered"); > } > + } > } > > diff --git a/sql/xa.cc b/sql/xa.cc > --- a/sql/xa.cc > +++ b/sql/xa.cc > @@ -319,9 +342,16 @@ static void xid_cache_delete(THD *thd, XID_cache_element *&element) > > void xid_cache_delete(THD *thd, XID_STATE *xid_state) > { > - DBUG_ASSERT(xid_state->is_explicit_XA()); > - xid_cache_delete(thd, xid_state->xid_cache_element); > - xid_state->xid_cache_element= 0; > + if (xid_state->is_explicit_XA()) why it should be done here and not in the caller? > + { > + xid_cache_delete(thd, xid_state->xid_cache_element); > + xid_state->xid_cache_element= 0; > + } > +} > + > +void xid_cache_delete(THD *thd) > +{ > + xid_cache_delete(thd, &thd->transaction->xid_state); > } > > > @@ -547,7 +577,14 @@ bool trans_xa_prepare(THD *thd) > } > else > { > - thd->transaction->xid_state.xid_cache_element->xa_state= XA_PREPARED; > + if (thd->transaction->xid_state.xid_cache_element) > + { > + thd->transaction->xid_state.xid_cache_element->xa_state= XA_PREPARED; > + } > + else > + { > + DBUG_ASSERT(thd->rgi_slave->is_parallel_exec); what do you mean by that? That parallel slave doesn't add XA transactions to the xid_cache? > + } > MYSQL_SET_TRANSACTION_XA_STATE(thd->m_transaction_psi, XA_PREPARED); > res= thd->variables.pseudo_slave_mode || thd->slave_thread ? > slave_applier_reset_xa_trans(thd) : 0; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: d2de9a89d80: MDEV-31531 Remove my_casedn_str() and my_caseup_str()
by Sergei Golubchik 19 Dec '23

19 Dec '23

Hi, Alexander, Looks good. A couple of minor comments below On Dec 19, Alexander Barkov wrote: > commit d2de9a89d80 > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Fri Jun 23 13:24:02 2023 +0400 > > diff --git a/include/m_ctype.h b/include/m_ctype.h > index 9d13989d1fe..65ccf869992 100644 > --- a/include/m_ctype.h > +++ b/include/m_ctype.h > @@ -1512,6 +1540,14 @@ size_t my_copy_fix_mb(CHARSET_INFO *cs, > /* Functions for 8bit */ > extern size_t my_caseup_str_8bit(CHARSET_INFO *, char *); > extern size_t my_casedn_str_8bit(CHARSET_INFO *, char *); > +static inline size_t my_caseup_str_latin1(char *str) caseup variant doesn't seem to be used anywhere > +{ > + return my_caseup_str_8bit(&my_charset_latin1, str); > +} > +static inline size_t my_casedn_str_latin1(char *str) > +{ > + return my_casedn_str_8bit(&my_charset_latin1, str); > +} > extern size_t my_caseup_8bit(CHARSET_INFO *, > const char *src, size_t srclen, > char *dst, size_t dstlen); > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 09af5523c76..3166c9e56cf 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2543,6 +2541,37 @@ static void push_new_user(const ACL_USER &user) > } > > > +/** > + Make a database name on mem_root from a String, > + apply lower-case conversion if lower_case_table_names says so. > + Perform database name length limit validation. > + > + @param thd - the THD, to get the warning text from > + @param mem_root - allocate the result on this memory root > + @param dbstr - the String, e.g. with Field::val_str() result > + > + @return - {NULL,0} in case of EOM or a bad database name, > + or a good database name otherwise. > +*/ > +static LEX_STRING make_and_check_db_name(MEM_ROOT *mem_root, > + const String &dbstr) > +{ > + LEX_STRING dbls= lower_case_table_names ? > + lex_string_casedn_root(mem_root, files_charset_info, > + dbstr.ptr(), dbstr.length()) : > + lex_string_strmake_root(mem_root, > + dbstr.ptr(), dbstr.length()); > + if (!dbls.str) > + return LEX_STRING{NULL, 0}; // EOM what's the difference between this and return {NULL, 0}; > + if (dbls.length > SAFE_NAME_LEN) > + { > + sql_print_warning(ER_DEFAULT(ER_WRONG_DB_NAME), dbls.str); > + return LEX_STRING{NULL, 0}; // Bad name > + } > + return dbls; // Good name > +} > + > + > /* > Initialize structures responsible for user/db-level privilege checking > and load information about grants from open privilege tables. > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index a84fd29affc..f61e5085e47 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -4186,6 +4187,18 @@ bool LEX::copy_db_to(LEX_CSTRING *to) > return thd->copy_db_to(to); > } > > + > +Lex_ident_db_normalized LEX::copy_db_normalized() > +{ > + if (sphead && sphead->m_name.str) > + { > + DBUG_ASSERT(sphead->m_db.str && sphead->m_db.length); you missed that DBUG_ASSERT(sphead->m_db.str); DBUG_ASSERT(sphead->m_db.length); > + return thd->to_ident_db_normalized_with_error(sphead->m_db); > + } > + return thd->copy_db_normalized(); > +} > + > + > /** > Initialize offset and limit counters. > > diff --git a/storage/perfschema/pfs_program.cc b/storage/perfschema/pfs_program.cc > index de456610519..f3734cb383b 100644 > --- a/storage/perfschema/pfs_program.cc > +++ b/storage/perfschema/pfs_program.cc > @@ -118,31 +118,21 @@ static void set_program_key(PFS_program_key *key, > */ > > char *ptr= &key->m_hash_key[0]; > + const char *end= ptr + sizeof(key->m_hash_key) - 1; > > ptr[0]= object_type; > ptr++; > > if (object_name_length > 0) > - { > - char tmp_object_name[COL_OBJECT_NAME_SIZE + 1]; > - memcpy(tmp_object_name, object_name, object_name_length); > - tmp_object_name[object_name_length]= '\0'; > - my_casedn_str(system_charset_info, tmp_object_name); > - memcpy(ptr, tmp_object_name, object_name_length); > - ptr+= object_name_length; > - } > + ptr+= system_charset_info->casedn(object_name, object_name_length, > + ptr, end - ptr); > ptr[0]= 0; > ptr++; > > if (schema_name_length > 0) > - { > - char tmp_schema_name[COL_OBJECT_SCHEMA_SIZE + 1]; > - memcpy(tmp_schema_name, schema_name, schema_name_length); > - tmp_schema_name[schema_name_length]='\0'; > - my_casedn_str(system_charset_info, tmp_schema_name); > - memcpy(ptr, tmp_schema_name, schema_name_length); > - ptr+= schema_name_length; > - } > + ptr+= system_charset_info->opt_casedn(schema_name, schema_name_length, > + ptr, end - ptr, > + lower_case_table_names); why system_charset_info and not files_charset_info ? > ptr[0]= 0; > ptr++; > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

MTR failures on official Debian buildd after MariaDB 1:10.11.6-1 upload
by Otto Kekäläinen 17 Dec '23

17 Dec '23

Hi! I just wanted to let you know there are a bunch of MTR failures/regressions at https://buildd.debian.org/status/package.php?p=mariadb which I need to fix or add to skip-lists. The previous upload of MariaDB 1:10.11.5-3 had more 'green' than ever before, and now there are regressions on 6 archs. [image: image.png]

3 6

Re: 2821cf816a4: MDEV-31531 Remove my_casedn_str() and my_caseup_str()
by Sergei Golubchik 16 Dec '23

16 Dec '23

Hi, Alexander, > commit 2821cf816a4 > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Fri Jun 23 13:24:02 2023 +0400 > > MDEV-31531 Remove my_casedn_str() and my_caseup_str() please, explain here why > > diff --git a/include/m_ctype.h b/include/m_ctype.h > index 9d13989d1fe..d2629121cfe 100644 > --- a/include/m_ctype.h > +++ b/include/m_ctype.h > @@ -848,6 +844,26 @@ struct charset_info_st > return (cset->casedn)(this, src, srclen, dst, dstlen); > } > > + /* Convert to a lower-cased 0-terminated string */ > + size_t casedn_z(const char *src, size_t srclen, > + char *dst, size_t dstlen) const > + { > + DBUG_ASSERT(dstlen); DBUG_ASSERT(src != dst) ? > + size_t len= casedn(src, srclen, dst, dstlen - 1); > + dst[len]= '\0'; > + return len; > + } > + > + /* Convert to a upper-cased 0-terminated string */ > + size_t caseup_z(const char *src, size_t srclen, > + char *dst, size_t dstlen) const > + { > + DBUG_ASSERT(dstlen); > + size_t len= caseup(src, srclen, dst, dstlen - 1); > + dst[len]= '\0'; > + return len; > + } > + > uint caseup_multiply() const > { > return (cset->caseup_multiply)(this); > @@ -1512,6 +1528,14 @@ size_t my_copy_fix_mb(CHARSET_INFO *cs, > /* Functions for 8bit */ > extern size_t my_caseup_str_8bit(CHARSET_INFO *, char *); > extern size_t my_casedn_str_8bit(CHARSET_INFO *, char *); > +static inline size_t my_caseup_str_latin1(char *str) > +{ > + return my_caseup_str_8bit(&my_charset_latin1, str); > +} > +static inline size_t my_casedn_str_latin1(char *str) > +{ > + return my_casedn_str_8bit(&my_charset_latin1, str); > +} may be my_charset_ascii? with an assert that it's indeed ascii? and a comment that it's used for generated temp table names and such > extern size_t my_caseup_8bit(CHARSET_INFO *, > const char *src, size_t srclen, > char *dst, size_t dstlen); > diff --git a/sql/char_buffer.h b/sql/char_buffer.h > index 7b855f5401c..1091775259d 100644 > --- a/sql/char_buffer.h > +++ b/sql/char_buffer.h > @@ -71,25 +76,92 @@ class CharBuffer > m_buff[m_length]= '\0'; > return *this; > } > + CharBuffer<buff_sz> & copy_caseup(CHARSET_INFO *cs, const LEX_CSTRING &str) > + { > + DBUG_ASSERT(!buffer_overlaps(str)); > + m_length= cs->cset->caseup(cs, str.str, str.length, m_buff, buff_sz); > + DBUG_ASSERT(is_sane()); > + m_buff[m_length]= '\0'; // See comments in copy_casedn() > + return *this; > + } > CharBuffer<buff_sz> & copy_casedn(CHARSET_INFO *cs, const LEX_CSTRING &str, > bool casedn) > { > casedn ? copy_casedn(cs, str) : copy_bin(str); > return *this; > } > + > + // Append one character > + CharBuffer<buff_sz> & append_char(char ch) > + { > + DBUG_ASSERT(is_sane()); > + if (available_size()) > + { > + m_buff[m_length++]= ch; > + m_buff[m_length]= '\0'; > + } > + DBUG_ASSERT(is_sane()); > + return *this; > + } > + > + // Append a string > + CharBuffer<buff_sz> & append_bin(const LEX_CSTRING &str) why _bin ? and why even have a suffix, and not just append() here and above? > + { > + DBUG_ASSERT(is_sane()); > + DBUG_ASSERT(!buffer_overlaps(str)); > + size_t len= MY_MIN(available_size(), str.length); > + memcpy(m_buff + m_length, str.str, len); > + m_length+= len; > + DBUG_ASSERT(is_sane()); > + m_buff[m_length]= '\0'; > + return *this; > + } > + > // Append a string with casedn conversion > CharBuffer<buff_sz> & append_casedn(CHARSET_INFO *cs, const LEX_CSTRING &str) > { > DBUG_ASSERT(is_sane()); > DBUG_ASSERT(!buffer_overlaps(str)); > size_t casedn_length= cs->casedn(str.str, str.length, > - m_buff + m_length, buff_sz - m_length); > + m_buff + m_length, available_size()); > + m_length+= casedn_length; > + DBUG_ASSERT(is_sane()); > + m_buff[m_length]= '\0'; > + return *this; > + } > + > + CharBuffer<buff_sz> & append_opt_casedn(CHARSET_INFO *cs, > + const LEX_CSTRING &str, > + bool casedn) > + { > + return casedn ? append_casedn(cs, str) : append_bin(str); > + } > + > + // Append a string with caseup conversion > + CharBuffer<buff_sz> & append_caseup(CHARSET_INFO *cs, const LEX_CSTRING &str) > + { > + DBUG_ASSERT(is_sane()); > + DBUG_ASSERT(!buffer_overlaps(str)); > + size_t casedn_length= cs->caseup(str.str, str.length, > + m_buff + m_length, available_size()); > m_length+= casedn_length; > DBUG_ASSERT(is_sane()); > m_buff[m_length]= '\0'; > return *this; > } > > + CharBuffer<buff_sz> & truncate(size_t length) > + { > + DBUG_ASSERT(is_sane()); > + if (m_length > length) > + { > + m_length= length; > + m_buff[m_length]= '\0'; > + DBUG_ASSERT(is_sane()); > + } > + return *this; > + } > + > LEX_CSTRING to_lex_cstring() const > { > return LEX_CSTRING{m_buff, m_length}; > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index bbd09a59267..824ed42ec91 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -3769,20 +3769,15 @@ int group_concat_key_cmp_with_order_with_nulls(void *arg, const void *key1_arg, > } > > > -static void report_cut_value_error(THD *thd, uint row_count, const char *fname) > +static void report_cut_value_error(THD *thd, uint row_count, > + const LEX_CSTRING &fname) > { > - size_t fn_len= strlen(fname); > - char *fname_upper= (char *) my_alloca(fn_len + 1); > - if (!fname_upper) > - fname_upper= (char*) fname; // Out of memory > - else > - memcpy(fname_upper, fname, fn_len+1); > - my_caseup_str(&my_charset_latin1, fname_upper); > + CharBuffer<NAME_LEN> fname_upper; > + fname_upper.copy_caseup(&my_charset_latin1, fname); why bother? we don't do it in other error cases > push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > ER_CUT_VALUE_GROUP_CONCAT, > ER_THD(thd, ER_CUT_VALUE_GROUP_CONCAT), > - row_count, fname_upper); > + row_count, fname_upper.ptr()); > - my_afree(fname_upper); > } > > > diff --git a/sql/lex_ident.h b/sql/lex_ident.h > index f39273b7da5..db96e40cab8 100644 > --- a/sql/lex_ident.h > +++ b/sql/lex_ident.h > @@ -45,6 +46,10 @@ class Lex_ident_fs: public LEX_CSTRING > bool is_in_lower_case() const; > bool ok_for_lower_case_names() const; > #endif > + bool check_db_name_quick() const please, use a name that makes it clear what the return value is Not if (str.check_db_name_quick()) but if (str.is_invalid_db_name()) > + { > + return !length || length > NAME_LEN || str[length-1] == ' '; > + } > }; > > > @@ -160,4 +171,95 @@ class DBNameBuffer: public CharBuffer<SAFE_NAME_LEN + MY_CS_MBMAXLEN> > }; > > > +class Identifier_chain2 > +{ > + LEX_CSTRING m_name[2]; > +public: > + Identifier_chain2() > + :m_name{Lex_cstring(), Lex_cstring()} > + { } > + Identifier_chain2(const LEX_CSTRING &a, const LEX_CSTRING &b) > + :m_name{a, b} > + { } > + > + const LEX_CSTRING& operator [] (size_t i) const > + { > + return m_name[i]; > + } > + > + static Identifier_chain2 split(const LEX_CSTRING &txt) > + { > + DBUG_ASSERT(txt.str[txt.length] == '\0'); // Expect 0-terminated input > + const char *dot= strchr(txt.str, '.'); > + if (!dot) > + return Identifier_chain2(Lex_cstring(), txt); > + size_t length0= dot - txt.str; > + Lex_cstring name0(txt.str, length0); > + Lex_cstring name1(txt.str + length0 + 1, txt.length - length0 - 1); > + return Identifier_chain2(name0, name1); > + } > + > + // Export as a qualified name string: 'db.name' > + size_t make_qname(char *dst, size_t dstlen) const > + { > + return my_snprintf(dst, dstlen, "%.*s.%.*s", > + (int) m_name[0].length, m_name[0].str, > + (int) m_name[1].length, m_name[1].str); > + } > + > + // Export as a qualified name string, allocate on mem_root. > + bool make_qname(MEM_ROOT *mem_root, LEX_CSTRING *dst) const > + { > + const uint dot= !!m_name[0].length; > + char *tmp; > + /* format: [pkg + dot] + name + '\0' */ > + dst->length= m_name[0].length + dot + m_name[1].length; > + if (unlikely(!(dst->str= tmp= (char*) alloc_root(mem_root, > + dst->length + 1)))) > + return true; > + snprintf(tmp, dst->length + 1, "%.*s%.*s%.*s", > + (int) m_name[0].length, (m_name[0].length ? m_name[0].str : ""), > + dot, ".", > + (int) m_name[1].length, m_name[1].str); > + return false; > + } > + > + /* > + Build a separated two step name, e.g. "ident1/ident2", 0-terminated. > + with an optional lower-case conversion. may be should be inside InnoDB? > + @param [OUT] dst - the destination > + @param dst_size - number of bytes available in dst > + @param sep - the separator character > + @param casedn - whether to convert components to lower case > + @return - number of bytes written to "dst", not counting > + the trailing '\0' byte. > + */ > + size_t make_sep_name_opt_casedn(char *dst, size_t dst_size, > + int sep, bool casedn) const > + { > + /* > + Minimal possible buffer is 4 bytes: 'd/t\0' > + where 'd' is the database name, 't' is the table name. > + Callers should never pass a smaller buffer. > + */ > + DBUG_ASSERT(dst_size > 4); > + DBUG_ASSERT(m_name[0].length + m_name[1].length + 2 < FN_REFLEN - 1); > + if (casedn) > + { > + CHARSET_INFO *cs= &my_charset_utf8mb3_general_ci; > + char *ptr= dst, *end= dst + dst_size; > + ptr+= cs->casedn(m_name[0].str, m_name[0].length, ptr, end - ptr - 2); > + *ptr++= (char) sep; > + ptr+= cs->casedn_z(m_name[1].str, m_name[1].length, ptr, end - ptr); > + return ptr - dst; > + } > + memcpy(dst, m_name[0].str, m_name[0].length); > + dst[m_name[0].length] = (char) sep; > + /* Copy the name and null-byte. */ > + memcpy(dst + m_name[0].length + 1, m_name[1].str, m_name[1].length + 1); > + return m_name[0].length + 1/*slash*/ + m_name[1].length; s/slash/sep/ > + } > +}; > + > + > #endif // LEX_IDENT_INCLUDED > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 6030202adc7..0fc1d4ce962 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -5777,19 +5778,29 @@ int Table_map_log_event::do_apply_event(rpl_group_info *rgi) > /* Step the query id to mark what columns that are actually used. */ > thd->set_query_id(next_query_id()); > > + db_mem_alloced= tname_mem_alloced= NAME_LEN * > + (lower_case_table_names ? 1 : > + files_charset_info->casedn_multiply()) > + + 1 /* for '\0' */; 1. may be db_mem_length/tname_mem_length instead of NAME_LEN? 2. is the ?: condition right? Looks like it's inverted > if (!(memory= my_multi_malloc(PSI_INSTRUMENT_ME, MYF(MY_WME), > &table_list, (uint) sizeof(RPL_TABLE_LIST), > - &db_mem, (uint) NAME_LEN + 1, > - &tname_mem, (uint) NAME_LEN + 1, > + &db_mem, (uint) db_mem_alloced, > + &tname_mem, (uint) tname_mem_alloced, > NullS))) > DBUG_RETURN(HA_ERR_OUT_OF_MEM); > > - db_mem_length= strmov(db_mem, m_dbnam) - db_mem; > - tname_mem_length= strmov(tname_mem, m_tblnam) - tname_mem; > if (lower_case_table_names) > { > - my_casedn_str(files_charset_info, (char*)tname_mem); > - my_casedn_str(files_charset_info, (char*)db_mem); > + db_mem_length= files_charset_info->casedn_z(m_dbnam, m_dblen, > + db_mem, db_mem_alloced); > + tname_mem_length= files_charset_info->casedn_z(m_tblnam, m_tbllen, > + tname_mem, > + tname_mem_alloced); > + } > + else > + { > + db_mem_length= strmov(db_mem, m_dbnam) - db_mem; > + tname_mem_length= strmov(tname_mem, m_tblnam) - tname_mem; > } > > /* call from mysql_client_binlog_statement() will not set rli->mi */ > diff --git a/sql/sp.cc b/sql/sp.cc > index 0a8aa24789b..537d29fb67e 100644 > --- a/sql/sp.cc > +++ b/sql/sp.cc > @@ -2281,12 +2281,20 @@ Sp_handler::sp_exist_routines(THD *thd, TABLE_LIST *routines) const > for (routine= routines; routine; routine= routine->next_global) > { > sp_name *name; > - LEX_CSTRING lex_db; > + LEX_CSTRING lex_db= thd->make_ident_opt_casedn(routine->db, > + lower_case_table_names); > + if (!lex_db.str) > + DBUG_RETURN(TRUE); // EOM // EOM, error was already sent right? (also, in other places) > LEX_CSTRING lex_name; > - thd->make_lex_string(&lex_db, routine->db.str, routine->db.length); > thd->make_lex_string(&lex_name, routine->table_name.str, > routine->table_name.length); Rather inconsistent :( May be, makes sense to create LEX_CSTRING make_lex_string(const char* str, size_t length) const ? > + /* > + routine->db was earlier tested with check_db_name(). > + Now it's lower-cased according to lower_case_table_names. > + It's safe to make a Lex_ident_db_normalized. > + */ > - name= new sp_name(&lex_db, &lex_name, true); > + name= new (thd->mem_root) sp_name(Lex_ident_db_normalized(lex_db), > + lex_name, true); > sp_object_found= sp_find_routine(thd, name, false) != NULL; > thd->get_stmt_da()->clear_warning_info(thd->query_id); > if (! sp_object_found) > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 09af5523c76..70dde7dc0ed 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2543,6 +2543,37 @@ static void push_new_user(const ACL_USER &user) > } > > > +/** > + Make a database name on mem_root from a String, > + apply lower-case conversion if lower_case_table_names says so. > + Perform database name length limit validation. > + > + @param thd - the THD, to get the warning text from > + @param mem_root - allocate the result on this memory root > + @param dbstr - the String, e.g. with Field::val_str() result > + > + @return - {NULL,0} in case of EOM or a bad database name, > + or a good database name otherwise. > +*/ > +static LEX_STRING make_and_check_db_name(THD *thd, MEM_ROOT *mem_root, > + const String &dbstr) > +{ > + LEX_STRING dbls= lower_case_table_names ? > + lex_string_casedn_root(mem_root, files_charset_info, > + dbstr.ptr(), dbstr.length()) : > + lex_string_strmake_root(mem_root, > + dbstr.ptr(), dbstr.length()); > + if (!dbls.str) > + return LEX_STRING{NULL, 0}; // EOM > + if (dbls.length > SAFE_NAME_LEN) > + { > + sql_print_warning(ER_THD(thd, ER_WRONG_DB_NAME), dbls.str); It's an error log, should every message be localized according to connection-specific locale? > + return LEX_STRING{NULL, 0}; // Bad name > + } > + return dbls; // Good name > +} > + > + > /* > Initialize structures responsible for user/db-level privilege checking > and load information about grants from open privilege tables. > @@ -3680,26 +3702,27 @@ privilege_t acl_get(const char *host, const char *ip, > { > privilege_t host_access(ALL_KNOWN_ACL), db_access(NO_ACL); > uint i; > - size_t key_length; > - char key[ACL_KEY_LENGTH],*tmp_db,*end; > + const char *tmp_db; > acl_entry *entry; > DBUG_ENTER("acl_get"); > > - tmp_db= strmov(strmov(key, safe_str(ip)) + 1, user) + 1; > - end= strnmov(tmp_db, db, key + sizeof(key) - tmp_db); > - > - if (end >= key + sizeof(key)) // db name was truncated > + constexpr size_t key_data_size= ACL_KEY_LENGTH + 2; > + CharBuffer<key_data_size + MY_CS_MBMAXLEN> key; why ACL_KEY_LENGTH + 2 + MY_CS_MBMAXLEN ? > + key.append_bin(Lex_cstring_strlen(safe_str(ip))); > + key.append_char('\0'); > + key.append_bin(Lex_cstring_strlen(user)); > + key.append_char('\0'); there was strmov(strmov(...)) pattern here before. as far as I can see, you can chain too: key.append_bin().append_char() .append_bin().append_char(); let's use that > + tmp_db= key.end(); > + key.append_opt_casedn(files_charset_info, Lex_cstring_strlen(db), > + lower_case_table_names); > + db= tmp_db; > + > + if (key.length() >= key_data_size) // db name was truncated > DBUG_RETURN(NO_ACL); // no privileges for an invalid db name > > - if (lower_case_table_names) > - { > - my_casedn_str(files_charset_info, tmp_db); > - db=tmp_db; > - } > - key_length= (size_t) (end-key); > - > mysql_mutex_lock(&acl_cache->lock); > - if (!db_is_pattern && (entry=acl_cache->search((uchar*) key, key_length))) > + if (!db_is_pattern && > + (entry= acl_cache->search((uchar*) key.ptr(), key.length()))) > { > db_access=entry->access; > mysql_mutex_unlock(&acl_cache->lock); > @@ -5630,28 +5662,26 @@ static GRANT_NAME *name_hash_search(HASH *name_hash, > const char *user, const char *tname, > bool exact, bool name_tolower) > { > - char helping[SAFE_NAME_LEN*2+USERNAME_LENGTH+3]; > - char *hend = helping + sizeof(helping); > - uint len; > + constexpr size_t key_data_size= SAFE_NAME_LEN * 2 + USERNAME_LENGTH + 3; > + CharBuffer<key_data_size + MY_CS_MBMAXLEN> key; > GRANT_NAME *grant_name,*found=0; > HASH_SEARCH_STATE state; > > - char *db_ptr= strmov(helping, user) + 1; > - char *tname_ptr= strnmov(db_ptr, db, hend - db_ptr) + 1; > - if (tname_ptr > hend) > - return 0; // invalid name = not found > - char *end= strnmov(tname_ptr, tname, hend - tname_ptr) + 1; > - if (end > hend) > + key.append_bin(Lex_cstring_strlen(user)); > + key.append_char('\0'); > + key.append_bin(Lex_cstring_strlen(db)); is db already normalized? If yes, then, please, document it with DBUG_ASSERT(ok_for_lower_case_names()); > + key.append_char('\0'); > + key.append_opt_casedn(files_charset_info, Lex_cstring_strlen(tname), > + name_tolower); > + key.append_char('\0'); > + if (key.length() >= key_data_size) > return 0; // invalid name = not found > > - len = (uint) (end - helping); > - if (name_tolower) > - my_casedn_str(files_charset_info, tname_ptr); > - for (grant_name= (GRANT_NAME*) my_hash_first(name_hash, (uchar*) helping, > - len, &state); > + for (grant_name= (GRANT_NAME*) my_hash_first(name_hash, (uchar*) key.ptr(), > + key.length(), &state); > grant_name ; > - grant_name= (GRANT_NAME*) my_hash_next(name_hash,(uchar*) helping, > - len, &state)) > + grant_name= (GRANT_NAME*) my_hash_next(name_hash, (uchar*) key.ptr(), > + key.length(), &state)) > { > if (exact) > { > @@ -8826,33 +8856,28 @@ static bool check_grant_db_routine(THD *thd, const char *db, HASH *hash) > bool check_grant_db(THD *thd, const char *db) > { > Security_context *sctx= thd->security_ctx; > - char helping [SAFE_NAME_LEN + USERNAME_LENGTH+2], *end; > - char helping2 [SAFE_NAME_LEN + USERNAME_LENGTH+2], *tmp_db; > - uint len, UNINIT_VAR(len2); > + constexpr size_t key_data_size= SAFE_NAME_LEN + USERNAME_LENGTH + 2; > + CharBuffer<key_data_size + MY_CS_MBMAXLEN> key, key2; > bool error= TRUE; > > - tmp_db= strmov(helping, sctx->priv_user) + 1; > - end= strnmov(tmp_db, db, helping + sizeof(helping) - tmp_db); > - > - if (end >= helping + sizeof(helping)) // db name was truncated > - return 1; // no privileges for an invalid db name > - > - if (lower_case_table_names) > - { > - end = tmp_db + my_casedn_str(files_charset_info, tmp_db); > - db=tmp_db; > - } > + key.append_bin(Lex_cstring_strlen(sctx->priv_user)); > + key.append_char('\0'); > + key.append_opt_casedn(files_charset_info, Lex_cstring_strlen(db), > + lower_case_table_names); > + key.append_char('\0'); > > - len= (uint) (end - helping) + 1; > + if (key.length() >= key_data_size) // db name was truncated > + return 1; // no privileges for an invalid db name > > /* > If a role is set, we need to check for privileges here as well. > */ > if (sctx->priv_role[0]) > { > - end= strmov(helping2, sctx->priv_role) + 1; > - end= strnmov(end, db, helping2 + sizeof(helping2) - end); > - len2= (uint) (end - helping2) + 1; > + key2.append_bin(Lex_cstring_strlen(sctx->priv_role)); > + key2.append_char('\0'); > + key2.append_bin(Lex_cstring_strlen(db)); No append_opt_casedn() here? Looks wrong. > + key2.append_char('\0'); > } > > > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 214118d8431..decb9a9064e 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -4918,11 +4948,46 @@ class THD: public THD_count, /* this must be first */ > my_message(ER_NO_DB_ERROR, ER(ER_NO_DB_ERROR), MYF(0)); > return TRUE; > } > + return false; > + } > + > + /* > + Copy the current database to the argument. Use the current arena to > + allocate memory for a deep copy: current database may be freed after > + a statement is parsed but before it's executed. > + > + Can only be called by owner of thd (no mutex protection) > + */ > + bool copy_db_to(LEX_CSTRING *to) > + { > + if (check_if_current_db_is_set_with_error()) > + return true; > > to->str= strmake(db.str, db.length); > to->length= db.length; > return to->str == NULL; /* True on error */ > } > + > + /* > + Make a normalized copy of the current database. > + Raise an error if no current database is set. > + Note, in case of lower_case_table_names==2, thd->db can contain the > + name in arbitrary case typed by the user, so it must be lower-cased. > + For other lower_case_table_names values the name is already in > + its normalized case, so it's copied as is. > + */ > + Lex_ident_db_normalized copy_db_normalized() > + { > + if (check_if_current_db_is_set_with_error()) > + return Lex_ident_db_normalized(); > + LEX_STRING ident= make_ident_opt_casedn(db, lower_case_table_names == 2); LEX_STRING or LEX_CSTRING ? > + /* > + A non-empty thd->db is always known to satisfy check_db_name(). > + So after optional lower-casing above it's safe to > + make Lex_ident_db_normalized. > + */ > + return Lex_ident_db_normalized(ident); > + } > /* Get db name or "". Use for printing current db */ > const char *get_db() > { return safe_str(db.str); } > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index a84fd29affc..8cafad65538 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -4186,6 +4186,18 @@ bool LEX::copy_db_to(LEX_CSTRING *to) > return thd->copy_db_to(to); > } > > + > +Lex_ident_db_normalized LEX::copy_db_normalized() > +{ > + if (sphead && sphead->m_name.str) > + { > + DBUG_ASSERT(sphead->m_db.str && sphead->m_db.length); Please, DBUG_ASSERT(sphead->m_db.str); DBUG_ASSERT(sphead->m_db.length); > + return thd->to_ident_db_normalized_with_error(sphead->m_db); > + } > + return thd->copy_db_normalized(); > +} > + > + > /** > Initialize offset and limit counters. > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 1fcafd6ec08..0f39a13632a 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -3856,30 +3859,31 @@ static bool show_status_array(THD *thd, const char *wild, > names until lp:1306875 has been fixed. > TODO: remove once lp:1306875 has been addressed. > */ > - if (!(*prefix) && !strncasecmp(name_buffer, "wsrep", strlen("wsrep"))) > + if (!prefix.length && > + !strncasecmp(name_buffer.ptr(), "wsrep", strlen("wsrep"))) > { > is_wsrep_var= TRUE; > } > #endif /* WITH_WSREP */ > > if (ucase_names) > - my_caseup_str(system_charset_info, name_buffer); > + name_buffer.append_caseup(system_charset_info, var_name); > else > { > - my_casedn_str(system_charset_info, name_buffer); > - DBUG_ASSERT(name_buffer[0] >= 'a'); > - DBUG_ASSERT(name_buffer[0] <= 'z'); > - > + name_buffer.append_casedn(system_charset_info, var_name); > // WSREP_TODO: remove once lp:1306875 has been addressed. > if (IF_WSREP(is_wsrep_var == FALSE, 1) && > status_var) > - name_buffer[0]-= 'a' - 'A'; > + { > + char *ptr= (char*) name_buffer.ptr(); > + if (ptr[0] >= 'a' && ptr[0] <= 'z') > + ptr[0]-= 'a' - 'A'; > + } is it still needed? > } > > > restore_record(table, s->default_values); > - table->field[0]->store(name_buffer, strlen(name_buffer), > - system_charset_info); > + table->field[0]->store(name_buffer.to_lex_cstring(), system_charset_info); > > /* > Compare name for types that can't return arrays. We do this to not > @@ -4355,18 +4361,15 @@ bool get_lookup_field_values(THD *thd, COND *cond, bool fix_table_name_case, > > if (lower_case_table_names && !rc) > { > - /* > - We can safely do in-place upgrades here since all of the above cases > - are allocating a new memory buffer for these strings. > - */ > if (lookup_field_values->db_value.str && lookup_field_values->db_value.str[0]) > - my_casedn_str(system_charset_info, > - (char*) lookup_field_values->db_value.str); > + lookup_field_values->db_value= thd->make_ident_casedn( > + lookup_field_values->db_value); > + > if (fix_table_name_case && > lookup_field_values->table_value.str && > lookup_field_values->table_value.str[0]) > - my_casedn_str(system_charset_info, > - (char*) lookup_field_values->table_value.str); > + lookup_field_values->table_value= thd->make_ident_casedn( > + lookup_field_values->table_value); Why is it even needed? If db/table names are compared case insensitively, it shouldn't matter if they're lowercased or not, should it? > } > > return rc; > diff --git a/sql/sql_type_fixedbin.h b/sql/sql_type_fixedbin.h > index 8aaa7582cd5..e9728b78a3d 100644 > --- a/sql/sql_type_fixedbin.h > +++ b/sql/sql_type_fixedbin.h > @@ -267,8 +267,7 @@ class Type_handler_fbt: public Type_handler > void print(String *str, enum_query_type query_type) override > { > StringBuffer<FbtImpl::max_char_length()+64> tmp; > - tmp.append(singleton()->name().lex_cstring()); > - my_caseup_str(&my_charset_latin1, tmp.c_ptr()); > + tmp.copy_caseup(&my_charset_latin1, singleton()->name().lex_cstring()); why bother? > str->append(tmp); > str->append('\''); > m_value.to_string(&tmp); > diff --git a/storage/innobase/dict/dict0load.cc b/storage/innobase/dict/dict0load.cc > index bf59fed9ee4..104a8dc8780 100644 > --- a/storage/innobase/dict/dict0load.cc > +++ b/storage/innobase/dict/dict0load.cc > @@ -2955,7 +2955,7 @@ dict_load_foreign( > > foreign->foreign_table_name = mem_heap_strdupl( > foreign->heap, (char*) field, len); > - dict_mem_foreign_table_name_lookup_set(foreign, TRUE); > + foreign->foreign_table_name_lookup_set(); please, ask Marko to review InnoDB changes > > const size_t foreign_table_name_len = len; > const size_t table_name_len = strlen(table_name); > diff --git a/storage/perfschema/pfs_instr_class.h b/storage/perfschema/pfs_instr_class.h > index f353c410d4c..d083db63968 100644 > --- a/storage/perfschema/pfs_instr_class.h > +++ b/storage/perfschema/pfs_instr_class.h > @@ -265,6 +265,40 @@ struct PFS_table_share_key > char m_hash_key[PFS_TABLESHARE_HASHKEY_SIZE]; > /** Length in bytes of @c m_hash_key. */ > uint m_key_length; > + > + size_t available_length() const > + { > + return sizeof(m_hash_key) - m_key_length; > + } > + char *end() > + { > + return m_hash_key + m_key_length; > + } > + // Append and 0-terminate a string with an optional lower-case conversion > + void append_opt_casedn_z(CHARSET_INFO *cs, > + const char *str, size_t length, > + bool casedn) > + { > + DBUG_ASSERT(length <= sizeof(m_hash_key)); // Expect valid db/tbl names > + MY_STRCOPY_STATUS st; > + size_t dst_length= available_length(); > + if (dst_length > 0) > + { > + dst_length--; > + /* > + The code below uses copy_fix() instead of memcpy() to make > + sure we don't break a multi-byte character in the middle. > + */ > + m_key_length+= (uint) (casedn ? > + cs->casedn(str, length, end(), dst_length) : > + cs->copy_fix(end(), dst_length, > + str, length, length, &st)); may be let's not do copy_fix here? there are table names and in a perfschema, for PFS_table_share_key. memcpy would be enough. > + m_hash_key[m_key_length++]= '\0'; > + } > + } > + void set(bool temporary, > + const char *schema_name, size_t schema_name_length, > + const char *table_name, size_t table_name_length); > }; > > /** Table index or 'key' */ > diff --git a/storage/perfschema/pfs_program.cc b/storage/perfschema/pfs_program.cc > index de456610519..6c35a745d30 100644 > --- a/storage/perfschema/pfs_program.cc > +++ b/storage/perfschema/pfs_program.cc > @@ -118,31 +118,20 @@ static void set_program_key(PFS_program_key *key, > */ > > char *ptr= &key->m_hash_key[0]; > + const char *end= ptr + sizeof(key->m_hash_key) - 1; > > ptr[0]= object_type; > ptr++; > > if (object_name_length > 0) > - { > - char tmp_object_name[COL_OBJECT_NAME_SIZE + 1]; > - memcpy(tmp_object_name, object_name, object_name_length); > - tmp_object_name[object_name_length]= '\0'; > - my_casedn_str(system_charset_info, tmp_object_name); > - memcpy(ptr, tmp_object_name, object_name_length); > - ptr+= object_name_length; > - } > + ptr+= system_charset_info->casedn(object_name, object_name_length, > + ptr, end - ptr); > ptr[0]= 0; > ptr++; > > if (schema_name_length > 0) > - { > - char tmp_schema_name[COL_OBJECT_SCHEMA_SIZE + 1]; > - memcpy(tmp_schema_name, schema_name, schema_name_length); > - tmp_schema_name[schema_name_length]='\0'; > - my_casedn_str(system_charset_info, tmp_schema_name); > - memcpy(ptr, tmp_schema_name, schema_name_length); > - ptr+= schema_name_length; > - } > + ptr+= system_charset_info->casedn(schema_name, schema_name_length, > + ptr, end - ptr); That's strange. casedn of schema when lower_case_table_names==0 ? > ptr[0]= 0; > ptr++; > Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 4

review for MDEV-32839 LONG UNIQUE gives error when used with REPLACE
by Alexander Barkov 12 Dec '23

12 Dec '23

Hello Sergei, - This patch: https://github.com/MariaDB/server/commit/4f2d0541b501a7a27bf44aa16821d09395… cleanup: remove innodb-specific code around update_auto_increment() is OK to push. Please write in the commit comments that it was a dead code and that InnoDB team does know why this code was there. - This patch https://github.com/MariaDB/server/commit/c8d0f3370b7d77ecb07f8f1357da6bb1f0… MDEV-32839 LONG UNIQUE gives error when used with REPLACE is OK. I have only a proposal about tests. You added tests above the "# End of 10.5 tests" line, but the patch is for 10.6. Also please add tests for at lease these engines: - MyISAM - InnoDB - Spider as well as partitioned tables for the same engines, because your patch for ha_partition.cc changes the code related to need_info_for_auto_inc(), which has a special implementation in ha_partition and ha_spider. - This patch: https://github.com/MariaDB/server/commit/94f26086663b7c8dbcd6b26930b64de36a… cleanup: remove partition-specific code around update_auto_increment() > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 25bbcfdd75ce0..e174ef8d21940 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -4308,6 +4308,7 @@ int ha_partition::external_lock(THD *thd, int lock_type) > m_part_info->part_expr->walk(&Item::register_field_in_read_map, 1, 0); > if ((error= m_part_info->vers_set_hist_part(thd))) > goto err_handler; > + need_info_for_auto_inc(); auto-increment code is often done under a condition like this: if (table->next_number_field && buf == table->record[0]) { some_auto_increment code; } This includes the code that you are removing from ha_partition::write_row() Is it ok to call need_info_for_auto_inc() unconditionally here? > } > DBUG_RETURN(0); > > @@ -4623,33 +4624,8 @@ int ha_partition::write_row(const uchar * buf) > */ > if (have_auto_increment) > { > - if (!table_share->next_number_keypart) > - if (unlikely(error= update_next_auto_inc_val())) > - goto exit; Here there was an extra condition !table_share->next_number_keypart before calling update_next_auto_inc_val(). Why was it needed? Why it's not needed in the new version of external_lock() before the call for need_info_for_auto_inc()? Also, the old code checked the return value. The new code int external_code() does not check the result of need_info_for_auto_inc(). Can't the behavior change because of this change? > - > - /* > - If we have failed to set the auto-increment value for this row, > - it is highly likely that we will not be able to insert it into > - the correct partition. We must check and fail if necessary. > - */ > if (unlikely(error= update_auto_increment())) > goto exit; > - > - /* > - Don't allow generation of auto_increment value the partitions handler. > - If a partitions handler would change the value, then it might not > - match the partition any longer. > - This can occur if 'SET INSERT_ID = 0; INSERT (NULL)', > - So allow this by adding 'MODE_NO_AUTO_VALUE_ON_ZERO' to sql_mode. > - The partitions handler::next_insert_id must always be 0. Otherwise > - we need to forward release_auto_increment, or reset it for all > - partitions. > - */ > - if (table->next_number_field->val_int() == 0) > - { > - table->auto_increment_field_not_null= TRUE; > - thd->variables.sql_mode|= MODE_NO_AUTO_VALUE_ON_ZERO; > - } Why is it safe to remove this code? > } > old_map= dbug_tmp_use_all_columns(table, &table->read_set); > error= m_part_info->get_partition_id(m_part_info, &part_id, &func_value); > @@ -11012,10 +10988,7 @@ void ha_partition::get_auto_increment(ulonglong offset, ulonglong increment, > else > { > THD *thd= ha_thd(); > - /* > - This is initialized in the beginning of the first write_row call. > - */ > - DBUG_ASSERT(part_share->auto_inc_initialized); > + update_next_auto_inc_val(); > /* > Get a lock for handling the auto_increment in part_share > for avoiding two concurrent statements getting the same number. > diff --git a/sql/ha_partition.h b/sql/ha_partition.h > index d75e38ec272a5..0ac8f696fcf55 100644 > --- a/sql/ha_partition.h > +++ b/sql/ha_partition.h > @@ -1408,9 +1408,8 @@ class ha_partition final :public handler > { > ulonglong nr= (((Field_num*) field)->unsigned_flag || > field->val_int() > 0) ? field->val_int() : 0; > + update_next_auto_inc_val(); > lock_auto_increment(); > - DBUG_ASSERT(part_share->auto_inc_initialized || > - !can_use_for_auto_inc_init()); > /* must check when the mutex is taken */ > if (nr >= part_share->next_auto_inc_val) > part_share->next_auto_inc_val= nr + 1;

2 2

MDEV-31404 Review
by Kristian Nielsen 11 Dec '23

11 Dec '23

Hi Monty, here is my review of the patch for MDEV-31404: > commit d51086a9f1b1b6a66c7ba8eb6a6876b433fa118f (origin/bb-11.4-MDEV-31404) > Author: Monty <monty(a)mariadb.org> > Date: Sun Dec 3 21:42:44 2023 +0200 > > MDEV-31404 Implement binlog_space_limit > > binlog_space_limit is a variable in Percona server used to limit the total > size of all binary logs. > > This implementation is based on code from Percona server 5.7. General comment: The --slave-connections-needed-for-purge defaults to 1. I think this means that --log-expire-days will have no effect on systems that have binlog enabled but no slaves configured. This is something to be aware of for backwards compatibility. What I do not like about this is the consequence it will have for all systems that have binlog enabled with some --expire-log-days set but no slaves configured. After upgrade, these systems will silently start accumulating binlogs without limit, possibly filling up disk eventually. On the other hand, it seems good to have --slave-connections-needed-for-purge enabled for the new --max-binlog-total-size option; as you pointed out, otherwise a single large transaction could immediately cause deletion of all binlogs when no slaves are connected. And having --slave-connections-needed-for-purge affect purge from --max-binlog-total-size but not for --expire-log-days seems inconsistent. In the end, maybe it isn't too serious. Many small and default-configured systems will probably be able to run for ages without binlogs getting excessive due to low traffic; and it will be easy for DBAs to just delete the binlogs if needed, or google the --slave-connections-needed-for-purge if they care enough. Two suggestions to make it easier for the DBAs to know how to handle this: 1. How about emitting a note or warning in the error log, if purge from --log-expire-days=N gets delayed for more than 2*N because no slaves are connected and --slave-connections-needed-for-purge=1 ? Note: Purge of binlogs prevented for <seconds> because no slaves are connected. To allow purge of binlogs with no slaves connected, set --slave-connections-needed-for-purge=0 2. Mention in the changelogs for 11.4 that binlogs will by default no longer be purged when no slaves are configured, and point to some documentation about how to handle this for the DBA. I think with those two suggestions it should be fine. Detailed comments on the patch below. - Kristian. > diff --git a/mysql-test/suite/binlog_encryption/binlog_index-master.opt b/mysql-test/suite/binlog_encryption/binlog_index-master.opt > new file mode 100644 > index 00000000000..f2673d08020 > --- /dev/null > +++ b/mysql-test/suite/binlog_encryption/binlog_index-master.opt > @@ -0,0 +1 @@ > +--slave_connections_needed_for_purge=0 Is this needed? If so, would it be better to put this in a general my.cnf for the entire binlog_encryption suite, like you did for mysql-test/suite/binlog/my.cnf ? So that a similar .opt will not need to be added manually for each new binlog_encryption test that will need it. > --- a/mysql-test/suite/perfschema/t/show_sanity.test > +++ b/mysql-test/suite/perfschema/t/show_sanity.test > @@ -533,6 +533,7 @@ insert into test.sanity values > ("JUNK: GLOBAL-ONLY", "I_S.SESSION_VARIABLES", "MAX_BINLOG_CACHE_SIZE"), > ("JUNK: GLOBAL-ONLY", "I_S.SESSION_VARIABLES", "MAX_BINLOG_SIZE"), > ("JUNK: GLOBAL-ONLY", "I_S.SESSION_VARIABLES", "MAX_BINLOG_STMT_CACHE_SIZE"), > + ("JUNK: GLOBAL-ONLY", "I_S.SESSION_VARIABLES", "MAX_BINLOGS_SIZE"), I think this is a left-over from previous naming, should probably be "MAX_BINLOG_TOTAL_SIZE ? > +FLUSH LOGS; > +FLUSH LOGS; > +FLUSH LOGS; > +source include/show_binary_logs.inc; > +show status like "binlog_disk_use"; I think you will need --source include/wait_for_binlog_checkpoint.inc after each of these FLUSH LOGS. Otherwise random delay of binlog checkpoint could cause the binlog_disk_use value to fluctuate and get rare random test failures. (Or alternatively consider if you could omit the select of binlog_disk_use completely, since it will probably differ between versions as event formats change.) Adding a small sleep in binlog_background_thread() just before calling mysql_bin_log.mark_xid_done() can usually reproduce the random failures from delayed binlog checkpoint. > +INSERT INTO t1 VALUES (0,repeat("a",3000)); > +show status like "binlog_disk_use"; > +Variable_name Value > +Binlog_disk_use 3848 > +# First binary should be binary.000004 > +show binary logs; > +Log_name File_size > +binary.000004 # > +INSERT INTO t1 VALUES (2,repeat("b",10)); > +# First binary should be binary.000004 > +show binary logs; > +Log_name File_size > +binary.000004 # > +binary.000005 # Here, I don't understand why binary.000004 is not purged. The binary.000004 contains the large transaction 'repeat("a", 3000)', so the binlog total size is too large. And the transaction 'repeat("n",10)' caused a binlog rotation. So why does that not cause binary.000004 to be purged due to --max-binlog-total-size=1500 ? > +show status like "binlog_disk_use"; > +Variable_name Value > +Binlog_disk_use 3647 > +INSERT INTO t1 VALUES (7,repeat("g",3000)); > +# binary.000001 should be deleted now > +show binary logs; > +Log_name File_size > +binary.000002 # > +binary.000003 # Similarly here, why binary.000002 is not purged? > +INSERT INTO t1 VALUES (4,repeat("d",3000)); > +--echo # First binary should be binary.000011 > +source include/show_binary_logs.inc; > + > +RESET MASTER; Idea: Here you could (or ask one of the other devs) add a test that the SHOW STATUS LIKE "binlog_disk_use" is equal to the size of the files on disk, to check that the running calculation of disk use doesn't miss the odd byte or two. > + if ((longlong) (binlog_space_total + binlog_pos - found_space <= > + binlog_space_limit)) > + break; // Found enough The (longlong) cast here looks misplaced. It's casting the result of the (X <= Y) comparison. I think the cast can just be removed, all the variables seem to be ulonglong type. > +static bool waiting_for_slave_to_change_binlog= 0; > +static ulonglong purge_sending_new_binlog_file= 0; I think a short comment here on these variables would be good, explaining their purpose. And same on the declaration of Atomic_counter<ulonglong> sending_new_binlog_file. > + DBUG_ASSERT(!is_relay_log || binlog_xid_count_list.is_empty()); > + if (!is_relay_log) > { > - I_List_iterator<xid_count_per_binlog> it(binlog_xid_count_list); > - while ((b= it++) && > - 0 != strncmp(log_file_name_arg+dirname_length(log_file_name_arg), > - b->binlog_name, b->binlog_name_len)) > - ; > + xid_count_per_binlog *b; > + mysql_mutex_lock(&LOCK_xid_list); Good catch that we were uselessly doing xid_count operations on the relay log, thanks! > +int MYSQL_BIN_LOG::count_binlog_space() > +{ > + int error; > + LOG_INFO log_info; > + DBUG_ENTER("count_binlog_space"); > + if (is_relay_log || !binlog_space_limit) > + DBUG_RETURN(0); Maybe this if (is_relay_log || !binlog_space_limit) could be an assert? We probably should not be calling this function at all in that case. It looks like that binlog_space_limit is already checked in all callers, not sure about relay log. (A small point, just something to consider as you like). > @@ -7608,7 +7807,8 @@ int MYSQL_BIN_LOG::rotate(bool force_rotate, bool* check_purge) > @retval > nonzero - error in rotating routine. > */ > -void MYSQL_BIN_LOG::purge() > + > +void MYSQL_BIN_LOG::purge(bool all) Would be good with a small comment here explaining the meaning of the new parameter `all`. Minor stuff: Commit message: > Binlog file file sizes are checked on startup Typo (s/file file/file/). > when updating the binary log on on binary log rotation Typo (s/on on/and on/ ?) > Without this option max-binlog-total-size would potentially deleted > binlogs needed by slaves on server startup or when a slave disconnects Typo s/deleted/delete/. > +static Sys_var_ulonglong Sys_max_binlog_total_size( > + "max_binlog_total_size", > + "Maximum space to use for all binary logs. Extra logs are deleted on " > + "server start, log rotation, FLUSH LOGGS or when writing to binlog. " > + "Default is 0, which means no size restrictions." > + "See also slave_connections_needed_for_purge", Missing space at end of line (s/restrictions."/restrictions. "/). > +static Sys_var_ulong Sys_slave_connections_needed_for_purge( > + "slave_connections_needed_for_purge", > + "Number of minimum connected slaves needed for automatic binary " > + "log purge with max_binlog_total_size, binlog_expire_logs_seconds " > + "or binlog_expire_logs_days.", Suggestion for better wording: "Minimum number of connected slaves required for automatic binary " "log purge with max_binlog_total_size, binlog_expire_logs_seconds " "or binlog_expire_logs_days."

2 2

Re: d25ab9fdfa1: MDEV-31608 - Connector/NET fails to connect since 10.10
by Sergei Golubchik 30 Nov '23

30 Nov '23

Hi, Vladislav, On Nov 29, Vladislav Vaintroub wrote: > revision-id: d25ab9fdfa1 (mariadb-10.11.2-62-gd25ab9fdfa1) > parent(s): 034848c6c27 > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2023-10-08 19:37:20 +0200 > message: > > MDEV-31608 - Connector/NET fails to connect since 10.10 > > Connector/NET does not expect collation IDs returned by "show collations" > to be NULL, runs into an exception. > > The fix is to determine connector/net using its connection attributes, > exclude null-id collations from "show collations".The patch introduces > new old_mode NO_NULL_COLLATION_IDs, that is automatically set, > once MySQL Connector/NET connection is determined. I don't like this hack at all, but I don't have any better ideas, so just one question below > A test was added, that uses MySql.Data from powershell - only works > if MySql.Data is installed into GAC (i.e with C/NET MSI package) please, make sure that MySql.Data is installed on at least one buildbot builder. > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index bff55f7a431..bea993305be 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -6386,7 +6386,8 @@ int fill_schema_collation(THD *thd, TABLE_LIST *tables, COND *cond) > character set prefix). > */ > if (is_context && > - cmp(tmp_cl->cs_name, Lex_cstring(STRING_WITH_LEN("utf8mb4")))) > + ((thd->variables.old_behavior & OLD_MODE_NO_NULL_COLLATION_IDS) || > + cmp(tmp_cl->cs_name, Lex_cstring(STRING_WITH_LEN("utf8mb4"))))) wouldn't it be better to do instead - bool is_context= cmp(context_collation_name, full_collation_name); + bool is_context= cmp(context_collation_name, full_collation_name) + && !(thd->variables.old_behavior & OLD_MODE_NO_NULL_COLLATION_IDS); ? > continue; > restore_record(table, s->default_values); > table->field[0]->store(context_collation_name, scs); Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: [PATCH] MDEV-32551: "Read semi-sync reply magic number error" warnings on master
by Kristian Nielsen 23 Nov '23

23 Nov '23

Hi Monty, > From: Michael Widenius <monty(a)mariadb.com> > > commit 7abf83f86829ed94de618af4749eb96d5379bd90 (origin/bb-10.6-monty) > Author: Michael Widenius <monty(a)mariadb.com> > Date: Wed Nov 8 16:57:58 2023 -0700 > > MDEV-32551: "Read semi-sync reply magic number error" warnings on master I read through the patch, looks good. A few comments below, mostly minor things and typos that you can consider as you like. - Kristian. > - We should in 11.3 changed the default value for > rpl-semi-sync-master-wait-no-slave from TRUE to FALSE as the TRUE > Sys_semisync_master_wait_no_slave( > "rpl_semi_sync_master_wait_no_slave", > - "Wait until timeout when no semi-synchronous replication slave " > - "available (enabled by default).", > + "Wait until timeout when no semi-synchronous replication slave is " > + "available.", > GLOBAL_VAR(rpl_semi_sync_master_wait_no_slave), > CMD_LINE(OPT_ARG), DEFAULT(TRUE), > - NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0), > - ON_UPDATE(fix_rpl_semi_sync_master_wait_no_slave)); > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0)); > -my_bool rpl_semi_sync_master_wait_no_slave = 1; > +my_bool rpl_semi_sync_master_wait_no_slave = 0; There seems to be som inconsistency here: - The commit is based on 10.6, but the commit comment says 11.3 - The option help text is changed in the commit - The "DEFAULT(TRUE)" in the sysvar definition is _not_ changed in the commit - The C initialization values of the my_bool rpl_semi_sync_master_wait_no_slave _is_ changed in the commit. Should it be DEFAULT(FALSE) also in this patch, and the commit message say "10.6"? Or should these changes be removed from the 10.6 patch and put in 11.3 instead? At least this looks inconsistent. > diff --git a/mysql-test/include/wait_for_pattern_in_file.inc b/mysql-test/include/wait_for_pattern_in_file.inc > new file mode 100644 > index 00000000000..d088cb125c2 > --- /dev/null > +++ b/mysql-test/include/wait_for_pattern_in_file.inc > @@ -0,0 +1,54 @@ > +# ==== Purpose ==== > +# > +# Waits until pattern comes into log file or until a timeout is reached. Nice addition, this will be useful for other things! > +--let $keep_include_silent= 1 Why do you set this in the .inc? (Normally this is set in the .test files that want to skip the printout). +let SEARCH_SILENT=1; Maybe the .inc should restore the old value at the end (in case a .test will at some point set it globally for its own calls to include/search_pattern_in_file.inc) > +--let $keep_include_silent= 0 Here, better restore the original value (in case some .test has set it to 1 for itself). > +# Test is binlog format indepdendent, so save resources (Typo s/indepdendent/independent/) > +# The followowing command is likely to cause the slave master is not yet setup > +# for semi-sync (Typo s/followowing/following/) > - strcpy(m_reply_file_name, ""); > - strcpy(m_wait_file_name, ""); > + m_reply_file_name[0]= m_wait_file_name[0]= 0; Interestingly, GCC actually creates identical code for these (it knows to optimize strcpy()). (I'm just mentioning this as I know we both are interested in low-level code details, not asking you to revert the change). Example code: ----------------------------------------------------------------------- #include <string.h> struct X { char m_a[100]; char m_b[10]; void init1(); void init2(); }; void X::init1() { strcpy(m_a, ""); strcpy(m_b, ""); } void X::init2() { m_a[0]= m_b[0]= 0; } ----------------------------------------------------------------------- Result from gcc -O3: ----------------------------------------------------------------------- .type _ZN1X5init1Ev, @function _ZN1X5init1Ev: movb $0, (%rdi) movb $0, 100(%rdi) ret .type _ZN1X5init2Ev, @function _ZN1X5init2Ev: movb $0, 100(%rdi) movb $0, (%rdi) ret ----------------------------------------------------------------------- Identical, except that we now initialize in the opposite order ;-) > +#ifndef DBUG_OFF > +void dbug_verify_no_duplicate_slaves(Slave_ilist *m_slaves, THD *thd) > +{ You could make this (debug-only) function "static". > + Master is not responding (gone away?) or it has turned semi sync > + off. Turning off semi-sync repsonces as there is no point in sending (Typo s/repsonces/responses/)

2 1

Re: dadcf5ed7b1: MDEV-24486 Add `table_privileges` view to the `sys` schema
by Sergei Golubchik 21 Nov '23

21 Nov '23

Hi, Oleg, ok to push. Just one comment below. On Nov 20, Oleg Smirnov wrote: > revision-id: dadcf5ed7b1 (mariadb-11.0.1-259-gdadcf5ed7b1) > parent(s): b8f9f796ff8 > author: Oleg Smirnov > committer: Oleg Smirnov > timestamp: 2023-11-20 19:35:36 +0700 > message: > > MDEV-24486 Add `table_privileges` view to the `sys` schema > > The existing INFORMATION_SCHEMA.TABLE_PRIVILEGES displays only those > privileges that were specifically granted on the table level, > whereas it may be useful to see privileges granted at the database > and global level. > > diff --git a/scripts/sys_schema/NEWS.md b/scripts/sys_schema/NEWS.md > index 7aeefb7edf9..12ce1382b75 100644 > --- a/scripts/sys_schema/NEWS.md > +++ b/scripts/sys_schema/NEWS.md > @@ -1,5 +1,12 @@ > # Change history for the MySQL sys schema > > +## 1.5.2 (11/20/23) > + > +### Improvements > + > +* A new `table_privileges` view was added, which displays privileges to tables granted at all levels (user, schema, table) is it "privileges to tables"? I feel like "privileges on tables" would be more correct. It's GRANT privilege ON object TO user same it two other places. > + > + > ## 1.5.1 (07/07/16) Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

3 3

Question about STACK_DIRETION fix removal in merge commit 6c05edf
by Otto Kekäläinen 20 Nov '23

20 Nov '23

Hi Marko! I noticed you removed from debian/rules a section that flipped -DSTACK_DIRECTION=-1/1 based on architecture in the merge commit https://github.com/MariaDB/server/commit/6c05edfdcd335b9587ba140ebef5d9b082… Was this intentional? I am not sure, but it looks like a mistake to me, so I wanted to confirm. The code originates from cross-compilation fix in Debian in https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/f0ba31e1565fe…, which was later upstreamed to 10.5 (but not on 10.4) in https://github.com/MariaDB/server/commit/ea56841997995283f04e85e957c8d7f315…. - Otto

2 1

Re: 73097e18b77: MDEV-32771 Server crash upon online alter with concurrent XA
by Sergei Golubchik 14 Nov '23

14 Nov '23

Hi, Nikita, ok to push. spelling and renaming suggestions below On Nov 13, Nikita Malyavin wrote: > revision-id: 73097e18b77 (mariadb-11.2.1-20-g73097e18b77) > parent(s): f7646d890b9 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-11-13 15:28:19 +0100 > message: > > MDEV-32771 Server crash upon online alter with concurrent XA > > In case of a non-recovery XA rollback/commit in the same connection, > thon->rollback is called instead of rollback_by_xid, Though reviously, "previously" > thd_ha_data was moved to thd->transaction->xid_state.xid in hton->prepare. > > Like it wasn't enough, XA PREPARE can be skipped upon user and thus we > can end up in hton->commit/rollback with and unprepared XA, so checking > xid_state.is_explicit_XA is not enough -- we should check > xid_state.get_state_code() == XA_PREPARED, which will also guarantee > is_explicit_XA() == true. > --- a/sql/online_alter.cc > +++ b/sql/online_alter.cc > @@ -315,17 +315,35 @@ int online_alter_savepoint_rollback(handlerton *hton, THD *thd, sv_id_t sv_id) > > static int online_alter_commit(handlerton *hton, THD *thd, bool all) > { > - int res= online_alter_end_trans(get_cache_list(hton, thd), thd, > - ending_trans(thd, all), true); > - cleanup_tables(thd); > + int res; > + bool commit= ending_trans(thd, all); > + if (thd->transaction->xid_state.get_state_code() == XA_PREPARED && commit) > + { > + res= hton->commit_by_xid(hton, thd->transaction->xid_state.get_xid()); > + // cleanup is done by prepare() this is conusing, there's no prepare after commit_by_xid. may be you wanted to say "cleanup was already done by prepare()" ? > + } > + else > + { > + res= online_alter_end_trans(get_cache_list(hton, thd), thd, commit, true); > + cleanup_tables(thd); > + } > return res; > }; > > static int online_alter_rollback(handlerton *hton, THD *thd, bool all) > { > - int res= online_alter_end_trans(get_cache_list(hton, thd), thd, > - ending_trans(thd, all), false); > - cleanup_tables(thd); > + int res; > + bool commit= ending_trans(thd, all); 'commit' is weird here, on the rollback path. may be you rename it to, like "is_ending_tran" or "end_tran" or "is_persistent" or something along these lines. And then 'commit' in online_alter_commit too, for consistency. > + if (thd->transaction->xid_state.get_state_code() == XA_PREPARED && commit) > + { > + res= hton->rollback_by_xid(hton, thd->transaction->xid_state.get_xid()); > + // cleanup is done by prepare() same as in online_alter_commit > + } > + else > + { > + res= online_alter_end_trans(get_cache_list(hton, thd), thd, commit, false); > + cleanup_tables(thd); > + } > return res; > }; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: MDEV-27576 Use reverse index for max/min optimization
by Sergey Petrunia 13 Nov '23

13 Nov '23

Hi Yuchen, > commit c4c4bd8e47e5bfd4d0a650e286ea9daa3e6276f0 > Author: Yuchen Pei <ycp(a)mariadb.com> > Date: Thu Nov 2 11:55:42 2023 +1100 > > MDEV-27576 Use reverse index for max/min optimization > > We add a field in st_table_ref to indicate that the key used is > a descending index, which will flip the functions and flags used in > get_index_max_value() and get_index_min_value(), that allows correct > optimization for max/min for descending index. Generally the patch looks good. I have only a few cosmetic comments. Please rename the test file from mdev_27576.test to something more descriptive like desc_index_min_max.test (following desc_index_range.test) Also please rename the table in the test to t1. > diff --git a/sql/sql_select.h b/sql/sql_select.h > index e0bbadadc69..45d7527280a 100644 > --- a/sql/sql_select.h > +++ b/sql/sql_select.h > @@ -126,6 +126,7 @@ typedef struct st_table_ref > uint key_parts; ///< num of ... > uint key_length; ///< length of key_buff > int key; ///< key no > + bool reverse; ///< key is reverse > uchar *key_buff; ///< value to look for with key > uchar *key_buff2; ///< key_buff+key_length > store_key **key_copy; // It's actually not "key is reverse" but "last used key part is reverse"... also this member is never set for the most common scenario of TABLE_REF use, which is ref access. The member is set in a rather trivial logic in find_key_for_maxmin(), and then it's used in get_index_min_value() and get_index_max_value(). I suggest that we do without this member. get_index_min_value() and get_index_max_value() can check: bool reverse= table->key_info[ref->key].key_part[ref->key_part]->key_part_flag & HA_REVERSE_SORT > diff --git a/sql/opt_sum.cc b/sql/opt_sum.cc > index 794ec40fc66..1f94a4f4dc7 100644 > --- a/sql/opt_sum.cc > +++ b/sql/opt_sum.cc > @@ -114,7 +114,8 @@ static int get_index_min_value(TABLE *table, TABLE_REF *ref, > int error; > > if (!ref->key_length) > - error= table->file->ha_index_first(table->record[0]); > + error= ref->reverse ? table->file->ha_index_last(table->record[0]) : > + table->file->ha_index_first(table->record[0]); Please add { ... } as the if-branch is a multi-line statement now. > else > { > /* > @@ -206,8 +210,12 @@ static int get_index_max_value(TABLE *table, TABLE_REF *ref, uint range_fl) > table->file->ha_index_read_map(table->record[0], ref->key_buff, > make_prev_keypart_map(ref->key_parts), > range_fl & NEAR_MAX ? > - HA_READ_BEFORE_KEY : > - HA_READ_PREFIX_LAST_OR_PREV) : > + (ref->reverse ? HA_READ_AFTER_KEY : > + HA_READ_BEFORE_KEY) : > + (ref->reverse ? HA_READ_KEY_OR_NEXT : > + HA_READ_PREFIX_LAST_OR_PREV)) : > + ref->reverse ? > + table->file->ha_index_first(table->record[0]) : > table->file->ha_index_last(table->record[0])); Multiple levels of x?y:z statements are hard to read. Please change to if (ref->key_length) { ... } else { ... } BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Blog: http://petrunia.net

2 2

Re: 28deba0a611: MDEV-32101 CREATE PACKAGE [BODY] for sql_mode=DEFAULT
by Sergei Golubchik 10 Nov '23

10 Nov '23

Hi, Alexander, Looks great! Just a couple of questions below On Nov 07, Alexander Barkov wrote: > revision-id: 28deba0a611 (mariadb-11.0.1-239-g28deba0a611) > parent(s): e9573c05965 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-09-22 14:38:52 +0400 > message: > > MDEV-32101 CREATE PACKAGE [BODY] for sql_mode=DEFAULT > > This patch adds PACKAGE support with SQL/PSM dialect for sql_mode=DEFAULT: > > - CREATE PACKAGE > - DROP PACKAGE > - CREATE PACKAGE BODY > - DROP PACKAGE BODY > - Package function and procedure invocation from outside of the package: > -- using two step identifiers > SELECT pkg.f1(); > CALL pkg.p1() > > -- using three step identifiers > SELECT db.pkg.f1(); > CALL db.pkg.p1(); > > This is a non-standard MariaDB extension. > > However, later this code can be used to implement > the SQL Standard and DB2 dialects of CREATE MODULE. > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index af2ec678daa..fca39b63312 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -1285,7 +1285,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t *yystacksize); > %left TRANSACTION_SYM TIMESTAMP PERIOD_SYM SYSTEM USER COMMENT_SYM > > %left PREC_BELOW_SP_OBJECT_TYPE > -%left FUNCTION_SYM > +%left PACKAGE_MARIADB_SYM FUNCTION_SYM Why PACKAGE_ORACLE_SYM and PACKAGE_ALLMODES_SYM don't need it? (same for BODY_ORACLE_SYM and BODY_ALLMODES_SYM) > > > /* > @@ -3049,6 +3059,8 @@ sp_handler: > | PROCEDURE_SYM { $$= &sp_handler_procedure; } > | PACKAGE_ORACLE_SYM { $$= &sp_handler_package_spec; } > | PACKAGE_ORACLE_SYM BODY_ORACLE_SYM { $$= &sp_handler_package_body; } > + | PACKAGE_MARIADB_SYM { $$= &sp_handler_package_spec; } > + | PACKAGE_MARIADB_SYM BODY_MARIADB_SYM { $$= &sp_handler_package_body; } Why not to use here PACKAGE_ALLMODES_SYM and BODY_ALLMODES_SYM? (also, I'd personally call them simply PACKAGE_SYM and BODY_SYM) > ; > > > @@ -19428,35 +19489,33 @@ create_routine: > (Item_result) $8, $10)) > MYSQL_YYABORT; > } > - | create_or_replace definer_opt PACKAGE_ORACLE_SYM > + | create_or_replace definer_opt PACKAGE_ALLMODES_SYM > opt_if_not_exists sp_name opt_create_package_chistics_init > { > sp_package *pkg; > if (unlikely(!(pkg= Lex-> > - create_package_start(thd, > - SQLCOM_CREATE_PACKAGE, > - &sp_handler_package_spec, > - $5, $1 | $4)))) > + create_package_start(thd, &sp_handler_package_spec, > + $5, $1 | $4, > + Lex->sp_chistics)))) > MYSQL_YYABORT; > pkg->set_c_chistics(Lex->sp_chistics); Looks redundant > Lex->sphead->set_body_start(thd, YYLIP->get_cpp_tok_start()); > } > sp_tail_is > opt_package_specification_element_list END > - remember_end_opt opt_sp_name > + remember_end_opt opt_trailing_sp_name > { > if (unlikely(Lex->create_package_finalize(thd, $5, $12, $11))) > MYSQL_YYABORT; > } > - | create_or_replace definer_opt PACKAGE_ORACLE_SYM BODY_ORACLE_SYM > + | create_or_replace definer_opt PACKAGE_ALLMODES_SYM BODY_ALLMODES_SYM > opt_if_not_exists sp_name opt_create_package_chistics_init > { > sp_package *pkg; > if (unlikely(!(pkg= Lex-> > - create_package_start(thd, > - SQLCOM_CREATE_PACKAGE_BODY, > - &sp_handler_package_body, > - $6, $1 | $5)))) > + create_package_start(thd, &sp_handler_package_body, > + $6, $1 | $5, > + Lex->sp_chistics)))) > MYSQL_YYABORT; > pkg->set_c_chistics(Lex->sp_chistics); same > Lex->sphead->set_body_start(thd, YYLIP->get_cpp_tok_start()); Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: 2342aaf46eb: MDEV-30973 Implement debug logging for slave_transaction_retries
by Sergei Golubchik 07 Nov '23

07 Nov '23

Hi, Aleksey, On Nov 07, Aleksey Midenkov wrote: > revision-id: 2342aaf46eb (mariadb-11.0.1-256-g2342aaf46eb) > parent(s): 80bc5df1c80 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2023-10-10 13:11:19 +0300 > message: > > MDEV-30973 Implement debug logging for slave_transaction_retries > > Configuration options: > > log_slave_retries > log_slave_retries_max 1. a commit comment of >100 lines that's a bit too much, I think. 2. comments in MDEV-30973 say that one should be able to have everything in one file, namely in the error log. I don't see how it's possible in your commit. 3. one of the reasons for not simply writing into erorr log was "you get what are relatively large entries in there which can make it hard/annoying to also look for the normal, one-line warning and error output". But as far as I can see your slave retry entries are not large, but normal one-line messages. I'd say they can fit into the error log just fine. I've still commented below in the assumption that the separate log implementation will stay. But I honestly don't see a reason for it anymore. > diff --git a/mysql-test/suite/rpl/t/slave_deadlock.test b/mysql-test/suite/rpl/t/slave_deadlock.test > new file mode 100644 > index 00000000000..9fcda7c9b71 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/slave_deadlock.test > @@ -0,0 +1,194 @@ > +--source include/have_debug.inc > +--source include/have_innodb.inc > +--source include/have_binlog_format_statement.inc > + > +# On --repeat test infinitely increments thread_id and finally fails --sync_with_master at log_pos 1754: > +# > +# 231009 20:58:15 server id 1 end_log_pos 1754 CRC32 0x7e5aecd1 Query thread_id=561 exec_time=0 error_code=0 xid=0 I'm sorry, I don't understand that. Could you elaborate please? > +# > +# force_restart seems to fix this > + > +--source include/force_restart.inc > + > +# Note that test requires big enough --debug-sync-timeout unfair scheduling at high load. Sorry, cannot parse that. Could you elaborate please? > +# slave_deadlock.opt sets this to 1 hour > + > +--let $rpl_topology=1->2 > +--source include/rpl_init.inc > + > +--echo # > +--echo # MDEV-30973 Enhance output of innodb deadlocks (SAMU-124, SAMU-131) > +--echo # > +--connection server_2 > +SET @old_dbug= @@global.debug_dbug; > +set @old_parallel_threads= @@global.slave_parallel_threads; > +set @old_domain_parallel_threads= @@global.slave_domain_parallel_threads; > +set @old_parallel_mode= @@global.slave_parallel_mode; > +set @old_slave_transaction_retries= @@global.slave_transaction_retries; > +set @old_innodb_lock_wait_timeout= @@global.innodb_lock_wait_timeout; > +set @old_log_slave_retries= @@global.log_slave_retries; > +set @old_log_slave_retries_file= @@global.log_slave_retries_file; > +set @old_log_slave_retries_max= @@global.log_slave_retries_max; All this makes little sense, it's simpler to put all settings into the .opt file generally it's better to do your way, to have less server restarts, but you have an .opt file already, you can as well use it. except if you decide to remove the .opt file - then nevermind, the above it good. > +--source include/stop_slave.inc > +change master to master_use_gtid= slave_pos; > +set global slave_parallel_threads= 10; > +set global slave_domain_parallel_threads= 1; > +set global slave_parallel_mode= optimistic; > +set global slave_transaction_retries= 4; > +set global innodb_lock_wait_timeout= 1; ... > diff --git a/mysql-test/suite/versioning/r/rpl_row.result b/mysql-test/suite/versioning/r/rpl_row.result > index c161f68d0bd..6343fbecdec 100644 > --- a/mysql-test/suite/versioning/r/rpl_row.result > +++ b/mysql-test/suite/versioning/r/rpl_row.result > @@ -37,12 +37,7 @@ select *, unix_timestamp(row_start) as row_start, unix_timestamp(row_end) as row > x y row_start row_end > 1 1 12345.000000 2147483647.999999 > set timestamp= default; > -### INSERT INTO `test`.`t1` > -### SET > -### @1=1 > -### @2=1 > -### @3=12345.000000 > -### @4=2147483647.999999 why? > +/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/; > connection slave; > select * from t1; > x y > diff --git a/sql/log.cc b/sql/log.cc > index 877315c60de..78e874fda5b 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -1188,9 +1188,17 @@ bool Log_to_file_event_handler:: > log_error(enum loglevel level, const char *format, > va_list args) > { > - return vprint_msg_to_log(level, format, args); > + return vprint_msg_to_log(stderr, level, format, args); > } > > +#ifdef HAVE_REPLICATION > +bool Log_to_file_event_handler::log_slave_retry(const char *format, va_list args) > +{ > + DBUG_ASSERT(slave_retries_file.get()); > + return vprint_msg_to_log(slave_retries_file.get(), NO_LEVEL, format, args); I don't think you should use vprint_msg_to_log and add a NO_LEVEL hack. vprint_msg_to_log also prints into the windows event log, into wsrep json log, and it locks LOCK_error_log. You don't need any of that, just create your own function, it's just one fprintf() anyway. > +} > +#endif > + > void Log_to_file_event_handler::init_pthread_objects() > { > mysql_log.init_pthread_objects(); > @@ -9752,13 +9787,15 @@ static void print_buffer_to_file(enum loglevel level, const char *buffer, > } > } > > +#ifndef EMBEDDED_LIBRARY > mysql_mutex_lock(&LOCK_error_log); > +#endif > > skr= my_time(0); > localtime_r(&skr, &tm_tmp); > start=&tm_tmp; > > - fprintf(stderr, "%d-%02d-%02d %2d:%02d:%02d %lu [%s] %.*s%.*s\n", > + fprintf(file, "%d-%02d-%02d %2d:%02d:%02d [T%lu] %s%.*s%.*s\n", eh... no. don't change the format of existing log files, please. > start->tm_year + 1900, > start->tm_mon+1, > start->tm_mday, > @@ -9834,6 +9872,22 @@ void sql_print_error(const char *format, ...) > } > > > +#ifdef HAVE_REPLICATION > +void sql_print_error2(const char *format, ...) s/sql_print_error2/print_slave_error/ ? > +{ > + va_list args; > + DBUG_ENTER("sql_print_error"); > + > + va_start(args, format); > + error_log_print(ERROR_LEVEL, format, args); > + slave_retries_print(format, args); > + va_end(args); > + > + DBUG_VOID_RETURN; > +} > +#endif > + > + > void sql_print_warning(const char *format, ...) > { > va_list args; > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 8cc3ad51ee9..4f6b2273597 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -765,6 +765,90 @@ char *opt_relay_logname = 0, *opt_relaylog_index_name=0; > char *opt_logname, *opt_slow_logname, *opt_bin_logname; > char *opt_binlog_index_name=0; > > +#ifdef HAVE_REPLICATION > +char default_slave_retries_path[FN_REFLEN]; > +bool opt_slave_retries= false; > +char *opt_slave_retries_path= NULL; > +uint opt_slave_retries_max_log= 0; > +#endif > +Slave_retries_file slave_retries_file; > + > +bool Slave_retries_file::open() > +{ > +#ifdef HAVE_REPLICATION > + DBUG_ASSERT(opt_slave_retries); > + DBUG_ASSERT(opt_slave_retries_path); > + file= fopen(opt_slave_retries_path, "a"); my_fopen or my_open and anyway, why a new class, why not to use MYSQL_LOG like general and slow logs do? > + if (!file) > + { > + sql_print_error("Open of log_slave_retries '%s' failed: %s (%d)", > + opt_slave_retries_path, strerror(errno), errno); This is inconsistent, the server doesn't print this for other log files. I've just tried with general log and got 2023-11-06 17:58:19 0 [ERROR] mysqld: File '/tmp' not found (Errcode: 21 "Is a directory") 2023-11-06 17:58:19 0 [ERROR] Could not use /tmp for logging (error 21). Turning logging off for the whole duration of the MariaDB server process. To turn it on again: fix the cause, shutdown the MariaDB server and restart it. Please, do the same > + return true; > + } > + else > + { > + slave_retries_print("%s (mariadbd %s) starting as process %lu ...", > + my_progname, server_version, (ulong) getpid()); Any reason, why it shouldn't be the same as in the error log? sql_print_information("Starting MariaDB %s source revision %s as process %lu", server_version, SOURCE_REVISION, (ulong) getpid()); > + } > +#endif > + return false; > +} > + > +void Slave_retries_file::close() > +{ > +#ifdef HAVE_REPLICATION > + if (file) > + { > + acquire(); > + fclose(file); > + release(); > + } > + file= NULL; > +#endif > +} > + > +void Slave_retries_file::init() > +{ > +#ifdef HAVE_REPLICATION > + mysql_mutex_init(key_LOCK_global_system_variables, &mutex, MY_MUTEX_INIT_FAST); > +#endif > +} > + > +void Slave_retries_file::destroy() > +{ > +#ifdef HAVE_REPLICATION > + mysql_mutex_destroy(&mutex); > +#endif > +} > + > +bool Slave_retries_file::acquire() > +{ > +#ifdef HAVE_REPLICATION > + if (!file) > + return false; > + mysql_mutex_lock(&mutex); > + return true; > +#else > + return false; > +#endif > +} > + > +void Slave_retries_file::release() > +{ > +#ifdef HAVE_REPLICATION > + mysql_mutex_unlock(&mutex); > +#endif > +} > + > +void Slave_retries_file::flush() > +{ > + if (acquire()) > + { > + fflush(file); > + release(); > + } > +} > + > /* Static variables */ > > my_bool opt_stack_trace; > @@ -5055,6 +5145,35 @@ static int init_server_components() > "this server. However this will be ignored as the " > "--log-bin option is not defined."); > } > + > + slave_retries_file.init(); > + if (!log_error_file_ptr[0]) > + fn_format(default_slave_retries_path, pidfile_name, mysql_data_home, > + "-retries.err", MY_REPLACE_EXT); > + else > + fn_format(default_slave_retries_path, log_error_file_ptr, mysql_data_home, > + "-retries.err", MY_REPLACE_EXT); that's not how all other log names work, grep mysqld.cc for opt_slow_logname > + if (opt_slave_retries_path && !opt_abort) > + { > + if (!opt_slave_retries_path[0]) > + { > + if (!default_slave_retries_path[0]) > + { > + /* fn_format failed */ > + opt_slave_retries_path= NULL; > + opt_slave_retries= false; > + } > + else > + { > + opt_slave_retries_path= my_strdup(key_memory_Sys_var_charptr_value, > + default_slave_retries_path, MYF(0)); > + } > + } > + > + if (opt_slave_retries && > + slave_retries_file.open()) > + unireg_abort(1); > + } > #endif > > if (opt_bin_log) > diff --git a/sql/mysqld.h b/sql/mysqld.h > index 69cfc0cad56..5f1561ca10d 100644 > --- a/sql/mysqld.h > +++ b/sql/mysqld.h > @@ -822,6 +848,8 @@ enum options_mysqld > OPT_KEY_CACHE_CHANGED_BLOCKS_HASH_SIZE, > OPT_LOG_BASENAME, > OPT_LOG_ERROR, > + OPT_SLAVE_RETRIES, > + OPT_SLAVE_RETRIES_LOG, why do you need them? > OPT_LOWER_CASE_TABLE_NAMES, > OPT_PLUGIN_LOAD, > OPT_PLUGIN_LOAD_ADD, > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index d742bc6c1c3..c53ca0f0097 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -10662,6 +10663,15 @@ char *thd_get_error_context_description(THD *thd, char *buffer, > /* Don't wait if LOCK_thd_data is used as this could cause a deadlock */ > if (!mysql_mutex_trylock(&thd->LOCK_thd_data)) > { > + if (thd->rgi_slave) is rgi_slave and its content protected by thd->LOCK_thd_data ? > + { > + rpl_group_info *rgi= thd->rgi_slave; > + len= my_snprintf(header, sizeof(header), > + ", GTID %u-%u-%llu", rgi->current_gtid.domain_id, > + rgi->current_gtid.server_id, rgi->current_gtid.seq_no); > + str.append(header, len); > + } > + > if (const char *info= thread_state_info(thd)) > { > str.append(' '); > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index ebb64b340b0..11c321f06dc 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -5862,6 +5862,61 @@ Sys_slave_trans_retry_interval( > "slave_transaction_retry_errors", > GLOBAL_VAR(slave_trans_retry_interval), CMD_LINE(REQUIRED_ARG), > VALID_RANGE(0, 3600), DEFAULT(0), BLOCK_SIZE(1)); > + > +static bool update_slave_retries_log(sys_var *self, THD* thd, enum_var_type type) > +{ > + if (opt_slave_retries) > + { > + if (!slave_retries_file.get() && slave_retries_file.open()) > + { > + my_error(ER_CANT_OPEN_FILE, MYF(0), opt_slave_retries_path, errno); > + opt_slave_retries= false; > + return true; > + } > + } > + else > + slave_retries_file.close(); > + return false; > +} > + > +static Sys_var_mybool Sys_slave_retries_log( > + "log_slave_retries", "Log transaction retries", > + GLOBAL_VAR(opt_slave_retries), > + CMD_LINE(OPT_ARG, OPT_SLAVE_RETRIES), DEFAULT(false), > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0), ON_UPDATE(update_slave_retries_log)); > + > + > +static bool check_slave_retries_file(sys_var *self, THD *thd, set_var *var) > +{ > + if (opt_slave_retries) > + { > + my_error(ER_NOT_SUPPORTED_YET, MYF(0), "modifying while log_slave_retries is ON"); as always, please don't hard-code error messages. you can use my_error(ER_OPTION_PREVENTS_STATEMENT, MYF(0), "log_slave_retries=ON"); > + return true; > + } > + return false; > +} > + > +static bool update_slave_retries_file(sys_var *self, THD* thd, enum_var_type type) > +{ > + if (!opt_slave_retries_path) > + opt_slave_retries_path= my_strdup(key_memory_Sys_var_charptr_value, > + default_slave_retries_path, MYF(0)); sys_var.cc can handle that for you, see PREALLOCATED in e.g. opt_slow_logname also, you don't reopen the log when its name changes, how comes? > + return false; > +} > + > +static Sys_var_charptr Sys_slave_retries_log_path( > + "log_slave_retries_file", "Log file path for transaction retries", > + GLOBAL_VAR(opt_slave_retries_path), > + CMD_LINE(REQUIRED_ARG, OPT_SLAVE_RETRIES_LOG), DEFAULT(NULL), > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(check_slave_retries_file), > + ON_UPDATE(update_slave_retries_file)); > + > + > +static Sys_var_uint Sys_slave_retries_log_max( > + "log_slave_retries_max", "Log max N transaction retries " > + "(default 0 means infinity)", just "(0 means infinity)". Default is printed separately > + GLOBAL_VAR(opt_slave_retries_max_log), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0, UINT_MAX), DEFAULT(0), BLOCK_SIZE(1)); > #endif > > static bool check_locale(sys_var *self, THD *thd, set_var *var) > diff --git a/storage/innobase/lock/lock0lock.cc b/storage/innobase/lock/lock0lock.cc > index 0e9cef669d4..8a8eeaf39f6 100644 > --- a/storage/innobase/lock/lock0lock.cc > +++ b/storage/innobase/lock/lock0lock.cc > @@ -49,6 +49,7 @@ Created 5/7/1996 Heikki Tuuri > #include <debug_sync.h> > > #include <set> > +#include <stdio.h> I presume Marko has reviewed (or will review) InnoDB changes > > #ifdef WITH_WSREP > #include <mysql/service_wsrep.h> > diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result > index 77d6d10b3e6..274c034d455 100644 > --- a/mysql-test/main/mysqld--help.result > +++ b/mysql-test/main/mysqld--help.result > @@ -1673,6 +1678,9 @@ log-isam myisam.log > log-output FILE > log-queries-not-using-indexes FALSE > log-short-format FALSE > +log-slave-retries FALSE > +log-slave-retries-file (No default value) this is another sign that log-slave-retries-file doesn't behave like other log filename variables. > +log-slave-retries-max 0 > log-slave-updates FALSE > log-slow-admin-statements TRUE > log-slow-disabled-statements sp Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: [PATCH] MDEV-32673 Handle InnoDB binlog position recovery for RESET MASTER or binlog name change
by Kristian Nielsen 07 Nov '23

07 Nov '23

Hi Marko, Thanks for the patch. It looks good, just a few small comments inline. Approved after fixing the check for !is_relay_log. I have pushed an extra commit on top of bb-10.4-MDEV-32673 with a testcase (feel free to squash it onto your commit). The tests pass with your patch (but fails without it). - Kristian. > diff --git a/sql/handler.h b/sql/handler.h > index 6085111bc25..134212a0ad7 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1438,6 +1438,7 @@ struct handlerton > enum_binlog_command binlog_command, > const char *query, uint query_length, > const char *db, const char *table_name); > + void (*reset_binlog_pos)(const char *name, uint64_t pos); I think there should be a comment describing what this new call does. Suggestion: /* Called at server startup or RESET MASTER when the new binlog is opened. Informs the storage engine of the change in binlog file name and position. */ > diff --git a/sql/log.cc b/sql/log.cc > index 59730c3205a..e46f3796b63 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -4327,6 +4327,8 @@ bool MYSQL_BIN_LOG::reset_logs(THD *thd, bool create_new_log, > err: > if (error == 1) > name= const_cast<char*>(save_name); > + else > + ha_reset_binlog_pos(get_log_fname(), BIN_LOG_HEADER_SIZE); Sorry, I forgot something when we discussed this. This function is also used for resetting the relay logs. So it needs a check for !is_relay_log. For example: --- a/sql/log.cc +++ b/sql/log.cc @@ -4331,6 +4331,10 @@ bool MYSQL_BIN_LOG::reset_logs(THD *thd, bool create_new_log, if (!is_relay_log) { xid_count_per_binlog *b; + + if (!error) + ha_reset_binlog_pos(get_log_fname(), BIN_LOG_HEADER_SIZE); + /* Remove all entries in the xid_count list except the last. Normally we will just be deleting all the entries that we waited for to > diff --git a/storage/innobase/trx/trx0rseg.cc b/storage/innobase/trx/trx0rseg.cc > index be52373d03b..3ffa003588d 100644 > --- a/storage/innobase/trx/trx0rseg.cc > +++ b/storage/innobase/trx/trx0rseg.cc > @@ -826,3 +826,52 @@ trx_rseg_update_binlog_offset(byte* rseg_header, const trx_t* trx, mtr_t* mtr) > mlog_write_string(p, binlog_name, len, mtr); > } > } > + > +void trx_rseg_reset_binlog_pos(const char *name, uint64_t pos) > +{ > + mtr_t mtr; > + /* Invalidate the information on the TRX_SYS page. */ > + mtr.start(); > + if (buf_block_t *sys= trx_sysf_get(&mtr, true)) > + { > + byte *magic= TRX_SYS + TRX_SYS_MYSQL_LOG_INFO + > + TRX_SYS_MYSQL_LOG_MAGIC_N_FLD + sys->frame; > + if (mach_read_from_4(magic)) > + mlog_memset(magic, > + TRX_SYS_MYSQL_LOG_NAME + TRX_SYS_MYSQL_LOG_NAME_LEN, 0, > + &mtr); > + > + const uint32_t page_no= trx_sysf_rseg_get_page_no(sys, 0); > + if (page_no != FIL_NULL) > + { > + const uint32_t space_id= trx_sysf_rseg_get_space(sys, 0); > + ut_ad(space_id == TRX_SYS_SPACE); > + buf_block_t *block= buf_page_get(page_id_t(space_id, page_no), > + 0, RW_X_LATCH, &mtr); > + mlog_write_ull(TRX_RSEG + TRX_RSEG_BINLOG_OFFSET + block->frame, > + pos, &mtr); > + byte *p= TRX_RSEG + TRX_RSEG_BINLOG_NAME + block->frame; > + const size_t len= strlen(name) + 1; > + if (memcmp(name, p, len)) > + mlog_write_string(p, reinterpret_cast<const byte*>(name), len, &mtr); > + } > + > + /* Invalidate the information in the other rollback segment headers. */ > + for (ulint rseg_id= 1; rseg_id < TRX_SYS_N_RSEGS; rseg_id++) > + { > + const uint32_t page_no= trx_sysf_rseg_get_page_no(sys, rseg_id); > + if (page_no != FIL_NULL) > + { > + const uint32_t space_id= trx_sysf_rseg_get_space(sys, rseg_id); > + buf_block_t *block= buf_page_get(page_id_t(space_id, page_no), > + 0, RW_X_LATCH, &mtr); > + byte *name= TRX_RSEG + TRX_RSEG_BINLOG_NAME + block->frame; > + if (*name) > + mlog_write_ulint(name, 0, MLOG_1BYTE, &mtr); > + } > + } > + } > + mtr.commit(); > + /* Durably write the change. */ > + log_write_up_to(mtr.commit_lsn(), true); > +}

1 0

Re: 3aea3bd7422: MDEV-23294 Segfault or assertion upon MyISAM repair
by Sergei Golubchik 06 Nov '23

06 Nov '23

Hi, Aleksey, On Oct 23, Aleksey Midenkov wrote: > revision-id: 3aea3bd7422 (mariadb-10.4.30-176-g3aea3bd7422) > parent(s): 4fa6383c937 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2023-09-26 20:20:46 +0300 > message: > > MDEV-23294 Segfault or assertion upon MyISAM repair > > When computing vcol expression some items use current_thd and that was > not set in MyISAM repair thread. Since all the repair threads belong > to one connection and items should not write into THD we can utilize > table THD for that. > > diff --git a/storage/myisam/ha_myisam.cc b/storage/myisam/ha_myisam.cc > index ddab8bcf741..27151548704 100644 > --- a/storage/myisam/ha_myisam.cc > +++ b/storage/myisam/ha_myisam.cc > @@ -714,6 +714,8 @@ static int compute_vcols(MI_INFO *info, uchar *record, int keynum) > /* This mutex is needed for parallel repair */ > mysql_mutex_lock(&info->s->intern_lock); > TABLE *table= (TABLE*)(info->external_ref); > + if (!current_thd) > + set_current_thd(table->in_use); why not to do it in thr_find_all_keys(), unconditionally? you'd avoid an if() executed per row. Also, I'd suggest a comment, like /* To evaluate vcols we must have current_thd set. This will set current_thd in all threads to the same THD, but it's safe, because vcols are always evaluated under info->s->intern_lock. */ set_current_thd((TABLE*)(info->external_ref)->in_use); > table->move_fields(table->field, record, table->field[0]->record_ptr()); > if (keynum == -1) // update all vcols > { Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

Re: 0f0ef93c4d4: MDEV-32444 Data from orphaned XA transaction is lost after online alter
by Sergei Golubchik 02 Nov '23

02 Nov '23

Hi, Nikita, See below. A small suggestion for the test. And a rather larger comment about extending xid_t, as you, likely, expected :) On Oct 19, Nikita Malyavin wrote: > revision-id: 0f0ef93c4d4 (mariadb-11.2.1-15-g0f0ef93c4d4) > parent(s): 7f00853f2f1 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-10-13 13:39:36 +0400 > diff --git a/mysql-test/main/alter_table_online_debug.result b/mysql-test/main/alter_table_online_debug.result > index 1ebb19965eb..154aec003ca 100644 > --- a/mysql-test/main/alter_table_online_debug.result > +++ b/mysql-test/main/alter_table_online_debug.result > @@ -1563,6 +1563,103 @@ connection default; > drop table t1, t2; > set @@binlog_format=default; > set debug_sync= reset; > +# MDEV-32444 Data from orphaned XA transaction is lost after online alter > +create table t (a int primary key) engine=innodb; > +insert into t values (1); > +# XA commit > +set debug_sync= 'alter_table_online_downgraded signal downgraded wait_for go'; > +alter table t force, algorithm=copy, lock=none; > +connection con1; > +set debug_sync= 'now wait_for downgraded'; > +xa begin 'x1'; > +update t set a = 2 where a = 1; > +xa end 'x1'; > +xa prepare 'x1'; > +disconnect con1; > +connection con2; > +xa commit 'x1'; > +set debug_sync= 'now signal go'; > +connection default; > +select * from t; > +a > +2 > +# XA rollback > +set debug_sync= 'alter_table_online_downgraded signal downgraded wait_for go'; > +alter table t force, algorithm=copy, lock=none; > +connect con1, localhost, root,,; > +xa begin 'x2'; > +insert into t values (53); > +xa end 'x2'; > +xa prepare 'x2'; > +disconnect con1; > +connection con2; > +xa rollback 'x2'; > +set debug_sync= 'now signal go'; > +connection default; > +select * from t; > +a > +2 > +# XA transaction is left uncommitted > +# end then is rollbacked after alter fails > +set debug_sync= 'alter_table_online_downgraded signal downgraded wait_for go'; > +set statement innodb_lock_wait_timeout=0, lock_wait_timeout= 0 > +for alter table t force, algorithm=copy, lock=none; > +connect con1, localhost, root,,; > +xa begin 'xuncommitted'; > +insert into t values (3); > +xa end 'xuncommitted'; > +xa prepare 'xuncommitted'; > +set debug_sync= 'now signal go'; > +disconnect con1; > +connection default; > +ERROR HY000: Lock wait timeout exceeded; try restarting transaction > +xa rollback 'xuncommitted'; > +select * from t; > +a > +2 > +# Same, but commit > +set debug_sync= 'alter_table_online_downgraded signal downgraded wait_for go'; > +set statement innodb_lock_wait_timeout=0, lock_wait_timeout= 0 > +for alter table t force, algorithm=copy, lock=none; instead of `alter table t force`, better do something that changes the table, so that you could see whether alter was executed or aborted. For example, you can change the column type int->bigint and back. not only here, but everywhere. Or add/drop column, this has an added benefit that you won't need a separate `show create table`, your already existing `select * from t` will show whether the extra column exists > +connect con1, localhost, root,,; > +xa begin 'committed_later'; > +insert into t values (3); > +xa end 'committed_later'; > +xa prepare 'committed_later'; > +set debug_sync= 'now signal go'; > +disconnect con1; > +connection default; > +ERROR HY000: Lock wait timeout exceeded; try restarting transaction > +xa commit 'committed_later'; > +select * from t; > +a > +2 > +3 > +# Commit, but error in statement, and there is some stmt data to rollback > +set debug_sync= 'alter_table_online_downgraded signal downgraded wait_for go'; > +alter table t force, algorithm=copy, lock=none; > +connect con1, localhost, root,,; > +set debug_sync= 'now wait_for downgraded'; > +xa begin 'x1'; > +insert into t values (4), (3); > +ERROR 23000: Duplicate entry '3' for key 'PRIMARY' > +insert into t values (5); > +xa end 'x1'; > +xa prepare 'x1'; > +disconnect con1; > +connection con2; > +xa commit 'x1'; > +set debug_sync= 'now signal go'; > +connection default; > +select * from t; > +a > +2 > +3 > +5 > +connect con1, localhost, root,,; > +connection default; > +drop table t; > +set debug_sync= reset; > disconnect con1; > disconnect con2; > # > diff --git a/sql/handler.h b/sql/handler.h > index 07b392c010a..4622e49f883 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -903,6 +905,7 @@ struct xid_t { > long gtrid_length; > long bqual_length; > char data[XIDDATASIZE]; // not \0-terminated ! > + Online_alter_cache_list *online_alter_cache; well, no, we've talked about it. Even the comment above the structure says that you cannot change it. make a separate structure, xid_and_online_alter_cache_t or xid_t_internal or whatever. but xid_t should be defined identically everywhere. For example: https://github.com/berkeleydb/libdb/blob/master/src/dbinc/xa.h === okay, now I've seen how you used it and I'd say it's _such a rare use case_ that you can store your online_alter_cache in the XID_cache_element and look it up there. Or even in a separate data structure, that will be empty most of the time anyway. > > xid_t() = default; /* Remove gcc warning */ > bool eq(struct xid_t *xid) const > diff --git a/sql/online_alter.cc b/sql/online_alter.cc > index 8b8c81319d1..19abe57914d 100644 > --- a/sql/online_alter.cc > +++ b/sql/online_alter.cc > @@ -334,14 +336,57 @@ static int online_alter_log_init(void *p) > online_alter_hton->savepoint_rollback_can_release_mdl= > [](handlerton *hton, THD *thd){ return true; }; > > - online_alter_hton->commit= [](handlerton *hton, THD *thd, bool all) > - { return online_alter_end_trans(hton, thd, all, true); }; > - online_alter_hton->rollback= [](handlerton *hton, THD *thd, bool all) > - { return online_alter_end_trans(hton, thd, all, false); }; > - online_alter_hton->commit_by_xid= [](handlerton *hton, XID *xid) > - { return online_alter_end_trans(hton, current_thd, true, true); }; > - online_alter_hton->rollback_by_xid= [](handlerton *hton, XID *xid) > - { return online_alter_end_trans(hton, current_thd, true, false); }; > + online_alter_hton->commit= [](handlerton *hton, THD *thd, bool all) -> int > + { > + int res= online_alter_end_trans(get_cache_list(hton, thd), thd, > + ending_trans(thd, all), true); > + cleanup_tables(thd); > + return res; > + }; > + online_alter_hton->rollback= [](handlerton *hton, THD *thd, bool all) -> int > + { > + int res= online_alter_end_trans(get_cache_list(hton, thd), thd, > + ending_trans(thd, all), false); > + cleanup_tables(thd); > + return res; > + }; > + > + > + online_alter_hton->recover= [](handlerton*, XID*, uint){ return 0; }; > + online_alter_hton->prepare= [](handlerton *hton, THD *thd, bool all) -> int > + { > + auto &cache_list= get_cache_list(hton, thd); > + int res= 0; > + if (ending_trans(thd, all)) > + { > + thd->transaction->xid_state.set_online_alter_cache(&cache_list); > + thd_set_ha_data(thd, hton, NULL); > + } > + else > + { > + res= online_alter_end_trans(cache_list, thd, false, true); is that possible? > + } > + > + cleanup_tables(thd); > + return res; > + }; > + online_alter_hton->commit_by_xid= [](handlerton *hton, XID *xid) -> int > + { > + int res= online_alter_end_trans(*xid->online_alter_cache, current_thd, > + true, true); > + delete xid->online_alter_cache; > + xid->online_alter_cache= NULL; > + return res; > + }; > + online_alter_hton->rollback_by_xid= [](handlerton *hton, XID *xid) -> int > + { > + int res= online_alter_end_trans(*xid->online_alter_cache, current_thd, > + true, false); > + delete xid->online_alter_cache; > + xid->online_alter_cache= NULL; > + return res; > + }; > + > > online_alter_hton->drop_table= [](handlerton *, const char*) { return -1; }; > online_alter_hton->flags= HTON_NOT_USER_SELECTABLE | HTON_HIDDEN Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 6

Re: 2d937b62c33: MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in row_upd_sec_index_entry (debug) | Corruption
by Sergei Golubchik 02 Nov '23

02 Nov '23

Hi, Alexander, Few questions and suggestions below. Nothing major. On Oct 27, Alexander Barkov wrote: > commit 2d937b62c33 > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Mon Apr 4 14:50:21 2022 +0400 > > MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in row_upd_sec_index_entry (debug) | Corruption > would it make sense to change the MDEV title in Jira to better describe the problem? I sometimes do it with my bugs. Like "LPAD in vcol created in ORACLE mode makes table corrupted in non-ORACLE" (I tried to make it short, appropriate for a title) and, of course, change it in the comment and in the test file to match. > The crash happened with an indexed virtual column whose > value is evaluated using a function that has a different meaning > in sql_mode='' vs sql_mode=ORACLE: > > diff --git a/mysql-test/suite/compat/oracle/r/func_concat.result b/mysql-test/suite/compat/oracle/r/func_concat.result > index 392d579707a..17ca4be078a 100644 > --- a/mysql-test/suite/compat/oracle/r/func_concat.result > +++ b/mysql-test/suite/compat/oracle/r/func_concat.result > @@ -211,14 +211,14 @@ SET sql_mode=ORACLE; > CREATE VIEW v1 AS SELECT 'foo'||NULL||'bar' AS test; > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > -v1 CREATE VIEW "v1" AS select concat_operator_oracle(concat_operator_oracle('foo',NULL),'bar') AS "test" latin1 latin1_swedish_ci > +v1 CREATE VIEW "v1" AS select concat(concat('foo',NULL),'bar') AS "test" latin1 latin1_swedish_ci > SELECT * FROM v1; > test > foobar > SET sql_mode=DEFAULT; > SHOW CREATE VIEW v1; > View Create View character_set_client collation_connection > -v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select concat_operator_oracle(concat_operator_oracle('foo',NULL),'bar') AS `test` latin1 latin1_swedish_ci > +v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select oracle_schema.concat(oracle_schema.concat('foo',NULL),'bar') AS `test` latin1 latin1_swedish_ci please, add a mysqldump test. Like, create a table with virtual columns, check, defaults in the default sql mode. and create a table in oracle mode. Also, a view in the default mode and a view in oracle mode. and then mysqldump, to see that it dumps and restores everything correctly may be stored routines/triggers/etc, if you'd like, but they aren't directly relevant to this MDEV, as far as I understand. > SELECT * FROM v1; > test > foobar > diff --git a/mysql-test/suite/compat/oracle/r/func_decode.result b/mysql-test/suite/compat/oracle/r/func_decode.result > index 2809e971be3..1870a1ec0d5 100644 > --- a/mysql-test/suite/compat/oracle/r/func_decode.result > +++ b/mysql-test/suite/compat/oracle/r/func_decode.result > @@ -1,8 +1,8 @@ > SET sql_mode=ORACLE; > SELECT DECODE(10); > -ERROR 42000: Incorrect parameter count in the call to native function 'DECODE' > +ERROR 42000: Incorrect parameter count in the call to native function 'oracle_schema.DECODE' Hmm, may be this should say DECODE as before? you know, falls under the case "if you don't change sql_mode back and forth, you won't see schema-qualified names" > SELECT DECODE(10,10); > -ERROR 42000: Incorrect parameter count in the call to native function 'DECODE' > +ERROR 42000: Incorrect parameter count in the call to native function 'oracle_schema.DECODE' > SELECT DECODE(10,10,'x10'); > DECODE(10,10,'x10') > x10 > diff --git a/mysql-test/suite/compat/oracle/r/vcol_innodb.result b/mysql-test/suite/compat/oracle/r/vcol_innodb.result > new file mode 100644 > index 00000000000..9fa97c75c10 > --- /dev/null > +++ b/mysql-test/suite/compat/oracle/r/vcol_innodb.result > @@ -0,0 +1,51 @@ > +SET @table_open_cache=@@GLOBAL.table_open_cache; why do you need to manipulate table_open_cache? to trigger a reopen? Just do flush tables, it's explicit, more readable and more... controllable. > +SET sql_mode=''; > +CREATE TABLE t (d INT,b VARCHAR(1),c CHAR(1),g CHAR(1) GENERATED ALWAYS AS (SUBSTR(b,0,0)) VIRTUAL,PRIMARY KEY(b),KEY g(g)) ENGINE=InnoDB; > +INSERT INTO t VALUES (0); > +ERROR 21S01: Column count doesn't match value count at row 1 > +SET sql_mode='ORACLE'; > +INSERT INTO t SET c=REPEAT (1,0); > +Warnings: > +Warning 1364 Field 'b' doesn't have a default value > +ALTER TABLE t CHANGE COLUMN a b INT; > diff --git a/sql/item_func.h b/sql/item_func.h > index 76a997c33fb..cdbefb82541 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -56,8 +56,40 @@ class Item_func :public Item_func_or_sum, > bool check_argument_types_can_return_date(uint start, uint end) const; > bool check_argument_types_can_return_time(uint start, uint end) const; > void print_cast_temporal(String *str, enum_query_type query_type); > + > + void print_schema_qualified_name(String *to, > + const LEX_CSTRING &schema_name, > + const char *function_name) const I don't see why you'd need this helper. is it something that was used in earlier versions of the patch? > + { > + // e.g. oracle_schema.func() > + to->append(schema_name); > + to->append('.'); > + to->append(function_name); > + } > + > + void print_sql_mode_qualified_name(String *to, > + enum_query_type query_type, > + const char *function_name) const > + { > + const Schema *func_schema= schema(); > + if (!func_schema || func_schema == Schema::find_implied(current_thd)) > + to->append(function_name); > + else > + print_schema_qualified_name(to, func_schema->name(), function_name); > + } > + > + void print_sql_mode_qualified_name(String *to, enum_query_type query_type) > + const > + { > + return print_sql_mode_qualified_name(to, query_type, func_name()); > + } I don't see why you need this helper either, you never use print_sql_mode_qualified_name with the last argument being not func_name(). So you can remove this helper and the third argument of print_sql_mode_qualified_name. > + > public: > > + // Print an error message for a builtin-schema qualified function call > + static void wrong_param_count_error(const LEX_CSTRING &schema_name, > + const LEX_CSTRING &func_name); > + > table_map not_null_tables_cache; > > enum Functype { UNKNOWN_FUNC,EQ_FUNC,EQUAL_FUNC,NE_FUNC,LT_FUNC,LE_FUNC, > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index ae078dbb22f..92d5e196da4 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -2170,13 +2170,31 @@ bool Item_func_trim::fix_length_and_dec() > > void Item_func_trim::print(String *str, enum_query_type query_type) > { > + LEX_CSTRING suffix= {STRING_WITH_LEN("_oracle")}; > if (arg_count == 1) > { > - Item_func::print(str, query_type); > + if (query_type & QT_FOR_FRM) > + { > + // 10.3 downgrade compatibility for FRM > + str->append(func_name()); > + if (schema() == &oracle_schema_ref) > + str->append(suffix); > + } > + else > + print_sql_mode_qualified_name(str, query_type, func_name()); > + print_args_parenthesized(str, query_type); > return; > } > - str->append(Item_func_trim::func_name()); > - str->append(func_name_ext()); > + > + if (query_type & QT_FOR_FRM) > + { > + // 10.3 downgrade compatibility for FRM > + str->append(Item_func_trim::func_name()); > + if (schema() == &oracle_schema_ref) > + str->append(suffix); > + } > + else > + print_sql_mode_qualified_name(str, query_type, Item_func_trim::func_name()); it'd be simpler if you move the above block that prints the function name before if (arg_count == 1) also you won't need suffix, but can do like in all other functions str->append(STRING_WITH_LEN("trim_oracle"); > str->append('('); > str->append(mode_name()); > str->append(' '); > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 71f592a3852..bb53d1a510a 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -2084,7 +2100,64 @@ bool Lex_input_stream::get_7bit_or_8bit_ident(THD *thd, uchar *last_char) > } > > > -int Lex_input_stream::scan_ident_sysvar(THD *thd, Lex_ident_cli_st *str) > +/* > + Resolve special SQL functions that have a qualified syntax in sql_yacc.yy. > + These functions are not listed in the native function registry > + because of a special syntax, or a reserved keyword: > + > + mariadb_schema.SUBSTRING('a' FROM 1 FOR 2) -- Special syntax I didn't find it in Oracle's manual, by the way > + mariadb_schema.TRIM(BOTH ' ' FROM 'a') -- Special syntax > + mariadb_schema.REPLACE('a','b','c') -- Verb keyword > +*/ > + > +int Lex_input_stream::find_keyword_qualified_special_func(Lex_ident_cli_st *str, > + uint length) const > +{ > + /* > + There are many other special functions, see the following grammar rules: > + function_call_keyword > + function_call_nonkeyword > + Here we resolve only those that have a qualified syntax to handle > + different behavior in different @@sql_mode settings. > + > + Other special functions do not work in qualified context: > + SELECT mariadb_schema.year(now()); -- Function year is not defined > + SELECT mariadb_schema.now(); -- Function now is not defined > + > + We don't resolve TRIM_ORACLE here, because it does not have > + a qualified syntax yet. Search for "trim_operands" in sql_yacc.yy > + to find more comments. > + */ > diff --git a/sql/sql_schema.h b/sql/sql_schema.h > index 1174bc7a83f..2c52646f2ea 100644 > --- a/sql/sql_schema.h > +++ b/sql/sql_schema.h > @@ -77,5 +98,6 @@ class Schema > > > extern Schema mariadb_schema; > +extern const Schema &oracle_schema_ref; What's the difference between these two definitions. Do you expect someone will need to change mariadb_schema? > > #endif // SQL_SCHEMA_H_INCLUDED Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Eliminating sprintf deprecation warnings (maybe others as well)
by Stein Vidar Hagfors Haugan 27 Oct '23

27 Oct '23

Hi! First time here, so my apologies if I'm violating some spoken or unspoken rules :) I'm developing/maintaining two specialized storage engines(*) and I have one issue: There are numerous uses of sprintf() in the general mariadb codebase, and each one generates a warning during compilation when doing a plain cmake/make build (on MacOS & RHEL7 at least). I am a bit fanatical about hunting down and eliminating warnings if at all possible, no matter how small and insignificant - if you become accustomed to routinely getting warnings during compilation, you can easily overlook an important one. So I am considering to contribute with a patch that will eliminate all sprintf() instances by converting them to snprintf(destination, N, ...). Of course, hunting down the correct N for each instance can be a *lot* of work (I presume this is why this hasn't been done already). So I'm proposing to make a macro UNSAFE_SNPRINTF(destination, ...) which would expand to snprintf(destination, 10000, ...) and replace all current sprintf() calls with that. Yes, 10000 is a ridiculously high number that makes it look like we're introducing a huge security risk, but but the current sprintf() calls have an implicit N = infinity. In some instances, the compiler sniffs out the actual size of the buffer and complains - in that case I would change the code to an appropriate regular snprintf() call. Would doing this be worthwile? I.e., is there a good chance that such a patch would actually be accepted into the codebase? If so, I might also do the same with other warnings (I believe there are some type-punning warnings at least). Sincerely, Stein Haugan *) The engines are for internal use so far, but might make them public at some point

2 2

Re: [PATCH 4/4] MDEV-31273: Precompute binlog checksums
by Michael Widenius 26 Oct '23

26 Oct '23

Hi! Review of MDEV-31273: Precompute binlog checksums On Fri, Aug 25, 2023 at 10:16 AM Kristian Nielsen <knielsen(a)knielsen-hq.org> wrote: > > Compute binlog checksums (when enabled) already when writing events > into the statement or transaction caches, where before it was done > when the caches are copied to the real binlog file. This moves the > checksum computation outside of holding LOCK_log, improving > scalabitily. DBUG_RETURN(my_b_copy_to_cache(from_cache, to_cache, SIZE_T_MAX)); You could use from_cache->end_of_file instead of SIZE_T_MAX <cut> uchar checksum_opt; Wouldn't it be better to have this as an "enum_binlog_checksum_alg" to avoid some casts ? > } > else if (data) > return (enum enum_binlog_checksum_alg)data->checksum_opt; > else > return BINLOG_CHECKSUM_ALG_OFF; > } You can remove the else's <cut> ulong *param_ptr_binlog_cache_disk_use, bool precompute_checksums) : stmt_cache(precompute_checksums), trx_cache(precompute_checksums), last_commit_pos_offset(0), using_xa(FALSE), xa_xid(0) { stmt_cache.set_binlog_cache_info(param_max_binlog_stmt_cache_size, param_ptr_binlog_stmt_cache_use, This code was a bit confusing as we first initialize stmt_cache and trx_cache with checksum and then we initialize them again in the next two lines. I understand why and this does not have to be changed, but it does look a bit strange. The other option would be to have set_binlog_cache_info() take checksum as an argument(). <cut> This one confused me a bit: Log_event_writer writer(file, 0, checksum_alg, &crypto); Log_event_writer writer(file, cache_data, checksum_alg, &crypto); Why this change? It may at least affect calls to: void Log_event_writer::add_status(enum_logged_status status) { if (likely(cache_data)) Fix (after discussions on slack): Remove the following lines in MYSQL_BIN_LOG::write_event(Log_event *ev, binlog_cache_data *cache_data, if (cache_data) cache_data->add_status(ev->logged_status()); <cut> > Don't attempt to precompute checksums if: > - Disabled by user request, --binlog-legacy-event-pos > - Binlog is encrypted, cannot use precomputed checksums > - WSREP/Galera. Why Galera? Would be good to explain this in the commit comment. <cut> > /* > If possible, just copy the cache over byte-by-byte with pre-computed > checksums. > */ > if (likely(binlog_checksum_options == cache_data->checksum_opt) && > likely(!crypto.scheme) && > likely(!opt_binlog_legacy_event_pos)) > { > int res= my_b_copy_all_to_cache(cache, &log_file); > status_var_add(thd->status_var.binlog_bytes_written, my_b_tell(cache)); > DBUG_RETURN(res ? ER_ERROR_ON_WRITE : 0); > } I was just wondering if this could sometimes be optimized to write directly to the real binlog file without another cache in between. <cut> bool precomputed_checksums= (cache_data->checksum_opt != BINLOG_CHECKSUM_ALG_OFF); uint old_checksum_len= precomputed_checksums ? BINLOG_CHECKSUM_LEN : 0; Why two variables when one can as easily do: uint old_checksum_len= ((cache_data->checksum_opt != BINLOG_CHECKSUM_ALG_OFF) ? BINLOG_CHECSUM_LEN : 0); In current write_cache() code, group is a bad name. Please rename 'group' to 'log_file_pos'. By the way, great that you removed the 'end_log_file_pos' variable, which also was a very confusing name related how it was used! > /* > Any old precomputed checksum must _not_ be written here. Instead, it > must be discarded; the new checksum, if needed, is written by > writer.write_footer(). > */ > if (ev_len > old_checksum_len) > { > uint bytes_to_skip= > old_checksum_len - std::min(old_checksum_len, ev_len - chunk); > if (writer.write_data(cache->read_pos, chunk - bytes_to_skip)) > goto error_in_write; > } The above is likely wrong as for long events we would execute the inner part multiple times, while there is only on checksum. As checksum is last in the event, why not just do 'even_len-= old_checksum' Before the loop to copy the event and then disregard 'old_checksum_len' bytes from the cache at the end of the loop? Anyway, to find bugs like this, we need to have a test case with events that are bigger than the IO_CACHE size for cache. Setting binlog_file_cache_size to 8192 (min value) should make it easy to test this. <cut> > binlog_checksum_options= value; > my_atomic_storeul_explicit(&binlog_checksum_options, value, MY_MEMORY_ORDER_RELAXED); Atom is not needed as we have a lock on binlog. ...... Something totally different. I noticed in MYSQL_BIN_LOG::write_cache(): group= (size_t)my_b_tell(&log_file); val= uint4korr(header + LOG_POS_OFFSET) + group + end_log_pos_inc; int4store(header + LOG_POS_OFFSET, val); The first log entry is probably ok, as we have done a rotate() event before. However the cache can have a lot of log_events (as part of a transaction). The end_log_pos after the first event can be wrong as it may not fit into 4 bytes. My understanding is that this is not a problem anymore as we are not using end_log_pos anymore. However I still wonder if rotate() should not only consider the current log file size but also the size of all events we plan to write to the log and do a rotate if the total new log file size > 4G. Regards, Monty

2 1

Re: [PATCH 3/4] MDEV-31273: Refactor MYSQL_BIN_LOG::write_cache()
by Michael Widenius 26 Oct '23

26 Oct '23

Hi! On Fri, Aug 25, 2023 at 10:16 AM Kristian Nielsen <knielsen(a)knielsen-hq.org> wrote: > > Preparatory patch for pre-computing binlog checksums outside of holding > LOCK_log. > > The existing code for MYSQL_BIN_LOG::write_cache() was needlessly complex > and very hard to understand and modify for handling the new case where > pre-computed checksums are already present in the IO_CACHE. MDEV-31273: Refactor MYSQL_BIN_LOG::write_cache() > if (likely(length > LOG_EVENT_HEADER_LEN)) > { > header= cache->read_pos; > cache->read_pos+= LOG_EVENT_HEADER_LEN; > length-= LOG_EVENT_HEADER_LEN; > } > else > { > size_t sofar= length; > size_t remain= LOG_EVENT_HEADER_LEN - sofar; > header= &header_buf[0]; > memcpy(header, cache->read_pos, sofar); > cache->read_pos+= sofar; > > while (hdr_offs < length) > length= my_b_fill(cache); > if (!length) > goto error_in_read; > size_t chunk= std::min(length, remain); > memcpy(header + sofar, cache->read_pos, chunk); > sofar+= chunk; > remain-= chunk; > cache->read_pos+= chunk; > length-= chunk; > } while (unlikely(remain > 0)); > } > The above can be replaced with: if (my_b_read(cache, header, LOG_EVENT_HEAD_LENGTH)) goto error_in_read; It is almost exactly as efficent as the above (one extra if) and avoids using cache internals. Note that you do not have to do call my_fill() if you use my_b_read(). my_b_read() will return 0 if it was able to read all data. In case of end of file, my_b_read() will return 1 and info->error will be 0. If needed, you can als find out how much data left to read from IO_CACHE: left_data_to_read= (cache->end_of_file - my_b_tell(cache)) (I can make an inline function of that if needed). /* Write the rest of the event. */ > while (ev_len > 0) > { > if (length == 0) > length= my_b_fill(cache); > if (!length) > goto error_in_read; > -> while (ev_len > 0) { if (length == 0) { if (!(length= my_b_fill(cache))); goto error_in_read; } <cut> uint chunk= std::min(ev_len, (uint)length); I would have prefer to have MY_MIN() used (like the rest of the code). (not critical) Regards, Monty

2 1

Re: ce9ce585e72: MDEV-31184 Remove parser tokens DECODE_MARIADB_SYM and DECODE_ORACLE_SYM
by Sergei Golubchik 23 Oct '23

23 Oct '23

Hi, Alexander, On Oct 21, Alexander Barkov wrote: > revision-id: ce9ce585e72 (mariadb-10.4.30-103-gce9ce585e72) > parent(s): 1fde785315e > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-08-31 13:49:19 +0400 > message: > > MDEV-31184 Remove parser tokens DECODE_MARIADB_SYM and DECODE_ORACLE_SYM > > Changing the code handling sql_mode-dependent function DECODE(): > > - removing parser tokens DECODE_MARIADB_SYM and DECODE_ORACLE_SYM > - removing the DECODE() related code from sql_yacc.yy/sql_yacc_ora.yy > - adding handling of DECODE() with help of a new Create_func_func_decode please, add a test for DECODE_MARIADB_SYM that changes from ER_PARSE_ERROR to ER_WRONG_PARAMCOUNT_TO_NATIVE_FCT. Why sql_yacc.yy had rules for DECODE_ORACLE_SYM and vice versa? Was it possible to get DECODE_ORACLE_SYM in sql_yacc.yy? > diff --git a/mysql-test/suite/compat/oracle/r/func_decode.result b/mysql-test/suite/compat/oracle/r/func_decode.result > index b49bad93627..2809e971be3 100644 > --- a/mysql-test/suite/compat/oracle/r/func_decode.result > +++ b/mysql-test/suite/compat/oracle/r/func_decode.result > @@ -1,8 +1,8 @@ > SET sql_mode=ORACLE; > SELECT DECODE(10); > -ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ')' at line 1 > +ERROR 42000: Incorrect parameter count in the call to native function 'DECODE' > SELECT DECODE(10,10); > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index cdc04d93708..048741b6ca1 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -11027,18 +11024,6 @@ function_call_nonkeyword: > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | DECODE_MARIADB_SYM '(' expr ',' expr ')' > - { > - $$= new (thd->mem_root) Item_func_decode(thd, $3, $5); > - if (unlikely($$ == NULL)) > - MYSQL_YYABORT; > - } > - | DECODE_ORACLE_SYM '(' expr ',' decode_when_list_oracle ')' > - { > - $5->push_front($3, thd->mem_root); > - if (unlikely(!($$= new (thd->mem_root) Item_func_decode_oracle(thd, *$5)))) > - MYSQL_YYABORT; > - } > | EXTRACT_SYM '(' interval FROM expr ')' > { > $$=new (thd->mem_root) Item_extract(thd, $3, $5); > @@ -12209,25 +12194,6 @@ when_list_opt_else: > } > ; > > -decode_when_list_oracle: > - expr ',' expr > - { > - $$= new (thd->mem_root) List<Item>; > - if (unlikely($$ == NULL) || > - unlikely($$->push_back($1, thd->mem_root)) || > - unlikely($$->push_back($3, thd->mem_root))) > - MYSQL_YYABORT; > - > - } > - | decode_when_list_oracle ',' expr > - { > - $$= $1; > - if (unlikely($$->push_back($3, thd->mem_root))) > - MYSQL_YYABORT; > - } > - ; > - > - > /* Equivalent to <table reference> in the SQL:2003 standard. */ > /* Warning - may return NULL in case of incomplete SELECT */ Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 1

Re: 56fa1da9c67: MDEV-30048 Prefix keys for CHAR work differently for MyISAM vs InnoDB
by Sergei Golubchik 23 Oct '23

23 Oct '23

Hi, Alexander, The actual code change (replacing strnncollsp with strnncollsp_nchars) is ok. But I didn't like the functions you've created. The *_ft_* family is quite confusing, I would not expect to see fulltext-specific functions my_compare.h. I would expect to see there functions which are generic and named by what they're doing, not by where they should be used. ha_compare_char_fixed and ha_compare_char_varying are better. Names are clearer and it's kind of understandable what they do. I suggest you remove *_ft_* functions. fulltext code should, I guess, just use charset_info->coll->strnncoll() everywhere. Not strnncollsp(), because words normally don't have trailing spaces, and if some custom pluggable parser returns words with trailing spaces, they're likely a part of the word and should be compared too. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org On Oct 17, Alexander Barkov wrote: > revision-id: 56fa1da9c67 (mariadb-10.4.28-90-g56fa1da9c67) > parent(s): ed2adc8c6f9 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-04-07 14:54:17 +0400 > message: > > MDEV-30048 Prefix keys for CHAR work differently for MyISAM vs InnoDB > > Also fixes: MDEV-30050 Inconsistent results of DISTINCT with NOPAD > > Problem: > > Key segments for CHAR columns where compared using strnncollsp() > for engines MyISAM and Aria. > > This did not work correct in case if the engine applyied trailing > space compression. > > Fix: > > Replacing ha_compare_text() with a number of new functions: > > - ha_compare_ft_text_full() > - ha_compare_ft_text_prefix() > - ha_compare_ft_text() > - ha_compare_char_varying() > - ha_compare_char_fixed() > > The code branch corresponding to comparison of CHAR column keys > (HA_KEYTYPE_TEXT segment type) now uses ha_compare_char_fixed() > which calls strnncollsp_nchars(). > > For the rest of the code: > - comparison of VARCHAR/TEXT column keys > (HA_KEYTYPE_VARTEXT1, HA_KEYTYPE_VARTEXT2 segments types) > - comparison in the fulltext code > this patch does not change the behaviour.

2 3

Re: [PATCH] MDEV-31949 parallel slave xa Round-Robin distribution
by Kristian Nielsen 22 Oct '23

22 Oct '23

Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: Hi Andrei, > commit 96bd9e6b780f3738b5008b89aae6b0b086f15943 > Author: Andrei <andrei.elkin(a)mariadb.com> > Date: Sat Aug 19 19:49:25 2023 +0300 > XA-Prepare group of events > > XA START xid > ... > XA END xid > XA PREPARE xid > > and its XA-"complete" terminator > > XA COMMIT or > XA ROLLBACK > > are made distributed Round-Robin across slave parallel workers. > The former hash-based policy was proven to attribute to execution > latency through creating a big - many times larger than the size > of the worker pool - queue of binlog-ordered transactions > to commit. > > Acronyms and notations used below: > > XAP := XA-Prepare event or the whole prepared XA group of events > XAC := XA-"complete", which is a solitary group of events > |W| := the size of the slave worker pool > Subscripts like `_k' denote order in a corresponding sequence > (e.g binlog file). Here is my review of this patch. > The list is arranged as a sliding window with the size of 2*|W| to account > a possibility of XAP_k -> XAP_k+2|W|-1 the largest (in the group-of-events > count sense) dependency. I spent a lot of effort trying to understand why the factor 2 in the size of 2*|W| occurs. Since each transaction must wait_for_prior_commit for the prior transaction, there should never be more than |W| transactions active. In your example with |W|=4, you say that > Worker #4 can take on its T_8 when T_1 is yet at the beginning of its processing But that does not seem possible. T_8 cannot start until T_1 (and 2..4) have done wakeup_subsequent_commits(), which surely shouldn't happen "at the beginning of its processing", but only at the end. But I think I now understand. The problem is that the XA code calls wakeup_subsequent_commits _before_ updating the XID hash (in XA PREPARE) / deleting from the XID hash (in XA COMMIT/ROLLBACK). Right? This I think is the fundamental issue to address. The wait_for_prior_commit mechanism is to ensure that the prior transaction has completed its commit, which means the state of the transaction is "committed" in memory. Thus, the XID hash, which records information about the status of the (XA) transactions, must also be updated before wakeup_prior_commit may be done. I can see that the XA PREPARE/COMMIT/ROLLBACK already uses the trx_group_commit_leader() code. So what you should do is to arrange for the update of / delete from the XID hash happens inside there, just before the wakeup_subsequent_commits() gets called: if (current->cache_mngr->using_xa && likely(!current->error) && DBUG_EVALUATE_IF("skip_commit_ordered", 0, 1)) { mysql_mutex_lock(&current->thd->LOCK_thd_data); run_commit_ordered(current->thd, current->all); mysql_mutex_unlock(&current->thd->LOCK_thd_data); } current->thd->wakeup_subsequent_commits(current->error); I checked the code, the XID hash update/delete currently happens shortly after that in the code, so there should be no problems moving it in there. [Incidentally, I also noticed that the XA PREPARE/COMMIT does not use the commit_ordered mechanism. While this seems (I think) unrelated to this patch, I think it is something you need to look into. The commit_ordered mechansim is central to ensuring that InnoDB and binlog commit in the same order, and I don't see how this is guaranteed for the current XA code (and I suspect this might also mean that binlog recovery will be broken in some cases).] Once the XID hash update/delete is moved as described, we have the very nice property that after an XA PREPARE/COMMIT/ROLLBACK T_i has done wakeup_subsequent_commits(), it is safe for a following T_j that refers to the same XID to replicate. This means that you can now use the existing wait_for_prior_commit mechanism to handle the dependencies between XA event groups with the same XID. Thus, there is no need to introduce a separate (and very complicated) wait mechanism of xid_cache_insert_maybe_wait() and xid_cache_search_maybe_wait(), solely for parallel replication of user XA. Thus, in the SQL driver thread, using your sliding window (which need only be of size |W| I believe), you can mark a "T1: XA COMMIT <xid> " followed by "T2: XA PREPARE <xid>" of a duplicate XID, that T2 must do a wait_for_prior_commit(T1) before running. This is simple, it can use the existing mechanims for that, using entry->last_committed_sub_id. Just like rgi->wait_commit_sub_id and rgi->wait_commit_group_info, we can introduce eg.: rgi->pre_dependency_sub_id rgi->pre_dependency_group_info and then do a wait_for_prior_commit(&rgi->pre_dependency_group_info->commit_orderer) if the rgi->pre_dependency_sub_id > entry->last_committed_sub_id) Similarly, the sliding window can record for "T3: XA PREPARE" and "T4: XA COMMIT", that T4 should do a wait_for_prior_commit(T3) before running. Then T4 will be sure that the XA transaction is ready. This will also be a good preparation for later introducing more of this kind of pre-calculated dependencies to the parallel replication scheduling, along the lines of the "balanced applier" that you have written about previously. These ideas I believe have tremendous potential, and handling this in a generic way is a very big improvement. I'm not suggesting to implement more than what is necessary now for XA, but keep it in mind that this dependency calculation in the SQL driver thread can be used for other than user XA in the future. You can also name the new introduced dependency fields appropriately (ie. with generic names not specific to XA when applicable). This will greatly simplify the patch, I believe; and more importantly it will integrate the XA-required scheduling in a clean and generic way in the parallel replication code. Following is more detailed comments on the patch: > - DBUG_ASSERT( > - !(thd->rgi_slave && > - !thd->rgi_slave->worker_error && > - thd->rgi_slave->did_mark_start_commit) || > - (thd->transaction->xid_state.is_explicit_XA() || > - (thd->rgi_slave->gtid_ev_flags2 & Gtid_log_event::FL_PREPARED_XA))); > - > + DBUG_ASSERT(!(thd->rgi_slave && > + !thd->rgi_slave->worker_error && > + thd->rgi_slave->did_mark_start_commit) || > + (thd->transaction->xid_state.is_explicit_XA() || > + (thd->rgi_slave->gtid_ev_flags2 & > + (Gtid_log_event::FL_PREPARED_XA | > + Gtid_log_event::FL_COMPLETED_XA)))); > if (thd->rgi_slave && > !thd->rgi_slave->worker_error && > thd->rgi_slave->did_mark_start_commit) > thd->rgi_slave->unmark_start_commit(); Why is it allowed to rollback here while did_mark_start_commit is true for XA PREPARE or XA COMMIT? The comment above should be extended to explain this. And why does the following if () statement then still do an emergency unmark_start_commit(), when the condition is not caught by the assertion? It looks like something is wrong in the earlier code, the intention here is that it should never be necessary to unmark_start_commit() here, and if it is, then it is a bug flagged by the assertion. > @@ -1751,7 +1753,8 @@ binlog_flush_cache(THD *thd, binlog_cache_mngr *cache_mngr, > > if ((using_stmt && !cache_mngr->stmt_cache.empty()) || > (using_trx && !cache_mngr->trx_cache.empty()) || > - thd->transaction->xid_state.is_explicit_XA()) > + (thd->transaction->xid_state.is_explicit_XA() || > + (thd->rgi_slave && thd->rgi_slave->is_async_xac))) Why is this extra condition thd->rgi_slave->is_async_xac necessary? There are many of these conditions spread around the code in the patch. I think perhaps it is because the transaction is somehow not "connected" to the THD? Because it is in the XID cache? But the situation must be similar on the master, if the XA COMMIT happens in a different transaction from XA PREPARE. So this should be done the same way on the slave, so that the XA transaction gets connected to the THD of the worker thread processing it, and so that these extra rgi_slave->is_async_xac conditions are not needed. It is very fragile to have such conditions spread around the code, it will be impossible to avoid bugs due to forgetting such extra condition on one place or another. > + /* > + While xid_state.get_xid() is a robust method to access `xid` > + it can't be used on slave by the asynchronously running XA-"complete". > + In the latter case thd->lex->xid is safely accessible. > + */ > + buflen= serialize_with_xid(is_async_xac? thd->lex->xid : > + thd->transaction->xid_state.get_xid(), Same here, it should be possible to use the same xid_state in the slave thread as on the master, so we avoid special cases all over the code for the slave threads. > +static bool acquire_xid(THD *thd) > +{ > + bool rc= false; > + > + if (thd->rgi_slave && thd->rgi_slave->is_async_xac && > + thd->rgi_slave->gtid_ev_flags2 & Gtid_log_event::FL_COMPLETED_XA) > + { > + XID_STATE &xid_state= thd->transaction->xid_state; This function actually seems to do part of "XA transaction gets connected to the THD". But it's called from binlog_commit_by_xid() and binlog_rollback_by_xid(), which means it has to have yet another of these special-condition checks for thd->rgi_slave->is_async_xac etc. Instead, call this from the code that applies the "XA COMMIT" event (eg. in log_event_server.cc), and make sure it fully connects the XA transaction to the the THD. Then you get rid of the checks for is_async_xac in a lot of places, and also avoid polluting with replication details the code than runs the original transaction on the master. > + rc= binlog_commit(thd, TRUE, FALSE); > + thd->ha_data[binlog_hton->slot].ha_info[1].reset(); > + } > + if (!rc) > + { > + rc= acquire_xid(thd); > + } > + if (thd->is_current_stmt_binlog_disabled()) > + { > + thd->wakeup_subsequent_commits(rc); > + } So IIUC, here we first binlog the XA COMMIT and then wakeup_subsequent_commits(). But the engine commit only happens afterwards, in ha_commit_or_rollback_by_xid. Why wakeup_subsequent_commits() here? It seems too early, the transaction is not yet committed in the engine, how do you avoid that the commits in the engine will happen in the wrong order? And then a mariabackup might take a backup with T2 committed but T1 not and not have a valid replication position to provision a slave. And also, is this skipping the binlog transaction coordinator and two-phase commit with the engines? And in particular, commit_ordered()? Then again, it seems we have the problem that the engine can commit in the opposite order from the binlog. I wonder if this will also affect crash recovery if we crash with different commit order in binlog and engine? So I think wakeup_subsequent_commits() here is wrong, should be removed. > @@ -2185,7 +2293,9 @@ int binlog_commit(THD *thd, bool all, bool ro_1pc) > } > > if (cache_mngr->trx_cache.empty() && > - (thd->transaction->xid_state.get_state_code() != XA_PREPARED || > + ((thd->transaction->xid_state.get_state_code() != XA_PREPARED && > + !(thd->rgi_slave && thd->rgi_slave->is_parallel_exec && > + thd->lex->sql_command == SQLCOM_XA_COMMIT)) || As explained above, this condition (and the similar in binlog_rollback() should be avoided. > @@ -10510,13 +10624,20 @@ int TC_LOG_BINLOG::unlog_xa_prepare(THD *thd, bool all) > > binlog_cache_mngr *cache_mngr= thd->binlog_setup_trx_data(); > int cookie= 0; > + int rc= 0; > + > + if (thd->rgi_slave && thd->is_current_stmt_binlog_disabled()) > + { > + rc= thd->wait_for_prior_commit(); > + if (rc == 0) > + thd->wakeup_subsequent_commits(rc); > + return rc; > + } Why is this necessary? If it is really necessary, then put it in the code that applies the XA PREPARE event, not as a random special case in generic code. > - bool rc= false; > - > > rc= write_empty_xa_prepare(thd, cache_mngr); // normally gains need_unlog > > static bool write_empty_xa_prepare(THD *thd, binlog_cache_mngr *cache_mngr) > { > return binlog_commit_flush_xa_prepare(thd, true, cache_mngr); > } Smaller point: since you're fixing the type of `rc` to be int (which is correct), also fix the return type of write_empty_xa_prepare() to be int and not bool - as binlog_commit_flush_xa_prepare() as well as unlog_xa_prepare() return int, not bool. > @@ -3314,16 +3314,22 @@ Gtid_log_event::Gtid_log_event(THD *thd_arg, uint64 seq_no_arg, > XID_STATE &xid_state= thd->transaction->xid_state; > if (is_transactional) > { > - if (xid_state.is_explicit_XA() && > - (thd->lex->sql_command == SQLCOM_XA_PREPARE || > - xid_state.get_state_code() == XA_PREPARED)) > + bool is_async_xac= false; > + if ((xid_state.is_explicit_XA() && > + (thd->lex->sql_command == SQLCOM_XA_PREPARE || > + xid_state.get_state_code() == XA_PREPARED)) || > + (is_async_xac= (thd->rgi_slave && thd->rgi_slave->is_async_xac))) Again, this is the generic GTID event contructor, it shouldn't need this kind of logic. It's very strange that a simple class constructor returns something different depending on which thread it runs in! Hopefully this is no longer necessary after XA transaction gets properly connected to the THD. But else, it should be handled by passing in suitable flag from the caller, or by the caller setting the required modifications after construction. > +using std::max; Please don't. Using std::max() explicit is not long and makes it explicit what `max` implementation is used. > @@ -760,7 +762,8 @@ convert_kill_to_deadlock_error(rpl_group_info *rgi) > return; > err_code= thd->get_stmt_da()->sql_errno(); > if ((rgi->speculation == rpl_group_info::SPECULATE_OPTIMISTIC && > - err_code != ER_PRIOR_COMMIT_FAILED) || > + (err_code != ER_PRIOR_COMMIT_FAILED && > + err_code != ER_XAER_NOTA)) || Why? What is special about the ER_XAER_NOTA error? Is it really safe to speculatively run the XA COMMIT in parallel, and then retry it for any other error than ER_XAER_NOTA ? > @@ -2467,6 +2463,7 @@ free_rpl_parallel_entry(void *element) > } > mysql_cond_destroy(&e->COND_parallel_entry); > mysql_mutex_destroy(&e->LOCK_parallel_entry); > + e->concurrent_xaps_window.~Dynamic_array(); > my_free(e); No, let's not do this, calling a destructor explicitly in a POD managed by my_malloc() / my_free(). Why not just make concurrent_xaps_window a pointer to the Dynamic_array and manage with new / delete? Then you can also allocate it lazily, so it will not need to be allocated at all for the 99.9xx% of users who are not replicating user XA transactions. The size of the sliding window is fixed anyway (until slave_parallel_threads is changed), so why use Dynamic_array at all, why not just allocate a plain array of the right size? > +template <> > +struct std::hash<XID> > +{ > + std::size_t operator()(const XID& xid) const > + { > + return my_hash_sort(&my_charset_bin, xid.key(), xid.key_length()); I don't understand, what is the purpose of introducing this function object? Why not just call my_hash_sort() directly (seems it's only used in one place) and avoid this code-obfuscation? > + Dynamic_array<std::pair<std::size_t, uint32>> concurrent_xaps_window; This is dangerous as Dynamic_array has a destructor but rpl_parallel_entry is a POD where we don't call any constructor/destructor. Should instead use a pointer to an object constructed with new (or not use Dynamic_array at all as suggested above). > + /* > + When true indicates that the user xa transaction is going to > + complete (with COMMIT or ROLLBACK) by the worker thread, > + *while* another worker is still preparing it. Once the latter is done > + the xid will be acquired and the flag gets reset. > + */ > + bool is_async_xac; > + I don't understand this comment. Surely it is not possible to *complete*, with XA COMMIT, a transaction before the corresponding XA PREPARE has completed (possibly in another worker thread)? Do you mean that it is possible for a worker thread W2 to *start* applying the XA COMMIT speculatively, before another worker W1 has completed the XA PREPARE? But I think this is not necessary, see below the discussion that the XA COMMIT worker can just do a wait_for_prior_commit() on the event group of its XA PREPARE. Then this should not be necessary. > @@ -137,7 +137,7 @@ template <class Elem> class Dynamic_array > */ > Elem& at(size_t idx) > { > - DBUG_ASSERT(idx < array.elements); > + DBUG_ASSERT(idx < max_size()); > return *(((Elem*)array.buffer) + idx); No, this cannot possibly be right. This is a patch about XA replication, why would it suddenly allow accessing deleted elements of any Dynamic_array used in the server?!? Surely you can allocate the elements you need for your sliding window. In fact, since the window is fixed size anyway, why use Dynamic_array at all? @@ -2367,6 +2367,15 @@ struct wait_for_commit event group is fully done. */ bool wakeup_blocked; + /* + The condition variable servers as a part of facilities to handle various + commit time additional dependency between groups of replication events, e.g + XA-Prepare -> XA-Commit, or XA-Prepare -> XA-Prepare all with the same xid. + */ + mysql_cond_t COND_wait_commit_dep; No, this doesn't belong in struct wait_for_commit. wait_for_commit is a low-level mechanism for ordering commits, it should not need any knowledge even of replication, certainly not of user-XA replication. The COND_wait_commit_dep isn't even used in any function related to wait_for_commit or even sql_class.cc, it's only used in sql/xa.cc. [But as discussed at the start, we can use the existing wait_for_prior_commit instead of introducing this new mechanism just for replicating user XA.] > -static XID_cache_element *xid_cache_search(THD *thd, XID *xid) > +XID_cache_element *xid_cache_search(THD *thd, XID *xid) Why remove the `static`? The function is not used outside of sql/xa.cc > @@ -254,16 +259,221 @@ static XID_cache_element *xid_cache_search(THD *thd, XID *xid) > + if (thd->rgi_slave && thd->rgi_slave->is_parallel_exec) > + { > + DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_COMMIT || > + thd->lex->sql_command == SQLCOM_XA_ROLLBACK); > + thd->rgi_slave->is_async_xac= true; Here again we have some logic that relates to replication being inserted somewhat randomly in low-level code, requiring this thd->rgi_slave->is_parallel_exec condition. Instead of doing this, the caller higher up should handle this depending on the return of xid_cache_search(). This way, code paths that are not related to replication don't get affected by the replication-specific logic. > +bool xid_cache_insert_maybe_wait(THD* thd) > +{ This new wait mechanism just for parallel replication of user XA seems very complex. Spin loop, and extra locking and condition variables and locking on a lock-free hash. As explained at the start, let's instead avoid introducing a new mechanism and use the existing wait_for_prior_commit, which is central to parallel replication and highly optimized for this. IIUC, the real problem here is that after wakeup_subsequent_commits() of XA COMMIT, the XID is still in the xid_hash. This is because of this code in trans_xa_commit(): ha_commit_or_rollback_by_xid(thd->lex->xid, !res); if (!res && thd->is_error()) { // hton completion error retains xs/xid in the cache, // unless there had been already one as reflected by `res`. res= true; goto _end_external_xid; } xid_cache_delete(thd, xs); xid_deleted= true; ha_commit_or_rollback_by_xid() ends up in queue_for_group_commit() which will do the wakeup_subsequent_commits. But the xid_cache_delete() happens just after that. This is too late. The wakeup_subsequent_commits() must not occur until after the transaction is fully committed in memory. So instead, move the xid_cache_delete() call so it happens inside the group commit code. Then when wakeup_subsequent_commits() is done after XA COMMIT, the XID entry will be gone. It would go here in MYSQL_BIN_LOG::trx_group_commit_leader(): if (current->cache_mngr->using_xa && likely(!current->error) && DBUG_EVALUATE_IF("skip_commit_ordered", 0, 1)) { mysql_mutex_lock(&current->thd->LOCK_thd_data); run_commit_ordered(current->thd, current->all); mysql_mutex_unlock(&current->thd->LOCK_thd_data); } current->thd->wakeup_subsequent_commits(current->error); I noticed that commit_ordered is not done for user XA transactions in replication. The direct reason for that seems to be that they do not use log_and_order(), which sets cache_mngr->using_xa. Can you explain why the XA PREPARE and XA COMMIT is not following the commit_ordered protocol, and what mechanism exists instead to ensure that everything still works correctly? For example that commit order in binlog and InnoDB will be the same, and that binlog recovery works? And finally, just for the record, I still do not agree with the way replication of user XA has been changed (aka MDEV-32020), and this patch does not fix the underlying problem. But changing it to use the wait_for_prior_commit() mechanism as explained at the start will improve the integration with the parallel replication code and be a step towards solving also some of the other underlying issues. - Kristian.

1 0

Locking thd->LOCK_thd_data during group commit
by Kristian Nielsen 21 Oct '23

21 Oct '23

Hi Serg, I came upon this commit by you: > commit 6b685ea7b0776430d45b095cb4be3ef0739a3c04 > Author: Sergei Golubchik <serg(a)mariadb.org> > Date: Wed Sep 28 18:55:15 2022 +0200 > > correctness assert > > thd_get_ha_data() can be used without a lock, but only from the > current thd thread, when calling from anoher thread it *must* > be protected by thd->LOCK_thd_data > > * fix group commit code to take thd->LOCK_thd_data > * remove innobase_close_connection() from the innodb background thread, > it's not needed after 87775402cd0c and was failing the assert with > current_thd==0 > @@ -8512,7 +8512,11 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) > ++num_commits; > if (current->cache_mngr->using_xa && likely(!current->error) && > DBUG_EVALUATE_IF("skip_commit_ordered", 0, 1)) > + { > + mysql_mutex_lock(&current->thd->LOCK_thd_data); > run_commit_ordered(current->thd, current->all); > + mysql_mutex_unlock(&current->thd->LOCK_thd_data); > + } This seems _really_ expensive :-( This code runs for every transaction commit in replication, and it runs under a global LOCK_commit_ordered which needs to be held for as short as possible. The commit message doesn't mention anything about what goes wrong during the group commit code. And the patch doesn't have any test case that shows the problem, it just adds an assertion that will trigger during the group commit. So what is the actual reason for this change? Note that at this point, the group commit leader thread _is_ effectively acting as if it was each of the participating threads, the other threads are blocked from running during commit_ordered. If it is just so that we can have this assertion, then that needs to be rolled back. We _really_ don't want to take/release N mutexes in release builds during LOCK_commit_ordered, just to have an assertion in debug builds. If there is some reason, then the correct fix could be to set current_thd for the duration of run_commit_ordered, which will satisfy the assertion and presumably the actual bug? Well, it depends _what_ the actual reason/bug is. - Kristian.

3 5

Re: Review of MDEV-31273: Eliminate Log_event::checksum_alg
by Kristian Nielsen 17 Oct '23

17 Oct '23

Hi Monty, Lots of good comments, thanks! Replies inline: Michael Widenius <michael.widenius(a)gmail.com> writes: > enum enum_binlog_checksum_alg > > In c++ we can drop the first 'enum' > You can fix that by doing: Ack, will fix. > event.select_checksum_alg() > > This is called a lot. Would it not be easier to store the checksum_alg > in the event when it is created and access the variable directly? > On the other hand, the number of calls to event.select_checksum_alg() may > not change as we will only create one Yes, this is called only once per event, when writing the event into the binlog or the event cache. So it seems unnecessary to use extra memory to store it. And we don't necessarily even have binlog_cache_data when constructing the event to pass to select_checksum_alg(), so I think it is correct to select the checksum algorithm to use only when we write the event. > @@ -1258,11 +1258,10 @@ Log_event* Log_event::read_log_event(const > uchar *buf, uint event_len, > > if (ev) > { > - ev->checksum_alg= alg; > #ifdef MYSQL_CLIENT > if (ev->checksum_alg != BINLOG_CHECKSUM_ALG_OFF && > ev->checksum_alg != BINLOG_CHECKSUM_ALG_UNDEF) > ev->crc= uint4korr(buf + (event_len)); > + ev->read_checksum_alg= alg; > > Why move the setting of read_checksum_alg inside MYQL_CLIENT? It is because with this patch, ev->checksum_alg is no longer needed in the server (the class member is removed). It is needed in the client (mysqlbinlog), so I added a new member read_checksum_alg, which is the algorithm that was used to read the event. This is needed so that mysqlbinlog can output the checksum (if any) in textual form. But it is not needed in the server > To ensure that this is not wrongly used, please add > #ifdef MYSQL_CLIENT > > here: > > class Log_event: > > #ifdef MYSQL_CLIENT > + enum enum_binlog_checksum_alg read_checksum_alg; > #endif Yes, agree, that is the intention - the read_checksum_alg is only present in the class in the client, not in the server (and we want to avoid using memory for it in the server). This code is already in the #else branch of #ifdef MYSQL_SERVER: #ifdef MYSQL_SERVER ... #else /* The checksum algorithm used (if any) when the event was read. */ enum enum_binlog_checksum_alg read_checksum_alg; I thought this would have the same effect as your suggestion with #ifdef MYSQL_CLIENT. It probably does, but it seems somewhat complex when MYSQL_CLIENT and MYSQL_SERVER are defined. Maybe they can even both be defined, though probably not where log_event.h is included? It's a bit confusing that we have both MYSQL_CLIENT and MYSQL_SERVER used for conditional compilation in log_event.h. But probably not something that should be cleaned up in this patch. Do you think I should still add the #ifdef MYSQL_CLIENT as you suggested (even though it should be redundant in the #else branch of #ifdef MYSQL_SERVER)? It can't hurt after all, even if we always have MYSQL_CLIENT == !MYSQL_SERVER. > in Format_description_log_event(const uchar > > We have: > > else > { > checksum_alg= BINLOG_CHECKSUM_ALG_UNDEF; > used_checksum_alg= BINLOG_CHECKSUM_ALG_OFF; > } > > The used_checksum_alg is not needed as you set it at function start. Sorry, I don't understand? At the start we set it to "undef": + used_checksum_alg= BINLOG_CHECKSUM_ALG_UNDEF; This value will remain if there's an error return; else the else branch will change it to "off". Did you miss that it's set differently at the start of the function and in this else { } branch? Or did I miss something? > sql/log_event.h > > enum enum_binlog_checksum_alg set_checksum_alg(enum > enum_binlog_checksum_alg alg) > As we are just copying and restoring checksum_length, why not just do > > Format_description_log_event::write(Log_event_writer *writer) > > org_checksum_length= checksum_length... > .... > checksum_length= org_checksum_length; > > instead of calling set_checksum_alg() again ? > (The last set we can even do undonditionally which saves us an if) Agree, that code looks silly, I don't know why I did it that way. I'll fix it, remove the function set_checksum_alg() and just do: org_checksum_length= writer->checksum_length; writer->checksum_length= BINLOG_CHECKSUM_LEN; ... writer->checksum_length= org_checksum_length; > log_event_server.cc > > enum enum_binlog_checksum_alg Log_event::select_checksum_alg() > > Much simpler, but we do not check if cache_type is correct > > DBUG_ASSERT(((get_type_code() != ROTATE_EVENT && > get_type_code() != STOP_EVENT) || > get_type_code() != FORMAT_DESCRIPTION_EVENT) || > cache_type == Log_event::EVENT_NO_CACHE); > > Do you think this is not needed? > (This could be added just before: > > return BINLOG_CHECKSUM_ALG_OFF; Agree, I was happy to remove a lot of complexity and the need for a number of asserts, but this assertion could be kept. I'll add it back. > Anyway, please remove the else after: > > if (cache_type == Log_event::EVENT_NO_CACHE) > return (enum_binlog_checksum_alg)binlog_checksum_options; Ack will do. > Format_description_log_event::write(Log_event_writer *writer) > > > uint8 checksum_byte= (uint8) (used_checksum_alg != BINLOG_CHECKSUM_ALG_UNDEF ? > used_checksum_alg : BINLOG_CHECKSUM_ALG_OFF); > > Is it possible the alg can be UNDEF? > You have several asserts in the code that this should not happen. Yeah, honestly, I'm not sure. It's one of the annoying things in the code that I did not get to fix, that we have both the UNDEF and the OFF state for the checksum algorithm. Format_description code is not performance critical (only once per binlog file), so I suggest I leave the extra check for UNDEF as defensive code, but add an assertion that it is not UNDEF and see if it passes the testsuite. Does that sound ok? > slave.cc > > @@ -1870,10 +1873,8 @@ static int get_master_version_and_clock(MYSQL* > mysql, Master_info* mi) > until it has received a new FD_m. > */ > mi->rli.relay_log.description_event_for_queue->checksum_alg= > mi->rli.relay_log.relay_log_checksum_alg; > > Not sure why this is removed. > > Are these guaranteed to be the same here? > If yes, why not add an assert for this? This patch removes Log_event::checksum_alg and instead adds a field Format_description_log_event::used_checksum_alg which gets set in the Format_description_log_event constructor. So this assignment - mi->rli.relay_log.description_event_for_queue->checksum_alg= - mi->rli.relay_log.relay_log_checksum_alg; is replaced by this code just above which serves the same purpose: - Format_description_log_event(4, mysql->server_version); + Format_description_log_event(4, mysql->server_version, + mi->rli.relay_log.relay_log_checksum_alg); So I think adding an assert DBUG_ASSERT(mi->rli.relay_log.description_event_for_queue->checksum_alg == mi->rli.relay_log.relay_log_checksum_alg); is somewhat redundant, but let me know if you still think it will be useful and I'll add it. - Kristian.

2 1

Re: [MariaDB commits] [PATCH 1/4] MDEV-31273: Replace Log_event::writer with function parameter
by Kristian Nielsen 16 Oct '23

16 Oct '23

Hi Monty, [Following up on review on IRC] Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > Replace Log_event::writer with just passing the writer object as a > function parameter to Log_event::write(). We wondered how the compiled code will look when passing the writer object as a parameter to Log_event::write() vs. temporarily storing it in the Log_event object. Interestingly, the code is actually "better" after my patch despite an extra parameter being passed along. This is because the compiler can cache the function parameter in a register across nested calls. While with a class member, the compiler cannot determine that the member will not be changed by the nested calls, so the value needs to be repeatedly re-loaded from memory. Some more details on this finding below. My reason for the change was probably more general, based on this code: > - ev->writer= this; > - int res= ev->write(); > - IF_DBUG(ev->writer= 0,); // writer must be set before every Log_event::write > + int res= ev->write(this); It's clear that the intention with this code is to pass the Log_event_writer object `this` to the ev->write() method, and it's a strange way to do it to temporarily set it as a member in the object rather than using a function parameter. Needing an IF_DBUG() to enforce/check that this is done correctly is a warning sign, and I was worried that as I did further changes I would introduce subtle bugs this way. It is interesting that this makes the code clearer not only for the developer (me), but also for the compiler in this case. Function parameter passing is quite well optimized on amd64, but it was still good to see this in detail in this case. And as you predicted we learned something from it! As we discussed, I will extend the commit message to explain these reasons better. Thanks for the suggestion; it's always good in the reviews to question the reason for changes like this. To check the compiled code in detail, I did a release build (BUILD/compile-pentium64-max) with and without this patch, and disassembled the User_var_log_event::write() function, which has a number of calls now passing the writer object as a function parameter: > - return write_header(event_length) || > - write_data(buf, sizeof(buf)) || > - write_data(name, name_len) || > - write_data(buf1, buf1_length) || > - write_data(pos, val_len) || > - write_data(&flags, unsigned_len) || > - write_footer(); > + return write_header(writer, event_length) || > + write_data(writer, buf, sizeof(buf)) || > + write_data(writer, name, name_len) || > + write_data(writer, buf1, buf1_length) || > + write_data(writer, pos, val_len) || > + write_data(writer, &flags, unsigned_len) || > + write_footer(writer); It looks like this would need to pass the `writer` argument many times to called functions. However, write_data() and write_footer() are actually inline functions: bool write_data(const uchar *buf, size_t data_length) { return writer->write_data(buf, data_length); } bool write_footer() { return writer->write_footer(); } So it's actually only write_header() where we now pass an extra parameter. For write_data() and write_footer() it was already being passed as the (implicit) `this` parameter. With the patch, the compiler saves the `writer` parameter in register r12 and uses it for each call: movq %rsi, %r12 ... movq %r12, %rsi # `writer` pointer movq %rbx, %rdi # `this` pointer call _ZN9Log_event12write_headerEP16Log_event_writerm@PLT ... movl $4, %edx # `data_length` parameter movq %r12, %rdi # `writer` pointer call _ZN16Log_event_writer10write_dataEPKhm@PLT ... leaq 137(%rbx), %rsi # `buf` parameter movq %r14, %rdx # `data_length` parameter movq %r12, %rdi # `writer` pointer call _ZN16Log_event_writer10write_dataEPKhm@PLT ... movq %r12, %rdi # `writer` pointer call _ZN16Log_event_writer12write_footerEv@PLT Before the patch, the code for User_var_log_event::write() loads this->writer for each call except write_header(): movq %rbx, %rdi # Here we don't pass `writer`, write_header() loads it from Log_event::writer call _ZN9Log_event12write_headerEm@PLT ... movq 88(%rbx), %rdi # this->writer leaq -126(%rbp), %rsi # `buf` parameter movl $4, %edx # `data_length` parameter call _ZN16Log_event_writer10write_dataEPKhm@PLT ... movq 88(%rbx), %rdi # this->writer leaq 153(%rbx), %rsi # `buf` parameter movq %r13, %rdx # `data_length` parameter call _ZN16Log_event_writer10write_dataEPKhm@PLT ... movq 88(%rbx), %rdi # this->writer call _ZN16Log_event_writer12write_footerEv@PLT I think the reason for the repeated loads of this->writer from 88(%rbx) is because the compiler cannot be sure that this->writer will not be modified by calls to write_data(). So it needs to reload the pointer each time. With the patch the pointer is a function parameter which cannot be modified by nested calls, so it can be saved in a register. Thus, we save some memory loads with my patch for each call of write_data() / write_footer(), which should be a bit faster (though perhaps not noticeably so). In general, passing parameters to functions is cheap (on amd64), as they are passed in registers, and register-to-register moves are very fast on high-performance CPUs, while memory loads have a relative high latency, even when they hit the cache. And like we see here, using local variables helps the compiler do alias analysis and cache values in registers, while this is a lot harder for the compiler to do for class members. - Kristian.

1 0

Re: d4c173ae6df: MDEV-27595 Backport SQL service, introduced by MDEV-19275.
by Sergei Golubchik 16 Oct '23

16 Oct '23

Hi, Alexey, On Oct 16, Alexey Botchkov wrote: > revision-id: d4c173ae6df (mariadb-10.4.30-117-gd4c173ae6df) > parent(s): 837b777bfd7 > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2023-09-05 13:18:12 +0400 > message: > > MDEV-27595 Backport SQL service, introduced by MDEV-19275. > > Fixes to make SQL SERVICE working. > > Conflicts: > storage/spider/spd_table.cc ^^^ better remove these two lines. normally git removes them automatically, and there's no spd_table.cc in this commit anyway. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 837b777bfd7: MDEV-27595 Backport SQL service, introduced by MDEV-19275.
by Sergei Golubchik 16 Oct '23

16 Oct '23

Hi, Alexey, On Oct 16, Alexey Botchkov wrote: > revision-id: 837b777bfd7 (mariadb-10.4.30-116-g837b777bfd7) > parent(s): a49b9314c16 > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2023-09-05 13:18:12 +0400 > message: > > MDEV-27595 Backport SQL service, introduced by MDEV-19275. > > The SQL SERVICE backported into the 10.4. > > diff --git a/plugin/test_sql_service/CMakeLists.txt b/plugin/test_sql_service/CMakeLists.txt > new file mode 100644 > index 00000000000..615508bdc4e > --- /dev/null > +++ b/plugin/test_sql_service/CMakeLists.txt > @@ -0,0 +1,18 @@ > +# Copyright (C) 2013 Alexey Botchkov and SkySQL Ab > +# > +# This program is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; version 2 of the License. > +# > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program; if not, write to the Free Software > +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA > + > +SET(SOURCES test_sql_service.c) > + > +MYSQL_ADD_PLUGIN(test_sql_service ${SOURCES} MODULE_ONLY) COMPONENT Test > diff --git a/plugin/test_sql_service/COPYING b/plugin/test_sql_service/COPYING > new file mode 100644 > index 00000000000..6e475df5526 > --- /dev/null > +++ b/plugin/test_sql_service/COPYING > @@ -0,0 +1,339 @@ > + GNU GENERAL PUBLIC LICENSE No need to, we already have one in the CMAKE_SOURCE_DIR, why do you want to copy it just for one plugin? Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: About Release Model
by Kristian Nielsen 16 Oct '23

16 Oct '23

Hi Sergei, Sergei Golubchik <serg(a)mariadb.org> writes: > You've said that it's unclear how the release model works, when we > release what and where a feature can go. > > I've documented the current release model in the knowledge base, > https://mariadb.com/kb/en/mariadb-release-model/ > would you like to take a look and see if it answers your questions? This is a fine overview page, and if this info was not previously available, then it's surely good to have it documented. However, it doesn't help explain how to meaningfully participate as a core developer in the releases. Where does the discussion happen about the progress and planning of the release, and addressing problems? How to partipate in those discussions? How to push a patch into a preview release, or a release, how to make a release? Etc. My assumption is that this is all restricted to employees? But even that isn't explained on the page. - Kristian.

2 3

Question about lacking git commit message in dc9498b
by Otto Kekäläinen 12 Oct '23

12 Oct '23

Hi Sergei and others! While reviewing changes in MariaDB 10.11.5 I was wondering about this commit: https://github.com/mariadb/server/commit/dc9498beb6f79f08edd02fb7f7153e5150… It has only the default message from issuing 'git revert' and a human written explanation *why* the revert was done is missing. AFAIK the reverted commit https://github.com/MariaDB/server/commit/1d0e3d80d84c2b375a444cdbb41c28afd7… was correct and good - so why was it reverted?

2 1

Re: 6cb526bd363: MDEV-32123: require_secure_transport doesn't allow TCP connections
by Sergei Golubchik 10 Oct '23

10 Oct '23

Hi, Dmitry, Looks pretty good! Basically no comments, some minor nitpicking below, feel free to ignore. On Oct 10, Dmitry Shulga wrote: > revision-id: 6cb526bd363 (mariadb-11.2.1-9-g6cb526bd363) > parent(s): 872ed5342d8 > author: Dmitry Shulga > committer: Dmitry Shulga > timestamp: 2023-10-03 18:07:35 +0700 > message: > > MDEV-32123: require_secure_transport doesn't allow TCP connections > > In case the option require_secure_transport is on the user can't > establish a secure ssl connection over TCP protocol. Inability to set up > a ssl session over TCP was caused by the fact that a type of client's > connection was checked before ssl handshake performed (ssl handshake > happens at the function acl_authenticate()). At that moment vio type has > the value VIO_TYPE_TCPIP for client connection that uses TCP transport. > In result, checking for allowable vio type for fails despite the fact > that SSL session being established. To fix the issue move checking of > vio type for allowable values inside the function > parse_client_handshake_packet() > right after client's capabilities discovered that SSL is not requested > by the client. > > diff --git a/mysql-test/main/require_secure_transport.result b/mysql-test/main/require_secure_transport.result > index 94474d0fa0e..8fa273cbae4 100644 > --- a/mysql-test/main/require_secure_transport.result > +++ b/mysql-test/main/require_secure_transport.result > @@ -1,8 +1,13 @@ > -CREATE TABLE t1 (t int(1)); > SET GLOBAL require_secure_transport=ON; > -ERROR HY000: Connections using insecure transport are prohibited while --require_secure_transport=ON. > +ERROR 08004: Connections using insecure transport are prohibited while --require_secure_transport=ON. I wonder why sqlstate has changed now, as far as I remember it was the previous patch that changed it. Was the test skipped until now? > +SELECT (VARIABLE_VALUE <> '') AS have_ssl FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='Ssl_cipher'; > +have_ssl > +1 > +disconnect with_ssl; > connection default; > SET GLOBAL require_secure_transport=OFF; > +SELECT (VARIABLE_VALUE <> '') AS have_ssl FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='Ssl_cipher'; > +have_ssl > +0 > disconnect without_ssl; > connection default; > -DROP TABLE t1; > diff --git a/mysql-test/main/require_secure_transport.test b/mysql-test/main/require_secure_transport.test > index e238e732423..f3da73a5167 100644 > --- a/mysql-test/main/require_secure_transport.test > +++ b/mysql-test/main/require_secure_transport.test > @@ -1,15 +1,19 @@ > -- source include/have_ssl_communication.inc > -CREATE TABLE t1 (t int(1)); > SET GLOBAL require_secure_transport=ON; > --disable_query_log > --error ER_SECURE_TRANSPORT_REQUIRED > connect without_ssl,localhost,root,,,,,TCP NOSSL; > + > +connect with_ssl,localhost,root,,,,,TCP SSL; > --enable_query_log I'd suggest to remove disable/enable_query_log and let connect commmand show in the result file. It'll make them easier to read. You'll need replace_result before the failing connect to replace the port away. > +SELECT (VARIABLE_VALUE <> '') AS have_ssl FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='Ssl_cipher'; > +disconnect with_ssl; > + > connection default; > SET GLOBAL require_secure_transport=OFF; > --disable_query_log > connect without_ssl,localhost,root,,,,,TCP NOSSL; > --enable_query_log > +SELECT (VARIABLE_VALUE <> '') AS have_ssl FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='Ssl_cipher'; > disconnect without_ssl; > connection default; > -DROP TABLE t1; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: 217a7a7ceff: online alter: rework savepoints
by Sergei Golubchik 09 Oct '23

09 Oct '23

Hi, Nikita, On Sep 29, Nikita Malyavin wrote: > revision-id: 217a7a7ceff (mariadb-11.2.1-10-g217a7a7ceff) > parent(s): ad6a305d84d > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-09-23 11:46:36 +0400 > message: > > online alter: rework savepoints > > Use standard handlerton functions for savepoint add/rollback. > > To identify the savepoint, the pointer passed is used. > > Every table that has online alter in progress maintains a list of > savepoints independently. > > Also this removes setting a value to a global variable savepoint_alloc_size > without any protection, which was a race condition bug. > > --- a/sql/online_alter.cc > +++ b/sql/online_alter.cc > @@ -6,7 +6,19 @@ > > static handlerton *online_alter_hton; > > -using Online_alter_cache_list= ilist<online_alter_cache_data>; > +using sv_id_t= void *; typedef, please > + > +struct Online_alter_cache_list: ilist<online_alter_cache_data> > +{ > + sv_id_t savepoint_id= 0; > +}; > @@ -281,13 +308,11 @@ static int online_alter_log_init(void *p) > { > online_alter_hton= (handlerton *)p; > online_alter_hton->db_type= DB_TYPE_ONLINE_ALTER; > - online_alter_hton->savepoint_offset= sizeof(my_off_t); > + online_alter_hton->savepoint_offset= 1; you can even use 0, I guess. Otherwise it'll still be 8, as the server will need to align allocation, I suppose. But I suggest to try to swap the implementation. Make online_alter_hton->savepoint_offset=sizeof(void*); and in savepoint_set create a list or an array of all offsets and store it in the savepoint. Something like this: struct { uint savepoints; my_off_t log_prev_pos[1]; } oasv; int online_alter_savepoint_set(handlerton *hton, THD *thd, oasv **sv) { uint count= 0, i=0; for (auto &cache: cache_list) count++; DBUG_ASSERT(count); *sv= malloc(sizeof(oasv) + sizeof(my_off_t)*(count - 1)); (*sv)->savepoints= count; for (auto &cache: cache_list) (*sv)->log_prev_pos[i]= cache.get_byte_position(); } and so on. You just need to make sure you always use push_back and not push_front for adding caches to the cache_list. instead of a structure with a counter you can use an array with MY_OFF_T_UNDEF (not 0!) at the end. > online_alter_hton->close_connection= online_alter_close_connection; > > - online_alter_hton->savepoint_set= // Done by online_alter_savepoint_set > - [](handlerton *, THD *, void *){ return 0; }; > - online_alter_hton->savepoint_rollback= // Done by online_alter_savepoint_rollback > - [](handlerton *, THD *, void *){ return 0; }; > + online_alter_hton->savepoint_set= online_alter_savepoint_set; > + online_alter_hton->savepoint_rollback= online_alter_savepoint_rollback; > online_alter_hton->savepoint_rollback_can_release_mdl= > [](handlerton *hton, THD *thd){ return true; }; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 2

Re: ad6a305d84d: online alter: use thd->ha_data to store cache_list
by Sergei Golubchik 09 Oct '23

09 Oct '23

Hi, Nikita, On Sep 29, Nikita Malyavin wrote: > revision-id: ad6a305d84d (mariadb-11.2.1-9-gad6a305d84d) > parent(s): 132c56f1325 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-09-23 11:46:36 +0400 > message: > > online alter: use thd->ha_data to store cache_list nice > diff --git a/sql/online_alter.cc b/sql/online_alter.cc > index 3255fc06c3e..5ad2156cd21 100644 > --- a/sql/online_alter.cc > +++ b/sql/online_alter.cc > @@ -6,6 +6,7 @@ > > static handlerton *online_alter_hton; > > +using Online_alter_cache_list= ilist<online_alter_cache_data>; please, use typedef for that > class online_alter_cache_data: public Sql_alloc, public ilist_node<>, > public binlog_cache_data > @@ -246,7 +268,11 @@ int online_alter_savepoint_rollback(THD *thd, LEX_CSTRING name) > > static int online_alter_close_connection(handlerton *hton, THD *thd) > { > - DBUG_ASSERT(thd->online_alter_cache_list.empty()); > + Ha_data &ha= thd->ha_data[online_alter_hton->slot]; > + auto *cache_list= (Online_alter_cache_list*)ha.ha_ptr; why not thd_get_ha_data()? and don't forget thd_set_ha_data(thd, online_alter_hton, 0); THD can be reused, I suppose, so let's play safe. > + > + DBUG_ASSERT(!cache_list || cache_list->empty()); > + delete cache_list; > return 0; > } Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

2 3

Re: MariaDB 10.11.5 mtr --big results from amd64/arm64/armhf/ppc64el/riscv64/s390x
by Otto Kekäläinen 09 Oct '23

09 Oct '23

Hi! > > I just wanted to share the summary of mariadb-test-run with --big from > > all available architectures on Launchpad > > (https://launchpad.net/~otto/+archive/ubuntu/mariadb/+builds?build_text=&bui…) > > I have tried to fix type_test_double and maria.maria-purge in 10.6. > With 'tried', I mean that I have not been able to repeat the test, but > I have a clue what is going on. > I will push later today. > Any chance you can try to run bb-10.6-monty on launchpad to test my fixes? I cherry-picked these two commits: https://github.com/MariaDB/server/commit/8e093d7597664290b0b5e0eb401b3a180f… https://github.com/MariaDB/server/commit/630cb99823f8e83545ffa16d60c9ac75cd… .. and pushed to https://launchpad.net/~otto/+archive/ubuntu/mariadb/+builds?build_text=&bui… and indeed both tests are now fixed: maria.maria-purge w1 [ pass ] 2134 type_test.type_test_double w1 [ pass ] 59 (from https://launchpadlibrarian.net/691041067/buildlog_ubuntu-mantic-amd64.maria…) Thanks Monty! Last failing that remains after Monty's and Daniel's fixes is: - unit.conc_connection: CONC-640/MDEV-30885

1 0

MariaDB 10.11.5 mtr --big results from amd64/arm64/armhf/ppc64el/riscv64/s390x
by Otto Kekäläinen 07 Oct '23

07 Oct '23

Hi! I just wanted to share the summary of mariadb-test-run with --big from all available architectures on Launchpad (https://launchpad.net/~otto/+archive/ubuntu/mariadb/+builds?build_text=3D&= build_state=3Dall): s390x Completed: Failed 13/5644 tests, 99.77% were successful. Failing test(s): innodb.row_size_error_log_warnings_3 disks.disks_notembedded type_test.type_test_double maria.maria-purge unit.conc_connection main.func_json_notembedded riscv64 Completed: Failed 6/6289 tests, 99.90% were successful. Failing test(s): disks.disks_notembedded type_test.type_test_double unit.conc_connection ppc64el Completed: Failed 9/6291 tests, 99.86% were successful. Failing test(s): disks.disks_notembedded type_test.type_test_double maria.maria-purge unit.conc_connection armhf Too many failed: Failed 19/1767 tests, 98.92% were successful. Failing test(s): federated.federatedx_versioning spider/bg.basic_sql spider/bg.spider_fixes_part spider.basic_sql spider.spider_fixes_part versioning.online versioning.partition period.versioning arm64 completed: Failed 9/6291 tests, 99.86% were successful. Failing test(s): disks.disks_notembedded type_test.type_test_double maria.maria-purge unit.conc_connection amd64 Completed: Failed 9/6291 tests, 99.86% were successful. Failing test(s): disks.disks_notembedded type_test.type_test_double maria.maria-purge unit.conc_connection As you can see, these seem to affect all architectures: - maria.maria-purge: MDEV-31349, easily repeatable failure on all architectures on Debian CI, Salsa CI and Launchpad - type_test.type_test_double: MDEV-22243 wasn't actually fixed yet, and failure visible on all architectures and platforms tested - unit.conc_connection: CONC-640/MDEV-30885 visible on all architectures tested - disk.disk{_notembedded}: MDEV-21587 Of these, Daniel Black is currently fixing the item 4 in the list (disk.disk) at https://github.com/MariaDB/server/pull/2777. For the items 1-3 help would be needed. The armhf situation does not map into a single MDEV. Do we have any Raspberry Pi enthusiasts here who would like to run the mariadb-test-run suite on their Pi and try to fix some of the failures? Overall it is nice to see nearly 6k tests pass on so many architectures! The Debian builds at https://buildd.debian.org/status/package.php?mariadb only run post-build the main suite (~1060 tests), and some architectures don't run any tests (e.g. m68k), just the build, but still nice to see the architecture support situation being very good at the moment for 10.11.5 across official and in-official Debian architectures as well :) Many thanks in particular to Daniel, Marko, Sergei, Tuukka, Faustin and Andrew for helping to hunt for fixes to the MariaDB code and tests to have wide architecture support! - Otto

1 0

Re: 132c56f1325: online alter: extract the source to a separate file
by Sergei Golubchik 29 Sep '23

29 Sep '23

Hi, Nikita, On Sep 29, Nikita Malyavin wrote: > revision-id: 132c56f1325 (mariadb-11.2.1-8-g132c56f1325) > parent(s): a71b13433f9 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-09-23 11:46:36 +0400 > message: > > online alter: extract the source to a separate file > > Move all the functions dedicated to online alter to a newly created > online_alter.cc. Copyright/GPL header in both new files, please. Also one comment below. > With that, make many functions static and simplify the static functions > naming. > > Also, rename binlog_log_row_online_alter -> online_alter_log_row. > > diff --git a/sql/log_event.h b/sql/log_event.h > index 848ad0cead9..35142e034f3 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -729,19 +729,6 @@ enum Log_event_type > }; > > > -/* > - Bit flags for what has been writing to cache. Used to > - discard logs with table map events but not row events and > - nothing else important. This is stored by cache. > -*/ > - > -enum enum_logged_status > -{ > - LOGGED_TABLE_MAP= 1, > - LOGGED_ROW_EVENT= 2, > - LOGGED_NO_DATA= 4, > - LOGGED_CRITICAL= 8 > -}; > > static inline bool LOG_EVENT_IS_QUERY(enum Log_event_type type) > { > @@ -979,6 +966,19 @@ typedef struct st_print_event_info > } PRINT_EVENT_INFO; > #endif > > +/** > + Bit flags for what has been writing to cache. Used to > + discard logs with table map events but not row events and > + nothing else important. This is stored by cache. > +*/ > +enum enum_logged_status > +{ > + LOGGED_TABLE_MAP= 1, > + LOGGED_ROW_EVENT= 2, > + LOGGED_NO_DATA= 4, > + LOGGED_CRITICAL= 8 > +}; > + why did you move it? looks unintentional. Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: a71b13433f9: MDEV-32126 Assertion fails upon online ALTER and binary log enabled
by Sergei Golubchik 29 Sep '23

29 Sep '23

Hi, Nikita, On Sep 29, Nikita Malyavin wrote: > revision-id: a71b13433f9 (mariadb-11.2.1-7-ga71b13433f9) > parent(s): 883e7e5f7e7 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-09-22 17:55:14 +0400 > message: > > MDEV-32126 Assertion fails upon online ALTER and binary log enabled > > Assertion `!writer.checksum_len || writer.remains == 0' fails upon > concurrent online ALTER and transactions with failing statements and binary > log enabled. > Also another assertion, `pos != (~(my_off_t) 0)', fails in my_seek, upon > reinit_io_cache, on a simplified test. This means that IO_CACHE wasn't > properly initialized, or had an error before. > > The overall problem is a deep interference with the effect of an installed > binlog_hton. As I wrote in Jira, please, provide a better explanation here. I had to debug the issue myself and as far as I can see, the problem happens because you register binlog_hton for a transaction, but doesn't prepare the cache manager for it. This can be fixed simply with @@ -2279,6 +2279,12 @@ int binlog_log_row_online_alter(TABLE* table, const ucha> trans_register_ha(thd, false, binlog_hton, 0); if (thd->in_multi_stmt_transaction_mode()) trans_register_ha(thd, true, binlog_hton, 0); + if (mysql_bin_log.is_open()) + { + binlog_cache_mngr *cache_mngr= thd->binlog_get_cache_mngr(); + if (cache_mngr && cache_mngr->trx_cache.get_prev_position() == MY_OFF_T_UNDEF) + thd->binlog_set_stmt_begin(); + } } which solves all cases from the bug report. Don't get me wrong, I don't suggest this fix, it's kind of hackish, and I liked how other pieces of the puzzle fell into a place after a separate handlerton, like savepoints and online_alter_cache_list. I'm just saying that "magical interference with the binlog_hton" doesn't explain the problem. At all. > Solution: > Extract online alter operations into a separate handlerton. > > --- a/sql/CMakeLists.txt > +++ b/sql/CMakeLists.txt > @@ -218,6 +218,8 @@ MYSQL_ADD_PLUGIN(partition ha_partition.cc STORAGE_ENGINE DEFAULT STATIC_ONLY > RECOMPILE_FOR_EMBEDDED) > MYSQL_ADD_PLUGIN(sql_sequence ha_sequence.cc STORAGE_ENGINE MANDATORY STATIC_ONLY > RECOMPILE_FOR_EMBEDDED) > +MYSQL_ADD_PLUGIN(online_alter_log log.cc STORAGE_ENGINE MANDATORY STATIC_ONLY > +NOT_EMBEDDED) better add it next to builtin_maria_binlog_plugin, instead of via MYSQL_ADD_PLUGIN. > ADD_LIBRARY(sql STATIC ${SQL_SOURCE}) > MAYBE_DISABLE_IPO(sql) > diff --git a/sql/log.cc b/sql/log.cc > index ee78844464e..132ec254adf 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -12522,3 +12512,61 @@ void wsrep_register_binlog_handler(THD *thd, bool trx) > } > > #endif /* WITH_WSREP */ > + > +static int online_alter_close_connection(handlerton *hton, THD *thd) > +{ > + DBUG_ASSERT(thd->online_alter_cache_list.empty()); > + return 0; > +} > + > +handlerton *online_alter_hton; > + > +int online_alter_log_init(void *p) > +{ > + online_alter_hton= (handlerton *)p; > + online_alter_hton->db_type= DB_TYPE_ONLINE_ALTER; > + online_alter_hton->savepoint_offset= sizeof(my_off_t); > + online_alter_hton->close_connection= online_alter_close_connection; > + > + online_alter_hton->savepoint_set= // Done by online_alter_savepoint_set > + [](handlerton *, THD *, void *){ return 0; }; > + online_alter_hton->savepoint_rollback= // Done by online_alter_savepoint_rollback > + [](handlerton *, THD *, void *){ return 0; }; > + online_alter_hton->savepoint_rollback_can_release_mdl= > + [](handlerton *hton, THD *thd){ return true; }; > + > + online_alter_hton->commit= [](handlerton *, THD *thd, bool all) > + { return binlog_online_alter_end_trans(thd, all, true); }; > + online_alter_hton->rollback= [](handlerton *, THD *thd, bool all) > + { return binlog_online_alter_end_trans(thd, all, false); }; > + online_alter_hton->commit_by_xid= [](handlerton *hton, XID *xid) > + { return binlog_online_alter_end_trans(current_thd, true, true); }; > + online_alter_hton->rollback_by_xid= [](handlerton *hton, XID *xid) > + { return binlog_online_alter_end_trans(current_thd, true, false); }; should it be so? commit_by_xid/rollback_by_xid take XID as an argument ad you ignore it. If commit_by_xid/rollback_by_xid shouldn't be called for online alter, don't set them at all. Or set to something that always fails (and add an assert too). > + > + online_alter_hton->drop_table= [](handlerton *, const char*) { return -1; }; yeah. Like this. > + online_alter_hton->flags= HTON_NOT_USER_SELECTABLE | HTON_HIDDEN > + | HTON_NO_ROLLBACK; better remove HTON_NO_ROLLBACK. As far as I can see it doesn't do anything here anyway, so the only effect it has - everyone reading the code will stop and thing "why HTON_NO_ROLLBACK is here?" Takes at least 5-10 minutes to look through all places where HTON_NO_ROLLBACK is tested to see that online_alter_hton cannot be there. > + return 0; > +} > + > +struct st_mysql_storage_engine online_alter_storage_engine= > +{ MYSQL_HANDLERTON_INTERFACE_VERSION }; > + > +maria_declare_plugin(online_alter_log) > +{ > + MYSQL_STORAGE_ENGINE_PLUGIN, > + &online_alter_storage_engine, > + "online_alter_log", > + "MariaDB PLC", "plc" lowercase > + "This is a pseudo storage engine to represent the online alter log in a transaction", > + PLUGIN_LICENSE_GPL, > + online_alter_log_init, > + NULL, > + 0x0100, // 1.0 > + NULL, // no status vars > + NULL, // no sysvars > + "1.0", > + MariaDB_PLUGIN_MATURITY_STABLE > +} > +maria_declare_plugin_end; Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: f9c1546bd7d: MDEV-32123: require_secure_transport doesn't allow TCP connections
by Sergei Golubchik 28 Sep '23

28 Sep '23

Hi, Dmitry, On Sep 28, Dmitry Shulga wrote: > revision-id: f9c1546bd7d (mariadb-11.2.1-6-gf9c1546bd7d) > parent(s): c4cbafe0c4c > author: Dmitry Shulga > committer: Dmitry Shulga > timestamp: 2023-09-28 16:20:36 +0700 > message: > > MDEV-32123: require_secure_transport doesn't allow TCP connections > > diff --git a/mysql-test/main/require_secure_transport.test b/mysql-test/main/require_secure_transport.test > index e238e732423..4ad18afb96d 100644 > --- a/mysql-test/main/require_secure_transport.test > +++ b/mysql-test/main/require_secure_transport.test > @@ -5,11 +5,16 @@ SET GLOBAL require_secure_transport=ON; > --error ER_SECURE_TRANSPORT_REQUIRED > connect without_ssl,localhost,root,,,,,TCP NOSSL; > --enable_query_log > + > +connect with_ssl,localhost,root,,,,,TCP SSL; where's a check that a connection was established successfully and it indeed uses SSL with require_secure_transport enabled? also, you didn't add any tests for --require-secure-transport=1 > +disconnect with_ssl; > + > connection default; > SET GLOBAL require_secure_transport=OFF; > --disable_query_log > connect without_ssl,localhost,root,,,,,TCP NOSSL; > --enable_query_log > disconnect without_ssl; > + > connection default; > DROP TABLE t1; > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 1f00a8670a6..3ce1c363c86 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -14519,6 +14546,22 @@ bool acl_authenticate(THD *thd, uint com_change_user_pkt_len) > > if (initialized) // if not --skip-grant-tables > { > + /* > + Check whether the option require_secure_transport is on and in case > + it is true that the secured connection type is used, that is either > + unix socket or named pipe or ssl is in use. > + */ > + if(check_require_secured_transport(thd)) > + { > + Host_errors errors; > + errors.m_ssl= 1; > + inc_host_errors(mpvio.auth_info.thd->security_ctx->ip, &errors); > + status_var_increment(thd->status_var.access_denied_errors); > + my_error(ER_SECURE_TRANSPORT_REQUIRED, MYF(0)); > + > + DBUG_RETURN(1); > + } No, no. This is wrong. The point of "require_secure_transport" is to make sure that *everything* is sent over the secure transport. In particular, passwords must never be sent if the transport is not secure. You cannot check for it after the authentication, you must abort the connection is soon as possible. (technically, the server can do nothing to prevent the client from sending passwords in plain-text over the insecure connection. but practically clients use mysql_native_password by default, so the password won't leak) Regards, Sergei Chief Architect, MariaDB Server and security(a)mariadb.org

1 0

Re: [PATCH] MDEV-31872 Deprecate ENCODE()/DECODE()
by Kristian Nielsen 26 Sep '23

26 Sep '23

Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > From: Sergei Golubchik <serg(a)mariadb.org> > @@ -2724,6 +2725,10 @@ bool Item_func_encode::seed() > hash_password(rand_nr, key->ptr(), key->length()); > sql_crypt.init(rand_nr); > > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, ER_WARN_DEPRECATED_SYNTAX, > + ER_THD(thd, ER_WARN_DEPRECATED_SYNTAX_NO_REPLACEMENT), > + func_name_cstring().str); // since 11.3 > + NAK. As I already explained in the MDEV-31872, deprecating these functions is not appropriate and will hurt users, not help them. Don't do this. You need to see this from the user's point of view. She is a DBA, she has an application with columns encoded with ENCODE(). What do you expect her to do, rewrite the application because you or someone does not like these functions? That's very disrespectful. If you want to give a warning for these functions, then you need to create an SQL mode or some other handle for the DBA to disable the warning and keep their application running. And the warning should explain why it's there, and what should be done to avoid it. And you need to carefully justify how burdening the DBA with this task of tracking down the source of the new warning and the way to remove it, is worth the benefit. And deprecation suggests eventual removal, which is even worse, please don't do that. You haven't even updated the documentation https://mariadb.com/kb/en/encode/ to eg. say "encode" or "scramble" instead of encrypt. "base64_encode" also does not do encryption, nor is rot13 cryptographically safe, but that doesn't justify removing them either. Most columns in mariadb databases are not encrypted or encoded at all, how is ENCODE() any less secure that that? - Kristian.

3 10

Re: 025ca90e5df: MDEV-32101 CREATE PACKAGE [BODY] for sql_mode=DEFAULT
by Sergei Golubchik 22 Sep '23

22 Sep '23

Hi, Alexander, On Sep 15, Alexander Barkov wrote: > revision-id: 025ca90e5df (mariadb-11.0.1-231-g025ca90e5df) > parent(s): 8ad1e26b1ba > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-09-05 17:33:27 +0400 > message: > > MDEV-32101 CREATE PACKAGE [BODY] for sql_mode=DEFAULT > > This patch adds PACKAGE support with SQL/PSM dialect for sql_mode=DEFAULT: > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 7452f6a083a..22c60bd4277 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -18454,6 +18518,22 @@ drop_routine: > if (Lex->stmt_drop_procedure($3, $4)) > MYSQL_YYABORT; > } > + | DROP PACKAGE_MARIADB_SYM opt_if_exists sp_name > + { > + LEX *lex= Lex; > + lex->set_command(SQLCOM_DROP_PACKAGE, $3); > + if (unlikely(lex->sphead)) > + my_yyabort_error((ER_SP_NO_DROP_SP, MYF(0), "PACKAGE")); > + lex->spname= $4; > + } > + | DROP PACKAGE_MARIADB_SYM BODY_MARIADB_SYM opt_if_exists sp_name > + { > + LEX *lex= Lex; > + lex->set_command(SQLCOM_DROP_PACKAGE_BODY, $4); > + if (unlikely(lex->sphead)) > + my_yyabort_error((ER_SP_NO_DROP_SP, MYF(0), "PACKAGE BODY")); > + lex->spname= $5; > + } > ; at least here and in SHOW you can use package: PACKAGE_MARIADB_SYM | PACKAGE_ORACLE_SYM ; and then move drop_routine out of ifdef > > > @@ -18498,6 +18578,248 @@ create_routine: > (Item_result) $8, $10)) > MYSQL_YYABORT; > } > + > + | create_or_replace definer_opt PACKAGE_MARIADB_SYM > + opt_if_not_exists sp_name opt_create_package_chistics_init > + remember_name > + { > + sp_package *pkg; > + if (unlikely(!(pkg= Lex-> > + create_package_start(thd, > + SQLCOM_CREATE_PACKAGE, > + &sp_handler_package_spec, > + $5, $1 | $4)))) > + MYSQL_YYABORT; > + pkg->set_c_chistics(Lex->sp_chistics); > + } > + opt_sqlpsm_package_specification_element_list END > + remember_end_opt > + { > + if (unlikely(Lex->create_package_finalize(thd, $5, NULL, $7, $11))) > + MYSQL_YYABORT; > + } > + | create_or_replace definer_opt PACKAGE_MARIADB_SYM BODY_MARIADB_SYM > + opt_if_not_exists sp_name opt_create_package_chistics_init > + remember_name > + { > + sp_package *pkg; > + if (unlikely(!(pkg= Lex-> > + create_package_start(thd, > + SQLCOM_CREATE_PACKAGE_BODY, > + &sp_handler_package_body, > + $6, $1 | $5)))) > + MYSQL_YYABORT; > + pkg->set_c_chistics(Lex->sp_chistics); > + Lex->sp_block_init(thd); > + } > + sqlpsm_package_implementation_declare_section > + { > + if (unlikely(Lex->sp_block_with_exceptions_finalize_declarations(thd))) > + MYSQL_YYABORT; > + } > + sqlpsm_package_implementation_executable_section > + { > + $10.hndlrs+= $12.hndlrs; > + if (unlikely(Lex->sp_block_finalize(thd, $10))) > + MYSQL_YYABORT; > + } > + remember_end_opt > + { > + if (unlikely(Lex->create_package_finalize(thd, $6, NULL, $8, $14))) > + MYSQL_YYABORT; > + } this can also use `package` rule as above and be moved outside of %ifdef. you'll only need %ifdef MARIADB sp_tail_is: _empty ; opt_sp_name: _empty { $$= NULL; } ; %else sp_tail_is: IS | AS ; opt_sp_name: _empty { $$= NULL; } | sp_name { $$= $1; } ; %endif > + ; > + > + > +opt_sqlpsm_package_specification_element_list: > + _empty > + | sqlpsm_package_specification_element_list > + ; and so on, this can be called simply opt_package_specification_element_list if placed under %ifdef MARIADB > + > +sqlpsm_package_specification_element_list: > + sqlpsm_package_specification_element > + | sqlpsm_package_specification_element_list > + sqlpsm_package_specification_element > + ; ... Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: 14a12f98eb8: MDEV-15935 Clean up string sys_var types.
by Sergei Golubchik 14 Sep '23

14 Sep '23

Hi, Yuchen, ok to push, after making one change (const) below. On Sep 13, Yuchen Pei wrote: > revision-id: 14a12f98eb8 (mariadb-11.0.1-232-g14a12f98eb8) > parent(s): e987b9350cb > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-09-13 11:50:41 +1000 > message: > > MDEV-15935 Clean up string sys_var types. > > Merge sys_var_charptr with sys_var_charptr_base, as well as merge > Sys_var_session_lexstring into Sys_var_lexstring. Also refactored > update methods of sys_var_charptr accordingly. > > Because the class is more generic, session_update() calls > sys_var_charptr::session_update() which does not assume a buffer field > associated with THD, but instead call strdup/free, we get rid of > THD::default_master_connection_buff accordingly You could've mentioned here "this also makes THD smaller by 192 bytes, and there can be many thousands of concurrent THDs" I mean, it's a good result, one of the benefits of your cleanup. > diff --git a/sql/sys_vars.inl b/sql/sys_vars.inl > index 31ebbae01ca..5bac91a0db3 100644 > --- a/sql/sys_vars.inl > +++ b/sql/sys_vars.inl > @@ -492,14 +492,18 @@ class Sys_var_mybool: public Sys_var_typelib > Backing store: char* > > @note > - This class supports only GLOBAL variables, because THD on destruction > - does not destroy individual members of SV, there's no way to free > - allocated string variables for every thread. > + > + Note that the memory management for SESSION_VAR's is manual, the > + value must be strdup'ed in THD::init() and freed in > + plugin_thdvar_cleanup(). TODO: it should be done automatically when > + we'll have more session string variables to justify it. Maybe some > + kind of a loop over all variables, like sys_var_end() in set_var.cc? > */ > -class Sys_var_charptr_base: public sys_var > +class Sys_var_charptr: public sys_var > { > + size_t max_length= 2000; Should be const, right? No need to store max_length in every Sys_var_charptr instance, if you aren't going to modify it per instance. > public: > - Sys_var_charptr_base(const char *name_arg, > + Sys_var_charptr(const char *name_arg, > const char *comment, int flag_args, ptrdiff_t off, size_t size, > CMD_LINE getopt, > const char *def_val, PolyLock *lock=0, Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: 9db04b3dc73: MDEV-15935 Adding global/session system var redirect_url
by Sergei Golubchik 13 Sep '23

13 Sep '23

Hi, Yuchen, Thanks. Ok to push both commits into preview-11.3-preview branch On Sep 13, Yuchen Pei wrote: > revision-id: 9db04b3dc73 (mariadb-11.0.1-233-g9db04b3dc73) > parent(s): 14a12f98eb8 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-09-13 11:50:42 +1000 > message: > > MDEV-15935 Adding global/session system var redirect_url > > Adding a global/session var `redirect_url' of string type. The initial > value is empty. Can be supplied in mysqld with --redirect-url or set > in --init-connect. A valid redirect_url should be of the format > > {mysql,mariadb}://host[:port] > > where <host> is an arbitrary string not containing colons, and <port> > is a number between 0 and 65535 inclusive. > > The variable will be used by the server to notify clients that they > should connect to another server, specified by the value of the > variable, if not empty. > > The notification is done by the inclusion of the variable in > session_track_system_variable. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: 8ac17273b14: MDEV-15935 Adding global/session system var redirect_url
by Sergei Golubchik 13 Sep '23

13 Sep '23

Hi, Yuchen, Right. This is pretty much the whole feature. Mostly ok, few minor comments below. On Sep 08, Yuchen Pei wrote: > revision-id: 8ac17273b14 (mariadb-11.0.1-104-g8ac17273b14) > parent(s): 365faf7229f > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-09-08 14:46:50 +1000 > message: > > MDEV-15935 Adding global/session system var redirect_url > > Adding a global/session var `redirect_url' of string type. The initial > value is empty. Can be supplied in mysqld with --redirect-url or set > in --init-connect. A valid redirect_url should be of the format > > {mysql,mariadb}://host[:port] > > where <host> is an arbitrary string not containing colons, and <port> > is a number between 0 and 65535 inclusive. > > The variable will be used by the server to notify clients that they > should connect to another server, specified by the value of the > variable, if not empty. > > The notification is done by the inclusion of the variable in > session_track_system_variable. > > diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > index fa3bf6b4354..85dc924fe5f 100644 > --- a/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_server_embedded.result > @@ -3232,6 +3232,16 @@ NUMERIC_BLOCK_SIZE 1 > ENUM_VALUE_LIST NULL > READ_ONLY NO > COMMAND_LINE_ARGUMENT REQUIRED > +VARIABLE_NAME REDIRECT_URL > +VARIABLE_SCOPE SESSION > +VARIABLE_TYPE VARCHAR > +VARIABLE_COMMENT URL to redirect to, if not empty see if you can make it a bit more descriptive. if not, ok as is. hmm, and a second thought. it's in the sysvars_server_embedded test. it doesn't make sense there (embedded doesn't accept connections, and doesn't have session tracking, iirc), so, may be hide it under #ifndef EMBEDDED_LIBRARY ? > +NUMERIC_MIN_VALUE NULL > +NUMERIC_MAX_VALUE NULL > +NUMERIC_BLOCK_SIZE NULL > +ENUM_VALUE_LIST NULL > +READ_ONLY NO > +COMMAND_LINE_ARGUMENT REQUIRED > VARIABLE_NAME REQUIRE_SECURE_TRANSPORT > VARIABLE_SCOPE GLOBAL > VARIABLE_TYPE BOOLEAN > diff --git a/sql/sql_class.h b/sql/sql_class.h > index cf2b5b96a22..c86a76f6a08 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -878,6 +878,7 @@ typedef struct system_variables > > Time_zone *time_zone; > char *session_track_system_variables; > + char *redirect_url; > > /* Some wsrep variables */ > ulonglong wsrep_trx_fragment_size; > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 4724e2200e6..70aba9cc4d5 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -3307,6 +3309,10 @@ void plugin_thdvar_init(THD *thd) > #ifndef EMBEDDED_LIBRARY > thd->session_tracker.sysvars.init(thd); > #endif > + thd->variables.redirect_url= > + my_strdup(PSI_INSTRUMENT_ME, global_system_variables.redirect_url, and here, key_memory_Sys_var_charptr_value > + MYF(MY_WME | MY_THREAD_SPECIFIC)); > + > DBUG_VOID_RETURN; > } > > @@ -3375,6 +3381,8 @@ void plugin_thdvar_cleanup(THD *thd) > #ifndef EMBEDDED_LIBRARY > thd->session_tracker.sysvars.deinit(thd); > #endif > + my_free(thd->variables.redirect_url); > + thd->variables.redirect_url= 0; also, perhaps in cleanup_variables() ? > my_free((char*) thd->variables.default_master_connection.str); > thd->variables.default_master_connection.str= 0; > thd->variables.default_master_connection.length= 0; > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 064c60ab2b0..a93e68f40d7 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -1343,6 +1343,61 @@ Sys_init_connect( > DEFAULT(""), &PLock_sys_init_connect, NOT_IN_BINLOG, > ON_CHECK(check_init_string)); > > +/** > + Validate a redirect_url string. > + > + A valid string is either empty, or of the format mysql://host[:port], > + where host is an arbitrary string without any colon ':'. > + > + @param str A string to validate > + @param len Length of the string > + @retval false The string is valid > + @retval true The string is invalid > +*/ > +static bool sysvar_validate_redirect_url(sys_var *, THD *, set_var *var) > +{ > + char *str= var->save_result.string_value.str; > + size_t len= var->save_result.string_value.length; > + /* Empty string is valid */ > + if (len == 0) > + return false; > + const char* end= str + len; > + if (!strncmp(str, "mysql://", strlen("mysql://"))) use !strncmp(str, STRING_WITH_LEN("mysql://")) avoids no strlen and no need to write the string twice. same below. > + str+= strlen("mysql://"); ah. Then, I'd suggest instead LEX_CSTRING mysql_prefix={STRING_WITH_LEN("mysql://")}; if (!strncmp(str, mysql_prefix.str, mysql_prefix.length)) str+= mysql_prefix.length; > + else if (!strncmp(str, "mariadb://", strlen("mariadb://"))) > + str+= strlen("mariadb://"); > + else > + return true; > + /* Host name cannot be empty */ > + if (str == end) > + return true; > + /* Find the colon, if any */ > + while (str < end && *str != ':') > + str++; > + /* Found colon */ > + if (str < end) > + { > + /* Should have at least one number after the colon */ > + if (str + 1 == end) > + return true; > + int p= 0; > + while (str < end && isdigit(*++str)) > + p= p * 10 + (*str - '0'); > + /* Should be all numbers after the colon */ > + if (str < end) > + return true; > + /* The port number should be between 0 and 65535 */ > + if (p > 65535) > + return true; check for overflow too. you can either put the p>65535 check inside the loop or at the end check that, like, (str - colon_ptr) < 7. > + } > + return false; > +} > + > +static Sys_var_charptr Sys_redirect_url( > + "redirect_url", "URL to redirect to, if not empty", > + SESSION_VAR(redirect_url), CMD_LINE(REQUIRED_ARG), DEFAULT(""), > + NO_MUTEX_GUARD, NOT_IN_BINLOG, sysvar_validate_redirect_url, 0, 0); By convention, write ON_CHECK(sysvar_validate_redirect_url). And you don't need 0's at the end, so just NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(sysvar_validate_redirect_url)); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [MariaDB commits] [PATCH] MDEV-31872: Add --old_mode=COMPAT_DISCOURAGED
by Sergei Golubchik 12 Sep '23

12 Sep '23

Hi, Kristian, On Sep 12, Kristian Nielsen wrote: > > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index e25d293bbb7..18d08996aa8 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -3863,12 +3863,13 @@ static const char *old_mode_names[]= > "IGNORE_INDEX_ONLY_FOR_JOIN", // deprecated since 11.3 > "COMPAT_5_1_CHECKSUM", // deprecated since 11.3 > "LOCK_ALTER_TABLE_COPY", // deprecated since 11.3 > + "COMPAT_DISCOURAGED", > 0 > }; > > void old_mode_deprecated_warnings(THD *thd, ulonglong v) > { > - v &= ~OLD_MODE_DEFAULT_VALUE; > + v &= ~(OLD_MODE_DEFAULT_VALUE | OLD_MODE_COMPAT_DISCOURAGED); No, this is wrong. old-mode *by definition* is something that is a temporary workaround for something new, old-mode values are by design deprecated from the day one. So, OLD_MODE_COMPAT_DISCOURAGED shouldn't be on the line above, and thus it'll itself be deprecated and will cause a deprecation warning. Which is rather silly, as I wrote in a previous email, and that's why an old-mode is not a good solution for turning deprecated warnings away. > for (uint i=0; old_mode_names[i]; i++) > if ((1ULL<<i) & v) > { > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 4b96271c331..339430a2794 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -2720,7 +2720,16 @@ bool Item_func_encode::seed() > hash_password(rand_nr, key->ptr(), key->length()); > sql_crypt.init(rand_nr); > > - warn_deprecated<1103>(current_thd, func_name_cstring().str); > + if (!(current_thd->variables.old_behavior & OLD_MODE_COMPAT_DISCOURAGED)) > + { > + /* > + This function should not be removed despite the deprecation warning. > + That would cause problems for users who can then no longer access > + columns in their database that might have been inserted as > + ENCODE(str,pass_str). > + */ > + warn_deprecated<1103>(current_thd, func_name_cstring().str); > + } No, this is exactly what I wanted to prevent by adding a warn_deprecated<>() wrapper. If something is giving a warning "... is deprecated and will be removed" - it *has to be removed*. We don't want to lie to users to simply scare them. If we promise to do something, we should do it. If we aren't going to remove ENCODE/DECODE functions, we shouldn't deprecate them at all. Which is the result I'm leaning to right now. > > return FALSE; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: 75ea482c4fc: MDEV-30973 GTID in deadlock output
by Sergei Golubchik 11 Sep '23

11 Sep '23

Hi, Aleksey, On Sep 11, Aleksey Midenkov wrote: > revision-id: 75ea482c4fc (mariadb-11.0.1-230-g75ea482c4fc) > parent(s): fe62f70fb4b > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2023-09-01 20:30:24 +0300 > message: > > MDEV-30973 GTID in deadlock output > > Example: > > TRANSACTION 40, ACTIVE 0 sec starting index read > mysql tables in use 1, locked 1 > LOCK WAIT 2 lock struct(s), heap size 1152, 1 row lock(s) > MySQL thread id 23, OS thread handle 140201491547840, query id 310, GTID 3-1-1 Updating > delete from tu where a = 9999 and b = 'xxxx' > 2023-04-12 19:53:58 22 [Note] InnoDB: *** (1) WAITING FOR THIS LOCK TO BE GRANTED: > > Deterministic slave_deadlock.test > > Instead of "MDEV-10962 Deadlock with 3 concurrent DELETEs by unique key" > this one uses "MDEV-22698 Deadlock on concurrent acquisition from multiple indexes" > > --- > mysql-test/suite/rpl/t/slave_deadlock.test | 93 ++++++++++++++++++++++++++++++ > sql/sql_show.cc | 10 ++++ please move slave_deadlock.result from the following commit to this one. otherwise ok Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Implementing Binlog GTID Indexes (MDEV-4991)
by Kristian Nielsen 10 Sep '23

10 Sep '23

I am implementing Binlog GTID Indexes, to fix an old performance regression for GTID when slaves connect to the master. I now have a solid design and a working prototype, so I wanted to describe the work to encourage early comments and suggestions. I'm hoping this will make it to the next release (where is the place to actively follow/participate in the release planning?) Ever since I implemented GTID, connecting slaves needed to sequentially scan the leading portion of one binlog file to locate the GTID position to start at. This can be slow as binlog files default to 1GB in size, even worse so when binlog files can now be encrypted. I files MDEV-4991 for this 10 years ago, and it is embarrassing that this has somehow remained unfixed for so long :-/ But now I'm fixing it. The current implementation is in the work-in-progress branch knielsen_mdev4991: https://github.com/MariaDB/server/tree/knielsen_mdev4991 The code is already functional, with the testsuite passing. Still missing is binlog purge, index crash recovery, async path, and general testing and cleanup. The basic idea is for each binlog file master-bin.000001 to write an index file master-bin.000001.idx. The index contains a B+-Tree in which the keys are pairs of (GTID state, binlog offset). A connecting slave's GTID position can be looked up quickly in the tree to find the corresponding binlog offset (and gtid_binlog_state) to start replicating at. Similarly, a non-GTID connecting slave can quickly look up its starting offset (BINLOG_GTID_POS()) to obtain the GTID position corresponding to its starting offset. The index is written out to disk concurrently with the writing of the binlog file. Since the tree is written in sorted order and append-only, the implementation is significantly simplified over a general B+-Tree, and can also be described as a Log-structed Merge Tree with a Btree search structure on top. All GTID state keys (except the first) in the index are delta-compressed, storing only the GTIDs that changed since the last record (which typically will be only one). Additionally the index is sparse, storing only say every 1-in-10 GTIDs in the index (this will be configurable of course). Using a sparse index, the disk space is reduced at the cost only of scanning a few extra events of the binlog file to find the exact position requested. Actual benchmarking of space usage is TBO, but we can expect a typical binlog file of 1 GB containing say 3 million GTIDs to require something like 10 MB of index disk space (1% increase). The file format is page-based (unlike the binlog file), so that it can be written efficiently and easily read using random-access. The index is written in parallel with the binlog file, but no extra fsync()s are needed (except one at the end of the file when the index is closed and synced to disk). The writing of the index will happen asynchroneously from the binlog background thread; this way it will have minimal impact on the performance and scalability of the binlog and the contested LOCK_log. Connecting slaves can read from the currently-being-written ("hot") GTID index by accessing internal memory buffers of pages not yet written to disk. If the server crashes, the existing binlog recovery and binlog checkpoint mechanism will be used to re-create any incomplete indexes as part of the normal recovery scan of binlog files. If somehow a binlog index should be found corrupt or missing (could be eg. old server upgrade or backup script that omits the new .idx files), the code will gracefully fall back to the old (slower) way of sequentially scanning the binlog file to locate the slave starting position. I think this will be a great (if rather late) improvement to one remaining corner of the GTID implementation that currently has sub-optimal performance, and I'm exited to get this completed and reviewed/tested and added to hopefully the next MariaDB release. - Kristian.

1 0

Re: 365faf7229f: MDEV-15935 Clean up string sys_var types.
by Sergei Golubchik 08 Sep '23

08 Sep '23

Hi, Yuchen, On Sep 08, Yuchen Pei wrote: > revision-id: 365faf7229f (mariadb-11.0.1-103-g365faf7229f) > parent(s): 4f39ae75606 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-09-08 13:12:22 +1000 > message: > > MDEV-15935 Clean up string sys_var types. > > Merge sys_var_charptr with sys_var_charptr_base, as well as merge > Sys_var_session_lexstring into Sys_var_lexstring. Also refactored > update methods of sys_var_charptr accordingly. > > Because the class is more generic, session_update() calls > sys_var_charptr::session_update() which does not assume a buffer field > associated with THD, but instead call strdup/free, we get rid of > THD::default_master_connection_buff accordingly > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 18e796655bd..52fa6c27112 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1233,10 +1233,12 @@ void THD::init() > avoid temporary tables replication failure. > */ > variables.pseudo_thread_id= thread_id; > - variables.default_master_connection.str= default_master_connection_buff; > - ::strmake(default_master_connection_buff, > - global_system_variables.default_master_connection.str, > - variables.default_master_connection.length); > + variables.default_master_connection.str= > + my_strndup(PSI_INSTRUMENT_ME, may be key_memory_Sys_var_charptr_value instead of PSI_INSTRUMENT_ME? > + global_system_variables.default_master_connection.str, > + global_system_variables.default_master_connection.length, > + MYF(MY_WME | MY_THREAD_SPECIFIC)); > + > mysql_mutex_unlock(&LOCK_global_system_variables); > > user_time.val= start_time= start_time_sec_part= 0; > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index d39589ed1ab..4724e2200e6 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -3279,6 +3279,9 @@ void plugin_thdvar_init(THD *thd) > #ifndef EMBEDDED_LIBRARY > thd->session_tracker.sysvars.deinit(thd); > #endif > + my_free((char*) thd->variables.default_master_connection.str); > + thd->variables.default_master_connection.str= 0; > + thd->variables.default_master_connection.length= 0; shouldn't this go inside cleanup_variables() ? > > thd->variables= global_system_variables; > > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 4843450352e..064c60ab2b0 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -1355,12 +1355,11 @@ static bool check_master_connection(sys_var *self, THD *thd, set_var *var) > return false; > } > > -static Sys_var_session_lexstring Sys_default_master_connection( > +static Sys_var_lexstring Sys_default_master_connection( > "default_master_connection", > "Master connection to use for all slave variables and slave commands", > - SESSION_ONLY(default_master_connection), > - NO_CMD_LINE, > - DEFAULT(""), MAX_CONNECTION_NAME, ON_CHECK(check_master_connection)); > + SESSION_ONLY(default_master_connection), NO_CMD_LINE, DEFAULT(""), > + NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(check_master_connection)); some length limit would still be nice, I'd say. Doesn't have to be per variable, may be all Sys_var_lexstring could have some common limit, like, arbitrarily, 1K or 10K. > #endif > > static Sys_var_charptr_fscs Sys_init_file( > diff --git a/sql/sys_vars.inl b/sql/sys_vars.inl > index 31ebbae01ca..0f2a07d92fc 100644 > --- a/sql/sys_vars.inl > +++ b/sql/sys_vars.inl > @@ -576,6 +571,21 @@ class Sys_var_charptr_base: public sys_var > new_val= 0; > return new_val; > } > + char *session_update_prepare(set_var *var) do you need this session_update_prepare method? global_update_prepare() and global_update_finish() are used by Sys_var_sesvartrack. Getting rid of them is a task for another day :) > + { > + return update_prepare(var, MYF(MY_WME | MY_THREAD_SPECIFIC)); > + } > + bool session_update(THD *thd, set_var *var) override > + { > + char *new_val= session_update_prepare(var); > + my_free(session_var(thd, char*)); > + session_var(thd, char*)= new_val; > + return (new_val == 0 && var->save_result.string_value.str != 0); > + } > + char *global_update_prepare(set_var *var) > + { > + return update_prepare(var, MYF(MY_WME)); > + } > void global_update_finish(char *new_val) > { > if (flags & ALLOCATED) > @@ -583,50 +593,28 @@ class Sys_var_charptr_base: public sys_var > flags|= ALLOCATED; > global_var(char*)= new_val; > } > - bool global_update(THD *thd, set_var *var) > + bool global_update(THD *thd, set_var *var) override > { > - char *new_val= global_update_prepare(thd, var); > + char *new_val= global_update_prepare(var); > global_update_finish(new_val); > return (new_val == 0 && var->save_result.string_value.str != 0); > } > - void session_save_default(THD *thd, set_var *var)= 0; > - void global_save_default(THD *thd, set_var *var) > + void save_default(set_var *var) > { > char *ptr= (char*)(intptr)option.def_value; > var->save_result.string_value.str= ptr; > var->save_result.string_value.length= ptr ? strlen(ptr) : 0; > } > -}; > - > -class Sys_var_charptr: public Sys_var_charptr_base > -{ > -public: > - Sys_var_charptr(const char *name_arg, > - const char *comment, int flag_args, ptrdiff_t off, size_t size, > - CMD_LINE getopt, > - const char *def_val, PolyLock *lock=0, > - enum binlog_status_enum binlog_status_arg=VARIABLE_NOT_IN_BINLOG, > - on_check_function on_check_func=0, > - on_update_function on_update_func=0, > - const char *substitute=0) : > - Sys_var_charptr_base(name_arg, comment, flag_args, off, size, getopt, > - def_val, lock, binlog_status_arg, > - on_check_func, on_update_func, substitute) > + void session_save_default(THD *, set_var *var) override > { > - SYSVAR_ASSERT(scope() == GLOBAL); > - SYSVAR_ASSERT(size == sizeof(char *)); > + return save_default(var); no, session's default value is global's value. See any other Sys_var_xxx class, e.g. for Sys_var_mybool: void session_save_default(THD *thd, set_var *var) { var->save_result.ulonglong_value= (ulonglong)*(my_bool *)global_value_ptr(thd, 0); } You cannot have a test for that now, but let's have one in the redirect_url commit. > } > - > - bool session_update(THD *thd, set_var *var) > + void global_save_default(THD *, set_var *var) override > { > - DBUG_ASSERT(FALSE); > - return true; > + return save_default(var); > } > - void session_save_default(THD *thd, set_var *var) > - { DBUG_ASSERT(FALSE); } > }; > > - > class Sys_var_charptr_fscs: public Sys_var_charptr > { > using Sys_var_charptr::Sys_var_charptr; > diff --git a/mysql-test/suite/sys_vars/r/mdev_15935.result b/mysql-test/suite/sys_vars/r/mdev_15935.result > new file mode 100644 > index 00000000000..d20c2ad2f19 > --- /dev/null > +++ b/mysql-test/suite/sys_vars/r/mdev_15935.result > @@ -0,0 +1,12 @@ > +# > +# test that the global/session-only enforcement is still in place despite the merge of string sys_var classes > +# > +set session init_connect="whatever"; > +ERROR HY000: Variable 'init_connect' is a GLOBAL variable and should be set with SET GLOBAL > +set session wsrep_auto_increment_control="whatever"; > +ERROR HY000: Variable 'wsrep_auto_increment_control' is a GLOBAL variable and should be set with SET GLOBAL > +set global default_master_connection="whatever"; > +ERROR HY000: Variable 'default_master_connection' is a SESSION variable and can't be used with SET GLOBAL not sure you need that, it's checked in the very base class, you could've have affected that I'd say it's enough if existing tests for Sys_var_session_lexstring and Sys_var_sesvartrack variables succeed and ASAN/MSAN are happy > +# > +# end of test mdev_15935 > +# Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: 04b99a30a3f: MDEV-15935 Adding global/session var redirect_url
by Sergei Golubchik 08 Sep '23

08 Sep '23

Hi, Yuchen, On Sep 01, Yuchen Pei wrote: > revision-id: 04b99a30a3f (mariadb-11.0.1-103-g04b99a30a3f) > parent(s): 4f39ae75606 > author: Yuchen Pei > committer: Yuchen Pei > timestamp: 2023-08-16 11:31:16 +1000 > message: > > MDEV-15935 Adding global/session var redirect_url > > Adding a global/session var `redirect_url' of string type. The initial > value is empty. Can be supplied in mysqld with --redirect-url. A valid > redirect_url should be of the format > > {mysql,mariadb}://host[:port] > > where <host> is an arbitrary string not containing colons, and <port> > is a number between 0 and 65535 inclusive. > > diff --git a/mysql-test/main/redirect.opt b/mysql-test/main/redirect.opt move this to the sys_vars suite, please > --- /dev/null > +++ b/mysql-test/main/redirect.opt > @@ -0,0 +1 @@ > +--redirect_url=mysql://foobar ok, as long as this isn't a complete feature, but just one piece of it. but when client support will be implemented (CONC-599) and the server will actually send this to the connecting clients (MDEV-31609), this test will need to be fixed, as foobar is not something a client can be redirected to. Also, it'd be more interesting to try something like --init-connect='set @@redirect_url="mysql://foobar"' instead of the simple --redirect_url=mysql://foobar because init-connect is much more flexible, it can generate the url on the fly, it can send different users to different locations. So it'd make sense to test init-connect approach. > diff --git a/mysql-test/main/redirect.test b/mysql-test/main/redirect.test > new file mode 100644 > --- /dev/null > +++ b/mysql-test/main/redirect.test > @@ -0,0 +1,56 @@ > +--echo # > +--echo # MDEV-15935 Connection Redirection Mechanism in MariaDB Client/Server Protocol > +--echo # > + > +set @old_redirect_url=@@redirect_url; > +select @@global.redirect_url; > +set global redirect_url=default; > +select @@global.redirect_url; > +--error ER_WRONG_VALUE_FOR_VAR > +set global redirect_url="mariadb.org"; > +--error ER_WRONG_VALUE_FOR_VAR > +set global redirect_url="https://mariadb.org"; before doing all this fun, please remove redirect_url from the session_track_system_variables, or just set it to the empty string. > diff --git a/sql/sql_class.h b/sql/sql_class.h > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -3585,6 +3587,7 @@ class THD: public THD_count, /* this must be first */ > */ > LEX_CSTRING connection_name; > char default_master_connection_buff[MAX_CONNECTION_NAME+1]; > + char redirect_url_buff[MAX_CONNECTION_NAME+1]; This isn't such a good idea (I'm afraid, I missed it when default_master_connection_buff was added). A couple of years ago Monty have spent significant efforts trying to make THD smaller as it has grown to be too large. Now THD keeps growing again. Let's try not to encourage it, if we can help it. > uint8 password; /* 0, 1 or 2 */ > uint8 failed_com_change_user; > bool slave_thread; > diff --git a/sql/sys_vars.inl b/sql/sys_vars.inl > --- a/sql/sys_vars.inl > +++ b/sql/sys_vars.inl > @@ -637,6 +637,129 @@ class Sys_var_charptr_fscs: public Sys_var_charptr ... > +class Sys_var_redirect_url: public Sys_var_charptr_base please, no. new classes are needed only for variables with very special, unlike anything else, behavior. redirect_url is just a session string variable. See the comment at the top of sys_vars.cc But (!) there are no other session string variables yet. So I'd recomment to fix Sys_var_charptr to support session variables. And then create redirect_url out of it. Practically it might need no changes at all, if you put explicit strmake (or strdup/free) into THD::init like you did anyway. After that, I suspect, you might be able to remove Sys_var_charptr_base, Sys_var_sesvartrack, and Sys_var_session_lexstring, because Sys_var_charptr/Sys_var_lexstring will be able to serve all those use cases. > +{ > + static const size_t max_length = MAX_CONNECTION_NAME; > +public: > + Sys_var_redirect_url(const char *name_arg, > + const char *comment, > + CMD_LINE getopt, > + const char *def_val, PolyLock *lock= 0) : > + Sys_var_charptr_base(name_arg, comment, > + SESSION_VAR(redirect_url), > + getopt, def_val, lock, > + VARIABLE_NOT_IN_BINLOG, 0, 0, 0) > + { > + global_var(LEX_CSTRING).length= strlen(def_val); > + option.var_type|= GET_STR; > + *const_cast<SHOW_TYPE*>(&show_val_type)= SHOW_LEX_STRING; > + } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: 44e692d0dbc: MDEV-29167 new db-level SHOW CREATE ROUTINE privilege
by Sergei Golubchik 06 Sep '23

06 Sep '23

Hi, Oleksandr, On Sep 04, Oleksandr Byelkin wrote: > revision-id: 44e692d0dbc (mariadb-11.0.1-228-g44e692d0dbc) > parent(s): 9cb75f333fa > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2023-09-04 14:12:29 +0200 > message: > > MDEV-29167 new db-level SHOW CREATE ROUTINE privilege > > -- Create general_log if CSV is enabled. > diff --git a/scripts/mariadb_system_tables_fix.sql b/scripts/mariadb_system_tables_fix.sql > --- a/scripts/mariadb_system_tables_fix.sql > +++ b/scripts/mariadb_system_tables_fix.sql > @@ -706,6 +706,9 @@ ALTER TABLE db modify Delete_history_priv enum('N','Y') COLLATE utf8mb3_general_ > > UPDATE user SET Delete_history_priv = Super_priv WHERE @had_user_delete_history_priv = 0; > > +ALTER TABLE db ADD Show_create_routine_priv enum('N','Y') COLLATE utf8mb3_general_ci NOT NULL DEFAULT 'N' AFTER Delete_history_priv; > +ALTER TABLE db modify Show_create_routine_priv enum('N','Y') COLLATE utf8mb3_general_ci NOT NULL DEFAULT 'N'; Why two statements? > + > ALTER TABLE user ADD plugin char(64) CHARACTER SET latin1 DEFAULT '' NOT NULL AFTER max_user_connections, > ADD authentication_string TEXT NOT NULL AFTER plugin; > ALTER TABLE user CHANGE auth_string authentication_string TEXT NOT NULL; > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -9003,7 +9008,7 @@ bool check_routine_level_acl(THD *thd, const char *db, const char *name, > NULL, db, > sctx->priv_role, > name, sph, 0))) > - no_routine_acl= !(grant_proc->privs & SHOW_PROC_ACLS); > + no_routine_acl= !(grant_proc->privs & SHOW_PROC_NO_DEF_ACLS); Oh, now I get it. All the time up until this line I thought that SHOW_PROC_NO_DEF_ACLS means "privileges that *not a definer* of the routine needs to have" And now I see that it means to "show procedure without definition". Please, rename to something, like, SHOW_PROC_EXISTANCE or SHOW_PROC_NO_BODY or even SHOW_PROC_WITHOUT_DEFINITION. But SHOW_PROC_NO_DEF is rather misleading. > } > mysql_rwlock_unlock(&LOCK_grant); > return no_routine_acl; > @@ -9237,7 +9242,7 @@ static const char *command_array[]= > "CREATE USER", "EVENT", "TRIGGER", "CREATE TABLESPACE", "DELETE HISTORY", > "SET USER", "FEDERATED ADMIN", "CONNECTION ADMIN", "READ_ONLY ADMIN", > "REPLICATION SLAVE ADMIN", "REPLICATION MASTER ADMIN", "BINLOG ADMIN", > - "BINLOG REPLAY", "SLAVE MONITOR" > + "BINLOG REPLAY", "SLAVE MONITOR", "SHOW CREATE ROUTINE" > }; > > static uint command_lengths[]= > @@ -9250,7 +9255,7 @@ static uint command_lengths[]= > 11, 5, 7, 17, 14, > 8, 15, 16, 15, > 23, 24, 12, > - 13, 13 > + 13, 13, 19 May be you'll change these two arrays into one of LEX_CSTRING's ? in a separate commit, of course. > }; > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -6676,7 +6677,8 @@ int store_schema_params(THD *thd, TABLE *table, TABLE *proc_table, > DBUG_RETURN(0); > > if (!full_access) > - full_access= !strcmp(sp_user, definer.str); > + full_access= !check_db_routine_access(thd, SHOW_CREATE_ROUTINE_ACL, > + db.str, name.str, sph, TRUE); Wait, I thought we agreed that to see a routine body one has to be either the definer or granted SHOW CREATE ROUTINE. As far as I can see check_db_routine_access doesn't check for the user being the definer. > if (!full_access && > check_some_routine_access(thd, db.str, name.str, sph)) > DBUG_RETURN(0); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 3

Re: 82a49812507: MDEV-22597 Add views for periods in information_schema
by Sergei Golubchik 05 Sep '23

05 Sep '23

Hi, Nikita, On Aug 31, Nikita Malyavin wrote: > revision-id: 82a49812507 (mariadb-11.0.1-228-g82a49812507) > parent(s): 23b15d24eb3 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-08-30 22:49:36 +0400 > message: > > MDEV-22597 Add views for periods in information_schema > > Two new information_schema views are added: > * PERIOD table -- columns TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, > PERIOD_NAME, START_COLUMN_NAME, END_COLUMN_NAME. > * KEY_PERIOD_USAGE -- works similar to KEY_COLUMN_USAGE, but for periods. > Columns CONSTRAINT_CATALOG, CONSTRAINT_SCHEMA, CONSTRAINT_NAME, > TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, PERIOD_NAME > > Two new columns are added to the COLUMNS view: > IS_SYSTEM_TIME_PERIOD_START, IS_SYSTEM_TIME_PERIOD_END - contain YES/NO. > > diff --git a/mysql-test/main/information_schema-big.result b/mysql-test/main/information_schema-big.result > index 94c8d274fad..ad0451293c9 100644 > --- a/mysql-test/main/information_schema-big.result > +++ b/mysql-test/main/information_schema-big.result good, you didn't forget to update big and embedded tests > diff --git a/mysql-test/suite/period/r/i_s_notembedded.result b/mysql-test/suite/period/r/i_s_notembedded.result > new file mode 100644 > index 00000000000..879376cc8b4 > --- /dev/null > +++ b/mysql-test/suite/period/r/i_s_notembedded.result > @@ -0,0 +1,52 @@ > +select * from information_schema.periods; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME PERIOD START_COLUMN_NAME END_COLUMN_NAME > +create or replace table t1 (id int primary key, s timestamp(6), e timestamp(6), > +period for mytime(s,e)); > +create or replace table t2 (id int primary key, s timestamp(6), e timestamp(6), > +period for mytime(s,e)) with system versioning; > +show columns from t1; > +Field Type Null Key Default Extra > +id int(11) NO PRI NULL > +s timestamp(6) NO NULL > +e timestamp(6) NO NULL > +select * from information_schema.periods where table_schema = 'test'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME PERIOD START_COLUMN_NAME END_COLUMN_NAME > +def test t2 SYSTEM_TIME row_start row_end I don't think so, they're INVISIBLE_SYSTEM. for all practical purposes they don't exist as columns in the table. Shouldn't be shown either. But don't remove this test, let's keep it to show that row_start/row_end are *not* shown here. And add table t3 with full standard definition of period for system_time. it should be shown all right. > +def test t2 mytime s e > +def test t1 mytime s e > +create user periods_hidden@localhost; ... > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 1dc60a01f38..876d6c6f244 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -6165,11 +6166,83 @@ static void store_variable_type(THD *thd, const sp_variable *spvar, > } > } > > +static int store_schema_period_record(THD *thd, TABLE_LIST *tl, > + TABLE *schema_table, > + const LEX_CSTRING *db_name, > + const LEX_CSTRING *table_name, > + const TABLE_SHARE::period_info_t &period) > +{ > + TABLE_SHARE *s= tl->table->s; > + const CHARSET_INFO *cs= system_charset_info; > + > + int err= schema_table->field[0]->store(STRING_WITH_LEN("def"), cs); > + if(!err) err= schema_table->field[1]->store(db_name, cs); > + if(!err) err= schema_table->field[2]->store(table_name, cs); > + if(!err) err= schema_table->field[3]->store(period.name, cs); well, ok. but as you've seen elsewhere these go without checks, as these calls generally cannot fail, they don't even allocate any memory. > + > + if (err) return err; > + > + int period_field= 4; > + for (auto *field: {period.start_field(s), period.end_field(s)}) > + { > +#ifndef NO_EMBEDDED_ACCESS_CHECKS > + bool col_granted= get_column_grant(thd, &tl->grant, > + db_name->str, table_name->str, > + field->field_name.str) & COL_ACLS; > + if (col_granted) > +#endif > + { > + schema_table->field[period_field]->set_notnull(); > + err= schema_table->field[period_field]->store(field->field_name, cs); > + } > + > + if (err) > + return err; > + period_field++; > + } > + > + return schema_table_store_record(thd, schema_table); > +} > + > + > +static int > +get_schema_period_records(THD *thd, TABLE_LIST *tl, > + TABLE *schema_table, //!< @ref > + //!< Show::periods_fields_info > + bool res, > + const LEX_CSTRING *db_name, > + const LEX_CSTRING *table_name) > +{ > + TABLE *table= tl->table; > + > + if (!table || (!table->s->period.name && !table->versioned())) > + return 0; > + > +#ifndef NO_EMBEDDED_ACCESS_CHECKS > + check_access(thd, SELECT_ACL, db_name->str, > + &tl->grant.privilege, 0, 0, MY_TEST(tl->schema_table)); > + if (is_temporary_table(tl)) > + { > + tl->grant.privilege|= TMP_TABLE_ACLS; > + } > +#endif > + int err= 0; > + if (table->versioned()) > + err= store_schema_period_record(thd, tl, schema_table, db_name, table_name, > + table->s->vers); > + if (!err && table->s->period.name) > + err= store_schema_period_record(thd, tl, schema_table, db_name, table_name, > + table->s->period); > + > + return err; > +} > > -static int get_schema_column_record(THD *thd, TABLE_LIST *tables, > - TABLE *table, bool res, > - const LEX_CSTRING *db_name, > - const LEX_CSTRING *table_name) > +static > +int get_schema_column_record(THD *thd, TABLE_LIST *tables, > + TABLE *table, //!< @ref Show::columns_fields_info > + bool res, > + const LEX_CSTRING *db_name, > + const LEX_CSTRING *table_name) > { > LEX *lex= thd->lex; > const char *wild= lex->wild ? lex->wild->ptr() : NullS; > @@ -7513,30 +7592,41 @@ static int get_schema_triggers_record(THD *thd, TABLE_LIST *tables, > DBUG_RETURN(0); > } > > +static int > +store_key_column_usage(TABLE *table, const LEX_CSTRING &db_name, > + const LEX_CSTRING &table_name, > + const LEX_CSTRING &key_name, > + const LEX_CSTRING &col_name) > +{ > + CHARSET_INFO *cs= system_charset_info; > + static const LEX_CSTRING def{STRING_WITH_LEN("def")}; > + const LEX_CSTRING *values[] { &def, &db_name, &key_name, > + &def, &db_name, &table_name, &col_name }; > + int err = 0; > + for (uint i = 0; i < array_elements(values) && !err; i++) > + table->field[i]->store(values[i], cs); you forgot `err=` here :) but as I wrote above, it's not needed here anyway > + return 0; > +} > > static void > store_key_column_usage(TABLE *table, const LEX_CSTRING *db_name, > const LEX_CSTRING *table_name, const char *key_name, > - size_t key_len, const char *con_type, size_t con_len, > + size_t key_len, const char *col_name, size_t col_len, > longlong idx) > { > - CHARSET_INFO *cs= system_charset_info; > - table->field[0]->store(STRING_WITH_LEN("def"), cs); > - table->field[1]->store(db_name->str, db_name->length, cs); > - table->field[2]->store(key_name, key_len, cs); > - table->field[3]->store(STRING_WITH_LEN("def"), cs); > - table->field[4]->store(db_name->str, db_name->length, cs); > - table->field[5]->store(table_name->str, table_name->length, cs); > - table->field[6]->store(con_type, con_len, cs); > + store_key_column_usage(table, *db_name, *table_name, {key_name, key_len}, > + {col_name, col_len}); pretty, but better, please, put it back. You can use the old store_key_column_usage from your get_schema_key_period_usage_record() just the same. There's no need to have two store_key_column_usage() functions. > table->field[7]->store((longlong) idx, TRUE); > } > > > -static int get_schema_key_column_usage_record(THD *thd, > - TABLE_LIST *tables, > - TABLE *table, bool res, > - const LEX_CSTRING *db_name, > - const LEX_CSTRING *table_name) > +static int > +get_schema_key_column_usage_record(THD *thd, TABLE_LIST *tables, > + TABLE *table, > + //!< @ref Show::key_column_usage_fields_info > + bool res, > + const LEX_CSTRING *db_name, > + const LEX_CSTRING *table_name) > { > DBUG_ENTER("get_schema_key_column_usage_record"); > if (res) > @@ -9703,6 +9824,18 @@ ST_FIELD_INFO key_column_usage_fields_info[]= > CEnd() > }; > > +ST_FIELD_INFO key_period_usage_fields_info[]= > +{ > + Column("CONSTRAINT_CATALOG", Catalog(), NOT_NULL, OPEN_FULL_TABLE), > + Column("CONSTRAINT_SCHEMA", Name(), NOT_NULL, OPEN_FULL_TABLE), > + Column("CONSTRAINT_NAME", Name(), NOT_NULL, OPEN_FULL_TABLE), > + Column("TABLE_CATALOG", Catalog(), NOT_NULL, OPEN_FULL_TABLE), > + Column("TABLE_SCHEMA", Name(), NOT_NULL, OPEN_FULL_TABLE), > + Column("TABLE_NAME", Name(), NOT_NULL, OPEN_FULL_TABLE), > + Column("PERIOD_NAME", Name(), NOT_NULL, OPEN_FULL_TABLE), Cannot they all be OPEN_FRM_ONLY? > + CEnd() > +}; > + > > ST_FIELD_INFO table_names_fields_info[]= > { > @@ -10208,6 +10358,8 @@ ST_SCHEMA_TABLE schema_tables[]= > {0, 0, 0, 0, 0, 0, 0, 0, 0, 0} > }; > > +static_assert(array_elements(schema_tables) == SCH_ENUM_SIZE + 1, > + "Update enum_schema_tables as well."); +1 > > int initialize_schema_table(st_plugin_int *plugin) > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 6

Re: 8114fb63bbb: MDEV-31684: Add timezone information to DATE_FORMAT
by Sergei Golubchik 04 Sep '23

04 Sep '23

Hi, Rucha, Few comments, after I saw the commit in the branch: On Sep 04, Rucha Deodhar wrote: > revision-id: 8114fb63bbb (mariadb-11.0.1-229-g8114fb63bbb) > parent(s): 7ba9c7fb84b > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2023-09-01 11:01:09 +0530 > message: > > MDEV-31684: Add timezone information to DATE_FORMAT > > Before starting to go over the format string, prepare the current time > zone information incase '%z' or '%Z' is encountered. > This information can be obtained as given below: > > A) If timezone is not set ( meaning we are working with system timezone): > Get the MYSQL_TIME representation for current time and GMT time using > current thread variable for timezone and timezone variable for UTC > respectively. This MYSQL_TIME variable will be used to calculate time > difference. Also convert current time in second to tm structure to > get system timezone information. > > B) If timezone is set as offset: > Get timezone information using current timezone information and store > in appropriate variable. > > C) If timezone is set as some place (example: Europe/Berlin) > Get timezone information by searching the timezone. During internal > timezone search, information like timeoffset from UTC and abbrevation > is stored in another relevant structure. Hence use the same information. > > diff --git a/mysql-test/main/date_formats.result b/mysql-test/main/date_formats.result > index 463cce39520..23aeffedef7 100644 > --- a/mysql-test/main/date_formats.result > +++ b/mysql-test/main/date_formats.result > @@ -573,3 +573,67 @@ time_format('01 02:02:02', '%T') > select time_format('2001-01-01 02:02:02', '%T'); > time_format('2001-01-01 02:02:02', '%T') > 02:02:02 > +# > +# Beginning of 11.3 test > +# > +# MDEV-31684: Add timezone information to DATE_FORMAT > +# > +SET @old_timezone= @@time_zone; > +SET TIME_ZONE='Japan'; > +SELECT DATE_FORMAT('2009-10-04 22:23:00', '%z %Z') AS current_timezone; > +current_timezone > ++0900 CJT > +SELECT DATE_FORMAT(CURDATE(), '%z %Z') AS current_tz; 1) please, don't use CURDATE/CURRENT_TIMESTAMP/etc. The time zone abbreviation depends on the date: $ TZ=Europe/Berlin date -d '2020-01-01 1:2:3' Wed Jan 1 01:02:03 CET 2020 $ TZ=Europe/Berlin date -d '2020-06-01 1:2:3' Mon Jun 1 01:02:03 CEST 2020 See? Could be CET or CEST. Japan doesn't seem to have daylight saving, so it'll be the same abbreviation throughout the year, but it can still change at some point. So always use a fixed date, like in your first test. 2) why is it CJT at all? $ TZ=Japan date -d '2009-10-04 22:23:00' +'%z %Z' +0900 JST it looks like it should be JST? ... 3) also, please test that CET/CEST example. Or, rather, MET/MEST/MST/MMT example - the data for these abbreviations are already present in mariadb_test_data_timezone.sql That is, add lines like SET @@time_zone='Europe/Moscow'; SELECT DATE_FORMAT('....', '%z %Z'); SELECT DATE_FORMAT('....', '%z %Z'); SELECT DATE_FORMAT('....', '%z %Z'); and with different dates it should return different abbreviations for the same time zone. > +# > +# output depends on system time information. Hence replacing > +# to avoid result diff test failures. > +# > +SET @@time_zone= default; > +SELECT DATE_FORMAT('2009-10+|-HH:MM ABC22:23:00', '%z %Z') AS current_timezone; 1) I don't understand what you're testing here :( 2) Anyway, note that you can put, say, --timezone=Europe/Berlin into the date_formats.opt file and the server will be restarted with this value in TZ. That is, you can force the system time zone to be a specific fixed value. > diff --git a/scripts/mariadb_test_data_timezone.sql b/scripts/mariadb_test_data_timezone.sql > index 8d07d413cef..83bb548478d 100644 > --- a/scripts/mariadb_test_data_timezone.sql > +++ b/scripts/mariadb_test_data_timezone.sql > @@ -14,8 +14,8 @@ > -- along with this program; if not, write to the Free Software > -- Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA > > -INSERT INTO time_zone_name (Name, Time_Zone_id) ... > -INSERT INTO time_zone (Time_zone_id, Use_leap_seconds) ... > -INSERT INTO time_zone_transition (Time_zone_id, Transition_time, Transition_type_id) ... > -INSERT INTO time_zone_transition_type (Time_zone_id, Transition_type_id, `Offset`, Is_DST, Abbreviation) ... > -INSERT INTO time_zone_leap_second (Transition_time, Correction) ... > +INSERT INTO time_zone_name (Name, Time_Zone_id) VALUES ... ('India/Kolkata', 6); > +INSERT INTO time_zone (Time_zone_id, Use_leap_seconds) ... > +INSERT INTO time_zone_transition (Time_zone_id, Transition_time, Transition_type_id) ... > +INSERT INTO time_zone_transition_type (Time_zone_id, Transition_type_id, `Offset`, Is_DST, Abbreviation) ... > +INSERT INTO time_zone_leap_second (Transition_time, Correction) ... 1. Why? I didn't see where you use this new value in tests. 2. Is it really India/Kolkata? The official https://www.iana.org/time-zones seems to have Asia/Kolkata. > diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc > index 26adc4eddaa..f8d3fe7dfe8 100644 > --- a/sql/item_timefunc.cc > +++ b/sql/item_timefunc.cc > @@ -468,20 +468,111 @@ static bool extract_date_time(THD *thd, DATE_TIME_FORMAT *format, > DBUG_RETURN(1); > } > > +/* > + > + Gets local time in MYSQL_TIME structure and tm struct. > + > +*/ > +void get_timezone_offset_information(THD *thd, Time_zone *tz, int &hr, > + int &min, char* abbrevation, > + bool &is_neg) > +{ > + if (!tz) > + return; > + > + if (tz->get_name()->length() == tz_SYSTEM_name.length() && > + !strncmp((const char*)(tz->get_name()->ptr()), > + (const char*)(tz_SYSTEM_name.ptr()), 6)) > + { > + struct tm tm_local_time; > + time_t time_sec= my_time(0); > + MYSQL_TIME local_TIME, gmt_TIME; > + ulonglong seconds_diff; > + ulong microsec_diff; > + > + /* Get local time in MYSQL_TIME struct. */ > + thd->variables.time_zone->gmt_sec_to_TIME(&local_TIME, > + thd->query_start()); > + /* Get GMT time in MYSQL_TIME struct. */ > + my_tz_UTC->gmt_sec_to_TIME(&gmt_TIME, thd->query_start()); > + > + /* We need above two MYSQL_TIME to get time difference. */ > + is_neg= calc_time_diff(&local_TIME, &gmt_TIME, 1, > + &seconds_diff, &microsec_diff); > + > + /* > + once we have time difference in seconds_diff, > + convert it into hr and min. > + */ > + hr= (int)(seconds_diff/3600L); > + int temp= (int)(seconds_diff%3600L); > + min= temp/60L; > + > + /* > + Get timezone abbrevation. It is stored in tm structure, > + so convert time to tm struct and copy it. > + */ 1) tm structure also has the offset, you don't need to calculate it manually. 2) you should use not my_time(0) but the value of the first DATE_FORMAT() argument > + localtime_r(&time_sec, &tm_local_time); > +# ifdef __USE_MISC > + int len= strlen((tm_local_time.tm_zone)); > + strncpy(abbrevation, (tm_local_time.tm_zone), len); > + abbrevation[len]= '\0'; > +# endif > + } > + else > + { > + /* > + Information for "named" timezone and offset can be obtained from their > + relevant class. So get the timezone information first. > + */ > + struct tz tmp; > + tz->get_timezone_information(&tmp); > + is_neg= tmp.is_behind; > + > + /* Get time in hour and min. */ > + time_t time_in_sec= abs(tmp.seconds_offset); > + hr= (int)(time_in_sec/3600L); > + int temp= (int)(time_in_sec%3600L); > + min= temp/60L; > + > + /* Copy the abbrevation. */ > + strncpy(abbrevation, (const char*)(tmp.abbrevation), > + strlen((const char*)(tmp.abbrevation))); > + abbrevation[strlen((const char*)(tmp.abbrevation))+1]='\0'; > + } > + > + return; > +} > + > > /** > Create a formatted date/time value in a string. > */ > > -static bool make_date_time(const String *format, const MYSQL_TIME *l_time, > - timestamp_type type, const MY_LOCALE *locale, > - String *str) > +static bool make_date_time(THD *thd, const String *format, > + const MYSQL_TIME *l_time, timestamp_type type, > + const MY_LOCALE *locale, String *str) > { > char intbuff[15]; > uint hours_i; > uint weekday; > ulong length; > const char *ptr, *end; > + int diff_hr=0, diff_min=0; > + bool curr_time_behind_utc= false; > + char abbrevation[8]; > + > + Time_zone* curr_timezone= my_tz_find(thd, > + thd->variables.time_zone->get_name()); > + > + /* > + precalculate local time and timezone information because > + %z or %Z maybe present in the format. > + */ > + memset(abbrevation, 0, sizeof(abbrevation)); > + get_timezone_offset_information(thd, curr_timezone, diff_hr, > + diff_min, abbrevation, > + curr_time_behind_utc); Just a thought. get_timezone_offset_information() doesn't seem to be very cheap. May be do it only when %z or %Z is requested? Like case 'z': if (!have_tz_info) { memset(abbrevation, 0, sizeof(abbrevation)); get_timezone_offset_information(thd, curr_timezone, diff_hr, diff_min, abbrevation, curr_time_behind_utc); have_tz_info= true; } Or better something with less code duplication :) > str->length(0); > > @@ -699,6 +790,31 @@ static bool make_date_time(const String *format, const MYSQL_TIME *l_time, > str->append_with_prefill(intbuff, length, 1, '0'); > break; > > + case 'z': > + { > + if (curr_time_behind_utc) > + str->append("-", 1, system_charset_info); > + else > + str->append("+", 1, system_charset_info); > + > + if (diff_hr/10 == 0) > + str->append("0", 1, system_charset_info); > + length= (uint) (int10_to_str(diff_hr, intbuff, 10) - intbuff); > + str->append_with_prefill(intbuff, length, 1, '0'); > + if (diff_min/10 == 0) > + str->append("0", 1, system_charset_info); > + length= (uint) (int10_to_str(diff_min, intbuff, 10) - intbuff); > + str->append_with_prefill(intbuff, length, 1, '0'); why do you append "0" manually, if you use append_with_prefill(), which, I suppose, does it automatically? or, if you add "0" manually, you should use simple append(). > + } > + break; > + > + case 'Z': > + { > + str->append(abbrevation, > + (uint) strlen(abbrevation), > + system_charset_info); > + } > + break; > default: > str->append(*ptr); > break; > @@ -1912,7 +2028,7 @@ String *Item_func_date_format::val_str(String *str) > > /* Create the result string */ > str->set_charset(collation.collation); > - if (!make_date_time(format, &l_time, > + if (!make_date_time(thd, format, &l_time, > is_time_format ? MYSQL_TIMESTAMP_TIME : > MYSQL_TIMESTAMP_DATE, > lc, str)) > diff --git a/sql/tztime.cc b/sql/tztime.cc > index e77a7529332..4662fa082e0 100644 > --- a/sql/tztime.cc > +++ b/sql/tztime.cc > @@ -55,15 +55,10 @@ > > /* > Now we don't use abbreviations in server but we will do this in future. > + Edit: Startd needing abbrevation in server as part of task MDEV-31684. typo > */ > -#if defined(TZINFO2SQL) || defined(TESTTIME) > -#define ABBR_ARE_USED > -#else > -#if !defined(DBUG_OFF) > -/* Let use abbreviations for debug purposes */ > -#undef ABBR_ARE_USED > -#define ABBR_ARE_USED > -#endif /* !defined(DBUG_OFF) */ > +#ifndef ABBR_ARE_USED > +#define ABBR_ARE_USED > #endif /* defined(TZINFO2SQL) || defined(TESTTIME) */ > > /* Structure describing local time type (e.g. Moscow summer time (MSD)) */ > @@ -1008,11 +1003,6 @@ TIME_to_gmt_sec(const MYSQL_TIME *t, const TIME_ZONE_INFO *sp, uint *error_code) > > #if !defined(TESTTIME) && !defined(TZINFO2SQL) > > -/* > - String with names of SYSTEM time zone. > -*/ > -static const String tz_SYSTEM_name("SYSTEM", 6, &my_charset_latin1); > - > > /* > Instance of this class represents local time zone used on this system > @@ -1031,6 +1021,7 @@ class Time_zone_system : public Time_zone > virtual my_time_t TIME_to_gmt_sec(const MYSQL_TIME *t, uint *error_code) const; > virtual void gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const; > virtual const String * get_name() const; > + virtual void get_timezone_information(struct tz* curr_tz) const; > }; > > > @@ -1097,6 +1088,12 @@ Time_zone_system::gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const > adjust_leap_second(tmp); > } > > +void > +Time_zone_system::get_timezone_information(struct tz* curr_tz) const > +{ this is weird. You have a special if (tz->get_name()->length() == tz_SYSTEM_name.length() && !strncmp((const char*)(tz->get_name()->ptr()), (const char*)(tz_SYSTEM_name.ptr()), 6)) to detect SYSTEM time zone by name in get_timezone_offset_information(). why not to put that logic here, in the dedicated method, like you did for other time zones? > + return; > +} > + > > /* > Get name of time zone > @@ -1128,6 +1125,7 @@ class Time_zone_utc : public Time_zone > uint *error_code) const; > virtual void gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const; > virtual const String * get_name() const; > + virtual void get_timezone_information(struct tz* curr_tz) const; > }; > > > @@ -1175,6 +1173,12 @@ Time_zone_utc::gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const > adjust_leap_second(tmp); > } > > +void > +Time_zone_utc::get_timezone_information(struct tz* curr_tz) const > +{ what is printed for UTC? Please, add a test for that. > + return; > +} > + > > /* > Get name of time zone > @@ -1210,6 +1214,7 @@ class Time_zone_db : public Time_zone > virtual my_time_t TIME_to_gmt_sec(const MYSQL_TIME *t, uint *error_code) const; > virtual void gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const; > virtual const String * get_name() const; > + virtual void get_timezone_information(struct tz* curr_tz) const; > private: > TIME_ZONE_INFO *tz_info; > const String *tz_name; > @@ -1280,6 +1285,16 @@ Time_zone_db::gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const > adjust_leap_second(tmp); > } > > +void > +Time_zone_db::get_timezone_information(struct tz* curr_tz) const > +{ > + curr_tz->seconds_offset= tz_info->revtis->rt_offset; > + curr_tz->is_behind= tz_info->revtis->rt_offset < 0 ? true : false; > + strncpy((char*)(curr_tz->abbrevation), (const char*)(tz_info->chars), > + strlen((const char*)(tz_info->chars))+1); > + curr_tz->abbrevation[strlen((const char*)(tz_info->chars))+1]='\0'; here and below (in Time_zone_offset) 1. why casts? 2. you use strncpy incorrectly, the third argument specifies the size of the buffer, like strncpy(curr_tz->abbrevation, tz_info->chars, sizeof(curr_tz->abbrevation)); but better use strmake, it'll do the zero-termination for you strmake(curr_tz->abbrevation, tz_info->chars, sizeof(curr_tz->abbrevation)-1); > +} > + > > /* > Get name of time zone > @@ -1309,6 +1324,7 @@ class Time_zone_offset : public Time_zone > uint *error_code) const; > virtual void gmt_sec_to_TIME(MYSQL_TIME *tmp, my_time_t t) const; > virtual const String * get_name() const; > + virtual void get_timezone_information(struct tz* curr_tz) const; > /* > This have to be public because we want to be able to access it from > my_offset_tzs_get_key() function > @@ -1434,6 +1450,16 @@ Time_zone_offset::get_name() const > return &name; > } > > +void > +Time_zone_offset::get_timezone_information(struct tz* curr_tz) const > +{ > + curr_tz->seconds_offset= offset; > + const char *name= get_name()->ptr(); > + curr_tz->is_behind= name[0] == '+' ? false : true; > + strncpy((char*)(curr_tz->abbrevation), name, this->get_name()->length()); > + curr_tz->abbrevation[this->get_name()->length()]= '\0'; > +} > + > > static Time_zone_utc tz_UTC; > static Time_zone_system tz_SYSTEM; > diff --git a/sql/tztime.h b/sql/tztime.h > index 6d8af62ecd4..57e4aa2021b 100644 > --- a/sql/tztime.h > +++ b/sql/tztime.h > @@ -26,6 +26,11 @@ > #include "sql_list.h" /* Sql_alloc */ > #include "sql_string.h" /* String */ > > +/* > + String with names of SYSTEM time zone. > +*/ > +static const String tz_SYSTEM_name("SYSTEM", 6, &my_charset_latin1); No, sorry. It's a .h file included in many .cc files. You'll have a separate `static const String` in every single source file that includes this header. > + > class THD; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: 23b15d24eb3: cleanup: add Field::store_yesno
by Sergei Golubchik 31 Aug '23

31 Aug '23

Hi, Nikita, On Aug 31, Nikita Malyavin wrote: > revision-id: 23b15d24eb3 (mariadb-11.0.1-227-g23b15d24eb3) > parent(s): cb37c99dd87 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-08-30 13:18:52 +0400 > message: > > cleanup: add Field::store_yesno I agree, it's a good idea to have a helper for storing YES/NO in the field. But I think Field class itself is a wrong place for it. YES/NO is exclusively INFORMATION_SCHEMA thing and it feels kind of wrong to have it in the base class. I'd do a separate function, like store_yes_no(). Or something. Also, note that you missed few places where it's needed, like pos=(uchar*) ((field->flags & NOT_NULL_FLAG) ? "NO" : "YES"); table->field[6]->store((const char*) pos, strlen((const char*) pos), cs); or if (updatable_view) table->field[5]->store(STRING_WITH_LEN("YES"), cs); else table->field[5]->store(STRING_WITH_LEN("NO"), cs); both in sql_show.cc. There are more. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: Client-server protocol improvement proposal: Connection redirection
by Otto Kekäläinen 28 Aug '23

28 Aug '23

Hi Pali! On Fri, 18 Aug 2023 at 11:47, Pali <pali(a)cpan.org> wrote: > CCing choroba (for Perl's DBD::MariaDB) and replying to developers@ ML. > > I have read proposal and the first thing which worries me is how the > proposal handles security aspect of the whole solution. > ... > If TLS is used then any protocol redirection information must happen > inside of the TLS tunnel. Thanks for the comments! Actually Daniel's submission has already this covered and the redirect works only if the connection is TLS protected: * https://github.com/MariaDB/server/pull/2681, server-side implementation * https://github.com/mariadb-corporation/mariadb-connector-c/pull/226, MariaDB Connector/C implementation

1 0

Re: [MariaDB commits] [PATCH 4/4] MDEV-31273: Precompute binlog checksums
by Marko Mäkelä 25 Aug '23

25 Aug '23

On Fri, Aug 25, 2023 at 10:16 AM Kristian Nielsen via commits <commits(a)lists.mariadb.org> wrote: > > Compute binlog checksums (when enabled) already when writing events > into the statement or transaction caches, where before it was done > when the caches are copied to the real binlog file. This moves the > checksum computation outside of holding LOCK_log, improving > scalabitily. > > At stmt/trx cache write time, the final end_log_pos values are not > known, so with this patch these will be set to 0. [...] > Checksums cannot be pre-computed when binlog encryption is enabled, as > encryption relies on correct end_log_pos to provide part of the > nonce/IV. Checksum pre-computation is also disabled for WSREP/Galera. Have you thought about a format change that would address these issues? In MDEV-14425 https://github.com/MariaDB/server/commit/685d958e38b825ad9829be311f26729ccc… I redesigned the InnoDB log file format in a way that allows checksums to be computed before the final position of the records (the log sequence number of LSN) is known. For encryption, an additional 64-bit nonce will be written; this currently is the LSN at the time the encryption was scheduled. Tablespace identifiers, page numbers and file names are not encrypted, because these were never fully encrypted in innodb_encrypt_tables=ON either. This allows the metadata to be used as initialization vectors, and also avoids the need for backup tools to know any encryption keys. Thanks to MDEV-27774, not only InnoDB computes checksums while not holding any mutex, but multiple threads can copy their mini-transactions to the shared log_sys.buf while holding a shared lock_sys.latch. An exclusive latch will only be needed during a log checkpoint when we need to guarantee that all changes up to a particular LSN have been written to the log buffer. This would not be possible without the format change that was implemented in MDEV-14425. > +/* Convenience macros since ulong is 32 or 64 bit depending on platform. */ > +#if SIZEOF_LONG == 4 > +#define my_atomic_storeul(P, D) my_atomic_store32((int32 volatile *)(P), (D)) In all currently supported MariaDB Server versions, C++11 is available. Is there a particular reason why you would not use std::atomic<size_t> here? Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB plc

2 3

Client-server protocol improvement proposal: Connection redirection
by Lenski, Daniel 18 Aug '23

18 Aug '23

Hi! I would like to propose a new feature in the MariaDB client-server protocol<https://mariadb.com/kb/en/clientserver-protocol>: application-layer redirection of client connections. We want the MariaDB server to be able to tell clients connecting to it, “Sorry, this server is unavailable. Connect to an alternate server instead.” This mechanism is inspired by HTTP 302 (“temporary redirect”) mechanism familiar to all developers of web applications, and is intended to have similar semantics and security properties, since these have now been widely deployed and tested for decades. I have submitted a minimal but viable implementation of this at: * https://github.com/MariaDB/server/pull/2681, server-side implementation * https://github.com/mariadb-corporation/mariadb-connector-c/pull/226, MariaDB Connector/C implementation I am seeking advice and opinions on these, and will proceed to finalize this in the following 1-2 months. In its current form, this implementation allows the MariaDB server administrator to set two variables: * SET GLOBAL SERVER_REDIRECT_MODE={ON,OFF} (the default is OFF) * SET GLOBAL SERVER_REDIRECT_TARGET='my-new-server.example.com' (or 192.168.0.123:3307, or new-server.com:3308, etc) When SERVER_REDIRECT_MODE is set to on, the server will stop accepting new TCP-based connections (unless they are to the extra port<https://mariadb.com/kb/en/thread-pool-in-mariadb/#configuring-the-extra-port>, which is intended for emergency administrative use), and it will instead respond to all clients with an error packet<https://mariadb.com/kb/en/err_packet/>: $ mariadb --host my-server.example.com ERROR 4189 (HY000): |Server is redirecting clients to 'my-new-server.example.com:3307'|my-new-server.example.com:3307 When an appropriately-updated client receives this error packet (error code 4196, message formatted as |Human readable message|<value of SERVER_REDIRECT_TARGET>), it will parse the redirection target out from the error message and attempt a new connection: $ updated-mariadb --host my-server.example.com -e 'select @@hostname' Got server redirect to 'my-new-server.example.com' (port 3307) +---------------------------+ | @@hostname | +---------------------------+ | my-new-server.example.com | +---------------------------+ The feature is gracefully backwards-compatible. Old clients that do not support redirection will fail and show the message in a form from which humans can deduce what happened. New clients that do support it will follow it to the new server. The feature is also designed to be extensible in a forwards-compatible way. The SERVER_REDIRECT_MODE system variable could be expanded to support additional redirection modes. For example, it could be augmented to redirect new connections only from specific IP subnets, or to direct the client to interpret the SERVER_REDIRECT_TARGET variable in a new form (such as a<https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-jdbc-url…> JDBC-compatible mysql:// URL<https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-jdbc-url…> as proposed in MDEV-15935<https://jira.mariadb.org/browse/MDEV-15935>). While researching and designing this connection redirection mechanism, I discovered or rediscovered a couple of major TLS-related security vulnerabilities in the MariaDB Connector/C client [1, 2]. These vulnerabilities affect all MariaDB users who care about the confidentiality and authenticity of their client/server connections, and they become even more severe if a connection redirection mechanism exists — because this entails the client trusting the server to send it instructions for creating a new network connection. Unless these are resolved, any connection redirection mechanism will be susceptible to hijacking and redirection to an attacker-controlled server even if the client thinks it is using TLS/SSL. What are the use cases for this connection redirection feature? Even in this simple, initial form, the feature is already useful for database administrators who want to gracefully shut down a MariaDB server (for example, to upgrade it), and to redirect new client connections to a replica or alternate server that can accept the traffic. Server-side connection redirection can be combined with existing MariaDB server features, such as SET GLOBAL READ_ONLY=ON and KILLing of idle or long-running connections, in order to build a full switchover solution for shunting traffic from one server to another as gracefully as possible. Currently, MariaDB database administrators tend to use proxies and wrappers (such as HikariCP) to redirect client connections from one MariaDB server to another. Supporting this redirection mechanism directly in the server, and in the client-server protocol, will allow many simple use cases of redirection without extra software, making MariaDB easier and simpler to manage for minimal downtime situations. Please post on this mailing list what you think about this suggestion. I am looking forward to polishing this feature and submitting it in a form that meets the quality requirements for new features from MariaDB core developers. Thank you! Daniel Lenski Amazon RDS MySQL/MariaDB engine team --- Notes on TLS vulnerabilities in MariaDB Connector/C: [1] CONC-648<https://jira.mariadb.org/browse/CONC-648>, reported by me on 5 June: when a Connector/C-based client is using connections secured by TLS/SSL, Connector/C will nevertheless receive plain-text, unencrypted error packets prior to the TLS handshake, and present them to the client in a way that’s indistinguishable from TLS-protected error packets sent by the server after the TLS handshake. This longstanding vulnerability is a consequence of the tangled and unsystematic way that the Connector/C codebase interleaves the MariaDB application-layer authentication protocol with the transport-layer security protocol. My standalone mariadb-connector-c PR#223<https://github.com/mariadb-corporation/mariadb-connector-c/pull/223> fixes the most straightforward manifestation of this vulnerability. Until this issue is resolved in Connector/C, it will be trivial for MITM attackers to inject plaintext connection redirection packets into MariaDB client/server connections, and thereby redirect clients to attacker-controlled servers despite those clients’ intention to use only TLS-secured connections. [2] MDEV-28634<https://jira.mariadb.org/browse/MDEV-28634>, reported almost 3 years ago by Geoff Montee: Connector/C clients which specify --ssl without --ssl-verify-server-cert will allow their connections to be silently downgraded to unencrypted, plaintext connections. This vulnerability can be used by on-path attackers to trivially downgrade most MariaDB client-server connections from TLS-encrypted to plaintext. My standalone mariadb-connector-c PR#224<https://github.com/mariadb-corporation/mariadb-connector-c/pull/224> fixes this. It is a backwards-incompatible change in that clients which specify --ssl (and thus clearly expect TLS-secured connections) will no longer succeed in connecting to servers which don’t support TLS. This should be considered a good thing: MariaDB Connector/C-based clients have been misleading their users about whether their connections are guaranteed to be encrypted, and will no longer do so. Essentially all software that purports to secure network connections with TLS have implemented similar changes, most of it about a decade ago when the pervasiveness of such attacks<https://en.wikipedia.org/wiki/Global_surveillance_disclosures_(2013%E2%80%9…> (by ISPs, censors, intelligenceagencies, etc) became very widely-known.

8 28

Re: [Commits] cb06612a9da: MDEV-31655: Parallel replication deadlock victim preference code errorneously removed
by Kristian Nielsen 15 Aug '23

15 Aug '23

Hi Marko, I'd like to ask you to review the below patch for MDEV-31655. https://jira.mariadb.org/browse/MDEV-31655 https://github.com/MariaDB/server/commit/cb06612a9da09a7981ada84768793f2ff3… This patch restores code in InnoDB that was originally part of the parallel replication implementation. It makes InnoDB choose the right victim when two transactions in parallel replication deadlock with each other. The code was erroneously removed as part of the VATS work in InnoDB. I saw that you later deprecated and eventually removed the VATS code. I saw that the InnoDB deadlock code is substantially changed in 10.6, so I will need to rework the patch in that version. But I wanted to ask your opinion about this 10.4 version first. In the patch, note in particular the modification to has_higher_priority(), which I think has the effect of choosing which of two waiting transactions to grant a lock to. I don't remember adding this myself, so it might have been added as part of VATS work. But it sounds appropriate; if T1 and T2 are both waiting for a lock, there is no point in granting it to T2 over T1, as then T2 will just be immediately killed and rolled back by replication. - Kristian. Kristian Nielsen via commits <commits(a)lists.mariadb.org> writes: > revision-id: cb06612a9da09a7981ada84768793f2ff3ef842c (mariadb-10.4.30-3-gcb06612a9da) > parent(s): dbe5c20b755b87f67d87990c3baabc866667e41b > author: Kristian Nielsen > committer: Kristian Nielsen > timestamp: 2023-07-11 00:31:29 +0200 > message: > > MDEV-31655: Parallel replication deadlock victim preference code errorneously removed > > Restore code to make InnoDB choose the second transaction as a deadlock > victim if two transactions deadlock that need to commit in-order for > parallel replication. This code was erroneously removed when VATS was > implemented in InnoDB. > > Also add a test case for InnoDB choosing the right deadlock victim. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > > --- > .../suite/binlog_encryption/rpl_parallel.result | 42 ++++++++++++- > mysql-test/suite/rpl/r/rpl_parallel.result | 42 ++++++++++++- > mysql-test/suite/rpl/t/rpl_parallel.test | 71 +++++++++++++++++++++- > sql/rpl_parallel.cc | 7 ++- > sql/sql_class.cc | 43 +++++++++++++ > storage/innobase/lock/lock0lock.cc | 12 ++++ > storage/innobase/trx/trx0trx.cc | 12 ++++ > 7 files changed, 225 insertions(+), 4 deletions(-) > > diff --git a/mysql-test/suite/binlog_encryption/rpl_parallel.result b/mysql-test/suite/binlog_encryption/rpl_parallel.result > index b75a66a634a..b24ff7ba53d 100644 > --- a/mysql-test/suite/binlog_encryption/rpl_parallel.result > +++ b/mysql-test/suite/binlog_encryption/rpl_parallel.result > @@ -2,6 +2,7 @@ include/master-slave.inc > [connection master] > connection server_2; > SET @old_parallel_threads=@@GLOBAL.slave_parallel_threads; > +SET @old_parallel_mode=@@GLOBAL.slave_parallel_mode; > SET GLOBAL slave_parallel_threads=10; > ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first > include/stop_slave.inc > @@ -1680,13 +1681,52 @@ a > 2000 > SELECT * FROM t2 WHERE a>=2000 ORDER BY a; > a > +MDEV-31655: Parallel replication deadlock victim preference code erroneously removed > +connection server_1; > +CREATE TABLE t7 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; > +BEGIN; > +COMMIT; > +include/save_master_gtid.inc > +connection server_2; > +include/sync_with_master_gtid.inc > +include/stop_slave.inc > +set @@global.slave_parallel_threads= 5; > +set @@global.slave_parallel_mode= conservative; > +SET @old_dbug= @@GLOBAL.debug_dbug; > +SET GLOBAL debug_dbug= "+d,rpl_mdev31655_zero_retries"; > +connection master; > +SET @old_dbug= @@SESSION.debug_dbug; > +SET SESSION debug_dbug="+d,binlog_force_commit_id"; > +SET @commit_id= 1+1000; > +SET @commit_id= 2+1000; > +SET @commit_id= 3+1000; > +SET @commit_id= 4+1000; > +SET @commit_id= 5+1000; > +SET @commit_id= 6+1000; > +SET @commit_id= 7+1000; > +SET @commit_id= 8+1000; > +SET @commit_id= 9+1000; > +SET @commit_id= 10+1000; > +SET SESSION debug_dbug= @old_dbug; > +SELECT COUNT(*), SUM(a*100*b) FROM t7; > +COUNT(*) SUM(a*100*b) > +10 225000 > +include/save_master_gtid.inc > +connection server_2; > +include/start_slave.inc > +include/sync_with_master_gtid.inc > +SET GLOBAL debug_dbug= @old_dbug; > +SELECT COUNT(*), SUM(a*100*b) FROM t7; > +COUNT(*) SUM(a*100*b) > +10 225000 > connection server_2; > include/stop_slave.inc > SET GLOBAL slave_parallel_threads=@old_parallel_threads; > +SET GLOBAL slave_parallel_mode=@old_parallel_mode; > include/start_slave.inc > SET DEBUG_SYNC= 'RESET'; > connection server_1; > DROP function foo; > -DROP TABLE t1,t2,t3,t4,t5,t6; > +DROP TABLE t1,t2,t3,t4,t5,t6,t7; > SET DEBUG_SYNC= 'RESET'; > include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/r/rpl_parallel.result b/mysql-test/suite/rpl/r/rpl_parallel.result > index 9b2e68d366e..ef89d954faa 100644 > --- a/mysql-test/suite/rpl/r/rpl_parallel.result > +++ b/mysql-test/suite/rpl/r/rpl_parallel.result > @@ -2,6 +2,7 @@ include/master-slave.inc > [connection master] > connection server_2; > SET @old_parallel_threads=@@GLOBAL.slave_parallel_threads; > +SET @old_parallel_mode=@@GLOBAL.slave_parallel_mode; > SET GLOBAL slave_parallel_threads=10; > ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first > include/stop_slave.inc > @@ -1679,13 +1680,52 @@ a > 2000 > SELECT * FROM t2 WHERE a>=2000 ORDER BY a; > a > +MDEV-31655: Parallel replication deadlock victim preference code erroneously removed > +connection server_1; > +CREATE TABLE t7 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; > +BEGIN; > +COMMIT; > +include/save_master_gtid.inc > +connection server_2; > +include/sync_with_master_gtid.inc > +include/stop_slave.inc > +set @@global.slave_parallel_threads= 5; > +set @@global.slave_parallel_mode= conservative; > +SET @old_dbug= @@GLOBAL.debug_dbug; > +SET GLOBAL debug_dbug= "+d,rpl_mdev31655_zero_retries"; > +connection master; > +SET @old_dbug= @@SESSION.debug_dbug; > +SET SESSION debug_dbug="+d,binlog_force_commit_id"; > +SET @commit_id= 1+1000; > +SET @commit_id= 2+1000; > +SET @commit_id= 3+1000; > +SET @commit_id= 4+1000; > +SET @commit_id= 5+1000; > +SET @commit_id= 6+1000; > +SET @commit_id= 7+1000; > +SET @commit_id= 8+1000; > +SET @commit_id= 9+1000; > +SET @commit_id= 10+1000; > +SET SESSION debug_dbug= @old_dbug; > +SELECT COUNT(*), SUM(a*100*b) FROM t7; > +COUNT(*) SUM(a*100*b) > +10 225000 > +include/save_master_gtid.inc > +connection server_2; > +include/start_slave.inc > +include/sync_with_master_gtid.inc > +SET GLOBAL debug_dbug= @old_dbug; > +SELECT COUNT(*), SUM(a*100*b) FROM t7; > +COUNT(*) SUM(a*100*b) > +10 225000 > connection server_2; > include/stop_slave.inc > SET GLOBAL slave_parallel_threads=@old_parallel_threads; > +SET GLOBAL slave_parallel_mode=@old_parallel_mode; > include/start_slave.inc > SET DEBUG_SYNC= 'RESET'; > connection server_1; > DROP function foo; > -DROP TABLE t1,t2,t3,t4,t5,t6; > +DROP TABLE t1,t2,t3,t4,t5,t6,t7; > SET DEBUG_SYNC= 'RESET'; > include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_parallel.test b/mysql-test/suite/rpl/t/rpl_parallel.test > index 9ba7a30f2eb..d43cec4df34 100644 > --- a/mysql-test/suite/rpl/t/rpl_parallel.test > +++ b/mysql-test/suite/rpl/t/rpl_parallel.test > @@ -13,6 +13,7 @@ > > --connection server_2 > SET @old_parallel_threads=@@GLOBAL.slave_parallel_threads; > +SET @old_parallel_mode=@@GLOBAL.slave_parallel_mode; > --error ER_SLAVE_MUST_STOP > SET GLOBAL slave_parallel_threads=10; > --source include/stop_slave.inc > @@ -2203,16 +2204,84 @@ SELECT * FROM t1 WHERE a>=2000 ORDER BY a; > SELECT * FROM t2 WHERE a>=2000 ORDER BY a; > > > +--echo MDEV-31655: Parallel replication deadlock victim preference code erroneously removed > +# The problem was that InnoDB would choose the wrong deadlock victim. > +# Create a lot of transactions that can cause deadlocks, and use error > +# injection to check that the first transactions in each group is never > +# selected as deadlock victim. > +--let $rows= 10 > +--let $transactions= 5 > +--let $gcos= 10 > + > +--connection server_1 > +CREATE TABLE t7 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB; > +BEGIN; > +--disable_query_log > +--let $i= 0 > +while ($i < 10) { > + eval INSERT INTO t7 VALUES ($i, 0); > + inc $i; > +} > +--enable_query_log > +COMMIT; > +--source include/save_master_gtid.inc > + > +--connection server_2 > +--source include/sync_with_master_gtid.inc > +--source include/stop_slave.inc > +eval set @@global.slave_parallel_threads= $transactions; > +set @@global.slave_parallel_mode= conservative; > +SET @old_dbug= @@GLOBAL.debug_dbug; > +# This error injection will allow no retries for GTIDs divisible by 1000. > +SET GLOBAL debug_dbug= "+d,rpl_mdev31655_zero_retries"; > + > +--connection master > +SET @old_dbug= @@SESSION.debug_dbug; > +SET SESSION debug_dbug="+d,binlog_force_commit_id"; > + > +--let $j= 1 > +while ($j <= $gcos) { > + eval SET @commit_id= $j+1000; > + --let $i= 0 > + while ($i < $transactions) { > + --disable_query_log > + eval SET SESSION gtid_seq_no= 1000 + 1000*$j + $i; > + BEGIN; > + --let $k= 0 > + while ($k < $rows) { > + eval UPDATE t7 SET b=b+1 WHERE a=(($i+$k) MOD $rows); > + inc $k; > + } > + COMMIT; > + --enable_query_log > + inc $i; > + } > + inc $j; > +} > + > +SET SESSION debug_dbug= @old_dbug; > +SELECT COUNT(*), SUM(a*100*b) FROM t7; > + > +--source include/save_master_gtid.inc > + > +--connection server_2 > +--source include/start_slave.inc > +--source include/sync_with_master_gtid.inc > +SET GLOBAL debug_dbug= @old_dbug; > +SELECT COUNT(*), SUM(a*100*b) FROM t7; > + > + > # Clean up. > --connection server_2 > --source include/stop_slave.inc > SET GLOBAL slave_parallel_threads=@old_parallel_threads; > +SET GLOBAL slave_parallel_mode=@old_parallel_mode; > --source include/start_slave.inc > SET DEBUG_SYNC= 'RESET'; > > --connection server_1 > DROP function foo; > -DROP TABLE t1,t2,t3,t4,t5,t6; > +DROP TABLE t1,t2,t3,t4,t5,t6,t7; > SET DEBUG_SYNC= 'RESET'; > > --source include/rpl_end.inc > diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc > index b550315d69f..1aeb1257c4a 100644 > --- a/sql/rpl_parallel.cc > +++ b/sql/rpl_parallel.cc > @@ -1385,8 +1385,13 @@ handle_rpl_parallel_thread(void *arg) > err= dbug_simulate_tmp_error(rgi, thd);); > if (unlikely(err)) > { > + ulong max_retries= slave_trans_retries; > convert_kill_to_deadlock_error(rgi); > - if (has_temporary_error(thd) && slave_trans_retries > 0) > + DBUG_EXECUTE_IF("rpl_mdev31655_zero_retries", > + if ((rgi->current_gtid.seq_no % 1000) == 0) > + max_retries= 0; > + ); > + if (has_temporary_error(thd) && max_retries > 0) > err= retry_event_group(rgi, rpt, qev); > } > } > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 8ed3f8a9c5e..e6ed7ca1cc4 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5247,6 +5247,49 @@ thd_need_ordering_with(const MYSQL_THD thd, const MYSQL_THD other_thd) > return 0; > } > > + > +/* > + If the storage engine detects a deadlock, and needs to choose a victim > + transaction to roll back, it can call this function to ask the upper > + server layer for which of two possible transactions is prefered to be > + aborted and rolled back. > + > + In parallel replication, if two transactions are running in parallel and > + one is fixed to commit before the other, then the one that commits later > + will be prefered as the victim - chosing the early transaction as a victim > + will not resolve the deadlock anyway, as the later transaction still needs > + to wait for the earlier to commit. > + > + The return value is -1 if the first transaction is prefered as a deadlock > + victim, 1 if the second transaction is prefered, or 0 for no preference (in > + which case the storage engine can make the choice as it prefers). > +*/ > +extern "C" int > +thd_deadlock_victim_preference(const MYSQL_THD thd1, const MYSQL_THD thd2) > +{ > + rpl_group_info *rgi1, *rgi2; > + > + if (!thd1 || !thd2) > + return 0; > + > + /* > + If the transactions are participating in the same replication domain in > + parallel replication, then request to select the one that will commit > + later (in the fixed commit order from the master) as the deadlock victim. > + */ > + rgi1= thd1->rgi_slave; > + rgi2= thd2->rgi_slave; > + if (rgi1 && rgi2 && > + rgi1->is_parallel_exec && > + rgi1->rli == rgi2->rli && > + rgi1->current_gtid.domain_id == rgi2->current_gtid.domain_id) > + return rgi1->gtid_sub_id < rgi2->gtid_sub_id ? 1 : -1; > + > + /* No preferences, let the storage engine decide. */ > + return 0; > +} > + > + > extern "C" int thd_non_transactional_update(const MYSQL_THD thd) > { > return(thd->transaction.all.modified_non_trans_table); > diff --git a/storage/innobase/lock/lock0lock.cc b/storage/innobase/lock/lock0lock.cc > index b1cf2152cd6..469d03eaa06 100644 > --- a/storage/innobase/lock/lock0lock.cc > +++ b/storage/innobase/lock/lock0lock.cc > @@ -49,6 +49,8 @@ Created 5/7/1996 Heikki Tuuri > #include <mysql/service_wsrep.h> > #endif /* WITH_WSREP */ > > +extern "C" int thd_deadlock_victim_preference(const MYSQL_THD thd1, const MYSQL_THD thd2); > + > /** Lock scheduling algorithm */ > ulong innodb_lock_schedule_algorithm; > > @@ -1538,6 +1540,16 @@ static bool has_higher_priority(lock_t *lock1, lock_t *lock2) > } else if (!lock_get_wait(lock2)) { > return false; > } > + // Ask the upper server layer if any of the two trx should be prefered. > + int preference = thd_deadlock_victim_preference(lock1->trx->mysql_thd, > + lock2->trx->mysql_thd); > + if (preference == -1) { > + // lock1 is preferred as a victim, so lock2 has higher priority > + return false; > + } else if (preference == 1) { > + // lock2 is preferred as a victim, so lock1 has higher priority > + return true; > + } > return lock1->trx->start_time_micro <= lock2->trx->start_time_micro; > } > > diff --git a/storage/innobase/trx/trx0trx.cc b/storage/innobase/trx/trx0trx.cc > index 7cd95878b0c..0771d764fb6 100644 > --- a/storage/innobase/trx/trx0trx.cc > +++ b/storage/innobase/trx/trx0trx.cc > @@ -52,6 +52,9 @@ Created 3/26/1996 Heikki Tuuri > #include <set> > #include <new> > > +extern "C" > +int thd_deadlock_victim_preference(const MYSQL_THD thd1, const MYSQL_THD thd2); > + > /** The bit pattern corresponding to TRX_ID_MAX */ > const byte trx_id_max_bytes[8] = { > 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff > @@ -1906,6 +1909,15 @@ trx_weight_ge( > { > ibool a_notrans_edit; > ibool b_notrans_edit; > + int pref; > + > + /* First ask the upper server layer if it has any preference for which > + to prefer as a deadlock victim. */ > + pref= thd_deadlock_victim_preference(a->mysql_thd, b->mysql_thd); > + if (pref < 0) > + return FALSE; > + else if (pref > 0) > + return TRUE; > > /* If mysql_thd is NULL for a transaction we assume that it has > not edited non-transactional tables. */ > _______________________________________________ > commits mailing list -- commits(a)lists.mariadb.org > To unsubscribe send an email to commits-leave(a)lists.mariadb.org

2 7

Re: 9c52eede2c8: MDEV-31646 Online alter applies binlog cache limit to cache writes
by Sergei Golubchik 15 Aug '23

15 Aug '23

Hi, Nikita, On Aug 14, Nikita Malyavin wrote: > revision-id: 9c52eede2c8 (mariadb-11.0.1-175-g9c52eede2c8) > parent(s): 209c894b431 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2023-08-11 16:29:37 +0400 > message: > > MDEV-31646 Online alter applies binlog cache limit to cache writes > > 1. Make online disk writes unlimited, same as filesort does. > 2. Make proper error handling -- in 32-bit build IO_CACHE capacity limit is > 4GB, so it is quite possible to overfill there. > 3. Event_log::write_cache complicated with event reparsing, and as it was > proven by QA, contains some mistakes. Rewrite introbuce a simpler and much > faster version, not featuring reparsing and therefore copying a whole > buffer at once. This also disables checksums and crypto. > > As a result, online alter is untied of several binlog variables, which was > a second aim of this patch. > > diff --git a/mysql-test/main/alter_table_online_debug.result b/mysql-test/main/alter_table_online_debug.result > index 021267a15f5..3a615372ed5 100644 > --- a/mysql-test/main/alter_table_online_debug.result > +++ b/mysql-test/main/alter_table_online_debug.result > @@ -409,23 +409,23 @@ set debug_sync= 'now SIGNAL start_replication'; > set debug_sync= 'now WAIT_FOR applied'; > select stage, progress from INFORMATION_SCHEMA.PROCESSLIST where id = @con; > stage progress > -3 51.220 > +3 53.390 why is that? > set debug_sync= 'now SIGNAL proceed WAIT_FOR applied'; > select stage, progress from INFORMATION_SCHEMA.PROCESSLIST where id = @con; > stage progress > diff --git a/sql/log.cc b/sql/log.cc > index d9e81700463..8c57487d57e 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2286,13 +2286,21 @@ int binlog_log_row_online_alter(TABLE* table, const uchar *before_record, > MY_BITMAP *old_rpl_write_set= table->rpl_write_set; > table->rpl_write_set= &table->s->all_set; > > + table->online_alter_cache->store_prev_position(); > int error= (*log_func)(thd, table, table->s->online_alter_binlog, > - table->online_alter_cache, true, > + table->online_alter_cache, > + table->file->has_transactions_and_rollback(), > before_record, after_record); > > table->rpl_write_set= old_rpl_write_set; > > - return unlikely(error) ? HA_ERR_RBR_LOGGING_FAILED : 0; > + if (unlikely(error)) > + { > + table->online_alter_cache->restore_prev_position(); what is this for? > + return HA_ERR_RBR_LOGGING_FAILED; > + } > + > + return 0; > } > > static void > @@ -7748,7 +7763,7 @@ static int binlog_online_alter_end_trans(THD *thd, bool all, bool commit) > { > DBUG_ASSERT(cache.cache_log.type != READ_CACHE); > mysql_mutex_lock(binlog->get_log_lock()); > - error= binlog->write_cache(thd, &cache.cache_log); > + error= binlog->write_cache_raw(thd, &cache.cache_log); > mysql_mutex_unlock(binlog->get_log_lock()); > } > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1