- developers - lists.mariadb.org

[Maria-developers] Interaction between rpl_slave_state and rpl_binlog_state
by Sachin Setiya 28 Nov '17

28 Nov '17

Hi Kristian, Andrei I have some question related to rpl_slave_state. Suppose A circular async replication between A < -- > B (gtid_ignore_duplicates on) Now, we set some temp server_id on server A , lets say `X`. Now the problem is each event group which originates from A is executed 2 times. For example we insert into table t1 and gtid is 0-X-2. The event goes to slave B B applies it, And send it back to A, Since its server_is different from A global server_id , this event is not filtered in queue_event(). Now event goes to handle_sql thread , And check_duplicate_gtid function is called. Since rpl_slave_state::hash does not have a element whose sequence_no is >= 2, So this event is applied again. Andrei suggested a solution of checking rpl_binlog_state in check_duplicate_gtid, This solution solves some problem but creates some problem. (multi_source.gtid_ignore_duplicates fails in master_sync, test for mdev_10715 also fails ). Reason for this is sync_with_master calls master_gtid_wait , which internally calls gtid_waiting::wait_for_gtid , and this checks rpl_salve_state::hash. Although this method can also be patched for to look for rpl_gtid_global_binlog_state. But I am not sure whether this will solve all problem or create new problem. Another solution might be somehow update rpl_slave_state::hash when we write gtid_event in log. But this does not make sense. rpl_slave_state should be used for slave replication usage. I think we need a more better solution for this. -- Regards sachin

3 4

[Maria-developers] Add hidden level 2/3 (PSEDUO_HIDDEN_COLUMN / COMPLETELY_HIDDEN_COLUMN) to mariadb
by Sachin Setiya 27 Nov '17

27 Nov '17

Hello Everyone! Currently I am implementing 10177. But I have one doubt I am not sure how to implement a function which will create hidden level 2/3 field for server. Suppose we create a function (say add_hidden_field(.....)) which will add the field to table T. But the issue is where should we call this function in mysql_prepare_create_table / mysql_prepare_alter_table or init_from_binary_frm_image Doing it from init_from_binary_frm_image make sense because this way server and slave frm of table T will be same (We have decided that we wont log hidden level 2/3 field to binlog) , But the issue is optimizer wont be ready to decide whether we require a hidden level 3 field at that time. We can also do it from mysql_prepare_create_table , but I dont think this wont be good idea because at this time optimizer wont be ready plus this will complicate binlog handling. Or may be we should have a function which can modify table object any time. But I am not sure if it is possible/good idea ? What you guys think ? -- Regards Sachin Setiya Software Engineer at MariaDB

2 4

Re: [Maria-developers] [Commits] b2571e5: Removed ut_crc32_byte_by_byte: never used
by Marko Mäkelä 23 Nov '17

23 Nov '17

Indeed, the functions and function pointers were never used. On Thu, Nov 23, 2017 at 12:36 PM, Sergey Vojtovich <svoj(a)mariadb.org> wrote: > revision-id: b2571e534f9eed873cc9b1ce5f48b109c1ee0fcc > (mariadb-10.3.2-65-gb2571e5) > parent(s): 0bfe5e349210c01404d20dfc8660d5224bd9c658 > committer: Sergey Vojtovich > timestamp: 2017-11-23 14:35:50 +0400 > message: > > Removed ut_crc32_byte_by_byte: never used > > --- > storage/innobase/include/ut0crc32.h | 4 ---- > storage/innobase/ut/ut0crc32.cc | 48 ------------------------------ > ------- > 2 files changed, 52 deletions(-) > > diff --git a/storage/innobase/include/ut0crc32.h > b/storage/innobase/include/ut0crc32.h > index 36b389b..509ac09 100644 > --- a/storage/innobase/include/ut0crc32.h > +++ b/storage/innobase/include/ut0crc32.h > @@ -51,10 +51,6 @@ extern ut_crc32_func_t ut_crc32; > when converting byte strings to integers internally. */ > extern ut_crc32_func_t ut_crc32_legacy_big_endian; > > -/** Pointer to CRC32-byte-by-byte calculation function (byte order > agnostic, > -but very slow). */ > -extern ut_crc32_func_t ut_crc32_byte_by_byte; > - > extern const char* ut_crc32_implementation; > > #endif /* ut0crc32_h */ > diff --git a/storage/innobase/ut/ut0crc32.cc b/storage/innobase/ut/ > ut0crc32.cc > index 386d32f..6383e6d 100644 > --- a/storage/innobase/ut/ut0crc32.cc > +++ b/storage/innobase/ut/ut0crc32.cc > @@ -93,10 +93,6 @@ ut_crc32_func_t ut_crc32; > when converting byte strings to integers internally. */ > ut_crc32_func_t ut_crc32_legacy_big_endian; > > -/** Pointer to CRC32-byte-by-byte calculation function (byte order > agnostic, > -but very slow). */ > -ut_crc32_func_t ut_crc32_byte_by_byte; > - > /** Text description of CRC32 implementation */ > const char* ut_crc32_implementation; > > @@ -409,26 +405,6 @@ ut_crc32_legacy_big_endian_hw( > > return(~crc); > } > - > -/** Calculates CRC32 using hardware/CPU instructions. > -This function processes one byte at a time (very slow) and thus it does > -not depend on the byte order of the machine. > -@param[in] buf data over which to calculate CRC32 > -@param[in] len data length > -@return CRC-32C (polynomial 0x11EDC6F41) */ > -uint32_t > -ut_crc32_byte_by_byte_hw( > - const byte* buf, > - ulint len) > -{ > - uint32_t crc = 0xFFFFFFFFU; > - > - while (len > 0) { > - ut_crc32_8_hw(&crc, &buf, &len); > - } > - > - return(~crc); > -} > #endif /* defined(__GNUC__) && defined(__x86_64__) */ > > /* CRC32 software implementation. */ > @@ -669,28 +645,6 @@ ut_crc32_legacy_big_endian_sw( > return(~crc); > } > > -/** Calculates CRC32 in software, without using CPU instructions. > -This function processes one byte at a time (very slow) and thus it does > -not depend on the byte order of the machine. > -@param[in] buf data over which to calculate CRC32 > -@param[in] len data length > -@return CRC-32C (polynomial 0x11EDC6F41) */ > -uint32_t > -ut_crc32_byte_by_byte_sw( > - const byte* buf, > - ulint len) > -{ > - uint32_t crc = 0xFFFFFFFFU; > - > - ut_a(ut_crc32_slice8_table_initialized); > - > - while (len > 0) { > - ut_crc32_8_sw(&crc, &buf, &len); > - } > - > - return(~crc); > -} > - > /********************************************************************//** > Initializes the data structures used by ut_crc32*(). Does not do any > allocations, would not hurt if called twice, but would be pointless. */ > @@ -701,7 +655,6 @@ ut_crc32_init() > ut_crc32_slice8_table_init(); > ut_crc32 = ut_crc32_sw; > ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_sw; > - ut_crc32_byte_by_byte = ut_crc32_byte_by_byte_sw; > ut_crc32_implementation = "Using generic crc32 instructions"; > > #if defined(__GNUC__) && defined(__x86_64__) > @@ -735,7 +688,6 @@ ut_crc32_init() > if (features_ecx & 1 << 20) { > ut_crc32 = ut_crc32_hw; > ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_hw; > - ut_crc32_byte_by_byte = ut_crc32_byte_by_byte_hw; > ut_crc32_implementation = "Using SSE2 crc32 instructions"; > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation DON’T MISS M|18 MariaDB User Conference February 26 - 27, 2018 New York City https://m18.mariadb.com/

1 0

Re: [Maria-developers] [Commits] f28af02: Removed HW acceleration for big endian checksum
by Marko Mäkelä 23 Nov '17

23 Nov '17

Looks OK to me. This can only slightly punish the very few people who are upgrading from a buggy MySQL 5.6 or MariaDB 10.0 or 10.1 installation on a big-endian machine (such as SPARC) into a newer version. And those users could invoke innochecksum to rewrite the page checksums. On Thu, Nov 23, 2017 at 12:57 PM, Sergey Vojtovich <svoj(a)mariadb.org> wrote: > revision-id: f28af027a0439d760c6d26eb67a932e97eb665d7 > (mariadb-10.3.2-66-gf28af02) > parent(s): b2571e534f9eed873cc9b1ce5f48b109c1ee0fcc > committer: Sergey Vojtovich > timestamp: 2017-11-23 14:53:16 +0400 > message: > > Removed HW acceleration for big endian checksum > > Big endian checksum is needed to workaround 10+ years old bug, where > checksum > was calculated incorrectly on big endian hardware. We can still checksum > such > tablespaces using software implementation of CRC32. > > --- > storage/innobase/include/ut0crc32.h | 4 +- > storage/innobase/ut/ut0crc32.cc | 89 +----------------------------- > ------- > 2 files changed, 3 insertions(+), 90 deletions(-) > > diff --git a/storage/innobase/include/ut0crc32.h > b/storage/innobase/include/ut0crc32.h > index 509ac09..527a3e81 100644 > --- a/storage/innobase/include/ut0crc32.h > +++ b/storage/innobase/include/ut0crc32.h > @@ -47,9 +47,9 @@ typedef uint32_t (*ut_crc32_func_t)(const byte* > ptr, ulint len); > /** Pointer to CRC32 calculation function. */ > extern ut_crc32_func_t ut_crc32; > > -/** Pointer to CRC32 calculation function, which uses big-endian byte > order > +/** CRC32 calculation function, which uses big-endian byte order > when converting byte strings to integers internally. */ > -extern ut_crc32_func_t ut_crc32_legacy_big_endian; > +extern uint32_t ut_crc32_legacy_big_endian(const byte* buf, ulint len); > > extern const char* ut_crc32_implementation; > > diff --git a/storage/innobase/ut/ut0crc32.cc b/storage/innobase/ut/ > ut0crc32.cc > index 6383e6d..36c52b7 100644 > --- a/storage/innobase/ut/ut0crc32.cc > +++ b/storage/innobase/ut/ut0crc32.cc > @@ -89,10 +89,6 @@ mysys/my_perf.c, contributed by Facebook under the > following license. > /** Pointer to CRC32 calculation function. */ > ut_crc32_func_t ut_crc32; > > -/** Pointer to CRC32 calculation function, which uses big-endian byte > order > -when converting byte strings to integers internally. */ > -ut_crc32_func_t ut_crc32_legacy_big_endian; > - > /** Text description of CRC32 implementation */ > const char* ut_crc32_implementation; > > @@ -238,37 +234,6 @@ ut_crc32_64_hw( > *len -= 8; > } > > -/** Calculate CRC32 over 64-bit byte string using a hardware/CPU > instruction. > -The byte string is converted to a 64-bit integer using big endian byte > order. > -@param[in,out] crc crc32 checksum so far when this function is called, > -when the function ends it will contain the new checksum > -@param[in,out] data data to be checksummed, the pointer will be > advanced > -with 8 bytes > -@param[in,out] len remaining bytes, it will be decremented with 8 */ > -inline > -void > -ut_crc32_64_legacy_big_endian_hw( > - uint32_t* crc, > - const byte** data, > - ulint* len) > -{ > - uint64_t data_int = *reinterpret_cast<const > uint64_t*>(*data); > - > -#ifndef WORDS_BIGENDIAN > - data_int = ut_crc32_swap_byteorder(data_int); > -#else > - /* Currently we only support x86_64 (little endian) CPUs. In case > - some big endian CPU supports a CRC32 instruction, then maybe we > will > - NOT need a byte order swap here. */ > -#error Dont know how to handle big endian CPUs > -#endif /* WORDS_BIGENDIAN */ > - > - *crc = ut_crc32_64_low_hw(*crc, data_int); > - > - *data += 8; > - *len -= 8; > -} > - > /** Calculates CRC32 using hardware/CPU instructions. > @param[in] buf data over which to calculate CRC32 > @param[in] len data length > @@ -355,56 +320,6 @@ ut_crc32_hw( > > return(~crc); > } > - > -/** Calculates CRC32 using hardware/CPU instructions. > -This function uses big endian byte ordering when converting byte sequence > to > -integers. > -@param[in] buf data over which to calculate CRC32 > -@param[in] len data length > -@return CRC-32C (polynomial 0x11EDC6F41) */ > -uint32_t > -ut_crc32_legacy_big_endian_hw( > - const byte* buf, > - ulint len) > -{ > - uint32_t crc = 0xFFFFFFFFU; > - > - /* Calculate byte-by-byte up to an 8-byte aligned address. After > - this consume the input 8-bytes at a time. */ > - while (len > 0 && (reinterpret_cast<uintptr_t>(buf) & 7) != 0) { > - ut_crc32_8_hw(&crc, &buf, &len); > - } > - > - while (len >= 128) { > - /* This call is repeated 16 times. 16 * 8 = 128. */ > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - } > - > - while (len >= 8) { > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - } > - > - while (len > 0) { > - ut_crc32_8_hw(&crc, &buf, &len); > - } > - > - return(~crc); > -} > #endif /* defined(__GNUC__) && defined(__x86_64__) */ > > /* CRC32 software implementation. */ > @@ -600,7 +515,7 @@ integers. > @param[in] len data length > @return CRC-32C (polynomial 0x11EDC6F41) */ > uint32_t > -ut_crc32_legacy_big_endian_sw( > +ut_crc32_legacy_big_endian( > const byte* buf, > ulint len) > { > @@ -654,7 +569,6 @@ ut_crc32_init() > { > ut_crc32_slice8_table_init(); > ut_crc32 = ut_crc32_sw; > - ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_sw; > ut_crc32_implementation = "Using generic crc32 instructions"; > > #if defined(__GNUC__) && defined(__x86_64__) > @@ -687,7 +601,6 @@ ut_crc32_init() > > if (features_ecx & 1 << 20) { > ut_crc32 = ut_crc32_hw; > - ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_hw; > ut_crc32_implementation = "Using SSE2 crc32 instructions"; > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation DON’T MISS M|18 MariaDB User Conference February 26 - 27, 2018 New York City https://m18.mariadb.com/

1 0

Re: [Maria-developers] [Commits] 14bd406: Less code to depend on ut_crc32_init()
by Marko Mäkelä 23 Nov '17

23 Nov '17

Thanks, this looks OK to me. I guess the function ut_crc32_init() will be made optional separately. On Thu, Nov 23, 2017 at 1:22 PM, Sergey Vojtovich <svoj(a)mariadb.org> wrote: > revision-id: 14bd406970c9bb18c744a452359aa4505f325b2d > (mariadb-10.3.2-67-g14bd406) > parent(s): f28af027a0439d760c6d26eb67a932e97eb665d7 > committer: Sergey Vojtovich > timestamp: 2017-11-23 15:22:10 +0400 > message: > > Less code to depend on ut_crc32_init() > > --- > storage/innobase/include/ut0crc32.h | 1 + > storage/innobase/ut/ut0crc32.cc | 20 +++++++------------- > 2 files changed, 8 insertions(+), 13 deletions(-) > > diff --git a/storage/innobase/include/ut0crc32.h > b/storage/innobase/include/ut0crc32.h > index 527a3e81..32ad066 100644 > --- a/storage/innobase/include/ut0crc32.h > +++ b/storage/innobase/include/ut0crc32.h > @@ -51,6 +51,7 @@ extern ut_crc32_func_t ut_crc32; > when converting byte strings to integers internally. */ > extern uint32_t ut_crc32_legacy_big_endian(const byte* buf, ulint len); > > +/** Text description of CRC32 implementation */ > extern const char* ut_crc32_implementation; > > #endif /* ut0crc32_h */ > diff --git a/storage/innobase/ut/ut0crc32.cc b/storage/innobase/ut/ > ut0crc32.cc > index 36c52b7..48f6b82 100644 > --- a/storage/innobase/ut/ut0crc32.cc > +++ b/storage/innobase/ut/ut0crc32.cc > @@ -86,12 +86,6 @@ mysys/my_perf.c, contributed by Facebook under the > following license. > #include "univ.i" > #include "ut0crc32.h" > > -/** Pointer to CRC32 calculation function. */ > -ut_crc32_func_t ut_crc32; > - > -/** Text description of CRC32 implementation */ > -const char* ut_crc32_implementation; > - > /** Swap the byte order of an 8 byte integer. > @param[in] i 8-byte integer > @return 8-byte integer */ > @@ -125,6 +119,13 @@ ut_crc32_power8( > { > return crc32c_vpmsum(0, buf, len); > } > + > +ut_crc32_func_t ut_crc32 = ut_crc32_power8; > +const char* ut_crc32_implementation = "Using POWER8 crc32 > instructions"; > +#else > +uint32_t ut_crc32_sw(const byte* buf, ulint len); > +ut_crc32_func_t ut_crc32 = ut_crc32_sw; > +const char* ut_crc32_implementation = "Using generic crc32 > instructions"; > #endif > > #if defined(__GNUC__) && defined(__x86_64__) > @@ -568,8 +569,6 @@ ut_crc32_init() > /*===========*/ > { > ut_crc32_slice8_table_init(); > - ut_crc32 = ut_crc32_sw; > - ut_crc32_implementation = "Using generic crc32 instructions"; > > #if defined(__GNUC__) && defined(__x86_64__) > uint32_t vend[3]; > @@ -603,10 +602,5 @@ ut_crc32_init() > ut_crc32 = ut_crc32_hw; > ut_crc32_implementation = "Using SSE2 crc32 instructions"; > } > - > -#elif defined(HAVE_CRC32_VPMSUM) > - ut_crc32 = ut_crc32_power8; > - ut_crc32_implementation = "Using POWER8 crc32 instructions"; > #endif > - > } > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation DON’T MISS M|18 MariaDB User Conference February 26 - 27, 2018 New York City https://m18.mariadb.com/

1 0

[Maria-developers] IB: trx_t:: start_time and start_time_micro
by Aleksey Midenkov 21 Nov '17

21 Nov '17

Hello! Different semantics of these 2 members doesn't look good: time_t start_time; /*!< time the state last time became TRX_STATE_ACTIVE */ ib_uint64_t start_time_micro; /*!< start time of transaction in microseconds */ Either `start_time_micro` should be drawn to the base of `start_time` or it should be explicitly commented that it has nothing to do with `start_time`. I.e. there is nothing wrong on algorithmic level, but such data members may (and certainly would) lead developers to misuse of `start_time_micro`. -- All the best, Aleksey Midenkov @midenok

2 2

[Maria-developers] Please review MDEV-11292 Item_func_sp::fix_length_and_dec() does not set collation derivation properly
by Alexander Barkov 20 Nov '17

20 Nov '17

Hello Alexey, Please review a patch for MDEV-11292 (for 10.3). Thanks.

2 2

[Maria-developers] MariaDB GUI: BSL 1.1 - Debate
by Juan Telleria 19 Nov '17

19 Nov '17

Dear MariaDb Developers, I am writing because I would like to open a debate whether if MariaDB, now that has gained investment from Microsoft, Alibaba, and the European Union, shall create its own SQL Graphical User Interface, as good as (or better than) MySQL Workbench. I agree that MariaDB includes by default HeidiSQL in its Windows builds, but in my opinion, if MariaDB really wants to look Oracle's MySQL face to face, HeidiSQL is not nearly as good as MySQL Workbench. If this wants to be done: 1st) The brand new GUI shall have a Business Source Licence v1.1, which only companies which invest in MariaDB would be able to use. 2nd) In 3 years now, MariaDB would be better positioned and have a fully Open Source GUI which would allow MariaDB to look face to face to Oracle. 3rd) And in 4/5 years now, MariaDB Corporation shall be brave enough to to continue Developing this GUI as a truly open source product, without the BSL, and adopting a purely GPLv2 with incremental improvements, distributing it through the MariaDB Foundation Webpage, as individuals or other companies apart from MariaDB Corporation, will not support the product if it is not totally Open Source. So BSL v1.1 might prove useful to launch the project, but with time, it shall have a GPLv2 for gaining investment from third companies. ¿What do you thing on the topic? Thank you all, Juan Telleria

1 0

[Maria-developers] Please review MDEV-13788 Server crash when issuing bad SQL partition syntax
by Alexander Barkov 16 Nov '17

16 Nov '17

Hello Sergei, Can you please review a patch for MDEV-13788, for 5.5? Thanks!

2 3

Re: [Maria-developers] MDEV-10173.
by Sergei Golubchik 15 Nov '17

15 Nov '17

Hi, Alexey! On Nov 14, Alexey Botchkov wrote: > Hi, Serg. > > I'd like your opinion about this issue. > Georg complains about weird column metadata > https://jira.mariadb.org/browse/MDEV-10173 > > Personally i see slightly different result, but basically same thing. > -------------------------------- > MariaDB [(none)]> select from_unixtime("0"); > Field 1: `from_unixtime("0")` > Catalog: `def` > Database: `` > Table: `` > Org_table: `` > Type: DATETIME > Collation: binary (63) > Length: 26 > Max_length: 19 > Decimals: 39 > Flags: BINARY Right, I get used to 31 being NOT_FIXED_DEC. But recently Monty changed it to 39. > +---------------------+ > | from_unixtime("0") | > +---------------------+ > | 1970-01-01 04:00:00 | > +---------------------+ > 1 row in set (0.00 sec) > -------------------------------- > > From what i see it is not a bug. The 'Decimals: 39' is not a real > precision here, but just a hint to be flexible converting the data to > the string. The 'Length: 26' takes into account the possible > 'second_part' of 6 digits. Agree. > Georg said on this: "Well, the client "trusts" the metadata. If there > is decimal 31, it appends 31 decimal places and returned truncation > error the metadata (MYSQL_FIELD) describes the data which are sent > from server to client. If there are no decimal places, it should be > zero. " Yes, the cient trusts the metadata, but if decimals is NOT_FIXED_DEC it means "unknown", not 31 (or 39). If the number of decimals for a timestamp is unknown, the max length is 26: "YYYY-MM-DD hh:mm:ss.uuuuuu" > As an improvement i can see using '7' instead of '39' as an > 'NOT_FIXED_DEC' marker for the Item_temporal_func. Why? One wouldn't be able simply to compare with NOT_FIXED_DEC, it'll be type dependent. > Or we can just document that '39' value so clients interpret it right. Isn't it documented? > Serg, maybe you can suggest something? Close as "Not a bug" :) Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0