March 2011 - discuss - lists.mariadb.org

Re: [Maria-discuss] Dynamic columns Library
by Michael Widenius 05 Apr '11

05 Apr '11

Hi! I am cc:ing maria-discuss as you accidently forgot to also email the patch there. >>>>> "Oleksandr" == Oleksandr Byelkin <sanja(a)askmonty.org> writes: Oleksandr> Hi! Oleksandr> Here it is dynamic columns library. I do not think that you will have time to review it completely, but please look at it. Oleksandr> Some notes: Oleksandr> - I changed indices from int to uint + some other minor changes in prototypes. Oleksandr> - dynamic_column_exists() has no way to report error (what to do?) I assume that same problem exists for all the other functions. Two options: - Change function type from my_bool to int where -1 means something went bad (like bad format for the blob). - Add an 'error' argument to all functions that is set to <> 0 if something goes wrong. I think we should go with first option, as this is how many other MariaDB functions work. Oleksandr> - inability to get length without keeping data in the same order as headers (because we store only offsets) lead to +1 memove call (i.e we prefer size to speed of update) I think this is ok. We can proably avoid the +1 memmove call by combining the call when we add things to the header to when we place the data in it's right place. For example, on insert <h1><h3><data1><data3> When adding h2 we would first do: <h1><h3><data1><data3> -> <h1><h3><data1><place for h2 + data2><data3> -> <h1><h3><data1><place for h2><data2><data3> -> <h1><h2><h3><data1><data2><data3> In other words, one memove of all data (trough 2 calls) Oleksandr> - sorry for absence of comment but I prefer unit testing and finishing the library in time. ok. > === added file 'include/ma_dyncol.h' > --- include/ma_dyncol.h 1970-01-01 00:00:00 +0000 > +++ include/ma_dyncol.h 2011-03-27 22:36:38 +0000 > @@ -0,0 +1,87 @@ > +#ifndef ma_dyncol_h > +#define ma_dyncol_h > + > +#include <my_global.h> You can assume that my_global.h is done. > +#include <my_sys.h> > +#include <m_string.h> > +#include <decimal.h> > +#include <my_decimal_limits.h> > +#include <mysql_time.h> > + > +typedef DYNAMIC_STRING DYNAMIC_COLUMN; > + > +enum enum_dynamic_column_type > +{ > + DYN_COL_NULL= 0, > + DYN_COL_INT, > + DYN_COL_UINT, > + DYN_COL_DOUBLE, > + DYN_COL_STRING, > + DYN_COL_DECIMAL, > + DYN_COL_DATETIME, > + DYN_COL_DATE, > + DYN_COL_TIME > +}; > + > +typedef enum enum_dynamic_column_type DYNAMIC_COLUMN_TYPE; > + > +struct st_dynamic_column_value > +{ > + DYNAMIC_COLUMN_TYPE type; > + union > + { > + long long long_value; > + unsigned long long ulong_value; > + double double_value; > + struct { > + LEX_STRING string_value; > + CHARSET_INFO *charset; > + }; > + struct { > + decimal_digit_t decimal_buffer[DECIMAL_BUFF_LENGTH]; > + decimal_t decimal_value; > + }; > + MYSQL_TIME time_value; > + }; > +}; > + > +typedef struct st_dynamic_column_value DYNAMIC_COLUMN_VALUE; > + > +my_bool dynamic_column_create(DYNAMIC_COLUMN *str, > + uint column_nr, > + DYNAMIC_COLUMN_VALUE *value); > + > +my_bool dynamic_column_create_many(DYNAMIC_COLUMN *str, > + uint column_count, > + uint *column_numbers, > + DYNAMIC_COLUMN_VALUE *values); > + > +my_bool dynamic_column_update(DYNAMIC_COLUMN *org, uint column_nr, > + DYNAMIC_COLUMN_VALUE *value); > + > +my_bool dynamic_column_delete(DYNAMIC_COLUMN *org, uint column_nr); > + > +/* Returns 1 if value exists */ > +my_bool dynamic_column_exists(DYNAMIC_COLUMN *org, uint column_nr); > + > +/* List of not NULL columns */ > +my_bool dynamic_column_list(DYNAMIC_COLUMN *org, DYNAMIC_ARRAY *array_of_uint); > + > +/* > + TODO: ask - return 1 on error, if column do not exists it is NULL > +*/ Yes! Same for all the above functions. > +my_bool dynamic_column_get(DYNAMIC_COLUMN *org, int column_nr, > + DYNAMIC_COLUMN_VALUE *store_it_here); > + > +#define dynamic_column_column_free(V) dynstr_free(V) > + > +#define dynamic_column_value_init(V) (V)->type= DYN_COL_NULL > + > +void dynamic_column_value_free(DYNAMIC_COLUMN_VALUE *value); > + > +/* > + Prepare value for using as decimal > +*/ > +void dynamic_column_prepare_decimal(DYNAMIC_COLUMN_VALUE *value); > + > +#endif > === modified file 'mysys/Makefile.am' > --- mysys/Makefile.am 2010-11-19 21:33:47 +0000 > +++ mysys/Makefile.am 2011-03-24 13:54:12 +0000 > @@ -57,7 +57,7 @@ > my_handler.c my_netware.c my_largepage.c \ > my_memmem.c stacktrace.c \ > my_windac.c my_access.c base64.c my_libwrap.c \ > - wqueue.c > + wqueue.c ma_dyncol.c > libmysys_la_LDFLAGS = $(AM_LDFLAGS) @WRAPLIBS@ > libmysys_la_LIBADD = $(ZLIB_LIBS) > === added file 'mysys/ma_dyncol.c' > --- mysys/ma_dyncol.c 1970-01-01 00:00:00 +0000 > +++ mysys/ma_dyncol.c 2011-03-27 22:36:38 +0000 > @@ -0,0 +1,1084 @@ > +#include <ma_dyncol.h> > + > +/* > + Flag byte bits > + > + 2 bits which determinate size of offset in the header > + > + 1 byte > + 2 byte > + 3 byte > + 3 byte > +*/ > +#define DYNCOL_FLG_OFFSET1 0 > +#define DYNCOL_FLG_OFFSET2 1 > +#define DYNCOL_FLG_OFFSET3 2 > +#define DYNCOL_FLG_OFFSET4 3 I don't see where you use the above or understand how you would use this. > +/* mask to get above bits */ > +#define DYNCOL_FLG_OFFSET 3 > +/* All known flags mask */ > +#define DYNCOL_FLG_KNOWN 3 > + > + > +/* dynamic column size reserve */ > +#define DYNCOL_SYZERESERVE 80 > + > + > +static my_bool dynamic_column_time_add(DYNAMIC_COLUMN *str, > + MYSQL_TIME *value); > +static my_bool dynamic_column_date_add(DYNAMIC_COLUMN *str, > + MYSQL_TIME *value); > +static my_bool > +dynamic_column_time_read_int(DYNAMIC_COLUMN_VALUE *store_it_here, > + uchar *data, size_t length); > +static my_bool > +dynamic_column_date_read_int(DYNAMIC_COLUMN_VALUE *store_it_here, > + uchar *data, size_t length); > + > + > +my_bool dynamic_column_init_str(DYNAMIC_COLUMN *str, size_t size) > +{ > + if (!size) > + size= DYNCOL_SYZERESERVE; > + /* > + Make string with no fields (empty header) > + - First \0 is flags > + - other four \0 is 0 fileds counter > + */ > + if (init_dynamic_string(str, NULL, > + size + 5, DYNCOL_SYZERESERVE)) > + return TRUE; > + return dynstr_append_mem(str, "\0\0\0\0\0", 5); > +} Why is an empty data 5 bytes instead of 1 or even 0? > + > +/* > + How many bytes needed to store val as variable length integer where > + fist bit indicate continuation of the sequence > +*/ > +size_t dynamic_column_var_uint_bytes(ulonglong val) > +{ > + size_t len= 0; > + do > + { > + len++; > + val>>= 7; > + } while (val); > + return len; > +} > + > +my_bool dynamic_column_var_uint_add(DYNAMIC_COLUMN *str, ulonglong val) > +{ > + if (dynstr_realloc(str, 10)) // max what we can use > + return TRUE; > + > + do > + { > + ulonglong rest= val >> 7; > + str->str[str->length++]= ((val & 0x7f) | (rest ? 0x80 : 0x00)); > + val= rest; > + } while (val); > + return FALSE; > +} > +ulonglong dynamic_column_var_uint_get(uchar *data, size_t *len) > +{ > + ulonglong val= 0; > + *len= 0; > + for(;;) > + { > + val+= (((ulonglong)(data[len[0]] & 0x7f)) << (len[0]*7)); > + if (!(data[len[0]] & 0x80)) > + break; > + len[0]++; > + } > + len[0]++; > + return val; > +} > + > +/* > + How many bytes needed to store val as unsigned > +*/ > + > +size_t dynamic_column_uint_bytes(ulonglong val) > +{ > + size_t len= 0; > + do > + { > + len++; > + val>>= 8; > + } while (val); > + return len; > +} > + > +my_bool dynamic_column_uint_add(DYNAMIC_COLUMN *str, ulonglong val) > +{ > + if (dynstr_realloc(str, 8)) // max what we can use > + return TRUE; > + do > + { > + str->str[str->length++]= val & 0xff; > + val>>= 8; > + } while (val); > + return FALSE; > +} Better to change all the _add' columns to take a pointer where to store the data, not a dynamic column. This makes it easier to do things like only one memmove In other words: void dynamic_column_uint_add(uchar *data, ulonglong val) { for ( ; val ; val>>= 8) *data++= val & 0xff; return FALSE; } We must test for val first to get 0 stored in 0 bytes A nice change is that no of the _add primitives can fail anymore, so we can make all of them void. > +my_bool dynamic_column_uint_read(DYNAMIC_COLUMN_VALUE *store_it_here, > + uchar *data, > + size_t length) > +{ > + ulonglong value= 0; > + size_t i; > + for (i= 0; i < length; i++) > + { > + value+= ((ulonglong)data[i]) << (i*8); > + } > + store_it_here->ulong_value= value; > + return FALSE; > +} Send pointer to store_it_here->ulong_value to the function. (Makes it more general) > + > +/* > + How many bytes needed to store val as signed in following encoding: > + 0 -> 0 > + -1 -> 1 > + 1 -> 2 > + -2 -> 3 > + 2 -> 4 > + ... > +*/ > + > +size_t dynamic_column_sint_bytes(longlong val) > +{ > + return dynamic_column_uint_bytes((val << 1) ^ > + (val < 0 ? ULL(0xffffffffffffffff) : 0)); > +} > + > +my_bool dynamic_column_sint_add(DYNAMIC_COLUMN *str, longlong val) > +{ > + return dynamic_column_uint_add(str, > + (val << 1) ^ > + (val < 0 ? ULL(0xffffffffffffffff) : 0)); > +} > +my_bool dynamic_column_sint_read(DYNAMIC_COLUMN_VALUE *store_it_here, > + uchar *data, > + size_t length) > +{ > + ulonglong val; > + dynamic_column_uint_read(store_it_here, data, length); > + val= store_it_here->ulong_value; > + if (val & 1) > + val= (val >> 1) ^ ULL(0xffffffffffffffff); > + else > + val>>= 1; > + store_it_here->long_value= (longlong)(val); > + return FALSE; > +} Quite clever. The idea I had to store these are: - Store positive integers as DYN_COL_UINT - Store negative numbers as DYN_COL_INT, which just makes them positive and uses the DYN_COL_UINT functions to store the. > + > +size_t dynamic_column_value_len(DYNAMIC_COLUMN_VALUE *value) > +{ > + switch (value->type) { > + case DYN_COL_NULL: > + return 0; > + case DYN_COL_INT: > + return dynamic_column_sint_bytes(value->long_value); > + case DYN_COL_UINT: > + return dynamic_column_uint_bytes(value->ulong_value); > + case DYN_COL_DOUBLE: > + return 8; > + case DYN_COL_STRING: > + return (dynamic_column_var_uint_bytes(value->charset->number) + > + value->string_value.length); > + case DYN_COL_DECIMAL: > + return (dynamic_column_var_uint_bytes(value->decimal_value.intg) + > + dynamic_column_var_uint_bytes(value->decimal_value.frac) + > + decimal_bin_size(value->decimal_value.intg + > + value->decimal_value.frac, > + value->decimal_value.frac)); We should probably store the value of dynamic_column_var_uint_bytes(...) cols in a parameter (size_t lengths[3]) to make things easier for the caller. > + case DYN_COL_DATETIME: > + /* date+time in bits: 14 + 4 + 5 + 5 + 6 + 6 + 20 + 1 61bits = 8 bytes */ > + return 8; > + case DYN_COL_DATE: > + /* date in dits: 14 + 4 + 5 = 23bits ~= 3bytes*/ > + return 3; > + case DYN_COL_TIME: > + /* time in bits: 5 + 6 + 6 + 20 + 1 = 38bits ~= 5bytes*/ > + return 5; > + default: > + DBUG_ASSERT(0); > + return 0; > + } > +} > + > +my_bool dynamic_column_double_add(DYNAMIC_COLUMN *str, double val) > +{ > + if (dynstr_realloc(str, 8)) > + return TRUE; > + float8store(str->str + str->length, val); > + str->length+= 8; > + return FALSE; > +} See comment for adding to integer for how to change this > + > +my_bool dynamic_column_double_read(DYNAMIC_COLUMN_VALUE *store_it_here, > + uchar *data, > + size_t length) > +{ > + if (length != 8) > + return TRUE; > + float8get(store_it_here->double_value, data); > + return FALSE; > +} > + > + > +my_bool dynamic_column_string_add(DYNAMIC_COLUMN *str, LEX_STRING *string, > + CHARSET_INFO *charset) > +{ > + if (dynamic_column_var_uint_add(str, charset->number)) > + return TRUE; > + return dynstr_append_mem(str, string->str, string->length); > +} If we just send data, and not DYNAMIC_COLUMN's, this should be changed to: my_bool dynamic_column_string_add(uchar *data, LEX_STRING *string, uint charset_number, size_t charset_storage_size) { dynamic_column_var_uint_add(data, charset_number)) memmove(data + charset_storage_size, string->str, string->length); } > + > +my_bool dynamic_column_string_read(DYNAMIC_COLUMN_VALUE *store_it_here, > + uchar *data, > + size_t length) > +{ > + size_t len; > + store_it_here->charset= get_charset(dynamic_column_var_uint_get(data, &len), > + MYF(0)); > + if (store_it_here->charset == NULL || len > length) > + return FALSE; Should return -1 > + data+= len; > + store_it_here->string_value.length= (length-= len); > + if (!(store_it_here->string_value.str= malloc(length + 1))) > + return FALSE; > + store_it_here->type= DYN_COL_STRING; // just to correctly destroy the value > + memcpy(store_it_here->string_value.str, data, length); > + store_it_here->string_value.str[length]= '\0'; // just to be safe Don't malloc it. Just return a pointer to the string in the dynamic data! (Yes, now \0, but we can live with that for the additional speed of not having to do malloc for every access). > + return TRUE; > +} > + > + > +my_bool dynamic_column_decimal_add(DYNAMIC_COLUMN *str, > + decimal_t *value) > +{ > + uint bin_size= decimal_bin_size(value->intg + value->frac, value->frac); > + if (dynamic_column_var_uint_add(str, value->intg) || > + dynamic_column_var_uint_add(str, value->frac) || > + dynstr_realloc(str, bin_size)) > + return TRUE; > + decimal2bin(value, (uchar *)str->str + str->length, value->intg + value->frac, > + value->frac); > + str->length+= bin_size; > + return FALSE; > +} To change this, we need to know the sizes for the intg and frac lengths. Can be done by storing them in an array when we calculate them. C > + > +void dynamic_column_prepare_decimal(DYNAMIC_COLUMN_VALUE *value) > +{ > + value->decimal_value.buf= value->decimal_buffer; > + value->decimal_value.len= DECIMAL_BUFF_LENGTH; > + // just to be safe > + value->type= DYN_COL_DECIMAL; > + decimal_make_zero(&value->decimal_value); > +} > + > +my_bool dynamic_column_decimal_read(DYNAMIC_COLUMN_VALUE *store_it_here, > + uchar *data, > + size_t length __attribute__((unused))) > +{ > + size_t len; > + uint intg, frac; > + > + dynamic_column_prepare_decimal(store_it_here); > + intg= dynamic_column_var_uint_get(data, &len); > + data+= len; > + frac= dynamic_column_var_uint_get(data, &len); > + data+= len; > + if (bin2decimal(data, &store_it_here->decimal_value, intg + frac, frac) != > + E_DEC_OK) > + return TRUE; > + return FALSE; > +} > + > + > +my_bool dynamic_column_date_time_add(DYNAMIC_COLUMN *str, MYSQL_TIME *value) > +{ > + /* > + 0<----year----><mn><day>00!<hr-><min-><sec-><---microseconds---> > + 12345678901234123412345 11234512345612345612345678901234567890 > + <123456><123456><123456><123456><123456><123456><123456><123456> > + */ > + return (dynamic_column_date_add(str, value) || > + dynamic_column_time_add(str, value)); > +} > + > + > +my_bool dynamic_column_date_time_read(DYNAMIC_COLUMN_VALUE *store_it_here, > + uchar *data, > + size_t length) > +{ > + /* > + 0<----year----><mn><day>00!<hr-><min-><sec-><---microseconds---> > + 12345678901234123412345 11234512345612345612345678901234567890 > + <123456><123456><123456><123456><123456><123456><123456><123456> > + */ > + if (length != 8) > + goto err; > + store_it_here->time_value.time_type= MYSQL_TIMESTAMP_DATETIME; > + if (dynamic_column_date_read_int(store_it_here, data, 3) || > + dynamic_column_time_read_int(store_it_here, data + 3, 5)) > + goto err; > + return FALSE; > + > +err: > + store_it_here->time_value.time_type= MYSQL_TIMESTAMP_ERROR; > + return TRUE; > +} > + > + > +my_bool dynamic_column_time_add(DYNAMIC_COLUMN *str, MYSQL_TIME *value) > +{ > + uchar *buf= ((uchar *)str->str) + str->length; > + if (dynstr_realloc(str, 5)) > + return TRUE; > + if (value->time_type == MYSQL_TIMESTAMP_NONE || > + value->time_type == MYSQL_TIMESTAMP_ERROR || > + value->time_type == MYSQL_TIMESTAMP_DATE) > + value->neg= value->second_part= value->hour= > + value->minute= value->second= 0; > + DBUG_ASSERT(value->hour <= 23); note that for time arguments, hour can be bigger than 23 ! (as this is elapsed time, not a wall-clock time) > + DBUG_ASSERT(value->minute <= 59); > + DBUG_ASSERT(value->second <= 59); > + DBUG_ASSERT(value->second_part <= 999999); > + /* > + 00!<hr-><min-><sec-><---microseconds---> > + 11234512345612345612345678901234567890 > + <123456><123456><123456><123456><123456> > + */ > + buf[0]= (value->second_part & 0xff); > + buf[1]= ((value->second_part & 0xff00) >> 8); > + buf[2]= (((value->second & 0xf) << 4) | > + ((value->second_part & 0xf0000) >> 16)); > + buf[3]= ((value->minute << 2) | ((value->second & 0x30) >> 4)); > + buf[4]= ((value->neg ? 0x20 : 0) | value->hour); Good but hard to read and verify. Another way would to store this in an uint and then just use int5store() to store it. We need to store at least the same precision of time as MariaDB is doing in a normal field. This means a hour up to TIME_MAX_HOUR which is 838, which means you need to use another byte for the hour. > + str->length+= 5; > + return FALSE; > +} > + > + > +my_bool dynamic_column_time_read(DYNAMIC_COLUMN_VALUE *store_it_here, > + uchar *data, > + size_t length) > +{ > + store_it_here->time_value.year= store_it_here->time_value.month= > + store_it_here->time_value.day= 0; No reason to reset the parts. Either the function should work or we should get an error. > + store_it_here->time_value.time_type= MYSQL_TIMESTAMP_TIME; > + return dynamic_column_time_read_int(store_it_here, data, length); > +} > + > + > +my_bool dynamic_column_time_read_int(DYNAMIC_COLUMN_VALUE *store_it_here, > + uchar *data, > + size_t length) > +{ > + if (length != 5) > + goto err; > + /* > + 00!<hr-><min-><sec-><---microseconds---> > + 11234512345612345612345678901234567890 > + <123456><123456><123456><123456><123456> > + */ > + store_it_here->time_value.second_part= (data[0] | > + (data[1] << 8) | > + ((data[2] & 0xf) << 16)); > + store_it_here->time_value.second= ((data[2] >> 4) | > + ((data[3] & 0x3) << 4)); > + store_it_here->time_value.minute= (data[3] >> 2); > + store_it_here->time_value.hour= (data[4] & 0x1f); > + store_it_here->time_value.neg= ((data[4] & 0x20) ? 1 : 0); > + if (store_it_here->time_value.second > 59 || > + store_it_here->time_value.minute > 59 || > + store_it_here->time_value.hour > 23 || > + store_it_here->time_value.second_part > 999999) > + goto err; > + return FALSE; Hour should not be checked. > + > +err: > + store_it_here->time_value.time_type= MYSQL_TIMESTAMP_ERROR; > + return TRUE; > +} > + > + > +my_bool dynamic_column_date_add(DYNAMIC_COLUMN *str, MYSQL_TIME *value) > +{ > + uchar *buf= ((uchar *)str->str) + str->length; > + if (dynstr_realloc(str, 3)) > + return TRUE; > + if (value->time_type == MYSQL_TIMESTAMP_NONE || > + value->time_type == MYSQL_TIMESTAMP_ERROR || > + value->time_type == MYSQL_TIMESTAMP_TIME) > + value->year= value->month= value->day = 0; > + DBUG_ASSERT(value->year <= 9999); > + DBUG_ASSERT(value->month <= 12); > + DBUG_ASSERT(value->day <= 31); > + /* > + 0<----year----><mn><day> > + 012345678901234123412345 > + <123456><123456><123456> > + */ > + buf[0]= (value->day | > + ((value->month & 0x7) << 5)); > + buf[1]= ((value->month >> 3) | ((value->year & 0x7F) << 1)); > + buf[2]= (value->year >> 7); > + str->length+= 3; > + return FALSE; > +} > + > +my_bool dynamic_column_date_read(DYNAMIC_COLUMN_VALUE *store_it_here, > + uchar *data, > + size_t length) > +{ > + store_it_here->time_value.neg= store_it_here->time_value.second_part= > + store_it_here->time_value.hour= store_it_here->time_value.minute= > + store_it_here->time_value.second= 0; Don't reset the above. > + store_it_here->time_value.time_type= MYSQL_TIMESTAMP_DATE; > + return dynamic_column_date_read_int(store_it_here, data, length); > +} > + > +my_bool dynamic_column_date_read_int(DYNAMIC_COLUMN_VALUE *store_it_here, > + uchar *data, > + size_t length) > +{ > + if (length != 3) > + goto err; > + /* > + 0<----year----><mn><day> > + 12345678901234123412345 > + <123456><123456><123456> > + */ > + store_it_here->time_value.day= (data[0] & 0x1f); > + store_it_here->time_value.month= (((data[1] & 0x1) << 3) | > + (data[0] >> 5)); > + store_it_here->time_value.year= ((((uint)data[2]) << 7) | > + (data[1] >> 1)); > + if (store_it_here->time_value.day > 31 || > + store_it_here->time_value.month > 12 || > + store_it_here->time_value.year > 9999) > + goto err; > + return FALSE; > + > +err: > + store_it_here->time_value.time_type= MYSQL_TIMESTAMP_ERROR; > + return TRUE; > +} > + > + > +my_bool data_add(DYNAMIC_COLUMN *str, DYNAMIC_COLUMN_VALUE *value) > +{ Change to take uchar *data instead of DYNAMIC_COLUMN > + switch (value->type) { > + case DYN_COL_INT: > + return dynamic_column_sint_add(str, value->long_value); > + case DYN_COL_UINT: > + return dynamic_column_uint_add(str, value->ulong_value); > + case DYN_COL_DOUBLE: > + return dynamic_column_double_add(str, value->double_value); > + case DYN_COL_STRING: > + return dynamic_column_string_add(str, &value->string_value, > + value->charset); > + case DYN_COL_DECIMAL: > + return dynamic_column_decimal_add(str, &value->decimal_value); > + case DYN_COL_DATETIME: > + /* date+time in bits: 14 + 4 + 5 + 5 + 6 + 6 40bits = 5 bytes */ > + return dynamic_column_date_time_add(str, &value->time_value); > + case DYN_COL_DATE: > + /* date in dits: 14 + 4 + 5 = 23bits ~= 3bytes*/ > + return dynamic_column_date_add(str, &value->time_value); > + case DYN_COL_TIME: > + /* time in bits: 5 + 6 + 6 = 17bits ~= 3bytes*/ > + return dynamic_column_time_add(str, &value->time_value); > + default: > + DBUG_ASSERT(0); > + return TRUE; > + } > +} > + > + > +size_t dynamic_column_offset_bytes(size_t data_length) > +{ > + if (data_length < 0x1f) // all 1 value is reserved > + return 1; > + if (data_length < 0x1fff) // all 1 value is reserved > + return 2; > + if (data_length < 0x1fffff) // all 1 value is reserved > + return 3; > + if (data_length < 0x1fffffff) // all 1 value is reserved > + return 4; > + return 5; > +} > + > + > +void type_and_offset_store(uchar *place, size_t offset_size, > + DYNAMIC_COLUMN_TYPE type, > + size_t offset) > +{ > + ulong val = (((ulong) offset) << 3) | (type - 1);; Make 3 a define. Remove double ; > + DBUG_ASSERT(type != DYN_COL_NULL); > + DBUG_ASSERT(((type - 1) & (~7)) == 0); // fit in 3 bits type <= 8 would also work... > + switch (offset_size) { > + case 1: > + DBUG_ASSERT(offset < 0x1f); // all 1 value is reserved > + place[0]= (uchar)val; > + break; > + case 2: > + DBUG_ASSERT(offset < 0x1fff); // all 1 value is reserved > + int2store(place, val); > + break; > + case 3: > + DBUG_ASSERT(offset < 0x1fffff); // all 1 value is reserved > + int3store(place, val); > + break; > + case 4: > + DBUG_ASSERT(offset < 0x1fffffff); // all 1 value is reserved > + int4store(place, val); > + break; > + default: > + DBUG_ASSERT(0); // impossible > + } > +} > + > + > +void type_and_offset_read(DYNAMIC_COLUMN_TYPE *type, > + size_t *offset, > + uchar *place, size_t offset_size) > +{ > + ulong val; > + switch (offset_size) { > + case 1: > + val= (ulong)place[0]; > + break; > + case 2: > + val= uint2korr(place); > + break; > + case 3: > + val= uint3korr(place); > + break; > + case 4: > + val= uint4korr(place); > + break; > + default: > + DBUG_ASSERT(0); // impossible > + } > + *type= (val & 0x7) + 1; > + *offset= val >> 3; > +} > + > + > +int column_sort(const void *a, const void *b) > +{ > + return **((uint **)a) - **((uint **)b); > +} > + > +#define FIXED_HEADER_SIZE 5 > + > +my_bool dynamic_new_column_add(DYNAMIC_COLUMN *str, > + size_t header_size, > + size_t offset_size, > + uint column_count, > + uint non_null_count, > + size_t data_size, > + uint *column_numbers, > + DYNAMIC_COLUMN_VALUE *values) > +{ > + uchar *header_end; > + uint **columns_order; > + uint i; > + uint entry_size= 2 + offset_size; > + my_bool err= TRUE; > + bzero(str, sizeof(DYNAMIC_COLUMN)); // just to make dynstr_free() working > + if (!(columns_order= malloc(sizeof(uint*)*column_count))) > + return TRUE; > + if (dynamic_column_init_str(str, > + data_size + header_size + DYNCOL_SYZERESERVE)) > + goto err; > + > + /* sort columns for the header */ > + for (i= 0; i < column_count; i++) > + columns_order[i]= column_numbers + i; > + qsort(columns_order, (size_t)column_count, sizeof(uint*), &column_sort); > + /* check numbers */ -> Change to 'ensure that there is no duplicates' > + for (i= 0; i < column_count - 1; i++) > + if (columns_order[i][0] == columns_order[i+1][0]) > + goto err; > + > + DBUG_ASSERT(str->length == FIXED_HEADER_SIZE); > + str->str[0]|= (offset_size - 1); // size of offset > + int4store(str->str + 1, non_null_count); // columns number Lets limit the number of columns to 65536 (2 bytes) Don't understand why we store non_null_count in header Don't we need to store total number of columns ? I think we should store also null's in the dynamic columns as someone may want to distinguish between a non existing columns and a column with not a set value. We could store this either as an extended type or as datetime with size 0. > + DBUG_ASSERT(str->max_length > str->length + header_size); > + str->length+= header_size; // reserve place for header > + header_end= (uchar *)str->str + FIXED_HEADER_SIZE; > + for (i= 0; i < column_count; i++) > + { > + uint ord= columns_order[i] - column_numbers; > + if (values[ord].type != DYN_COL_NULL) > + { Better to not skip these. > + int2store(header_end, column_numbers[ord]); > + type_and_offset_store(header_end + 2, offset_size, make '2' a define > + values[ord].type, > + str->length - header_size - FIXED_HEADER_SIZE); > + if (data_add(str, values + ord)) > + goto err; Add a note that this can only fail for wrong data, not for memory allocation as everything is already allocated. > + header_end+= entry_size; > + } > + } > + err= FALSE; > +err: > + free(columns_order); > + return err; > +} > + > + > +my_bool dynamic_column_create_many(DYNAMIC_COLUMN *str, > + uint column_count, > + uint *column_numbers, > + DYNAMIC_COLUMN_VALUE *values) > +{ > + size_t data_size= 0; > + size_t header_size, offset_size; > + uint i; > + int not_null_column_count= column_count; > + for (i= 0; i < column_count; i++) > + { > + data_size+= dynamic_column_value_len(values + i); It would be good to store the parts lengths for the values somewhere. Same as for the 'order' parts. One nice way to do this would be to add some 'private' members to the DYANMIC_COLUMN_VALUE struct: uint length[2] /* To store part lengths when packing values */ uint column_number; /* Temp value for column number */ > + if (values[i].type == DYN_COL_NULL) > + not_null_column_count--; > + } > + if ((offset_size= dynamic_column_offset_bytes(data_size)) >= 5) > + { > + /* TODO: error - no more space for data */ > + return TRUE; > + } > + /* header entry is column number + offset&type */ > + header_size= not_null_column_count * (offset_size + 2); > + > + return dynamic_new_column_add(str, > + header_size, offset_size, > + column_count, > + not_null_column_count, > + data_size, > + column_numbers, values); > +} > + > + > +my_bool dynamic_column_create(DYNAMIC_COLUMN *str, uint column_nr, > + DYNAMIC_COLUMN_VALUE *value) > +{ > + return dynamic_column_create_many(str, 1, &column_nr, value); > +} We also need functions: Initialize the DYNAMIC_COLUMN. (Basicly just a call to init_dynamic_string()) > + > +int header_compar(const void *a, const void *b) > +{ > + uint va= uint2korr((uchar*)a), vb= uint2korr((uchar*)b); > + return (va > vb ? 1 : (va < vb ? -1 : 0)); > +} > + /* If the column doesn't exist, *entry_pos contains the place in the header where the data should be stored. */ > +my_bool find_column(DYNAMIC_COLUMN_TYPE *type, uchar **data, size_t *length, > + uchar *header, size_t offset_size, uint column_count, > + uchar *data_end, uint num, uchar **entry_pos) > +{ > + uchar *entry; > + size_t offset, offset_next; > + size_t header_size, entry_size; > + DYNAMIC_COLUMN_TYPE type_next; > + uchar key[2+4]; > + if (!entry_pos) > + entry_pos= &entry; > + > + entry_size= 2 + offset_size; > + header_size= entry_size * column_count; > + int2store(key, num); > + entry= bsearch(key, header, (size_t)column_count, entry_size, > + &header_compar); Change to: uint mid, start, end; int flag; start= 0; end= column_count -1; mid= 1; while (start != end) { uint val; mid= (start + end) / 2; val= uint2korr(header+ mid * entry_size); if ((flag= CMP_NUM(num, val)) >= 0) end= mid; else start= mid+1; } if (start != mid) { val= uint2korr(header+ start * entry_size); flag= CMP_NUM(num, val); } entry= header+ start * entry_size; if (flag < 0) entry+= entry_size; /* Point at next bigger key */ > + if (!entry) This should now be flag != 0 > + { > + *type= DYN_COL_NULL; > + *entry_pos= NULL; > + return FALSE; > + } > + type_and_offset_read(type, &offset, entry + 2, offset_size); > + *data= header + header_size + offset; > + DBUG_ASSERT(*data < data_end); Should not be an assert, but an error condition. Add /* If not last entry */ > + if (entry + entry_size < header + header_size) > + { > + type_and_offset_read(&type_next, &offset_next, entry + entry_size + 2, > + offset_size); > + *length= offset_next - offset; > + } > + else > + *length= data_end - *data; > + *entry_pos= entry; > + return FALSE; > +} > + > +my_bool dynamic_column_get(DYNAMIC_COLUMN *str, int column_nr, > + DYNAMIC_COLUMN_VALUE *store_it_here) > +{ > + uchar *data; > + size_t offset_size, length; > + uint column_count; > + > + if (!str || str->length < FIXED_HEADER_SIZE || > + (str->str[0] & (~DYNCOL_FLG_KNOWN))) > + goto err; Having a length 0 string should be seen as ok. > + > + offset_size= (str->str[0] & DYNCOL_FLG_OFFSET) + 1; > + column_count= uint4korr(str->str + 1); > + > + if (find_column(&store_it_here->type, &data, &length, (uchar*)str->str + 5, > + offset_size, column_count, (uchar*)str->str + str->length, > + column_nr, NULL)) > + goto err; > + > + switch (store_it_here->type) > + { > + case DYN_COL_INT: > + dynamic_column_sint_read(store_it_here, data, length); > + break; > + case DYN_COL_UINT: > + dynamic_column_uint_read(store_it_here, data, length); > + break; > + case DYN_COL_DOUBLE: > + dynamic_column_double_read(store_it_here, data, length); > + break; > + case DYN_COL_STRING: > + dynamic_column_string_read(store_it_here, data, length); > + break; > + case DYN_COL_DECIMAL: > + dynamic_column_decimal_read(store_it_here, data, length); > + break; > + case DYN_COL_DATETIME: > + dynamic_column_date_time_read(store_it_here, data, length); > + break; > + case DYN_COL_DATE: > + dynamic_column_date_read(store_it_here, data, length); > + break; > + case DYN_COL_TIME: > + dynamic_column_time_read(store_it_here, data, length); > + break; > + case DYN_COL_NULL: > + break; > + default: > + goto err; > + } > + return FALSE; > + > +err: > + store_it_here->type= DYN_COL_NULL; Wouldn't it be better to set: store_it_here->type= DYN_COL_ERROR; > + return TRUE; > +} > + > + > +void dynamic_column_value_free(DYNAMIC_COLUMN_VALUE *value) > +{ > + switch (value->type) > + { > + case DYN_COL_INT: > + case DYN_COL_UINT: > + case DYN_COL_DOUBLE: > + case DYN_COL_DECIMAL: > + case DYN_COL_DATETIME: > + case DYN_COL_DATE: > + case DYN_COL_TIME: > + case DYN_COL_NULL: > + break; // no need cleanup; > + case DYN_COL_STRING: > + free(value->string_value.str); No need for the above as we will point directly to the data. > + value->string_value.str= NULL; //just to be safe > + value->type= DYN_COL_NULL; > + break; > + default: > + DBUG_ASSERT(0); > + } > +} > + > + For this one we proably need a couple of different return values: -1 wrong data in DYNAMIC_COLUMN 0 Column was not part of data 1 Column was part of data and is now deleted. > +my_bool dynamic_column_delete(DYNAMIC_COLUMN *str, uint column_nr) > +{ > + uchar *data, *header_entry, *read, *write; > + size_t offset_size, new_offset_size, length, entry_size, new_entry_size, > + header_size, new_header_size, data_size, new_data_size, > + deleted_entry_offset; > + uint column_count, i; > + DYNAMIC_COLUMN_TYPE type; > + > + if (!str || str->length < FIXED_HEADER_SIZE || > + (str->str[0] & (~DYNCOL_FLG_KNOWN))) > + return TRUE; > + > + offset_size= (str->str[0] & DYNCOL_FLG_OFFSET) + 1; > + column_count= uint4korr(str->str + 1); > + > + if (column_count == 0) > + return FALSE; // no columns > + > + if (find_column(&type, &data, &length, (uchar*)str->str + 5, > + offset_size, column_count, (uchar*)str->str + str->length, > + column_nr, &header_entry)) > + return TRUE; We need to have find_column to have different return values for wrong data, did not find it, found it. With this we can do the above as: if ((res= find_column()) <= 0) return res; > + > + if (type == DYN_COL_NULL) > + return FALSE; // no such column Which allows us to skip the above. > + if (column_count == 1) > + { > + /* delete the only column */ > + str->length= FIXED_HEADER_SIZE; > + bzero(str->str, FIXED_HEADER_SIZE); > + return FALSE; Better to make the string 0 size. > + } > + > + entry_size= 2 + offset_size; > + header_size= entry_size * column_count; > + data_size= str->length - FIXED_HEADER_SIZE - header_size; > + > + new_data_size= data_size - length; > + new_offset_size= dynamic_column_offset_bytes(new_data_size); > + DBUG_ASSERT(new_offset_size <= offset_size); > + new_entry_size= 2 + new_offset_size; > + new_header_size= new_entry_size * (column_count - 1); > + > + deleted_entry_offset= (data - (uchar*)str->str) - > + header_size - FIXED_HEADER_SIZE; > + > + > + /* rewrite header*/ > + str->str[0]|= (new_offset_size - 1); // size of offset > + int4store(str->str + 1, column_count - 1); // columns number > + for (i= 0, write= read= (uchar *)str->str + FIXED_HEADER_SIZE; > + i < column_count; > + i++, read+= entry_size, write+= new_entry_size) > + { > + size_t offs; > + uint nm; > + DYNAMIC_COLUMN_TYPE tp; > + if (read == header_entry) > + { > +#ifndef DBUG_OFF > + nm= uint2korr(read); > + type_and_offset_read(&tp, &offs, read + 2, > + offset_size); > + DBUG_ASSERT(nm == column_nr); > + DBUG_ASSERT(offs == deleted_entry_offset); > +#endif > + write-= new_entry_size; // do not move writer > + continue; // skip removed field > + } > + > + nm= uint2korr(read), > + type_and_offset_read(&tp, &offs, read + 2, > + offset_size); > + > + if (offs > deleted_entry_offset) > + offs-= length; // data stored after removed data > + > + int2store(write, nm); > + type_and_offset_store(write + 2, new_offset_size, tp, offs); > + } Please optimize that if new_offset_size == old_offset_size then we should start from header_entry. You should also make the above a function as update also needs to do this! > + > + /* move data */ > + { > + size_t first_chunk_len= (data - (uchar *)str->str) - > + FIXED_HEADER_SIZE - header_size; > + size_t second_chunk_len= new_data_size - first_chunk_len; > + if (first_chunk_len) > + memmove(str->str + FIXED_HEADER_SIZE + new_header_size, > + str->str + FIXED_HEADER_SIZE + header_size, > + first_chunk_len); > + if (second_chunk_len) > + memmove(str->str + > + FIXED_HEADER_SIZE + new_header_size + first_chunk_len, > + str->str + > + FIXED_HEADER_SIZE + header_size + first_chunk_len + length, > + second_chunk_len); > + } > + > + /* fix str length */ > + DBUG_ASSERT(str->length >= > + FIXED_HEADER_SIZE + new_header_size + new_data_size); > + str->length= FIXED_HEADER_SIZE + new_header_size + new_data_size; > + > + return FALSE; > +} > + > + > +my_bool dynamic_column_update(DYNAMIC_COLUMN *str, uint column_nr, > + DYNAMIC_COLUMN_VALUE *value) > +{ > + size_t offset_size, new_offset_size, data_size, new_data_size, > + add_data_size, entry_size, new_entry_size, > + header_size, new_header_size, additional_size, next_offs, > + data_ins_offset; > + uint column_count, i; > + uchar *read, *write; > + my_bool added; > + > + /* TODO: optimize update without deleting */ > + if (dynamic_column_delete(str, column_nr)) > + return TRUE; The problem with doing delete + insert later is that the whole function needs to be rewritten and you can't use any of the old code (for columns that exists). So I suggest you write this ASAP! > + > + if (value->type == DYN_COL_NULL) > + return FALSE; Should be stored. > + > + if (!str || str->length < FIXED_HEADER_SIZE || > + (str->str[0] & (~DYNCOL_FLG_KNOWN))) > + return TRUE; length of 0 should be regarded as ok. > + offset_size= (str->str[0] & DYNCOL_FLG_OFFSET) + 1; > + column_count= uint4korr(str->str + 1); > + > + add_data_size= dynamic_column_value_len(value); > + > + if (column_count == 0) > + { > + /* offset size of one record could be minimal */ > + return dynamic_new_column_add(str, 1 * (1 + 2), 1, 1, 1, > + new_data_size, &column_nr, value); > + } > + > + entry_size= 2 + offset_size; > + header_size= entry_size * column_count; > + data_size= str->length - FIXED_HEADER_SIZE - header_size; > + > + new_data_size= data_size + add_data_size; > + new_offset_size= dynamic_column_offset_bytes(new_data_size); > + DBUG_ASSERT(new_offset_size >= offset_size); > + new_entry_size= 2 + new_offset_size; > + new_header_size= new_entry_size * (column_count + 1); > + > + additional_size= add_data_size + (new_header_size - header_size); > + > + if (dynstr_realloc(str, additional_size)) > + return TRUE; > + > + /* move data to the string end */ > + str->length+= additional_size; > + memmove(str->str + FIXED_HEADER_SIZE + new_header_size + add_data_size, > + str->str + FIXED_HEADER_SIZE + header_size, > + data_size); Data should be placed on it's right position, not at end! You can do now do that with find_column() ! > + /* rewrite header */ > + next_offs= str->length - FIXED_HEADER_SIZE - new_header_size; > + for (i= column_count, > + read= (uchar *)str->str + FIXED_HEADER_SIZE + header_size - entry_size, > + write= ((uchar *)str->str + FIXED_HEADER_SIZE + > + (new_entry_size * column_count)); > + i > 0; > + i--, read-= entry_size, write-= new_entry_size) > + { > + size_t offs; > + uint nm; > + DYNAMIC_COLUMN_TYPE tp; > + > + nm= uint2korr(read); > + type_and_offset_read(&tp, &offs, read + 2, offset_size); > + > + if (!added && column_nr > nm) > + { > + /* place to put the new column */ > + int2store(write, column_nr); > + type_and_offset_store(write + 2, new_offset_size, > + value->type, > + (data_ins_offset= next_offs - add_data_size)); > + /* move previous data */ > + memmove(str->str + FIXED_HEADER_SIZE + new_header_size, > + str->str + FIXED_HEADER_SIZE + new_header_size + add_data_size, > + next_offs - add_data_size); > + added= TRUE; > + write-= new_entry_size; > + } > + int2store(write, nm); > + if (!added) > + offs+= add_data_size; > + type_and_offset_store(write + 2, new_offset_size, tp, offs); > + next_offs= offs; > + } We should just have one function to rewrite the headers! To make life easer, maybe we should assume that header size change operation is a very unlikely one and accept to do an extra memmove when header offset changes? This would allow us to have a simple function that just rewrites the header (from a given point) and another that increases/decreases the header size + all data ? Then all operations are basicly: - If offset_pointer_size changes, fix data - Add / remove things from header and data (with new offset size) - Update header The whole things above would then be very trivial: - realloc str->length if it needs to grow - If offset_pointer_size changes, fix data and update entry_pos - Update data (and create new header entry last) - memmove(entry, entry + entry_size, header_end - entry); - Write data at entry > + if (!added) > + { > + /* put the new column before others */ > + int2store(write, column_nr); > + type_and_offset_store(write + 2, new_offset_size, > + value->type, (data_ins_offset= 0)); > + } > + str->str[0]|= (new_offset_size - 1); // size of offset > + int4store(str->str + 1, column_count + 1); // columns number > + > + { > + /* insert data */ > + size_t save_len= str->length; > + str->length= data_ins_offset + FIXED_HEADER_SIZE + new_header_size; > + if (data_add(str, value)) > + return TRUE; > + str->length= save_len; > + } > + > + return FALSE; > +} > + > + > +my_bool dynamic_column_exists(DYNAMIC_COLUMN *str, uint column_nr) > +{ > + uchar *data; > + size_t offset_size, length; > + uint column_count; > + DYNAMIC_COLUMN_TYPE type; > + > + if (!str || str->length < FIXED_HEADER_SIZE || > + (str->str[0] & (~DYNCOL_FLG_KNOWN))) > + return TRUE; length 0 should return FALSE > + > + offset_size= (str->str[0] & DYNCOL_FLG_OFFSET) + 1; > + column_count= uint4korr(str->str + 1); > + > + if (column_count == 0) > + return FALSE; // no columns > + > + if (find_column(&type, &data, &length, (uchar*)str->str + 5, > + offset_size, column_count, (uchar*)str->str + str->length, > + column_nr, NULL)) > + return FALSE; // error but we can't report it > + > + return type != DYN_COL_NULL; > +} > + > +my_bool dynamic_column_list(DYNAMIC_COLUMN *str, DYNAMIC_ARRAY *array_of_uint) > +{ > + uchar *read; > + size_t offset_size, entry_size; > + uint column_count, i; > + > + if (!str || str->length < FIXED_HEADER_SIZE || > + (str->str[0] & (~DYNCOL_FLG_KNOWN))) > + return TRUE; > + > + offset_size= (str->str[0] & DYNCOL_FLG_OFFSET) + 1; > + column_count= uint4korr(str->str + 1); > + entry_size= 2 + offset_size; > + > + if (init_dynamic_array(array_of_uint, sizeof(uint), > + column_count, 10 CALLER_INFO)) > + return TRUE; -> my_init_dynamic_array2(array_of_uint, sizeof(uint), NULL, column_count,0) > + > + for (i= 0, read= (uchar *)str->str + FIXED_HEADER_SIZE; > + i < column_count; > + i++, read+= entry_size) > + { > + uint nm= uint2korr(read); > + if (insert_dynamic(array_of_uint, (uchar *)&nm)) > + { > + delete_dynamic(array_of_uint); > + return TRUE; > + } > + } > + return FALSE; > +} > === modified file 'sql/my_decimal.h' > --- sql/my_decimal.h 2010-11-24 22:57:34 +0000 > +++ sql/my_decimal.h 2011-03-24 16:30:27 +0000 > @@ -32,32 +32,7 @@ > #include <decimal.h> > C_MODE_END > -#define DECIMAL_LONGLONG_DIGITS 22 > -#define DECIMAL_LONG_DIGITS 10 > -#define DECIMAL_LONG3_DIGITS 8 > - > -/** maximum length of buffer in our big digits (uint32). */ > -#define DECIMAL_BUFF_LENGTH 9 > - > -/* the number of digits that my_decimal can possibly contain */ > -#define DECIMAL_MAX_POSSIBLE_PRECISION (DECIMAL_BUFF_LENGTH * 9) > - > - > -/** > - maximum guaranteed precision of number in decimal digits (number of our > - digits * number of decimal digits in one our big digit - number of decimal > - digits in one our big digit decreased by 1 (because we always put decimal > - point on the border of our big digits)) > -*/ > -#define DECIMAL_MAX_PRECISION (DECIMAL_MAX_POSSIBLE_PRECISION - 8*2) > -#define DECIMAL_MAX_SCALE 30 > -#define DECIMAL_NOT_SPECIFIED 31 > - > -/** > - maximum length of string representation (number of maximum decimal > - digits + 1 position for sign + 1 position for decimal point) > -*/ > -#define DECIMAL_MAX_STR_LENGTH (DECIMAL_MAX_POSSIBLE_PRECISION + 2) > +#include <my_decimal_limits.h> > /** > maximum size of packet length. > === modified file 'unittest/mysys/Makefile.am' > --- unittest/mysys/Makefile.am 2010-11-30 21:11:03 +0000 > +++ unittest/mysys/Makefile.am 2011-03-25 07:46:32 +0000 > @@ -25,4 +25,4 @@ > EXTRA_DIST = CMakeLists.txt > noinst_PROGRAMS = bitmap-t base64-t my_atomic-t lf-t waiting_threads-t \ > - my_vsnprintf-t > + my_vsnprintf-t ma_dyncol-t > === added file 'unittest/mysys/ma_dyncol-t.c' > --- unittest/mysys/ma_dyncol-t.c 1970-01-01 00:00:00 +0000 > +++ unittest/mysys/ma_dyncol-t.c 2011-03-27 22:53:44 +0000 > @@ -0,0 +1,690 @@ > +#include <my_global.h> > +#include <ma_dyncol.h> > +#include <tap.h> > + > + > + > +void test_value_single_null() > +{ > + int rc= FALSE; > + DYNAMIC_COLUMN_VALUE val, res; > + DYNAMIC_COLUMN str; > + /* init values */ > + val.type= DYN_COL_NULL; > + dynamic_column_value_init(&res); > + /* create column */ > + if (dynamic_column_create(&str, 1, &val)) > + goto err; > + dynstr_append(&str, "\1"); str.length--; //check for overflow > + /* read column */ > + if (dynamic_column_get(&str, 1, &res)) > + goto err; > + rc= (res.type == DYN_COL_NULL); > +err: > + ok(rc, "%s", "NULL"); > + /* cleanup */ > + dynamic_column_column_free(&str); > + dynamic_column_value_free(&val); > + dynamic_column_value_free(&res); > +} > + > +void test_value_single_uint(ulonglong num, const char *name) > +{ > + int rc= FALSE; > + DYNAMIC_COLUMN_VALUE val, res; > + DYNAMIC_COLUMN str; > + /* init values */ > + val.type= DYN_COL_UINT; > + val.ulong_value= num; > + dynamic_column_value_init(&res); > + /* create column */ > + if (dynamic_column_create(&str, 1, &val)) > + goto err; > + dynstr_append(&str, "\1"); str.length--; //check for overflow > + /* read column */ > + if (dynamic_column_get(&str, 1, &res)) > + goto err; > + rc= (res.type == DYN_COL_UINT) && (res.ulong_value == num); > + num= res.ulong_value; > +err: > + ok(rc, "%s - %llu", name, num); > + /* cleanup */ > + dynamic_column_column_free(&str); > + dynamic_column_value_free(&val); > + dynamic_column_value_free(&res); > +} > + > +void test_value_single_sint(longlong num, const char *name) > +{ > + int rc= FALSE; > + DYNAMIC_COLUMN_VALUE val, res; > + DYNAMIC_COLUMN str; > + /* init values */ > + val.type= DYN_COL_INT; > + val.long_value= num; > + dynamic_column_value_init(&res); > + /* create column */ > + if (dynamic_column_create(&str, 1, &val)) > + goto err; > + dynstr_append(&str, "\1"); str.length--; //check for overflow > + /* read column */ > + if (dynamic_column_get(&str, 1, &res)) > + goto err; > + rc= (res.type == DYN_COL_INT) && (res.long_value == num); > + num= res.ulong_value; > +err: > + ok(rc, "%s - %lld", name, num); > + /* cleanup */ > + dynamic_column_column_free(&str); > + dynamic_column_value_free(&val); > + dynamic_column_value_free(&res); > +} > + > + > +void test_value_single_double(double num, const char *name) > +{ > + int rc= FALSE; > + DYNAMIC_COLUMN_VALUE val, res; > + DYNAMIC_COLUMN str; > + /* init values */ > + val.type= DYN_COL_DOUBLE; > + val.double_value= num; > + dynamic_column_value_init(&res); > + /* create column */ > + if (dynamic_column_create(&str, 1, &val)) > + goto err; > + dynstr_append(&str, "\1"); str.length--; //check for overflow > + /* read column */ > + if (dynamic_column_get(&str, 1, &res)) > + goto err; > + rc= (res.type == DYN_COL_DOUBLE) && (res.double_value == num); > + num= res.ulong_value; > +err: > + ok(rc, "%s - %lf", name, num); > + /* cleanup */ > + dynamic_column_column_free(&str); > + dynamic_column_value_free(&val); > + dynamic_column_value_free(&res); > +} > + > +void test_value_single_decimal(const char *num) > +{ > + char *end= (((char*)num) + strlen(num)); > + char buff[80]; > + int rc= FALSE; > + int length= 80; > + DYNAMIC_COLUMN_VALUE val, res; > + DYNAMIC_COLUMN str; > + > + /* init values */ > + dynamic_column_prepare_decimal(&val); // special procedure for decimal!!! > + if (string2decimal(num, &val.decimal_value, &end) != E_DEC_OK) > + goto err; > + dynamic_column_value_init(&res); > + > + /* create column */ > + if (dynamic_column_create(&str, 1, &val)) > + goto err; > + dynstr_append(&str, "\1"); str.length--; //check for overflow > + /* read column */ > + if (dynamic_column_get(&str, 1, &res)) > + goto err; > + rc= ((res.type == DYN_COL_DECIMAL) && > + (decimal_cmp(&res.decimal_value, &val.decimal_value) == 0)); > + decimal2string(&res.decimal_value, buff, &length, 0, 0, ' '); > +err: > + ok(rc, "%s - %s", num, buff); > + /* cleanup */ > + dynamic_column_column_free(&str); > + dynamic_column_value_free(&val); > + dynamic_column_value_free(&res); > +} > + > +static CHARSET_INFO *charset_list[]= > +{ > +#ifdef HAVE_CHARSET_big5 > + &my_charset_big5_chinese_ci, > + &my_charset_big5_bin, > +#endif > +#ifdef HAVE_CHARSET_euckr > + &my_charset_euckr_korean_ci, > + &my_charset_euckr_bin, > +#endif > +#ifdef HAVE_CHARSET_gb2312 > + &my_charset_gb2312_chinese_ci, > + &my_charset_gb2312_bin, > +#endif > +#ifdef HAVE_CHARSET_gbk > + &my_charset_gbk_chinese_ci, > + &my_charset_gbk_bin, > +#endif > +#ifdef HAVE_CHARSET_latin1 > + &my_charset_latin1, > + &my_charset_latin1_bin, > +#endif > +#ifdef HAVE_CHARSET_sjis > + &my_charset_sjis_japanese_ci, > + &my_charset_sjis_bin, > +#endif > +#ifdef HAVE_CHARSET_tis620 > + &my_charset_tis620_thai_ci, > + &my_charset_tis620_bin, > +#endif > +#ifdef HAVE_CHARSET_ujis > + &my_charset_ujis_japanese_ci, > + &my_charset_ujis_bin, > +#endif > +#ifdef HAVE_CHARSET_utf8 > + &my_charset_utf8_general_ci, > +#ifdef HAVE_HAVE_UCA_COLLATIONS > + &my_charset_utf8_unicode_ci, > +#endif > + &my_charset_utf8_bin, > +#endif > +}; > + > + > +void test_value_single_string(const char *string, size_t len, > + CHARSET_INFO *cs) > +{ > + int rc= FALSE; > + DYNAMIC_COLUMN_VALUE val, res; > + DYNAMIC_COLUMN str; > + > + /* init values */ > + val.type= DYN_COL_STRING; > + val.string_value.str= (char*)string; > + val.string_value.length= len; > + val.charset= cs; > + dynamic_column_value_init(&res); > + > + /* create column */ > + if (dynamic_column_create(&str, 1, &val)) > + goto err; > + dynstr_append(&str, "\1"); str.length--; //check for overflow > + /* read column */ > + if (dynamic_column_get(&str, 1, &res)) > + goto err; > + rc= ((res.type == DYN_COL_STRING) && > + (res.string_value.length == len) && > + (memcmp(res.string_value.str, string, len) == 0) && > + (res.charset->number == cs->number)); > +err: > + ok(rc, "'%s' - '%s' %u %u-%s", string, > + res.string_value.str, (uint)res.string_value.length, > + (uint)res.charset->number, res.charset->name); > + /* cleanup */ > + val.string_value.str= NULL; // we did not allocated it > + dynamic_column_column_free(&str); > + dynamic_column_value_free(&val); > + dynamic_column_value_free(&res); > +} > + > +void test_value_single_date(uint year, uint month, uint day, const char *name) > +{ > + int rc= FALSE; > + DYNAMIC_COLUMN_VALUE val, res; > + DYNAMIC_COLUMN str; > + /* init values */ > + val.type= DYN_COL_DATE; > + val.time_value.time_type= MYSQL_TIMESTAMP_DATE; > + val.time_value.year= year; > + val.time_value.month= month; > + val.time_value.day= day; > + dynamic_column_value_init(&res); > + /* create column */ > + if (dynamic_column_create(&str, 1, &val)) > + goto err; > + dynstr_append(&str, "\1"); str.length--; //check for overflow > + /* read column */ > + if (dynamic_column_get(&str, 1, &res)) > + goto err; > + rc= ((res.type == DYN_COL_DATE) && > + (res.time_value.time_type == MYSQL_TIMESTAMP_DATE) && > + (res.time_value.year == year) && > + (res.time_value.month == month) && > + (res.time_value.day == day)); > +err: > + ok(rc, "%s - %04u-%02u-%02u", name, year, month, day); > + /* cleanup */ > + dynamic_column_column_free(&str); > + dynamic_column_value_free(&val); > + dynamic_column_value_free(&res); > +} > + > +void test_value_single_time(uint neg, uint hour, uint minute, uint second, > + uint mic, const char *name) > +{ > + int rc= FALSE; > + DYNAMIC_COLUMN_VALUE val, res; > + DYNAMIC_COLUMN str; > + /* init values */ > + val.type= DYN_COL_TIME; > + val.time_value.time_type= MYSQL_TIMESTAMP_TIME; > + val.time_value.neg= neg; > + val.time_value.hour= hour; > + val.time_value.minute= minute; > + val.time_value.second= second; > + val.time_value.second_part= mic; > + dynamic_column_value_init(&res); > + /* create column */ > + if (dynamic_column_create(&str, 1, &val)) > + goto err; > + dynstr_append(&str, "\1"); str.length--; //check for overflow > + /* read column */ > + if (dynamic_column_get(&str, 1, &res)) > + goto err; > + rc= ((res.type == DYN_COL_TIME) && > + (res.time_value.time_type == MYSQL_TIMESTAMP_TIME) && > + (res.time_value.neg == (int)neg) && > + (res.time_value.hour == hour) && > + (res.time_value.minute == minute) && > + (res.time_value.second == second) && > + (res.time_value.second_part == mic)); > +err: > + ok(rc, "%s - %c%02u:%02u:%02u.%06u", name, (neg ? '-' : '+'), > + hour, minute, second, mic); > + /* cleanup */ > + dynamic_column_column_free(&str); > + dynamic_column_value_free(&val); > + dynamic_column_value_free(&res); > +} > + > + > +void test_value_single_datetime(uint neg, uint year, uint month, uint day, > + uint hour, uint minute, uint second, > + uint mic, const char *name) > +{ > + int rc= FALSE; > + DYNAMIC_COLUMN_VALUE val, res; > + DYNAMIC_COLUMN str; > + /* init values */ > + val.type= DYN_COL_DATETIME; > + val.time_value.time_type= MYSQL_TIMESTAMP_DATETIME; > + val.time_value.neg= neg; > + val.time_value.year= year; > + val.time_value.month= month; > + val.time_value.day= day; > + val.time_value.hour= hour; > + val.time_value.minute= minute; > + val.time_value.second= second; > + val.time_value.second_part= mic; > + dynamic_column_value_init(&res); > + /* create column */ > + if (dynamic_column_create(&str, 1, &val)) > + goto err; > + dynstr_append(&str, "\1"); str.length--; //check for overflow > + /* read column */ > + if (dynamic_column_get(&str, 1, &res)) > + goto err; > + rc= ((res.type == DYN_COL_DATETIME) && > + (res.time_value.time_type == MYSQL_TIMESTAMP_DATETIME) && > + (res.time_value.neg == (int)neg) && > + (res.time_value.year == year) && > + (res.time_value.month == month) && > + (res.time_value.day == day) && > + (res.time_value.hour == hour) && > + (res.time_value.minute == minute) && > + (res.time_value.second == second) && > + (res.time_value.second_part == mic)); > +err: > + ok(rc, "%s - %c %04u-%02u-%02u %02u:%02u:%02u.%06u", name, (neg ? '-' : '+'), > + year, month, day, hour, minute, second, mic); > + /* cleanup */ > + dynamic_column_column_free(&str); > + dynamic_column_value_free(&val); > + dynamic_column_value_free(&res); > +} > + > + > +void test_value_multi(ulonglong num0, > + longlong num1, > + double num2, > + const char *num3, > + const char *string4, size_t len4, CHARSET_INFO *cs4, > + uint year5, uint month5, uint day5, > + uint neg6, uint hour6, uint minute6, > + uint second6, uint mic6, > + uint neg7, uint year7, uint month7, uint day7, > + uint hour7, uint minute7, uint second7, > + uint mic7, > + uint *column_numbers, > + const char *name) > +{ > + char *end3= (((char*)num3) + strlen(num3)); > + int rc= FALSE; > + uint i; > + DYNAMIC_COLUMN_VALUE val[9], res[9]; > + DYNAMIC_COLUMN str; > + /* init values */ > + val[0].type= DYN_COL_UINT; > + val[0].ulong_value= num0; > + val[1].type= DYN_COL_INT; > + val[1].long_value= num1; > + val[2].type= DYN_COL_DOUBLE; > + val[2].double_value= num2; > + dynamic_column_prepare_decimal(val + 3); // special procedure for decimal!!! > + if (string2decimal(num3, &val[3].decimal_value, &end3) != E_DEC_OK) > + goto err; > + val[4].type= DYN_COL_STRING; > + val[4].string_value.str= (char*)string4; > + val[4].string_value.length= len4; > + val[4].charset= cs4; > + val[5].type= DYN_COL_DATE; > + val[5].time_value.time_type= MYSQL_TIMESTAMP_DATE; > + val[5].time_value.year= year5; > + val[5].time_value.month= month5; > + val[5].time_value.day= day5; > + val[6].type= DYN_COL_TIME; > + val[6].time_value.time_type= MYSQL_TIMESTAMP_TIME; > + val[6].time_value.neg= neg6; > + val[6].time_value.hour= hour6; > + val[6].time_value.minute= minute6; > + val[6].time_value.second= second6; > + val[6].time_value.second_part= mic6; > + val[7].type= DYN_COL_DATETIME; > + val[7].time_value.time_type= MYSQL_TIMESTAMP_DATETIME; > + val[7].time_value.neg= neg7; > + val[7].time_value.year= year7; > + val[7].time_value.month= month7; > + val[7].time_value.day= day7; > + val[7].time_value.hour= hour7; > + val[7].time_value.minute= minute7; > + val[7].time_value.second= second7; > + val[7].time_value.second_part= mic7; > + val[8].type= DYN_COL_NULL; > + for (i= 0; i < 9; i++) > + dynamic_column_value_init(res + i); > + /* create column */ > + if (dynamic_column_create_many(&str, 9, column_numbers, val)) > + goto err; > + dynstr_append(&str, "\1"); str.length--; //check for overflow > + /* read column */ > + for (i= 0; i < 9; i++) > + if (dynamic_column_get(&str, column_numbers[i], res + i)) > + goto err; > + rc= ((res[0].type == DYN_COL_UINT) && > + (res[0].ulong_value == num0) && > + (res[1].type == DYN_COL_INT) && > + (res[1].long_value == num1) && > + (res[2].type == DYN_COL_DOUBLE) && > + (res[2].double_value == num2) && > + (res[3].type == DYN_COL_DECIMAL) && > + (decimal_cmp(&res[3].decimal_value, &val[3].decimal_value) == 0) && > + (res[4].type == DYN_COL_STRING) && > + (res[4].string_value.length == len4) && > + (memcmp(res[4].string_value.str, string4, len4) == 0) && > + (res[4].charset->number == cs4->number) && > + (res[5].type == DYN_COL_DATE) && > + (res[5].time_value.time_type == MYSQL_TIMESTAMP_DATE) && > + (res[5].time_value.year == year5) && > + (res[5].time_value.month == month5) && > + (res[5].time_value.day == day5) && > + (res[6].type == DYN_COL_TIME) && > + (res[6].time_value.time_type == MYSQL_TIMESTAMP_TIME) && > + (res[6].time_value.neg == (int)neg6) && > + (res[6].time_value.hour == hour6) && > + (res[6].time_value.minute == minute6) && > + (res[6].time_value.second == second6) && > + (res[6].time_value.second_part == mic6) && > + (res[7].type == DYN_COL_DATETIME) && > + (res[7].time_value.time_type == MYSQL_TIMESTAMP_DATETIME) && > + (res[7].time_value.neg == (int)neg7) && > + (res[7].time_value.year == year7) && > + (res[7].time_value.month == month7) && > + (res[7].time_value.day == day7) && > + (res[7].time_value.hour == hour7) && > + (res[7].time_value.minute == minute7) && > + (res[7].time_value.second == second7) && > + (res[7].time_value.second_part == mic7) && > + (res[8].type == DYN_COL_NULL)); > +err: > + ok(rc, "%s", name); > + /* cleanup */ > + val[4].string_value.str= NULL; // we did not allocated it > + dynamic_column_column_free(&str); > + for (i= 0; i < 9; i++) > + { > + dynamic_column_value_free(val + i); > + dynamic_column_value_free(res + i); > + } > +} > + > + > +void test_value_multi_same_num() > +{ > + int rc= FALSE; > + uint i; > + DYNAMIC_COLUMN_VALUE val[5]; > + uint column_numbers[]= {3,4,5,3,6}; // same column numbers > + DYNAMIC_COLUMN str; > + /* init values */ > + for (i= 0; i < 5; i++) > + val[i].type= DYN_COL_NULL; > + /* create column */ > + if (!dynamic_column_create_many(&str, 5, column_numbers, val)) > + goto err; > + rc= TRUE; > +err: > + ok(rc, "%s", "same column numbers check"); > + /* cleanup */ > + dynamic_column_column_free(&str); > + for (i= 0; i < 5; i++) > + dynamic_column_value_free(val + i); > +} > + > + > +void test_update_multi(uint *column_numbers, uint *column_values, > + my_bool *null_values, int only_add, int all) > +{ > + int rc= FALSE; > + int i, j; > + DYNAMIC_COLUMN str; > + DYNAMIC_COLUMN_VALUE val; > + > + val.type= DYN_COL_UINT; > + val.ulong_value= column_values[0]; > + if (dynamic_column_create(&str, column_numbers[0], &val)) > + goto err; > + dynamic_column_value_free(&val); > + for (i= 1; i < all; i++) > + { > + val.type= (null_values[i] ? DYN_COL_NULL : DYN_COL_UINT); > + val.ulong_value= column_values[i]; > + if (dynamic_column_update(&str, column_numbers[i], &val)) > + goto err; > + dynamic_column_value_free(&val); > + > + /* check value(s) */ > + for (j= i; j >= (i < only_add ? 0 : i); j--) > + { > + if (dynamic_column_get(&str, column_numbers[j], &val)) > + goto err; > + if (null_values[j]) > + { > + if (val.type != DYN_COL_NULL || > + dynamic_column_exists(&str, column_numbers[j])) > + goto err; > + } > + else > + { > + if (val.type != DYN_COL_UINT || > + val.ulong_value != column_values[j] || > + !dynamic_column_exists(&str, column_numbers[j])) > + goto err; > + } > + dynamic_column_value_free(&val); > + } > + if (i < only_add) > + { > + DYNAMIC_ARRAY num; > + if (dynamic_column_list(&str, &num)) > + goto err; > + /* cross check arrays */ > + if ((int)num.elements != i + 1) > + { > + delete_dynamic(&num); > + goto err; > + } > + for(j= 0; j < i + 1; j++) > + { > + int k; > + for(k= 0; > + k < i + 1 && column_numbers[j] != > + *dynamic_element(&num, k, uint*); > + k++); > + if (k >= i + 1) > + { > + delete_dynamic(&num); > + goto err; > + } > + for(k= 0; > + k < i + 1 && column_numbers[k] != > + *dynamic_element(&num, j, uint*); > + k++); > + if (k >= i + 1) > + { > + delete_dynamic(&num); > + goto err; > + } > + } > + delete_dynamic(&num); > + } > + } > + > + rc= TRUE; > +err: > + ok(rc, "%s", "add/delete/update"); > + /* cleanup */ > + dynamic_column_column_free(&str); > + dynamic_column_value_free(&val); > +} > + > + > +int main(int argc __attribute__((unused)), char **argv) > +{ > + uint i; > + MY_INIT(argv[0]); > + plan(53); > + char *big_string= malloc(1024*1024); > + if (!big_string) > + exit(1); > + for (i= 0; i < 1024*1024; i++) > + big_string[i]= ('0' + (i % 10)); > + test_value_single_null(); > + test_value_single_uint(0, "0"); > + test_value_single_uint(0xffffffffffffffff, "0xffffffffffffffff"); > + test_value_single_uint(0xaaaaaaaaaaaaaaaa, "0xaaaaaaaaaaaaaaaa"); > + test_value_single_uint(0x5555555555555555, "0x5555555555555555"); > + test_value_single_uint(27652, "27652"); > + test_value_single_sint(0, "0"); > + test_value_single_sint(1, "1"); > + test_value_single_sint(-1, "-1"); > + test_value_single_sint((longlong)0x7fffffffffffffff, > + "0x7fffffffffffffff"); > + test_value_single_sint((longlong)0xaaaaaaaaaaaaaaaa, > + "0xaaaaaaaaaaaaaaaa"); > + test_value_single_sint((longlong)0x5555555555555555, > + "0x5555555555555555"); > + test_value_single_sint((longlong)0x8000000000000000, > + "0x8000000000000000"); > + test_value_single_double(0.0, "0.0"); > + test_value_single_double(1.0, "1.0"); > + test_value_single_double(-1.0, "-1.0"); > + test_value_single_double(1.0e100, "1.0e100"); > + test_value_single_double(1.0e-100, "1.0e-100"); > + test_value_single_double(9999999999999999999999999999999999999.0, > + "9999999999999999999999999999999999999.0"); > + test_value_single_double(-9999999999999999999999999999999999999.0, > + "-9999999999999999999999999999999999999.0"); > + test_value_single_decimal("0"); > + test_value_single_decimal("1"); > + test_value_single_decimal("-1"); > + test_value_single_decimal("9999999999999999999999999999999"); > + test_value_single_decimal("-9999999999999999999999999999999"); > + test_value_single_decimal("0.9999999999999999999999999999999"); > + test_value_single_decimal("-0.9999999999999999999999999999999"); > + test_value_single_string("", 0, charset_list[0]); > + test_value_single_string("1234567890", 10, charset_list[0]); > + test_value_single_string("nulls\0\0\0\0\0", 10, charset_list[0]); > + sprintf(big_string, "%x", 0x7a); > + test_value_single_string(big_string, 0x7a, charset_list[0]); > + sprintf(big_string, "%x", 0x80); > + test_value_single_string(big_string, 0x80, charset_list[0]); > + sprintf(big_string, "%x", 0x7ffa); > + test_value_single_string(big_string, 0x7ffa, charset_list[0]); > + sprintf(big_string, "%x", 0x8000); > + test_value_single_string(big_string, 0x8000, charset_list[0]); > + sprintf(big_string, "%x", 1024*1024); > + test_value_single_string(big_string, 1024*1024, charset_list[0]); > + free(big_string); > + test_value_single_date(0, 0, 0, "zero date"); > + test_value_single_date(9999, 12, 31, "max date"); > + test_value_single_date(2011, 3, 26, "some date"); > + test_value_single_time(0, 0, 0, 0, 0, "zero time"); > + test_value_single_time(1, 23, 59, 59, 999999, "min time"); > + test_value_single_time(0, 23, 59, 59, 999999, "max time"); > + test_value_single_time(0, 21, 36, 20, 28, "some time"); > + test_value_single_datetime(0, 0, 0, 0, 0, 0, 0, 0, "zero datetime"); > + test_value_single_datetime(1, 9999, 12, 31, 23, 59, 59, 999999, > + "min datetime"); > + test_value_single_datetime(0, 9999, 12, 31, 23, 59, 59, 999999, > + "max datetime"); > + test_value_single_datetime(0, 2011, 3, 26, 21, 53, 12, 3445, > + "some datetime"); > + { > + uint column_numbers[]= {100,1,2,3,4,5,6,7,8}; > + test_value_multi(0, 0, 0.0, "0", > + "", 0, charset_list[0], > + 0, 0, 0, > + 0, 0, 0, 0, 0, > + 0, 0, 0, 0, 0, 0, 0, 0, > + column_numbers, > + "zero data"); > + } > + { > + uint column_numbers[]= {10,1,12,37,4,57,6,76,87}; > + test_value_multi(0xffffffffffffffff, 0x7fffffffffffffff, > + 99999999.999e120, "9999999999999999999999999999999", > + big_string, 1024*1024, charset_list[0], > + 9999, 12, 31, > + 0, 23, 59, 59, 999999, > + 0, 9999, 12, 31, 23, 59, 59, 999999, > + column_numbers, > + "much data"); > + } > + { > + uint column_numbers[]= {101,12,122,37,24,572,16,726,77}; > + test_value_multi(37878, -3344, > + 2873.3874, "92743.238984789898", > + "string", 6, charset_list[0], > + 2011, 3, 26, > + 1, 23, 23, 20, 333, > + 0, 2011, 3, 26, 23, 23, 53, 334, > + column_numbers, > + "zero data"); > + } > + test_value_multi_same_num(); > + { > + uint column_numbers[]= {1,2,3,4,5,6,7,2, 3, 4}; > + uint column_values[]= {1,2,3,4,5,6,7,0,30,40}; > + my_bool null_values[]= {0,0,0,0,0,0,0,1, 0, 0}; > + > + test_update_multi(column_numbers, column_values, null_values, 7, 10); > + } > + { > + uint column_numbers[]= {4,3,2,1, 1,2,3,4}; > + uint column_values[]= {4,3,2,1, 0,0,0,0}; > + my_bool null_values[]= {0,0,0,0, 1,1,1,1}; > + > + test_update_multi(column_numbers, column_values, null_values, 4, 8); > + } > + { > + uint column_numbers[]= {4,3,2,1, 4,3,2,1}; > + uint column_values[]= {4,3,2,1, 0,0,0,0}; > + my_bool null_values[]= {0,0,0,0, 1,1,1,1}; > + > + test_update_multi(column_numbers, column_values, null_values, 4, 8); > + } > + return 0; > +} Regards, Monty

2 2

[Maria-discuss] Bundle more relevant cnf files in MariaDB package
by Haidong Ji 04 Apr '11

04 Apr '11

Hi, I alluded to this in IRC maybe a couple of weeks back. That is, it is really to have intelligent default for MariaDB servers. In particular, the bundled sample my.cnf files, under /usr/local/mysql/support-files, such as my-small.cnf, my-medium.cnf, my-large.cnf, and my-huge.cnf, are laughable in today's world. In the my-huge.cnf, it talks about memory between 1G and 2G as huge. It will be good that MariaDB can bundle a different set, like what has been mentioned here: http://datastrangler.com/windpipe/2011/02/24/custom-mysql-config-files-to-e… I think the sample cnf files in the above blog post is a good starting point. Thanks, Haidong "Alex" Ji PS: I modified http://kb.askmonty.org/v/installing-mariadb-binary-tarballs with the addition of modifying $PATH so a casual user will be able to use various MariaDB clients after the install.

6 14

[Maria-discuss] Segfaulting with signal 6
by Lance Albertson 30 Mar '11

30 Mar '11

I've run into another weird problem where MariaDB is segfaulting and dieing on Gentoo Hardened Linux. I can't figure out if its a weird situation with my Gentoo environment on the machine or something with MariaDB itself. I don't have this problem on the other machine running MariaDB. It seems to happen under load about once or twice a day. I have a nearly identical machine that doesn't have any of these problems. I'm running mariadb-5.1.50 under Hardened Gentoo. The compiled flags are: --- Configuration summary for MariaDB Server version 5.1.50-MariaDB * Installation prefix: /usr * System type: pc-linux-gnu * Host CPU: x86_64 * C Compiler: x86_64-pc-linux-gnu-gcc (Gentoo Hardened 4.4.4-r2 p1.2, pie-0.4.5) 4.4.4 * C++ Compiler: x86_64-pc-linux-gnu-g++ (Gentoo Hardened 4.4.4-r2 p1.2, pie-0.4.5) 4.4.4 * Debug enabled: no * Community Features: yes --- $ ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --libexecdir=/usr/sbin --sysconfdir=/etc/mysql --localstatedir=/var/lib/mysql --sharedstatedir=/usr/share/mysql --libdir=/usr/lib64/mysql --includedir=/usr/include/mysql --with-low-memory --with-client-ldflags=-lstdc++ --enable-thread-safe-client --with-comment=Gentoo Linux mariadb-5.1.50 --without-docs --without-big-tables --enable-local-infile --with-extra-charsets=all --with-mysqld-user=mysql --with-server --with-unix-socket-path=/var/run/mysqld/mysqld.sock --without-libwrap --enable-shared --enable-static --without-debug --with-charset=utf8 --with-collation=utf8_general_ci --without-embedded-privilege-control --without-embedded-server --with-ssl=/usr --enable-assembler --with-geometry --with-readline --with-zlib-dir=/usr/ --without-pstack --with-plugindir=/usr/lib64/mysql/plugin --enable-community-features --disable-profiling --with-maria-tmp-tables --with-libevent --with-plugins=csv,myisam,myisammrg,heap,archive,blackhole,maria,xtradb --without-plugin-example --without-plugin-ibmdb2i --without-plugin-partition --without-plugin-federated --without-plugin-ndbcluster --without-plugin-pbxt I haven't opened a bug with Gentoo yet (I'm actually developer myself). I figured I'd ask here first. Any help would be great. Thanks! Here's the output I have from the log file: mysqld: malloc.c:5153: malloc_consolidate: Assertion `nextchunk->fd_nextsize->bk_nextsize == nextchunk' failed. 110329 13:11:25 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=134217728 read_buffer_size=262144 max_used_connections=376 max_threads=401 threads_connected=80 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 445322 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. thd: 0x3518dccbaa0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x3519ebd3ea0 thread_stack 0x3c000 /usr/lib64/mysql/libmysys.so.0(my_print_stacktrace+0x3c) [0x3547393ab0c] /usr/sbin/mysqld(handle_segfault+0x3cc) [0x7d0ebd1c1c] /lib/libpthread.so.0(+0xf010) [0x35473f86010] /lib/libc.so.6(gsignal+0x35) [0x35472264135] /lib/libc.so.6(abort+0x180) [0x35472265550] /lib/libc.so.6(+0x72c6a) [0x354722a4c6a] /lib/libc.so.6(+0x72f2d) [0x354722a4f2d] /lib/libc.so.6(+0x751e4) [0x354722a71e4] /lib/libc.so.6(__libc_malloc+0x70) [0x354722a9900] /usr/sbin/mysqld(+0x5ad9cf) [0x7d0ee529cf] /usr/sbin/mysqld(+0x5f3bfb) [0x7d0ee98bfb] /usr/sbin/mysqld(+0x5ed71b) [0x7d0ee9271b] /usr/sbin/mysqld(+0x588aa3) [0x7d0ee2daa3] /usr/sbin/mysqld(rr_sequential(READ_RECORD*)+0x29) [0x7d0ecc4b69] /usr/sbin/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x99) [0x7d0ec430b9] /usr/sbin/mysqld(+0x39e524) [0x7d0ec43524] /usr/sbin/mysqld(JOIN::exec()+0xbd7) [0x7d0ec53b47] /usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st _order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x1b2) [0x7d0ec551a 2] /usr/sbin/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x19c) [0x7d0ec55c5c] /usr/sbin/mysqld(+0x33955a) [0x7d0ebde55a] /usr/sbin/mysqld(mysql_execute_command(THD*)+0x405) [0x7d0ebe07f5] /usr/sbin/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x2f3) [0x7d0ebe5cf3] /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x53a) [0x7d0ebe6b2a] /usr/sbin/mysqld(libevent_thread_proc+0x182) [0x7d0ebda682] /lib/libpthread.so.0(+0x68c4) [0x35473f7d8c4] /lib/libc.so.6(clone+0x6d) [0x35472302b4d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort... thd->query at 0x3518424f018 is an invalid pointer thd->thread_id=14399 thd->killed=NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. -- Lance Albertson Systems Administrator / Architect Open Source Lab Network Services Oregon State University

3 3

Re: [Maria-discuss] client library shared library version changed in 5.5.10
by Sergei Golubchik 28 Mar '11

28 Mar '11

Hi, Monty! On Mar 18, Michael Widenius wrote: > >>>>> "MARK" == MARK CALLAGHAN <mdcallag(a)gmail.com> writes: > > > I think there was discussion about how client library shared library > > versions during the Lisbon meeting. > > >> From http://dev.mysql.com/doc/refman/5.5/en/news-5-5-10.html > >>>> > > Incompatible Change: The shared library version of the client library > > was increased to 18 to reflect ABI changes, and avoid compatibility > > problems with the client library in MySQL 5.1. Note that this is an > > incompatible change between 5.5.10 and earlier 5.5 versions, so client > > programs that use the 5.5 client library should be recompiled against > > the 5.5.10 client library. > > Serg, as you have now probably have full insight into the MariaDB 5.5 > code, is it possible to keep the API compatible with MariaDB 5.1 / > MySQL 5.1 to not force all applications to have to be recompiled when > upgrading to 5.5 ? The API incompatibility was caused by migration to CMake. If you want to keep libraries compatible, if's something that Wlad may need to look into. Regards, Sergei

4 6

[Maria-discuss] Suggested sample my.cnf for MariaDB package
by Haidong Ji 25 Mar '11

25 Mar '11

All, I proposed updating the packaged my.cnf sample files before. After some thinking and deliberation, here are my thoughts on the sample my.cnf: 1. There are so many factors that affect settings in my.cnf so it is impossible to please everybody. There is a lot of debates on this. 2. There are values in having just ONE sample my.cnf to get beginners started 3. It should be short 4. It should incorporate best practices, to the extent that best practices can be agreed 5. It should benefit as many people as possible My proposal is pasted below. Comments welcome. At the same time, constant back-and-forth bickering won't benefit much either, in my opinion. Thanks, Haidong "Alex" Ji # MariaDB/Percona/Oracle MySQL programs look for option files in a set of # locations which depend on the deployment platform. # You can copy this option file to one of those # locations. For information about these locations, see: # http://dev.mysql.com/doc/mysql/en/option-files.html # # For detailed definition of settings, see URL below: # http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html # Please do your own research, consult various resources # and experts for optimal settings for your own needs. # Some basic ones are provided below to get you started. # The following options will be passed to all MySQL clients [client] port = 3306 socket = /var/lib/mysql/mysql.sock # The MySQL server [mysqld] port = 3306 socket = /var/lib/mysql/mysql.sock user=mysql log-bin = mysql-bin log-slow-queries sync_binlog = 1 expire_logs_days= 10 wait_timeout = 30 #server-id = last octet of IP address #character-set-server = utf8 [mysqldump] quick

4 6

Re: [Maria-discuss] Suggested sample my.cnf for MariaDB package
by Haidong Ji 20 Mar '11

20 Mar '11

Hi Monty, > The problem is that a too simple my.cnf will please very few (mainly > desktop users) and leave the a bit more advanced user to search on > the net and get 'anything' that looks remotely right. > I respectfully disagree. I think the sample my.cnf is targeted for desktop users, medium-sized company users, and small business owners whose expertise is low and frustration level can run high. A generic, simple, and short my.cnf file serves that constituents well, gains their confidence, and is conducive in building up their comfort and skill level gradually, that they would seek information themselves on the web and/or ask expert for help as the need for performance/scalability rises. But that is AFTER they are comfortable enough with the ease, power, and simplicity of MariaDB. In other words, my view is that its target audience is not big size social network sites, Facebook, Google, and other money-rich financial companies. Those guys probably don't need the built-in sample anyway. Personally, I feel a minimalistic file works the best. This is not a debate of enterprise-worthiness of MariaDB/Percona/MySQL. It is. It's just that I think the sample's target is small scale users, relatively speaking. > I prefer the other approach taken by most unix programs (like postfix) > have taken where you have most important options in the config file, > but the not common ones are commented, with a good explanation of when > to enable the option. This is makes it much faster to get good > options without having to go trough a lot of manuals and web sites... This will make it long, which tends to cause confusion. A typical user/developer will be overwhelmed by the slew of jargons, which is a turn-off, IMHO. Given the importance of key_buffer_cache, however, perhaps that should be mentioned as a comment. > I also personally like the original approach with small, medium, large and > huge default config files, but now with many storage engines these are > also not as valid as they originally where. (And I agree with Arjen > that many of the values needs to be updates). I actually went ahead with that assumption as well initially, but after some thinking, I feel that contradict with the simplicity rule, so I stopped. But the idea is still good, in my mind. So perhaps we can put my-1gb.cnf, my-4gb.cnf, my-32gb.cnf on the knowledgebase? > > What should (at least) be done is to create new sections to the my.cnf > files that one can easily enable if one uses different storage > engines. I would also like to see more options and better comments in > the defaults file. > I agree, but I think we should NOT put in samples for every storage engine under the sun. Once again, simplicity rules in my mind, so we need only cover 2: MyISAM (Aria when Aria is mature enough to phase out MyISAM) and InnoDB. I do realize that in the sample I sent out, I didn't put any for InnoDB, which is probably not a good idea. > I have this week actually added to MariaDB 5.2 some new startup > options and groups to be able to make the option files smaller. > > For example, now in 5.2 both the server and clients are reading the > 'client-server' group, so one doesn't have to specify the socket and > port twice. > > I also added and --log-basedir so that one can specify the name for > all log files with one command (instead of having to use 8+ different > options) and new groups [mariadb] and [client-mariadb] so that one can > use the same config file for MySQL and MariaDB running on the same > computer. > That's fantastic! In summary, it appears that we disagree quite a bit on this. And the main point of contention, I think, is who is the target audience. There is definitely a need to replace/update the current ones, though. What about we package a minimalist sample my.cnf, and put a line in it pointing to a web page on KnowledgeBase that has more samples available with better explanation. Better yet, it is a wiki so the knowledge can be crowd-sourced. Haidong

1 0

[Maria-discuss] client library shared library version changed in 5.5.10
by MARK CALLAGHAN 18 Mar '11

18 Mar '11

I think there was discussion about how client library shared library versions during the Lisbon meeting. >From http://dev.mysql.com/doc/refman/5.5/en/news-5-5-10.html >>> Incompatible Change: The shared library version of the client library was increased to 18 to reflect ABI changes, and avoid compatibility problems with the client library in MySQL 5.1. Note that this is an incompatible change between 5.5.10 and earlier 5.5 versions, so client programs that use the 5.5 client library should be recompiled against the 5.5.10 client library. >>> -- Mark Callaghan mdcallag(a)gmail.com

1 0

[Maria-discuss] Installation problems of 5.2 in debian squueze
by Πρεκατές Αλέξανδρος 13 Mar '11

13 Mar '11

Hi. in debian squeeze trying to install mariaDB5.2 i get a conflict with mysql- core used by aconandi. what should i do? Thanks. aprekates

2 4

[Maria-discuss] askmonty.org vs kb.askmonty.org
by MARK CALLAGHAN 13 Mar '11

13 Mar '11

Why must I have separate logins for askmonty.org and kb.askmonty.org. I was confused by this and assume others will be confused by it. This isn't efficient. After logging into kb.askmonty.org and going to my profile page, there is no option to change my password. I am "Mdcallag" on both kb.askmonty.org and askmonty.org After logging into kb.askmonty.org I get "403 ERROR: Access denied" on everything including: http://kb.askmonty.org/v/plans-for-56 http://kb.askmonty.org/v/legal http://kb.askmonty.org/v/about -- Mark Callaghan mdcallag(a)gmail.com

1 0

Re: [Maria-discuss] Bundle more relevant cnf files in MariaDB package
by Haidong Ji 12 Mar '11

12 Mar '11

cool, Kurt von Finck, I will do it. I couldn't find a good enough translation in English of the “遣将不如激将” phrase, but I like your message. I will dig around the source code base in Launchpad, and get this damn thing started. I bet those config files are old relic from 10 years ago, or even more. 可八方，好，“遣将不如激将”，我会按你的建议去做！ On Fri, Mar 11, 2011 at 5:23 PM, Kurt von Finck <kurt(a)montyprogram.com> wrote: > On Fri, Mar 11, 2011 at 2:56 PM, Haidong Ji <haidong.ji(a)gmail.com> wrote: >> Hi, >> >> I alluded to this in IRC maybe a couple of weeks back. That is, it is >> really to have intelligent default for MariaDB servers. >> >> In particular, the bundled sample my.cnf files, under >> /usr/local/mysql/support-files, such as my-small.cnf, my-medium.cnf, >> my-large.cnf, and my-huge.cnf, are laughable in today's world. In the >> my-huge.cnf, it talks about memory between 1G and 2G as huge. It will >> be good that MariaDB can bundle a different set, like what has been >> mentioned here: >> >> http://datastrangler.com/windpipe/2011/02/24/custom-mysql-config-files-to-e… >> >> I think the sample cnf files in the above blog post is a good starting point. >> >> Thanks, >> Haidong "Alex" Ji >> >> PS: I modified http://kb.askmonty.org/v/installing-mariadb-binary-tarballs >> with the addition of modifying $PATH so a casual user will be able to >> use various MariaDB clients after the install. > > Not to sound snarky, but MariaDB is an open project. I would encourage > you to create sample my.cnf files, and submit them with bzr for review > and approval by a maria-captain. > > The real beauty of an open project is the ability to DIY for the > improvements and changes you think are important. Just Do It. :) > > -- > ./k > > Kurt von Finck > Chief Community And Communications Officer > Monty Program > http://montyprogram.com >

1 0