- developers - lists.mariadb.org

[Maria-developers] Valgrind failure in Buildbot
by Kristian Nielsen 30 Oct '14

30 Oct '14

Hi Sergey, We briefly discussed the new valgrind error in buildbot after your push on IRC: https://buildbot.askmonty.org/buildbot/builders/work-amd64-valgrind/builds/… ==13161== Conditional jump or move depends on uninitialised value(s) ==13161== at 0x4C280AA: memcpy (mc_replace_strmem.c:115) ==13161== by 0x723F6B: Field_varstring::get_key_image(unsigned char*, unsigned int, Field::imagetype) (field.cc:6932) ==13161== by 0x818D5B: key_copy(unsigned char*, unsigned char*, st_key*, unsigned int, bool) (key.cc:146) ==13161== by 0x6DD6FE: Mrr_ordered_index_reader::interrupt_read() (multi_range_read.cc:448) ==13161== by 0x6DE605: Mrr_ordered_rndpos_reader::refill_from_index_reader() (multi_range_read.cc:688) ==13161== by 0x6DE7D2: Mrr_ordered_rndpos_reader::refill_buffer(bool) (multi_range_read.cc:618) ==13161== by 0x6DFB27: DsMrr_impl::dsmrr_init(handler*, st_range_seq_if*, void*, unsigned int, unsigned int, st_handler_buffer*) (multi_range_read.cc:966) ==13161== by 0x91AAE9: ha_innobase::multi_range_read_init(st_range_seq_if*, void*, unsigned int, unsigned int, st_handler_buffer*) (ha_innodb.cc:13961) ==13161== by 0x6DA60A: JOIN_TAB_SCAN_MRR::open() (sql_join_cache.cc:3867) ==13161== by 0x6D7363: JOIN_CACHE::join_matching_records(bool) (sql_join_cache.cc:2249) ==13161== by 0x6D5647: JOIN_CACHE::join_records(bool) (sql_join_cache.cc:2086) ==13161== by 0x5F8259: sub_select_cache(JOIN*, st_join_table*, bool) (sql_select.cc:16717) ==13161== by 0x6D71BA: JOIN_CACHE::generate_full_extensions(unsigned char*) (sql_join_cache.cc:2397) ==13161== by 0x6D74E6: JOIN_CACHE::join_matching_records(bool) (sql_join_cache.cc:2289) ==13161== by 0x6D5647: JOIN_CACHE::join_records(bool) (sql_join_cache.cc:2086) ==13161== by 0x5F8219: sub_select_cache(JOIN*, st_join_table*, bool) (sql_select.cc:16697) I took a quick look. The problem seems to be that the code is copying out the fields of a key from table->record[0] (key_copy()), however, the first length byte of the value is uninitialised. I think it's the field col_varchar_key in the table, testcase is appended below. I checked a bit back up the callstack. The field col_varchar_key is reported as uninitialised already at the start of Mrr_ordered_rndpos_reader::refill_from_index_reader(), but I'm not really familiar with when and how table->record[0] is supposed to be initialised during multi-range read. As you said on IRC, your recent patch is related to interrupt_read(), so probably best if you take a look and see if you can figure out something. Incidentally, I noticed something that looks wrong in key_copy(to_key, from_record, key_info, key_length, with_zerofill) The comments say that the key is copied from `from_record', however actually the function only works if the from_record passed in is table->record[0] (or more precisely, the same record as is installed in key_info->key_part->field). Because it checks the null bit in from_record, but copies the data from table->record[0]. However, this is not the problem in this particular failure (as in this case from_record == table->record[0]), just struck me as something that could easily cause surprises later... Below is a reduced test case that triggers the valgrind error. It did not trigger for me with a debug build, but _did_ trigger (in 5.5) with this build command: EXTRA_FLAGS="-O3 -fno-omit-frame-pointer -Wno-uninitialized" AM_EXTRA_MAKEFLAGS="VERBOSE=1" BUILD/compile-amd64-valgrind-max - Kristian. ----------------------------------------------------------------------- -- source include/have_xtradb.inc set optimizer_switch='mrr=on,mrr_sort_keys=on,index_condition_pushdown=on'; CREATE TABLE t1 ( pk int(11) NOT NULL AUTO_INCREMENT, col_int_key int(11) NOT NULL, col_varchar_key varchar(1) NOT NULL, col_varchar_nokey varchar(1) NOT NULL, PRIMARY KEY (pk), KEY col_int_key (col_int_key), KEY col_varchar_key (col_varchar_key,col_int_key) ) ENGINE=InnoDB; INSERT INTO t1 VALUES (10,8,'v','v'), (11,8,'f','f'), (12,5,'v','v'), (13,8,'s','s'), (14,8,'a','a'), (15,6,'p','p'), (16,7,'z','z'), (17,2,'a','a'), (18,5,'h','h'), (19,7,'h','h'), (20,2,'v','v'), (21,9,'v','v'), (22,142,'b','b'), (23,3,'y','y'), (24,0,'v','v'), (25,3,'m','m'), (26,5,'z','z'), (27,9,'n','n'), (28,1,'d','d'), (29,107,'a','a'); CREATE TABLE t2 ( pk int(11) NOT NULL AUTO_INCREMENT, col_int_key int(11) NOT NULL, col_varchar_key varchar(1) NOT NULL, col_varchar_nokey varchar(1) NOT NULL, PRIMARY KEY (pk), KEY col_int_key (col_int_key), KEY col_varchar_key (col_varchar_key,col_int_key) ) ENGINE=InnoDB; INSERT INTO t2 VALUES (10,3,'i','i'), (11,186,'x','x'), (12,1,'g','g'), (13,8,'q','q'), (14,226,'m','m'), (15,133,'p','p'), (16,6,'e','e'), (17,3,'t','t'), (18,8,'j','j'), (19,5,'h','h'), (20,7,'w','w'); set join_cache_level=6; set join_buffer_size=1536; SELECT count(*), sum(table1.col_int_key*table2.pk) FROM t2 AS table1, t1 AS table2, t2 AS table3 WHERE table3.col_varchar_nokey = table2.col_varchar_key AND table3.pk > table2.col_varchar_nokey ; -----------------------------------------------------------------------

1 0

Re: [Maria-developers] Rev 4323: Set proper alternative names on RHEL7: they should match Fedora names
by Sergei Golubchik 30 Oct '14

30 Oct '14

Hi, Sergey! I cannot verify whether rhel7 alternative names should match Fedora names or they should be something different. Assuming they should match, your patch is ok. On Oct 09, Sergey Vojtovich wrote: > ------------------------------------------------------------ > revno: 4323 > revision-id: svoj(a)mariadb.org-20141009112748-eq3xwoyobgnz0cxj > parent: svoj(a)mariadb.org-20141008141031-xv2x7vrq269202vf > committer: Sergey Vojtovich <svoj(a)mariadb.org> > branch nick: 5.5 > timestamp: Thu 2014-10-09 15:27:48 +0400 > message: > Set proper alternative names on RHEL7: they should match Fedora names. > === modified file 'cmake/cpack_rpm.cmake' > --- a/cmake/cpack_rpm.cmake 2014-10-08 07:35:00 +0000 > +++ b/cmake/cpack_rpm.cmake 2014-10-09 11:27:48 +0000 > @@ -182,7 +182,7 @@ IF(RPM MATCHES "(rhel|centos)5") > ELSEIF(RPM MATCHES "(rhel|centos)6") > ALTERNATIVE_NAME("client" "mysql") > ALTERNATIVE_NAME("shared" "mysql-libs") > -ELSEIF(RPM MATCHES "fedora") > +ELSEIF(RPM MATCHES "fedora" OR RPM MATCHES "(rhel|centos)7") > SET(epoch 1) # this is fedora > ALTERNATIVE_NAME("client" "mariadb") > ALTERNATIVE_NAME("client" "mysql") Regards, Sergei

2 2

[Maria-developers] Fwd: [debian-mysql] Backporting the mysql_no_login plugin
by Colin Charles 30 Oct '14

30 Oct '14

This seems to get a +1 for backporting by at least Honza (RH), so I am wondering if we do this in the 5.5 branch too, that is shipping in many distributions. Begin forwarded message: > From: "Norvald H. Ryeng" <norvald.ryeng(a)oracle.com> > Subject: [debian-mysql] Backporting the mysql_no_login plugin > Date: 24 October 2014 15:49:34 GMT+8 > To: "pkg-mysql-maint(a)lists.alioth.debian.org" <pkg-mysql-maint(a)lists.alioth.debian.org>, "Honza Horak" <hhorak(a)redhat.com>, "Roman Drahtmueller" <draht(a)suse.de> > > Hi package maintainers, > > We have a new plugin in MySQL 5.7 that makes it possible to have > accounts that can't log in: > > CREATE USER foo@localhost IDENTIFIED WITH 'mysql_no_login'; > > The mysql_no_login plugin simply denies all login attempts. This is > useful for users that are created, e.g., to serve as proxy users, or > as owners of stored programs/functions, views or events. > > This new plugin doesn't fix known security defects in the server, but > does provide new and better means to harden security. Best practices > for security include application of least-required privileges, and in > some cases, that means no client connections for privileged > accounts. This new plugin provides means to implement such > restrictions in a standard way. > > Because of the security benefits, we'd like to discuss backporting it > to 5.6. Like you, we don't like big changes to GA releases, but this > time we think it has a good use case, it's safe and has a very low > risk of regressions: > > - Since this is a plugin, it doesn't touch server code > - All new code is in a plugin that must be enabled explicitly by the > DBA > - The code itself is very simple. It's only one line of "real" code > (unconditionally return authentication failure), plus necessary > plugin plumbing to fill out the plugin API. > > If we backport this to 5.6, there are multiple ways to avoid it: > > - Apply a patch from us to remove the plugin > - Don't build it > - Build it, but don't ship it > - Build and ship it, but don't use it (in any case, the DBA has to > enable it and alter the user accounts to use it) > > So what do you think about backporting this? The only thing you'll > notice is one more file in the plugins directory. > > Regards, > > Norvald > > _______________________________________________ > pkg-mysql-maint mailing list > pkg-mysql-maint(a)lists.alioth.debian.org > http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-mysql-maint -- Colin Charles, Chief Evangelist, MariaDB Corporation blog: http://bytebot.net/blog/| t: +6-012-204-3201 | Skype: colincharles

2 1

Re: [Maria-developers] [Commits] b28e6bf: MDEV-6945 MDEV-6948 MDEV-6956 MDEV-6957 MDEV-6953 MDEV-6954 MDEV-6955
by Sergei Golubchik 29 Oct '14

29 Oct '14

Hi, Sanja! On Oct 29, sanja(a)mariadb.com wrote: > --- > mysql-test/r/set_statement.result | 14 ++++++++++++++ > mysql-test/r/set_statement_profiling.result | 4 ++++ > mysql-test/t/set_statement.test | 18 ++++++++++++++++++ > mysql-test/t/set_statement_profiling.test | 9 +++++++++ Did you put set_statement_debug.test in a separate commit? That's good. > sql/set_var.h | 4 +++- > sql/share/errmsg-utf8.txt | 2 ++ > sql/sql_parse.cc | 5 +++++ > sql/sys_vars.cc | 20 +++++++++++--------- > sql/sys_vars.h | 1 + > 9 files changed, 67 insertions(+), 10 deletions(-) > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index bb42807..1633169 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7113,3 +7113,5 @@ ER_STATEMENT_TIMEOUT 70100 > eng "Query execution was interrupted (max_statement_time exceeded)" > ER_SUBQUERIES_NOT_SUPPORTED 42000 > eng "%s does not support subqueries or stored functions." > +ER_SET_STATEMENT_NOT_SUPPORTED 42000 > + eng "Variable '%s' is not supported in SET STATEMENT." Use the text eng "The system variable %.200s cannot be set in SET STATEMENT." > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 6252e89..fd77e5f 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -997,7 +997,7 @@ static bool check_master_connection(sys_var *self, THD *thd, set_var *var) > static Sys_var_session_lexstring Sys_default_master_connection( > "default_master_connection", > "Master connection to use for all slave variables and slave commands", > - SESSION_ONLY(default_master_connection), > + NO_SET_STMT SESSION_ONLY(default_master_connection), Why should this be prohibited? I'd rather have it supported. Use case: setting many connection variables for a single master, like in SET STATEMENT default_master_connection=foobar SET max_relay_log_size=10000000, sql_slave_skip_counter=5; Admittedly, this use case, while valid, isn't a strong argument at the moment - we don't have many per-master variables yet. When we'll have more, this will become more important. So, if fixing default_master_connection is not easy - keep it NO_SET_STMT, it can be enabled later. Why, exactly, does it fail? > NO_CMD_LINE, IN_SYSTEM_CHARSET, > DEFAULT(""), MAX_CONNECTION_NAME, ON_CHECK(check_master_connection)); > #endif > @@ -3527,12 +3528,12 @@ static Sys_var_bit Sys_unique_checks( > #ifdef ENABLED_PROFILING > static Sys_var_bit Sys_profiling( > "profiling", "profiling", > - SESSION_VAR(option_bits), NO_CMD_LINE, OPTION_PROFILING, > + NO_SET_STMT SESSION_VAR(option_bits), NO_CMD_LINE, OPTION_PROFILING, Why should this be prohibited? I'd rather have it supported. Use case: one usually needs to profile a single statement, not everything. > DEFAULT(FALSE)); > > static Sys_var_ulong Sys_profiling_history_size( > "profiling_history_size", "Limit of query profiling memory", > - SESSION_VAR(profiling_history_size), CMD_LINE(REQUIRED_ARG), > + NO_SET_STMT SESSION_VAR(profiling_history_size), CMD_LINE(REQUIRED_ARG), > VALID_RANGE(0, 100), DEFAULT(15), BLOCK_SIZE(1)); > #endif > > @@ -3566,7 +3567,8 @@ static bool check_skip_replication(sys_var *self, THD *thd, set_var *var) > > static Sys_var_bit Sys_skip_replication( > "skip_replication", "skip_replication", > - SESSION_ONLY(option_bits), NO_CMD_LINE, OPTION_SKIP_REPLICATION, > + NO_SET_STMT SESSION_ONLY(option_bits), > + NO_CMD_LINE, OPTION_SKIP_REPLICATION, Why should this be prohibited? I'd rather have it supported. Use case: skip replication of a single statement. A generic version of FLUSH NO_WRITE_TO_BINLOG TABLE REPAIR NO_WRITE_TO_BINLOG TABLE ANALYZE NO_WRITE_TO_BINLOG TABLE OPTIMIZE NO_WRITE_TO_BINLOG TABLE and all ALTER TABLE ... PARTITION variants: ALTER TABLE ... REBUILD PARTITION NO_WRITE_TO_BINLOG ALTER TABLE ... OPTIMIZE PARTITION NO_WRITE_TO_BINLOG ALTER TABLE ... ANALYZE PARTITION NO_WRITE_TO_BINLOG ALTER TABLE ... REPAIR PARTITION NO_WRITE_TO_BINLOG ALTER TABLE ... COALESCE PARTITION NO_WRITE_TO_BINLOG ALTER TABLE ... ADD PARTITION NO_WRITE_TO_BINLOG ALTER TABLE ... REORGANIZE PARTITION NO_WRITE_TO_BINLOG > DEFAULT(FALSE), NO_MUTEX_GUARD, NOT_IN_BINLOG, > ON_CHECK(check_skip_replication)); > > @@ -3688,7 +3690,7 @@ static ulonglong read_rand_seed(THD *thd) > static Sys_var_session_special Sys_rand_seed1( > "rand_seed1", "Sets the internal state of the RAND() " > "generator for replication purposes", > - sys_var::ONLY_SESSION, NO_CMD_LINE, > + NO_SET_STMT sys_var::ONLY_SESSION, NO_CMD_LINE, > VALID_RANGE(0, ULONG_MAX), BLOCK_SIZE(1), > NO_MUTEX_GUARD, IN_BINLOG, ON_CHECK(0), > ON_UPDATE(update_rand_seed1), ON_READ(read_rand_seed)); > @@ -3706,7 +3708,7 @@ static bool update_rand_seed2(THD *thd, set_var *var) > static Sys_var_session_special Sys_rand_seed2( > "rand_seed2", "Sets the internal state of the RAND() " > "generator for replication purposes", > - sys_var::ONLY_SESSION, NO_CMD_LINE, > + NO_SET_STMT sys_var::ONLY_SESSION, NO_CMD_LINE, > VALID_RANGE(0, ULONG_MAX), BLOCK_SIZE(1), > NO_MUTEX_GUARD, IN_BINLOG, ON_CHECK(0), > ON_UPDATE(update_rand_seed2), ON_READ(read_rand_seed)); I'd rather fix rand_seed1/rand_seed2 variables not to be special anymore. Like, move them to SV structure, remove special behavior and so on. Regards, Sergei

1 0

Re: [Maria-developers] [Commits] b18ddd2: Added 'mariadb-galera-server' metapackage to install all
by Sergei Golubchik 28 Oct '14

28 Oct '14

Hi, Nirbhay! On Oct 23, nirbhay(a)skysql.com wrote: > > Added 'mariadb-galera-server' metapackage to install all > necessary Galera-related dependencies. > > --- > debian/dist/Debian/control | 13 ++++++++++--- > debian/dist/Ubuntu/control | 13 ++++++++++--- > 2 files changed, 20 insertions(+), 6 deletions(-) > > diff --git a/debian/dist/Debian/control b/debian/dist/Debian/control > index e68f027..bc48dbc 100644 > --- a/debian/dist/Debian/control > +++ b/debian/dist/Debian/control > @@ -194,9 +194,7 @@ Recommends: libhtml-template-perl > Pre-Depends: mariadb-common, adduser (>= 3.40), debconf > Depends: mariadb-client-10.1 (>= ${source:Version}), libdbi-perl, > perl (>= 5.6), ${shlibs:Depends}, ${misc:Depends}, psmisc, passwd, > - lsb-base (>= 3.0-10), mariadb-server-core-10.1 (>= ${binary:Version}), > - galera (>=25.2), rsync, lsof, socat | netcat, grep, gawk, iproute, > - coreutils, findutils > + lsb-base (>= 3.0-10), mariadb-server-core-10.1 (>= ${binary:Version}) > Provides: mariadb-server, mysql-server, virtual-mysql-server > Conflicts: mariadb-server (<< ${source:Version}), mysql-server (<< ${source:Version}), > mysql-server-4.1, mysql-server-5.0, mysql-server-5.1, mysql-server-5.5, Looks ok. Few comments: 1. What about RPM? 2. Did you compare with, say, non-galera 10.0 - just to make sure all dependencies you removed are, indeed, only for galera and the server doesn't need any of that? 3. I'd also add /etc/mysql/conf.d/galera.cnf (/etc/my.cnf.d/galera.cnf in rpms) that enables wsrep, and may be sets other reasonable defaults for galera. Regards, Sergei

1 0

[Maria-developers] DisconnectEx() problem
by Delveri chick 28 Oct '14

28 Oct '14

Hi i was tracing why mysql_socket_shutdown(...) won't call DisconnectEx(...) as reported in the TODO list and realised DisconnectEx(...) cannot be called in include/mysql/psi/mysql_socket.h. Possible solution would be to add it here or include the *.h where it it defined. Where is it defined please? Regards Delveri

2 1

Re: [Maria-developers] [Commits] 18a7470: MDEV-6939 : Dots in file names of configuration files
by Sergei Golubchik 28 Oct '14

28 Oct '14

Hi, Nirbhay! 1. Why 10.1? 1.0 would be fine too, may be even 5.5 2. In cases like this I usually make two commits, one for a cleanup and one for a bug fix. Anyway, the fix is ok to push, of course. On Oct 24, nirbhay(a)mariadb.com wrote: > revision-id: 18a7470a563c30bffbfe787e4e5406b687597a44 > parent(s): cc656e4d8538326d85e84adbe48d0a3739d55674 > committer: Nirbhay Choubey > branch nick: server > timestamp: 2014-10-24 16:34:54 -0400 > message: > > MDEV-6939 : Dots in file names of configuration files > > Use fn_ext2() to get the file extension from last occurrence > of FN_EXTCHAR ('.') instead. Also made some cosmetic changes > in mysys/mf_fn_ext.c. > > --- > mysys/mf_fn_ext.c | 11 ++++++----- > mysys/my_default.c | 2 +- > 2 files changed, 7 insertions(+), 6 deletions(-) > Regards, Sergei

1 0

[Maria-developers] DATE and GROUP_MIN_MAX optimization
by Alexander Barkov 28 Oct '14

28 Oct '14

Hello Sergey, I'm working on pluggable data types and moving some pieces of the code into new classes for data type handlers. I noticed that GROUP_MIN_MAX optimization works fine with a TIME column with temporal and string literals but does not work for an integer literal: # This script creates demo data: DROP TABLE IF EXISTS t1; CREATE TABLE t1 (id INT, b TIME, KEY(id,b)) ENGINE=MyISAM; INSERT INTO t1 VALUES (1,10),(1,11),(1,12),(1,13),(1,14),(1,15),(1,16),(2,10),(2,11),(2,12),(2,13),(2,14),(2,15),(2,16); SELECT * FROM t1; # Now three EXPLAIN queries: explain SELECT id, MIN(b),MAX(b) FROM t1 WHERE b<TIME'10:10:10' GROUP BY id; explain SELECT id, MIN(b),MAX(b) FROM t1 WHERE b<'10:10:10' GROUP BY id; explain SELECT id, MIN(b),MAX(b) FROM t1 WHERE b<10 GROUP BY id; The first and the second EXPLAIN queries tell that 8 rows are to be examined; tracing the code in debugger shows that group_min_max optimization is active: > mysql> explain SELECT id, MIN(b),MAX(b) FROM t1 WHERE b<TIME'10:10:10' GROUP BY id; > +------+-------------+-------+-------+---------------+------+---------+------+------+---------------------------------------+ > | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | > +------+-------------+-------+-------+---------------+------+---------+------+------+---------------------------------------+ > | 1 | SIMPLE | t1 | range | NULL | id | 9 | NULL | 8 | Using where; Using index for group-by | > +------+-------------+-------+-------+---------------+------+---------+------+------+---------------------------------------+ > 1 row in set (0.00 sec) > > mysql> explain SELECT id, MIN(b),MAX(b) FROM t1 WHERE b<'10:10:10' GROUP BY id; > +------+-------------+-------+-------+---------------+------+---------+------+------+---------------------------------------+ > | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | > +------+-------------+-------+-------+---------------+------+---------+------+------+---------------------------------------+ > | 1 | SIMPLE | t1 | range | NULL | id | 9 | NULL | 8 | Using where; Using index for group-by | > +------+-------------+-------+-------+---------------+------+---------+------+------+---------------------------------------+ > 1 row in set (0.00 sec) > The third EXPLAIN query tells that all 14 rows is to be examined, and tracing in debugging shows that group_min_max is not enabled: > mysql> explain SELECT id, MIN(b),MAX(b) FROM t1 WHERE b<10 GROUP BY id; > +------+-------------+-------+-------+---------------+------+---------+------+------+--------------------------+ > | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | > +------+-------------+-------+-------+---------------+------+---------+------+------+--------------------------+ > | 1 | SIMPLE | t1 | index | NULL | id | 9 | NULL | 14 | Using where; Using index | > +------+-------------+-------+-------+---------------+------+---------+------+------+--------------------------+ > 1 row in set (0.01 sec) Any reasons not to use group_min_max for TIME column in a combination with an integer literal? I have a feeling that it should work. What about decimal and real literals? Thanks.

1 0

[Maria-developers] Problem with KVM version in Buildbot?
by Kristian Nielsen 24 Oct '14

24 Oct '14

Hi Daniel, We have a problem with a build in Buildbot: https://internal.askmonty.org/buildbot/builders/kvm-rpm-centos7_0-x86_64/bu… + qemu-img create -b /kvm/vms/vm-centos7_0-x86_64-build.qcow2 -f qcow2 /dev/shm/vm-tmp-2302.qcow2 qemu-img: '' uses a qcow2 feature which is not supported by this qemu version: QCOW version 3 qemu-img: Could not open '/kvm/vms/vm-centos7_0-x86_64-build.qcow2' Failed to clone base image, aborting This failure happens on the build host `terrier'. I assume the problem is that the image was created on a machine with a newer KVM? The host terrier seems to have a rather old one: knielsen@terrier:~> kvm --version QEMU emulator version 0.13.0 (qemu-kvm-devel), Copyright (c) 2003-2008 Fabrice Bellard How do you think we should solve the problem? Should we upgrade terrier? Or should we alternatively make sure to create new images in a backward-compatible version (and re-create affected images) ? Also, another issue related to VM images. Our old images were created with rather small disk size. I noticed that some new images are created much larger, for example: $ qemu-img info /kvm/vms/vm-sid-amd64-build.qcow2 image: /kvm/vms/vm-sid-amd64-build.qcow2 file format: qcow2 virtual size: 40G (42949672960 bytes) disk size: 3.1G cluster_size: 65536 Format specific information: compat: 0.10 That is fine, I think, it is indeed annoying when we run out of space in a build VM. However, the reason I noticed this is because host bb02 ran out of space in /run/shm/ due to a 16 GB copy of this image. So when we use larger VM images, we cannot use the optimisation to have the temporary copies in ramdisk, lest we get disk full errors. If the virtual size is larger to (ram disk size) / (max_builds), then we need to disable using ramdisk for buildbot for that image. I will try to fix the config to not use ramdisk for existing images that have too large virtual size. But it is something to keep in mind for the future. - Kristian.

3 3

Re: [Maria-developers] SQL Accelerator FPGA/ASIC
by Nasser Ghoseiri 24 Oct '14

24 Oct '14

Dear Jan, Thank you for your response. I have background in SQL (MS, Sybase, MySQL, Postgre), so luckily I'm not a beginner in query writing, however, I have absolutely no background in how databases work on background (other than hash-table). The idea is not really SIMD here, but a specially designed hardware to perform database functions much faster than the conventional DISK<->RAM<->CACHE<->CPU model. What I'm struggling to figure is that the latency caused by a enterprise-grade SQL script (not very correct and advanced code, but not very low-quality either), is Memory-Related or Arithmetic related or both? Here is what can be made for example: 1) A card with 16 DDR3 Chips on board and a Central FPGA, allowing very fast access to data, and like 100 small CPUs integrated in the FPGA to process a query. (a query that was written to take advantage of these features however) 2) A card with XDR Memory chips and like 32GB of Flash Chips to achieve very high-speed storage, retrieval (SSD<->RAM) and an FPGA that has many cores inside. A query is executed by FPGA pulling large chunk of data from Flashes, putting them in XDR RAM chips and processes them in a parallel manner 3) A card with XDR Memory chips and an FPGA on board, connected to a PCI Express 8x which receives all the data it needs from the host computer, and does the parallel processing inside itself, returning it back to the host. 4) A card with an FPGA inside, having 100 small CPUs and PCI Express to take commands from host computer and process in a parallel manner 5) A card with an FPGA, not having small processors inside, but special circuitry that does either lookup, sorting, etc in a very fast manner These are a few examples that I gave. The difference among them is the notion of having RAM, having a cold-storage, having an FPGA with many CPUs inside or an FPGA with special dedicated circuitry that performs special functions in parallel that is difficult for a CPU to handle (imaging adding 300 numbers to another 300 numbers in a single cycle, where a CPU with SIMD would take much longer, or maybe a string processor that processes many string columns with PATINDEX, etc in parallel where a CPU would be slow as it can't handle them in parallel). So what I'm researching is whether beneath the SQL pyramid, there is a memory intensive operation going on or logic-intensive. What to look for and what to aim for... Thanks in advance for your help, Nasser On Friday, October 24, 2014 7:49 PM, Jan Lindström <jan.lindstrom(a)mariadb.com> wrote: Hi, This idea of SIMD (single instruction multiple data) processing is not totally new one, similarly the idea to perform SQL-operations inside GPU or GPGA is not new. In traditional relational databases problematic is the fact that e.g. in your example TableX contains several columns, picking columns A and B from pages that reside first on disk, then on main-memory and finally on L1-L3 cache is not cheap, and then they are not on continuous memory. This is because page in cache would contain values for other columns that we not even need. In columnar database architecture this would be a lot easier, you just feed column containing values for A and B directly to SIMD operation and every page in main memory would contain a lot more values to process compared to traditional relational database where page would contain also values for columns that we really do not even need to process the result set of query. Anyway, I find the proposal interesting and challenging. R: Jan Lindström On Thu, Oct 23, 2014 at 3:39 PM, Nasser Ghoseiri <cdmcsd(a)yahoo.com> wrote: Dear Serg, > > >Following our little chat in IRC, I'm writing this email to explain in more detail what the idea is. My name is Nasser GHOSEIR (Founder, CTO of Butterfly labs), which I must note that this project is not from Butterfly Labs, >but will be a new company in Europe. > > >Our idea is to find a way to accelerate SQL query processing by either: > > >1) Creating an FPGA solution (Which will later evolve into ASIC), that has like 400 processors in itself, allow distribute calculation of some kind >2) Creating an FPGA/ASIC solution that performs large number of unrelated tasks (such as addition, multiplication, etc) in parallel >3) Creating an FPGA/ASIC solution that allows very high-speed access to data with some PRE-PROCESSING involved to accelerate the calculation >4) Creating a very high-speed storage solution, but no pre-processing. > > >An example to give is for 200,000 rows, imagine: "SELECT A+B, C FROM TableX WHERE C > 0". If there are 200,000 records, the processor has to perform 200,000 additions (A+B). >A CPU will handle these additions one-by-one (or maybe few-by-few if SSE, etc is used). However, an FPGA/ASIC solution can perform 1000 additions in a single-cycle. This results in acceleration. > > >Now, to what extend this can be effective, or what other solutions (maybe string processing?) can be implemented to accelerate the SQL processing is a question to me right now. But we do have experience in >making extremely fast processors (our Butterfly Labs Monarch chip performs 400 billion double-SHA256 hashes in 1 second, around 20,000 times faster than the best intel XEON processor can do only twenty >million hashes per second). > > >The project is in brain-storming phase, and we are aware that this solution will be useful to large enterprises, or companies that have to deal with 200,000 rows in a single "SELECT" query. The idea is to integrate >some features in hardware, and then re-write a portion of MariaDB to take advantage of the new resources. Also, it is possible that users/companies will need to re-write their query to make it compliant. > > >You can reach my by mail at cdmcsd(a)yahoo.com or by phone at +33 6 72 17 26 19 (France). > > > > > > >Best Regards, >Nasser GHOSEIRI > > > > > > > > >_______________________________________________ >Mailing list: https://launchpad.net/~maria-developers >Post to : maria-developers(a)lists.launchpad.net >Unsubscribe : https://launchpad.net/~maria-developers >More help : https://help.launchpad.net/ListHelp > >

1 0