- developers - lists.mariadb.org

Re: [Maria-developers] Gsoc 2016 Mdev 371 Unique index for blob
by Sergei Golubchik 15 Mar '16

15 Mar '16

Hi, Sachin! First: please send questions like this to maria-developers(a)lists.launchpad.net it's a public mailing list dedicated to MySQL and MariaDB internals, source code, and related things. I am subscribed, so I'll see you mail there, and you may be sure I will, because it won't be accidentally catched by my spam filter, or sorted out in some obscure folder. Furthermore other subscribers will see your question and could reply if I will be not available (e.g. I could be travelling). Thank … [View More]

1 0

Re: [Maria-developers] [Commits] a0c06ba: Preliminary implementation for the aggregate sum function as a window function
by Sergey Petrunia 14 Mar '16

14 Mar '16

Hi Vicentiu, I think it add_helper() is a really poor choice of name when the function can do removal. Maybe add_or_remove() name would be better? On Mon, Mar 14, 2016 at 03:44:48PM +0200, Vicentiu Ciorbaru wrote: > revision-id: a0c06ba1edb54c8c4705189c0455137a85658297 (mariadb-10.1.8-154-ga0c06ba) > parent(s): ce8a0d8e19e7bf1486a2c80ff6b7a30ef35bf99f > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2016-03-14 15:42:00 +0200 > message: > > … [View More]Preliminary implementation for the aggregate sum function as a window function > > This implementation does not deal with the case where removal of > elements from the window frame causes the item to turn to a null value. > > --- > mysql-test/r/win_sum.result | 42 ++++++++++++++++++++++++++++++++++++++++++ > mysql-test/t/win_sum.test | 32 ++++++++++++++++++++++++++++++++ > sql/item_sum.cc | 29 +++++++++++++++++++++++++---- > sql/item_sum.h | 4 ++++ > sql/sql_window.cc | 1 + > 5 files changed, 104 insertions(+), 4 deletions(-) > > diff --git a/mysql-test/r/win_sum.result b/mysql-test/r/win_sum.result > new file mode 100644 > index 0000000..1db6c6e > --- /dev/null > +++ b/mysql-test/r/win_sum.result > @@ -0,0 +1,42 @@ > +create table t1 ( > +pk int primary key, > +a int, > +b int, > +c real > +); > +insert into t1 values > +(101 , 0, 10, 1.1), > +(102 , 0, 10, 2.1), > +(103 , 1, 10, 3.1), > +(104 , 1, 10, 4.1), > +(108 , 2, 10, 5.1), > +(105 , 2, 20, 6.1), > +(106 , 2, 20, 7.1), > +(107 , 2, 20, 8.15), > +(109 , 4, 20, 9.15), > +(110 , 4, 20, 10.15), > +(111 , 5, NULL, 11.15), > +(112 , 5, 1, 12.25), > +(113 , 5, NULL, 13.35), > +(114 , 5, NULL, 14.50), > +(115 , 5, NULL, 15.65); > +select pk, a, b, sum(b) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING), > +sum(c) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) > +from t1; > +pk a b sum(b) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) sum(c) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) > +101 0 10 20 3.2 > +102 0 10 20 3.2 > +103 1 10 20 7.199999999999999 > +104 1 10 20 7.199999999999999 > +105 2 20 40 13.2 > +106 2 20 60 21.35 > +107 2 20 50 20.35 > +108 2 10 30 13.250000000000002 > +109 4 20 40 19.3 > +110 4 20 40 19.3 > +111 5 NULL 1 23.4 > +112 5 1 1 36.75 > +113 5 NULL 1 40.1 > +114 5 NULL 0 43.5 > +115 5 NULL 0 30.15 > +drop table t1; > diff --git a/mysql-test/t/win_sum.test b/mysql-test/t/win_sum.test > new file mode 100644 > index 0000000..3c12b08 > --- /dev/null > +++ b/mysql-test/t/win_sum.test > @@ -0,0 +1,32 @@ > +create table t1 ( > + pk int primary key, > + a int, > + b int, > + c real > +); > + > + > +insert into t1 values > +(101 , 0, 10, 1.1), > +(102 , 0, 10, 2.1), > +(103 , 1, 10, 3.1), > +(104 , 1, 10, 4.1), > +(108 , 2, 10, 5.1), > +(105 , 2, 20, 6.1), > +(106 , 2, 20, 7.1), > +(107 , 2, 20, 8.15), > +(109 , 4, 20, 9.15), > +(110 , 4, 20, 10.15), > +(111 , 5, NULL, 11.15), > +(112 , 5, 1, 12.25), > +(113 , 5, NULL, 13.35), > +(114 , 5, NULL, 14.50), > +(115 , 5, NULL, 15.65); > + > +--sorted_result > +select pk, a, b, sum(b) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING), > + sum(c) over (partition by a order by pk ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING) > + > +from t1; > + > +drop table t1; > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index c78c206..3f4853a 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -1318,25 +1318,39 @@ void Item_sum_sum::fix_length_and_dec() > bool Item_sum_sum::add() > { > DBUG_ENTER("Item_sum_sum::add"); > + add_helper(false); > + DBUG_RETURN(0); > +} > + > +void Item_sum_sum::add_helper(bool perform_removal) > +{ > + DBUG_ENTER("Item_sum_sum::add_helper"); > if (Item_sum_sum::result_type() == DECIMAL_RESULT) > { > my_decimal value; > const my_decimal *val= aggr->arg_val_decimal(&value); > if (!aggr->arg_is_null(true)) > { > - my_decimal_add(E_DEC_FATAL_ERROR, dec_buffs + (curr_dec_buff^1), > - val, dec_buffs + curr_dec_buff); > + if (perform_removal) > + my_decimal_sub(E_DEC_FATAL_ERROR, dec_buffs + (curr_dec_buff ^ 1), > + dec_buffs + curr_dec_buff, val); > + else > + my_decimal_add(E_DEC_FATAL_ERROR, dec_buffs + (curr_dec_buff ^ 1), > + val, dec_buffs + curr_dec_buff); > curr_dec_buff^= 1; > null_value= 0; > } > } > else > { > - sum+= aggr->arg_val_real(); > + if (perform_removal) > + sum-= aggr->arg_val_real(); > + else > + sum+= aggr->arg_val_real(); > if (!aggr->arg_is_null(true)) > null_value= 0; > } > - DBUG_RETURN(0); > + DBUG_VOID_RETURN; > } > > > @@ -1386,6 +1400,13 @@ my_decimal *Item_sum_sum::val_decimal(my_decimal *val) > return val_decimal_from_real(val); > } > > +void Item_sum_sum::remove() > +{ > + DBUG_ENTER("Item_sum_sum::remove"); > + add_helper(true); > + DBUG_VOID_RETURN; > +} > + > /** > Aggregate a distinct row from the distinct hash table. > > diff --git a/sql/item_sum.h b/sql/item_sum.h > index f1f5b22..24dfd9b 100644 > --- a/sql/item_sum.h > +++ b/sql/item_sum.h > @@ -773,6 +773,10 @@ class Item_sum_sum :public Item_sum_num, > return has_with_distinct() ? "sum(distinct " : "sum("; > } > Item *copy_or_same(THD* thd); > + void remove(); > + > +private: > + void add_helper(bool perform_removal); > }; > > > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index df9899e..f0d74b1 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -1584,6 +1584,7 @@ bool JOIN::process_window_functions(List<Item> *curr_fields_list) > } > case Item_sum::COUNT_FUNC: > case Item_sum::SUM_BIT_FUNC: > + case Item_sum::SUM_FUNC: > { > /* > Frame-aware window function computation. It does one pass, but > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog [View Less]

2 1

[Maria-developers] Gsoc 2016 Mdev 371 Unique index for blob
by Sachin Setia 14 Mar '16

14 Mar '16

Hello Developers My name is sachin , a third year undergrad student in India.I am interested in gsoc 2016 I was debugging the maria db for following query create table c4(a1 int primary key,b1 int unique,c1 blob) engine=myisam; In mysql_prepare_create_table there is a function name prepare_create_field which sets the sql_field pack flag . for a1 field it sets the pack flag =27 . do not quiet get what it means.

2 1

Re: [Maria-developers] c0b1148: MDEV-9659 : AWS KMS encryption plugin
by Sergei Golubchik 12 Mar '16

12 Mar '16

Hi, Wlad! Just a couple of comments: On Mar 11, wlad(a)mariadb.com wrote: > + # Relax AWS C++ SDK unreasonably high requirements for CMake version. Use replace utility (from MariaDB build) > + # to patch their CMakeLists.txt Nice solution. But you might need to add a dependency, to ensure that this external project target is built after replace. Unless cmake can, somehow, derive it automatically (you can test it by building just this specific target in a clean tree). > + MY_DIR … [View More]

2 1

Re: [Maria-developers] 66cd0f2: MDEV-9704: ALTER TABLE does not work from client
by Sergei Golubchik 12 Mar '16

12 Mar '16

Hi, OleksandrByelkin! On Mar 11, OleksandrByelkin wrote: > revision-id: 66cd0f2f02bb45707b38e983186ab8e7aae21da9 (mariadb-10.1.8-156-g66cd0f2) > parent(s): 75d8544c0bc611e82078b9e5f59caef9105e4892 > committer: Oleksandr Byelkin > timestamp: 2016-03-11 16:55:57 +0100 > message: > > MDEV-9704: ALTER TABLE does not work from client > > Do not try to send progress packets if client was not informed about them. > > --- > sql/sql_acl.cc | 2 -- > 1 file … [View More]

1 0

Re: [Maria-developers] MDEV-9659 Encryption plugin that uses AWS Key management service
by Sergei Golubchik 12 Mar '16

12 Mar '16

Hi, Wlad! On Mar 03, wlad(a)mariadb.com wrote: > revision-id: c1ed3c0267da2ece7664e9f876314979114b2186 (mariadb-10.1.11-26-gc1ed3c0) > parent(s): a62db8186e40a62366bc86f49a0195e9b687d86d > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2016-03-03 21:29:29 +0100 > message: > > MDEV-9659 Encryption plugin that uses AWS Key management service Looks pretty good, thanks! See comment below... > diff --git a/cmake/plugin.cmake b/cmake/plugin.… [View More]cmake > index c24239c..361cb59 100644 > --- a/cmake/plugin.cmake > +++ b/cmake/plugin.cmake > @@ -74,7 +74,9 @@ MACRO(MYSQL_ADD_PLUGIN) > SET(compat "with${compat}") > ENDIF() > > - IF (compat STREQUAL ".") > + IF (ARG_DISABLED) > + SET(howtobuild NO) > + ELSEIF (compat STREQUAL ".") > SET(howtobuild DYNAMIC) > ELSEIF (compat STREQUAL "with.") > IF (NOT ARG_MODULE_ONLY) > @@ -122,7 +124,7 @@ MACRO(MYSQL_ADD_PLUGIN) > > # Build either static library or module > IF (PLUGIN_${plugin} MATCHES "(STATIC|AUTO|YES)" AND NOT ARG_MODULE_ONLY > - AND NOT ARG_DISABLED AND NOT ARG_CLIENT) > + AND NOT ARG_CLIENT) > > IF(CMAKE_GENERATOR MATCHES "Makefiles|Ninja") > # If there is a shared library from previous shared build, > @@ -178,8 +180,7 @@ MACRO(MYSQL_ADD_PLUGIN) > SET (mysql_optional_plugins ${mysql_optional_plugins} PARENT_SCOPE) > ENDIF() > ELSEIF(PLUGIN_${plugin} MATCHES "(DYNAMIC|AUTO|YES)" > - AND NOT ARG_STATIC_ONLY AND NOT WITHOUT_DYNAMIC_PLUGINS > - AND NOT ARG_DISABLED) > + AND NOT ARG_STATIC_ONLY AND NOT WITHOUT_DYNAMIC_PLUGINS) > > ADD_VERSION_INFO(${target} MODULE SOURCES) > ADD_LIBRARY(${target} MODULE ${SOURCES}) a bit more of the explanation would've been nice. in the changeset comment or - even better - pushing this change separately. you can select what files to include in the commit, you know... > diff --git a/plugin/aws_key_management/CMakeLists.txt b/plugin/aws_key_management/CMakeLists.txt > new file mode 100644 > index 0000000..ec9ee46 > --- /dev/null > +++ b/plugin/aws_key_management/CMakeLists.txt > @@ -0,0 +1,144 @@ > +# We build parts of AWS C++ SDK as CMake external project > +# The restrictions of the SDK (https://github.com/awslabs/aws-sdk-cpp/blob/master/README.md) > +# are > + > +# - OS : Windows,Linux or OSX > +# - C++11 compiler : VS2013+, gcc 4.7+, clang 3.3+ > +# - libcurl development package needs to be present on Unixes > +# > +# Since we're using ExternalProject that reads from git repository > +# we also require GIT to be present on the build machine > + > + > +# Give message why the building this plugin is skipped (only if -DVERBOSE is defined) > +# and and bail out. > +MACRO(SKIP_AWS_PLUGIN msg) > + IF(DEFINED VERBOSE) VERBOSE - is it used somewhere else? Or you've invented it on the spot for this plugin? I understand that it can be used in other cmake files, but is it? > + MESSAGE(STATUS "Skip aws_key_management - ${msg}") > + RETURN() > + ENDIF() > +ENDMACRO() Ok, as discussed on irc, I'd suggest to add here a check whether AWS SDK is already installed. Like, CheckLibraryExists or CheckSymbolExists. And if not - then try to download and compile it (which would require new cmake, git, curl, etc). > +IF(CMAKE_VERSION VERSION_LESS "2.8.8") > + SKIP_AWS_PLUGIN("CMake is too old") > +ENDIF() > + > +FIND_PACKAGE(Git) > +IF(NOT GIT_FOUND) > + SKIP_AWS_PLUGIN("no GIT") > +ENDIF() > + > +INCLUDE(ExternalProject) > +IF (NOT(WIN32 OR APPLE OR (CMAKE_SYSTEM_NAME MATCHES "Linux"))) > + SKIP_AWS_PLUGIN("OS unsupported by AWS SDK") > +ENDIF() > + > +IF(UNIX) > + FIND_PACKAGE(CURL) > + IF(NOT CURL_FOUND) > + SKIP_AWS_PLUGIN("AWS C++ SDK requires libcurl development package") > + ENDIF() > + SET(PIC_FLAG -fPIC) > +ENDIF() > + > +SET(CXX11_FLAGS) > +SET(OLD_COMPILER_MSG "AWS SDK requires c++11 -capable compiler (minimal supported versions are g++ 4.7, clang 3.3, VS2103)") > +IF(CMAKE_CXX_COMPILER_ID MATCHES "GNU") > + EXECUTE_PROCESS(COMMAND ${CMAKE_CXX_COMPILER} -dumpversion OUTPUT_VARIABLE GCC_VERSION) eh? I thought cmake knows compiler versions... let me see This one: CMAKE_<LANG>_COMPILER_VERSION > + IF (GCC_VERSION VERSION_LESS 4.7) > + SKIP_AWS_PLUGIN("${OLD_COMPILER_MSG}") > + ENDIF() > + SET(CXX11_FLAGS "-std=c++11") Instead of checking the compiler version, I would rather suggest to check for a feature that you need to use. Like MY_CHECK_CXX_COMPILER_FLAG(-std=c++11) IF (NOT HAVE_CXX__STD_C__11) SKIP_AWS_PLUGIN("${OLD_COMPILER_MSG}") ENDIF() Something like that. Not sure about HAVE_CXX__STD_C__11 name, though. > +ELSEIF (CMAKE_CXX_COMPILER_ID MATCHES "Clang") > + IF (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 3.3) > + SKIP_AWS_PLUGIN("${OLD_COMPILER_MSG}") > + ENDIF() > + SET(CXX11_FLAGS "-stdlib=libc++") Same. > +ELSEIF(MSVC) > + IF (MSVC_VERSION LESS 1800) > + SKIP_AWS_PLUGIN("${OLD_COMPILER_MSG}") > + ENDIF() > +ELSE() > + SKIP_AWS_PLUGIN("Compiler not supported by AWS C++ SDK") > +ENDIF() > + > +IF(WIN32) > + SET(EXTRA_SDK_CMAKE_FLAGS -DCMAKE_CXX_FLAGS_DEBUGOPT="" -DCMAKE_EXE_LINKER_FLAGS_DEBUGOPT="") For consistency you can put this inside ELSEIF(MSVC), that's where you set options for gcc and clang. > +ENDIF() > + > +FIND_PROGRAM(PERL_EXECUTABLE perl) > + > +IF(PERL_EXECUTABLE) > + # CMake minimimal version for building AWS SDK can be less than 3.1.2 > + # Downgrade this requirement. > + SET(AWS_SDK_PATCH_COMMAND > + PATCH_COMMAND ${PERL_EXECUTABLE} -i.bak -pe s/3.1.2/2.8/g ${CMAKE_BINARY_DIR}/aws-sdk-cpp/CMakeLists.txt) 1. Without perl it'll require 3.1.2, right? 2. Couldn't you do that with cmake alone? 3. Is that safe at all - you pull the latest version from git, they can start using cmake-3.1.2 features any time. > +ENDIF() > + > +ExternalProject_Add( > + aws_sdk_cpp > + GIT_REPOSITORY "https://github.com/awslabs/aws-sdk-cpp.git" > + UPDATE_COMMAND "" > + ${AWS_SDK_PATCH_COMMAND} > + SOURCE_DIR "${CMAKE_BINARY_DIR}/aws-sdk-cpp" > + CMAKE_ARGS > + -DBUILD_ONLY=aws-cpp-sdk-kms -DSTATIC_LINKING=1 > + "-DCMAKE_CXX_FLAGS_DEBUG=${CMAKE_CXX_FLAGS_DEBUG} ${PIC_FLAG}" > + "-DCMAKE_CXX_FLAGS_RELWITHDEBINFO=${CMAKE_CXX_FLAGS_RELWITHDEBINFO} ${PIC_FLAG}" > + "-DCMAKE_CXX_FLAGS_RELEASE=${CMAKE_CXX_FLAGS_RELEASE} ${PIC_FLAG}" > + "-DCMAKE_CXX_FLAGS_MINSIZEREL=${CMAKE_CXX_FLAGS_MINSIZEREL} ${PIC_FLAG}" > + ${EXTRA_SDK_CMAKE_FLAGS} > + -DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_DIR}/aws_sdk_cpp > + TEST_COMMAND "" > +) > + > +# Figure out where AWS installs SDK libraries > +# The below is defined in AWS SDK's CMakeLists.txt > +# (and their handling is weird, every OS has special install directory) > +if(WIN32) > + SET(SDK_INSTALL_BINARY_PREFIX "windows") > +ELSEIF(APPLE) > + SET(SDK_INSTALL_BINARY_PREFIX "mac") > +ELSEIF(UNIX) > + SET(SDK_INSTALL_BINARY_PREFIX "linux") > +ENDIF() > +IF(NOT APPLE) > + IF(CMAKE_SIZEOF_VOID_P EQUAL 8) > + SET(SDK_INSTALL_BINARY_PREFIX "${SDK_INSTALL_BINARY_PREFIX}/intel64") > + ELSE() > + SET(SDK_INSTALL_BINARY_PREFIX "${SDK_INSTALL_BINARY_PREFIX}/ia32") > + ENDIF() > +ENDIF() > +IF(CMAKE_CONFIGURATION_TYPES) > + SET(SDK_INSTALL_BINARY_PREFIX "${SDK_INSTALL_BINARY_PREFIX}/${CMAKE_CFG_INTDIR}") > +ENDIF() > + > +# We do not need to build the whole SDK , just 2 of its libs > +set(AWS_SDK_LIBS aws-cpp-sdk-core aws-cpp-sdk-kms) > + > +FOREACH(lib ${AWS_SDK_LIBS}) > + ADD_LIBRARY(${lib} STATIC IMPORTED) > + ADD_DEPENDENCIES(${lib} aws_sdk_cpp) > + SET(loc "${CMAKE_BINARY_DIR}/aws_sdk_cpp/lib/${SDK_INSTALL_BINARY_PREFIX}/${CMAKE_STATIC_LIBRARY_PREFIX}${lib}${CMAKE_STATIC_LIBRARY_SUFFIX}") > + SET_TARGET_PROPERTIES(${lib} PROPERTIES IMPORTED_LOCATION ${loc}) > + IF(WIN32) > + SET_TARGET_PROPERTIES(${lib} PROPERTIES IMPORTED_LINK_INTERFACE_LIBRARIES "bcrypt;winhttp;wininet;userenv") > + ELSE() > + SET_TARGET_PROPERTIES(${lib} PROPERTIES IMPORTED_LINK_INTERFACE_LIBRARIES "${SSL_LIBRARIES};${CURL_LIBRARIES}") > + ENDIF() > +ENDFOREACH() > + > +IF(CMAKE_SYSTEM_NAME MATCHES "Linux") > + # Need whole-archive , otherwise static libraries are not linked > + SET(AWS_SDK_LIBS -Wl,--whole-archive ${AWS_SDK_LIBS} -Wl,--no-whole-archive) > +ENDIF() > + > +SET_TARGET_PROPERTIES(aws_sdk_cpp PROPERTIES EXCLUDE_FROM_ALL TRUE) > +INCLUDE_DIRECTORIES(${CMAKE_BINARY_DIR}/aws_sdk_cpp/include) > +SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${CXX11_FLAGS}") > +MYSQL_ADD_PLUGIN(aws_key_management aws_key_management_plugin.cc > + MODULE_ONLY COMPONENT aws-key-management > + LINK_LIBRARIES ${AWS_SDK_LIBS} DISABLED) > diff --git a/plugin/aws_key_management/aws_key_management_plugin.cc b/plugin/aws_key_management/aws_key_management_plugin.cc > new file mode 100644 > index 0000000..fa7d042 > --- /dev/null > +++ b/plugin/aws_key_management/aws_key_management_plugin.cc > @@ -0,0 +1,611 @@ > +/* > + Copyright (c) 2016 MariaDB Corporation > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > + > + > +#include <my_global.h> > +#include <my_pthread.h> > +#include <my_sys.h> > +#include <mysql/plugin_encryption.h> > +#include <my_crypt.h> > +#include <string.h> > +#include <stdio.h> > +#include <stdlib.h> > +#include <mysqld_error.h> > +#include <base64.h> > +#include <map> > +#include <algorithm> > +#include <string> > +#include <vector> > +#include <iterator> > +#include <sstream> > +#include <fstream> > +#ifndef _WIN32 > +#include <sys/types.h> > +#include <dirent.h> > +#endif Use cmake checks when there are some, HAVE_DIRENT_H, HAVE_SYS_TYPES_H, HAVE_STRING_H, HAVE_STDLIB_H. Hmm, HAVE_STDLIB_H doesn't make sense to me :) could you remove it from configure.cmake, please? In a separate changeset. May be HAVE_SYS_TYPES_H too, not sure. But do use HAVE_DIRENT_H and HAVE_STDLIB_H above instead of _WIN32. > +#include <aws/core/client/AWSError.h> > +#include <aws/core/utils/logging/AWSLogging.h> > +#include <aws/core/utils/logging/ConsoleLogSystem.h> > +#include <aws/kms/KMSClient.h> > +#include <aws/kms/model/DecryptRequest.h> > +#include <aws/kms/model/DecryptResult.h> > +#include <aws/kms/model/GenerateDataKeyWithoutPlaintextRequest.h> > +#include <aws/kms/model/GenerateDataKeyWithoutPlaintextResult.h> > +#include <aws/core/utils/Outcome.h> > +using namespace std; > +using namespace Aws::KMS; > +using namespace Aws::KMS::Model; > +using namespace Aws::Utils::Logging; > +extern void sql_print_error(const char *format, ...); > +extern void sql_print_warning(const char *format, ...); > +extern void sql_print_information(const char *format, ...); > + > + > +/* Plaintext key info struct */ > +struct KEY_INFO > +{ > + unsigned int key_id; > + unsigned int key_version; > + unsigned int length; > + unsigned char data[MY_AES_MAX_KEY_LENGTH]; > + bool load_failed; /* if true, do not attempt to reload?*/ > +public: > + KEY_INFO() : key_id(0), key_version(0), length(0), load_failed(false){}; > +}; > +#define KEY_ID_AND_VERSION(key_id,version) ((longlong)key_id << 32 | version) > + > +/* Cache for the latest version, per key id */ > +static std::map<uint, uint> latest_version_cache; > + > +/* Cache for plaintext keys */ > +static std::map<ulonglong, KEY_INFO> key_info_cache; > + > +static const char *key_spec_names[]={ "AES_128", "AES_256", 0 }; > + > +/* Plugin variables */ > +static char* master_key_id; > +static unsigned long key_spec; > +static unsigned long log_level; > +static int rotate_key; > + > +/* AWS functionality*/ > +static int aws_decrypt_key(const char *path, KEY_INFO *info); > +static int aws_generate_datakey(uint key_id, uint version); > + > +static int extract_id_and_version(const char *name, uint *id, uint *ver); > +static unsigned int get_latest_key_version(unsigned int key_id); > +static unsigned int get_latest_key_version_nolock(unsigned int key_id); > +static int load_key(KEY_INFO *info); > + > +/* Mutex to serialize access to caches */ > +static pthread_mutex_t mtx; Use PSI, please. That's mysql_mutex_t, etc. > + > +static Aws::KMS::KMSClient *client; > + > +/* Redirect AWS trace to error log */ > +class MySQLLogSystem : public Aws::Utils::Logging::FormattedLogSystem > +{ > +public: > + > + using Base = FormattedLogSystem; > + MySQLLogSystem(LogLevel logLevel) : > + Base(logLevel) > + { > + } > + virtual LogLevel GetLogLevel(void) const override > + { > + return (LogLevel)log_level; > + } > + virtual ~MySQLLogSystem() > + { > + } > + > +protected: > + virtual void ProcessFormattedStatement(Aws::String&& statement) override > + { > +#ifdef _WIN32 > + /* > + On Windows, we can't use C runtime functions to write to stdout, > + because we compile with static C runtime, so plugin has a stdout > + different from server. Thus we're using WriteFile(). > + */ > + DWORD nSize= (DWORD)statement.size(); > + DWORD nWritten; > + const char *s= statement.c_str(); > + HANDLE h= GetStdHandle(STD_OUTPUT_HANDLE); > + > + WriteFile(h, s, nSize, &nWritten, NULL); > +#else > + printf("%s", statement.c_str()); printf? You have declared sql_print_information above, why wouldn't you use it here? > +#endif > + } > +}; > + > +/* > + Read filenames from the current directory. > + This is used in plugin initilization. We check for > + plugin specific names aws-kms-key.<id>.<version> > + to find out what keys/key versions are already there. > +*/ > +static vector<string> read_dir() > +{ > + vector<string> v; > + DBUG_ENTER("read_dir"); > +#ifdef _WIN32 > + WIN32_FIND_DATA ffd; > + HANDLE h= FindFirstFile("*", &ffd); > + if (h == INVALID_HANDLE_VALUE) > + { > + DBUG_RETURN(v); > + } > + do > + { > + /* handle file possibly updates caches*/ > + v.push_back(ffd.cFileName); > + } while (FindNextFile(h, &ffd)); > + FindClose(h); > +#else > + DIR *dir= opendir("."); > + if (!dir) > + { > + DBUG_RETURN(v); > + } > + struct dirent *dp; > + while ((dp= readdir(dir)) != NULL) > + { > + v.push_back(dp->d_name); > + } > + closedir(dir); > +#endif > + DBUG_RETURN(v); ok. although you could've used my_dir and save yourself some troubles :) > +} > + > + > +/* > + Plugin initialization. > + > + Create KMS client and scan datadir to find out which keys and versions > + are present. > +*/ > +static int plugin_init(void *p) > +{ > + DBUG_ENTER("plugin_init"); > + client = new KMSClient(); > + if (!client) > + { > + sql_print_error("Can not initialize KMS client"); > + DBUG_RETURN(-1); > + } > + InitializeAWSLogging(Aws::MakeShared<MySQLLogSystem>("aws_key_management_plugin", (Aws::Utils::Logging::LogLevel) log_level)); > + pthread_mutex_init(&mtx, NULL); > + vector<string> filenames= read_dir(); > + for (size_t i= 0; i < filenames.size(); i++) > + { > + KEY_INFO info; > + if (extract_id_and_version(filenames[i].c_str(), &info.key_id, &info.key_version) == 0) > + { > + key_info_cache[KEY_ID_AND_VERSION(info.key_id, info.key_version)]= info; > + latest_version_cache[info.key_id]= max(info.key_version, latest_version_cache[info.key_id]); > + } > + } > + DBUG_RETURN(0); > +} > + > + > + > +static int plugin_deinit(void *p) > +{ > + DBUG_ENTER("plugin_deinit"); > + latest_version_cache.clear(); > + key_info_cache.clear(); > + pthread_mutex_destroy(&mtx); > + delete client; > + ShutdownAWSLogging(); > + DBUG_RETURN(0); > +} > + > +/* Generate filename to store the ciphered key */ > +static void format_keyfile_name(char *buf, size_t size, uint key_id, uint version) > +{ > + snprintf(buf, size, "aws-kms-key.%u.%u", key_id, version); > +} > + > +/* Extract key id and version from file name */ > +static int extract_id_and_version(const char *name, uint *id, uint *ver) > +{ > + int len; > + int n= sscanf(name, "aws-kms-key.%u.%u%n", id, ver, &len); > + if (n == 2 && *id > 0 && *ver > 0 && len == (int)strlen(name)) > + return 0; > + return 1; > +} > + > +/* > + Decrypt key stored in aws-kms-key.<id>.<version> > + Cache the decrypted key. > +*/ > +static int load_key(KEY_INFO *info) > +{ > + int ret; > + char path[256]; > + DBUG_ENTER("load_key"); > + DBUG_PRINT("enter", ("id=%u,ver=%u", info->key_id, info->key_version)); > + format_keyfile_name(path, sizeof(path), info->key_id, info->key_version); > + ret= aws_decrypt_key(path, info); > + if (ret) > + info->load_failed= true; > + > + latest_version_cache[info->key_id]= max(latest_version_cache[info->key_id], info->key_version); > + key_info_cache[KEY_ID_AND_VERSION(info->key_id, info->key_version)]= *info; > + > + if (!ret) > + { > + sql_print_information("AWS KMS plugin: loaded key %u, version %u, key length %u bit", > + info->key_id, info->key_version,(uint)info->length*8); > + } > + else > + { > + sql_print_warning("AWS KMS plugin: key %u, version %u could not be decrypted", > + info->key_id, info->key_version); > + } > + DBUG_RETURN(ret); > +} > + > + > +/* > + Get latest version for the key. > + > + If key is not decrypted yet, this function also decrypt the key > + and error will be returned if decryption fails. > + > + The reason for that is that InnoDB crashes > + in case errors are returned by get_key(), Shall we rather fix that in InnoDB? Do you remember where it crashes? > + > + A new key will be created if it does not exist, provided there is > + valid master_key_id. > +*/ > +static unsigned int get_latest_key_version(unsigned int key_id) > +{ > + unsigned int ret; > + DBUG_ENTER("get_latest_key_version"); > + > + pthread_mutex_lock(&mtx); > + ret= get_latest_key_version_nolock(key_id); why do you need get_latest_key_version_nolock if you never use it directly? > + pthread_mutex_unlock(&mtx); > + > + DBUG_PRINT("info", ("key=%u,ret=%u", key_id, ret)); > + DBUG_RETURN(ret); > +} > + > +static unsigned int get_latest_key_version_nolock(unsigned int key_id) > +{ > + KEY_INFO info; > + uint ver; > + DBUG_ENTER("get_latest_key_version_nolock"); > + ver= latest_version_cache[key_id]; > + if (ver > 0) > + { > + info= key_info_cache[KEY_ID_AND_VERSION(key_id, ver)]; > + } > + if (info.load_failed) > + { > + /* Decryption failed previously, don't retry */ > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + } > + else if (ver > 0) > + { > + /* Key encrypted already, return it*/ may be s/encrypted already/exists/ ? > + if (info.length > 0) > + DBUG_RETURN(ver); > + } > + else // (ver == 0) > + { > + /* Generate a new key, version 1 */ > + if (!master_key_id[0]) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Can't generate key %u, because aws_key_management_master_key_id parameter is not set", > + MYF(ME_JUST_WARNING), key_id); > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + } > + if (aws_generate_datakey(key_id, 1) != 0) > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + info.key_id= key_id; > + info.key_version= 1; > + info.length= 0; > + } > + > + if (load_key(&info)) > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + DBUG_RETURN(info.key_version); > +} > + > + > +/* > + Decrypt a file with KMS > +*/ > +static int aws_decrypt_key(const char *path, KEY_INFO *info) > +{ > + DBUG_ENTER("aws_decrypt_key"); > + > + /* Read file content into memory */ > + ifstream ifs(path, ios::binary | ios::ate); > + if (!ifs.good()) > + { > + sql_print_error("can't open file %s", path); > + } > + size_t pos = (size_t)ifs.tellg(); > + std::vector<char> contents(pos); > + ifs.seekg(0, ios::beg); > + ifs.read(&contents[0], pos); > + > + /* Decrypt data the with AWS */ > + DecryptRequest request; > + Aws::Utils::ByteBuffer byteBuffer((unsigned char *)contents.data(), pos); > + request.SetCiphertextBlob(byteBuffer); > + DecryptOutcome outcome = client->Decrypt(request); > + if (!outcome.IsSuccess()) > + { > + sql_print_error("AWS KMS plugin: Decrypt failed for %s : %s", path, > + outcome.GetError().GetMessage().c_str()); > + DBUG_RETURN(-1); > + } > + Aws::Utils::ByteBuffer plaintext = outcome.GetResult().GetPlaintext(); > + size_t len = plaintext.GetLength(); > + > + if (len > (int)sizeof(info->data)) > + { > + sql_print_error("AWS KMS plugin: encoding key too large for %s", path); > + DBUG_RETURN(ENCRYPTION_KEY_BUFFER_TOO_SMALL); > + } > + memcpy(info->data, plaintext.GetUnderlyingData(), len); > + info->length= len; > + DBUG_RETURN(0); > +} > + > + > +/* Generate a new datakey and store it a file */ > +static int aws_generate_datakey(uint keyid, uint version) > +{ > + > + DBUG_ENTER("aws_generate_datakey"); > + GenerateDataKeyWithoutPlaintextRequest request; > + request.SetKeyId(master_key_id); > + request.SetKeySpec(DataKeySpecMapper::GetDataKeySpecForName(key_spec_names[key_spec])); > + > + GenerateDataKeyWithoutPlaintextOutcome outcome; > + outcome= client->GenerateDataKeyWithoutPlaintext(request); > + if (!outcome.IsSuccess()) > + { > + sql_print_error("AWS KMS plugin : GenerateDataKeyWithoutPlaintext failed : %s - %s", > + outcome.GetError().GetExceptionName().c_str(), > + outcome.GetError().GetMessage().c_str()); > + DBUG_RETURN(-1); > + } > + > + string out; > + char filename[20]; > + Aws::Utils::ByteBuffer byteBuffer = outcome.GetResult().GetCiphertextBlob(); > + > + format_keyfile_name(filename, sizeof(filename), keyid, version); > + int fd= my_open(filename, O_RDWR | O_CREAT, 0); > + if (fd < 0) > + { > + sql_print_error("AWS KMS plugin: Can't create file %s", filename); > + DBUG_RETURN(-1); > + } > + size_t len= byteBuffer.GetLength(); > + if (my_write(fd, byteBuffer.GetUnderlyingData(), len, 0) != len) > + { > + sql_print_error("AWS KMS plugin: can't write to %s", filename); > + my_close(fd, 0); > + my_delete(filename, 0); > + DBUG_RETURN(-1); > + } > + my_close(fd, 0); > + sql_print_information("AWS KMS plugin: generated encrypted datakey for key id=%u, version=%u", > + keyid, version); > + DBUG_RETURN(0); > +} > + > +/* Key rotation for a single key */ > +static int rotate_single_key(uint key_id) > +{ > + uint ver; > + ver= latest_version_cache[key_id]; > + > + if (!ver) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "key %u does not exist", MYF(ME_JUST_WARNING), key_id); > + return -1; > + } > + else if (aws_generate_datakey(key_id, ver + 1)) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "Could not generate datakey for key id= %u, ver= %u", > + MYF(ME_JUST_WARNING), key_id, ver); > + return -1; > + } > + else > + { > + KEY_INFO info; > + info.key_id= key_id; > + info.key_version = ver + 1; > + if (load_key(&info)) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "Could not load datakey for key id= %u, ver= %u", > + MYF(ME_JUST_WARNING), key_id, ver); > + return -1; > + } > + } > + return 0; > +} > + > +/* Key rotation for all key ids */ > +static int rotate_all_keys() > +{ > + int ret= 0; > + for (map<uint, uint>::iterator it= latest_version_cache.begin(); it != latest_version_cache.end(); it++) > + { > + ret= rotate_single_key(it->first); > + if (ret) > + break; > + } > + return ret; > +} > + > +static void update_rotate(MYSQL_THD, struct st_mysql_sys_var *, void *, const void *val) > +{ > + if (!master_key_id[0]) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "aws_key_management_master_key_id must be set to generate new data keys", MYF(ME_JUST_WARNING)); > + return; > + } > + pthread_mutex_lock(&mtx); > + rotate_key= *(int *)val; > + switch (rotate_key) > + { > + case 0: > + break; > + case -1: > + rotate_all_keys(); > + break; > + default: > + rotate_single_key(rotate_key); > + break; > + } > + rotate_key= 0; > + pthread_mutex_unlock(&mtx); > +} > + > +static unsigned int get_key( > + unsigned int key_id, > + unsigned int version, > + unsigned char* dstbuf, > + unsigned int* buflen) > +{ > + KEY_INFO info; > + > + DBUG_ENTER("get_key"); > + pthread_mutex_lock(&mtx); > + info= key_info_cache[KEY_ID_AND_VERSION(key_id, version)]; > + if (info.length == 0 && !info.load_failed) > + { > + info.key_id= key_id; > + info.key_version= version; > + load_key(&info); > + } > + pthread_mutex_unlock(&mtx); > + > + if (info.load_failed) > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + if (*buflen < info.length) > + { > + *buflen= info.length; > + DBUG_RETURN(ENCRYPTION_KEY_BUFFER_TOO_SMALL); > + } > + *buflen= info.length; > + memcpy(dstbuf, info.data, info.length); > + DBUG_RETURN(0); > +} > + > + > +/* Plugin defs */ > +struct st_mariadb_encryption aws_key_management_plugin= { > + MariaDB_ENCRYPTION_INTERFACE_VERSION, > + get_latest_key_version, > + get_key, > + // use default encrypt/decrypt functions > + 0, 0, 0, 0, 0 > +}; > + > + > +static TYPELIB key_spec_typelib = > +{ > + array_elements(key_spec_names) - 1, "", > + key_spec_names, NULL > +}; > + > +const char *log_level_names[] = > +{ > + "Off", > + "Fatal", > + "Error", > + "Warn", > + "Info", > + "Debug", > + "Trace", > + 0 > +}; > + > +static TYPELIB log_level_typelib = > +{ > + array_elements(log_level_names) - 1, "", > + log_level_names, NULL > +}; > + > +static MYSQL_SYSVAR_STR(master_key_id, master_key_id, > + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_MEMALLOC, > + "Key id for master encryption key. Used to create new datakeys. If not set, no new keys will be created", > + NULL, NULL, ""); > + > +static MYSQL_SYSVAR_ENUM(key_spec, key_spec, > + PLUGIN_VAR_RQCMDARG, > + "Encryption algorithm used to create new keys.", > + NULL, NULL, 0, &key_spec_typelib); > + > + > +static MYSQL_SYSVAR_ENUM(log_level, log_level, > + PLUGIN_VAR_RQCMDARG, > + "Logging for AWS API", > + NULL, NULL, 0, &log_level_typelib); > + > + > +static MYSQL_SYSVAR_INT(rotate_key, rotate_key, > + PLUGIN_VAR_RQCMDARG, > + "Set this variable to key id to perform rotation of the key. Specify -1 to rotate all keys", > + NULL, update_rotate, 0, -1, INT_MAX, 1); > + > +static struct st_mysql_sys_var* settings[]= { > + MYSQL_SYSVAR(master_key_id), > + MYSQL_SYSVAR(key_spec), > + MYSQL_SYSVAR(rotate_key), > + MYSQL_SYSVAR(log_level), > + NULL > +}; > + > +/* > + Plugin library descriptor > +*/ > +maria_declare_plugin(aws_key_management) > +{ > + MariaDB_ENCRYPTION_PLUGIN, > + &aws_key_management_plugin, > + "aws_key_management", > + "MariaDB Corporation", > + "AWS key management plugin", > + PLUGIN_LICENSE_GPL, > + plugin_init, > + plugin_deinit, > + 0x0100, > + NULL, > + settings, > + "1.0", > + MariaDB_PLUGIN_MATURITY_EXPERIMENTAL > +} > +maria_declare_plugin_end; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org [View Less]

2 2

[Maria-developers] GSoC 2016 MHA GTID support.
by Charles Dirk 11 Mar '16

11 Mar '16

Hi, I am a computer science student at the university of Cape Town and wish to participate in GSoC 2016 with MariaDB Foundation. I am interested if working on making MHA understand GTIDs. I am proficient in C/C+, Python/Perl and Im equally familiar with replication. I look forward to hearing from mentors soon. Cheers, Charles.

3 3

[Maria-developers] MDEV-9709 Unexpected modification of value and warning about out of range value upon ALTER
by Alexander Barkov 11 Mar '16

11 Mar '16

Hi Elena, can you please review a patch for MDEV-9709? I'm quite sure about the code change, it's trivial. Please have a look into the tests. Thanks.

1 0

[Maria-developers] GSoC 2016 Maxscale Filter
by Lisa Brinson 08 Mar '16

08 Mar '16

Hi everyone, My name is Lisa Brinson and I am a junior Computer science student at Oregon State University. I wish to participate in an open source project through GSoC 2016. I am proficient in C and C++ as well as git. I have looked through the proposed projects and I have developed interest in the Maxscale tasks especially the filters. I have been going through the filter API and have also looked at some sample filters in server/modules/filter directory. I will try to write my first filter(… [View More]

3 4

[Maria-developers] Questions regarding GSoC 2016 and Import / Export
by Dan Ungureanu 08 Mar '16

08 Mar '16

Hello, I am a second year student currently attending University Politehnica of Bucharest and I am interested in contributing to MariaDB during this year's Google Summer of Code. I have looked at the ideas you published in the Knowledge Base [1] and two projects caught my attention. The first one is related to the import and export process related to dynamic columns [2] and the second one is related to the GIS enhancements [3]. I am willing to implement both of them, but I am not sure … [View More]whether the time will allow me properly implement both of them. I cannot estimate how much effort do these projects require as I am still trying to become acquainted with the code base. I hope I am not underestimating the projects and some guidance from one of the mentors (Oleksandr Byelkin, Georg Richter or Alexey Botchkov - Holyfoot) is certainly welcome. I will focus on "Import and export popular data formats from and to dynamic columns" and as soon as I am done with this one, hopefully I will be able to work on GIS enhancements as well. Before continuing, I would like to clarify some aspects related to these projects. Import and export popular data formats from and to dynamic columns: - Can be split in four main tasks: a) adding support for arrays (vectors) in dynamic columns. Currently, dynamic columns only store dictionaries (maps); b) implement a library that can import (parse) and export (build in a specific format) data to and from dynamic columns; c) provide interface to access to the new import and export mechanism; d) provide unit tests for the new features. - The second part (b) of the project is mostly standalone and requires little edits to the project and the third part (c) will be mostly shared by the server and the C connector. - The outcome of (b) is going to be a minimal parsing and building library for various formats. - Some of the formats that I plan to support are: JSON, XML, CSV (maybe XLS too?), PHP Array, YAML, SQL (export as INSERT INTO statement?), etc. One of the key aspects I have to keep in mind when designing the library is to provide a extensible interface to quickly implement new formats in the future. - Probably, most of my work will be located in: - ./server/sql/item_*.(cc|h) - implement user defined functions - ./server/mysys/ma_dyncol.c - provide interface for user defined functions - ./server/include/ma_dyncol.h - ./mariadb-connector-c/libmariadb/mariadb_dyncol.c - provide interface in C Connector - ./mariadb-connector-c/include/mariadb_dyncol.h GIS enhancements: - This project is based on extending the current functionality related to GIS. I am not very familiar with GIS and I still have to do more research based on documents [4], [5] and [6], but the biggest part of the project consists in defining user functions for handling GIS. - I found this compliance test suite [7] for PostGIS that may come in handy. Also, I have a few other questions related to how should I proceed: - I have already read the introductory developers guide [8], the contributing guidelines [9] and will submit the MariaDB Contributor Agreement [10] as soon as possible. Is there anything else I should do before getting started? - I found no template for Google Summer of Code applications. Is there anything in particular that mentors would like to see? I was thinking of organizing my application in multiple sections, some of the most important being: Synopsis, Benefits to the Users, Project Details, Deliverables, Project Schedule and my resume. I believe that I will receive more feedback related after I will have submitted my application. - Most issues already have an assignee. Is it fine if I try and send pull requests for some of those bugs? I could not decide between whether I should send this mail to discuss or developers mailing list. Hopefully, I chose the right one. I am sorry for my long email and thank you for attention. I am looking for feedback, so please do not hesitate to send me any suggestion you might have. [1] https://mariadb.com/kb/en/mariadb/google-summer-of-code-2016/ [2] https://jira.mariadb.org/browse/CONC-125 [3] https://jira.mariadb.org/browse/MDEV-5813 [4] http://www.opengeospatial.org/standards/sfs [5] http://portal.opengeospatial.org/files/?artifact_id=25354 [6] http://www.opengeospatial.org/compliance/ [7] https://svn.osgeo.org/postgis/trunk/extras/ogc_test_suite/README [8] https://mariadb.org/get-involved/getting-started-for-developers/ [9] https://mariadb.com/kb/en/mariadb/contributing-code/ [10] https://mariadb.com/kb/en/mariadb/mca/ Best regards, Dan Ungureanu [View Less]

2 1