- developers - lists.mariadb.org

Re: [Maria-developers] MDEV-9659 Encryption plugin that uses AWS Key management service
by Sergei Golubchik 12 Mar '16

12 Mar '16

Hi, Wlad! On Mar 03, wlad(a)mariadb.com wrote: > revision-id: c1ed3c0267da2ece7664e9f876314979114b2186 (mariadb-10.1.11-26-gc1ed3c0) > parent(s): a62db8186e40a62366bc86f49a0195e9b687d86d > author: Vladislav Vaintroub > committer: Vladislav Vaintroub > timestamp: 2016-03-03 21:29:29 +0100 > message: > > MDEV-9659 Encryption plugin that uses AWS Key management service Looks pretty good, thanks! See comment below... > diff --git a/cmake/plugin.cmake b/cmake/plugin.cmake > index c24239c..361cb59 100644 > --- a/cmake/plugin.cmake > +++ b/cmake/plugin.cmake > @@ -74,7 +74,9 @@ MACRO(MYSQL_ADD_PLUGIN) > SET(compat "with${compat}") > ENDIF() > > - IF (compat STREQUAL ".") > + IF (ARG_DISABLED) > + SET(howtobuild NO) > + ELSEIF (compat STREQUAL ".") > SET(howtobuild DYNAMIC) > ELSEIF (compat STREQUAL "with.") > IF (NOT ARG_MODULE_ONLY) > @@ -122,7 +124,7 @@ MACRO(MYSQL_ADD_PLUGIN) > > # Build either static library or module > IF (PLUGIN_${plugin} MATCHES "(STATIC|AUTO|YES)" AND NOT ARG_MODULE_ONLY > - AND NOT ARG_DISABLED AND NOT ARG_CLIENT) > + AND NOT ARG_CLIENT) > > IF(CMAKE_GENERATOR MATCHES "Makefiles|Ninja") > # If there is a shared library from previous shared build, > @@ -178,8 +180,7 @@ MACRO(MYSQL_ADD_PLUGIN) > SET (mysql_optional_plugins ${mysql_optional_plugins} PARENT_SCOPE) > ENDIF() > ELSEIF(PLUGIN_${plugin} MATCHES "(DYNAMIC|AUTO|YES)" > - AND NOT ARG_STATIC_ONLY AND NOT WITHOUT_DYNAMIC_PLUGINS > - AND NOT ARG_DISABLED) > + AND NOT ARG_STATIC_ONLY AND NOT WITHOUT_DYNAMIC_PLUGINS) > > ADD_VERSION_INFO(${target} MODULE SOURCES) > ADD_LIBRARY(${target} MODULE ${SOURCES}) a bit more of the explanation would've been nice. in the changeset comment or - even better - pushing this change separately. you can select what files to include in the commit, you know... > diff --git a/plugin/aws_key_management/CMakeLists.txt b/plugin/aws_key_management/CMakeLists.txt > new file mode 100644 > index 0000000..ec9ee46 > --- /dev/null > +++ b/plugin/aws_key_management/CMakeLists.txt > @@ -0,0 +1,144 @@ > +# We build parts of AWS C++ SDK as CMake external project > +# The restrictions of the SDK (https://github.com/awslabs/aws-sdk-cpp/blob/master/README.md) > +# are > + > +# - OS : Windows,Linux or OSX > +# - C++11 compiler : VS2013+, gcc 4.7+, clang 3.3+ > +# - libcurl development package needs to be present on Unixes > +# > +# Since we're using ExternalProject that reads from git repository > +# we also require GIT to be present on the build machine > + > + > +# Give message why the building this plugin is skipped (only if -DVERBOSE is defined) > +# and and bail out. > +MACRO(SKIP_AWS_PLUGIN msg) > + IF(DEFINED VERBOSE) VERBOSE - is it used somewhere else? Or you've invented it on the spot for this plugin? I understand that it can be used in other cmake files, but is it? > + MESSAGE(STATUS "Skip aws_key_management - ${msg}") > + RETURN() > + ENDIF() > +ENDMACRO() Ok, as discussed on irc, I'd suggest to add here a check whether AWS SDK is already installed. Like, CheckLibraryExists or CheckSymbolExists. And if not - then try to download and compile it (which would require new cmake, git, curl, etc). > +IF(CMAKE_VERSION VERSION_LESS "2.8.8") > + SKIP_AWS_PLUGIN("CMake is too old") > +ENDIF() > + > +FIND_PACKAGE(Git) > +IF(NOT GIT_FOUND) > + SKIP_AWS_PLUGIN("no GIT") > +ENDIF() > + > +INCLUDE(ExternalProject) > +IF (NOT(WIN32 OR APPLE OR (CMAKE_SYSTEM_NAME MATCHES "Linux"))) > + SKIP_AWS_PLUGIN("OS unsupported by AWS SDK") > +ENDIF() > + > +IF(UNIX) > + FIND_PACKAGE(CURL) > + IF(NOT CURL_FOUND) > + SKIP_AWS_PLUGIN("AWS C++ SDK requires libcurl development package") > + ENDIF() > + SET(PIC_FLAG -fPIC) > +ENDIF() > + > +SET(CXX11_FLAGS) > +SET(OLD_COMPILER_MSG "AWS SDK requires c++11 -capable compiler (minimal supported versions are g++ 4.7, clang 3.3, VS2103)") > +IF(CMAKE_CXX_COMPILER_ID MATCHES "GNU") > + EXECUTE_PROCESS(COMMAND ${CMAKE_CXX_COMPILER} -dumpversion OUTPUT_VARIABLE GCC_VERSION) eh? I thought cmake knows compiler versions... let me see This one: CMAKE_<LANG>_COMPILER_VERSION > + IF (GCC_VERSION VERSION_LESS 4.7) > + SKIP_AWS_PLUGIN("${OLD_COMPILER_MSG}") > + ENDIF() > + SET(CXX11_FLAGS "-std=c++11") Instead of checking the compiler version, I would rather suggest to check for a feature that you need to use. Like MY_CHECK_CXX_COMPILER_FLAG(-std=c++11) IF (NOT HAVE_CXX__STD_C__11) SKIP_AWS_PLUGIN("${OLD_COMPILER_MSG}") ENDIF() Something like that. Not sure about HAVE_CXX__STD_C__11 name, though. > +ELSEIF (CMAKE_CXX_COMPILER_ID MATCHES "Clang") > + IF (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 3.3) > + SKIP_AWS_PLUGIN("${OLD_COMPILER_MSG}") > + ENDIF() > + SET(CXX11_FLAGS "-stdlib=libc++") Same. > +ELSEIF(MSVC) > + IF (MSVC_VERSION LESS 1800) > + SKIP_AWS_PLUGIN("${OLD_COMPILER_MSG}") > + ENDIF() > +ELSE() > + SKIP_AWS_PLUGIN("Compiler not supported by AWS C++ SDK") > +ENDIF() > + > +IF(WIN32) > + SET(EXTRA_SDK_CMAKE_FLAGS -DCMAKE_CXX_FLAGS_DEBUGOPT="" -DCMAKE_EXE_LINKER_FLAGS_DEBUGOPT="") For consistency you can put this inside ELSEIF(MSVC), that's where you set options for gcc and clang. > +ENDIF() > + > +FIND_PROGRAM(PERL_EXECUTABLE perl) > + > +IF(PERL_EXECUTABLE) > + # CMake minimimal version for building AWS SDK can be less than 3.1.2 > + # Downgrade this requirement. > + SET(AWS_SDK_PATCH_COMMAND > + PATCH_COMMAND ${PERL_EXECUTABLE} -i.bak -pe s/3.1.2/2.8/g ${CMAKE_BINARY_DIR}/aws-sdk-cpp/CMakeLists.txt) 1. Without perl it'll require 3.1.2, right? 2. Couldn't you do that with cmake alone? 3. Is that safe at all - you pull the latest version from git, they can start using cmake-3.1.2 features any time. > +ENDIF() > + > +ExternalProject_Add( > + aws_sdk_cpp > + GIT_REPOSITORY "https://github.com/awslabs/aws-sdk-cpp.git" > + UPDATE_COMMAND "" > + ${AWS_SDK_PATCH_COMMAND} > + SOURCE_DIR "${CMAKE_BINARY_DIR}/aws-sdk-cpp" > + CMAKE_ARGS > + -DBUILD_ONLY=aws-cpp-sdk-kms -DSTATIC_LINKING=1 > + "-DCMAKE_CXX_FLAGS_DEBUG=${CMAKE_CXX_FLAGS_DEBUG} ${PIC_FLAG}" > + "-DCMAKE_CXX_FLAGS_RELWITHDEBINFO=${CMAKE_CXX_FLAGS_RELWITHDEBINFO} ${PIC_FLAG}" > + "-DCMAKE_CXX_FLAGS_RELEASE=${CMAKE_CXX_FLAGS_RELEASE} ${PIC_FLAG}" > + "-DCMAKE_CXX_FLAGS_MINSIZEREL=${CMAKE_CXX_FLAGS_MINSIZEREL} ${PIC_FLAG}" > + ${EXTRA_SDK_CMAKE_FLAGS} > + -DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_DIR}/aws_sdk_cpp > + TEST_COMMAND "" > +) > + > +# Figure out where AWS installs SDK libraries > +# The below is defined in AWS SDK's CMakeLists.txt > +# (and their handling is weird, every OS has special install directory) > +if(WIN32) > + SET(SDK_INSTALL_BINARY_PREFIX "windows") > +ELSEIF(APPLE) > + SET(SDK_INSTALL_BINARY_PREFIX "mac") > +ELSEIF(UNIX) > + SET(SDK_INSTALL_BINARY_PREFIX "linux") > +ENDIF() > +IF(NOT APPLE) > + IF(CMAKE_SIZEOF_VOID_P EQUAL 8) > + SET(SDK_INSTALL_BINARY_PREFIX "${SDK_INSTALL_BINARY_PREFIX}/intel64") > + ELSE() > + SET(SDK_INSTALL_BINARY_PREFIX "${SDK_INSTALL_BINARY_PREFIX}/ia32") > + ENDIF() > +ENDIF() > +IF(CMAKE_CONFIGURATION_TYPES) > + SET(SDK_INSTALL_BINARY_PREFIX "${SDK_INSTALL_BINARY_PREFIX}/${CMAKE_CFG_INTDIR}") > +ENDIF() > + > +# We do not need to build the whole SDK , just 2 of its libs > +set(AWS_SDK_LIBS aws-cpp-sdk-core aws-cpp-sdk-kms) > + > +FOREACH(lib ${AWS_SDK_LIBS}) > + ADD_LIBRARY(${lib} STATIC IMPORTED) > + ADD_DEPENDENCIES(${lib} aws_sdk_cpp) > + SET(loc "${CMAKE_BINARY_DIR}/aws_sdk_cpp/lib/${SDK_INSTALL_BINARY_PREFIX}/${CMAKE_STATIC_LIBRARY_PREFIX}${lib}${CMAKE_STATIC_LIBRARY_SUFFIX}") > + SET_TARGET_PROPERTIES(${lib} PROPERTIES IMPORTED_LOCATION ${loc}) > + IF(WIN32) > + SET_TARGET_PROPERTIES(${lib} PROPERTIES IMPORTED_LINK_INTERFACE_LIBRARIES "bcrypt;winhttp;wininet;userenv") > + ELSE() > + SET_TARGET_PROPERTIES(${lib} PROPERTIES IMPORTED_LINK_INTERFACE_LIBRARIES "${SSL_LIBRARIES};${CURL_LIBRARIES}") > + ENDIF() > +ENDFOREACH() > + > +IF(CMAKE_SYSTEM_NAME MATCHES "Linux") > + # Need whole-archive , otherwise static libraries are not linked > + SET(AWS_SDK_LIBS -Wl,--whole-archive ${AWS_SDK_LIBS} -Wl,--no-whole-archive) > +ENDIF() > + > +SET_TARGET_PROPERTIES(aws_sdk_cpp PROPERTIES EXCLUDE_FROM_ALL TRUE) > +INCLUDE_DIRECTORIES(${CMAKE_BINARY_DIR}/aws_sdk_cpp/include) > +SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${CXX11_FLAGS}") > +MYSQL_ADD_PLUGIN(aws_key_management aws_key_management_plugin.cc > + MODULE_ONLY COMPONENT aws-key-management > + LINK_LIBRARIES ${AWS_SDK_LIBS} DISABLED) > diff --git a/plugin/aws_key_management/aws_key_management_plugin.cc b/plugin/aws_key_management/aws_key_management_plugin.cc > new file mode 100644 > index 0000000..fa7d042 > --- /dev/null > +++ b/plugin/aws_key_management/aws_key_management_plugin.cc > @@ -0,0 +1,611 @@ > +/* > + Copyright (c) 2016 MariaDB Corporation > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > + > + > +#include <my_global.h> > +#include <my_pthread.h> > +#include <my_sys.h> > +#include <mysql/plugin_encryption.h> > +#include <my_crypt.h> > +#include <string.h> > +#include <stdio.h> > +#include <stdlib.h> > +#include <mysqld_error.h> > +#include <base64.h> > +#include <map> > +#include <algorithm> > +#include <string> > +#include <vector> > +#include <iterator> > +#include <sstream> > +#include <fstream> > +#ifndef _WIN32 > +#include <sys/types.h> > +#include <dirent.h> > +#endif Use cmake checks when there are some, HAVE_DIRENT_H, HAVE_SYS_TYPES_H, HAVE_STRING_H, HAVE_STDLIB_H. Hmm, HAVE_STDLIB_H doesn't make sense to me :) could you remove it from configure.cmake, please? In a separate changeset. May be HAVE_SYS_TYPES_H too, not sure. But do use HAVE_DIRENT_H and HAVE_STDLIB_H above instead of _WIN32. > +#include <aws/core/client/AWSError.h> > +#include <aws/core/utils/logging/AWSLogging.h> > +#include <aws/core/utils/logging/ConsoleLogSystem.h> > +#include <aws/kms/KMSClient.h> > +#include <aws/kms/model/DecryptRequest.h> > +#include <aws/kms/model/DecryptResult.h> > +#include <aws/kms/model/GenerateDataKeyWithoutPlaintextRequest.h> > +#include <aws/kms/model/GenerateDataKeyWithoutPlaintextResult.h> > +#include <aws/core/utils/Outcome.h> > +using namespace std; > +using namespace Aws::KMS; > +using namespace Aws::KMS::Model; > +using namespace Aws::Utils::Logging; > +extern void sql_print_error(const char *format, ...); > +extern void sql_print_warning(const char *format, ...); > +extern void sql_print_information(const char *format, ...); > + > + > +/* Plaintext key info struct */ > +struct KEY_INFO > +{ > + unsigned int key_id; > + unsigned int key_version; > + unsigned int length; > + unsigned char data[MY_AES_MAX_KEY_LENGTH]; > + bool load_failed; /* if true, do not attempt to reload?*/ > +public: > + KEY_INFO() : key_id(0), key_version(0), length(0), load_failed(false){}; > +}; > +#define KEY_ID_AND_VERSION(key_id,version) ((longlong)key_id << 32 | version) > + > +/* Cache for the latest version, per key id */ > +static std::map<uint, uint> latest_version_cache; > + > +/* Cache for plaintext keys */ > +static std::map<ulonglong, KEY_INFO> key_info_cache; > + > +static const char *key_spec_names[]={ "AES_128", "AES_256", 0 }; > + > +/* Plugin variables */ > +static char* master_key_id; > +static unsigned long key_spec; > +static unsigned long log_level; > +static int rotate_key; > + > +/* AWS functionality*/ > +static int aws_decrypt_key(const char *path, KEY_INFO *info); > +static int aws_generate_datakey(uint key_id, uint version); > + > +static int extract_id_and_version(const char *name, uint *id, uint *ver); > +static unsigned int get_latest_key_version(unsigned int key_id); > +static unsigned int get_latest_key_version_nolock(unsigned int key_id); > +static int load_key(KEY_INFO *info); > + > +/* Mutex to serialize access to caches */ > +static pthread_mutex_t mtx; Use PSI, please. That's mysql_mutex_t, etc. > + > +static Aws::KMS::KMSClient *client; > + > +/* Redirect AWS trace to error log */ > +class MySQLLogSystem : public Aws::Utils::Logging::FormattedLogSystem > +{ > +public: > + > + using Base = FormattedLogSystem; > + MySQLLogSystem(LogLevel logLevel) : > + Base(logLevel) > + { > + } > + virtual LogLevel GetLogLevel(void) const override > + { > + return (LogLevel)log_level; > + } > + virtual ~MySQLLogSystem() > + { > + } > + > +protected: > + virtual void ProcessFormattedStatement(Aws::String&& statement) override > + { > +#ifdef _WIN32 > + /* > + On Windows, we can't use C runtime functions to write to stdout, > + because we compile with static C runtime, so plugin has a stdout > + different from server. Thus we're using WriteFile(). > + */ > + DWORD nSize= (DWORD)statement.size(); > + DWORD nWritten; > + const char *s= statement.c_str(); > + HANDLE h= GetStdHandle(STD_OUTPUT_HANDLE); > + > + WriteFile(h, s, nSize, &nWritten, NULL); > +#else > + printf("%s", statement.c_str()); printf? You have declared sql_print_information above, why wouldn't you use it here? > +#endif > + } > +}; > + > +/* > + Read filenames from the current directory. > + This is used in plugin initilization. We check for > + plugin specific names aws-kms-key.<id>.<version> > + to find out what keys/key versions are already there. > +*/ > +static vector<string> read_dir() > +{ > + vector<string> v; > + DBUG_ENTER("read_dir"); > +#ifdef _WIN32 > + WIN32_FIND_DATA ffd; > + HANDLE h= FindFirstFile("*", &ffd); > + if (h == INVALID_HANDLE_VALUE) > + { > + DBUG_RETURN(v); > + } > + do > + { > + /* handle file possibly updates caches*/ > + v.push_back(ffd.cFileName); > + } while (FindNextFile(h, &ffd)); > + FindClose(h); > +#else > + DIR *dir= opendir("."); > + if (!dir) > + { > + DBUG_RETURN(v); > + } > + struct dirent *dp; > + while ((dp= readdir(dir)) != NULL) > + { > + v.push_back(dp->d_name); > + } > + closedir(dir); > +#endif > + DBUG_RETURN(v); ok. although you could've used my_dir and save yourself some troubles :) > +} > + > + > +/* > + Plugin initialization. > + > + Create KMS client and scan datadir to find out which keys and versions > + are present. > +*/ > +static int plugin_init(void *p) > +{ > + DBUG_ENTER("plugin_init"); > + client = new KMSClient(); > + if (!client) > + { > + sql_print_error("Can not initialize KMS client"); > + DBUG_RETURN(-1); > + } > + InitializeAWSLogging(Aws::MakeShared<MySQLLogSystem>("aws_key_management_plugin", (Aws::Utils::Logging::LogLevel) log_level)); > + pthread_mutex_init(&mtx, NULL); > + vector<string> filenames= read_dir(); > + for (size_t i= 0; i < filenames.size(); i++) > + { > + KEY_INFO info; > + if (extract_id_and_version(filenames[i].c_str(), &info.key_id, &info.key_version) == 0) > + { > + key_info_cache[KEY_ID_AND_VERSION(info.key_id, info.key_version)]= info; > + latest_version_cache[info.key_id]= max(info.key_version, latest_version_cache[info.key_id]); > + } > + } > + DBUG_RETURN(0); > +} > + > + > + > +static int plugin_deinit(void *p) > +{ > + DBUG_ENTER("plugin_deinit"); > + latest_version_cache.clear(); > + key_info_cache.clear(); > + pthread_mutex_destroy(&mtx); > + delete client; > + ShutdownAWSLogging(); > + DBUG_RETURN(0); > +} > + > +/* Generate filename to store the ciphered key */ > +static void format_keyfile_name(char *buf, size_t size, uint key_id, uint version) > +{ > + snprintf(buf, size, "aws-kms-key.%u.%u", key_id, version); > +} > + > +/* Extract key id and version from file name */ > +static int extract_id_and_version(const char *name, uint *id, uint *ver) > +{ > + int len; > + int n= sscanf(name, "aws-kms-key.%u.%u%n", id, ver, &len); > + if (n == 2 && *id > 0 && *ver > 0 && len == (int)strlen(name)) > + return 0; > + return 1; > +} > + > +/* > + Decrypt key stored in aws-kms-key.<id>.<version> > + Cache the decrypted key. > +*/ > +static int load_key(KEY_INFO *info) > +{ > + int ret; > + char path[256]; > + DBUG_ENTER("load_key"); > + DBUG_PRINT("enter", ("id=%u,ver=%u", info->key_id, info->key_version)); > + format_keyfile_name(path, sizeof(path), info->key_id, info->key_version); > + ret= aws_decrypt_key(path, info); > + if (ret) > + info->load_failed= true; > + > + latest_version_cache[info->key_id]= max(latest_version_cache[info->key_id], info->key_version); > + key_info_cache[KEY_ID_AND_VERSION(info->key_id, info->key_version)]= *info; > + > + if (!ret) > + { > + sql_print_information("AWS KMS plugin: loaded key %u, version %u, key length %u bit", > + info->key_id, info->key_version,(uint)info->length*8); > + } > + else > + { > + sql_print_warning("AWS KMS plugin: key %u, version %u could not be decrypted", > + info->key_id, info->key_version); > + } > + DBUG_RETURN(ret); > +} > + > + > +/* > + Get latest version for the key. > + > + If key is not decrypted yet, this function also decrypt the key > + and error will be returned if decryption fails. > + > + The reason for that is that InnoDB crashes > + in case errors are returned by get_key(), Shall we rather fix that in InnoDB? Do you remember where it crashes? > + > + A new key will be created if it does not exist, provided there is > + valid master_key_id. > +*/ > +static unsigned int get_latest_key_version(unsigned int key_id) > +{ > + unsigned int ret; > + DBUG_ENTER("get_latest_key_version"); > + > + pthread_mutex_lock(&mtx); > + ret= get_latest_key_version_nolock(key_id); why do you need get_latest_key_version_nolock if you never use it directly? > + pthread_mutex_unlock(&mtx); > + > + DBUG_PRINT("info", ("key=%u,ret=%u", key_id, ret)); > + DBUG_RETURN(ret); > +} > + > +static unsigned int get_latest_key_version_nolock(unsigned int key_id) > +{ > + KEY_INFO info; > + uint ver; > + DBUG_ENTER("get_latest_key_version_nolock"); > + ver= latest_version_cache[key_id]; > + if (ver > 0) > + { > + info= key_info_cache[KEY_ID_AND_VERSION(key_id, ver)]; > + } > + if (info.load_failed) > + { > + /* Decryption failed previously, don't retry */ > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + } > + else if (ver > 0) > + { > + /* Key encrypted already, return it*/ may be s/encrypted already/exists/ ? > + if (info.length > 0) > + DBUG_RETURN(ver); > + } > + else // (ver == 0) > + { > + /* Generate a new key, version 1 */ > + if (!master_key_id[0]) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Can't generate key %u, because aws_key_management_master_key_id parameter is not set", > + MYF(ME_JUST_WARNING), key_id); > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + } > + if (aws_generate_datakey(key_id, 1) != 0) > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + info.key_id= key_id; > + info.key_version= 1; > + info.length= 0; > + } > + > + if (load_key(&info)) > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + DBUG_RETURN(info.key_version); > +} > + > + > +/* > + Decrypt a file with KMS > +*/ > +static int aws_decrypt_key(const char *path, KEY_INFO *info) > +{ > + DBUG_ENTER("aws_decrypt_key"); > + > + /* Read file content into memory */ > + ifstream ifs(path, ios::binary | ios::ate); > + if (!ifs.good()) > + { > + sql_print_error("can't open file %s", path); > + } > + size_t pos = (size_t)ifs.tellg(); > + std::vector<char> contents(pos); > + ifs.seekg(0, ios::beg); > + ifs.read(&contents[0], pos); > + > + /* Decrypt data the with AWS */ > + DecryptRequest request; > + Aws::Utils::ByteBuffer byteBuffer((unsigned char *)contents.data(), pos); > + request.SetCiphertextBlob(byteBuffer); > + DecryptOutcome outcome = client->Decrypt(request); > + if (!outcome.IsSuccess()) > + { > + sql_print_error("AWS KMS plugin: Decrypt failed for %s : %s", path, > + outcome.GetError().GetMessage().c_str()); > + DBUG_RETURN(-1); > + } > + Aws::Utils::ByteBuffer plaintext = outcome.GetResult().GetPlaintext(); > + size_t len = plaintext.GetLength(); > + > + if (len > (int)sizeof(info->data)) > + { > + sql_print_error("AWS KMS plugin: encoding key too large for %s", path); > + DBUG_RETURN(ENCRYPTION_KEY_BUFFER_TOO_SMALL); > + } > + memcpy(info->data, plaintext.GetUnderlyingData(), len); > + info->length= len; > + DBUG_RETURN(0); > +} > + > + > +/* Generate a new datakey and store it a file */ > +static int aws_generate_datakey(uint keyid, uint version) > +{ > + > + DBUG_ENTER("aws_generate_datakey"); > + GenerateDataKeyWithoutPlaintextRequest request; > + request.SetKeyId(master_key_id); > + request.SetKeySpec(DataKeySpecMapper::GetDataKeySpecForName(key_spec_names[key_spec])); > + > + GenerateDataKeyWithoutPlaintextOutcome outcome; > + outcome= client->GenerateDataKeyWithoutPlaintext(request); > + if (!outcome.IsSuccess()) > + { > + sql_print_error("AWS KMS plugin : GenerateDataKeyWithoutPlaintext failed : %s - %s", > + outcome.GetError().GetExceptionName().c_str(), > + outcome.GetError().GetMessage().c_str()); > + DBUG_RETURN(-1); > + } > + > + string out; > + char filename[20]; > + Aws::Utils::ByteBuffer byteBuffer = outcome.GetResult().GetCiphertextBlob(); > + > + format_keyfile_name(filename, sizeof(filename), keyid, version); > + int fd= my_open(filename, O_RDWR | O_CREAT, 0); > + if (fd < 0) > + { > + sql_print_error("AWS KMS plugin: Can't create file %s", filename); > + DBUG_RETURN(-1); > + } > + size_t len= byteBuffer.GetLength(); > + if (my_write(fd, byteBuffer.GetUnderlyingData(), len, 0) != len) > + { > + sql_print_error("AWS KMS plugin: can't write to %s", filename); > + my_close(fd, 0); > + my_delete(filename, 0); > + DBUG_RETURN(-1); > + } > + my_close(fd, 0); > + sql_print_information("AWS KMS plugin: generated encrypted datakey for key id=%u, version=%u", > + keyid, version); > + DBUG_RETURN(0); > +} > + > +/* Key rotation for a single key */ > +static int rotate_single_key(uint key_id) > +{ > + uint ver; > + ver= latest_version_cache[key_id]; > + > + if (!ver) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "key %u does not exist", MYF(ME_JUST_WARNING), key_id); > + return -1; > + } > + else if (aws_generate_datakey(key_id, ver + 1)) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "Could not generate datakey for key id= %u, ver= %u", > + MYF(ME_JUST_WARNING), key_id, ver); > + return -1; > + } > + else > + { > + KEY_INFO info; > + info.key_id= key_id; > + info.key_version = ver + 1; > + if (load_key(&info)) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "Could not load datakey for key id= %u, ver= %u", > + MYF(ME_JUST_WARNING), key_id, ver); > + return -1; > + } > + } > + return 0; > +} > + > +/* Key rotation for all key ids */ > +static int rotate_all_keys() > +{ > + int ret= 0; > + for (map<uint, uint>::iterator it= latest_version_cache.begin(); it != latest_version_cache.end(); it++) > + { > + ret= rotate_single_key(it->first); > + if (ret) > + break; > + } > + return ret; > +} > + > +static void update_rotate(MYSQL_THD, struct st_mysql_sys_var *, void *, const void *val) > +{ > + if (!master_key_id[0]) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "aws_key_management_master_key_id must be set to generate new data keys", MYF(ME_JUST_WARNING)); > + return; > + } > + pthread_mutex_lock(&mtx); > + rotate_key= *(int *)val; > + switch (rotate_key) > + { > + case 0: > + break; > + case -1: > + rotate_all_keys(); > + break; > + default: > + rotate_single_key(rotate_key); > + break; > + } > + rotate_key= 0; > + pthread_mutex_unlock(&mtx); > +} > + > +static unsigned int get_key( > + unsigned int key_id, > + unsigned int version, > + unsigned char* dstbuf, > + unsigned int* buflen) > +{ > + KEY_INFO info; > + > + DBUG_ENTER("get_key"); > + pthread_mutex_lock(&mtx); > + info= key_info_cache[KEY_ID_AND_VERSION(key_id, version)]; > + if (info.length == 0 && !info.load_failed) > + { > + info.key_id= key_id; > + info.key_version= version; > + load_key(&info); > + } > + pthread_mutex_unlock(&mtx); > + > + if (info.load_failed) > + DBUG_RETURN(ENCRYPTION_KEY_VERSION_INVALID); > + if (*buflen < info.length) > + { > + *buflen= info.length; > + DBUG_RETURN(ENCRYPTION_KEY_BUFFER_TOO_SMALL); > + } > + *buflen= info.length; > + memcpy(dstbuf, info.data, info.length); > + DBUG_RETURN(0); > +} > + > + > +/* Plugin defs */ > +struct st_mariadb_encryption aws_key_management_plugin= { > + MariaDB_ENCRYPTION_INTERFACE_VERSION, > + get_latest_key_version, > + get_key, > + // use default encrypt/decrypt functions > + 0, 0, 0, 0, 0 > +}; > + > + > +static TYPELIB key_spec_typelib = > +{ > + array_elements(key_spec_names) - 1, "", > + key_spec_names, NULL > +}; > + > +const char *log_level_names[] = > +{ > + "Off", > + "Fatal", > + "Error", > + "Warn", > + "Info", > + "Debug", > + "Trace", > + 0 > +}; > + > +static TYPELIB log_level_typelib = > +{ > + array_elements(log_level_names) - 1, "", > + log_level_names, NULL > +}; > + > +static MYSQL_SYSVAR_STR(master_key_id, master_key_id, > + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_MEMALLOC, > + "Key id for master encryption key. Used to create new datakeys. If not set, no new keys will be created", > + NULL, NULL, ""); > + > +static MYSQL_SYSVAR_ENUM(key_spec, key_spec, > + PLUGIN_VAR_RQCMDARG, > + "Encryption algorithm used to create new keys.", > + NULL, NULL, 0, &key_spec_typelib); > + > + > +static MYSQL_SYSVAR_ENUM(log_level, log_level, > + PLUGIN_VAR_RQCMDARG, > + "Logging for AWS API", > + NULL, NULL, 0, &log_level_typelib); > + > + > +static MYSQL_SYSVAR_INT(rotate_key, rotate_key, > + PLUGIN_VAR_RQCMDARG, > + "Set this variable to key id to perform rotation of the key. Specify -1 to rotate all keys", > + NULL, update_rotate, 0, -1, INT_MAX, 1); > + > +static struct st_mysql_sys_var* settings[]= { > + MYSQL_SYSVAR(master_key_id), > + MYSQL_SYSVAR(key_spec), > + MYSQL_SYSVAR(rotate_key), > + MYSQL_SYSVAR(log_level), > + NULL > +}; > + > +/* > + Plugin library descriptor > +*/ > +maria_declare_plugin(aws_key_management) > +{ > + MariaDB_ENCRYPTION_PLUGIN, > + &aws_key_management_plugin, > + "aws_key_management", > + "MariaDB Corporation", > + "AWS key management plugin", > + PLUGIN_LICENSE_GPL, > + plugin_init, > + plugin_deinit, > + 0x0100, > + NULL, > + settings, > + "1.0", > + MariaDB_PLUGIN_MATURITY_EXPERIMENTAL > +} > +maria_declare_plugin_end; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] GSoC 2016 MHA GTID support.
by Charles Dirk 11 Mar '16

11 Mar '16

Hi, I am a computer science student at the university of Cape Town and wish to participate in GSoC 2016 with MariaDB Foundation. I am interested if working on making MHA understand GTIDs. I am proficient in C/C+, Python/Perl and Im equally familiar with replication. I look forward to hearing from mentors soon. Cheers, Charles.

3 3

[Maria-developers] MDEV-9709 Unexpected modification of value and warning about out of range value upon ALTER
by Alexander Barkov 11 Mar '16

11 Mar '16

Hi Elena, can you please review a patch for MDEV-9709? I'm quite sure about the code change, it's trivial. Please have a look into the tests. Thanks.

1 0

[Maria-developers] GSoC 2016 Maxscale Filter
by Lisa Brinson 08 Mar '16

08 Mar '16

Hi everyone, My name is Lisa Brinson and I am a junior Computer science student at Oregon State University. I wish to participate in an open source project through GSoC 2016. I am proficient in C and C++ as well as git. I have looked through the proposed projects and I have developed interest in the Maxscale tasks especially the filters. I have been going through the filter API and have also looked at some sample filters in server/modules/filter directory. I will try to write my first filter(a hello world filter) to get familiar with the API. I will be submitting 3 proposals for the 3 Maxscale filters hoping one will be selected. Best regards, Lisa.

3 4

[Maria-developers] Questions regarding GSoC 2016 and Import / Export
by Dan Ungureanu 08 Mar '16

08 Mar '16

Hello, I am a second year student currently attending University Politehnica of Bucharest and I am interested in contributing to MariaDB during this year's Google Summer of Code. I have looked at the ideas you published in the Knowledge Base [1] and two projects caught my attention. The first one is related to the import and export process related to dynamic columns [2] and the second one is related to the GIS enhancements [3]. I am willing to implement both of them, but I am not sure whether the time will allow me properly implement both of them. I cannot estimate how much effort do these projects require as I am still trying to become acquainted with the code base. I hope I am not underestimating the projects and some guidance from one of the mentors (Oleksandr Byelkin, Georg Richter or Alexey Botchkov - Holyfoot) is certainly welcome. I will focus on "Import and export popular data formats from and to dynamic columns" and as soon as I am done with this one, hopefully I will be able to work on GIS enhancements as well. Before continuing, I would like to clarify some aspects related to these projects. Import and export popular data formats from and to dynamic columns: - Can be split in four main tasks: a) adding support for arrays (vectors) in dynamic columns. Currently, dynamic columns only store dictionaries (maps); b) implement a library that can import (parse) and export (build in a specific format) data to and from dynamic columns; c) provide interface to access to the new import and export mechanism; d) provide unit tests for the new features. - The second part (b) of the project is mostly standalone and requires little edits to the project and the third part (c) will be mostly shared by the server and the C connector. - The outcome of (b) is going to be a minimal parsing and building library for various formats. - Some of the formats that I plan to support are: JSON, XML, CSV (maybe XLS too?), PHP Array, YAML, SQL (export as INSERT INTO statement?), etc. One of the key aspects I have to keep in mind when designing the library is to provide a extensible interface to quickly implement new formats in the future. - Probably, most of my work will be located in: - ./server/sql/item_*.(cc|h) - implement user defined functions - ./server/mysys/ma_dyncol.c - provide interface for user defined functions - ./server/include/ma_dyncol.h - ./mariadb-connector-c/libmariadb/mariadb_dyncol.c - provide interface in C Connector - ./mariadb-connector-c/include/mariadb_dyncol.h GIS enhancements: - This project is based on extending the current functionality related to GIS. I am not very familiar with GIS and I still have to do more research based on documents [4], [5] and [6], but the biggest part of the project consists in defining user functions for handling GIS. - I found this compliance test suite [7] for PostGIS that may come in handy. Also, I have a few other questions related to how should I proceed: - I have already read the introductory developers guide [8], the contributing guidelines [9] and will submit the MariaDB Contributor Agreement [10] as soon as possible. Is there anything else I should do before getting started? - I found no template for Google Summer of Code applications. Is there anything in particular that mentors would like to see? I was thinking of organizing my application in multiple sections, some of the most important being: Synopsis, Benefits to the Users, Project Details, Deliverables, Project Schedule and my resume. I believe that I will receive more feedback related after I will have submitted my application. - Most issues already have an assignee. Is it fine if I try and send pull requests for some of those bugs? I could not decide between whether I should send this mail to discuss or developers mailing list. Hopefully, I chose the right one. I am sorry for my long email and thank you for attention. I am looking for feedback, so please do not hesitate to send me any suggestion you might have. [1] https://mariadb.com/kb/en/mariadb/google-summer-of-code-2016/ [2] https://jira.mariadb.org/browse/CONC-125 [3] https://jira.mariadb.org/browse/MDEV-5813 [4] http://www.opengeospatial.org/standards/sfs [5] http://portal.opengeospatial.org/files/?artifact_id=25354 [6] http://www.opengeospatial.org/compliance/ [7] https://svn.osgeo.org/postgis/trunk/extras/ogc_test_suite/README [8] https://mariadb.org/get-involved/getting-started-for-developers/ [9] https://mariadb.com/kb/en/mariadb/contributing-code/ [10] https://mariadb.com/kb/en/mariadb/mca/ Best regards, Dan Ungureanu

2 1

[Maria-developers] GSoC 2016 Maxscale Luafilter
by Pedro Moratta 08 Mar '16

08 Mar '16

Hi, My name in Pedro Mora and I am a MS Computer Science student at Milano Institute of technology. I wish to work on Maxscale filters during GSoC 2016. I am interested in working on the Lua filter because I am familiar with Lua scripting and C/C++ programming as well as working Knowledge of Maxscale. I have been looking at some sample filters but could not find the luafilter mentioned as it seems the github branch https://github.com/mariadb-corporation/MaxScale/blob/luafilter/server/modul… <https://github.com/mariadb-corporation/MaxScale/blob/luafilter/server/modul…> no longer exists. Please I would love to know where the current repo for the Luafilter is located so as to see what had been done and what we would need to enhance this summer. I look forward to feedback soon. Regards, Pedro

1 0

[Maria-developers] Enhancing mysqlbinlog (MDEV-4989 and MDEV-5993)
by Dan Ungureanu 08 Mar '16

08 Mar '16

Hello, I have been looking into MDEV-4989 [1] and MDEV-5993 [2] and as far as I know, the features requested in those two tickets have already been implemented in MySQL. However, the team decided to approach this problem in a different manner and focus more on parallel replication so the GTID implementations diverged. Now, my question is: for starters, is it enough to consider the code from MySQL as some sort of guideline when implementing these features in MariaDB? Kristian Nielsen did a great job explaining GTID in his blog posts and even though the mechanism behind the scene seems to be pretty simple, the codebase is quite massive and I have a hard time going through all of it. >From what I can understand so far, I do not believe that porting the code from MySQL to MariaDB is possible since there are a few key aspects that differ. Is that correct? I hope I made myself clear and I look forward to hearing from you. [1] https://jira.mariadb.org/browse/MDEV-4989 [2] https://jira.mariadb.org/browse/MDEV-5993 Best regards, Dan Ungureanu

2 1

Re: [Maria-developers] [Commits] e19e45b: MDEV-9651 - Simplify audit event dispatching
by Sergei Golubchik 07 Mar '16

07 Mar '16

Hi, Sergey! On Mar 01, Sergey Vojtovich wrote: > revision-id: e19e45b23c07ef8d0679ca0ed7d93b70625e1a26 (mariadb-10.1.8-141-ge19e45b) > parent(s): e9f6c8167e238107643b9ef6228ca23976afb654 > committer: Sergey Vojtovich > timestamp: 2016-03-01 10:21:42 +0400 > message: > > MDEV-9651 - Simplify audit event dispatching > > Simplified audit event dispatching call chain from: Ok to push. Btw, you there's a conveniency helper safe_strlen() that you could've used. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] GSoC 2016 Maxscale Filter Implementation.
by Jason Kravitz 07 Mar '16

07 Mar '16

Hi, I am Jay Kravitz an Informations systems student at Georgetown University and I would love to work on Maxscale during GSoC 2016. I am interested in the Filter projects specifically the filter to capture incoming operations for consumption in external sources. I have build Maxscale on Ubuntu 15 and I am currently looking at some filters in the codebase to understand the filter API. I will also submit a backup proposal for the query injection filter hoping I am given the opportunity to implement one. I look forward to hearing from anyone soon. Regards, JK.

1 0

[Maria-developers] main.delayed fails on timeout
by Alexander Barkov 03 Mar '16

03 Mar '16

Hello Monty, Sergei, Can you please have a look into: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-precise-amd64/builds… Can you please give any suggestions about what we can do to fix the problem with main.delayed failing sporadically? This problem is blocking us badly in Debian. Otto can try to disable this test temporary. But it would be nice to have a real fix. Any tips would be very appreciated. Thanks!

2 1