Re: [Maria-developers] [Maria-discuss] MariaDB encryption
Hi again,
What is the type of license of your code?
I asked internally about license, and it seems like we releasing dual gpl2/apache licensed code.
I would like to know, which interfaces from maria-DB you are using.
I don't 100% understand the question. We didn't write any actual encryption code, but used the one provided in openssl. Other than that, we didn't really "use interfaces", but rather added/modified functionality/interfaces here and there. Can you be more specific ? /Jonas On Sat, Jun 7, 2014 at 11:20 PM, Elmar Eperiesi-Beck <elmar@eperiesi-beck.de
wrote:
Hi! We (eperi) would be glad to do a joined work with Google. Our solution works with MS-SQL, Oracle and other DBs and we are currently porting it to MariaDB - and - as Monty said - its never to late to put some sources together and make the best for the open source community.
What is the type of license of your code?
Jonas, I am looking forward to connect to you directly.
Regards Elmar
Hi!
Hi Jonas, (same Jonas we know from NDBCLUSTER? :-) Good to see you again)
On 6 Jun 2014, at 02:31, Jonas Oreland <jonaso@google.com> wrote:
Hi there, I read this blog post
and wanted to inform you that we at Google has developed on-disk/block-level encryption for Innodb, aria (as used by temporary
http://monty-says.blogspot.com/2014/05/for-your-eyes-only-or-adding-better.h... tables), binlogs and temp-files.
The code is not yet published, but we expect it to be within a few weeks or so. We (of course?) think that it would be better if you instead of developing new code spent the time testing/reviewing ours.
We are out course happy to do this!
I'm happy to answer questions on the topic, and will let you know once we've published it.
The main question I have about the Innodb encryption is if it based on the compression code we did for fusion-io? The idea we had on our side was that by using the new compression hooks we could add encryption with very little changes to the Innodb code. Looking forward to when you are ready to publish the code so we can discuss your changes in detail.
This is great news!
From what I gather, from Monty's blog post (and a 1:1 we had some time back), this is something done by a partner/external company that has a mostly OSS solution, that we should integrate into 10.1
Yes, that's correct. It I would have known that Google was working on encryption I would have included them in my discussions with eperi. Fortunately it's not yet too late to do this. I am sure eperi would like to work on the Google code as a base!
That said, Google's release of something that works for InnoDB, Aria, binlogs, temp files (and presumably not too hard to add for MyISAM) is something we should definitely review and target for 10.1
Yes!
Regards, Monty
Hi, by "interfaces" I was looking for the Maria DB place/ function / hook... where you are enhancing the MariaDB Code. This would help me to understand what you are trying to do. Elmar
Am 17.06.2014 um 07:02 schrieb Jonas Oreland <jonaso@google.com>:
Hi again,
What is the type of license of your code?
I asked internally about license, and it seems like we releasing dual gpl2/apache licensed code.
I would like to know, which interfaces from maria-DB you are using.
I don't 100% understand the question. We didn't write any actual encryption code, but used the one provided in openssl. Other than that, we didn't really "use interfaces", but rather added/modified functionality/interfaces here and there.
Can you be more specific ?
/Jonas
On Sat, Jun 7, 2014 at 11:20 PM, Elmar Eperiesi-Beck <elmar@eperiesi-beck.de> wrote: Hi! We (eperi) would be glad to do a joined work with Google. Our solution works with MS-SQL, Oracle and other DBs and we are currently porting it to MariaDB - and - as Monty said - its never to late to put some sources together and make the best for the open source community.
What is the type of license of your code?
Jonas, I am looking forward to connect to you directly.
Regards Elmar
Hi!
Hi Jonas, (same Jonas we know from NDBCLUSTER? :-) Good to see you again)
On 6 Jun 2014, at 02:31, Jonas Oreland <jonaso@google.com> wrote:
Hi there, I read this blog post http://monty-says.blogspot.com/2014/05/for-your-eyes-only-or-adding-better.h... and wanted to inform you that we at Google has developed on-disk/block-level encryption for Innodb, aria (as used by temporary tables), binlogs and temp-files. The code is not yet published, but we expect it to be within a few weeks or so. We (of course?) think that it would be better if you instead of developing new code spent the time testing/reviewing ours.
We are out course happy to do this!
I'm happy to answer questions on the topic, and will let you know once we've published it.
The main question I have about the Innodb encryption is if it based on the compression code we did for fusion-io? The idea we had on our side was that by using the new compression hooks we could add encryption with very little changes to the Innodb code. Looking forward to when you are ready to publish the code so we can discuss your changes in detail.
This is great news!
From what I gather, from Monty's blog post (and a 1:1 we had some time back), this is something done by a partner/external company that has a mostly OSS solution, that we should integrate into 10.1
Yes, that's correct. It I would have known that Google was working on encryption I would have included them in my discussions with eperi. Fortunately it's not yet too late to do this. I am sure eperi would like to work on the Google code as a base!
That said, Google's release of something that works for InnoDB, Aria, binlogs, temp files (and presumably not too hard to add for MyISAM) is something we should definitely review and target for 10.1
Yes!
Regards, Monty
Hi again,
by "interfaces" I was looking for the Maria DB place/ function / hook... where you are enhancing the MariaDB Code.
I'm not sure how to convey this in a digestible form, attaching diffstats below. Not sure if it's helps :-( There are many aspects of it. And each of the sub-projects (innodb data, innodb log, maria, tempfiles, binlog) has "interesting" details. /Jonas storage/innodb has this diffstat: CMakeLists.txt | 2 btr/btr0cur.cc | 9 buf/buf0buf.cc | 213 +++++ buf/buf0checksum.cc | 8 buf/buf0dblwr.cc | 40 - buf/buf0flu.cc | 6 buf/buf0rea.cc | 7 dict/dict0load.cc | 8 fil/fil0crypt.cc | 1986 +++++++++++++++++++++++++++++++++++++++++++++++++++ fil/fil0fil.cc | 280 ++++++- fsp/fsp0fsp.cc | 36 handler/ha_innodb.cc | 110 ++ handler/i_s.cc | 292 +++++++ handler/i_s.h | 1 include/buf0buf.h | 60 + include/buf0buf.ic | 29 include/fil0fil.h | 266 ++++++ include/fsp0fsp.h | 9 include/log0crypt.h | 85 ++ include/log0log.h | 21 include/log0recv.h | 5 include/mtr0log.ic | 2 include/mtr0mtr.h | 8 include/srv0srv.h | 8 log/log0crypt.cc | 256 ++++++ log/log0log.cc | 93 ++ log/log0recv.cc | 35 mtr/mtr0log.cc | 4 row/row0import.cc | 3 srv/srv0srv.cc | 14 srv/srv0start.cc | 29 31 files changed, 3853 insertions(+), 72 deletions(-) storage/maria has this diffstat: CMakeLists.txt | 12 ha_maria.cc | 12 ma_bitmap.c | 63 ++-- ma_blockrec.c | 222 ++++++++------ ma_blockrec.h | 26 + ma_check.c | 49 +-- ma_checkpoint.c | 4 ma_close.c | 2 ma_create.c | 56 +++ ma_crypt.c | 464 ++++++++++++++++++++++++++++++ ma_crypt.h | 26 + ma_delete.c | 2 ma_key_recover.c | 10 ma_loghandler.c | 63 +--- ma_open.c | 48 ++- ma_pagecache.c | 154 ++++++--- ma_pagecache.h | 34 +- ma_pagecrc.c | 118 ++++--- ma_static.c | 1 ma_write.c | 24 - maria_def.h | 81 ++--- unittest/ma_pagecache_consist.c | 28 - unittest/ma_pagecache_rwconsist.c | 27 - unittest/ma_pagecache_rwconsist2.c | 27 - unittest/ma_pagecache_single.c | 27 - unittest/ma_test_loghandler_pagecache-t.c | 29 - 26 files changed, 1102 insertions(+), 507 deletions(-) A noticeable difference between innodb and maria is that we didn't implement encryption of the log for maria, as we only added support for temporary tables. For maria we also only added encryption support for BLOCK format but added all the features to this format so that it was usable for all temp-table scenarios. maria also doesn't have key-rotation feature like innodb has. I couldn't (as) easily extract diffstats for binlog and tempfile encryption. You have to wait for the code to get published... On Tue, Jun 17, 2014 at 7:29 AM, Elmar Eperiesi-Beck <elmar@eperiesi-beck.de
wrote:
Hi, by "interfaces" I was looking for the Maria DB place/ function / hook... where you are enhancing the MariaDB Code. This would help me to understand what you are trying to do.
Elmar
Am 17.06.2014 um 07:02 schrieb Jonas Oreland <jonaso@google.com>:
Hi again,
What is the type of license of your code?
I asked internally about license, and it seems like we releasing dual gpl2/apache licensed code.
I would like to know, which interfaces from maria-DB you are using.
I don't 100% understand the question. We didn't write any actual encryption code, but used the one provided in openssl. Other than that, we didn't really "use interfaces", but rather added/modified functionality/interfaces here and there.
Can you be more specific ?
/Jonas
On Sat, Jun 7, 2014 at 11:20 PM, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> wrote:
Hi! We (eperi) would be glad to do a joined work with Google. Our solution works with MS-SQL, Oracle and other DBs and we are currently porting it to MariaDB - and - as Monty said - its never to late to put some sources together and make the best for the open source community.
What is the type of license of your code?
Jonas, I am looking forward to connect to you directly.
Regards Elmar
Hi!
Hi Jonas, (same Jonas we know from NDBCLUSTER? :-) Good to see you again)
On 6 Jun 2014, at 02:31, Jonas Oreland <jonaso@google.com> wrote:
Hi there, I read this blog post
and wanted to inform you that we at Google has developed on-disk/block-level encryption for Innodb, aria (as used by temporary
http://monty-says.blogspot.com/2014/05/for-your-eyes-only-or-adding-better.h... tables), binlogs and temp-files.
The code is not yet published, but we expect it to be within a few weeks or so. We (of course?) think that it would be better if you instead of developing new code spent the time testing/reviewing ours.
We are out course happy to do this!
I'm happy to answer questions on the topic, and will let you know once we've published it.
The main question I have about the Innodb encryption is if it based on the compression code we did for fusion-io? The idea we had on our side was that by using the new compression hooks we could add encryption with very little changes to the Innodb code. Looking forward to when you are ready to publish the code so we can discuss your changes in detail.
This is great news!
From what I gather, from Monty's blog post (and a 1:1 we had some time back), this is something done by a partner/external company that has a mostly OSS solution, that we should integrate into 10.1
Yes, that's correct. It I would have known that Google was working on encryption I would have included them in my discussions with eperi. Fortunately it's not yet too late to do this. I am sure eperi would like to work on the Google code as a base!
That said, Google's release of something that works for InnoDB, Aria, binlogs, temp files (and presumably not too hard to add for MyISAM) is something we should definitely review and target for 10.1
Yes!
Regards, Monty
Hi, I agree with you. If we want to know, what Google has developed as encryption feature, we will have to wait for your source code to be published. In the meantime, you can find our concept for the encryption on our website: http://bit.ly/1slJyuI Feedback (negative and positive) from all of you is welcome - and needed! Best Regards Elmar Am 17.06.2014 um 12:50 schrieb Jonas Oreland <jonaso@google.com>:
Hi again,
by "interfaces" I was looking for the Maria DB place/ function / hook... where you are enhancing the MariaDB Code.
I'm not sure how to convey this in a digestible form, attaching diffstats below. Not sure if it's helps :-(
There are many aspects of it. And each of the sub-projects (innodb data, innodb log, maria, tempfiles, binlog) has "interesting" details.
/Jonas
storage/innodb has this diffstat: CMakeLists.txt | 2 btr/btr0cur.cc | 9 buf/buf0buf.cc | 213 +++++ buf/buf0checksum.cc | 8 buf/buf0dblwr.cc | 40 - buf/buf0flu.cc | 6 buf/buf0rea.cc | 7 dict/dict0load.cc | 8 fil/fil0crypt.cc | 1986 +++++++++++++++++++++++++++++++++++++++++++++++++++ fil/fil0fil.cc | 280 ++++++- fsp/fsp0fsp.cc | 36 handler/ha_innodb.cc | 110 ++ handler/i_s.cc | 292 +++++++ handler/i_s.h | 1 include/buf0buf.h | 60 + include/buf0buf.ic | 29 include/fil0fil.h | 266 ++++++ include/fsp0fsp.h | 9 include/log0crypt.h | 85 ++ include/log0log.h | 21 include/log0recv.h | 5 include/mtr0log.ic | 2 include/mtr0mtr.h | 8 include/srv0srv.h | 8 log/log0crypt.cc | 256 ++++++ log/log0log.cc | 93 ++ log/log0recv.cc | 35 mtr/mtr0log.cc | 4 row/row0import.cc | 3 srv/srv0srv.cc | 14 srv/srv0start.cc | 29 31 files changed, 3853 insertions(+), 72 deletions(-)
storage/maria has this diffstat: CMakeLists.txt | 12 ha_maria.cc | 12 ma_bitmap.c | 63 ++-- ma_blockrec.c | 222 ++++++++------ ma_blockrec.h | 26 + ma_check.c | 49 +-- ma_checkpoint.c | 4 ma_close.c | 2 ma_create.c | 56 +++ ma_crypt.c | 464 ++++++++++++++++++++++++++++++ ma_crypt.h | 26 + ma_delete.c | 2 ma_key_recover.c | 10 ma_loghandler.c | 63 +--- ma_open.c | 48 ++- ma_pagecache.c | 154 ++++++--- ma_pagecache.h | 34 +- ma_pagecrc.c | 118 ++++--- ma_static.c | 1 ma_write.c | 24 - maria_def.h | 81 ++--- unittest/ma_pagecache_consist.c | 28 - unittest/ma_pagecache_rwconsist.c | 27 - unittest/ma_pagecache_rwconsist2.c | 27 - unittest/ma_pagecache_single.c | 27 - unittest/ma_test_loghandler_pagecache-t.c | 29 - 26 files changed, 1102 insertions(+), 507 deletions(-)
A noticeable difference between innodb and maria is that we didn't implement encryption of the log for maria, as we only added support for temporary tables. For maria we also only added encryption support for BLOCK format but added all the features to this format so that it was usable for all temp-table scenarios. maria also doesn't have key-rotation feature like innodb has.
I couldn't (as) easily extract diffstats for binlog and tempfile encryption. You have to wait for the code to get published...
On Tue, Jun 17, 2014 at 7:29 AM, Elmar Eperiesi-Beck <elmar@eperiesi-beck.de> wrote: Hi, by "interfaces" I was looking for the Maria DB place/ function / hook... where you are enhancing the MariaDB Code. This would help me to understand what you are trying to do.
Elmar
Am 17.06.2014 um 07:02 schrieb Jonas Oreland <jonaso@google.com>:
Hi again,
What is the type of license of your code?
I asked internally about license, and it seems like we releasing dual gpl2/apache licensed code.
I would like to know, which interfaces from maria-DB you are using.
I don't 100% understand the question. We didn't write any actual encryption code, but used the one provided in openssl. Other than that, we didn't really "use interfaces", but rather added/modified functionality/interfaces here and there.
Can you be more specific ?
/Jonas
On Sat, Jun 7, 2014 at 11:20 PM, Elmar Eperiesi-Beck <elmar@eperiesi-beck.de> wrote: Hi! We (eperi) would be glad to do a joined work with Google. Our solution works with MS-SQL, Oracle and other DBs and we are currently porting it to MariaDB - and - as Monty said - its never to late to put some sources together and make the best for the open source community.
What is the type of license of your code?
Jonas, I am looking forward to connect to you directly.
Regards Elmar
Hi!
Hi Jonas, (same Jonas we know from NDBCLUSTER? :-) Good to see you again)
On 6 Jun 2014, at 02:31, Jonas Oreland <jonaso@google.com> wrote:
Hi there, I read this blog post http://monty-says.blogspot.com/2014/05/for-your-eyes-only-or-adding-better.h... and wanted to inform you that we at Google has developed on-disk/block-level encryption for Innodb, aria (as used by temporary tables), binlogs and temp-files. The code is not yet published, but we expect it to be within a few weeks or so. We (of course?) think that it would be better if you instead of developing new code spent the time testing/reviewing ours.
We are out course happy to do this!
I'm happy to answer questions on the topic, and will let you know once we've published it.
The main question I have about the Innodb encryption is if it based on the compression code we did for fusion-io? The idea we had on our side was that by using the new compression hooks we could add encryption with very little changes to the Innodb code. Looking forward to when you are ready to publish the code so we can discuss your changes in detail.
This is great news!
From what I gather, from Monty's blog post (and a 1:1 we had some time back), this is something done by a partner/external company that has a mostly OSS solution, that we should integrate into 10.1
Yes, that's correct. It I would have known that Google was working on encryption I would have included them in my discussions with eperi. Fortunately it's not yet too late to do this. I am sure eperi would like to work on the Google code as a base!
That said, Google's release of something that works for InnoDB, Aria, binlogs, temp files (and presumably not too hard to add for MyISAM) is something we should definitely review and target for 10.1
Yes!
Regards, Monty
well, for a first version, i think it's nice :) maybe more information about the key server should be nice about key file... if the attacker know the file and contents, he/she could decrypt the table/column? 2014-06-17 13:40 GMT-03:00 Elmar Eperiesi-Beck <elmar@eperiesi-beck.de>:
Hi, I agree with you. If we want to know, what Google has developed as encryption feature, we will have to wait for your source code to be published.
In the meantime, you can find our concept for the encryption on our website: http://bit.ly/1slJyuI Feedback (negative and positive) from all of you is welcome - and needed!
Best Regards Elmar
Am 17.06.2014 um 12:50 schrieb Jonas Oreland <jonaso@google.com>:
Hi again,
by "interfaces" I was looking for the Maria DB place/ function / hook... where you are enhancing the MariaDB Code.
I'm not sure how to convey this in a digestible form, attaching diffstats below. Not sure if it's helps :-(
There are many aspects of it. And each of the sub-projects (innodb data, innodb log, maria, tempfiles, binlog) has "interesting" details.
/Jonas
storage/innodb has this diffstat: CMakeLists.txt | 2 btr/btr0cur.cc | 9 buf/buf0buf.cc | 213 +++++ buf/buf0checksum.cc | 8 buf/buf0dblwr.cc | 40 - buf/buf0flu.cc | 6 buf/buf0rea.cc | 7 dict/dict0load.cc | 8 fil/fil0crypt.cc | 1986 +++++++++++++++++++++++++++++++++++++++++++++++++++ fil/fil0fil.cc | 280 ++++++- fsp/fsp0fsp.cc | 36 handler/ha_innodb.cc | 110 ++ handler/i_s.cc | 292 +++++++ handler/i_s.h | 1 include/buf0buf.h | 60 + include/buf0buf.ic | 29 include/fil0fil.h | 266 ++++++ include/fsp0fsp.h | 9 include/log0crypt.h | 85 ++ include/log0log.h | 21 include/log0recv.h | 5 include/mtr0log.ic | 2 include/mtr0mtr.h | 8 include/srv0srv.h | 8 log/log0crypt.cc | 256 ++++++ log/log0log.cc | 93 ++ log/log0recv.cc | 35 mtr/mtr0log.cc | 4 row/row0import.cc | 3 srv/srv0srv.cc | 14 srv/srv0start.cc | 29 31 files changed, 3853 insertions(+), 72 deletions(-)
storage/maria has this diffstat: CMakeLists.txt | 12 ha_maria.cc | 12 ma_bitmap.c | 63 ++-- ma_blockrec.c | 222 ++++++++------ ma_blockrec.h | 26 + ma_check.c | 49 +-- ma_checkpoint.c | 4 ma_close.c | 2 ma_create.c | 56 +++ ma_crypt.c | 464 ++++++++++++++++++++++++++++++ ma_crypt.h | 26 + ma_delete.c | 2 ma_key_recover.c | 10 ma_loghandler.c | 63 +--- ma_open.c | 48 ++- ma_pagecache.c | 154 ++++++--- ma_pagecache.h | 34 +- ma_pagecrc.c | 118 ++++--- ma_static.c | 1 ma_write.c | 24 - maria_def.h | 81 ++--- unittest/ma_pagecache_consist.c | 28 - unittest/ma_pagecache_rwconsist.c | 27 - unittest/ma_pagecache_rwconsist2.c | 27 - unittest/ma_pagecache_single.c | 27 - unittest/ma_test_loghandler_pagecache-t.c | 29 - 26 files changed, 1102 insertions(+), 507 deletions(-)
A noticeable difference between innodb and maria is that we didn't implement encryption of the log for maria, as we only added support for temporary tables. For maria we also only added encryption support for BLOCK format but added all the features to this format so that it was usable for all temp-table scenarios. maria also doesn't have key-rotation feature like innodb has.
I couldn't (as) easily extract diffstats for binlog and tempfile encryption. You have to wait for the code to get published...
On Tue, Jun 17, 2014 at 7:29 AM, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> wrote:
Hi, by "interfaces" I was looking for the Maria DB place/ function / hook... where you are enhancing the MariaDB Code. This would help me to understand what you are trying to do.
Elmar
Am 17.06.2014 um 07:02 schrieb Jonas Oreland <jonaso@google.com>:
Hi again,
What is the type of license of your code?
I asked internally about license, and it seems like we releasing dual gpl2/apache licensed code.
I would like to know, which interfaces from maria-DB you are using.
I don't 100% understand the question. We didn't write any actual encryption code, but used the one provided in openssl. Other than that, we didn't really "use interfaces", but rather added/modified functionality/interfaces here and there.
Can you be more specific ?
/Jonas
On Sat, Jun 7, 2014 at 11:20 PM, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> wrote:
Hi! We (eperi) would be glad to do a joined work with Google. Our solution works with MS-SQL, Oracle and other DBs and we are currently porting it to MariaDB - and - as Monty said - its never to late to put some sources together and make the best for the open source community.
What is the type of license of your code?
Jonas, I am looking forward to connect to you directly.
Regards Elmar
Hi!
Hi Jonas, (same Jonas we know from NDBCLUSTER? :-) Good to see you again)
On 6 Jun 2014, at 02:31, Jonas Oreland <jonaso@google.com> wrote:
Hi there, I read this blog post
and wanted to inform you that we at Google has developed on-disk/block-level encryption for Innodb, aria (as used by temporary
http://monty-says.blogspot.com/2014/05/for-your-eyes-only-or-adding-better.h... tables), binlogs and temp-files.
The code is not yet published, but we expect it to be within a few weeks or so. We (of course?) think that it would be better if you instead of developing new code spent the time testing/reviewing ours.
We are out course happy to do this!
I'm happy to answer questions on the topic, and will let you know once we've published it.
The main question I have about the Innodb encryption is if it based on the compression code we did for fusion-io? The idea we had on our side was that by using the new compression hooks we could add encryption with very little changes to the Innodb code. Looking forward to when you are ready to publish the code so we can discuss your changes in detail.
This is great news!
From what I gather, from Monty's blog post (and a 1:1 we had some time back), this is something done by a partner/external company that has a mostly OSS solution, that we should integrate into 10.1
Yes, that's correct. It I would have known that Google was working on encryption I would have included them in my discussions with eperi. Fortunately it's not yet too late to do this. I am sure eperi would like to work on the Google code as a base!
That said, Google's release of something that works for InnoDB, Aria, binlogs, temp files (and presumably not too hard to add for MyISAM) is something we should definitely review and target for 10.1
Yes!
Regards, Monty
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
humm, now i'm thinking as a data warehouse think about installing a server (server 1) in somewhere (maybe saara desert).... i connect the "server 1" to internet, and configure the server uri to point to my central server (server central), maybe at moon when the mysqld/mariadbd start, it will contact the central server and get all keys, or only get keys when i need? for example a server with 1000 tables and 1000 diferent keys, they are all stored at memory at boot time, or only when i need read/write access to that table? if i remove the internet link, the "server 1" will not read tables, right? in this case, if i have the keyfile in a pendrive, or a cd or dvd, could i redirect it to a key file and start database, as a backup solution?
At startup the keys will be read once and kept in memory. Normaly you are not going to encrypt 1000 tables, because you just encrypt the content that is confidential. But yes- each key has to be in the memory. Or you use an external encryption/key server that handels the encryption and the key-management outside the DB. We enhanced the concept, that it is possible to deliver the key manually at server startup. You can have it e.g. on a pendrive and start the server with the keys as a backup. Am 17.06.2014 um 18:55 schrieb Roberto Spadim <roberto@spadim.com.br>:
humm, now i'm thinking as a data warehouse think about installing a server (server 1) in somewhere (maybe saara desert).... i connect the "server 1" to internet, and configure the server uri to point to my central server (server central), maybe at moon
when the mysqld/mariadbd start, it will contact the central server and get all keys, or only get keys when i need? for example a server with 1000 tables and 1000 diferent keys, they are all stored at memory at boot time, or only when i need read/write access to that table?
if i remove the internet link, the "server 1" will not read tables, right? in this case, if i have the keyfile in a pendrive, or a cd or dvd, could i redirect it to a key file and start database, as a backup solution?
nice, check what i'm thinking about... 1) i start mariadb without keys i start my app here i must check that all tables are 'unlocked' and read to use, we will have a method to this? at mysql_connect i will check if keys are loaded, maybe a SHOW STATUS like 'encryption_keys_loaded' = 1 or 0 2) about externall acess to include encryption/key maybe a sql statment? INSERT INTO mysql.encrypt_keys (key,value) value (1,"abcdefg.....") just an idea about external key uploading or an external server (no problem) 2014-06-20 9:51 GMT-03:00 Elmar Eperiesi-Beck <elmar@eperiesi-beck.de>:
At startup the keys will be read once and kept in memory. Normaly you are not going to encrypt 1000 tables, because you just encrypt the content that is confidential. But yes- each key has to be in the memory. Or you use an external encryption/key server that handels the encryption and the key-management outside the DB.
We enhanced the concept, that it is possible to deliver the key manually at server startup. You can have it e.g. on a pendrive and start the server with the keys as a backup.
Am 17.06.2014 um 18:55 schrieb Roberto Spadim <roberto@spadim.com.br>:
humm, now i'm thinking as a data warehouse think about installing a server (server 1) in somewhere (maybe saara desert).... i connect the "server 1" to internet, and configure the server uri to point to my central server (server central), maybe at moon
when the mysqld/mariadbd start, it will contact the central server and get all keys, or only get keys when i need? for example a server with 1000 tables and 1000 diferent keys, they are all stored at memory at boot time, or only when i need read/write access to that table?
if i remove the internet link, the "server 1" will not read tables, right? in this case, if i have the keyfile in a pendrive, or a cd or dvd, could i redirect it to a key file and start database, as a backup solution?
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
1) thats a good point, we will extend our coding to mysql_connect 2) yes, we want to do this with an INSERT statement - a bit more complex, but yes…. We will update the concept paper and come back to you beginning of next week. Am 20.06.2014 um 16:28 schrieb Roberto Spadim <roberto@spadim.com.br>:
nice, check what i'm thinking about... 1) i start mariadb without keys i start my app here i must check that all tables are 'unlocked' and read to use, we will have a method to this? at mysql_connect i will check if keys are loaded, maybe a SHOW STATUS like 'encryption_keys_loaded' = 1 or 0
2) about externall acess to include encryption/key maybe a sql statment? INSERT INTO mysql.encrypt_keys (key,value) value (1,"abcdefg.....")
just an idea about external key uploading or an external server (no problem)
2014-06-20 9:51 GMT-03:00 Elmar Eperiesi-Beck <elmar@eperiesi-beck.de>:
At startup the keys will be read once and kept in memory. Normaly you are not going to encrypt 1000 tables, because you just encrypt the content that is confidential. But yes- each key has to be in the memory. Or you use an external encryption/key server that handels the encryption and the key-management outside the DB.
We enhanced the concept, that it is possible to deliver the key manually at server startup. You can have it e.g. on a pendrive and start the server with the keys as a backup.
Am 17.06.2014 um 18:55 schrieb Roberto Spadim <roberto@spadim.com.br>:
humm, now i'm thinking as a data warehouse think about installing a server (server 1) in somewhere (maybe saara desert).... i connect the "server 1" to internet, and configure the server uri to point to my central server (server central), maybe at moon
when the mysqld/mariadbd start, it will contact the central server and get all keys, or only get keys when i need? for example a server with 1000 tables and 1000 diferent keys, they are all stored at memory at boot time, or only when i need read/write access to that table?
if i remove the internet link, the "server 1" will not read tables, right? in this case, if i have the keyfile in a pendrive, or a cd or dvd, could i redirect it to a key file and start database, as a backup solution?
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
:) very nice I will wait :) Em sexta-feira, 20 de junho de 2014, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> escreveu:
1) thats a good point, we will extend our coding to mysql_connect
2) yes, we want to do this with an INSERT statement - a bit more complex, but yes….
We will update the concept paper and come back to you beginning of next week.
Am 20.06.2014 um 16:28 schrieb Roberto Spadim <roberto@spadim.com.br <javascript:;>>:
nice, check what i'm thinking about... 1) i start mariadb without keys i start my app here i must check that all tables are 'unlocked' and read to use, we will have a method to this? at mysql_connect i will check if keys are loaded, maybe a SHOW STATUS like 'encryption_keys_loaded' = 1 or 0
2) about externall acess to include encryption/key maybe a sql statment? INSERT INTO mysql.encrypt_keys (key,value) value (1,"abcdefg.....")
just an idea about external key uploading or an external server (no problem)
At startup the keys will be read once and kept in memory. Normaly you are not going to encrypt 1000 tables, because you just encrypt the content
2014-06-20 9:51 GMT-03:00 Elmar Eperiesi-Beck <elmar@eperiesi-beck.de <javascript:;>>: that
is confidential. But yes- each key has to be in the memory. Or you use an external encryption/key server that handels the encryption and the key-management outside the DB.
We enhanced the concept, that it is possible to deliver the key manually at server startup. You can have it e.g. on a pendrive and start the server with the keys as a backup.
Am 17.06.2014 um 18:55 schrieb Roberto Spadim <roberto@spadim.com.br <javascript:;>>:
humm, now i'm thinking as a data warehouse think about installing a server (server 1) in somewhere (maybe saara desert).... i connect the "server 1" to internet, and configure the server uri to point to my central server (server central), maybe at moon
when the mysqld/mariadbd start, it will contact the central server and get all keys, or only get keys when i need? for example a server with 1000 tables and 1000 diferent keys, they are all stored at memory at boot time, or only when i need read/write access to that table?
if i remove the internet link, the "server 1" will not read tables, right? in this case, if i have the keyfile in a pendrive, or a cd or dvd, could i redirect it to a key file and start database, as a backup solution?
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
=] any news? 2014-06-20 13:47 GMT-03:00 Roberto Spadim <roberto@spadim.com.br>:
:) very nice I will wait :)
Em sexta-feira, 20 de junho de 2014, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> escreveu:
1) thats a good point, we will extend our coding to mysql_connect
2) yes, we want to do this with an INSERT statement - a bit more complex, but yes….
We will update the concept paper and come back to you beginning of next week.
Am 20.06.2014 um 16:28 schrieb Roberto Spadim <roberto@spadim.com.br>:
nice, check what i'm thinking about... 1) i start mariadb without keys i start my app here i must check that all tables are 'unlocked' and read to use, we will have a method to this? at mysql_connect i will check if keys are loaded, maybe a SHOW STATUS like 'encryption_keys_loaded' = 1 or 0
2) about externall acess to include encryption/key maybe a sql statment? INSERT INTO mysql.encrypt_keys (key,value) value (1,"abcdefg.....")
just an idea about external key uploading or an external server (no problem)
At startup the keys will be read once and kept in memory. Normaly you are not going to encrypt 1000 tables, because you just encrypt the content
2014-06-20 9:51 GMT-03:00 Elmar Eperiesi-Beck <elmar@eperiesi-beck.de>: that
is confidential. But yes- each key has to be in the memory. Or you use an external encryption/key server that handels the encryption and the key-management outside the DB.
We enhanced the concept, that it is possible to deliver the key manually at server startup. You can have it e.g. on a pendrive and start the server with the keys as a backup.
Am 17.06.2014 um 18:55 schrieb Roberto Spadim <roberto@spadim.com.br>:
humm, now i'm thinking as a data warehouse think about installing a server (server 1) in somewhere (maybe saara desert).... i connect the "server 1" to internet, and configure the server uri to point to my central server (server central), maybe at moon
when the mysqld/mariadbd start, it will contact the central server and get all keys, or only get keys when i need? for example a server with 1000 tables and 1000 diferent keys, they are all stored at memory at boot time, or only when i need read/write access to that table?
if i remove the internet link, the "server 1" will not read tables, right? in this case, if i have the keyfile in a pendrive, or a cd or dvd, could i redirect it to a key file and start database, as a backup solution?
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
Of course - we are working with high pressure on this topic. Unfortunately, we had more problems than expected to get the code up and running - BUT we made it! Currently we have managed to implement a primitive xor encryption on the page level. We have solved some issues with checksums mainly by deactivating checksum validation for encrypted pages. This is still work in progress. We would rather want to recalculate checksums instead of deactivating them. We are now working on aes_cbc encryption. Problem with blockcyphers in general ist hat the cypher text can be larger than the plaintext. This means that the encrypted data will not fit into the default page size of 16k (4k or 8k). This has to do with aes_cbc padding, which rounds up the size to a divisor of the aes blocksize and in addition to that we need to store key id and iv for the encrypted page and we need to remember the original values of the non encrypted page. Currently we are using the FLUSH LSN field of the page header (Bytes 26-33) and the old_style checksum (bytes 16376 – 16379) to store some of our data. We are not quite sure if it is a good idea to reuse these fields. Maybe the community can help with this question. We were also thinking about extending the original page header of 38 Bytes by a crypto header. From our understanding it should be possible to extend the header by X bytes whenever it is a crypto pagetype, leaving less space for payload in that page. We do not yet know how to accomlish this and if it is a good idea to do so. We might create more problems with extending a header than we solve with it. From what we understand from page compression, extending a header seems to be non trivial. We are always open to ideas of how to overcome these page size problems and are willing to try different routes. So to sum it up - we think, that we are able to submit a first implementation by beginning of october.
Very nice :) working only with innodb or any engine? Em quinta-feira, 18 de setembro de 2014, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> escreveu:
Of course - we are working with high pressure on this topic. Unfortunately, we had more problems than expected to get the code up and running - BUT we made it!
Currently we have managed to implement a primitive xor encryption on the page level. We have solved some issues with checksums mainly by deactivating checksum validation for encrypted pages. This is still work in progress. We would rather want to recalculate checksums instead of deactivating them.
We are now working on aes_cbc encryption. Problem with blockcyphers in general ist hat the cypher text can be larger than the plaintext. This means that the encrypted data will not fit into the default page size of 16k (4k or 8k). This has to do with aes_cbc padding, which rounds up the size to a divisor of the aes blocksize and in addition to that we need to store key id and iv for the encrypted page and we need to remember the original values of the non encrypted page.
Currently we are using the FLUSH LSN field of the page header (Bytes 26-33) and the old_style checksum (bytes 16376 – 16379) to store some of our data. We are not quite sure if it is a good idea to reuse these fields. Maybe the community can help with this question.
We were also thinking about extending the original page header of 38 Bytes by a crypto header. From our understanding it should be possible to extend the header by X bytes whenever it is a crypto pagetype, leaving less space for payload in that page. We do not yet know how to accomlish this and if it is a good idea to do so. We might create more problems with extending a header than we solve with it. From what we understand from page compression, extending a header seems to be non trivial. We are always open to ideas of how to overcome these page size problems and are willing to try different routes.
So to sum it up - we think, that we are able to submit a first implementation by beginning of october.
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
We work with XtraDB in the first run and will extend it to others later on. Am 19.09.2014 um 15:04 schrieb Roberto Spadim <roberto@spadim.com.br>:
Very nice :) working only with innodb or any engine?
Em quinta-feira, 18 de setembro de 2014, Elmar Eperiesi-Beck <elmar@eperiesi-beck.de> escreveu: Of course - we are working with high pressure on this topic. Unfortunately, we had more problems than expected to get the code up and running - BUT we made it!
Currently we have managed to implement a primitive xor encryption on the page level. We have solved some issues with checksums mainly by deactivating checksum validation for encrypted pages. This is still work in progress. We would rather want to recalculate checksums instead of deactivating them.
We are now working on aes_cbc encryption. Problem with blockcyphers in general ist hat the cypher text can be larger than the plaintext. This means that the encrypted data will not fit into the default page size of 16k (4k or 8k). This has to do with aes_cbc padding, which rounds up the size to a divisor of the aes blocksize and in addition to that we need to store key id and iv for the encrypted page and we need to remember the original values of the non encrypted page.
Currently we are using the FLUSH LSN field of the page header (Bytes 26-33) and the old_style checksum (bytes 16376 – 16379) to store some of our data. We are not quite sure if it is a good idea to reuse these fields. Maybe the community can help with this question.
We were also thinking about extending the original page header of 38 Bytes by a crypto header. From our understanding it should be possible to extend the header by X bytes whenever it is a crypto pagetype, leaving less space for payload in that page. We do not yet know how to accomlish this and if it is a good idea to do so. We might create more problems with extending a header than we solve with it. From what we understand from page compression, extending a header seems to be non trivial.
We are always open to ideas of how to overcome these page size problems and are willing to try different routes.
So to sum it up - we think, that we are able to submit a first implementation by beginning of october.
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
very nice, good work :) 2014-09-19 10:17 GMT-03:00 Elmar Eperiesi-Beck <elmar@eperiesi-beck.de>:
We work with XtraDB in the first run and will extend it to others later on.
Am 19.09.2014 um 15:04 schrieb Roberto Spadim <roberto@spadim.com.br>:
Very nice :) working only with innodb or any engine?
Em quinta-feira, 18 de setembro de 2014, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> escreveu:
Of course - we are working with high pressure on this topic. Unfortunately, we had more problems than expected to get the code up and running - BUT we made it!
Currently we have managed to implement a primitive xor encryption on the page level. We have solved some issues with checksums mainly by deactivating checksum validation for encrypted pages. This is still work in progress. We would rather want to recalculate checksums instead of deactivating them.
We are now working on aes_cbc encryption. Problem with blockcyphers in general ist hat the cypher text can be larger than the plaintext. This means that the encrypted data will not fit into the default page size of 16k (4k or 8k). This has to do with aes_cbc padding, which rounds up the size to a divisor of the aes blocksize and in addition to that we need to store key id and iv for the encrypted page and we need to remember the original values of the non encrypted page.
Currently we are using the FLUSH LSN field of the page header (Bytes 26-33) and the old_style checksum (bytes 16376 – 16379) to store some of our data. We are not quite sure if it is a good idea to reuse these fields. Maybe the community can help with this question.
We were also thinking about extending the original page header of 38 Bytes by a crypto header. From our understanding it should be possible to extend the header by X bytes whenever it is a crypto pagetype, leaving less space for payload in that page. We do not yet know how to accomlish this and if it is a good idea to do so. We might create more problems with extending a header than we solve with it. From what we understand from page compression, extending a header seems to be non trivial. We are always open to ideas of how to overcome these page size problems and are willing to try different routes.
So to sum it up - we think, that we are able to submit a first implementation by beginning of october.
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
Hi there, our implementation is in production :) but has not yet been open sourced due to other tasks consuming time :( it has solved all problems enumerated above...and I think it would be much better to have one crypt implementation than two! i'll ask the pavel that does the open-sourcing for more info about what is the current state. /Jonas On Fri, Sep 19, 2014 at 12:31 AM, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> wrote:
Of course - we are working with high pressure on this topic. Unfortunately, we had more problems than expected to get the code up and running - BUT we made it!
Currently we have managed to implement a primitive xor encryption on the page level. We have solved some issues with checksums mainly by deactivating checksum validation for encrypted pages. This is still work in progress. We would rather want to recalculate checksums instead of deactivating them.
We are now working on aes_cbc encryption. Problem with blockcyphers in general ist hat the cypher text can be larger than the plaintext. This means that the encrypted data will not fit into the default page size of 16k (4k or 8k). This has to do with aes_cbc padding, which rounds up the size to a divisor of the aes blocksize and in addition to that we need to store key id and iv for the encrypted page and we need to remember the original values of the non encrypted page.
Currently we are using the FLUSH LSN field of the page header (Bytes 26-33) and the old_style checksum (bytes 16376 – 16379) to store some of our data. We are not quite sure if it is a good idea to reuse these fields. Maybe the community can help with this question.
We were also thinking about extending the original page header of 38 Bytes by a crypto header. From our understanding it should be possible to extend the header by X bytes whenever it is a crypto pagetype, leaving less space for payload in that page. We do not yet know how to accomlish this and if it is a good idea to do so. We might create more problems with extending a header than we solve with it. From what we understand from page compression, extending a header seems to be non trivial. We are always open to ideas of how to overcome these page size problems and are willing to try different routes.
So to sum it up - we think, that we are able to submit a first implementation by beginning of october.
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
Very good news Em terça-feira, 23 de setembro de 2014, Jonas Oreland <jonaso@google.com> escreveu:
Hi there,
our implementation is in production :) but has not yet been open sourced due to other tasks consuming time :(
it has solved all problems enumerated above...and I think it would be much better to have one crypt implementation than two! i'll ask the pavel that does the open-sourcing for more info about what is the current state.
/Jonas
On Fri, Sep 19, 2014 at 12:31 AM, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de <javascript:_e(%7B%7D,'cvml','elmar@eperiesi-beck.de');>> wrote:
Of course - we are working with high pressure on this topic. Unfortunately, we had more problems than expected to get the code up and running - BUT we made it!
Currently we have managed to implement a primitive xor encryption on the page level. We have solved some issues with checksums mainly by deactivating checksum validation for encrypted pages. This is still work in progress. We would rather want to recalculate checksums instead of deactivating them.
We are now working on aes_cbc encryption. Problem with blockcyphers in general ist hat the cypher text can be larger than the plaintext. This means that the encrypted data will not fit into the default page size of 16k (4k or 8k). This has to do with aes_cbc padding, which rounds up the size to a divisor of the aes blocksize and in addition to that we need to store key id and iv for the encrypted page and we need to remember the original values of the non encrypted page.
Currently we are using the FLUSH LSN field of the page header (Bytes 26-33) and the old_style checksum (bytes 16376 – 16379) to store some of our data. We are not quite sure if it is a good idea to reuse these fields. Maybe the community can help with this question.
We were also thinking about extending the original page header of 38 Bytes by a crypto header. From our understanding it should be possible to extend the header by X bytes whenever it is a crypto pagetype, leaving less space for payload in that page. We do not yet know how to accomlish this and if it is a good idea to do so. We might create more problems with extending a header than we solve with it. From what we understand from page compression, extending a header seems to be non trivial. We are always open to ideas of how to overcome these page size problems and are willing to try different routes.
So to sum it up - we think, that we are able to submit a first implementation by beginning of october.
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net <javascript:_e(%7B%7D,'cvml','maria-developers@lists.launchpad.net');> Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
Hi there, we just managed to get the problems solved much faster as assumed. Thanks to the community! Perhaps it makes sense to have just one crypto implementation to do not confuse users. You can find our Source Code under: https://github.com/eperi-GmbH/server . As fork of the actual 10.1 Code base. By end of this week we will finish the tests for the encrypted external key file and some other features and update the project. Please keep in mind, that you have to use the MariaDB Configfile. Please use the mode „innodb_file_per_table“ on the storage engine XtraDB. To create an encrypt table you can use: create table a (value varchar(100)) PAGE_ENCRYPTION=1 PAGE_ENCRYPTION_KEY=255; For the encryption we use aes-128 cvc - until end of this week, it is a fixed key to make testing easier. If you want to have page compression enabled also, you can use: create table b (value varchar(100)) PAGE_ENCRYPTION=1 PAGE_ENCRYPTION_KEY=255 PAGE_COMPRESSED=1; Example Configuration: [mysqld] innodb_file_per_table innodb_file_format=Barracuda Have fun with this implementation and feel free to play with it! We are looking forward to your comments. Am 23.09.2014 um 12:26 schrieb Jonas Oreland <jonaso@google.com>:
Hi there,
our implementation is in production :) but has not yet been open sourced due to other tasks consuming time :(
it has solved all problems enumerated above...and I think it would be much better to have one crypt implementation than two! i'll ask the pavel that does the open-sourcing for more info about what is the current state.
/Jonas
On Fri, Sep 19, 2014 at 12:31 AM, Elmar Eperiesi-Beck <elmar@eperiesi-beck.de> wrote: Of course - we are working with high pressure on this topic. Unfortunately, we had more problems than expected to get the code up and running - BUT we made it!
Currently we have managed to implement a primitive xor encryption on the page level. We have solved some issues with checksums mainly by deactivating checksum validation for encrypted pages. This is still work in progress. We would rather want to recalculate checksums instead of deactivating them.
We are now working on aes_cbc encryption. Problem with blockcyphers in general ist hat the cypher text can be larger than the plaintext. This means that the encrypted data will not fit into the default page size of 16k (4k or 8k). This has to do with aes_cbc padding, which rounds up the size to a divisor of the aes blocksize and in addition to that we need to store key id and iv for the encrypted page and we need to remember the original values of the non encrypted page.
Currently we are using the FLUSH LSN field of the page header (Bytes 26-33) and the old_style checksum (bytes 16376 – 16379) to store some of our data. We are not quite sure if it is a good idea to reuse these fields. Maybe the community can help with this question.
We were also thinking about extending the original page header of 38 Bytes by a crypto header. From our understanding it should be possible to extend the header by X bytes whenever it is a crypto pagetype, leaving less space for payload in that page. We do not yet know how to accomlish this and if it is a good idea to do so. We might create more problems with extending a header than we solve with it. From what we understand from page compression, extending a header seems to be non trivial.
We are always open to ideas of how to overcome these page size problems and are willing to try different routes.
So to sum it up - we think, that we are able to submit a first implementation by beginning of october.
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
=] nice, i will try soon 2014-09-23 15:50 GMT-03:00 Elmar Eperiesi-Beck <elmar@eperiesi-beck.de>:
Hi there, we just managed to get the problems solved much faster as assumed. Thanks to the community!
Perhaps it makes sense to have just one crypto implementation to do not confuse users. You can find our Source Code under: https://github.com/eperi-GmbH/server . As fork of the actual 10.1 Code base.
By end of this week we will finish the tests for the encrypted external key file and some other features and update the project. Please keep in mind, that you have to use the MariaDB Configfile. Please use the mode „innodb_file_per_table“ on the storage engine XtraDB.
To create an encrypt table you can use: create table a (value varchar(100)) PAGE_ENCRYPTION=1 PAGE_ENCRYPTION_KEY=255;
For the encryption we use aes-128 cvc - until end of this week, it is a fixed key to make testing easier.
If you want to have page compression enabled also, you can use: create table b (value varchar(100)) PAGE_ENCRYPTION=1 PAGE_ENCRYPTION_KEY=255 PAGE_COMPRESSED=1;
Example Configuration: [mysqld] innodb_file_per_table innodb_file_format=Barracuda
Have fun with this implementation and feel free to play with it! We are looking forward to your comments.
Am 23.09.2014 um 12:26 schrieb Jonas Oreland <jonaso@google.com>:
Hi there,
our implementation is in production :) but has not yet been open sourced due to other tasks consuming time :(
it has solved all problems enumerated above...and I think it would be much better to have one crypt implementation than two! i'll ask the pavel that does the open-sourcing for more info about what is the current state.
/Jonas
On Fri, Sep 19, 2014 at 12:31 AM, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> wrote: Of course - we are working with high pressure on this topic. Unfortunately, we had more problems than expected to get the code up and running - BUT we made it!
Currently we have managed to implement a primitive xor encryption on the page level. We have solved some issues with checksums mainly by deactivating checksum validation for encrypted pages. This is still work in progress. We would rather want to recalculate checksums instead of deactivating them.
We are now working on aes_cbc encryption. Problem with blockcyphers in general ist hat the cypher text can be larger than the plaintext. This means that the encrypted data will not fit into the default page size of 16k (4k or 8k). This has to do with aes_cbc padding, which rounds up the size to a divisor of the aes blocksize and in addition to that we need to store key id and iv for the encrypted page and we need to remember the original values of the non encrypted page.
Currently we are using the FLUSH LSN field of the page header (Bytes 26-33) and the old_style checksum (bytes 16376 – 16379) to store some of our data. We are not quite sure if it is a good idea to reuse these fields. Maybe the community can help with this question.
We were also thinking about extending the original page header of 38 Bytes by a crypto header. From our understanding it should be possible to extend the header by X bytes whenever it is a crypto pagetype, leaving less space for payload in that page. We do not yet know how to accomlish this and if it is a good idea to do so. We might create more problems with extending a header than we solve with it. From what we understand from page compression, extending a header seems to be non trivial.
We are always open to ideas of how to overcome these page size problems and are willing to try different routes.
So to sum it up - we think, that we are able to submit a first implementation by beginning of october.
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
Thank you for your feedback - What we want to achieve is, that the encryption key is stored outside the database. But you are right – if an attacker has access to the key and the encrypted value, he is able to decrypt the content. That means you have to store the key file in a place, where the attacker with system privileges is unable to access it. But the DB-Process must be able to have access at startup time. We have enhanced the concept with the possibility to enter the key manually during the startup phase if a keyfile does not exist. Or you use an external encryption/decryption server that does all the encryption/decryption operation in the key server and does not transfer any key. Than an attacker has to steal the database and the key server and have the access credentials to both... Am 17.06.2014 um 18:49 schrieb Roberto Spadim <roberto@spadim.com.br>:
well, for a first version, i think it's nice :) maybe more information about the key server should be nice about key file... if the attacker know the file and contents, he/she could decrypt the table/column?
2014-06-17 13:40 GMT-03:00 Elmar Eperiesi-Beck <elmar@eperiesi-beck.de>: Hi, I agree with you. If we want to know, what Google has developed as encryption feature, we will have to wait for your source code to be published.
In the meantime, you can find our concept for the encryption on our website: http://bit.ly/1slJyuI Feedback (negative and positive) from all of you is welcome - and needed!
Best Regards Elmar
Hi again, 1) we have not done column level encryption at all. 2) keys are managed in a separate module (which can be overridden by plugin, which we do for testing) 3) keys have 32 bit version 4) different key versions can exists simultaneously in database 5) for innodb we encrypt both datafiles and logfiles 6) encryption of innodb datafiles works roughly like this: - pages are encrypted with aes ctr - page 0 of each encrypted datafile contains IV - all pages except page 0 is encrypted - if compression is used, pages are encrypted after compression - pages are encrypted/decrypted by calls added to innodb-buffer-pool module - encrypted pages are "tagged" with which key-version encrypted it - different pages in database can be encrypted with different key version - when key-server module reports that a new key version is available, a background task will re-encrypt database with new key-version (key rotation) - key rotation is performed with configurable number of threads, that will perform configurable amount of IOPS. one can also configure how frequently pages shall be key-rotated (i.e max key "age") /Jonas On Tue, Jun 17, 2014 at 6:40 PM, Elmar Eperiesi-Beck <elmar@eperiesi-beck.de
wrote:
Hi, I agree with you. If we want to know, what Google has developed as encryption feature, we will have to wait for your source code to be published.
In the meantime, you can find our concept for the encryption on our website: http://bit.ly/1slJyuI Feedback (negative and positive) from all of you is welcome - and needed!
Best Regards Elmar
Am 17.06.2014 um 12:50 schrieb Jonas Oreland <jonaso@google.com>:
Hi again,
by "interfaces" I was looking for the Maria DB place/ function / hook... where you are enhancing the MariaDB Code.
I'm not sure how to convey this in a digestible form, attaching diffstats below. Not sure if it's helps :-(
There are many aspects of it. And each of the sub-projects (innodb data, innodb log, maria, tempfiles, binlog) has "interesting" details.
/Jonas
storage/innodb has this diffstat: CMakeLists.txt | 2 btr/btr0cur.cc | 9 buf/buf0buf.cc | 213 +++++ buf/buf0checksum.cc | 8 buf/buf0dblwr.cc | 40 - buf/buf0flu.cc | 6 buf/buf0rea.cc | 7 dict/dict0load.cc | 8 fil/fil0crypt.cc | 1986 +++++++++++++++++++++++++++++++++++++++++++++++++++ fil/fil0fil.cc | 280 ++++++- fsp/fsp0fsp.cc | 36 handler/ha_innodb.cc | 110 ++ handler/i_s.cc | 292 +++++++ handler/i_s.h | 1 include/buf0buf.h | 60 + include/buf0buf.ic | 29 include/fil0fil.h | 266 ++++++ include/fsp0fsp.h | 9 include/log0crypt.h | 85 ++ include/log0log.h | 21 include/log0recv.h | 5 include/mtr0log.ic | 2 include/mtr0mtr.h | 8 include/srv0srv.h | 8 log/log0crypt.cc | 256 ++++++ log/log0log.cc | 93 ++ log/log0recv.cc | 35 mtr/mtr0log.cc | 4 row/row0import.cc | 3 srv/srv0srv.cc | 14 srv/srv0start.cc | 29 31 files changed, 3853 insertions(+), 72 deletions(-)
storage/maria has this diffstat: CMakeLists.txt | 12 ha_maria.cc | 12 ma_bitmap.c | 63 ++-- ma_blockrec.c | 222 ++++++++------ ma_blockrec.h | 26 + ma_check.c | 49 +-- ma_checkpoint.c | 4 ma_close.c | 2 ma_create.c | 56 +++ ma_crypt.c | 464 ++++++++++++++++++++++++++++++ ma_crypt.h | 26 + ma_delete.c | 2 ma_key_recover.c | 10 ma_loghandler.c | 63 +--- ma_open.c | 48 ++- ma_pagecache.c | 154 ++++++--- ma_pagecache.h | 34 +- ma_pagecrc.c | 118 ++++--- ma_static.c | 1 ma_write.c | 24 - maria_def.h | 81 ++--- unittest/ma_pagecache_consist.c | 28 - unittest/ma_pagecache_rwconsist.c | 27 - unittest/ma_pagecache_rwconsist2.c | 27 - unittest/ma_pagecache_single.c | 27 - unittest/ma_test_loghandler_pagecache-t.c | 29 - 26 files changed, 1102 insertions(+), 507 deletions(-)
A noticeable difference between innodb and maria is that we didn't implement encryption of the log for maria, as we only added support for temporary tables. For maria we also only added encryption support for BLOCK format but added all the features to this format so that it was usable for all temp-table scenarios. maria also doesn't have key-rotation feature like innodb has.
I couldn't (as) easily extract diffstats for binlog and tempfile encryption. You have to wait for the code to get published...
On Tue, Jun 17, 2014 at 7:29 AM, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> wrote:
Hi, by "interfaces" I was looking for the Maria DB place/ function / hook... where you are enhancing the MariaDB Code. This would help me to understand what you are trying to do.
Elmar
Am 17.06.2014 um 07:02 schrieb Jonas Oreland <jonaso@google.com>:
Hi again,
What is the type of license of your code?
I asked internally about license, and it seems like we releasing dual gpl2/apache licensed code.
I would like to know, which interfaces from maria-DB you are using.
I don't 100% understand the question. We didn't write any actual encryption code, but used the one provided in openssl. Other than that, we didn't really "use interfaces", but rather added/modified functionality/interfaces here and there.
Can you be more specific ?
/Jonas
On Sat, Jun 7, 2014 at 11:20 PM, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> wrote:
Hi! We (eperi) would be glad to do a joined work with Google. Our solution works with MS-SQL, Oracle and other DBs and we are currently porting it to MariaDB - and - as Monty said - its never to late to put some sources together and make the best for the open source community.
What is the type of license of your code?
Jonas, I am looking forward to connect to you directly.
Regards Elmar
Hi!
Hi Jonas, (same Jonas we know from NDBCLUSTER? :-) Good to see you again)
On 6 Jun 2014, at 02:31, Jonas Oreland <jonaso@google.com> wrote:
Hi there, I read this blog post
and wanted to inform you that we at Google has developed on-disk/block-level encryption for Innodb, aria (as used by temporary
http://monty-says.blogspot.com/2014/05/for-your-eyes-only-or-adding-better.h... tables), binlogs and temp-files.
The code is not yet published, but we expect it to be within a few weeks or so. We (of course?) think that it would be better if you instead of developing new code spent the time testing/reviewing ours.
We are out course happy to do this!
I'm happy to answer questions on the topic, and will let you know once we've published it.
The main question I have about the Innodb encryption is if it based on the compression code we did for fusion-io? The idea we had on our side was that by using the new compression hooks we could add encryption with very little changes to the Innodb code. Looking forward to when you are ready to publish the code so we can discuss your changes in detail.
This is great news!
From what I gather, from Monty's blog post (and a 1:1 we had some time back), this is something done by a partner/external company that has a mostly OSS solution, that we should integrate into 10.1
Yes, that's correct. It I would have known that Google was working on encryption I would have included them in my discussions with eperi. Fortunately it's not yet too late to do this. I am sure eperi would like to work on the Google code as a base!
That said, Google's release of something that works for InnoDB, Aria, binlogs, temp files (and presumably not too hard to add for MyISAM) is something we should definitely review and target for 10.1
Yes!
Regards, Monty
2014-06-17 14:03 GMT-03:00 Jonas Oreland <jonaso@google.com>:
Hi again,
1) we have not done column level encryption at all.
nice, did you check that have an idea at mariadb atlassian mdevs, maybe this could help? https://mariadb.atlassian.net/browse/MDEV-4912
2) keys are managed in a separate module (which can be overridden by plugin, which we do for testing) 3) keys have 32 bit version 4) different key versions can exists simultaneously in database
nice
5) for innodb we encrypt both datafiles and logfiles 6) encryption of innodb datafiles works roughly like this: - pages are encrypted with aes ctr - page 0 of each encrypted datafile contains IV - all pages except page 0 is encrypted - if compression is used, pages are encrypted after compression - pages are encrypted/decrypted by calls added to innodb-buffer-pool module - encrypted pages are "tagged" with which key-version encrypted it - different pages in database can be encrypted with different key version - when key-server module reports that a new key version is available, a background task will re-encrypt database with new key-version (key rotation) - key rotation is performed with configurable number of threads, that will perform configurable amount of IOPS. one can also configure how frequently pages shall be key-rotated (i.e max key "age")
nice something like a i/o wrapper between filesystem calls and innodb, right?
/Jonas
On Tue, Jun 17, 2014 at 6:40 PM, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> wrote:
Hi, I agree with you. If we want to know, what Google has developed as encryption feature, we will have to wait for your source code to be published.
In the meantime, you can find our concept for the encryption on our website: http://bit.ly/1slJyuI Feedback (negative and positive) from all of you is welcome - and needed!
Best Regards Elmar
Am 17.06.2014 um 12:50 schrieb Jonas Oreland <jonaso@google.com>:
Hi again,
by "interfaces" I was looking for the Maria DB place/ function / hook... where you are enhancing the MariaDB Code.
I'm not sure how to convey this in a digestible form, attaching diffstats below. Not sure if it's helps :-(
There are many aspects of it. And each of the sub-projects (innodb data, innodb log, maria, tempfiles, binlog) has "interesting" details.
/Jonas
storage/innodb has this diffstat: CMakeLists.txt | 2 btr/btr0cur.cc | 9 buf/buf0buf.cc | 213 +++++ buf/buf0checksum.cc | 8 buf/buf0dblwr.cc | 40 - buf/buf0flu.cc | 6 buf/buf0rea.cc | 7 dict/dict0load.cc | 8 fil/fil0crypt.cc | 1986 +++++++++++++++++++++++++++++++++++++++++++++++++++ fil/fil0fil.cc | 280 ++++++- fsp/fsp0fsp.cc | 36 handler/ha_innodb.cc | 110 ++ handler/i_s.cc | 292 +++++++ handler/i_s.h | 1 include/buf0buf.h | 60 + include/buf0buf.ic | 29 include/fil0fil.h | 266 ++++++ include/fsp0fsp.h | 9 include/log0crypt.h | 85 ++ include/log0log.h | 21 include/log0recv.h | 5 include/mtr0log.ic | 2 include/mtr0mtr.h | 8 include/srv0srv.h | 8 log/log0crypt.cc | 256 ++++++ log/log0log.cc | 93 ++ log/log0recv.cc | 35 mtr/mtr0log.cc | 4 row/row0import.cc | 3 srv/srv0srv.cc | 14 srv/srv0start.cc | 29 31 files changed, 3853 insertions(+), 72 deletions(-)
storage/maria has this diffstat: CMakeLists.txt | 12 ha_maria.cc | 12 ma_bitmap.c | 63 ++-- ma_blockrec.c | 222 ++++++++------ ma_blockrec.h | 26 + ma_check.c | 49 +-- ma_checkpoint.c | 4 ma_close.c | 2 ma_create.c | 56 +++ ma_crypt.c | 464 ++++++++++++++++++++++++++++++ ma_crypt.h | 26 + ma_delete.c | 2 ma_key_recover.c | 10 ma_loghandler.c | 63 +--- ma_open.c | 48 ++- ma_pagecache.c | 154 ++++++--- ma_pagecache.h | 34 +- ma_pagecrc.c | 118 ++++--- ma_static.c | 1 ma_write.c | 24 - maria_def.h | 81 ++--- unittest/ma_pagecache_consist.c | 28 - unittest/ma_pagecache_rwconsist.c | 27 - unittest/ma_pagecache_rwconsist2.c | 27 - unittest/ma_pagecache_single.c | 27 - unittest/ma_test_loghandler_pagecache-t.c | 29 - 26 files changed, 1102 insertions(+), 507 deletions(-)
A noticeable difference between innodb and maria is that we didn't implement encryption of the log for maria, as we only added support for temporary tables. For maria we also only added encryption support for BLOCK format but added all the features to this format so that it was usable for all temp-table scenarios. maria also doesn't have key-rotation feature like innodb has.
I couldn't (as) easily extract diffstats for binlog and tempfile encryption. You have to wait for the code to get published...
On Tue, Jun 17, 2014 at 7:29 AM, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> wrote:
Hi, by "interfaces" I was looking for the Maria DB place/ function / hook... where you are enhancing the MariaDB Code. This would help me to understand what you are trying to do.
Elmar
Am 17.06.2014 um 07:02 schrieb Jonas Oreland <jonaso@google.com>:
Hi again,
What is the type of license of your code?
I asked internally about license, and it seems like we releasing dual gpl2/apache licensed code.
I would like to know, which interfaces from maria-DB you are using.
I don't 100% understand the question. We didn't write any actual encryption code, but used the one provided in openssl. Other than that, we didn't really "use interfaces", but rather added/modified functionality/interfaces here and there.
Can you be more specific ?
/Jonas
On Sat, Jun 7, 2014 at 11:20 PM, Elmar Eperiesi-Beck < elmar@eperiesi-beck.de> wrote:
Hi! We (eperi) would be glad to do a joined work with Google. Our solution works with MS-SQL, Oracle and other DBs and we are currently porting it to MariaDB - and - as Monty said - its never to late to put some sources together and make the best for the open source community.
What is the type of license of your code?
Jonas, I am looking forward to connect to you directly.
Regards Elmar
Hi!
Hi Jonas, (same Jonas we know from NDBCLUSTER? :-) Good to see you again)
On 6 Jun 2014, at 02:31, Jonas Oreland <jonaso@google.com> wrote:
Hi there, I read this blog post
and wanted to inform you that we at Google has developed on-disk/block-level encryption for Innodb, aria (as used by temporary
http://monty-says.blogspot.com/2014/05/for-your-eyes-only-or-adding-better.h... tables), binlogs and temp-files.
The code is not yet published, but we expect it to be within a few weeks or so. We (of course?) think that it would be better if you instead of developing new code spent the time testing/reviewing ours.
We are out course happy to do this!
I'm happy to answer questions on the topic, and will let you know once we've published it.
The main question I have about the Innodb encryption is if it based on the compression code we did for fusion-io? The idea we had on our side was that by using the new compression hooks we could add encryption with very little changes to the Innodb code. Looking forward to when you are ready to publish the code so we can discuss your changes in detail.
This is great news!
From what I gather, from Monty's blog post (and a 1:1 we had some time back), this is something done by a partner/external company that has a mostly OSS solution, that we should integrate into 10.1
Yes, that's correct. It I would have known that Google was working on encryption I would have included them in my discussions with eperi. Fortunately it's not yet too late to do this. I am sure eperi would like to work on the Google code as a base!
That said, Google's release of something that works for InnoDB, Aria, binlogs, temp files (and presumably not too hard to add for MyISAM) is something we should definitely review and target for 10.1
Yes!
Regards, Monty
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
-- Roberto Spadim SPAEmpresarial Eng. Automação e Controle
participants (3)
-
Elmar Eperiesi-Beck -
Jonas Oreland -
Roberto Spadim