[Maria-developers] [GSOC] ColumnStore: Add full support for DECIMAL
Hello, I took part in GSoC last year, and closed one of MariaDB tasks. Now I want to continue programming with MariaDB developers and solve a new task. I considered some of proposed tasks, and was interested in "ColumnStore: Add full support for DECIMAL". I think this task is doable in the given time and I might solve it. P.S. My github link: https://github.com/medvdanil Regards, Daniil Medvedev
Hi Daniil, Many thanks for your interest. We haven't fully specified it yet (I'll build a full specification later this week). But I suspect the following would be involved: * Implementation of methods to handle MariaDB's DECIMAL format * Support for a longer than 8-byte numeric column type (there is an InfiniDB tree with work for this already) * Modification of the primitives processor for the math * Modification of the function expression processor to handle the new type * Version upgrade support for DECIMAL from the current form to the new form In the mean time please let me know if you have any questions. Kind Regards Andrew (Senior Software Engineer, MariaDB ColumnStore) On 19/03/17 23:09, Даниил Медведев wrote:
Hello,
I took part in GSoC last year, and closed one of MariaDB tasks. Now I want to continue programming with MariaDB developers and solve a new task. I considered some of proposed tasks, and was interested in "ColumnStore: Add full support for DECIMAL". I think this task is doable in the given time and I might solve it.
P.S. My github link: https://github.com/medvdanil
Regards, Daniil Medvedev
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
-- Andrew Hutchings - LinuxJedi - http://www.linuxjedi.co.uk/
I looked at the code (for better understanding of problem), and now I have some questions: Do I understand correctly, that now it is stored as an int64 value with a scale and we want to extend it, and store maximum 65 digits as in documentation? https://github.com/mariadb-corporation/mariadb-columnstore-engine/blob/maste... We will use fixed size type or extandable? I think that I will find all answers of my questions in specification, and there is no point in asking more questions. Regards, Daniil Medvedev 2017-03-20 21:51 GMT+03:00 Andrew Hutchings <andrew@linuxjedi.co.uk>:
Hi Daniil,
Many thanks for your interest. We haven't fully specified it yet (I'll build a full specification later this week). But I suspect the following would be involved:
* Implementation of methods to handle MariaDB's DECIMAL format * Support for a longer than 8-byte numeric column type (there is an InfiniDB tree with work for this already) * Modification of the primitives processor for the math * Modification of the function expression processor to handle the new type * Version upgrade support for DECIMAL from the current form to the new form
In the mean time please let me know if you have any questions.
Kind Regards Andrew (Senior Software Engineer, MariaDB ColumnStore)
On 19/03/17 23:09, Даниил Медведев wrote:
Hello,
I took part in GSoC last year, and closed one of MariaDB tasks. Now I want to continue programming with MariaDB developers and solve a new task. I considered some of proposed tasks, and was interested in "ColumnStore: Add full support for DECIMAL". I think this task is doable in the given time and I might solve it.
P.S. My github link: https://github.com/medvdanil
Regards, Daniil Medvedev
_______________________________________________ Mailing list: https://launchpad.net/~maria-developers Post to : maria-developers@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-developers More help : https://help.launchpad.net/ListHelp
-- Andrew Hutchings - LinuxJedi - http://www.linuxjedi.co.uk/
Hi Daniil, Yes, it is currently stored as an int64. We want to extend it but also some of the internal functionality converts it to a double (see utils/funcexp/func_math.cpp for example). Every function would need to be fully tested to make sure that it uses DECIMAL end-to-end and handles overflow scenarios correctly. At the moment ColumnStore's integer storage can only handle fixed 1, 2, 4 and 8 byte columns. 2, 4 and 8 are used for DECIMAL. It should be somewhat easily extendible to 16 byte columns. I believe this will give us the capability to store the same amount of digits as MariaDB. If at CREATE TABLE it is determined that a DECIMAL will only require 8 bytes it should only use an 8 byte column, if it needs bigger it should use a 16 byte column. In addition the DDL code in ColumnStore will need modifying to handle the range. The ability to have 16 byte columns will also help us at a later date with other potential data types in the future too. Here is the first attempt at implementing 16byte columns in InfiniDB. This work has not made its way into ColumnStore yet and there are likely quite a few things this doesn't touch: https://github.com/infinidb/infinidb/commit/eb1e2df76bda95a43cd2948890bff98f... Kind Regards Andrew On 20/03/17 21:57, Даниил Медведев wrote:
I looked at the code (for better understanding of problem), and now I have some questions: Do I understand correctly, that now it is stored as an int64 value with a scale and we want to extend it, and store maximum 65 digits as in documentation? https://github.com/mariadb-corporation/mariadb-columnstore-engine/blob/maste...
We will use fixed size type or extandable?
I think that I will find all answers of my questions in specification, and there is no point in asking more questions.
Regards, Daniil Medvedev
2017-03-20 21:51 GMT+03:00 Andrew Hutchings <andrew@linuxjedi.co.uk <mailto:andrew@linuxjedi.co.uk>>:
Hi Daniil,
Many thanks for your interest. We haven't fully specified it yet (I'll build a full specification later this week). But I suspect the following would be involved:
* Implementation of methods to handle MariaDB's DECIMAL format * Support for a longer than 8-byte numeric column type (there is an InfiniDB tree with work for this already) * Modification of the primitives processor for the math * Modification of the function expression processor to handle the new type * Version upgrade support for DECIMAL from the current form to the new form
In the mean time please let me know if you have any questions.
Kind Regards Andrew (Senior Software Engineer, MariaDB ColumnStore)
On 19/03/17 23:09, Даниил Медведев wrote: > Hello, > > I took part in GSoC last year, and closed one of MariaDB tasks. Now I > want to continue programming with MariaDB developers and solve a new task. > I considered some of proposed tasks, and was interested in "ColumnStore: > Add full support for DECIMAL". > I think this task is doable in the given time and I might solve it. > > P.S. My github link: https://github.com/medvdanil > > Regards, > Daniil Medvedev > > > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-developers <https://launchpad.net/~maria-developers> > Post to : maria-developers@lists.launchpad.net <mailto:maria-developers@lists.launchpad.net> > Unsubscribe : https://launchpad.net/~maria-developers <https://launchpad.net/~maria-developers> > More help : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp> >
-- Andrew Hutchings - LinuxJedi - http://www.linuxjedi.co.uk/
-- Andrew Hutchings - LinuxJedi - http://www.linuxjedi.co.uk/
Hi Daniil, Just to keep you updated I have created a ticket with some basic information in it (which we have already gone though) and updated the MariaDB KB 2017 page accordingly. The ticket can be found here: https://jira.mariadb.org/browse/MCOL-641 Kind Regards Andrew On 20/03/17 22:52, Andrew Hutchings wrote:
Hi Daniil,
Yes, it is currently stored as an int64. We want to extend it but also some of the internal functionality converts it to a double (see utils/funcexp/func_math.cpp for example). Every function would need to be fully tested to make sure that it uses DECIMAL end-to-end and handles overflow scenarios correctly.
At the moment ColumnStore's integer storage can only handle fixed 1, 2, 4 and 8 byte columns. 2, 4 and 8 are used for DECIMAL. It should be somewhat easily extendible to 16 byte columns. I believe this will give us the capability to store the same amount of digits as MariaDB. If at CREATE TABLE it is determined that a DECIMAL will only require 8 bytes it should only use an 8 byte column, if it needs bigger it should use a 16 byte column.
In addition the DDL code in ColumnStore will need modifying to handle the range.
The ability to have 16 byte columns will also help us at a later date with other potential data types in the future too.
Here is the first attempt at implementing 16byte columns in InfiniDB. This work has not made its way into ColumnStore yet and there are likely quite a few things this doesn't touch:
https://github.com/infinidb/infinidb/commit/eb1e2df76bda95a43cd2948890bff98f...
Kind Regards Andrew
On 20/03/17 21:57, Даниил Медведев wrote:
I looked at the code (for better understanding of problem), and now I have some questions: Do I understand correctly, that now it is stored as an int64 value with a scale and we want to extend it, and store maximum 65 digits as in documentation? https://github.com/mariadb-corporation/mariadb-columnstore-engine/blob/maste...
We will use fixed size type or extandable?
I think that I will find all answers of my questions in specification, and there is no point in asking more questions.
Regards, Daniil Medvedev
2017-03-20 21:51 GMT+03:00 Andrew Hutchings <andrew@linuxjedi.co.uk <mailto:andrew@linuxjedi.co.uk>>:
Hi Daniil,
Many thanks for your interest. We haven't fully specified it yet (I'll build a full specification later this week). But I suspect the following would be involved:
* Implementation of methods to handle MariaDB's DECIMAL format * Support for a longer than 8-byte numeric column type (there is an InfiniDB tree with work for this already) * Modification of the primitives processor for the math * Modification of the function expression processor to handle the new type * Version upgrade support for DECIMAL from the current form to the new form
In the mean time please let me know if you have any questions.
Kind Regards Andrew (Senior Software Engineer, MariaDB ColumnStore)
On 19/03/17 23:09, Даниил Медведев wrote: > Hello, > > I took part in GSoC last year, and closed one of MariaDB tasks. Now I > want to continue programming with MariaDB developers and solve a new task. > I considered some of proposed tasks, and was interested in "ColumnStore: > Add full support for DECIMAL". > I think this task is doable in the given time and I might solve it. > > P.S. My github link: https://github.com/medvdanil > > Regards, > Daniil Medvedev > > > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-developers <https://launchpad.net/~maria-developers> > Post to : maria-developers@lists.launchpad.net <mailto:maria-developers@lists.launchpad.net> > Unsubscribe : https://launchpad.net/~maria-developers <https://launchpad.net/~maria-developers> > More help : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp> >
-- Andrew Hutchings - LinuxJedi - http://www.linuxjedi.co.uk/
-- Andrew Hutchings - LinuxJedi - http://www.linuxjedi.co.uk/
Hello, Ok, i'll write a proposal according to this description at the end of the week and will send the link to it. Regards 2017-03-27 17:07 GMT+03:00 Andrew Hutchings <andrew@linuxjedi.co.uk>:
Hi Daniil,
Just to keep you updated I have created a ticket with some basic information in it (which we have already gone though) and updated the MariaDB KB 2017 page accordingly.
The ticket can be found here: https://jira.mariadb.org/browse/MCOL-641
Kind Regards Andrew
On 20/03/17 22:52, Andrew Hutchings wrote:
Hi Daniil,
Yes, it is currently stored as an int64. We want to extend it but also some of the internal functionality converts it to a double (see utils/funcexp/func_math.cpp for example). Every function would need to be fully tested to make sure that it uses DECIMAL end-to-end and handles overflow scenarios correctly.
At the moment ColumnStore's integer storage can only handle fixed 1, 2, 4 and 8 byte columns. 2, 4 and 8 are used for DECIMAL. It should be somewhat easily extendible to 16 byte columns. I believe this will give us the capability to store the same amount of digits as MariaDB. If at CREATE TABLE it is determined that a DECIMAL will only require 8 bytes it should only use an 8 byte column, if it needs bigger it should use a 16 byte column.
In addition the DDL code in ColumnStore will need modifying to handle the range.
The ability to have 16 byte columns will also help us at a later date with other potential data types in the future too.
Here is the first attempt at implementing 16byte columns in InfiniDB. This work has not made its way into ColumnStore yet and there are likely quite a few things this doesn't touch:
https://github.com/infinidb/infinidb/commit/ eb1e2df76bda95a43cd2948890bff98f8785a805
Kind Regards Andrew
On 20/03/17 21:57, Даниил Медведев wrote:
I looked at the code (for better understanding of problem), and now I have some questions: Do I understand correctly, that now it is stored as an int64 value with a scale and we want to extend it, and store maximum 65 digits as in documentation? https://github.com/mariadb-corporation/mariadb-columnstore-engine/blob/ master/dbcon/execplan/treenode.h#L62
We will use fixed size type or extandable?
I think that I will find all answers of my questions in specification, and there is no point in asking more questions.
Regards, Daniil Medvedev
2017-03-20 21:51 GMT+03:00 Andrew Hutchings <andrew@linuxjedi.co.uk <mailto:andrew@linuxjedi.co.uk>>:
Hi Daniil,
Many thanks for your interest. We haven't fully specified it yet (I'll build a full specification later this week). But I suspect the following would be involved:
* Implementation of methods to handle MariaDB's DECIMAL format * Support for a longer than 8-byte numeric column type (there is an InfiniDB tree with work for this already) * Modification of the primitives processor for the math * Modification of the function expression processor to handle the new type * Version upgrade support for DECIMAL from the current form to the new form
In the mean time please let me know if you have any questions.
Kind Regards Andrew (Senior Software Engineer, MariaDB ColumnStore)
On 19/03/17 23:09, Даниил Медведев wrote: > Hello, > > I took part in GSoC last year, and closed one of MariaDB tasks. Now I > want to continue programming with MariaDB developers and solve a new task. > I considered some of proposed tasks, and was interested in "ColumnStore: > Add full support for DECIMAL". > I think this task is doable in the given time and I might solve it. > > P.S. My github link: https://github.com/medvdanil > > Regards, > Daniil Medvedev > > > > _______________________________________________ > Mailing list: https://launchpad.net/~maria-developers <https://launchpad.net/~maria-developers> > Post to : maria-developers@lists.launchpad.net <mailto:maria-developers@lists.launchpad.net> > Unsubscribe : https://launchpad.net/~maria-developers <https://launchpad.net/~maria-developers> > More help : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp> >
-- Andrew Hutchings - LinuxJedi - http://www.linuxjedi.co.uk/
-- Andrew Hutchings - LinuxJedi - http://www.linuxjedi.co.uk/
participants (2)
-
Andrew Hutchings -
Даниил Медведев