Hi Varun, On Mon, Apr 06, 2020 at 01:30:38PM +0530, Varun wrote:
revision-id: 38d7dc6171cba4d5d3cdadd424f1d4b6a2d43f75 (mariadb-10.5.0-571-g38d7dc6171c) parent(s): 139117528affc89b6e174231b86048ec2b03f686 author: Varun Gupta committer: Varun Gupta timestamp: 2020-04-06 13:30:17 +0530 message:
MDEV-21946: Server crash in store_length upon GROUP BY WITH ROLLUP with geometry field
overflow happening when we add suffix length to sortlength. Make sure that sortlength is at max UINT_MAX32
diff --git a/sql/filesort.cc b/sql/filesort.cc index 1f491df82eb..9b75d088289 100644 --- a/sql/filesort.cc +++ b/sql/filesort.cc @@ -2105,6 +2105,7 @@ Type_handler_string_result::sort_length(THD *thd, { CHARSET_INFO *cs; sortorder->length= item->max_length; + set_if_smaller(sortorder->length, thd->variables.max_sort_length); sortorder->original_length= item->max_length;
if (use_strnxfrm((cs= item->collation.collation))) @@ -2114,9 +2115,13 @@ Type_handler_string_result::sort_length(THD *thd, else if (cs == &my_charset_bin) { /* Store length last to be able to sort blob/varbinary */ - sortorder->suffix_length= suffix_length(sortorder->length); + sortorder->suffix_length= suffix_length(item->max_length); + DBUG_ASSERT(sortorder->length <= UINT_MAX32 - sortorder->suffix_length); sortorder->length+= sortorder->suffix_length; - sortorder->original_length+= sortorder->suffix_length; + if (sortorder->original_length >= UINT_MAX32 - sortorder->suffix_length) + sortorder->original_length= UINT_MAX32; + else + sortorder->original_length+= sortorder->suffix_length;
I'm wondering, why should original_length include suffix_length? The sort key has this sort suffix, but the original value does not? I tried removing the "else ..." part and it didn't seem to make any difference. I know that the line
- sortorder->original_length+= sortorder->suffix_length;
was there before this patch, but since the patch touches it, lets figure this out. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog