Re: [Maria-developers] 9243921c84b: Make all #sql temporary table names uniform
Hi, Michael! On Apr 13, Michael Widenius wrote:
revision-id: 9243921c84b (mariadb-10.5.2-128-g9243921c84b) parent(s): 8742d176bc2 author: Michael Widenius <monty@mariadb.com> committer: Michael Widenius <monty@mariadb.com> timestamp: 2020-04-09 22:02:06 +0300 message:
Make all #sql temporary table names uniform
The reason for this is to make all temporary file names similar and also to be able to figure out from where a #sql-xxx name orginates.
New format is for most cases: #sql-name-current_pid-thread_id[-increment] Where name is one of subselect, alter, exchange, temptable or backup
The execptions are:
typo, "exceptions"
ALTER PARTITION shadow files: #sql-shadow-'original_table_name'
Please, add a thread_id here at the end. normally MDL should ensure that no two threads can have a shadow for the same table at the same time, but we have enough bugs as it is to introduce another vector when two threads can overwrite each other temp files.
Names from the temp pool: #sql-name-current_pid-pool_number
Would you mind if I drop temp pool completely in 10.5? It was added by Jeremy in January 2001 with the comment Added --temp-pool option to mysqld. This will cause temporary files created to use a small set of filenames, to try and avoid problems in the Linux kernel. And I doubt it's still an issue in 2020. Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
Hi! <cut>
ALTER PARTITION shadow files: #sql-shadow-'original_table_name'
Please, add a thread_id here at the end. normally MDL should ensure that no two threads can have a shadow for the same table at the same time, but we have enough bugs as it is to introduce another vector when two threads can overwrite each other temp files.
That would make the file name even longer and I am not sure what happens if we table names goes much over NAME_LEN. I don't have time just now to check for possible name overruns (I think it should be save as most engines are using FN_REFLEN, but better safe than sorry.
Names from the temp pool: #sql-name-current_pid-pool_number
Would you mind if I drop temp pool completely in 10.5? It was added by Jeremy in January 2001 with the comment
Added --temp-pool option to mysqld. This will cause temporary files created to use a small set of filenames, to try and avoid problems in the Linux kernel.
And I doubt it's still an issue in 2020.
I think that reusing file names is still something that is a problem. We should do some tests for that before skipping this code. It's not a lot a code after all. Regard, Monty
Hi, Michael! On Apr 16, Michael Widenius wrote:
ALTER PARTITION shadow files: #sql-shadow-'original_table_name'
Please, add a thread_id here at the end. normally MDL should ensure that no two threads can have a shadow for the same table at the same time, but we have enough bugs as it is to introduce another vector when two threads can overwrite each other temp files.
That would make the file name even longer and I am not sure what happens if we table names goes much over NAME_LEN. I don't have time just now to check for possible name overruns (I think it should be save as most engines are using FN_REFLEN, but better safe than sorry.
Well, I'm certain that absolutely all temporary tables must have unique names. If you're concerned about name overruns use the pattern #sql-shadow-<thread_id>-<original_name> then any truncation will only affect the "original_name" part and it'll be fine. Regards, Sergei VP of MariaDB Server Engineering and security@mariadb.org
Monty, On Thu, Apr 16, 2020 at 3:32 PM Michael Widenius <michael.widenius@gmail.com> wrote:
ALTER PARTITION shadow files: #sql-shadow-'original_table_name'
Please, add a thread_id here at the end. normally MDL should ensure that no two threads can have a shadow for the same table at the same time, but we have enough bugs as it is to introduce another vector when two threads can overwrite each other temp files.
That would make the file name even longer and I am not sure what happens if we table names goes much over NAME_LEN. I don't have time just now to check for possible name overruns (I think it should be save as most engines are using FN_REFLEN, but better safe than sorry.
FN_REFLEN limits the path name to only 511 or 512 bytes. When I started using GNU/Linux in 1993, the maximum path name length was 4095 or 4096 bytes. Already back then, extfs or xiafs or something supported up to 255 bytes per path name component (name in a directory). The sad matter of fact is that with the filename-safe encoding (my_charset_filename) of MySQL 5.1, we can use up to 5 ASCII bytes per character. We can already easily blow the 255-byte limit of ext4fs or similar by creating a table, partition, and subpartition name consisting of (say) 64 $ characters (which will be encoded as @0024). The complete filename would be something like @0024#P#@0024#SP#@0024.frm (add more sequences of @0024). This could have been made a little worse in MySQL 5.1. In the old days (4.1 and 5.0), AFAIU we stored the file name in UTF-8 encoding (up to 3*64=192 bytes for database name, and 3*3*64+some bytes for table,partition,subpartition name). Also that would easily exceed the 255-byte path component limit of the Linux extfs and its successors. Side note: Windows NTFS uses UTF-16 encoding for file names and can support up to 32768 characters per path component when using some extension. But the selection of disallowed characters in file names is larger than the *nix one ('/' and NUL). So, the reasons for introducing my_charset_filename might not have gone completely away. Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation
Hi! On Fri, 17 Apr 2020, 15:52 Marko Mäkelä, <marko.makela@mariadb.com> wrote:
Monty,
ALTER PARTITION shadow files: #sql-shadow-'original_table_name'
Please, add a thread_id here at the end. normally MDL should ensure
On Thu, Apr 16, 2020 at 3:32 PM Michael Widenius <michael.widenius@gmail.com> wrote: that
no two threads can have a shadow for the same table at the same time, but we have enough bugs as it is to introduce another vector when two threads can overwrite each other temp files.
That would make the file name even longer and I am not sure what happens if we table names goes much over NAME_LEN. I don't have time just now to check for possible name overruns (I think it should be save as most engines are using FN_REFLEN, but better safe than sorry.
FN_REFLEN limits the path name to only 511 or 512 bytes. When I started using GNU/Linux in 1993, the maximum path name length was 4095 or 4096 bytes. Already back then, extfs or xiafs or something supported up to 255 bytes per path name component (name in a directory).
I understand the problem, but not the suggested solution? We can't just increase FN_REFLEN as it will increase stack and memory usage. And increasing it will not help when the filename part becomes too long. In reality this is more a theoretical (or hacker security problem) as long table names are not common. Don't remember a single real bug report about this. Regards, Monty
participants (3)
-
Marko Mäkelä -
Michael Widenius -
Sergei Golubchik