- discuss - lists.mariadb.org

[Maria-discuss] Some questions about mariadb-dump and mariabackup
by jkmdb＠w3.org 02 Aug '22

02 Aug '22

Hi! Sorry to bother you about this but I haven't found the answers in the doc. ** mariadb-dump Could you tell me what is the difference of calling mariadb-dump on innodb tables using --skip-lock-tables --single-transaction? >From what I read these seem to be time / performace optimizations for doing innodb tables dumps but if we omit them, we would get the same data dumped, even it takes longer. Could you confirm this? ** mariabackup We have a setup that stores the binary logs under /var/log/mysql (set up with the log_bin configuration option) and the databases themselves under /var/lib/mysql Suppose I call mariabackup to make a backup of the database as follows (some options obmitted): mariabackup --backup --target-dir=my_backup_dir Could you tell me if calling mariabackup --prepare --target-dir=my_backup_dir needs access to the binary logs stored under /var/log/mysql or if mariabackup made a copy of all that it needs in my_backup_dir? ** mariabackup and incremental backups This is basically the same question as the above. If i'm making incremental backups as follows (some options obmitted): mariabackup --backup \ --target-dir=my_backup_dir/inc1 --incremental-basedir=full_backup_dir When I do the prepare for the incremental backups, does mariabackup need access to the /var/log/mysql bin logs or do the full + each incremental backup dirs has all the info needed to do the prepare? mariabackup --prepare \ --target-dir=my_backup_dir/inc1 --incremental-basedir=full_backup_dir Many thanks in advance! --josé

3 2

[Maria-discuss] MariaDB server horribly slow on start
by Cédric Counotte 01 Aug '22

01 Aug '22

Hello all. I hope I'm at the right place to ask this question. I opened a bug here: https://jira.mariadb.org/browse/MDEV-28969, however I was told to use this mailing list. We have 4 MariaDB servers in a Galera Cluster and it happens that a server has to be restarted (be it for a crash which I have to open a bug for) or maintenance. When that happens, the restarted server is causing huge slow down on the whole cluster, and it lasts for 10 to 30 minutes at the very least! And by huge, I mean huge, we end up with 500 to 800 pending queries on all servers as you can see on attached screenshots I've attached the configuration of any server for reference in case this is the source of the issue. Any way to solve this would be greatly appreciated. Regards, 3C.

8 45

[Maria-discuss] Looking for advice where to start looking at a slow query case
by ngtech1ltd＠gmail.com 26 Jul '22

26 Jul '22

Hey, I have tried to run a local service which claims to be a YouTube alike which is named AVideo(used to be PHPTube). I loaded the service with 220k+ videos from YouTube and have used MariaDB as the DB backend. The service moved slowly as the size of the videos DB growed from 100k to 150k. I got a recommendation from the developer to use MySQL which didn’t made any sense to me. I tracked the issue to the SQL queries that the service runs. Currently I am upgrading my local servers so the DB cannot be downloaded for testing however I have seen a very interesting thing. I will first put the query here: ```sql SELECT u.*, v.*, c.iconClass, c.name as category, c.clean_name as clean_category,c.description as category_description, v.created as videoCreation, v.modified as videoModified FROM videos as v LEFT JOIN categories c ON categories_id = c.id LEFT JOIN users u ON v.users_id = u.id WHERE 2=2 AND u.status = 'a' AND v.status IN ('a','k','f') ORDER BY likes DESC LIMIT 36, 12; ``` The ANALYZE FORMAT=JSON for that query is: ```json { "query_block": { "select_id": 1, "r_loops": 1, "r_total_time_ms": 9643.807074, "filesort": { "sort_key": "v.likes desc", "r_loops": 1, "r_total_time_ms": 756.223544, "r_limit": 48, "r_used_priority_queue": true, "r_output_rows": 49, "r_sort_mode": "sort_key,rowid", "temporary_table": { "table": { "table_name": "u", "access_type": "ref", "possible_keys": ["PRIMARY", "users_status_IDX"], "key": "users_status_IDX", "key_length": "1", "used_key_parts": ["status"], "ref": ["const"], "r_loops": 1, "rows": 1, "r_rows": 1, "r_table_time_ms": 0.034852389, "r_other_time_ms": 0.020715696, "filtered": 100, "r_filtered": 100, "index_condition": "u.`status` = 'a'" }, "table": { "table_name": "v", "access_type": "ref", "possible_keys": [ "fk_videos_users_idx", "video_status_idx", "videos_status_index" ], "key": "fk_videos_users_idx", "key_length": "4", "used_key_parts": ["users_id"], "ref": ["AVideo.u.id"], "r_loops": 1, "rows": 391299, "r_rows": 990067, "r_table_time_ms": 2691.791742, "r_other_time_ms": 6129.887127, "filtered": 50.00025558, "r_filtered": 100, "attached_condition": "v.`status` in ('a','k','f')" }, "table": { "table_name": "c", "access_type": "eq_ref", "possible_keys": ["PRIMARY"], "key": "PRIMARY", "key_length": "4", "used_key_parts": ["id"], "ref": ["AVideo.v.categories_id"], "r_loops": 990067, "rows": 1, "r_rows": 1, "r_table_time_ms": 0.00546745, "r_other_time_ms": 0.569425897, "filtered": 100, "r_filtered": 100 } } } } } ``` While for the next: ```sql SELECT * FROM (SELECT u.id as uid , u.user as uuser , u.name as uname , u.email as uemail , u.password as upassword , u.created as ucreated , u.modified as umodified , u.isAdmin as uisAdmin , u.status as ustatus , u.photoURL as uphotoURL , u.lastLogin as ulastLogin , u.recoverPass as urecoverPass , u.backgroundURL as ubackgroundURL , u.canStream as ucanStream , u.canUpload as ucanUpload , u.canCreateMeet as ucanCreateMeet , u.canViewChart as ucanViewChart , u.about as uabout , u.channelName as uchannelName , u.emailVerified as uemailVerified , u.analyticsCode as uanalyticsCode , u.externalOptions as uexternalOptions , u.first_name as ufirst_name , u.last_name as ulast_name , u.address as uaddress , u.zip_code as uzip_code , u.country as ucountry , u.region as uregion , u.city as ucity , u.donationLink as udonationLink , u.extra_info as uextra_info , u.phone as uphone , u.is_company as uis_company , v.*, c.iconClass, c.name as category, c.clean_name as clean_category,c.description as category_description, v.created as videoCreation, v.modified as videoModified FROM videos as v LEFT JOIN categories c ON categories_id = c.id LEFT JOIN users u ON v.users_id = u.id WHERE 2=2 AND v.status IN ('a','k','f') ORDER BY likes DESC ) a WHERE a.ustatus IN ('a') LIMIT 1000, 900; ``` The ANALYZE FORMAT=JSON is the next: ```json { "query_block": { "select_id": 1, "r_loops": 1, "r_total_time_ms": 8.047324959, "table": { "table_name": "u", "access_type": "ref", "possible_keys": ["PRIMARY", "users_status_IDX"], "key": "users_status_IDX", "key_length": "1", "used_key_parts": ["status"], "ref": ["const"], "r_loops": 1, "rows": 1, "r_rows": 1, "r_table_time_ms": 0.045761237, "r_other_time_ms": 0.0201165, "filtered": 100, "r_filtered": 100, "index_condition": "u.`status` = 'a'" }, "table": { "table_name": "v", "access_type": "ref", "possible_keys": [ "fk_videos_users_idx", "video_status_idx", "videos_status_index" ], "key": "fk_videos_users_idx", "key_length": "4", "used_key_parts": ["users_id"], "ref": ["AVideo.u.id"], "r_loops": 1, "rows": 391299, "r_rows": 1900, "r_table_time_ms": 6.939235389, "r_other_time_ms": 1.010973728, "filtered": 50.00025558, "r_filtered": 100, "attached_condition": "v.`status` in ('a','k','f')" }, "table": { "table_name": "c", "access_type": "eq_ref", "possible_keys": ["PRIMARY"], "key": "PRIMARY", "key_length": "4", "used_key_parts": ["id"], "ref": ["AVideo.v.categories_id"], "r_loops": 1900, "rows": 1, "r_rows": 1, "r_table_time_ms": 0.007669714, "r_other_time_ms": 0.001959112, "filtered": 100, "r_filtered": 100 } } } ``` And what I am trying to understand is, what's the difference? Why would the same exact query will result with this issue? Just to notice that the next query will not run at all due to fields naming conflict between u and v: ```sql SELECT * FROM (SELECT u.* , v.*, c.iconClass, c.name as category, c.clean_name as clean_category,c.description as category_description, v.created as videoCreation, v.modified as videoModified FROM videos as v LEFT JOIN categories c ON categories_id = c.id LEFT JOIN users u ON v.users_id = u.id WHERE 2=2 AND v.status IN ('a','k','f') ORDER BY likes DESC ) a WHERE a.ustatus IN ('a') LIMIT 1000, 900; ``` I am not new to the DB world that much but it's the first time I am getting my hands dirty about such an issue. I will share the full DB later on today so it could be used as a learning material. Also, are there any big public databases that can be used for learning and testing purposes? Thanks, Eliezer ---- Eliezer Croitoru NgTech, Tech Support Mobile: +972-5-28704261 Email: ngtech1ltd(a)gmail.com Web: https://ngtech.co.il/ My-Tube: https://tube.ngtech.co.il/

3 10

[Maria-discuss] mariadb-connector-cpp and multiple threads
by jason＠youtorque.com 25 Jul '22

25 Jul '22

Using the [ MariaDB C++ Connector ]( https://github.com/mariadb-corporation/mariadb-connector-cpp ) on ubuntu 20.04, and gcc compiler. During load test of a webserver application it was discovered that anytime there was more than a single concurrent connection the server would crash. This was narrowed down to the connect member of the mariadb driver. The errors that were being reported were always one of the following two: terminate called after throwing an instance of 'std::logic_error' what(): basic_string::_M_construct null not validAborted (core dumped)terminate called after throwing an instance of 'std::invalid_argument' what(): stoiAborted (core dumped) As it wouldn't be feasible to post the source of the server, I have reproduced a minimal working example that uses only standard library and mariadb/conncpp.hpp: ``` #include <mariadb/conncpp.hpp>#include <iostream>#include <thread>#include <string>sql::Driver* driver = sql::mariadb::get_driver_instance();sql::SQLString connectionUrl = "jdbc:mariadb://localhost:3306/the_database_name";sql::Properties properties = {{"user", "the_username"}, {"password", "the_password"}};int main(){ // simulate 2 concurrent connections each making 500 requests std::vector<std::function<void()>> t1cbs; std::vector<std::function<void()>> t2cbs; for(int i = 0; i < 500; i++) // if does not fail for you at 500 up the number until it does { t1cbs.push_back([&]{ std::unique_ptr<sql::Connection> conn(driver->connect(connectionUrl, properties)); std::cout << "t1:" << conn->getHostname().c_str() << std::endl; }); // comment out this block to keep the second thread from executing, and you will see // that no errors occur t2cbs.push_back([&]{ std::unique_ptr<sql::Connection> conn(driver->connect(connectionUrl, properties)); std::cout << "t2:" << conn->getHostname().c_str() << std::endl; }); } std::thread t1([&]{ for(auto& cb : t1cbs) cb(); }); std::thread t2([&]{ for(auto& cb : t2cbs) cb(); }); t1.join(); t2.join(); return 0;} ``` I'm guessing the issue is on my end as I've just started working with this particular library and have probably missed or misunderstood something. Would someone be able to point me in the right direction as far as how I might go about creating database connections from within separate threads?

2 3

Re: [Maria-discuss] mysql_upgrade fails
by Majed Zouhairy 21 Jul '22

21 Jul '22

on centos 8, upgrade form mraidb 10.3 to 10.6 fails with: sudo mysql_upgrade -u root -p Enter password: Major version upgrade detected from 10.3.17-MariaDB to 10.6.8-MariaDB. Check required! Phase 1/7: Checking and upgrading mysql database Processing databases mysql mysql.column_stats OK mysql.columns_priv OK mysql.db OK mysql.event OK mysql.func OK mysql.help_category OK mysql.help_keyword OK mysql.help_relation OK mysql.help_topic OK mysql.host OK mysql.index_stats OK mysql.plugin OK mysql.proc OK mysql.procs_priv OK mysql.proxies_priv OK mysql.roles_mapping OK mysql.servers OK mysql.table_stats OK mysql.tables_priv OK mysql.time_zone OK mysql.time_zone_leap_second OK mysql.time_zone_name OK mysql.time_zone_transition OK mysql.time_zone_transition_type OK mysql.user OK Phase 2/7: Installing used storage engines... Skipped Phase 3/7: Fixing views Phase 4/7: Running 'mysql_fix_privilege_tables' ERROR 1005 (HY000) at line 694: Can't create table `mysql`.`innodb_table_stats` (errno: 184 "Tablespace already exists") ERROR 1005 (HY000) at line 698: Can't create table `mysql`.`innodb_index_stats` (errno: 184 "Tablespace already exists") ERROR 1005 (HY000) at line 702: Can't create table `mysql`.`transaction_registry` (errno: 184 "Tablespace already exists") ERROR 1005 (HY000) at line 774: Can't create table `mysql`.`gtid_slave_pos` (errno: 184 "Tablespace already exists") FATAL ERROR: Upgrade failed ________________________________ From: Vicențiu Ciorbaru <cvicentiu(a)gmail.com> Sent: Wednesday, April 21, 2021 3:51 PM To: Majed Zouhairy <majedzouhairy(a)hotmail.com> Cc: maria-discuss(a)lists.launchpad.net <maria-discuss(a)lists.launchpad.net> Subject: Re: [Maria-discuss] mysql_upgrade fails Hi Majed, What MariaDB version is this? What is the previous version and what is the current version you are upgrading to? Is this from a linux distro, or where did you get the MariaDB binary from? If it's a distro, which one? Vicențiu On Wed, 21 Apr 2021 at 15:31, Majed Zouhairy <majedzouhairy(a)hotmail.com<mailto:majedzouhairy@hotmail.com>> wrote: in last time's upgrade and today i got: sudo mysql_upgrade -u root -p Enter password: Phase 1/7: Checking and upgrading mysql database Processing databases mysql mysql.column_stats OK mysql.columns_priv OK mysql.db OK mysql.event OK mysql.func OK mysql.help_category OK mysql.help_keyword OK mysql.help_relation OK mysql.help_topic OK mysql.host OK mysql.index_stats OK mysql.plugin OK mysql.proc OK mysql.procs_priv OK mysql.proxies_priv OK mysql.roles_mapping OK mysql.servers OK mysql.table_stats OK mysql.tables_priv OK mysql.time_zone OK mysql.time_zone_leap_second OK mysql.time_zone_name OK mysql.time_zone_transition OK mysql.time_zone_transition_type OK mysql.user OK Phase 2/7: Installing used storage engines... Skipped Phase 3/7: Fixing views Phase 4/7: Running 'mysql_fix_privilege_tables' ERROR 1813 (HY000) at line 77: Tablespace for table '`mysql`.`innodb_table_stats`' exists. Please DISCARD the tablespace before IMPORT ERROR 1813 (HY000) at line 81: Tablespace for table '`mysql`.`innodb_index_stats`' exists. Please DISCARD the tablespace before IMPORT ERROR 1813 (HY000) at line 85: Tablespace for table '`mysql`.`transaction_registry`' exists. Please DISCARD the tablespace before IMPORT ERROR 1813 (HY000) at line 153: Tablespace for table '`mysql`.`gtid_slave_pos`' exists. Please DISCARD the tablespace before IMPORT ERROR 1146 (42S02) at line 642: Table 'mysql.innodb_index_stats' doesn't exist ERROR 1243 (HY000) at line 643: Unknown prepared statement handler (stmt) given to EXECUTE ERROR 1146 (42S02) at line 645: Table 'mysql.innodb_table_stats' doesn't exist ERROR 1243 (HY000) at line 646: Unknown prepared statement handler (stmt) given to EXECUTE ERROR 1146 (42S02) at line 650: Table 'mysql.innodb_index_stats' doesn't exist ERROR 1146 (42S02) at line 654: Table 'mysql.innodb_table_stats' doesn't exist ERROR 1146 (42S02) at line 657: Table 'mysql.innodb_table_stats' doesn't exist FATAL ERROR: Upgrade failed what to do? _______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss(a)lists.launchpad.net<mailto:maria-discuss@lists.launchpad.net> Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp

3 8

[Maria-discuss] Speeding up MyRocks engine
by Jan Křístek 24 Jun '22

24 Jun '22

Hello, I have already given up on MyRocks but could someone give some hint to take back my decision?. I have a setup with a heavy insert load with potentially long records (up to a kilobyte or two). I cannot get more than about 300-400 inserts per second with MyRocks. With the Aria engine, it's reaching up to 1900 inserts/sec with the same setup. The only config changes for MyRocks are: rocksdb-override-cf-options='cf1={compression=kLZ4Compression;bottommost_compression=kLZ4Compression;}' rocksdb_allow_concurrent_memtable_write = ON rocksdb_default_cf_options=block_based_table_factory={cache_index_and_filter_blocks=1;filter_policy=bloomfilter:10:false;whole_key_filtering=1};level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true;compaction_pri=kMinOverlappingRatio;compression=kLZ4Compression plugin-load-add=ha_rocksdb.so Both MyRocks and Aria are the same hardware, the same setup SSD with software Raid0 and LVM. With Aria I have switched on aria_group_commit to "soft" with 500 ms delay, as I understand, I might lose some data if the server is hard killed or crashes. I am willing to take the risk unless s/o suggests a major speedup for the MyRocks. I have switched TRANSACTIONAL=1 for the table. The table has partitions on both Aria and MyRocks. Thank you for any help in advance, Jan

1 0

[Maria-discuss] Change delimiter with server_audit plugin
by Philippe Maladjian 22 Jun '22

22 Jun '22

Hi there, In order to monitor a database and especially the requests that could be in error, I activated the server_audit plugin. Plugin setup: show global VARIABLES like '%server_audit%'; +-------------------------------+---------------------------------+ | Variable_name | Value | +-------------------------------+---------------------------------+ | server_audit_events | QUERY | | server_audit_excl_users | user | | server_audit_file_path | /var/log/mysql/server_audti.log | | server_audit_file_rotate_now | OFF | | server_audit_file_rotate_size | 1000000 | | server_audit_file_rotations | 9 | | server_audit_incl_users | user | | server_audit_logging | ON | | server_audit_mode | 0 | | server_audit_output_type | file | | server_audit_query_log_limit | 1024 | | server_audit_syslog_facility | LOG_USER | | server_audit_syslog_ident | mysql-server_auditing | | server_audit_syslog_info | | | server_audit_syslog_priority | LOG_ERR | +-------------------------------+---------------------------------+ My problem is that the separator between each value is the comma and bad luck it's the same separator as in the query. Example of a line: 20220622 08:35:03,sserver,user,localhost,14858,7002875,QUERY,database,'UPDATE sorderq SET `odlqty` = "6", `soqsta` = "3", `sdhnum` = "LIVHIL22062", `sddlin` = 24000 WHERE `sohnum` = "NORHIL220601" AND `soplin` = 24000 AND `soqseq` = 24000',0 [...] 20220622 08:35:03,server,user,localhost,14858,7002878,QUERY,database,'INSERT INTO sdeliveryd(`sdhnum`, `itmref`, `sddlin`, `netprinot`, `qty`, `rtnqty`, `tsicod0`, `tsicod1`, `tsicod2`, `tsicod3`, `tsicod4`, `rep1`) VALUES("LIVHIL22061", "1-11325", 26000, 0.0, "12", "0", "STA", "05", " ", " ", " ", "093")',0 It is the last field that interests me because it is the return code of the request. Now if I want to automate the error processing (last field <> 0) I simply cannot because due to the identical separator between the fields of the request and the fields of the log line, the number of columns varies and the return code never has the same position. Is it possible to change the field separator in the plugin? Thanks in advance

1 0

[Maria-discuss] debug.csr error?
by Dave Pawson 19 Jun '22

19 Jun '22

Using create table. error shown > python createRItbl.py Debug.csr Traceback (most recent call last): File "/home/dpawson/Dropbox/2022/sibc/software/mariadb/createRItbl.py", line 82, in <module> showcsr(csr) File "/home/dpawson/Dropbox/2022/sibc/software/mariadb/createRItbl.py", line 40, in showcsr for item in csr: File "/home/dpawson/.local/lib/python3.10/site-packages/mysql/connector/cursor_cext.py", line 718, in fetchone return self._fetch_row() File "/home/dpawson/.local/lib/python3.10/site-packages/mysql/connector/cursor_cext.py", line 689, in _fetch_row row = self._rows[self._next_row] TypeError: 'NoneType' object is not subscriptable Called from Python. Table= f"CREATE TABLE IF NOT EXISTS rinkincome (\ matchdate DATE ,\ session ENUM('rollup','league','club match') ,\ code ENUM {costcodes} ,\ cashOrCard ENUM('card','cash'),\ amount DECIMAL(5,2) );" where costcodes=('a','b') form. # print Table out to check syntax #print(Table) #sys.exit(0) csr.execute(Table) showcsr(csr) Nothing shown to indicate which list / structure I'm trying to iterate over? Any suggestions to chase down the fault please. Pretty new to mariadb. regards -- Dave Pawson XSLT XSL-FO FAQ. Docbook FAQ.

1 0

[Maria-discuss] Incorrect Documentation: Replicating JSON from MySQL to MariaDB
by Gordan Bobic 29 May '22

29 May '22

On this page: https://mariadb.com/kb/en/json-data-type/ it states: If you must use row-based replication and cannot change the MySQL master from JSON to TEXT, you can try to introduce an intermediate MySQL slave and change the column type from JSON to TEXT on it. Then you replicate from this intermediate slave to MariaDB. I can confirm that this does not work with MySQL 5.7. If you have a 5.7 master and you replicate to 5.7 slave, and you convert the slave's json columns to longtext, replication fails with this error: "Column X of table 'dbname.tablename' cannot be converted from type 'json' to type 'longtext'" Using longblob doesn't work either. So it seems there is currently no viable solution for replicating anything that involves JSON columns from MySQL to MariaDB.

2 2

[Maria-discuss] Custom Galera SST Method Documentation
by Gordan Bobic 27 May '22

27 May '22

Hi, I vaguely remember reading something years ago about implementing a custom SST method, but I cannot seem to find any documentation for it now. Is there any documentation on the interfaces that scripts need to implement and how to hook up said scripts into the config files?

2 2