May 2015 - discuss - lists.mariadb.org

[Maria-discuss] InnoDB: slow insert (too many fsyncs ?)
by Julien Muchembled 15 Jun '15

15 Jun '15

Hello, I am trying to optimize the insertion of 124 GB of data into InnoDB and it takes much more time than I expected. For the moment this is only a test before migrating prod, using a USB HDD. Prod has much faster storage but I prefer to check if there's anything wrong in my MariaDB setup. The source DB is read as less than 1MB/s with less than 20% CPU usage (<10% for mysqld and <10% for the client app doing the migration). I know that several unsafe settings can be used to import data but I won't be able to use them for the migration of prod. This is because the migration will be done transparently in background, the old db being accessed for not yet migrated data. For the same reason, indexes can't be disabled. Anyway, it was slow that for the moment, I only did a full test with unsafe settings: innodb_flush_log_at_trx_commit = 0 innodb_flush_method = nosync innodb_doublewrite = 0 sync_frm = 0 It took exactly 19.8 hours to import all the data. See attached file for all parameters and detailed statistics. (Sorry for the 'show status' outputs, no idea what I expected to get when I tried 'flush status', whereas 21G were already processed. So 'show status.20941778944' is the output just before the flush, and 'show status' is after everything was finished) The schema of tables can be found here: http://git.erp5.org/gitweb/neoppod.git/blob/HEAD:/neo/storage/database/mysq… 1. fsync Not sure if I read 'show status' correctly, but I'd say it's actually 6 fsyncs per second, and this matches what I see with strace. Why not 0 whereas I disabled them ? There's also less than 1 COMMIT per second, so even without disabling fsyncs I was surprised to see so many in the middle of transactions. 2. Write amplification During my first tests, I saw with iotop that mysqld wrote at 10MB/s. Which means a factor 10. With doublewrite disabled, it's rather a factor 5. These ratios are probably underestimated. For better interpretation of the statistics, I should add that: - source DB is uncompressed but the file format is very compact - most of the imported data ends up in the 'data.value' column, and most values are zlib-compressed, with also deduplication The resulting ibdata is only 70GB. Is this write amplification only caused by indexes & doublewrite ? Thanks, Julien

3 3

Re: [Maria-discuss] Dynamic Columns ORM
by Tom Worster 23 May '15

23 May '15

Hi Daniel, Yes, I shall add remarks about indexes to the docs. I didn't have any purpose for "token" other than I needed a name for the strings that go in the SQL that refer to dynamic columns. Is there a problem with that particular word or do you think it would be better to do without any such name? You're spot on about computed columns. From my point of view, doing that amounts to adding the column to the schema. The column name appears in DDL and the corresponding attribute is therefore no longer ad hoc in the the application. The same point applies (in some imaginary future) to adding an index to a dynamic column without needing a computed column. If attributes represented in the application are to be truly fluid then their corresponding dynamic column cannot be indexed, at least not as I understand indexes in MySQL. Thanks for taking a look. Tom From: Daniel Black <daniel.black(a)openquery.com.au> Date: Saturday, May 23, 2015 at 1:27 AM To: Tom Worster <fsb(a)thefsb.org>, maria-discuss email list <maria-discuss(a)lists.launchpad.net> Subject: Re: [Maria-discuss] Dynamic Columns ORM They are generally good for blob like information that is generally retrieved in a query with some structure around name->value pairs. You mentioned (forum) that there isn't an index on these items so you'll get more row scanning on these if people search on a Dynamic attribute tokens (not sure why you are adding "token" to the language here). This limitation to be mentioned more in the documentation you've written. You could use computed columns to extract a variable to index if really needed but you'll need to ask why isn't it a column in its own right. https://mariadb.com/kb/en/mariadb/virtual-computed-columns/ Enhancing frameworks to expose database features is a good thing as they often get omitted in the reduction to the lowest common factor (mysql-5.0?) and sometimes the lowest common factor across db implementations (ansi SQL(?)). -- -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.

2 1

[Maria-discuss] Dynamic Columns ORM
by Tom Worster 23 May '15

23 May '15

In January I was asking about Dynamic Columns because I had started working on an ORM extension for it. It is now tagged 0.1.0 in Github. There probably aren't many here using Yii Framework but at least this shows that some of us like and use Maria's Dynamic Columns. Thank you, Maria. I already got some feedback on IRC from Kolbe who hated it :'( and we'd welcome more of the same :-) https://github.com/tom--/dynamic-ar Yii Framework forum thread: http://www.yiiframework.com/forum/index.php/topic/63995-/ Previous thread started here: https://lists.launchpad.net/maria-discuss/msg02159.html tom[] The empty array without a dollar sign

2 1

[Maria-discuss] mysqldump Spider tables
by Nicolas Payart 21 May '15

21 May '15

Hi list, I am using Spider Storage Engine for multiple purpose. I often create spider tables as a drop-in replacement for Federated(x) tables. But one main difference I notice using spider SE vs Federated is that mysqldump dumps the whole data of remote spider tables whereas it only dumps the table schema of federated tables. I expected these two close storage engines to behave similarly... Is this an expected behaviour of Spider SE (and/or mysqldump)? Is there a workaround to make mysqldump dump only schema of Spider tables, just like it would do with a federated table? Of course, I can use the --no-data option but it's a bit annoying when you have a mix of innodb tables and spider tables in the same database. Thanks, -- Nicolas

2 2

[Maria-discuss] Query Cache with TTL and unconsistent values - remove memcache servers
by Roberto Spadim 20 May '15

20 May '15

Hi guys! I created a MDEV (feature request) sometime ago, https://mariadb.atlassian.net/browse/MDEV-6777 There's a patch ( http://ebergen.net/wordpress/2007/04/23/select-sql_cache_ttl/) but i want change a bit to work "better", it's using "SELECT SQL_CACHE_TTL ..." hint, and could be better change it to "SELECT SQL_CACHE_TTL=9999 ..." to easily change TTL instead of a fixed TTL set by "SET ttl=999;SELECT SQL_CACHE_TTL ..." query, could anyone help me with it? Why? With this we can reduce the memcache use in small/medium systems, reducing app side caching code too the difference between query cache and query cache with ttl is how it's removed from cache, one is removed with table invalidation (consistent cache), and other is removed with time to live even if table change (unconsistent cache) =] help is welcome, thnks guys -- Roberto Spadim

1 0

[Maria-discuss] MariaDB 10.1.4 Progress output
by Rhys Campbell 19 May '15

19 May '15

Note the progress output below. I've seen several like this. i.e... Stage: 1 of 72 'altering table' 350% of stage done. Is this intentional? [cid:image001.png@01D07B7A.526BD6D0] Rhys Campbell Database Administrator TradingScreen, Inc. 23 York House, 5th Floor London WC2B 6UJ Email: rhys.campbell(a)tradingscreen.com<mailto:rhys.campbell@tradingscreen.com> Follow TradingScreen on Twitter<http://twitter.com/#!/TradingScreen> , Facebook<http://www.facebook.com/pages/TradingScreen/214046251945650> and our blog Trading Smarter<tradingsmarter.tradingscreen.com> This message is intended only for the recipient(s) named above and may contain confidential information. If you are not an intended recipient, you should not review, distribute or copy this message. Please notify the sender immediately by e-mail if you have received this message in error and delete it from your system.

5 5

[Maria-discuss] MariaDB Galera Cluster 10.0.19 and 5.5.43 now available
by Daniel Bartholomew 18 May '15

18 May '15

The MariaDB project is pleased to announce the immediate availability of MariaDB Galera Cluster 5.5.43 and 10.0.19. These are Stable (GA) releases. See the Release Notes and Changelogs for details. - - Links - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - MariaDB Galera Cluster 10.0.19 - Release Notes: https://mariadb.com/kb/en/mariadb-galera-cluster-10019-release-notes/ - Changelog: https://mariadb.com/kb/en/mariadb-galera-cluster-10019-changelog/ - Downloads: https://downloads.mariadb.org/mariadb-galera/10.0.19/ MariaDB Galera Cluster 5.5.43 - Release Notes: https://mariadb.com/kb/en/mariadb-galera-cluster-5543-release-notes/ - Changelog: https://mariadb.com/kb/en/mariadb-galera-cluster-5543-changelog/ - Downloads: https://downloads.mariadb.org/mariadb-galera/5.5.43/ About MariaDB Galera Cluster: - https://mariadb.com/kb/en/what-is-mariadb-galera-cluster/ APT and YUM Repository Configuration Generator: - https://downloads.mariadb.org/mariadb/repositories/ - - MariaDB Webinars - - - - - - - - - - - - - - - - - - - - - - - - - There are several upcoming MariaDB-focused webinars and many previously held ones available to watch on an on-demand basis at: - https://mariadb.com/news-events/webinars - - MariaDB Books - - - - - - - - - - - - - - - - - - - - - - - - - - There is an ever-growing library of MariaDB books available to help you get the most out of MariaDB. See the MariaDB Books page for details and links: - https://mariadb.com/kb/en/mariadb/books/ - - User Feedback plugin - - - - - - - - - - - - - - - - - - - - - - - MariaDB includes a User Feedback plugin. This plugin is disabled by default. If enabled, it submits basic, completely anonymous MariaDB usage information. This information is used by the developers to track trends in MariaDB usage to better guide development efforts. If you would like to help make MariaDB better, please add "feedback=ON" to your my.cnf or my.ini file! See http://mariadb.com/kb/en/user-feedback-plugin for more information. - - Quality - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - The project always strives for quality, but in reality, nothing is perfect. Please take time to report any issues you encounter at: - http://mariadb.org/jira - - Support MariaDB - - - - - - - - - - - - - - - - - - - - - - - - - - If you would like to contribute to the MariaDB Foundation, please see the "contributing" and "donations" pages. We also have merchandise available in a cafepress store. All proceeds go to support the MariaDB Foundation. - https://mariadb.com/kb/en/contributing - https://mariadb.org/en/donate/ We hope you enjoy MariaDB! -- MariaDB Website - http://mariadb.org Twitter - http://twitter.com/mariadb Google+ - http://google.com/+mariadb Facebook - http://fb.com/MariaDB.dbms Knowledge Base - http://mariadb.com/kb

1 0

[Maria-discuss] query cache dump_at_shutdown/load_at_startup useful + no character set/collation?
by Daniel Black 17 May '15

17 May '15

I was testing out using the query_cache_info plugin (https://mariadb.com/kb/en/mariadb/query_cache_info-plugin/) to pre-cache data into a server after an upgrade. The read only slaves have a fairly high cache hit ratio. select concat('use ', STATEMENT_SCHEMA, ';', STATEMENT_TEXT, ';') as q INTO OUTFILE '/tmp/qc.txt' FIELDS ESCAPED BY '' TERMINATED BY '' from information_schema.QUERY_CACHE_INFO; Query OK, 7122 rows affected (0.03 sec) When loading these it seems the statements have a collation however this isn't stored in the query cache. $ mysql --force < /tmp/qc.txt > /dev/null ERROR 1267 (HY000) at line 119: Illegal mix of collations (latin1_bin,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '=' ERROR 1267 (HY000) at line 2390: Illegal mix of collations (latin1_bin,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '=' ERROR 1267 (HY000) at line 7569: Illegal mix of collations (latin1_bin,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '=' Attempts to force this with --default-character-set={utf8|utf8mb4} also resulted in the same error or same with (echo "set character_set_connection=utf8;"; cat /tmp/qc.txt; ) | time mysql --force.. Are query cache results actually predictable if they have the same statement running in a different session character set/collation? Anyone see a work around to get the restore file usable on all queries? Is asking for query cache dump/load as a feature request (similar to innodb_buffer_pool_dump_at_shutdown/innodb_buffer_pool_restore_at_startup) going to be useful to anyone? -- -- Daniel Black, Engineer @ Open Query (http://openquery.com.au) Remote expertise & maintenance for MySQL/MariaDB server environments.

2 2

[Maria-discuss] performance_schema
by Rhys Campbell 15 May '15

15 May '15

Hi All, Does anyone know if the performance_schema guarantees that every statement executed will appear in the events_statement_summary_by_digest table (excluding those that could be pushed out due to size restrictions). Or is it a "best effort" system? If it's a best effort system what are the circumstances in which a query may not make it to the performance schema? Rhys Campbell Database Administrator TradingScreen, Inc. 23 York House, 5th Floor London WC2B 6UJ Email: rhys.campbell(a)tradingscreen.com<mailto:rhys.campbell@tradingscreen.com> Follow TradingScreen on Twitter<http://twitter.com/#!/TradingScreen> , Facebook<http://www.facebook.com/pages/TradingScreen/214046251945650> and our blog Trading Smarter<tradingsmarter.tradingscreen.com> This message is intended only for the recipient(s) named above and may contain confidential information. If you are not an intended recipient, you should not review, distribute or copy this message. Please notify the sender immediately by e-mail if you have received this message in error and delete it from your system.

2 1

[Maria-discuss] Twitter handles of technical MariaDB guys
by Lukas Lehner 15 May '15

15 May '15

Hi You can recommend me Twitter handles of technical MariaDB guys? I wish to learn, I have immediate DB skills. Lukas

1 0