- discuss - lists.mariadb.org

[Maria-discuss] MariaDB and Intel TSX
by Reindl Harald 25 Jun '17

25 Jun '17

does MariaDB make usage of TSX on newer hardware or at least using glibc for the query cache mutex which would implicit benfit form it? just curious because next year we replace our cluster nodes and put them in a EVC mode where TSX instrcutions are supported https://en.wikipedia.org/wiki/Transactional_Synchronization_Extensions https://www.flamingspork.com/blog/2014/06/05/performance-impact-of-mysql-qu… https://lwn.net/Articles/534758/

2 2

[Maria-discuss] Help to compile MariaDB 10.2.6 - Debian 8.8
by Guillermo Céspedes Tabárez 23 Jun '17

23 Jun '17

Hello I need help to add in my script ( https://gist.github.com/dertin/8de14e458dd4f0d3acb5f0deff120951#file-instal… ) the compilation of "MariaDB 10.2.6" from its source code. Must work for Debian 8.8 32bit and 64bit Thank you

4 16

[Maria-discuss] memory allocators and MariaDB
by MARK CALLAGHAN 20 Jun '17

20 Jun '17

Do MariaDB 10.2 binaries use glibc malloc or get built with one of tcmalloc or jemalloc? Sorry for being lazy, as I have yet to install it. -- Mark Callaghan mdcallag(a)gmail.com

3 8

[Maria-discuss] mariadb 10.2.6 make test failures
by Bruce Dubbs 20 Jun '17

20 Jun '17

I am trying to build mariadb from source on a linux system. I used this procedure: mkdir build cd build cmake -DTOKUDB_OK=0 .. make make test The build goes fine, but the output of make test gives: 74% tests passed, 21 tests failed out of 81 Total Test time (real) = 42.80 sec The following tests FAILED: 23 - bulk1 (Failed) 24 - performance (Failed) 25 - basic-t (Failed) 26 - fetch (Failed) 27 - charset (Failed) 28 - logs (Failed) 29 - cursor (Failed) 30 - errors (Failed) 31 - view (Failed) 32 - ps (Failed) 33 - ps_bugs (Failed) 34 - sp (Failed) 35 - result (Failed) 36 - connection (Failed) 37 - misc (Failed) 38 - ps_new (Failed) 39 - sqlite3 (Failed) 40 - thread (Failed) 41 - features-10_2 (Failed) 42 - dyncol (Failed) 43 - async (Failed) Errors while running CTest All the failing tests have messages in the log like: # Couldn't establish connection to server (null). Error (2002): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) Bail out! Can't connect to a server. Aborting.... All the above tests appear to be new to version 10.2. I used the asme build procedure for version 10.1.23 and all tests pass, but none of the above tests show up on the earlier version either. What am I missing? If the tests need a server running, shouldn't the tests start the server? -- Bruce

3 4

[Maria-discuss] MariaDB new installation on Centos 6
by romeo kalebic 18 Jun '17

18 Jun '17

Hi everyone, I'm new user of MariaDB, and I install it on Centos 6.8 64bit. I install it using yum utility , according instruction on MariaDB site: https://mariadb.com/kb/en/mariadb/yum/. yum install MariaDB-server MariaDB-client finished without any problem. I can use mysql client to connect to databases without any problems. Then I try to compile some of my applications. To find out what library to use and where include files are I use mysql_config and mariadb_config. mysql_config output: [root@xms tests]# mysql_config Usage: /usr/bin/mysql_config [OPTIONS] Options: --cflags [-I/usr/include/mysql ] --include [-I/usr/include/mysql] --libs [-L/usr/lib64 -lmariadb] --libs_r [-L/usr/lib64 -lmariadb] --plugindir [/usr/lib64/mysql/plugin] --socket [/var/lib/mysql/mysql.sock] --port [0] --version [10.2.6] --libmysqld-libs [-L/usr/lib64 -lmysqld -lpthread -lz -lm -lrt -ldl -lssl -lcrypto -lcrypt -laio] --variable=VAR VAR is one of: pkgincludedir [/usr/include/mysql] pkglibdir [/usr/lib64] plugindir [/usr/lib64/mysql/plugin] [root@xms tests]# And mariadb_config returns: [root@xms tests]# mariadb_config Copyright 2011-2015 MariaDB Corporation AB Get compiler flags for using the MariaDB Connector/C. Usage: mariadb_config [OPTIONS] --cflags [-I/usr/include/mysql -I/usr/include/mysql/mysql] --include [-I/usr/include/mysql -I/usr/include/mysql/mysql] --libs [-L/usr/lib64/ -lmariadb -lpthread -ldl -lm -lssl -lcrypto] --libs_r [-L/usr/lib64/ -lmariadb -lpthread -ldl -lm -lssl -lcrypto] --libs_sys [-lpthread -ldl -lm -lssl -lcrypto] --version [10.2.5] --socket [/var/lib/mysql/mysql.sock] --port [3306] --plugindir [/usr/lib64/mysql/plugin] --tlsinfo [OpenSSL 1.0.1e] [root@xms tests]# First question is why these two outputs are different? I wrote Makefile and use -I and -L like this in mysql_config. gcc compiler finish with error that there is no mariadb library. What I have missed? Where is libmariadb located? Best Regards, Romeo <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_cam…> Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_cam…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

3 3

[Maria-discuss] Question about mariadb 10.2 "rows" keyword
by Rod Bio 14 Jun '17

14 Jun '17

Hi everyone, Apologies I am not a native english speaker so my english will be weird and database is not my specialty, but I would just like to ask something about mariadb specifically version 10.2 I just recently upgraded to 10.2 from 5.5 and somewhat we had problem displaying our data. Anyway we've traced the issue, after trying to replicate the database, to a specific table. What we found is that when we create a table with a column named "rows" it would return an error. e.g. MariaDB [test]> create table test (rows int); ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'rows int)' at line 1 Only by using `` as in `rows` to create the column would we be able to do it successfully. e.g. MariaDB [test]> create table test (`rows` int); Query OK, 0 rows affected (0.02 sec) I haven't had any other issue but just this specific keyword and we have not successfully seen any documentation regarding this. Thank you everyone in advance. -- Rod

3 2

Re: [Maria-discuss] Deadlocks in R/W phase of Sysbench
by J. Cassidy 10 Jun '17

10 Jun '17

All, as discussed yesterday, here is my setup and configuration. ************************************************************* 16 VCPU 64GB Memory, DB size 12 GB (In RAM), MariaDB 10.1.24* Single Image. No clustering or outside network interaction * SuSE SLES 12 SP2 using a 4.4.49-92 kernel * ************************************************************* ### Sysbench R/W - 64 Threads sysbench 0.5: multi-threaded system evaluation benchmark Running the test with following options: Number of threads: 64 Report intermediate results every 2 second(s) Initializing random number generator from seed (42). Threads started! [ 2s] threads: 64, tps: 51141.62, reads: 717023.15, writes: 204757.48, response time: 4.72ms (99%), errors: 54.00, reconnects: 0.00 [ 4s] threads: 64, tps: 47938.19, reads: 671195.72, writes: 191759.28, response time: 4.65ms (99%), errors: 5.00, reconnects: 0.00 [ 6s] threads: 64, tps: 46908.41, reads: 656790.25, writes: 187640.64, response time: 4.65ms (99%), errors: 2.50, reconnects: 0.00 [ 8s] threads: 64, tps: 45940.12, reads: 643191.64, writes: 183781.47, response time: 4.44ms (99%), errors: 3.00, reconnects: 0.00 [ 10s] threads: 64, tps: 45789.91, reads: 641083.20, writes: 183161.13, response time: 4.77ms (99%), errors: 2.00, reconnects: 0.00 [ 12s] threads: 64, tps: 45408.37, reads: 635825.16, writes: 181672.97, response time: 4.80ms (99%), errors: 1.50, reconnects: 0.00 [ 14s] threads: 64, tps: 44319.93, reads: 620440.07, writes: 177262.74, response time: 4.50ms (99%), errors: 0.50, reconnects: 0.00 [ 16s] threads: 64, tps: 44634.78, reads: 624843.97, writes: 178523.13, response time: 4.71ms (99%), errors: 0.50, reconnects: 0.00 . . . [ 572s] threads: 64, tps: 24980.71, reads: 349707.39, writes: 99924.32, response time: 5.92ms (99%), errors: 0.00, reconnects: 0.00 [ 574s] threads: 64, tps: 25337.03, reads: 354796.98, writes: 101355.13, response time: 6.19ms (99%), errors: 0.00, reconnects: 0.00 [ 576s] threads: 64, tps: 25196.31, reads: 352683.33, writes: 100777.24, response time: 6.49ms (99%), errors: 0.00, reconnects: 0.00 [ 578s] threads: 64, tps: 25235.61, reads: 353317.05, writes: 100941.94, response time: 5.94ms (99%), errors: 0.00, reconnects: 0.00 [ 580s] threads: 64, tps: 25241.11, reads: 353395.59, writes: 100956.96, response time: 6.09ms (99%), errors: 0.00, reconnects: 0.00 [ 582s] threads: 64, tps: 25146.18, reads: 352056.04, writes: 100599.73, response time: 5.96ms (99%), errors: 0.00, reconnects: 0.00 [ 584s] threads: 64, tps: 25242.59, reads: 353362.72, writes: 100978.85, response time: 6.21ms (99%), errors: 0.00, reconnects: 0.00 [ 586s] threads: 64, tps: 25297.02, reads: 354187.80, writes: 101166.58, response time: 5.89ms (99%), errors: 0.00, reconnects: 0.00 [ 588s] threads: 64, tps: 25070.60, reads: 350944.86, writes: 100283.39, response time: 6.33ms (99%), errors: 0.00, reconnects: 0.00 [ 590s] threads: 64, tps: 25115.20, reads: 351666.84, writes: 100466.81, response time: 6.21ms (99%), errors: 0.00, reconnects: 0.00 [ 592s] threads: 64, tps: 25165.11, reads: 352314.52, writes: 100668.94, response time: 5.98ms (99%), errors: 0.00, reconnects: 0.00 [ 594s] threads: 64, tps: 25165.99, reads: 352322.89, writes: 100703.97, response time: 6.30ms (99%), errors: 0.00, reconnects: 0.00 [ 596s] threads: 64, tps: 24854.73, reads: 347945.66, writes: 99370.90, response time: 6.17ms (99%), errors: 0.00, reconnects: 0.00 [ 598s] threads: 64, tps: 24888.16, reads: 348463.80, writes: 99556.16, response time: 6.21ms (99%), errors: 0.00, reconnects: 0.00 [ 600s] threads: 64, tps: 25158.22, reads: 352195.58, writes: 100633.38, response time: 5.94ms (99%), errors: 0.00, reconnects: 0.00 OLTP test statistics: queries performed: read: 256244548 write: 73212536 other: 36606172 total: 366063256 transactions: 18302990 (30454.11 per sec.) read/write requests: 329457084 (548179.44 per sec.) other operations: 36606172 (60908.54 per sec.) ignored errors: 192 (0.32 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 601.0023s total number of events: 18302990 total time taken by event execution: 38450.5507s response time: min: 0.42ms avg: 2.10ms max: 62.26ms approx. 99 percentile: 5.62ms Threads fairness: events (avg/stddev): 285984.2188/3788.79 execution time (avg/stddev): 600.7899/0.00 ================================================================================================================= ### Excerpt from MariaDB Error Log (Thousands of similiar messages). . . . 2017-06-08 8:10:44 4395980287248 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction 2017-06-08 8:10:44 4353081699600 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction 2017-06-08 8:10:44 4353087458576 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction 2017-06-08 8:10:44 4360227998992 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction 2017-06-08 8:10:44 4353073515792 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction 2017-06-08 8:10:44 4390705949968 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction 2017-06-08 8:10:44 4360227392784 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction 2017-06-08 8:10:44 4353087458576 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction 2017-06-08 8:10:44 4353073515792 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction 2017-06-08 8:10:44 4353087458576 [ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction . . . =================================================================================================================== The configuration below has been tried and tested and works across on different architectures e.g. S390X and S390X. [mysqld_safe] user=root password='' [mysqld] symbolic-links=0 open_files_limit = 65535 # was 102400 max_allowed_packet = 16M # was 768M max-connect-errors = 1000000 connect_timeout = 120 # was 60 performance-schema = false net_read_timeout = 720 log-warnings=2 #####files and sockets # innodb_log_group_home_dir= /data-lun22/logs pid-file=/tmp/mysqld.pid.sysbench port = 3306 # log-error=/var/log/mariadb/mariadb-error-john.log log-error=/var/log/mariadb/mariadb-error-john.log general_log_file=/var/log/mariadb/mariadb-john.log long_query_time = 0 slow_query_log = 0 # slow_query_log_file = /var/log/mariadb/mariadb-slowquery.log # general-log # general_log_file=/var/log/mariadb/mariadb-general.log ##### MariaDB temporary tables # tmpdir in RAM tmpdir = /mnt/tmpdir #####non innodb options (fixed) max_connections = 4000 back_log = 150 table_open_cache = 8000 # was 2048 # was 4000 # was 8000 key_buffer_size = 16M query_cache_type = 0 #was 1 # was 0 join_buffer_size = 32K sort_buffer_size = 32K autocommit = 0 thread-cache-size = 50 table-definition-cache = 1024 table-open-cache = 2048 ####fixed innodb options innodb_file_per_table = true innodb_open_files = 4096 # was 8192 # was 4096 innodb_data_file_path = ibdata1:250M:autoextend innodb_flush_log_at_trx_commit = 2 innodb_flush_method = O_DIRECT_NO_FSYNC innodb_log_buffer_size = 256M innodb_log_file_size = 4G # was 14G # was 4G innodb_log_files_in_group = 2 innodb_buffer_pool_size = 36G #was 8G # was 48G innodb_buffer_pool_instances = 8 # 16 # was 6 # was 32 innodb_adaptive_hash_index_partitions = 32 innodb_thread_concurrency = 0 ####tuning for RAM disk innodb_adaptive_flushing = 1 innodb_flush_neighbors = 0 innodb_io_capacity = 8000 # was 20000 innodb_io_capacity_max = 40000 innodb_lru_scan_depth = 4096 innodb_purge_threads = 2 innodb_read_io_threads = 2 innodb_write_io_threads = 2 <*> Note NMON reports 35% memory usage in the middle of the abovementioned Banchmark ergo the DB is in cache. No paging. ++ Note The R/W TPS rate are "respectable" based on observations from countless other runs ran on other configurations/architecture. The TPS rates would obviously improve if the deadlock situation outlined above was resolved. So there you have it. Any tips / pointers that would enlighten me are much appreciated. Regards, JC Privet Sergey, do not think I have a problem with my config(s). I am not on the machine at the moment. >From memory, with 16 Cores / 32 VCPU, 64 GB memory, buffer pools, logs etc. tailored to this configuration, the TPS rate is 64 threads - R/O 73K R/W 26K 128 threads - R/O 71K R/W 23K. I am putting the DB in RAM of course to obviate I/O issues. I am interested in seeing what is happening with the CPU caches and nest.. NMON tells me that for the R/O runs I have approximately 85% User, 12-14% System and the rest Idle - this stays consistent throughout the run (300 seconds R/O, 600 seconds R/W) which is reasonable. On the R/W runs, NMONs CPU usage display shows that some CPUs are barely firing, or only intermittently and the granularity I had with R/O is gone. This would seem to tie in with what I see in the MariaDB log - headwinds, violent storms and deadlocks. Any further information you require, please let me know. Regards, JC Hi! I just tried this on x86: 40 threads against single table having 10k rows. No deadlocks, "tps: 12372.40, reads: 173219.52, writes: 49490.51". This is with latest snapshot of 10.1 and sysbench 1.0. I believe you shouldn't see deadlocks on such a big data set. Probably something is wrong with your configs? Regards, Sergey On Thu, Jun 08, 2017 at 04:51:31PM +0200, J. Cassidy wrote: > > > Hello Mark, > > appreciate the reply. > > The OLTP table size and > OLTP size give me a DB size of appx 12GB. This is what I want. I have in the > meantime, looked at some older logs and see that whatever amount of threads I > specify (8, 16,32, 64 or 128), the deadlock messages are still surfacing in the > R/W phase > of Sysbench. I even dropped in two different MariaDB release levels > (10.0.19, 10.1.24) to see whether it would make a difference, deadlock still > there. I am using Sysbench 0.5 for these tests. > I am currently using MariaDB > 10.1.24 - built directly on the machine. S390x by the way, but the problem is > also > occuring on an X86-64 box. > > Regards, > > > JC > > > > I run sysbench frequently with MySQL > (not MariaDB yet) and don't recall ever seeing this error. But I use much > larger values for --oltp-table-size. You used 85937 and I use 1M or larger. > Maybe > there is too much data contention with smaller tables, even if a large number of > tables is used (128 in your case). I use a smaller number of tables - between 4 > and 20. On Thu, Jun 8, 2017 at 12:48 AM, J. Cassidy sean(a)jdcassidy.eu> wrote: > Hello all, > > > have used Sysbench extensively to test MariaDB CPU load factors between various > architectures. > > I am noticing massive amounts of "[ERROR] mysqld: > Deadlock found when trying to get lock; try restarting transaction" in the > MariaDB error log during the R/W phase. The Sysbench R/W TPS rates are still > respectable for my purposes, but how do I > correct this condition?? Can I > ignore this / must live with it? > I have tried three different MariaDB > releases (10.0.19, 10.1.12, 10.1.24) to narrow things down, but this message (x > 1000's) is happening across all releases in the Sysbench R/W phase. > > Here is the Sysbench command that drives the workload against MariaDB - > > sysbench --test=lua/oltp.lua --oltp_tables_count=128 --oltp-table-size=85937 > --rand-seed=42 --rand-type=uniform --num-threads=128 --oltp-read-only=off > --report-interval=2 --mysql-socket=/var/lib/mysql/mysql.sock --max-time=201 > --max-requests=0 --mysql-user=root --percentile=99 run > > Would anyone > have a tip / idea /pointer? > > > Regards, > > > JC > > > John Cassidy > > Obere Bühlstrasse 21 > 8700 Küsnacht > (ZH) > Switzerland / Suisse / Schweiz > > > Mobile: +49 152 58961601 (Germany) > Mobile: +352 621 577 149 (Luxembourg) > Mobile: +41 78 769 17 97 (CH) > Landline: +41 44 509 1957 > > http://www.jdcassidy.eu > > "Aut viam inveniam aut > faciam" - Hannibal. > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : > maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp > --Mark Callaghan > mdcallag(a)gmail.com > > John Cassidy > > Obere Bühlstrasse 21 > 8700 > Küsnacht (ZH) > Switzerland / Suisse / Schweiz > > > Mobile: > +49 152 58961601 (Germany) > Mobile: +352 621 577 149 (Luxembourg) > Mobile: > +41 78 769 17 97 (CH) > Landline: +41 44 509 1957 > > http://www.jdcassidy.eu > > "Aut viam inveniam aut faciam" - > Hannibal. > _______________________________________________ > Mailing list: https://launchpad.net/~maria-discuss > Post to : maria-discuss(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~maria-discuss > More help : https://help.launchpad.net/ListHelp John Cassidy Obere Bühlstrasse 21 8700 Küsnacht (ZH) Switzerland / Suisse / Schweiz Mobile: +49 152 58961601 (Germany) Mobile: +352 621 577 149 (Luxembourg) Mobile: +41 78 769 17 97 (CH) Landline: +41 44 509 1957 http://www.jdcassidy.eu "Aut viam inveniam aut faciam" - Hannibal. John Cassidy Obere Bühlstrasse 21 8700 Küsnacht (ZH) Switzerland / Suisse / Schweiz Mobile: +49 152 58961601 (Germany) Mobile: +352 621 577 149 (Luxembourg) Mobile: +41 78 769 17 97 (CH) Landline: +41 44 509 1957 http://www.jdcassidy.eu "Aut viam inveniam aut faciam" - Hannibal.

4 11

[Maria-discuss] logging original SQL statement along with ROW events
by Justin Swanhart 09 Jun '17

09 Jun '17

Hi, I seem to recall reading that MariaDB can annotate the binary log with the original SQL statement for a given set of ROW changes in the binary log, but I can't seem to find documentation on this on the wiki. Can MariaDB log the original STATEMENT along with the ROW changeset? If so, can you point me to the relevant documentation, or better yet the patch itself that makes this possible? Thanks, --Justin

3 3

[Maria-discuss] Deadlocks in R/W phase of Sysbench
by J. Cassidy 08 Jun '17

08 Jun '17

Hello all, have used Sysbench extensively to test MariaDB CPU load factors between various architectures. I am noticing massive amounts of "[ERROR] mysqld: Deadlock found when trying to get lock; try restarting transaction" in the MariaDB error log during the R/W phase. The Sysbench R/W TPS rates are still respectable for my purposes, but how do I correct this condition?? Can I ignore this / must live with it? I have tried three different MariaDB releases (10.0.19, 10.1.12, 10.1.24) to narrow things down, but this message (x 1000's) is happening across all releases in the Sysbench R/W phase. Here is the Sysbench command that drives the workload against MariaDB - sysbench --test=lua/oltp.lua --oltp_tables_count=128 --oltp-table-size=85937 --rand-seed=42 --rand-type=uniform --num-threads=128 --oltp-read-only=off --report-interval=2 --mysql-socket=/var/lib/mysql/mysql.sock --max-time=201 --max-requests=0 --mysql-user=root --percentile=99 run Would anyone have a tip / idea /pointer? Regards, JC John Cassidy Obere Bühlstrasse 21 8700 Küsnacht (ZH) Switzerland / Suisse / Schweiz Mobile: +49 152 58961601 (Germany) Mobile: +352 621 577 149 (Luxembourg) Mobile: +41 78 769 17 97 (CH) Landline: +41 44 509 1957 http://www.jdcassidy.eu "Aut viam inveniam aut faciam" - Hannibal.

3 6

[Maria-discuss] libmariadb.so.3 compatibility
by Michal Schorm 08 Jun '17

08 Jun '17

Hello, I'm currently quite confused about the *libmariadb* library. In MariaDB 10.1.24 there are: " libmysqlclient_r.so -> libmysqlclient.so " " libmysqlclient.so -> libmysqlclient.so.18 " " libmysqlclient.so.18 -> libmysqlclient.so.18.0.0 " " libmysqlclient.so.18.0.0 " In MariaDB 10.2.6 there are: " libmariadb.so -> libmariadb.so.3 " In MariaDB-connector-c v2.3.3 there are: " libmysqlclient.so -> libmariadb.so " " libmysqlclient_r.so -> libmariadb.so " " libmariadb.so -> libmariadb.so.2 " " libmariadb.so.2 " -- 1) is "libmariadb.so.2" compatibile with "libmysqlclient.so.18.0.0"? Both API and ABI? 2) is "libmariadb.so.3" compatibile with "libmysqlclient.so.18.0.0"? Both API and ABI? 3) is "libmariadb.so.2" compatibile with "libmariadb.so.3"? Both API and ABI? Many packages in Fedora depends on "libmysqlclient.so". I would like to pack 10.2 to Fedora, but I can't before all those packages move to the libmariadb library. It would be considerably easier transfer for me, if all those libraries were compatibile, so I could provide symlinks (simmilar as in the connector C), and provide time for all those packages in Fedora to move safely. -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

2 2