- discuss - lists.mariadb.org

[Maria-discuss] Troubleshooting Statement That Crashes MariaDB (mysqld got signal 11)
by Michael Caplan 04 Nov '21

04 Nov '21

Hello, I have a nothing fancy SQL statement that crashes MariaDB 10.3.31. The error log reports "mysqld got signal 11" with the below included mini core dump. The statement caused no issues with my prior install of 10.2.xx. With 10.3, the statement runs the majority of the time. The vast majority of the time. I have never dealt with such an issue with MariaDB, and am unsure where to start with troubleshooting this. Any tips? FYI, there is a corresponding bug recorded here: https://jira.mariadb.org/browse/MDEV-26835 Thanks, Mike 211014 16:13:53 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.3.31-MariaDB-1:10.3.31+maria~focal-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=587 max_threads=752 thread_count=81 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1784362 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f13e83b3768 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f1644e96dd8 thread_stack 0x49000 /usr/sbin/mysqld(my_print_stacktrace+0x32)[0x55d5a2f6f3a2] /usr/sbin/mysqld(handle_fatal_signal+0x55d)[0x55d5a2a46e8d] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f27569fc3c0] /usr/sbin/mysqld(+0x978f4f)[0x55d5a2bdbf4f] /usr/sbin/mysqld(_ZN7handler10ha_rnd_posEPhS0_+0x50)[0x55d5a2a4c6d0] /usr/sbin/mysqld(_Z16rr_from_pointersP11READ_RECORD+0x3e)[0x55d5a2b7cb2e] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x18e)[0x55d5a28869be] /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xbee)[0x55d5a28b0dce] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x37)[0x55d5a28b1137] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xf2)[0x55d5a28b1282] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x14b)[0x55d5a28b1bcb] /usr/sbin/mysqld(+0x5de9f1)[0x55d5a28419f1] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x5414)[0x55d5a284f6e4] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1f3)[0x55d5a2852253] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x105d)[0x55d5a285462d] /usr/sbin/mysqld(_Z10do_commandP3THD+0x12d)[0x55d5a28565dd] /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x246)[0x55d5a293f186] /usr/sbin/mysqld(handle_one_connection+0x3f)[0x55d5a293f35f] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f27569f0609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f2756915293] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f13e84a8b70): SELECT * FROM (( # # get postings from last statement and postings not on statements # SELECT a.*, COALESCE(ci.id, i.id) as invoice_id FROM ddx_lab_801317.postings a LEFT JOIN ddx_lab_801317.cases_invoices ci ON a.ref_type = 'CASE_INVOICE' AND a.ref_id = ci.id LEFT JOIN ddx_lab_801317.invoices i ON a.ref_type = 'INVOICE' AND a.ref_id = i.id WHERE ( a.statement_date IS NULL OR a.statement_date = '2021-09-27' ) AND a.practice_id='19686' ORDER BY post_date ) UNION ( # # get postings that have remaining balances # SELECT DISTINCT a.*, COALESCE(ci.id, i.id) as invoice_id FROM ddx_lab_801317.postings a LEFT JOIN ddx_lab_801317.cases_invoices ci ON a.ref_type = 'CASE_INVOICE' AND a.ref_id = ci.id LEFT JOIN ddx_lab_801317.invoices i ON a.ref_type = 'INVOICE' AND a.ref_id = i.id LEFT JOIN ddx_lab_801317.postings b ON a.ref_type = b.ref_type AND a.ref_id = b.ref_id AND a.id != b.id WHERE NOT (a.type = 'CREDIT' OR a.type = 'PAYMENT') AND ((SELECT SUM(b.amount)) IS NULL OR (SELECT SUM(a.amount)) + (SELECT SUM(b.amount)) > 0) AND a.practice_id='19686' GROUP BY a.id ORDER BY post_date ) UNION ( # # get partial payments and credits from older statements # SELECT a.*, COALESCE(ci.id, i.id) as invoice_id FROM ddx_lab_801317.postings a LEFT JOIN ddx_lab_801317.cases_invoices ci ON a.ref_type = 'CASE_INVOICE' AND a.ref_id = ci.id LEFT JOIN ddx_lab_801317.invoices i ON a.ref_type = 'INVOICE' AND a.ref_id = i.id WHERE ( a.statement_date < '2021-09-27' ) AND (a.type='PAYMENT' OR a.type='CREDIT') AND a.practice_id='19686' )) AS t1 ORDER BY post_date ASC, invoice_id ASC Connection ID (thread ID): 21014956 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains information that should help you find out what is causing the crash. Writing a core file... Working directory at /var/lib/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory unlimited unlimited bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 514974 514974 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core

2 2

[Maria-discuss] Incorrectly formatted XML files in mariadb-server sources
by Zuzana Miklankova 26 Oct '21

26 Oct '21

Hi, Fedora's rpminspect tool reports [1] some incorrectly formatted XMLs (most of them used for testsuite) within mariadb source files, namely they are: mysql-test/plugin/sphinx/sphinx/testdata.xml - undefined namespace mysql-test/std_data/loadxml.dat - different parser errors, seems to me like it is ill formed on purpose of the test mysql-test/plugin/connect/connect/std_data/nocs.xml - not proper utf-8 encoding mysql-test/std_data/loaddata/mdev9874.xml - not proper utf-8 encoding sql/share/mariadb/charsets/Index.xml - redefined attribute mysql-test/std_data/ldml/Index.xml - redefined attribute It does not seem (to me) that the ill-formatting breaks anything, and that in some cases it is desired. Specifically the loadxml.dat XML, where the test case counts with the form as it is, and Index.xml files, where modification to the XML format could change the meaning of the file. In other cases, it seems (to me) that the correction of the XMLs to a valid format could be possible. These are specifically the XMLs with encoding issues, where it should be sufficient to specify a used charset (ISO-8859-1 in both cases). This is also the case with the undefined namespace in testdata.xml, whose definition should be also possible. Would the correct approach to this issue be to suppress the rpminspect warnings, or should the specified XML files be modified to a valid XML form? Thanks for your opinions in advance. [1] https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/rpminspect-pi… Zuzana Miklánková Associate Software Engineer Red Hat <https://www.redhat.com> <https://www.redhat.com>

2 2

[Maria-discuss] MariaDB 10.4 shows a 2e308 cost on a optimization plan correctly preferred by 10.0
by David Strauss 25 Oct '21

25 Oct '21

I'm seeing a regression in query optimization after testing a database+query that was running well on MariaDB 10.0. The issue occurs on an installation of 10.4.18. Version 10.4 considers the plan that 10.0 used but dismisses it as having a cost of 2e308 [1]. The plan 10.4 chooses is highly non-optimal. Based on reviewing some source code, it looks like this sort of "max" cost should only result if summing/multiplying other costs would exceed that maximum. The usual steps of rebuilding tables and index statistics don't help, especially because the JSON output for the optimization plan already shows reasonable index use costs. (The 2e308 cost only starts appearing in a later section.) Because 10.0 lacks the same optimization tracing as 10.4, here is the EXPLAIN output for comparison [2]. Hopefully, there's just something simple amiss and posting here helps anyone else who encounters a 2e308 cost on query optimization plans. I tried to see if this may be fixed in 10.5+, but I haven't found much discussion online of this phenomenon. [1] https://gist.github.com/davidstrauss/becb03fa75ecac046cafe6f05da9f869#file-… [2] https://gist.github.com/davidstrauss/becb03fa75ecac046cafe6f05da9f869#file-…

1 0

[Maria-discuss] State of "alternative" engines and mariadb
by pslawek83 19 Oct '21

19 Oct '21

Hi Guys, what's the state of "alternative" engines (aria and rocks). AFAIR there were some plans to make aria much more advanced than myisam, transactional (transaction safe?) and more optimized for todays hardware. What's the status of RocksBD, basically on the mariadb website i can't find recent updates and most of sources on the internet are outdated. Is it 100% compatible with mariadb nowdays, is there a possibility to have "one-file-per-table" (one set of files) or at least per-database? If everything resides in shared space is there a possibility that one write error (eg. resulting from power loss) can make all rocks tables unreadable? Is it possible to efficiently drop eg. 100GB table partition if the table is using rocks? Last time i checked there were some issues with compaction for Rocks (some limit on DB size, which was rather small can't recall this currently). Is that resolved, and how much additional space is required to compact eg. 1tb database? How's the adoption and developement of both engines going? Are people using it instead of myisam/toku/inno? Is there a lot of effor put into making aria/rocks better? As i'll have to update many servers in the future what's generally the preferred way to go? Drop everything and just use innodb like Oracle did or move towards rocks/aria for everything which doesn't play with innodb well? Another interesting topic is that myisam is having performance issues because of spectre/meltdown mitigations and it seems to hit this engine the most (at least in 10.3, 10.4 as far as I recall). Are there some plans to optimize it for new CPUs (maybe it was already done) or all the effors are going into different storage engines and i should abandon myisam long term? Thanks

2 1

[Maria-discuss] Loading a backup created from 10.6.4 into newly installed 10.6.4 causes mysql.proc to appear corrupt
by Jeff Dyke 30 Sep '21

30 Sep '21

Failing process: Install mariadb from ppa using apt (happens through salt, fwiw) Download last night's backup from a 10.6.4 server, which is a replica from a galera cluster. run the script attached restore_mariabackup.sh Once this is done the database itself works, for our application, but any client trying to access anything that requires mysql.proc gets an error similar to: *Sep 30 19:29:07 devmaria /etc/mysql/debian-start[24656]: error : CorruptSep 30 19:29:07 devmaria /etc/mysql/debian-start[24656]: sys.x$schema_table_statistics* *Sep 30 19:29:07 devmaria /etc/mysql/debian-start[24656]: Error : Cannot load from mysql.proc. The table is probably corrupted * I just built two new servers, one that builds mariadb and then overwrites the data from a backup and one that just installs mariadb. The two different files base_install.proc.sql and backupedup.proc.sql. The error output from sudo mysql_upgrade on the server that received the backup is in mysql_upgrade.out There are more differences in the proc table if i dump the data as well, but i think much of it is driven by the utf8_mb4 rather than 3. Tried to keep this sucient and use attachments. This process has worked for me for until i upgraded to 10.6.4. Please ask anything you need. I think i've been using this similar process to build dev machines since about 10.4 Thanks Jeff

1 2

[Maria-discuss] A general question about performance of update command
by Lehrbass, Robert 27 Sep '21

27 Sep '21

Hi there, I'm new to MariaDB and I'm running tests on the update command. Server version 10.5.11. I'm finding that updates to a table with engine=columnstore are far slower than to a table with default engine=innodb. Obviously, there is no primary key in a table with engine=columnstore, whereas the innodb table does have a primary key. In my tests, for engine=columnstore, 10000 rows took 1995 seconds to be updated, but for engine=innodb, 10000 rows took only 7 seconds to transfer. The difference suggests columnstore takes about 285X longer than innodb. Is this to be expected? I don't need a lengthy answer, btw, just general guidance. Thanks very much, Rob

1 0

[Maria-discuss] DBSeeder Version 3.0.3
by Walter Weinmann 13 Sep '21

13 Sep '21

We are pleased to release version 3.0.3 of DBSeeder ( https://github.com/KonnexionsGmbH/db_seeder) DBSeeder also supports the latest version of MariaDB Server available on Docker Hub (DBMS 10.6.4-focal) and Maven Repository (JDBC 2.7.4). [image: Perf_Snap_3.0.3_win10.png]

2 2

[Maria-discuss] AWS Key management plugin key rotation in replication
by Reinis Rozitis 30 Aug '21

30 Aug '21

Hello, I wanted to clarify - does the key rotation (with AWS KMS backend) work in replication scenario? As in do the slaves (somehow) get also notified and perform the re-encryption if the key is rotated (on master) in background or you have to use different key for each slave for rotation to work? In https://mariadb.com/kb/en/encryption-key-management/#key-rotation it wasn't mentioned. wbr rr

2 3

[Maria-discuss] Issues with Upgrading from MariaDB 10.2.22 to 10.5.12
by Michael Caplan 25 Aug '21

25 Aug '21

Hi there. I'm going through an upgrade process I have done before with earlier versions of mariaDB and Mysql, but running into an issue. My goal is to create a new slave including upgrade process from MariaDB 10.2.22 (serverOld) to 10.5.12 on a new server (serverNew) (as inspired by the brighter mind at https://www.stephenrlang.com/2016/08/setting-up-mysql-master-slave-replicat… ) 1. rsync /var/lib/mysql from serverOld to serverNew 2. On serverOld: FLUSH TABLES WITH READ LOCK; SHOW MASTER STATUS; 3. rerun rsync from serverOld to serverNew 4. Release read lock from serverOld 5. Start mariadb on serverNew 6. run mysql_upgrade on serverNew 7. Celebrate and have a nap In actual practice step #5 failed numerous times before what I think is now an okay serverNew # Innodb settings between serverOld and serverNew #serverOld innodb-flush-method = O_DIRECT innodb-log-files-in-group = 2 innodb-log-file-size = 48M innodb-flush-log-at-trx-commit = 2 innodb-file-per-table = 1 innodb-buffer-pool-size = 128M #serverNew innodb-flush-method = O_DIRECT innodb-log-file-size = 48M innodb-flush-log-at-trx-commit = 2 innodb-file-per-table = 1 innodb-buffer-pool-size = 128M The only difference is innodb-log-files-in-group setting is removed in serverNew, as it has been depricated and removed in 10.5 # Upgrade after crash? When attempting to start serverNew, it would fail with the following error: [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.2.22. [ERROR] InnoDB: Plugin initialization aborted with error Generic error the log file on serverOld showed no evidence of crashing. None the less, I restarted serverOld, and reran the process above. Still the same issue step #5. # Remove ib_logfile* files I then tried forcing the recreation of the redo logs by removing them and then starting up serverNew. This resulted in the following scary errors: [ERROR] InnoDB: Page [page id: space=0, page number=1984] log sequence number 285722576186 is in the future! Current system log sequence number 259573448177. [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. # innodb_fast_shutdown = 0 The next attempt started with serverOld being restarted and innodb_fast_shutdown = 0 being set beforehand. I reran the above steps and still got stuck on step #5 with the following error: [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.2.22. [ERROR] InnoDB: Plugin initialization aborted with error Generic error This time, however, when I removed the ib_logfile* files, we seemed to have an okay startup: [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728 [Note] InnoDB: Completed initialization of buffer pool [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0 [Note] InnoDB: New log file created, LSN=288830457728 [Note] InnoDB: 128 rollback segments are active. [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" [Note] InnoDB: Creating shared tablespace for temporary tables [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... [Note] InnoDB: File './ibtmp1' size is now 12 MB. [Note] InnoDB: 10.5.12 started; log sequence number 0; transaction id 3037161218 [Note] Plugin 'FEEDBACK' is disabled. [Note] Recovering after a crash using /var/lib/mysql/mysql-bin [Note] Starting crash recovery... [Note] Crash recovery finished. [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool [Note] Server socket created on IP: '::'. [ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALUE_ON_ZERO','NO_BACKSLASH_ESCAPES','STRICT_TRANS_TABLES','STRICT_ALL_TABLES','NO_ZERO_IN_DATE','NO_ZERO_DATE','INVALID_DATES','ERROR_FOR_DIVISION_BY_ZERO','TRADITIONAL','NO_AUTO_CREATE_USER','HIGH_NOT_PRECEDENCE','NO_ENGINE_SUBSTITUTION','PAD_CHAR_TO_FULL_LENGTH','EMPTY_STRING_IS_NULL','SIMULTANEOUS_ASSIGNMENT'), found type set('REAL_AS_FLOAT','PIPES_AS_CONCAT','ANSI_QUOTES','IGNORE_SPACE','IGNORE_BAD_TABLE_OPTIONS','ONLY_FULL_GROUP_BY','NO_UNSIGNED_SUBTRACTION','NO_DIR_IN_CREATE','POSTGRESQL','ORACLE','MSSQL','DB2','MAXDB','NO_KEY_OPTIONS','NO_TABLE_OPTIONS','NO_FIELD_OPTIONS','MYSQL323','MYSQL40','ANSI','NO_AUTO_VALU [ERROR] mariadbd: Event Scheduler: An error occurred when initializing system tables. Disabling the Event Scheduler. [Note] Reading of all Master_info entries succeeded [Note] Added new Master_info '' to hash table [Note] /usr/sbin/mariadbd: ready for connections. Version: '10.5.12-MariaDB-1:10.5.12+maria~focal-log' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution Running mysql_upgrade completed without issue. I'm not feeling confident that this upgrade is actually okay. removing the ib_logfile* seems questionable as a requirement to get the upgrade done. Anyone have similar troubles? Recommendations on what my process should be? Thanks, Mike

3 3

[Maria-discuss] Adding MySQL database backup to rsnapshot for Linux Servers
by Turritopsis Dohrnii Teo En Ming 22 Aug '21

22 Aug '21

Subject: Adding MySQL database backup to rsnapshot for Linux Servers Author: Mr. Turritopsis Dohrnii Teo En Ming (TARGETED INDIVIDUAL) Country: Singapore Date: 22 August 2021 Sunday Singapore Time Type of Pulication: Plain Text Document Version: 20210822.01 DETAILED INSTRUCTIONS ===================== Looking for mysqldump ===================== # which mysqldump /usr/bin/mysqldump Testing if mysqldump works ========================== # /usr/bin/mysqldump --all-databases > mysqldump_all_databases.sql Check the filesize of MySQL dump ================================ # du -h mysqldump_all_databases.sql 23M mysqldump_all_databases.sql Backup rsnapshot configuration file first ========================================= # cd /etc/ # cp rsnapshot.conf rsnapshot.conf.backup20082021 Deploying MySQL backup script ============================= # cp /usr/share/doc/rsnapshot-1.4.3/utils/backup_mysql.sh /usr/local/bin/ # chown root.root /usr/local/bin/backup_mysql.sh # chmod o-w /usr/local/bin/backup_mysql.sh # chmod +x /usr/local/bin/backup_mysql.sh # cat /usr/local/bin/backup_mysql.sh ===BEGIN of backup_mysql.sh=== #!/bin/sh ############################################################################## # backup_mysql.sh # # by Nathan Rosenquist <nathan(a)rsnapshot.org> # http://www.rsnapshot.org/ # # This is a simple shell script to backup a MySQL database with rsnapshot. # # The assumption is that this will be invoked from rsnapshot. Also, since it # will run unattended, the user that runs rsnapshot (probably root) should have # a .my.cnf file in their home directory that contains the password for the # MySQL root user. For example: # # /root/.my.cnf (chmod 0600) # [client] # user = root # password = thepassword # host = localhost # # This script simply needs to dump a file into the current working directory. # rsnapshot handles everything else. ############################################################################## # $Id: backup_mysql.sh,v 1.6 2007/03/22 02:50:21 drhyde Exp $ umask 0077 # backup the database /usr/bin/mysqldump --all-databases > mysqldump_all_databases.sql # make the backup readable only by root /bin/chmod 600 mysqldump_all_databases.sql ===END of backup_mysql.sh=== Edit rsnapshot configuration file ================================== # nano /etc/rsnapshot.conf Under BACKUP POINTS / SCRIPTS add the following line ==================================================== backup_script /usr/local/bin/backup_mysql.sh secret.teo-en-ming-corp.com/mysqldump/ Test the rsnapshot configuration file for syntax errors ======================================================= # rsnapshot configtest require Lchown Lchown module loaded successfully Syntax OK Mr. Turritopsis Dohrnii Teo En Ming, 43 years old as of 22 August 2021, is a TARGETED INDIVIDUAL living in Singapore. He is an IT Consultant with a System Integrator (SI)/computer firm in Singapore. He is an IT enthusiast. -----BEGIN EMAIL SIGNATURE----- The Gospel for all Targeted Individuals (TIs): [The New York Times] Microwave Weapons Are Prime Suspect in Ills of U.S. Embassy Workers Link: https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html ******************************************************************************************** Singaporean Targeted Individual Mr. Turritopsis Dohrnii Teo En Ming's Academic Qualifications as at 14 Feb 2019 and refugee seeking attempts at the United Nations Refugee Agency Bangkok (21 Mar 2017), in Taiwan (5 Aug 2019) and Australia (25 Dec 2019 to 9 Jan 2020): [1] https://tdtemcerts.wordpress.com/ [2] https://tdtemcerts.blogspot.sg/ [3] https://www.scribd.com/user/270125049/Teo-En-Ming -----END EMAIL SIGNATURE-----

1 0