Hi all,

I am investigating the viability of using MariaDB as a datastore for streaming telemetry data sourced via gRPC. The data ingestion rate will be quite high (write once, read many with no change to the data), but the data can be buffered and bulk loaded in 10-60sec intervals. We are currently using MariaDB for OLTP purposes (Innodb) and also graphs (OQGRAPH) with no issues.

My journey so far has been mainly focused on Kafka but the more I dig the more future trouble I uncover. So many components required to solve even a simple problem. Now I am investigating if using our current tooling that our team knows already will get the job done.

Reading up on ColumnStore has also uncovered more gotchas and deprectations that I am comfortable with. Things that concern me are:

1. Is ColumnStore going to go the way of TokuDB? 2. The docs warn that data needs to be bulk inserted - not super ideal for streaming applications. Is streaming via cpimport viable? 3. The bulk write API's have been deprecated ( https://mariadb.com/kb/en/columnstore-bulk-write-sdk/) which then implies that streaming is not advised (Kafka connectors are thus also not viable). This slideshow talks about streaming with ColumnStore 1.1 https://www.slideshare.net/MariaDB/big-data-analytics-with-mariadb-columnsto... 4. Streaming data is a combination of fixed columns and key / value pairs. How to deal with the dynamic nature of the key / value pairs (see below for sample data that was sourced via gRPC and converted to OpenTSDB format - we can convert the data into any required format, this is just one example).

So some questions:

What is the current recommended streaming telemetry data store solution from MariaDB, or should I rather look at a custom solution (different tooling, more skills/re-skilling required)? Is there a writeup or reference design?

Any insights will be helpful.

Thanks.

put eos.intflagpktcounter.counter.pkt 1611032330 547181 intf=Port-Channel553 direction=in type=Multicast host=30.7.253.145 put eos.intflagcounter.counter.octets 1611032330 121113118 intf=Port-Channel553 direction=in host=30.7.253.145 put eos.intflagpktcounter.counter.pkt 1611032330 3649031 type=Ucast host=30.7.253.145 intf=Port-Channel553 direction=out put eos.intflagpktcounter.counter.pkt 1611032330 297600 host=30.7.253.145 intf=Port-Channel553 direction=in type=Ucast put eos.intflagcounter.counter.errors 1611032330 0 intf=Port-Channel553 direction=out host=30.7.253.145 put eos.totalcpu.cpu.utilization.total 1611032330 21 type=util host=30.7.253.145 put eos.totalcpu.cpu.utilization.total 1611032330 225275556 host=30.7.253.145 type=user put eos.corecpu.cpu.utilization.total 1611032330 42929390 type=system host=30.7.253.145 put eos.corecpu.cpu.utilization.total 1611032330 4005894390 type=idle host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.0 1611032330 55264287 type=user host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.0 1611032330 10865672 host=30.7.253.145 type=system put eos.corecpu.cpu.utilization.cpu.0 1611032330 1002366784 type=idle host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.1 1611032330 20 type=util host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.1 1611032330 50596732 type=user host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.1 1611032330 10883378 type=system host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.1 1611032330 1006988024 type=idle host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.2 1611032330 21 type=util host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.2 1611032330 58921103 type=user host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.2 1611032330 10405231 type=system host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.2 1611032330 999355549 type=idle host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.3 1611032330 21 type=util host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.3 1611032330 60493433 type=user host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.3 1611032330 10775108 type=system host=30.7.253.145 put eos.corecpu.cpu.utilization.cpu.3 1611032330 997184032 type=idle host=30.7.253.145 put eos.tempsensor.environment.temperature 1611032330 45.526831652731964 sensor=cell/1/TempSensor1 host=30.7.253.145 put eos.fanspeed.environment.cooling 1611032330 60.57142857142857 fan=Fan2/1 host=30.7.253.145 put eos.fanspeed.environment.cooling 1611032330 60.57142857142857 host=30.7.253.145 fan=Fan2/1 put eos.fanspeed.environment.cooling 1611032330 60.57142857142857 fan=Fan1/1 host=30.7.253.145 put eos.fanspeed.environment.cooling 1611032330 60.57142857142857 fan=Fan1/1 host=30.7.253.145 put eos.fanspeed.environment.cooling 1611032330 60.57142857142857 fan=Fan3/1 host=30.7.253.145 put eos.fanspeed.environment.cooling 1611032330 60.57142857142857 fan=Fan3/1 host=30.7.253.145 put eos.fanspeed.environment.cooling 1611032330 60.57142857142857 fan=Fan4/1 host=30.7.253.145 put eos.fanspeed.environment.cooling 1611032330 60.57142857142857 fan=Fan4/1 host=30.7.253.145 put eos.sfpfixedtemperature.xcvr.temperature 1611032330 38.0390625 host=30.7.253.145 intf=Ethernet3 put eos.sfpfixedrxtxpower.xcvr.power 1611032330 -2.0398103065285085 intf=Ethernet3 powerdirection=tx host=30.7.253.145 put eos.sfpfixedrxtxpower.xcvr.power 1611032330 -1.92938760082761 powerdirection=rx host=30.7.253.145 intf=Ethernet3 put eos.sfpfixedrxtxpower.xcvr.power 1611032330 -1.92938760082761 intf=Ethernet3 powerdirection=rx host=30.7.253.145