Hi Daniel

The original goal of this project was the generation of large amounts of anonymous and random data for relational database systems useful e.g. for stress testing. 

I am really pleased with your reaction to my announcement of DBSeeder version 2.9.0. I am now working on version 2.9.1 (see here: https://github.com/KonnexionsGmbH/db_seeder/tree/wwe_2.9.1) and will try to incorporate as much of your suggestions and issue reports as possible.

1. Compilation with OpenJDK 16

Unfortunately, I can't reproduce the problem. I am also of the opinion that I run OpenJDK 16 both under Windows 10:

openjdk version "16.0.1" 2021-04-20
OpenJDK Runtime Environment (build 16.0.1+9-24)
OpenJDK 64-Bit Server VM (build 16.0.1+9-24, mixed mode, sharing)

as well as under Ubuntu 20.04:

openjdk version "16.0.1" 2021-04-20
OpenJDK Runtime Environment (build 16.0.1+9-24)
OpenJDK 64-Bit Server VM (build 16.0.1+9-24, mixed mode, sharing)

The compilation should be done in the root directory of the db_seeder repository with the following command: 

gradle copyJarToLib

2. Docker Image Size

I created a custom image for the development and operation of DBSeeder. The size of the image is now only 1.73 GB. Details can be found here: https://hub.docker.com/repository/docker/konnexionsgmbh/db_seeder

3. README.md

I have restructured this document on your suggestion with the intention to present DBSeeder more clearly.

4. Docker Compose

I removed Docker Compose, which doesn't make much sense in combination with DBSeeder.

5. line 13 in script run_db_seeder.sh

I have corrected this issue.

6. Login Docker Hub

I removed the unnecessary login to Docker Hub. It seems that at some point I misinterpreted the problems described here: https://github.com/docker/hub-feedback/issues/1103.

7. Defaults in script run_db_seeder_single.sh

run_db_seeder_single is just an auxiliary script used by the script run_db_seeder and is not called directly (see: README.md diagram in section 4.1.2). Therefore the script variables in run_db_seeder_single have no default values unlike in run_db_seeder. The easiest way to customize these environment variables is directly in the run_db_seeder script.

8. Feature request

I added the following two environment variables in the run_db_seeder script specifically for MariaDB:

    DB_SEEDER_CHARACTER_SET_SERVER=utf8mb4
    DB_SEEDER_COLLATION_SERVER=utf8mb4_unicode_ci

9. MDEV-25434

When this extension is available I will gladly switch to health check on MariaDB (similar to Apache Derby for example).

10. Podman

I think there is no need for action on the part of DBSeeder?

11. JSON output

Here the effort seems to me too large, since the realization would have to take place DBMS specifically and this would go beyond the bounds with the currently 24 supported systems. In this context DBSeeder offers two functionalities: 

1. the database schema is generated from a JSON file - see in README.md section 2 and the files in the resources/json directory. 
2. runtime statistics are produced in tsv format - see README.md section 4.4 and files in resources/statistics directory

---

I hope that I have understood your suggestions and objections correctly and have offered satisfactory solutions. If you notice anything else then just let me know.

I plan to release version 2.9.1 by the end of June at the latest. It will also include all updates of the used DBMS Docker images and JDBC drivers.

Thanks again and best regards

Walter







On Mon, 7 Jun 2021 at 05:50, Daniel Black <daniel@mariadb.org> wrote:

Thanks Walter for letting everyone know.

I did play around with it as it might be useful in attempting to replicate sample data for users experiencing problems.

What was your intended purpose when creating this?


Suggestions/minor problems that I came up with testing it:

It doesn't compile with openjdk and I needed to install oraclejdk to test this. This wasn't documented (except .travis.yml that tipped me off).

docker.io/konnexionsgmbh/kxn_dev is a very big image - 7.56G

README.md
- the database list in section 1  - was moved into the 1.1 table this would be easier to read.
- Chines typo
- in reference to docker-compose.yml needs to document creating Transfer directory

run_db_seeder.sh:

line 13 - is this meant to be "rm -rf $PWD/tmp/* || sudo rm -rf $PWD/tmp/*"?

the docker username/password seem to be optional

It's unclear what username/password this is referring to in the output:
Username:
Password:
Login Succeeded!
================================================================================
Start ./scripts/run_db_seeder_single.sh

suggest using shell default values allowing user to override:
export DB_SEEDER_CONNECTION_PORT=${DB_SEEDER_CONNECTION_PORT:-3306}

Feature request, can there be an environment way to specify arguments/settings to the container (defaulting to utf8?)?
--character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci (is this required?)

src/main/java/ch/konnexions/db_seeder/jdbc/mariadb/MariadbSeeder.java the initialization could be replaced with:
diff --git a/scripts/run_db_seeder_setup_dbms.sh b/scripts/run_db_seeder_setup_dbms.sh
index 096bc61..c47cbab 100755
--- a/scripts/run_db_seeder_setup_dbms.sh
+++ b/scripts/run_db_seeder_setup_dbms.sh
@@ -387,18 +387,21 @@ if [ "${DB_SEEDER_DBMS_DB}" = "mariadb" ]; then
     echo "Docker create db_seeder_db (MariaDB Server ${DB_SEEDER_VERSION})"
 
     docker network create db_seeder_net  2>/dev/null || true
-    docker create -e        MYSQL_ROOT_PASSWORD=mariadb \
+    docker create -e        MARIADB_ROOT_PASSWORD="${DB_SEEDER_PASSWORD_SYS}" \
+                 -e        MARIADB_DATABASE="${DB_SEEDER_DATABASE}" \
+                 -e        MARIADB_USER="${DB_SEEDER_USER}" \
+                 -e        MARIADB_PASSWORD="${DB_SEEDER_PASSWORD}" \
                   --name    db_seeder_db \
                   --network db_seeder_net \
                   -p        "${DB_SEEDER_CONNECTION_PORT}":"${DB_SEEDER_CONTAINER_PORT}"/tcp \
-                  mariadb:"${DB_SEEDER_VERSION}" --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
+                  mariadb:"${DB_SEEDER_VERSION}" --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --port="${DB_SEEDER_CONNECTION_PORT}"
 
     echo "Docker start db_seeder_db (MariaDB Server ${DB_SEEDER_VERSION}) ..."
     if ! docker start db_seeder_db; then
         exit 255
     fi
 
    sleep 30

Once MDEV-25434 is done I hope the sleep 30 can be replaced. If there's any other MariaDB container features needed to work better please let me know (https://github.com/MariaDB/mariadb-docker/issues or https://jira.mariadb.org component Docker).

Testing with podman instead of docker with a alias docker=podman worked quite well too.

A useful feature would be a SQL statement, probably over information_schema and maybe table statistics columns that generates the JSON output from an existing schema.

Thanks for sharing


On Sun, Jun 6, 2021 at 1:58 AM Walter Weinmann <walter.at.konnexions@gmail.com> wrote:
We are happy to announce that version 2.9.0 of DBSeeder (https://github.com/KonnexionsGmbH/db_seeder) is now available for general use. DBSeeder also supports the latest version of MariaDB Server available on DockerHub (10.6.1).
_______________________________________________
Mailing list: https://launchpad.net/~maria-discuss
Post to     : maria-discuss@lists.launchpad.net
Unsubscribe : https://launchpad.net/~maria-discuss
More help   : https://help.launchpad.net/ListHelp