Hi Daniel The original goal of this project was the generation of large amounts of anonymous and random data for relational database systems useful e.g. for stress testing. I am really pleased with your reaction to my announcement of DBSeeder version 2.9.0. I am now working on version 2.9.1 (see here: https://github.com/KonnexionsGmbH/db_seeder/tree/wwe_2.9.1) and will try to incorporate as much of your suggestions and issue reports as possible. *1. Compilation with OpenJDK 16* Unfortunately, I can't reproduce the problem. I am also of the opinion that I run OpenJDK 16 both under Windows 10: *openjdk version "16.0.1" 2021-04-20* *OpenJDK Runtime Environment (build 16.0.1+9-24)* *OpenJDK 64-Bit Server VM (build 16.0.1+9-24, mixed mode, sharing)* as well as under Ubuntu 20.04: *openjdk version "16.0.1" 2021-04-20* *OpenJDK Runtime Environment (build 16.0.1+9-24)* *OpenJDK 64-Bit Server VM (build 16.0.1+9-24, mixed mode, sharing)* The compilation should be done in the root directory of the db_seeder repository with the following command: *gradle copyJarToLib* *2. Docker Image Size* I created a custom image for the development and operation of DBSeeder. The size of the image is now only 1.73 GB. Details can be found here: https://hub.docker.com/repository/docker/konnexionsgmbh/db_seeder. *3. README.md* I have restructured this document on your suggestion with the intention to present DBSeeder more clearly. *4. Docker Compose* I removed Docker Compose, which doesn't make much sense in combination with DBSeeder. *5. line 13 in script run_db_seeder.sh* I have corrected this issue. *6. Login Docker Hub* I removed the unnecessary login to Docker Hub. It seems that at some point I misinterpreted the problems described here: https://github.com/docker/hub-feedback/issues/1103. *7. Defaults in script run_db_seeder_single.sh* *run_db_seeder_single* is just an auxiliary script used by the script *run_db_seeder* and is not called directly (see: README.md diagram in section 4.1.2). Therefore the script variables in *run_db_seeder_single* have no default values unlike in *run_db_seeder*. The easiest way to customize these environment variables is directly in the *run_db_seeder* script. *8. Feature request* I added the following two environment variables in the *run_db_seeder* script specifically for MariaDB: DB_SEEDER_CHARACTER_SET_SERVER=utf8mb4 DB_SEEDER_COLLATION_SERVER=utf8mb4_unicode_ci *9. MDEV-25434* When this extension is available I will gladly switch to health check on MariaDB (similar to Apache Derby for example). *10. Podman* I think there is no need for action on the part of DBSeeder? *11. JSON output* Here the effort seems to me too large, since the realization would have to take place DBMS specifically and this would go beyond the bounds with the currently 24 supported systems. In this context DBSeeder offers two functionalities: 1. the database schema is generated from a JSON file - see in README.md section 2 and the files in the *resources/json* directory. 2. runtime statistics are produced in tsv format - see README.md section 4.4 and files in *resources/statistics* directory --- I hope that I have understood your suggestions and objections correctly and have offered satisfactory solutions. If you notice anything else then just let me know. I plan to release version 2.9.1 by the end of June at the latest. It will also include all updates of the used DBMS Docker images and JDBC drivers. Thanks again and best regards Walter On Mon, 7 Jun 2021 at 05:50, Daniel Black <daniel@mariadb.org> wrote:
Thanks Walter for letting everyone know.
I did play around with it as it might be useful in attempting to replicate sample data for users experiencing problems.
What was your intended purpose when creating this?
Suggestions/minor problems that I came up with testing it:
It doesn't compile with openjdk and I needed to install oraclejdk to test this. This wasn't documented (except .travis.yml that tipped me off).
docker.io/konnexionsgmbh/kxn_dev is a very big image - 7.56G
README.md - the database list in section 1 - was moved into the 1.1 table this would be easier to read. - Chines typo - in reference to docker-compose.yml needs to document creating Transfer directory
run_db_seeder.sh:
line 13 - is this meant to be "rm -rf $PWD/tmp/* || sudo rm -rf $PWD/tmp/*"?
the docker username/password seem to be optional
It's unclear what username/password this is referring to in the output: Username: Password: Login Succeeded!
================================================================================ Start ./scripts/run_db_seeder_single.sh
suggest using shell default values allowing user to override: export DB_SEEDER_CONNECTION_PORT=${DB_SEEDER_CONNECTION_PORT:-3306}
Feature request, can there be an environment way to specify arguments/settings to the container (defaulting to utf8?)? --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci (is this required?)
src/main/java/ch/konnexions/db_seeder/jdbc/mariadb/MariadbSeeder.java the initialization could be replaced with: diff --git a/scripts/run_db_seeder_setup_dbms.sh b/scripts/run_db_seeder_setup_dbms.sh index 096bc61..c47cbab 100755 --- a/scripts/run_db_seeder_setup_dbms.sh +++ b/scripts/run_db_seeder_setup_dbms.sh @@ -387,18 +387,21 @@ if [ "${DB_SEEDER_DBMS_DB}" = "mariadb" ]; then echo "Docker create db_seeder_db (MariaDB Server ${DB_SEEDER_VERSION})"
docker network create db_seeder_net 2>/dev/null || true - docker create -e MYSQL_ROOT_PASSWORD=mariadb \ + docker create -e MARIADB_ROOT_PASSWORD="${DB_SEEDER_PASSWORD_SYS}" \ + -e MARIADB_DATABASE="${DB_SEEDER_DATABASE}" \ + -e MARIADB_USER="${DB_SEEDER_USER}" \ + -e MARIADB_PASSWORD="${DB_SEEDER_PASSWORD}" \ --name db_seeder_db \ --network db_seeder_net \ -p "${DB_SEEDER_CONNECTION_PORT}":"${DB_SEEDER_CONTAINER_PORT}"/tcp \ - mariadb:"${DB_SEEDER_VERSION}" --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci + mariadb:"${DB_SEEDER_VERSION}" --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci --port="${DB_SEEDER_CONNECTION_PORT}"
echo "Docker start db_seeder_db (MariaDB Server ${DB_SEEDER_VERSION}) ..." if ! docker start db_seeder_db; then exit 255 fi
sleep 30
Once MDEV-25434 is done I hope the sleep 30 can be replaced. If there's any other MariaDB container features needed to work better please let me know (https://github.com/MariaDB/mariadb-docker/issues or https://jira.mariadb.org component Docker).
Testing with podman instead of docker with a alias docker=podman worked quite well too.
A useful feature would be a SQL statement, probably over information_schema and maybe table statistics columns that generates the JSON output from an existing schema.
Thanks for sharing
On Sun, Jun 6, 2021 at 1:58 AM Walter Weinmann < walter.at.konnexions@gmail.com> wrote:
We are happy to announce that version 2.9.0 of DBSeeder ( https://github.com/KonnexionsGmbH/db_seeder) is now available for general use. DBSeeder also supports the latest version of MariaDB Server available on DockerHub (10.6.1). _______________________________________________ Mailing list: https://launchpad.net/~maria-discuss Post to : maria-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~maria-discuss More help : https://help.launchpad.net/ListHelp