Hi,
Maybe a transitional setting that would create/update both the "legacy" frm file and a file with the new format when required could be a solution for backward compatibility.
In that case, it would read the new format definition file if present.
Another possibility could be that we could choose the default format and only newly created tables would use that format but it wouldnt modify the existing ones (or maybe via an ALTER).
It also could be of use to have a tool that could offline convert the files between the two format in case MariaDB would have to be downgraded or replaced by MySQL or another fork that does not support the new definition file format.