We had similar issues and we went through a couple of exercises to determine how file descriptors work with both mysql and mariadb. Please correct me if I'm wrong...this is what I've seen based on a number of repeated tests.
For innodb, regardless of how many queries you have or any other parameters, the number of fds the daemon will open is bound strictly by innodb_open_files. It won't exceed this and it won't reduce either once you hit the max until you restart the daemon
For myisam, it;s much trickier...table_open_cache controls not the number of file descriptors but the number of "tables". So from a system level, the number of file descriptors open for a myisam table would be 2 X number of partitions (because each partition consists of a myi file and an myd file, unlike innodb which is a single ibd file). Furthermore, the number of copies of each table that mariadb/mysql open depend on the concurrency/joins. You can test this by writing a query that joins the same table 4 times..in this case the number of file descriptors open is 4 X 2 X number of partitions. The same holds for concurrency as well. If you have 3 very slow queries hitting the same myisam table, the daemon opens up 3 copies of each file belonging to that table..so a total of 3 X 2 X number of partitions. (for innodb, the number of copies of each file descriptor open is always 1...so the number of fds is always = number of innodb partitons).
Another caveat for myisam is that the number represents not the number of tables, but the total number of "copies" that are permissible to be open. So if table_open_cache is 10, you can have 10 different tables or 10 copies of the same table or a mix. table_open_cache doesn't care how many partitons there are, so if each table has a thousand partitions, then you can find that with a table_open_cache of 10, you have a maximum of 10,000 fds open. Also keep in mind that table_open_cache affects innodb tables as well as myisam tables (not sure if this is intended, but you can try this by setting table_open_cache to 1 and doing a show create table on some innodb table...it will close myisam fds)
You can test the number of copies open per table with a simple shell script or command
/usr/sbin/lsof -p `cat /var/run/mysqld/mysqld.pid` | sed 1d | awk '{ print $9 }' | cut -d "/" -f 5 | grep -v '.MYI' | sort -n | uniq -c | cut -d '.' -f 1 | cut -d '#' -f 1 | uniq -c | awk '{ print $1*$2,$0 }' | sort -n -r
first column is total number of fds, second column is number of partitions, 3rd column is the number of copies of each file.
Also, for myisam you can close open file descriptors by doing a "flush tables" but this is a little risky as we've seen unpredictable behavior when flush runs on a system under load...esp if tables are being repaired, it has a small chance of messing things up.
A safer but hackier option to reduce file descriptor count is to lower the value of table_open_cache..this is a dynamic variable and can be lowered, at which point the daemon eventually closes file descriptors until the max is reached (i tested this by continuously hitting the db with very quick and light queries..the fds kept getting closed until the num tables reached table_open_cache. I usually do a show create table on some innodb table)
Thanks,
Rohan