Bugs in database management systems can be a nightmare for administrators and developers. In my recent experiments, I discovered a Signal 11 bug in multiple versions of MariaDB, resulting in mysqld process crashes. This blog post delves into the implications of this bug, providing valuable insights and understanding.
Unverified variables and experiment
I was reviewing the bug MDEV-31470, which focuses on the “wsrep_sst_method” variable accepting literally any value. This variable was supposed to accept only specified set of values but it doesn’t. It seems that there’s a lack of adequate value validation, enabling the assignment of potentially incorrect values.
I was curious and decided to see what else is really going unchecked! Using a loop in the command line, I attempted to set each database variable to an unexpected value, “change_is_inevitable,” and observed the outcome. The experiment involved testing multiple versions of MariaDB (10.4, 10.6, 11.2), Percona Server 8, and MySQL Community 8.
# Save commands for variable assignment to a wrong value in a file
mysql -BNe "show variables" | awk -F' ' '{print "set global " $1 "=change_is_inevitable;"}' > db.vars
# Execute the SQLs from file, one by one, verifying status
while read -r line; do echo "trying … $line"; echo $line | mysql; mysqladmin ping | grep -q "mysqld is alive" || (echo " #### looks like mysqld crashed @ $line" && sleep 5); echo; sleep 0.1; done < db.varsResults of the experiment
MySQL Community 8.0.33
We saw many variable getting the wrong value assigned. That said, MySQL was still running. It responded to queries, new connections etc.
| admin_ssl_ca | change_is_inevitable |
| admin_ssl_capath | change_is_inevitable |
| admin_ssl_cert | change_is_inevitable |
| admin_ssl_cipher | change_is_inevitable |
| admin_ssl_crl | change_is_inevitable |
| admin_ssl_crlpath | change_is_inevitable |
| admin_ssl_key | change_is_inevitable |
| admin_tls_ciphersuites | change_is_inevitable |
| general_log_file | change_is_inevitable |
| init_connect | change_is_inevitable |
| init_replica | change_is_inevitable |
| init_slave | change_is_inevitable |
| innodb_buffer_pool_filename | change_is_inevitable |
| mandatory_roles | change_is_inevitable |
| session_track_system_variables | change_is_inevitable |
| slow_query_log_file | change_is_inevitable |
| ssl_ca | change_is_inevitable |
| ssl_capath | change_is_inevitable |
| ssl_cert | change_is_inevitable |
| ssl_cipher | change_is_inevitable |
| ssl_crl | change_is_inevitable |
| ssl_crlpath | change_is_inevitable |
| ssl_key | change_is_inevitable |
| tls_ciphersuites | change_is_inevitable |Percona Server 8.0.33
Following variables accepted the wrong value. Also: TIL version = concat(version, version_comment).
| admin_ssl_ca | change_is_inevitable |
| admin_ssl_capath | change_is_inevitable |
| admin_ssl_cert | change_is_inevitable |
| admin_ssl_cipher | change_is_inevitable |
| admin_ssl_crl | change_is_inevitable |
| admin_ssl_crlpath | change_is_inevitable |
| admin_ssl_key | change_is_inevitable |
| admin_tls_ciphersuites | change_is_inevitable |
| buffered_error_log_filename | change_is_inevitable |
| general_log_file | change_is_inevitable |
| init_connect | change_is_inevitable |
| init_replica | change_is_inevitable |
| init_slave | change_is_inevitable |
| innodb_buffer_pool_filename | change_is_inevitable |
| mandatory_roles | change_is_inevitable |
| session_track_system_variables | change_is_inevitable |
| slow_query_log_file | change_is_inevitable |
| ssl_ca | change_is_inevitable |
| ssl_capath | change_is_inevitable |
| ssl_cert | change_is_inevitable |
| ssl_cipher | change_is_inevitable |
| ssl_crl | change_is_inevitable |
| ssl_crlpath | change_is_inevitable |
| ssl_key | change_is_inevitable |
| tls_ciphersuites | change_is_inevitable |
| validate_password.dictionary_file | change_is_inevitable |
| version | 8.0.33-25change_is_inevitable |
| version_comment | change_is_inevitable |
| version_suffix | change_is_inevitable |Introducing Signal 11 Crashes
MariaDB 10.4.30
This version causes MySQL to crash when we try setting a value to the variable. After my for-loop caused a crash I manually connected and ran this experiment. MySQL crashes after changing the variable session_track_system_variables while establishing a new connection!
MariaDB [(none)]> set global session_track_system_variables=change_is_inevitable;
ERROR 1193 (HY000): Unknown system variable 'change_is_inevitable'
MariaDB [(none)]> \s
--------------
mysql Ver 15.1 Distrib 10.4.30-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
Connection id: 1437
Current database:
Current user: root@localhost
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server: MariaDB
Server version: 10.4.30-MariaDB-1:10.4.30+maria~ubu2004-log mariadb.org binary distribution
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: utf8mb4
Db characterset: utf8mb4
Client characterset: utf8mb4
Conn. characterset: utf8mb4
UNIX socket: /var/run/mysqld/mysqld.sock
Uptime: 20 hours 24 min 12 sec
Threads: 7 Questions: 2178 Slow queries: 0 Opens: 31 Flush tables: 1 Open tables: 25 Queries per second avg: 0.029
--------------
MariaDB [(none)]> show global variables like 'session_track_system_variables';
+--------------------------------+-------+
| Variable_name | Value |
+--------------------------------+-------+
| session_track_system_variables | |
+--------------------------------+-------+
1 row in set (0.001 sec)
MariaDB [(none)]> exit
Bye
root@ip-172-31-92-119:~# mysql
ERROR 2013 (HY000): Lost connection to MySQL server at 'handshake: reading initial communication packet', system error: 11
root@ip-172-31-92-119:~## Stack trace
Thread pointer: 0x7f61ac00f4a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f61e8d21dd8 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x32)[0x555e3fe43252]
/usr/sbin/mysqld(handle_fatal_signal+0x55d)[0x555e3f8d676d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f61f514e420]
/lib/x86_64-linux-gnu/libc.so.6(+0x1886e5)[0x7f61f4d976e5]
/usr/sbin/mysqld(my_strdup+0x1d)[0x555e3fe3f72d]
/usr/sbin/mysqld(_ZN23Session_sysvars_tracker4initEP3THD+0x28)[0x555e3f60ec38]
/usr/sbin/mysqld(_Z18plugin_thdvar_initP3THD+0x1d2)[0x555e3f6da512]
/usr/sbin/mysqld(_ZN3THD4initEv+0x30)[0x555e3f67ac90]
/usr/sbin/mysqld(_ZN3THD11change_userEv+0x7c)[0x555e3f67b07c]
/usr/sbin/mysqld(_ZN3THD15reset_for_reuseEv+0x1e)[0x555e3f67b1fe]
/usr/sbin/mysqld(_ZN7CONNECT10create_thdEP3THD+0x28)[0x555e3f7cc6e8]
/usr/sbin/mysqld(_Z29one_thread_per_connection_endP3THDb+0x29d)[0x555e3f5f932d]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x16b)[0x555e3f7cca4b]
/usr/sbin/mysqld(handle_one_connection+0x3f)[0x555e3f7ccc2f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f61f5142609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f61f4d2e133]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): (null)
Connection ID (thread ID): 1437
Status: NOT_KILLED
This also failed in MariaDB 10.6 and latest MariaDB 11 and I have posted the details in the “post script” section.
Here is the bug report: https://jira.mariadb.org/browse/MDEV-31534
TL;DR
Setting session_track_system_variables to an out of range values will crash your MariaDB.
Do NOT run on MariaDB:
SET GLOBAL session_track_system_variables=crash_is_inevitable;
Conclusion
The discovery of this bug highlights the importance of thorough validation and error handling within database management systems. Variables should be designed with stringent checks to prevent potential crashes and ensure system stability. By exposing this vulnerability, I have raised awareness of the issue and initiated the bug reporting process.
Robust variable validation mechanisms must be implemented to protect against similar vulnerabilities in the future. As administrators and developers, it is crucial for us to remain vigilant and actively contribute to improving the reliability and security of our database systems.
PS
MariaDB 10.6.5
[root@centos_2 ~]# mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 7
Server version: 10.6.5-MariaDB-log MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> set global session_track_system_variables=change_is_inevitable;
ERROR 1193 (HY000): Unknown system variable 'change_is_inevitable'
MariaDB [(none)]> exit
Bye
[root@centos_2 ~]# mysql
ERROR 2013 (HY000): Lost connection to server at 'handshake: reading initial communication packet', system error: 11
[root@centos_2 ~]#
# Stack trace from error log
Thread pointer: 0x7f178c000a98
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f17ac173cc0 thread_stack 0x49000
??:0(my_print_stacktrace)[0x561e31791a4e]
??:0(handle_fatal_signal)[0x561e311e1e27]
sigaction.c:0(__restore_rt)[0x7f17c48c5630]
:0(__strlen_sse2_pminub)[0x7f17c3e498c1]
??:0(my_strdup)[0x561e3178e07c]
??:0(Session_sysvars_tracker::init(THD*))[0x561e30f05f66]
??:0(THD::init())[0x561e30f6ee9d]
??:0(THD::change_user())[0x561e30f6f34e]
??:0(THD::reset_for_reuse())[0x561e30f6f569]
??:0(CONNECT::create_thd(THD*))[0x561e310bf026]
??:0(do_handle_one_connection(CONNECT*, bool))[0x561e310bf425]
??:0(handle_one_connection)[0x561e310bf614]
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x561e31421bb2]
pthread_create.c:0(start_thread)[0x7f17c48bdea5]
??:0(__clone)[0x7f17c3dd8b0d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): (null)
Connection ID (thread ID): 7
Status: NOT_KILLEDMariaDB 11
MariaDB [(none)]> set global session_track_system_variables=change_is_inevitable;
ERROR 1193 (HY000): Unknown system variable 'change_is_inevitable'
MariaDB [(none)]> exit;
...
root@ip-172-31-92-119:~# mysql
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
ERROR 2013 (HY000): Lost connection to server at 'handshake: reading initial communication packet', system error: 11
# stack trace from error log
Thread pointer: 0x7f923c000fb8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f92404f7dd8 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x32)[0x556cf07df252]
/usr/sbin/mysqld(handle_fatal_signal+0x55d)[0x556cf027276d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f924c924420]
/lib/x86_64-linux-gnu/libc.so.6(+0x1886e5)[0x7f924c56d6e5]
/usr/sbin/mysqld(my_strdup+0x1d)[0x556cf07db72d]
/usr/sbin/mysqld(_ZN23Session_sysvars_tracker4initEP3THD+0x28)[0x556ceffaac38]
/usr/sbin/mysqld(_Z18plugin_thdvar_initP3THD+0x1d2)[0x556cf0076512]
/usr/sbin/mysqld(_ZN3THD4initEv+0x30)[0x556cf0016c90]
/usr/sbin/mysqld(_ZN3THD11change_userEv+0x7c)[0x556cf001707c]
/usr/sbin/mysqld(_ZN3THD15reset_for_reuseEv+0x1e)[0x556cf00171fe]
/usr/sbin/mysqld(_ZN7CONNECT10create_thdEP3THD+0x28)[0x556cf01686e8]
/usr/sbin/mysqld(_Z29one_thread_per_connection_endP3THDb+0x29d)[0x556ceff9532d]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x16b)[0x556cf0168a4b]
/usr/sbin/mysqld(handle_one_connection+0x3f)[0x556cf0168c2f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f924c918609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f924c504133]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): (null)
Connection ID (thread ID): 1274
Status: NOT_KILLED
3 comments
Interesting. I was not aware of this
Copying conclusion from ChatGPT…
Ah, you caught me red-handed! Turns out even AI models have a way with words. But hey, it’s all about leveraging the best resources available, right? Let’s call it a collaboration between human brilliance and AI assistance. So, what are your thoughts on the article itself? I promise, the rest of it is 100% original and filled with insights you won’t want to miss!
(Even this response is from ChatGPT, but c’mon, that’s not the point of this article you wanted to comment on.)