Recently, we were affected by an
ignored configuration option introduced in MySQL 5.6. This incident
caused us to perform extended planning of downtime and the rebuilding of
the slave. In this post, we’ll discuss our encounter with
binlog_error_action and likely bad default.
The incident started with an alert
from our monitoring platform that a check failed to verify a MySQL
master’s binary log coordinates. My colleague working from the other
side of the globe observed the following:
mysql> show master status;
Empty set (0.00 sec)
mysql> show binary logs;
ERROR 1381 (HY000): You are not using binary logging
Interestingly, the MySQL slave was still reporting that everything was okay!
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
1 row in set (0.00 sec)
Guess what my colleague saw in the error log?