Recently, we were affected by an
ignored configuration option introduced in MySQL 5.6. This incident
caused us to perform extended planning of downtime and the rebuilding of
the slave. In this post, we’ll discuss our encounter with
binlog_error_action and likely bad default.
The incident started with an alert
from our monitoring platform that a check failed to verify a MySQL
master’s binary log coordinates. My colleague working from the other
side of the globe observed the following:
mysql> show master status;
Empty set (0.00 sec)
mysql> show binary logs;
ERROR 1381 (HY000): You are not using binary logging
Interestingly, the MySQL slave was still reporting that everything was okay!
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: master
Master_User: replicator
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000186
Read_Master_Log_Pos: 1069066810
Relay_Log_File: mysqld-relay-bin.000245
Relay_Log_Pos: 1069067004
Relay_Master_Log_File: mysql-bin.000186
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 1069066810
Relay_Log_Space: 1069067296
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 23294781
Master_UUID: 7e2e60eb-39fb-11ea-a248-005056b589fd
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set: 7e2e60eb-39fb-11ea-a248-005056b589fd:50961813-81101493
Executed_Gtid_Set: 2366bbb5-39fb-11ea-a246-005056b5f82f:1-818,
7e2e60eb-39fb-11ea-a248-005056b589fd:1-81101493
Auto_Position: 1
1 row in set (0.00 sec)
Guess what my colleague saw in the error log?
Read More →