Troubleshooting MySQL: Encryption can’t find Master Key

If you’ve encountered the “Encryption can’t find master key” error while starting MySQL, you’re likely facing issues with the keyring plugin. I recently faced a similar issue on my server, and this blog walks through the steps I took to troubleshoot and fix it.

2024-07-10T08:58:38.337113Z 1 [ERROR] [MY-012657] [InnoDB] Encryption can't find master key, please check the keyring is loaded.
2024-07-10T08:58:38.337155Z 1 [ERROR] [MY-012226] [InnoDB] Encryption information in datafile: ./mysql_db/encrypted_table.ibd can't be decrypted, please confirm that keyring is loaded.
2024-07-10T09:15:41.979505Z 1 [ERROR] [MY-012179] [InnoDB] Could not find any file associated with the tablespace ID: 13.

Investigating MySQL Encryption Error

My immediate next steps was to confirm if selinux or anything else is blocking the access, but it wasn’t the case.

root@nitty_witty:~# getenforce
Permissive

There’s nothing in dmesg or syslog (/var/log/syslog | /var/log/messages) that could hint otherwise. Verified that the files are in place and have proper permissions.

root@nitty_witty:/data# grep ssl /etc/mysql/my.cnf
ssl-ca=/data/mysql/mysql-ssl/ca.pem
ssl-cert=/data/mysql/mysql-ssl/server-cert.pem
ssl-key=/data/mysql/mysql-ssl/server-key.pem
root@nitty_witty:/data#


root@nitty_witty:/data# ls -lhtrR mysql-ssl/
mysql-ssl/:
total 32K
-rwxr-x--- 1 mysql mysql 1.7K Jul 08  2024 server-key.pem
-rwxr-x--- 1 mysql mysql 1.1K Jul 08  2024 server-cert.pem
-rwxr-x--- 1 mysql mysql  451 Jul 08  2024 public_key.pem
-rwxr-x--- 1 mysql mysql 1.7K Jul 08  2024 private_key.pem
-rwxr-x--- 1 mysql mysql 1.7K Jul 08  2024 client-key.pem
-rwxr-x--- 1 mysql mysql 1.1K Jul 08  2024 client-cert.pem
-rwxr-x--- 1 mysql mysql 1.1K Jul 08  2024 ca.pem
-rwxr-x--- 1 mysql mysql 1.7K Jul 08  2024 ca-key.pem
root@nitty_witty:/data#

So, reread the error and analyzing what it’s saying:

Encryption error: MySQL couldn’t locate the encryption master key required to decrypt InnoDB tablespaces.
Tablespace error: Some tablespaces couldn’t be restored because they were encrypted, and without the master key, MySQL couldn’t access the data.

Since I see that the encryption keys are present and accessible, do we think somehow MySQL is not able to make sense of it? And because encryption key is not loaded, it couldn’t encrypt the tablespaces?

Wait… “encryption key is not loaded”?

What loads the encryption key? The keyring plugin…

are you getting it? Is it the case that keyring plugin is not loaded?

And then comes the golden lines from MySQL documentation:

A keyring component or plugin must be loaded early during the server startup sequence so that other components can access it as necessary during their own initialization. For example, the InnoDB storage engine uses the keyring for tablespace encryption, so a keyring component or plugin must be loaded and available prior to InnoDB initialization.

I added the following line in my.cnf and started MySQL successfully.

early-plugin-load=keyring_file.so

The issue occurred because I had dynamically enabled SSL and keyring encryption, which worked fine. But, I forgot to add this to the configuration / my.cnf file. This caused MySQL to fail during subsequent restart as the keyring plugin wasn’t being loaded before the InnoDB engine starts doing its thing.

Key Takeaways

Ensure to persist dynamic changes
It’s important to ensure that any dynamic changes are saved to the my.cnf configuration file. Failing to do so may result in the system working temporarily but failing on restart, as seen in my case.

Keyring Plugin needs early loading
The keyring plugin, responsible for managing encryption keys, must be loaded early during the MySQL startup process. If it’s not loaded in time, MySQL can’t access encrypted tables or files, causing startup failures.

SSL configuration
Even though SSL was properly configured in this case, verifying SSL settings and permissions is a good practice, especially when dealing with encrypted data.

Question for you

Do you think we should have early-plugin-load=keyring_file.so as default and always loaded? Why? Why not?

Conclusion

The MySQL startup issue I faced occurred because the keyring plugin was not loaded early enough in the startup process, preventing MySQL from accessing the encryption keys required to decrypt tablespaces. The root cause was my failure to persist the dynamic configuration in my.cnf.

early-plugin-load=keyring_file.so