{"id":3024,"date":"2023-09-05T10:08:34","date_gmt":"2023-09-05T10:08:34","guid":{"rendered":"http:\/\/kedar.nitty-witty.com\/?p=3024"},"modified":"2026-03-30T04:15:49","modified_gmt":"2026-03-30T04:15:49","slug":"how-to-percona-xtradb-cluster-8-0-upgrade-issues","status":"publish","type":"post","link":"https:\/\/kedar.nitty-witty.com\/blog\/how-to-percona-xtradb-cluster-8-0-upgrade-issues","title":{"rendered":"How to fix Percona XtraDB Cluster 8.0 Upgrade issues"},"content":{"rendered":"\n<p>Are you planning an upgrade for your Percona XtraDB Cluster (PXC)? Upgrading to PXC 8.0  can be a smooth process, but sometimes challenges arise that require careful troubleshooting. In this blog post, we will talk about 2 specific issues that you might face during an upgrade of Percona XtraDB Cluster or Galera. <\/p>\n\n\n\n<p>It is possible that you might already know of these as it&#8217;s been already been blogged about but consider this as a refresher.<\/p>\n\n\n<div class=\"wp-block-image is-resized\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"832\" src=\"http:\/\/kedar.nitty-witty.com\/wp-content\/uploads\/2023\/09\/image-1024x832.png\" alt=\"Percona XtraDB Cluster issues\" class=\"wp-image-3033\" srcset=\"https:\/\/kedar.nitty-witty.com\/blog\/wp-content\/uploads\/2023\/09\/image-1024x832.png 1024w, https:\/\/kedar.nitty-witty.com\/blog\/wp-content\/uploads\/2023\/09\/image-300x244.png 300w, https:\/\/kedar.nitty-witty.com\/blog\/wp-content\/uploads\/2023\/09\/image-768x624.png 768w, https:\/\/kedar.nitty-witty.com\/blog\/wp-content\/uploads\/2023\/09\/image-390x317.png 390w, https:\/\/kedar.nitty-witty.com\/blog\/wp-content\/uploads\/2023\/09\/image-820x667.png 820w, https:\/\/kedar.nitty-witty.com\/blog\/wp-content\/uploads\/2023\/09\/image.png 1026w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<\/div>\n\n\n<h2 class=\"wp-block-heading\">Upgrading from PXC 5.7 to PXC 8.0<\/h2>\n\n\n\n<p>Upgrading from PXC 5.7 to PXC 8.0 or Galera MySQL 8 typically follows a straightforward plan:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Backup Cluster Data:<\/strong> Begin by backing up your cluster data from one node to ensure data safety.<\/li>\n\n\n\n<li><strong>Shutdown Cluster Nodes:<\/strong> Temporarily shut down all cluster nodes to prepare for the upgrade.<\/li>\n\n\n\n<li><strong>Perform MySQL Upgrade:<\/strong> Upgrade MySQL on each node one at a time to the target version.<\/li>\n\n\n\n<li><strong>Bootstrap the Cluster:<\/strong> After upgrading, it&#8217;s time to bootstrap the cluster.<\/li>\n<\/ul>\n\n\n\n<p><\/p>\n\n\n\n<p>Note that you need to be prepared well and have performed proper upgrade testing before you migrate to the newer version. The Percona tool, pt-upgrade and MySQL&#8217;s upgrade checker utility are two amazing tools to help you with upgrading to MySQL 8. This blog is not exactly about How To upgrade but about the issues faced during an upgrade process.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Issue 1 &#8211; Bootstrapping PXC doesn&#8217;t bring up the cluster node<\/h2>\n\n\n\n<p>One common issue encountered during the upgrade is related to bootstrapping the Percona XtraDB Cluster. After upgrading PXC binaries and executing the bootstrap command the cluster not failed to start:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>root@production: mysql # systemctl start mysql@bootstrap\nJob for mysql@bootstrap.service failed because a timeout was exceeded. See \"systemctl status mysql@bootstrap.service\" and \"journalctl -xe\" for details.<\/code><\/pre>\n\n\n\n<p>For Galera Cluster node similar error may also appear while joining a cluster as follows:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Job for mysql.service failed because a timeout was exceeded<\/h3>\n\n\n\n<h6 class=\"wp-block-heading\">MySQL error log<\/h6>\n\n\n\n<pre class=\"wp-block-code\"><code>2023-07-13T09:24:04.598299Z 0 &#91;System] &#91;MY-011323] &#91;Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: \/var\/lib\/mysql\/mysqlx.sock\n2023-07-13T09:24:04.598348Z 0 &#91;System] &#91;MY-010931] &#91;Server] \/usr\/sbin\/mysqld: ready for connections. Version: '8.0.32-24.2' socket: '\/var\/lib\/mysql\/mysql.sock' port: 3306 Percona XtraDB\nCluster (GPL), Release rel24, Revision 2119e75, WSREP version 26.1.4.3.\n2023-07-13T09:24:04.598439Z 0 &#91;System] &#91;MY-013172] &#91;Server] Received SHUTDOWN from user . Shutting down mysqld (Version: 8.0.32-24.2).\n2023-07-13T09:24:04.598506Z 0 &#91;Note] &#91;MY-000000] &#91;WSREP] Received shutdown signal. Will sleep for 10 secs before initiating shutdown. pxc_maint_mode switched to SHUTDOWN\n\u2026\n2023-07-13T10:14:43.389622Z 0 &#91;Note] &#91;MY-000000] &#91;Galera] announce period timed out (pc.announce_timeout)\n\u2026\n2023-07-13T10:15:13.402826Z 0 &#91;ERROR] &#91;MY-000000] &#91;Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out)\nat gcomm\/src\/pc.cpp:connect():161\n2023-07-13T10:15:13.402836Z 0 &#91;ERROR] &#91;MY-000000] &#91;Galera] gcs\/src\/gcs_core.cpp:gcs_core_open():219: Failed to open backend connection: -110 (Connection timed out)\n2023-07-13T10:15:14.403073Z 0 &#91;Note] &#91;MY-000000] &#91;Galera] gcomm: terminating thread\n2023-07-13T10:15:14.403130Z 0 &#91;Note] &#91;MY-000000] &#91;Galera] gcomm: joining thread\n2023-07-13T10:15:14.403354Z 0 &#91;ERROR] &#91;MY-000000] &#91;Galera] gcs\/src\/gcs.cpp:gcs_open():1811: Failed to open channel 'prod-db' at 'gcomm:\/\/10.10.10.10,10.10.10.11,10.10.10.12': -110 (Con\nnection timed out)\n2023-07-13T10:15:14.403381Z 0 &#91;ERROR] &#91;MY-000000] &#91;Galera] gcs connect failed: Connection timed out\n2023-07-13T10:15:14.403401Z 0 &#91;ERROR] &#91;MY-000000] &#91;WSREP] Provider\/Node (gcomm:\/\/10.10.10.10,10.10.10.11,10.10.10.12) failed to establish connection with cluster (reason: 7)\n2023-07-13T10:15:14.403417Z 0 &#91;ERROR] &#91;MY-010119] &#91;Server] Aborting<\/code><\/pre>\n\n\n\n<h6 class=\"wp-block-heading\">The kernel log<\/h6>\n\n\n\n<pre class=\"wp-block-code\"><code>\n&#91;prod] percona@production: mysql@bootstrap.service.d $ sudo grep bootstrap \/var\/log\/messages | grep -v pmm\n...\nJul 13 10:37:15 production systemd: Starting Percona XtraDB Cluster with config \/etc\/sysconfig\/mysql.bootstrap...\nJul 13 10:38:45 production systemd: <strong>mysql@bootstrap.service start-pre operation timed out. Terminating<\/strong>.\nJul 13 10:38:51 production systemd: <strong>Failed to start Percona XtraDB Cluster<\/strong> with config \/etc\/sysconfig\/mysql.bootstrap.\nJul 13 10:38:51 production systemd: <strong>Unit mysql@bootstrap.service entered failed state.<\/strong>\n<\/code><\/pre>\n\n\n\n<p>Observing carefully all of them points to &#8220;timeout&#8221;. The Percona XtraDB Cluster couldn&#8217;t start before TimeoutStartSec (90s) and hence it was terminated.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Fixing Percona XtraDB Cluster Timeout Error<\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>systemctl edit mysql@bootstrap \n(OR systemctl edit mysql.service #as per your service definition)\n&#91;Service]\nTimeoutStartSec=6000\n\n$ systemctl daemon-reload\n\n$ systemctl show mysql.service -p TimeoutStartUSec\nTimeoutStartUSec=1h 40min<\/code><\/pre>\n\n\n\n<p>After the change in service config, MySQL (Percona XtraDB Cluster node) started cleanly with the bootstrap command.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Issue 2 &#8211; PXC Cluster node doesn&#8217;t join cluster<\/h2>\n\n\n\n<p>To get the other nodes join the bootstrapped Percona XtraDB Cluster<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;13\/07\/2023, 4:58:24,911 PM] &#91;prod] root@production: mysql # systemctl start mysql<br>&#91;13\/07\/2023, 4:58:24,912 PM] Job for mysql.service failed because the control process exited with error code. See \"systemctl status mysql.service\" and \"journalctl -xe\" for details.<\/code><\/pre>\n\n\n\n<p>The error output has following hints<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;Galera] Handshake failed: <strong>tlsv1 alert unknown ca<\/strong>\n\n&#91;Galera] handshake with remote endpoint ssl:\/\/10.0.0.1:4567 failed: asio.ssl:336031996: 'unknown protocol' ( 336031996: '<strong>error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol<\/strong>')<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>2023-07-13T13:04:13.369141Z 0 &#91;System] &#91;MY-010116] &#91;Server] \/usr\/sbin\/mysqld (mysqld 8.0.32-24.2) starting as process 8662\n2023-07-13T13:04:13.381273Z 0 &#91;ERROR] &#91;MY-000059] &#91;Server] <strong>SSL error: Unable to get private key from 'server-key.pem'.<\/strong>\n2023-07-13T13:04:13.381316Z 0 &#91;Warning] &#91;MY-013595] &#91;Server] <strong>Failed to initialize TLS for channel<\/strong>: mysql_main. See below for the description of exact issue.\n2023-07-13T13:04:13.381333Z 0 &#91;Warning] &#91;MY-010069] &#91;Server] <strong>Failed to set up SSL because of the following SSL library error: Unable to get private key<\/strong>\n2023-07-13T13:04:13.381340Z 0 &#91;Note] &#91;MY-000000] &#91;WSREP] New joining cluster node configured to use specified SSL artifacts\n2023-07-13T13:04:13.381379Z 0 &#91;Note] &#91;MY-000000] &#91;Galera] Loading provider \/usr\/lib64\/galera4\/libgalera_smm.so initial position: f0bc6f23-11bd-11ec-a9f8-8f101bd9a1be:2783648882\n2023-07-13T13:04:13.381394Z 0 &#91;Note] &#91;MY-000000] &#91;Galera] wsrep_load(): loading provider library '\/usr\/lib64\/galera4\/libgalera_smm.so'<\/code><\/pre>\n\n\n\n<p>This is a reminder that Percona XtraDB Cluster 8.0 by-default enables the <code>pxc_encrypt_cluster_traffic<\/code> and with that encrypting SST, IST, and replication traffic.<\/p>\n\n\n\n<p>Configuring <code>pxc-encrypt-cluster-traffic<\/code> is non-dynamic change and will require MySQL restart. It is important to ensure verifying this change while preparing for upgrades for seamless experience.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Fixing SSL Handshake Failures in PXC 8.0<\/h2>\n\n\n\n<p>You may either disable the pxc-encrypt-cluster-traffic mode (joking) OR preferably create \/ use the existing pem files to ship across the cluster. <\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Bootstrap the Percona XtraDB Cluster node with SSL keys.<\/li>\n\n\n\n<li>Transfer key files from the bootstrapped node to the remaining cluster nodes.<\/li>\n\n\n\n<li>Start the remaining nodes to have them join the cluster successfully.<\/li>\n<\/ul>\n\n\n\n<p>There&#8217;s an extensive blog I recommend reading <a href=\"https:\/\/www.percona.com\/blog\/percona-xtradb-cluster-8-0-behavior-change-for-pxc-encrypt-cluster-traffic\/\" target=\"_blank\" rel=\"noopener nofollow\" title=\"here\">by Marco on Percona Blog<\/a>.<\/p>\n\n\n\n<p>Alright, there ends this reminder blog for the upgrade to Percona XtraDB Cluster for MySQL 8.0.  But before you go, there is one other issues at MySQL 8 upgrade that you must not forget and one of such corner case is about <a href=\"https:\/\/kedar.nitty-witty.com\/blog\/mysql-8-timestamp-cannot-be-null-and-explicit_defaults_for_timestamp\" target=\"_blank\" rel=\"noopener\" title=\"\">default for timestamp columns change<\/a> in MySQL 8. This is an interesting use-case of handling an issue after the upgrade is done.<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"Are you planning an upgrade for your Percona XtraDB Cluster (PXC)? Upgrading to PXC 8.0 can be a smooth process, but sometimes challenges arise that require careful troubleshooting. In this&hellip;\n","protected":false},"author":1,"featured_media":3033,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[864,8,377,865],"tags":[689,686,688,810,809,687,383,690,1175,1174,684],"class_list":{"0":"post-3024","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-mysql-galera-cluster","8":"category-mysql","9":"category-mysql-articles","10":"category-percona-xtradb-cluster","11":"tag-bootstrap","12":"tag-connection-timed-out","13":"tag-fixing-percona-xtradb-cluster","14":"tag-job-for-mysql-service-failed-because-a-timeout-was-exceeded","15":"tag-mysqlbootstrap-service-failed-because-a-timeout-was-exceeded","16":"tag-operation-timed-out","17":"tag-percona-xtradb-cluster","18":"tag-pxc-bootstrap-error","19":"tag-pxc_encrypt_cluster_traffic","20":"tag-pxc-encrypt-cluster-traffic","21":"tag-upgrade-mysql-8"},"aioseo_notices":[],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/posts\/3024","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/comments?post=3024"}],"version-history":[{"count":12,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/posts\/3024\/revisions"}],"predecessor-version":[{"id":3559,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/posts\/3024\/revisions\/3559"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/media\/3033"}],"wp:attachment":[{"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/media?parent=3024"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/categories?post=3024"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kedar.nitty-witty.com\/blog\/wp-json\/wp\/v2\/tags?post=3024"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}