18 Jan

A tale of Corrupt InnoDB table, MySQL crash & recovery

I’m going to narrate you a story that happened around a crashing MyQL, Corrupted InnoDB table and finally the recovery by table restore. We will see how our database administrator detected the issue and what he did to resolve it.

A day in MySQL Database Consultant’s day was taking its shape while a friend called for help.

Friend: Hey, my mysql is crashing and website isn't functioning well. Everything is down. Can you help me?

Our database admin quickly jumps in and checks for the MySQL error log.

        2018-01-01T07:39:03.173398Z 0 [ERROR] InnoDB: Database page corruption on disk or a failed file read of page [page id: space=13701, page number=4603]. You may have to recover from a backup.
        2018-01-01T07:39:03.173428Z 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):
        InnoDB: End of page dump
        2018-01-01T07:39:03.265864Z 0 [Note] InnoDB: Uncompressed page, stored checksum in field1 4187651462, calculated checksums for field1: crc32 4128877936/1194051977, innodb 680941878, none 3735928559, stored checksum in field2 3735928559, calculated checksums for field2: crc32 4128877936/1194051977, innodb 1675940203, none 3735928559,  page LSN 400 3284265879, low 4 bytes of LSN at page end 3284252104, page number (if stored to page already) 4603, space id (if created with >= MySQL-4.1.1 and stored already) 13701
        InnoDB: Page may be an index page where index id is 33515
        2018-01-01T07:39:03.265911Z 0 [Note] InnoDB: Index 33515 is `PRIMARY` in table `nitty-witty`.`flat_address`
        2018-01-01T07:39:03.265919Z 0 [Note] InnoDB: It is also possible that your operating system has corrupted its own file cache and rebooting your computer removes the error. If the corrupt page is an index page. You can also try to fix the corruption by dumping, dropping, and reimporting the corrupt table. You can use CHECK TABLE to scan your table for corruption. Please refer to http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html for information about forcing recovery.
        2018-01-01T07:39:03.265944Z 0 [ERROR] [FATAL] InnoDB: Aborting because of a corrupt database page in the system tablespace. Or,  there was a failure in tagging the tablespace  as corrupt.
        2018-01-01 07:39:03 0x7f5fa0466700  InnoDB: Assertion failure in thread 140048687589120 in file ut0ut.cc line 916
        InnoDB: We intentionally generate a memory trap.
        InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
        InnoDB: If you get repeated assertion failures or crashes, even
        InnoDB: immediately after the mysqld startup, there may be
        InnoDB: corruption in the InnoDB tablespace. Please refer to
        InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
        InnoDB: about forcing recovery.
        07:39:03 UTC - mysqld got signal 6 ;
        This could be because you hit a bug. It is also possible that this binary
        or one of the libraries it was linked against is corrupt, improperly built,
        or misconfigured. This error can also be caused by malfunctioning hardware.
        Attempting to collect some information that could help diagnose the problem.
        As this is a crash and something is definitely wrong, the information
        collection process might fail.

        It is possible that mysqld could use up to
        key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 7539498 K  bytes of memory
        Hope that's ok; if not, decrease some variables in the equation.

        Thread pointer: 0x0
        Attempting backtrace. You can use the following information to find out
        where mysqld died. If you see no messages after this, something went
        terribly wrong...

Read More

09 Jan

Linux shell script: Pause (Suspend) a process with kill signals

Linux KILL command gives you killing opportunities to use SIGNALs. In this post we’re going to Pause (or suspend) and Start an ongoing Linux process.

Problem: The requirement here is to pause a process for certain time and run it only during off hours. The process itself doesn’t have this option and also we cannot stop it in-between, as it will start from the scratch again.

Solution: To resolve this we can think of sending out signals to the process to hint it to Pause or Progress.

A signal is an asynchronous notification sent to a process or to a specific thread within the same process in order to notify it of an event that occurred. When a signal is sent, the operating system interrupts the target process’ normal flow of execution to deliver the signal.

Read More

14 Dec

Decoding Siebel Audit Trail with PLSQL

In this post we will understand how to decode Seibel Audit Trail using PLSQL.

Siebel Audit Trail is feature in Siebel CRM which allows customer to track changes to important UI fields. Though it is very useful feature, the changes are stored in encoded text in DB. We are aiming to decode this complex data and allow user to extract complete Audit Trail information in files or tables. This can be further used to enhance customer experience.

This extract process is written completely in PL-SQL procedure. Similarly a script based solution would follow. This PL SQL procedure is tested with version 8.0 & 8.1. It should also be applicable for newer versions as well.
So let’s get started with this. We will go step by step and understand the purpose of this code while going along with a scenario.

WHAT is Audit Trail?

Read More

11 Apr

Reset root password on Galera Cluster (Percona or MariaDB)

We have known methods to reset MySQL root password but in this post we’ll see how that works for a Galera Cluster; may it be Percona XtraDB Cluster or MariaDB Galera cluster.

Basically a Galera Cluster is bunch of MySQL nodes working together in synchronous replication. The user credential is common to all (Ideally of-course)! Let’s see how straight  forward it is to reset password on a Percona Xtradb Cluster lab.

To reset the root password on a standard MySQL instance you may choose either of init-file or skip-grant-tables way. Read More

02 Mar

MySQL to Galera Cluster Migration, Deadlock, Back to basics

This post is a lab experiment learning from migration to the Percona Xtradb Cluster (Galera) and a very unexpected DEADLOCK scenario which took me back to basics.

(root@localhost) [test]>insert into app values (1);
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

Little background:

We needed to work on migrating the standard MySQL replication to a Galera (Percona Xtradb Cluster).

Previous configuration being one Master with two slaves detailed as follows:
server-1: MySQL community 5.6 serving as Master
server-2: MySQL community 5.6 serving as Slave
server-3: MySQL community 5.6 serving as Slave

We wanted to migrate above MySQL standard replication setup to Percona Xtradb Cluster (PXC 5.6) as following 3 nodes:

mysql to galera migration

Read More

-- Kedar Vaijanapurkar --