13 Aug

Galera cluster to AWS Aurora migration & HA_ERR_FOUND_DUPP_KEY

In this post we will see a case study of a Galera Cluster migration to AWS Aurora and quick solution to the replication issue.

A friend received an error in a Master-Master replication as follows:

Could not execute Write_rows event on table _database._table; Duplicate entry '65eJ8RmzASppBuQD2Iz73AAy8gPKIEmP-2018-08-03 08:30:03' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log mysql-bin-changelog.000010, end_log_pos 1107814

We talked about it and I immediately started blabbering cons of writing on both masters, how to handle, roles of apps and so on. He intervened and revealed, it is Galera Cluster replicated to Aurora and he is not writing on Aurora.

He was actually migrating his Galera Cluster to Aurora and master-master was to support his rollback plans. Consider following diagram for simplicity of understanding:

galera to aws migration – sample architecture

29 Jun

Restore A Table / Database From Full Backup – Yet Another Way

Restore requests are common and so are the restores of specific entities: a database, or one or more table(s). This has been discussed a lot and we have plenty of tools and solutions already available.
In this blog post we will cover an interesting solution that I came across when I received a restoration request from a client with a specific scenario.

The scenario? Well, the client was on a Windows server with 400GB of mysqldump and wanted to restore a table.

As Linux players we already know of some tools and techniques to export a table or database from mysqldump – for example, using sed command or using the script mysqldumpsplitter (based on sed itself). But on Windows we are powerless by not being able to use sed (we’re sad without sed.) Also, there was no cygwin to ease up the pain.

We had to come-up with a solution that works on Windows as well. During this discussion, my Pythian colleague, Romuller, suggested a cool but simple trick which enlightens us and offers one more way of exporting or recovering a table from a full mysqldump.

So the trick here is as follows:

29 Jan

Export MySQL database table to CSV (delimited / Excel) file

Today lets talk a little about converting a MySQL table to CSV (Excel). My friend was looking to export MySQL to Excel, I saw couple of questions for export MySQL tables to CSV on forums. Since I saw the question often, I thought of writing out all the ways I can think of for exporting Delimited (CSV / TSV / …) data from MySQL table. Pretty chewed & basic but frequent topic.

Following are the ways to export CSV data from MySQL database / table(s).

1. Using SELECT INTO … OUTFILE statement to export from MySQL to CSV

SELECT ... INTO OUTFILE writes the selected rows to a file. Column and line terminators can be specified to produce a specific output format. Just to mention, SELECT ... INTO OUTFILE is the complement of LOAD DATA INFILE, which you may use to load CSV (generally speaking delimited) files to MySQL.

Here’s a sample command to export “tablename” table of “db” database as a CSV:

SELECT * INTO OUTFILE '/csv_files/db.tablename.txt'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
FROM db.tablename;

18 Jan

A tale of Corrupt InnoDB table, MySQL crash & recovery

I’m going to narrate you a story that happened around a crashing MyQL, Corrupted InnoDB table and finally the recovery by table restore. We will see how our database administrator detected the issue and what he did to resolve it.

A day in MySQL Database Consultant’s day was taking its shape while a friend called for help.

Friend: Hey, my mysql is crashing and website isn't functioning well. Everything is down. Can you help me?

Our database admin quickly jumps in and checks for the MySQL error log.

2018-01-01T07:39:03.173398Z 0 [ERROR] InnoDB: Database page corruption on disk or a failed file read of page [page id: space=13701, page number=4603]. You may have to recover from a backup.
2018-01-01T07:39:03.173428Z 0 [Note] InnoDB: Page dump in ascii and hex (16384 bytes):
...
InnoDB: End of page dump
2018-01-01T07:39:03.265864Z 0 [Note] InnoDB: Uncompressed page, stored checksum in field1 4187651462, calculated checksums for field1: crc32 4128877936/1194051977, innodb 680941878, none 3735928559, stored checksum in field2 3735928559, calculated checksums for field2: crc32 4128877936/1194051977, innodb 1675940203, none 3735928559, page LSN 400 3284265879, low 4 bytes of LSN at page end 3284252104, page number (if stored to page already) 4603, space id (if created with >= MySQL-4.1.1 and stored already) 13701
InnoDB: Page may be an index page where index id is 33515
2018-01-01T07:39:03.265911Z 0 [Note] InnoDB: Index 33515 is `PRIMARY` in table `nitty-witty`.`flat_address`
2018-01-01T07:39:03.265919Z 0 [Note] InnoDB: It is also possible that your operating system has corrupted its own file cache and rebooting your computer removes the error. If the corrupt page is an index page. You can also try to fix the corruption by dumping, dropping, and reimporting the corrupt table. You can use CHECK TABLE to scan your table for corruption. Please refer to http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html for information about forcing recovery.
2018-01-01T07:39:03.265944Z 0 [ERROR] [FATAL] InnoDB: Aborting because of a corrupt database page in the system tablespace. Or, there was a failure in tagging the tablespace as corrupt.
2018-01-01 07:39:03 0x7f5fa0466700 InnoDB: Assertion failure in thread 140048687589120 in file ut0ut.cc line 916
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
07:39:03 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=536870912
read_buffer_size=16777216
max_used_connections=6
max_threads=214
thread_count=6
connection_count=6
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 7539498 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...

09 Jan

Linux shell script: Pause (Suspend) a process with kill signals

Linux KILL command gives you killing opportunities to use SIGNALs. In this post we’re going to Pause (or suspend) and Start an ongoing Linux process.

Problem: The requirement here is to pause a process for certain time and run it only during off hours. The process itself doesn’t have this option and also we cannot stop it in-between, as it will start from the scratch again.

Solution: To resolve this we can think of sending out signals to the process to hint it to Pause or Progress.

A signal is an asynchronous notification sent to a process or to a specific thread within the same process in order to notify it of an event that occurred. When a signal is sent, the operating system interrupts the target process’ normal flow of execution to deliver the signal.