Tag Archives: MySQL

03 May

Load columnar text file into MySQL table

Ever come accoss a situation where you have to load a flat text file into MySQL with data provided in single column!
Say, your data to be loaded in table looks something like this:

Company1
Street Name
City, MyState 8582
(999)999-999
another_Company
Another Street 2
New City, NWSTATE 8582
(111)111-1111

So how’d you go about loading a text file into MySQL where field values are given in a single column!!
In the above sample, data has fields like company-name, address-1, address-2 and phone given in a text file line by line.
Ofcourse you can write a piece of code in your choice of language: perl, shell, python…
But can you do this by sheer power of MySQL by using some commands or functions only?

Well, normally you will have to have a script for such data to be inserted into database. We always prefer the raw data / text file in the csv / tab separated (columns) file(s), which you can directly load to MySQL table as explained in my previous article: http://kedar.nitty-witty.com/blog/load-delimited-data-csv-excel-into-mysql-server

But that’s not the case here. And hence I have a workaround to share. With little more efforts you can also automate / scriptize this. [ I’m feeling bit lazy now. ;) ]

Follow these steps:
– Save above data in a file [say c:/main.txt considering windows]
– Connect to MySQL through commad prompt

– Create a main table to load all data to process later.

drop table if exists main; create table main ( id integer not null auto_increment primary key,alldata varchar(100), type int );

– Load data to MySQL table:

load data local infile 'C:/mainfile.txt' into table main lines terminated by '\r\n' (alldata) set id=null;
set @var=0;update main set type=if(id mod 4 = 1,@var:=@var+1,@var);

– Create table as per your requirement:

create table company_details (
company varchar(100),
street varchar(100),
citystatezip varchar(100),
phone varchar(100) );

– Generate insert statement:

select concat("insert into company_details (company,street,citystatezip,phone) values (", my_values, ");") from (select group_concat('"',alldata,'"') my_values from main group by type) X;

you may alternatively redirect this data to a txt file and load it back to mysql!

put above query in sql :
mysql -uroot -ppassword database < generate_insert_query.sql > inserts.sql

Finally load thus generated inserts to MySQL database table.
mysql -uroot -ppassword database < inserts.sql

I hope this helps many.

23 Apr

MySQL – Alter table-change datatype of column with foreign key

Schema alteration is a big headache especially when it has already reached to production and things get worse when the relations are involved.
Below is a short example for altering tables having foreign key constraints.

SQL for tables used in example:
create database temp;use temp;
drop table if exists city; drop table if exists country;
create table country ( c_id mediumint not null auto_increment primary key, c_name varchar(10)) engine=innodb;
insert into country (c_name) values ('india'),('uk'),('us');
create table city (city_id mediumint not null auto_increment primary key, country_id mediumint, city_name varchar(20), constraint fk_country_id foreign key (country_id) references country(c_id)) engine=innodb;
insert into city (country_id, city_name) values (1,'mumbai'), (2,'ahmedabad'), (2,'new york'), (2, 'la'), (3, 'london'), (3,'Birmingham');

We want to change the datatype of primary key c_id from mediumint to int.
Now, while you proceed to change the datatype of column in relation, you observe a typical mysql error.

mysql> alter table country modify column c_id int not null auto_increment;
ERROR 1025 (HY000): Error on rename of '.\temp\#sql-248_4' to '.\temp\country' (errno: 150)

“Error on the rename” doesn’t suggest you anything though you have a way out to know the actual error:

1) On your shell prompt fire:
$] perror 150
MySQL error code 150: Foreign key constraint is incorrectly formed

For more details you may look into last foreign key error section of innodb status:

2) In this case specifically you should check innodb status:
mysql> show engine innodb status;
...
-------------------------
LATEST FOREIGN KEY ERROR
-------------------------
120422 13:11:47 Error in foreign key constraint of table temp/city:
there is no index in referenced table which would contain
the columns as the first columns, or the data types in the
referenced table do not match the ones in table. Constraint:
,
CONSTRAINT "fk_country_id" FOREIGN KEY ("country_id") REFERENCES "country" ("c_id")
The index in the foreign key in table is "fk_country_id"
...

This clearly explains you the reason for error in table alteration. InnoDB does not permit the creation of a foreign key constraint where a column references a nonmatching column type. This restriction applies while you alter the parent or child table with Foreign Key Constraint.
So how ‘d you proceed? Here is one way to do it.

The steps to follow are:
– Alter all the referencing tables and remove Foreign Key relations.
– Alter all the referencing and referenced tables and modify column to new datatype.
– Alter all referencing tables create Foreign Key relations.

To ease out the task execute following SQL and you will get your queries that will help you changing the datatype step by step. You have to provide the parent table name and column that’s being referenced along with the datatype you want to modify to.

#COMMAND FOR DROP FOREIGNKEY RELATION
select
concat('ALTER TABLE ', table_name,' DROP FOREIGN KEY ',constraint_name,';') drop_fk_syntax
from information_schema.key_column_usage where constraint_schema = database() and referenced_table_name ='country' and referenced_column_name ='c_id';

+--------------------------------------------------+
| drop_fk_syntax |
+--------------------------------------------------+
| ALTER TABLE city DROP FOREIGN KEY fk_country_id; |
+--------------------------------------------------+

#COMMAND FOR APPLY COLUMN DEFINITION CHANGE TO PARENT TABLE
SELECT CONCAT('ALTER TABLE ', referenced_table_name,' MODIFY COLUMN ', referenced_column_name, ' bigint not null auto_increment;') modify_column
from information_schema.key_column_usage where constraint_schema = database() and referenced_table_name ='country' and referenced_column_name ='c_id';
+---------------------------------------------------------------------------+
| modify_column |
+---------------------------------------------------------------------------+
| ALTER TABLE country MODIFY COLUMN c_id bigint not null auto_increment; |
+---------------------------------------------------------------------------+

#COMMAND FOR APPLY COLUMN DEFINITION CHANGE TO CHILD TABLES
select concat('ALTER TABLE ', table_name,' MODIFY COLUMN ', column_name, ' bigint not null;') modify_column
from information_schema.key_column_usage where constraint_schema = database() and referenced_table_name ='country' and referenced_column_name ='c_id';

+------------------------------------------------------------+
| modify_column |
+------------------------------------------------------------+
| ALTER TABLE city MODIFY COLUMN country_id bigint not null; |
+------------------------------------------------------------+

#COMMAND FOR RESTORING FOREIGNKEY RELATION
select
concat('ALTER TABLE ',table_name,' ADD CONSTRAINT ',constraint_name ,' FOREIGN KEY (', column_name, ') REFERENCES ', referenced_table_name , '(', referenced_column_name,');') create_fk_syntax
from information_schema.key_column_usage where constraint_schema = database() and referenced_table_name ='country' and referenced_column_name ='c_id';

+--------------------------------------------------------------------------------------------------+
| create_fk_syntax |
+--------------------------------------------------------------------------------------------------+
| ALTER TABLE city ADD CONSTRAINT fk_country_id FOREIGN KEY (country_id) REFERENCES country(c_id); |
+--------------------------------------------------------------------------------------------------+

Execute above queries in sequence to modify your schema.

Always remember: “backup saves your life like no one else”.
Share your better ways.

30 Jan

Access Control in MySQL Stored Routines: DEFINER, INVOKER & SQL SECURITY

MySQL Stored Routines (functions and procedures) are not only used for improving performance but also they’re handy when it comes to enhancing security and restricting user access. This post briefs you about security aspects of stored routines by means of example.

By the definition of a Stored Routine,
– DEFINER clause specifies the creator of the stored routine.
– SQL SECURITY clause specifies the execution context of the stored routine.

Now let’s start with an example:

Create a very basic stored routine in test database:

DELIMITER //
CREATE PROCEDURE user_cnt()
BEGIN
SELECT COUNT(*) as total_user FROM mysql.user;
END;//
DELIMITER ;

Our current user is: root@localhost
[ You can see your current user by issuing select CURRENT_USER(); ]

Call the procedure:
CALL user_cnt()\G
total_user: 10

Here, We created a stored routine using root@localhost user.
– DEFINER: The user who creates this procedure. i.e. root@localhost
– SQL SECURITY: Defines under whose privileges the routine will be executed; defaults to DEFINER. i.e. root@localhost

As SQL SECURITY is set to DEFINER, a user even with only EXECUTE permission for routine can call and get the output of the stored routine regardless of whether that user has permission on mysql database or not.

Lets see how this works. Create a “execuser” with limited privileges:
GRANT EXECUTE ON test.* TO 'execuser'@'localhost' IDENTIFIED BY 'execuser' ;
FLUSH PRIVILEGEES;

Login using execuser to mysql prompt and call the procedure created under test database:
mysql> CALL user_cnt()\G
total_user: 10

Now check that below command fired by execuser fails due to lack of privileges:
mysql> select count(*) from mysql.user;
ERROR 1142 (42000): SELECT command denied to user 'execuser'@'localhost' for table 'user'

– Thus stored procedure allows us to restrict users to access tables directly but still getting access to certain data.
– User execuser@localhost is not having permission on mysql database and still he could get the data.
– As the SQL SECURITY was set to DEFINER (default), the execution of the routine happened under the security context of root@localhost user and returned the result.

Is this behaviour something that you don’t want to happen? Hmmm we have a way out here.
Lets rewrite the stored procedure as below:

DELIMITER //
DROP PROCEDURE IF EXISTS user_cnt;
CREATE DEFINER='root'@'localhost' PROCEDURE user_cnt()
SQL SECURITY INVOKER
BEGIN
SELECT COUNT(*) as total_user FROM mysql.user;
END;//
DELIMITER ;

– Did you note “SQL SECURITY INVOKER”!! That’s our saviour.
– We’ve specified the DEFINER attribute explicitly.
– SQL SECURITY decides under whose privileges the stored routine gets executed! Here it is INVOKER, the user that CALLs the routine!

Let’s check through execuser@localhost user:
mysql> call user_cnt();
ERROR 1142 (42000): SELECT command denied to user 'execuser'@'localhost' for table 'user'

So, this is now pretty clear that the stored routine tried to execute the SELECT query under INVOKER’s (execuser@localhost) privileges; and the routine failed as the INVOKER has no privileges on user table.

Finally, a few points:
Together DEFINER & SQL SECURITY clauses define the security context to be used during routine execution time.
The DEFINER attribute defaults to the current user & defines the creator of the Stored Routine.
SQL SECURITY defines the execution permission of the stored routine and defaults to DEFINER.

I hope this cleared the basics; thanks for reading.

06 Dec

Escaping “Lesson Learned” dealing with MySQL Databases & Case Sensitiviy

Do you ever need to transfer databases between servers? Different OSes?

Two common ways of data migration:
1. Create mysqldump and load it.
2. File transfer using SCP.

Case Sensitivity and Issue:
Yes, that can create big issues when you have to deal with systems having different case sensitivity.
E.g. on linux you can create directories with names “Kedar” or “kedar” which is not possible on windows!
It will be harmful when file-system itself restricting the names to lower case.
MySQL has a variable for that: lower_case_file_system.

Documentation says: This variable describes the case sensitivity of file names on the file system where the data directory is located. OFF means file names are case sensitive, ON means they are not case sensitive. This variable is read only because it reflects a file system attribute and setting it would have no effect on the file system.

Try loading data having different values for lower_case_file_system and you will get it!

How MySQL tries to solves this problem:
It asks user to handle the issue by changing system variable lower_case_table_names accordingly.
Check the options available:
0 : Table names are stored as specified and comparisons are case sensitive.
1 : Table names are stored in lowercase on disk and comparisons are not case sensitive.
2 : Table names are stored as given but compared in lowercase.

So where do I see the problem?
Here control is given to user with the variable lower_case_table_names and ignoring which can definitely lead to issues. These issues can be avoided by putting restrictions.
Also queries working on MAC / Windows may not work well on Linux / Other Flavours; cause problems on application side.

Documentation Says: You should not set this variable to 0 if you are running MySQL on a system that has case-insensitive file names (such as Windows or Mac OS X). If you set this variable to 0 on such a system and access MyISAM tablenames using different lettercases, index corruption may result.

Now, I recently read the change: PERFORMANCE_SCHEMA is renamed to lowercase in MySQL 5.5 to answer a bug [http://bugs.mysql.com/bug.php?id=57609] pointed by Marc Alff’s post [http://marcalff.blogspot.com/].

In one line I wish: lower_case_table_names, RIP!
I wish MySQL does the same for all tables in order to avoid case sensitivity problems :).

I feel MySQL should not allow upper cases. What do you think?

30 Nov

Using LOAD DATA INFILE with Stored Procedure Workaround-MySQL

Okay! So here we will use Load Data syntax to load file into MySQL Server in a Stored procedure. Yep! It’s a workaround.

Download MySQL UDF:

[root@localhost kedar]# wget http://www.mysqludf.org/lib_mysqludf_sys/lib_mysqludf_sys_0.0.3.tar.gz

[refer: http://www.mysqludf.org/]

Extract and Install:

[root@localhost kedar]# tar -xzvf lib_mysqludf_sys_0.0.3.tar.gz
install.sh
lib_mysqludf_sys.c
lib_mysqludf_sys.html
lib_mysqludf_sys.so
lib_mysqludf_sys.sql
Makefile

[root@localhost kedar]# sh install.sh
Compiling the MySQL UDF
gcc -Wall -I/usr/include/mysql -I. -shared lib_mysqludf_sys.c -o /usr/lib/lib_mysqludf_sys.so
MySQL UDF compiled successfully

Please provide your MySQL root password
Enter password:
MySQL UDF installed successfully

Create function sys_exec as follows:

CREATE FUNCTION sys_exec RETURNS INT SONAME ‘lib_mysqludf_sys.so';

sys_exec – executes an arbitrary command, and returns it’s exit code.

You also can similarly create functions:
sys_eval – executes an arbitrary command, and returns it’s output.
sys_get – gets the value of an environment variable.
sys_set – create an environment variable, or update the value of an existing environment variable.

Example – How to load txt file to MySQL using Stored Procedure & Load Data syntax:

Step-1. Creating table:

CREATE TABLE `t` ( `id` int(2) default NULL ) ENGINE=MyISAM

Step-2. Create a sample file to load:

vi loadtest.txt
1
2
3

Step-3. Create a shell script:

vi /tmp/load.sh
mysql -u mysql_user -p mysql_password -e “load data local infile \”$1\” into table $2;”

Step-4. Create a Stored Procedure:

DELIMITER $$
DROP PROCEDURE IF EXISTS `load_data_SP` $$
CREATE PROCEDURE `load_data_SP` (in_filepath varchar(100),in_db_table varchar(100))
BEGIN
declare exec_str varchar(500);
declare ret_val int;
set exec_str=concat(“sh /tmp/load.sh “,in_filepath,” “, in_db_table);
set ret_val=sys_exec(exec_str);
if ret_val=0 then
select “Success” as Result;
else
select “Please check file permissions and paths” as Result;
end if;
END $$
DELIMITER ;

Step 5. Execute:

CALL `load_data_SP`(‘/tmp/loadtest.txt’ , ‘test.t’);

…and that’s it Stored Procedure will return Success or Failure accordingly.

Make sure you’re having file permissions well set and MySQL can access the files.
Here I’ve kept the files under /tmp directory with chmod 777 & chown mysql:mysql to remove the permission-issue possibility.

Hope this helps.

-- Kedar Vaijanapurkar --