A lot of articles have been written on splitting mysqldump and grab required tables.
Long back when Shlomi had suggested a “sed” way, I actually shell scripted it, and now publishing.
This shell script will be grabbing the tables you want and pass it to tablename.sql.
It’s capable to understand regular expressions as I’ve added sed -r option.
Also MyDumpSplitter can split the dump in to individual table dumps.
You may directly download the script at the end of the page.
$>sh MyDumpSplitter.sh
Usage: sh MyDumpSplitter.sh DUMP-FILE-NAME — Extract all tables as a separate file from dump.
sh MyDumpSplitter.sh DUMP-FILE-NAME TABLE-NAME — Extract single table from dump.
sh MyDumpSplitter.sh DUMP-FILE-NAME -S TABLE-NAME-REGEXP – Extract tables from dump for specified regular expression.
How to use MyDumpSplitter to split tables from MySQL Dump: (Watch Video Below)
1. To create individual tables’ sql from sinlge dump:
sh MyDumpSplitter.sh database_fulldump.sql
Above command will create individual dumps for each table from main fulldump sql file.
They will be stored as table-name.sql in same directory.
2. To extract single table dump from sinlge dump:
sh MyDumpSplitter.sh database_fulldump.sql tablename
Above command will create dump for specified table from main fulldump sql file and store it to tablename.sql.
3. To extract tables matching matching certain regular expression from sinlge dump:
sh MyDumpSplitter.sh database_fulldump.sql -S tablename
Above command will extract all tables which matches regular expression “tablename(.*)” from main fulldump sql file and store each tabke matching to regular expression in to tablename.sql.
Watch how to use mysqldumpsplitter:
MyDumpSplitter.sh is a very basic script and can be added more features and modified further.
For eg. it can be parameterised to get individual table-names or single-dump of all extracted tables and others.
Suggestions are welcome. I’ll update it whenever time permits.
The shell script is as follows, you may also download it from end of the page:
#!/bin/sh
#SPLIT DUMP FILE INTO INDIVIDUAL TABLE DUMPS
# Text color variables
txtund=$(tput sgr 0 1) # Underline
txtbld=$(tput bold) # Bold
txtred=$(tput setaf 1) # Red
txtgrn=$(tput setaf 2) # Green
txtylw=$(tput setaf 3) # Yellow
txtblu=$(tput setaf 4) # Blue
txtpur=$(tput setaf 5) # Purple
txtcyn=$(tput setaf 6) # Cyan
txtwht=$(tput setaf 7) # White
txtrst=$(tput sgr0) # Text resetTARGET_DIR=”.”
DUMP_FILE=$1
TABLE_COUNT=0if [ $# = 0 ]; then
echo “${txtbld}${txtred}Usage: sh MyDumpSplitter.sh DUMP-FILE-NAME${txtrst} — Extract all tables as a separate file from dump.”
echo “${txtbld}${txtred} sh MyDumpSplitter.sh DUMP-FILE-NAME TABLE-NAME ${txtrst} — Extract single table from dump.”
echo “${txtbld}${txtred} sh MyDumpSplitter.sh DUMP-FILE-NAME -S TABLE-NAME-REGEXP ${txtrst} — Extract tables from dump for specified regular expression.”
exit;
elif [ $# = 1 ]; then
#Loop for each tablename found in provided dumpfile
for tablename in $(grep “Table structure for table ” $1 | awk -F”\`” {‘print $2′})
do
#Extract table specific dump to tablename.sql
sed -n “/^– Table structure for table \`$tablename\`/,/^– Table structure for table/p” $1 > $TARGET_DIR/$tablename.sql
TABLE_COUNT=$((TABLE_COUNT+1))
done;
elif [ $# = 2 ]; then
for tablename in $(grep -E “Table structure for table \`$2\`” $1| awk -F”\`” {‘print $2′})
do
echo “Extracting $tablename…”
#Extract table specific dump to tablename.sql
sed -n “/^– Table structure for table \`$tablename\`/,/^– Table structure for table/p” $1 > $TARGET_DIR/$tablename.sql
TABLE_COUNT=$((TABLE_COUNT+1))
done;
elif [ $# = 3 ]; thenif [ $2 = "-S" ]; then
for tablename in $(grep -E “Table structure for table \`$3″ $1| awk -F”\`” {‘print $2′})
do
echo “Extracting $tablename…”
#Extract table specific dump to tablename.sql
sed -n “/^– Table structure for table \`$tablename\`/,/^– Table structure for table/p” $1 > $TARGET_DIR/$tablename.sql
TABLE_COUNT=$((TABLE_COUNT+1))
done;
else
echo “${txtbld}${txtred} Please provide proper parameters. ${txtrst}”;
fi
fi#Summary
echo “${txtbld}$TABLE_COUNT Table extracted from $DUMP_FILE at $TARGET_DIR${txtrst}”
Download MyDumpSplitter.sh
Update: The script had some issues with regular expression export etc, which Kevin pointed me. I have updated the script and tested for the same.
Special thanks to mwvisa1.
You might also like::
Hi Kedar,
Good work!
I was also considering writing such a script (python). What I see is still missing is as follows:
- working with streaming (e.g. mysqldump mydb | mydumpsplit.sh)
- Managing those little thinks, such as disabling FK checks per split table, charsets…
- Handling triggers, stored routines.
PS, doing the reverse: loading the distinct tables back into mysql can be easily accomplished with Maatkit’s mk-parallel-restore (which also serves as a general-purpose parallelizing tool last I checked).
Thanks! This saved me a bunch of time. I had a 35 GB dump file from which I needed to extract two tables. It took about 28 minutes to process the file. That was faster than loading the whole dumpfile and then dumping just the tables I needed.
One issue I had was that the script didn’t separate tables with the same name, but in different databases. So, I ended up with one .sql file with the drop, create and insert statements for two tables. I was confused for a while, because the resulting table was much too small. Finally, I realized that the smaller second table was replacing the first table when I loaded that .sql file.
Also, when specifying the table name, a regular expression is used that causes any suffix to also match. I had a table named “tasks” and a much larger table named “tasks_archive”, but I wanted only the statements for the tasks table. I assume it took a little extra time to also write out the tasks_archive table statements, but it wasn’t that big of a deal, since I could obviously ignore it.
Hi Robert,
I’m glad to see it’s working as expected.
Now about your regexp, for sure if you want to extract “tasks” table you shouldn’t have used REGEXP mode… you should have directly specified the table name as “tasks_XYZ” will obviously a match to the regexp!
I hope I got what you’ve said?!!
Meanwhile, for extracting Databases from a fulldump following will help:
#Shell Script To Extract Database from Full Dump
for dbz in $(grep “\-\- Current Database” $1 | sed ‘s/`//g’ | awk -F “: ” {‘print $2′}|sort|uniq)
do
echo $dbz;
sed -n “/^– Current Database: \`$dbz\`/,/^– Current Database: \`/p” $1 > $dbz.db.sql
done;
For extracting databases from a mysql dump the following php script is much quicker. The shell script parses the file for table names, then passes the whole file to sed once per table. This takes quite a while for a large dump (I’m currently looking at a 30GB dump). The php script gets the tables, splitting the file in one pass. It uses little memory as it only holds the currently examined line in memory. The first argument to the script is the filename of the full dump. Script is quick and dirty, so no error checking or option to just get one db, but both could be added trivially. Hope someone finds this useful.
<?php
$db = null;
$outfp = null;
$infp = fopen($argv[1], 'r');
while($line = fgets($infp)){
if(preg_match('/– Current Database: `([^`]+)`/',$line,$matches)){
if($outfp){
fclose($outfp);
}
$db = $matches[1];
$outfp = fopen($db.'.db.sql','w');
}
if($outfp){
fwrite($outfp, $line);
}
}
Thank you! It works great. Exactly what I was looking for!
Thanks Nick,
That’s a great piece of code. Somehow I missed writing linux sed command for doing that!
For extracting database from a complete dump we may use following:
sed -n “/CREATE DATABASE.*\`DATABASE_NAME\`/,/CREATE DATABASE/p” COMPLETE_DUMP_FILE > DATABASE_NAME.sql
Thanks,
Kedar.
1. Script Updated.
2. Video “How to use” attached!!
p.s.: It’s my first screen cast – I’ll improve soon.
You saved my life ! Thanks dude !!!
I definitely owe you a beer, started scripting one out then did a quick google and found your scripts many thanks!
@guroot: Welcome dude!
@will pink: So where can we have it anyways ? Glad to know it helped as it meant to be
nice tips, thanks for share.
omg, thanx man!!
Hey, looks like this script is just like what I need. But, it doesn’t work for me.
Could you please help? I get this when run without any arguments. The filname is “sp.sh”. The output is:
: command not found
: command not found
sp.sh: line 23: syntax error near unexpected token `elif’
‘p.sh: line 23: `elif [ $# = 1 ]; then
Please help.
Hey Andrey, I think you might have copied it from browser, try downloading it. I didn’t understand when you say “I get this when run without any arguments”…
Thankyou so much! This saved me a lot of time..
I am unable to extract a table called billing from the full db dump.
cdr_backup# ./MyDumpSplitter.sh backup_feb15.log billing
0 Table extracted from backup_feb15.log at .
Please help me on this
Thanks,
Sandeep
Hi Sandeep,
I hope you’ve downloaded the MyDumpSplitter.sh file from download link.
Now, “backup_feb15.log” seems a log file than a backup file! Are you sure that its a mysqldump file?
-Kedar
Dear sir i found ur site today and today i have found your MyDumpSplitter script i have need this type of script from long period this is great for me and very very thanx to you
Came across your script. nice work.
Thought I’d pass this helpful tip to those who’s *.sql files are stored/saved as *.sql.gz (compressed) files.
To avoid having to uncompress a full backup dump first (especially if its very large; and disk space is a premium).. I may the following minor code mods..
change all grep cmds to zgrep
change all lines
sed -n “/^– Table structure for table \`$tablename\`/,/^– Table structure for table/p” $1 > $TARGET_DIR/$tablename.sql
to
zcat $1 | sed -n “/^– Table structure for table \`$tablename\`/,/^– Table structure for table/p” >$TARGET_DIR/$tablename.sql
I don’t know what to do, by my is something timeout error…
Let me know how may i help you.
Thanks a lot for the script! It was very helpfully for me!
here is a bug!
when i want to extract table which locate in the last of dump file.I get error like this:
ERROR 1231 (42000) at line 25: Variable ‘time_zone’ can’t be set to the value of ‘NULL’
Reason:extract extra infomation as follows:
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
Hi Alex,
Thanks for posting here.
Well I tested the script for the said scenario “fetching the last table”:
sh mysqldumpsplitter.sh SQL-FILE LAST-TABLE-NAME
and it worked fine for me.
Can you tell me your exact scenario?
Thanks,
Kedar.
Hello Kedar,
I tried to run the script but I got this errors:
mysqldumpslitter.sh:25 Syntax error: “elif” unexpected (expecting “then”)
Thank you
Hello I tried it again, but this time it worked as it should be. Thanks
Don’t you think sometimes it works exactly when we were about to leave hopes
keep visiting,
Kedar.
Thanks for this script! Just what I needed. Also glad to see the .gz file handling – saves a step.
Yep John…
Nice script very usefull
Hurrah, that’s what I was searching for, what a material! present here at this webpage, thanks admin of this website.
Looks great. What happens to the table schema?
Hey ST,
I didn’t get your question?