12 Nov

How to Get Stock Quote on Linux using Google ,Curl, Grep, Awk

If you are a Stock Market Addict and a Techie; normally working on Linux(es); this post will interest you as you still can get quote of your stock on your Linux prompt.

Consider, we have Google Finance url for Tata Consultancy Services Stock: http://www.google.com/finance?q=NSE:TCS

So, how do you get above stock price while you’re on Linux / Unix server?
It’s a sinlge line command and you get your stock’s current price. 🙂

curl --silent -X Get "http://www.google.com/finance?q=NSE:TCS" > /tmp/ChangeIsInevitable && cat /tmp/ChangeIsInevitable | grep -m1 -E 'span id="ref_' | awk -F ">" '{print $2}' | awk -F "<" '{print $1}'

This work for other Stock Exchanges like NYSE as well:

curl --silent -X Get "http://www.google.com/finance?q=nyse:goog" > /tmp/bgz && cat /tmp/bgz | grep -m1 -E 'span id="ref_' | awk -F ">" '{print $2}' | awk -F "<" '{print $1}' && date 617.19 Fri Nov 12 17:07:52 IST 2010

and that’s it. Well, I’m thinking thinking more about further extensions of this.
Special thanks goes to: Jignesh Bhai, Google, Curl, Grep, Awk.

05 Apr

In the world of NoSQL-Hands on Mongodb-1

I’m hearing a lot of “NoSQL” these days. To really understand how (and) does it works, I decided to give a try on MongoDB.

MongoDB (hu*mongo*us) is an open source, scalable, high-performance, schema-free, document-oriented database written in the C++ programming language.

MongoDB is not a Relational Database Management System. The database manages collections of JSON (JavaScript Object Notation)-like documents which are stored in a binary format referred to as BSON (Binary JSON).

This post will surely get you started with the MongoDB-NoSQL.

Installing MongoDB:

Download Mongodb from www.mongodb.org
Extract binaries
Create data directory – default for windows C:\data\db
Install service: Full\path\to\mongod.exe –install

If you want to change default data directory location:
Full\path\to\mongod.exe –dbpath=”\”full\path\to\”” –install

Troubleshooting Mongodb Installtion Error:
I faced while installing, if you get error like:
dbexit:
shutdown: going to close listening sockets.
shutdown: going to flush oplog…
shutdown: going to close sockets…
shutdown: waiting for fs preallocator…
shutdown: closing all files…
closeAllFiles() finished
dbexit: really exiting now

Go to: HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/
Check for MongoDB
Verify the value of ImagePath is proper

This seems to be caused by not giving full-path while installing.

Add Environment variable Path, add path/upto/mongo-bin-dir;

Okay, now we’re all set to checkout the MongoDB-NoSQL world.

MongoDB GUI Administration Tool:

There is a GUI Admin tool available phpmoadmin to simplify the MongoDB Operations.
Signle php script: phpmoadmin
Download PHP extension from php.net/manual/en/mongo.installation.php
Extract in ext directory of WAMP & Restart.

In case of error you may install Microsoft Visual C++ 2008 SP1 Redistributable Package (x86) [www.microsoft.com/downloads/details.aspx?familyid=A5C84275-3B97-4AB7-A40D-3802B2AF5FC2&displaylang=en]

MySQL to MongoDB Database Migration:

Create CSV from MySQL:
mysql> select * into outfile “c:/TABLENAME.csv” FIELDS TERMINATED BY ‘,’ OPTIONALLY ENCLOSED BY ‘”‘ LINES TERMINATED BY ‘\n’ from TABLENAME;
Query OK, 293911 rows affected (1 min 8.17 sec)

Import CSV to mongodb using mongoimport:
C:\>mongoimport -d DATABASENAME -c TABLENAME –file “c:\TABLENAME.csv” –type csv -f FIELD1,FIELD2…

Using MongoDB on Windows:

On Command Prompt:
C:\MongoDB\mongodb-win32-i386-1.4.0\bin>mongo.exe
MongoDB shell version: 1.4.0
url: test
connecting to: test
type “exit” to exit
type “help” for help

By default it’s connecting to test database.
Default mongodb port is 9999.

Basic Commands

show dbs : show database names
show collections : show collections in current database
show users : show users in current database
show profile : show most recent system.profile entries with time >= 1ms
use : set curent database to
db.help() : help on DB methods
db.foo.help() : help on collection methods
db.foo.find() : list objects in collection foo
db.foo.find( { a : 1 } ) : list objects in foo where a == 1 it result of the last line evaluated; use to further iterate

Simple Queries with MongoDB:

Set Current Database:
>use DATABASENAME

See Present Database:
> db
DATABASENAME

View collections (tables) under present db:
> db.getCollectionNames()
[ “TABLENAME”, “system.indexes” ]

SQL>Select count(*) from TABLENAME:
NoSQL> db.TABLENAME.count()
238735

SQL>select * from TABLENAME where field1 regexp ‘enrique’:
NoSQL>db.TABLENAME.find( { field1 : /enriqu/i} );

For simple find:
NoSQL>db.TABLENAME.find( { field1 : “value” } );

SQL>Select field1 from table where field2=”%search%”:
NoSQL>db.TABLENAME.find( { field2 : /search/i} , {field1 : 1 } );

SQL>Select All but one (field1) column:
NoSQL>db.TABLENAME.find( { field2 : /search/i} , {field1 : 0 } );

SQL>Select with sort order ascending and descending (1,-1):
NoSQL>db.TABLENAME.find( { ar_name : /enrique/i }, {field1 : 1 } ).sort({ field1 : -1})

MongoDB now seems easy to follow, but I’m still having hard time in taking on the Map/Reduce.
More in the next post; ofcourse documentation home is a good place, else you may export the pdf / html / xml documentation here.

29 Mar

MySQL related file types and basic information

This post covers the basic information of files that MySQL Server uses / creates for various tasks.

my.cnf :

It is the main configuration file for MySQL. You may find it under base directory in windows or under /etc/.

.sock :

mysqld creates a socket for programs to connect to and notes in this file. It is named as mysql.sock.

Pid file :

MySQL server write its process ID in this file. You may specify it by –pid-file=file_name otherwise it defaults to hostname of the server.

.frm :

MySQL represents each table by an .frm table format file, stores table definition in the database directory. It is stored as
.frm under data directory.

.MYD :

This is the extension of the Data files for MyISAM tables. It is stored as
.MYD under data directory.

.MYI :

This is the extension of the Index files for MyISAM tables. It is stored as
.MYI under data directory.

[ad#ad-2-300×250]

.ibd :

If you specify innodb_file_per_table option to my.cnf, InnoDB stores each table in its own .ibd file in the same MySQL database directory where the .frm file is created. For InnoDB tables, the data and the indexes are stored together in the .ibd file.

ibdata1 :

Shared tablespace / data file for InnoDB tables when innodb_file_per_table option is not specified in my.cnf, all tables are created in this shared tablespace.

ib_logfile0 & ib_logfile1 :

InnoDB log files. The information is logged in circular fashion. InnoDB applies changes recorded in logfiles to tablespace. This is also important for performance and recovery.

.db :

This is data file for BDB (BerkeleyDB) storage engine which stores table data and indexes.

db.opt :

In CREATE DATABASE command, create_specification options specify database characteristics. Database characteristics are stored in the db.opt file in the database directory.

error log :

This log contains information about mysqld start and stop events, any critical errors that occur while the server is running etc. In my.cnf you can specify log-error[=file_name] option and error log will be created under data directory file_name.err. If no name is specified MySQL will create hostname.err file.

slow query log :

This log contains queries that runs slower than specified long_query_time variable. You can specify log-slow-queries[=file_name] option. If no file name specified, MySQL will log slow queries in host_name-slow.log file under data directory.

general query log :

This file logs all the SQL sent from client about MySQL server and mysqld start up/down details. You may enable it by log[=file_name]. If file_name is not specified MySQL will create hostname.log under data directory.

#sql-55c_6.MYD / #sql-55c_6.frm / #sql-55c_6.MYI :

Temporary files created when you normally issue ALTER TABLE, MySQL creates a copy of table, apply changes, delete original table and then rename temporary table to original. The names are like #sql-…/MYD/MYI/frm.

binary log files :

These files contains events that describe database changes such as table creation operations or changes to table data. You may specify their name by –log-bin[=base_name] option. Default name is value of pid-file option (defaults to hostname of server).

relay log files :

These files contains events read from the binary log of the master and written by the slave I/O thread. Events in the relay log are executed on the slave as part of the SQL thread.

master.info :

This file contains the status and current configuration information for the slave’s connectivity to the master.

relay-log.info :

This file holds the status information about the execution point within the slave’s relay log.

.index :

To keep track of which binary log files have been used, mysqld creates a binary log index file. It contains the names of all used binary log files. You may change it by –log-bin-index[=file_name] option otherwise basename will be same as that of binary log file with default extension “.index”.
In case of replication, relay logs uses .index files for same purpose. The default relay log index file name is host_name-relay-bin.index in the data directory. You may change the name by –relay-log-index option.

.TMD:

This is an intermediate data file for a table that needs to recreate its data file. You can find this files normally during REPAIR TABLE operations.

TRG & TRN Files:

Trigger definitions are stored in plain text files in the directory that contains the schema objects.
The file tablename.TRN is the TRIGGERNAME file stored under database directory.
The file tablename.TRG is the TRIGGERS file. It represents all the table triggers attached to a given table, so this file can contain triggers for multiple events (BEFORE/AFTER, INSERT/UPDATE/DELETE).

.ARZ, .ARM & .ARN files:

These are files related to Archieve Storage Engine.

.ARZ is data files holds data for table, .ARM holds metadata while during table optimization process a .ARN file may appear.

20 Mar

Working with EditPlus Text Editor-Regular Expression How To

Editplus is a lot better than the regular text editor, Notepad.

From all it’s features I like RegExp Support the most, and than comes the block select feature.

Here are the quick lines to carry out regular tasks using regular expression in Editplus. It’s kinda downloadable cheetsheet list.

Remove all empty lines:

Find: “^\n” (Ignore double-quotes in all find/replace)

Replace: “”

Where,

^ – Beginning of the line

\n – New Line

Remove Multiple Spaces convert into single space:

Find: ” +”

Replace: ” “

Where,

+ – find one or more occurance of space character.

Comment multiple line of code:

Find: “^”

Replace: “#” or “//”

You may optionally use: Edit Menu > Format > Line Comment.

Generate Comma Separated List from new line delimited list:

Find: “\n”

Replace: “, “

This helps in even joining some of lines of code instead of replacing by comma you may replace it with “”.

Manipulate columns display order / punctuation:

Find: “([0-9]+)\t([a-zA-Z]+)”

Replace: “\2\t\1”

Where,

[0-9]+ – Finds one or more digits

[a-zA-Z]+ – Finds one or more characters

() – mark the block or capture the group

\2 – 2nd mark expression

Eg:

123 abc

345 cde

567 efg

Becomes:

abc 123

cde 345

efg 567

The Other Way:

– Press Alt+C

– Drag you mouse to select respective column and click

– Copy / Cut as required

[ad#ad-2-300×250]

Append / Add semicolon (any character) at the end of the line:

Find: “\n”

Replace: “;\n”

Enclose lines by quotes:

Find: “\n”

Replace: “‘\n'”

Delete all lines containing a given STRING:

Find: “^.*STRING.*$”

Replace: “”

Remove lines not containing a given STRING:

I don’t know how to do this!! 🙂

Convert tab separated file into insert statements:

TSV: abcd de4 iirn 34399

SQL: INSERT INTO TABLENAME VALUES (“abcd”, “de4”, “iirn”,”34399″);

Find: “(.*)\t(.*)\t(.*)\t(.*)”

Replace: “INSERT INTO TABLENAME VALUES (“\1”, “\2”, “\3″,”\4″);”

Format the telephone number:

Find: “([0-9][0-9][0-9])([0-9][0-9][0-9])([0-9].*)”

Replace: “\1-\2-\3”

Eg.:

Original: 1231231231

Formatted-1: 123-123-1231

Remove Brackets:

Find: “$|$”

Replace: “”

Where,

\( – Match (. \ is required to escape marking the expression.

| – or

Replace 1st occurrence of character:

Find: ” (.*)”

Replace: “-\1”

Where,

(.*) – matches everything and marks the block

** Make sure you ignore double-quotes(“) while writing in find / replace boxes.

EditPlus supports following regular expressions in Find, Replace and Find in Files command.

Expression – Description

\t – Tab character.
\n – New line.
. – Matches any character.
| – Either expression on its left and right side matches the target string.
[] – Any of the enclosed characters may match the target character.
[^] – None of the enclosed characters may match the target character.
* – Character to the left of asterisk in the expression should match 0 or more times.
+ – Character to the left of plus sign in the expression should match 1 or more times.
? – Character to the left of question mark in the expression should match 0 or 1 time.
^ – Expression to the right of ^ matches only when it is at the beginning of line.
$ – Expression to the left of $ matches only when it is at the end of line.
() – Affects evaluation order of expression and also used for tagged expression.
\ – Escape character. If you want to use character “\” itself, you should use “\\”.

Notable Features of Editplus are :

Spell checking
Regex-based find & replace
Encoding conversion
Newline conversion
Syntax highlighting
Multiple undo/redo
Rectangular block selection
Auto indentation
Code folding (Text folding)

Download pdf: Editplus-RegExp.

18 Mar

batch script to add remove prefix zero pad bulk file rename

This post has two batch scripts:
1. Batch script to rename files with zero padded number series-prefix

2. Batch script to remove prefix of perticular length

1. Batch script to rename files with zero padded number series-prefix

This script will accept file-type to be searched and lenght of zero-padded prefix to be attached.

Usage: RenZeroPad.bat

Consider you have to rename / arrange a lot of mp3 files in a perticular sequence:

File names before execution:
fileX.mp3
fileY.mp3
fileZ.mp3

Command on dos prompt: RenZeroPad.bat mp3 4

File names with attached prefix:
0001 fileX.mp3
0002 fileY.mp3
0003 fileZ.mp3

[ad#ad-2-300×250]

The Batch Script to add zero pad digits prefix:

@echo off
setLocal EnableDelayedExpansion

set /a cnt=1
for %%i in (*.%1) do (
call :Set0Pad %2

set newName=!str! %%i
ren “%%i” “!newName!”
)

:Set0Pad
set padcntr=0000000000%cnt%
set str=%padcntr:~-%1%
set renstr=%str%
set /a cnt+=1

** Download available at the end of the page.

2. Batch script to remove prefix of perticular length

This script will accept file-type to be searched and lenght of prefix to be removed.

Usage: RemovePrefix.bat

Consider you have to rename / remove prefixes from a bunch of files:

0001_fileX.doc
0002_fileY.doc
0003_fileZ.doc

Command on dos prompt: RemovePrefix.bat mp3 5

Files will be renamed with removed prefixes as follows:
fileX.doc
fileY.doc
fileZ.doc

The Batch Script to remove prefix:

@echo off
setLocal EnableDelayedExpansion

set /a cnt=1
for %%i in (*.%1) do (
set str=%%i
set newstr=!str:~%2!
ren “%%i” “!newstr!”
)

Download: RemovePrefix.bat and RenZeroPad.bat