Bash Shell Scripting - Basics

  January 03, 2016   , , , , , ,   No comments   Edit
Working with text

1. Read a CSV file line by line and extract specific filed, and store it in a variable.
file_name="data.csv"
while read line
do
field1=$(echo -e $line | gawk -F, '{ print $1}')
field2=$(echo -e $line | gawk -F, '{ print $2}')
done < $file_name

2. Find and repace a string in a file.
sed -i 's/foo/bar/g' file.txt

Arithematic Operations
1. Increment a variable
count=$((count+1))

2. Decrement a variable
count=$((count-1))

Working with date and time
1. Get current date in yy-mm-dd format.
current_date="`date +%y-%m-%d`"
echo $current_date

2. Get backup time stamp in yymmdd-hhmmss format and store it in a variable.
bkptime=("`date +%y%m%d-%H%M%S`")

3. Get yesterday's date in yyyy-mm-dd format.
yesterday_date="`date --date=yesterday +%Y/%m/%d`"
echo $yesterday_date

3. Get 10 days old date in yyyy-mm-dd format.
day=("`date --date=\"10 day ago\" +%Y-%m-%d`")

OR

daysold=10
day=("`date --date=\"$daysold day ago\" +%Y-%m-%d`")

4. Get last day of the month
To get last day of current month:
date -d "-$(date +%d) days 1 month" +%Y-%m-%d

To get last day of last month:
date -d "-$(date +%d) days 0 month" +%Y-%m-%d

To get last day of last before month
date -d "-$(date +%d) days -1 month" +%Y-%m-%d

To get last day of next month
date -d "-$(date +%d) days 2 month" +%Y-%m-%d

To get last day of next to next month
date -d "-$(date +%d) days 3 month" +%Y-%m-%d

If conditions

1. Check a vriable is having some value (it is non-zero)
if [ ! -z "$var" ]; then
     echo "Variable value is $val"
else
     echo "Variable has no value"
fi

2. Check if file exist
if [  -e "$file.csv" ]; then
     echo "File exist"
fi

3. Check if doesn't file exist
if [ !  -e "$file.csv" ]; then
     echo "File exist"
fi

4. Check if previously executed command is successful or not. If not successful exit from loop.
ssconvert "file.xls" "file.csv"
if [ "$?" == "0" ]; then
      echo "File converted successfully.."
else
exit
fi

Working with files

1. Covert EXCEL(.xls) file to CSV file.
ssconvert "file.xls" "file.csv"

2. Compress a directory using 7zip.
/usr/bin/7z a reportdir.7z reportdir > /dev/null

3. Upload a directory and its content to an FTP location
ftp_server="ftp.mydomain.com"
ftp_bkp_username="myusername"
ftp_bkp_password="mypassword"
FTP Over TLS
lftp -c "open -e \"set ftps:initial-prot \"\"; set ftp:ssl-force true; set ftp:ssl-protect-data true; mirror -R mydir; \" -u \"$ftp_bkp_username\",\"$ftp_bkp_password\" ftp://$ftp_server/mypath/"
FTP (without TLS)
ncftpput –R -u $ftp_bkp_username -p $ftp_bkp_password $ftp_server mypath mydir

4. Upload a file to an FTP location.
ftp_server="ftp.mydomain.com"
ftp_bkp_username="myusername"
ftp_bkp_password="mypassword"
FTP Over TLS
lftp -c "open -e \"set ftps:initial-prot \"\"; set ftp:ssl-force true; set ftp:ssl-protect-data true; put myfile; \" -u \"$ftp_bkp_username\",\"$ftp_bkp_password\" ftp://$ftp_server/mypath/"
FTP (without TLS)
ncftpput -u $ftp_bkp_username -p $ftp_bkp_password $ftp_server mypath myfile

5. Create an empty directory in an FTP location over TLS.
ftp_server="ftp.mydomain.com"
ftp_bkp_username="myusername"
ftp_bkp_password="mypassword"
lftp -c "open -e \"set ftps:initial-prot \"\"; set ftp:ssl-force true; set ftp:ssl-protect-data true; put myemptydir; \" -u \"$ftp_bkp_username\",\"$ftp_bkp_password\" ftp://$ftp_server/mypath/"


Working with user input
1. Read input from user.
echo -n "Enter date for processing data (yyyy/mm/dd) : "
read given_date
echo "You have entered $given_date"

Working with MySQL statements
1. Change a MySQL environment variable.
 db.cnf
[client]
user = root
password = mypassword
host = localhost

mysql --defaults-extra-file=db.cnf  -e "set global general_log='ON';"

2. Get the pids of sleeping MySQL connections and kill them whose sleep time is more than 100 sec.
sleeptime=100
/usr/bin/mysql -N -h $host -p$password -u $username -e "select id from information_schema.processlist where command='Sleep' and time > $sleeptime;" > kill_ids
while read id
do
/usr/bin/mysqladmin -h $host -p$password -u $username kill $id
done < kill_ids
rm -f kill_ids

3. Various options used while taking MySQL DB backup
/usr/bin/mysqldump -h $host --no-data  --routines -p$password -u $username $database > $database-Schema_only.sql
/usr/bin/mysqldump -h $host -p$password -u $username $database  --no-create-db --no-create-info --ignore-table=$database.cdrdetails --ignore-table=$database.clientsummary > $database-without_CDR_Data_tables.sql
--no-data Ã  Do not write any table row information i.e structure only
--routines Ã  Dump stored routines (procedures and functions)
--no-create-db Ã  Do not write CREATE DATABASE statements
--ignore-table=db_name.tbl_name Ã  Do not dump given table
--skip-add-drop-table Ã  Do not add a DROP TABLE statement before each CREATE TABLE statement

Sending Mail
1. Send mail with body text stored in a file.
message.txt
Hi
  Some text here
Thanks & Regards
xxxxxxxxxxxxx

mail -s "Log Rotation Failed" abc@xyz.com < message.txt

2. Send mail with attachment.
mail -a Report.txt -s "Log Rotation Failed" abc@xyz.com < message.txt

3. Send a mail with Cc
mail -a Report.txt -s "Log Rotation Failed" -c id@domain.com abc@xyz.com < message.txt

Working across multiple scripts
  1. There will be situations, where multiple scripts dependent on same data. Such data can be stored in one script, and can be exported to multiple scripts. This will help us in managing the data centrally that changes time to time.
00_CONFIG.sh
#!/bin/bash
host="192.168.10.23"
export host

database=mydbname
export database

port=3306
export port

serverids="1 2 3 4 5 6 7 8 9 10"
export serverids

recenthours="3 2 1"
export recenthours

mysql_username="myuserid"
export mysql_username

mysql_password="mypassword"
export mysql_password

01_DAILY DATA_VERIFY.sh
#!/bin/bash
source 00_CONFIG.sh

for id in `echo $serverids`
do
..................
..................
done

02_DAILY DATA_BACKUP.sh
#!/bin/bash
source 00_CONFIG.sh

for id in `echo $serverids`
do
..................
..................
done

03_DAILY DATA_DELETE.sh
#!/bin/bash
source 00_CONFIG.sh

for id in `echo $serverids`
do
..................
..................
done

Important Note: The above example will work when you run scripts manually in shell. In case if you want to schedule the jobs using crontab, you need to declare those variables (CONFIG.sh) in cron environment. In this case it looks like this.
# crontab -e 
host="192.168.10.23"
database=mydbname
port=3306
serverids="1 2 3 4 5 6 7 8 9 10"
recenthours="3 2 1"
mysql_username="myuserid"
mysql_password="mypassword"
30    5     *    *     *    /mypath_to_scripts/01_DAILY DATA_VERIFY.sh
59    23     *    *     *    /mypath_to_scripts/01_DAILY DATA_BACKUP.sh

0 comments:

Post a Comment

Subscribe to: Post Comments (Atom)