Bash Shell Scripting - Basics

By -
Working with text

1. Read a CSV file line by line and extract specific filed, and store it in a variable.
file_name="data.csv"
while read line
do
field1=$(echo -e $line | gawk -F, '{ print $1}')
field2=$(echo -e $line | gawk -F, '{ print $2}')
done < $file_name

2. Find and repace a string in a file.
sed -i 's/foo/bar/g' file.txt

Arithematic Operations
1. Increment a variable
count=$((count+1))

2. Decrement a variable
count=$((count-1))

Working with date and time
1. Get current date in yy-mm-dd format.
current_date="`date +%y-%m-%d`"
echo $current_date

2. Get backup time stamp in yymmdd-hhmmss format and store it in a variable.
bkptime=("`date +%y%m%d-%H%M%S`")

3. Get yesterday's date in yyyy-mm-dd format.
yesterday_date="`date --date=yesterday +%Y/%m/%d`"
echo $yesterday_date

3. Get 10 days old date in yyyy-mm-dd format.
day=("`date --date=\"10 day ago\" +%Y-%m-%d`")

OR

daysold=10
day=("`date --date=\"$daysold day ago\" +%Y-%m-%d`")

4. Get last day of the month
To get last day of current month:
date -d "-$(date +%d) days 1 month" +%Y-%m-%d

To get last day of last month:
date -d "-$(date +%d) days 0 month" +%Y-%m-%d

To get last day of last before month
date -d "-$(date +%d) days -1 month" +%Y-%m-%d

To get last day of next month
date -d "-$(date +%d) days 2 month" +%Y-%m-%d

To get last day of next to next month
date -d "-$(date +%d) days 3 month" +%Y-%m-%d

If conditions

1. Check a vriable is having some value (it is non-zero)
if [ ! -z "$var" ]; then
     echo "Variable value is $val"
else
     echo "Variable has no value"
fi

2. Check if file exist
if [  -e "$file.csv" ]; then
     echo "File exist"
fi

3. Check if doesn't file exist
if [ !  -e "$file.csv" ]; then
     echo "File exist"
fi

4. Check if previously executed command is successful or not. If not successful exit from loop.
ssconvert "file.xls" "file.csv"
if [ "$?" == "0" ]; then
      echo "File converted successfully.."
else
exit
fi

Working with files

1. Covert EXCEL(.xls) file to CSV file.
ssconvert "file.xls" "file.csv"

2. Compress a directory using 7zip.
/usr/bin/7z a reportdir.7z reportdir > /dev/null

3. Upload a directory and its content to an FTP location
ftp_server="ftp.mydomain.com"
ftp_bkp_username="myusername"
ftp_bkp_password="mypassword"
FTP Over TLS
lftp -c "open -e \"set ftps:initial-prot \"\"; set ftp:ssl-force true; set ftp:ssl-protect-data true; mirror -R mydir; \" -u \"$ftp_bkp_username\",\"$ftp_bkp_password\" ftp://$ftp_server/mypath/"
FTP (without TLS)
ncftpput –R -u $ftp_bkp_username -p $ftp_bkp_password $ftp_server mypath mydir

4. Upload a file to an FTP location.
ftp_server="ftp.mydomain.com"
ftp_bkp_username="myusername"
ftp_bkp_password="mypassword"
FTP Over TLS
lftp -c "open -e \"set ftps:initial-prot \"\"; set ftp:ssl-force true; set ftp:ssl-protect-data true; put myfile; \" -u \"$ftp_bkp_username\",\"$ftp_bkp_password\" ftp://$ftp_server/mypath/"
FTP (without TLS)
ncftpput -u $ftp_bkp_username -p $ftp_bkp_password $ftp_server mypath myfile

5. Create an empty directory in an FTP location over TLS.
ftp_server="ftp.mydomain.com"
ftp_bkp_username="myusername"
ftp_bkp_password="mypassword"
lftp -c "open -e \"set ftps:initial-prot \"\"; set ftp:ssl-force true; set ftp:ssl-protect-data true; put myemptydir; \" -u \"$ftp_bkp_username\",\"$ftp_bkp_password\" ftp://$ftp_server/mypath/"


Working with user input
1. Read input from user.
echo -n "Enter date for processing data (yyyy/mm/dd) : "
read given_date
echo "You have entered $given_date"

Working with MySQL statements
1. Change a MySQL environment variable.
 db.cnf
[client]
user = root
password = mypassword
host = localhost

mysql --defaults-extra-file=db.cnf  -e "set global general_log='ON';"

2. Get the pids of sleeping MySQL connections and kill them whose sleep time is more than 100 sec.
sleeptime=100
/usr/bin/mysql -N -h $host -p$password -u $username -e "select id from information_schema.processlist where command='Sleep' and time > $sleeptime;" > kill_ids
while read id
do
/usr/bin/mysqladmin -h $host -p$password -u $username kill $id
done < kill_ids
rm -f kill_ids

3. Various options used while taking MySQL DB backup
/usr/bin/mysqldump -h $host --no-data  --routines -p$password -u $username $database > $database-Schema_only.sql
/usr/bin/mysqldump -h $host -p$password -u $username $database  --no-create-db --no-create-info --ignore-table=$database.cdrdetails --ignore-table=$database.clientsummary > $database-without_CDR_Data_tables.sql
--no-data Ã  Do not write any table row information i.e structure only
--routines Ã  Dump stored routines (procedures and functions)
--no-create-db Ã  Do not write CREATE DATABASE statements
--ignore-table=db_name.tbl_name Ã  Do not dump given table
--skip-add-drop-table Ã  Do not add a DROP TABLE statement before each CREATE TABLE statement

Sending Mail
1. Send mail with body text stored in a file.
message.txt
Hi
  Some text here
Thanks & Regards
xxxxxxxxxxxxx

mail -s "Log Rotation Failed" abc@xyz.com < message.txt

2. Send mail with attachment.
mail -a Report.txt -s "Log Rotation Failed" abc@xyz.com < message.txt

3. Send a mail with Cc
mail -a Report.txt -s "Log Rotation Failed" -c id@domain.com abc@xyz.com < message.txt

Working across multiple scripts
  1. There will be situations, where multiple scripts dependent on same data. Such data can be stored in one script, and can be exported to multiple scripts. This will help us in managing the data centrally that changes time to time.
00_CONFIG.sh
#!/bin/bash
host="192.168.10.23"
export host

database=mydbname
export database

port=3306
export port

serverids="1 2 3 4 5 6 7 8 9 10"
export serverids

recenthours="3 2 1"
export recenthours

mysql_username="myuserid"
export mysql_username

mysql_password="mypassword"
export mysql_password

01_DAILY DATA_VERIFY.sh
#!/bin/bash
source 00_CONFIG.sh

for id in `echo $serverids`
do
..................
..................
done

02_DAILY DATA_BACKUP.sh
#!/bin/bash
source 00_CONFIG.sh

for id in `echo $serverids`
do
..................
..................
done

03_DAILY DATA_DELETE.sh
#!/bin/bash
source 00_CONFIG.sh

for id in `echo $serverids`
do
..................
..................
done

Important Note: The above example will work when you run scripts manually in shell. In case if you want to schedule the jobs using crontab, you need to declare those variables (CONFIG.sh) in cron environment. In this case it looks like this.
# crontab -e 
host="192.168.10.23"
database=mydbname
port=3306
serverids="1 2 3 4 5 6 7 8 9 10"
recenthours="3 2 1"
mysql_username="myuserid"
mysql_password="mypassword"
30    5     *    *     *    /mypath_to_scripts/01_DAILY DATA_VERIFY.sh
59    23     *    *     *    /mypath_to_scripts/01_DAILY DATA_BACKUP.sh

Comments

Post a Comment

Popular posts from this blog

grep: unknown device method

By Raaz -
Today while using grep command via rundeck I was through grep: unknown device method  error. And the reason is due to I am having ' - ' in my search pattern I was through this error. I mean search includes grep "-search.this" /path/to/file Then you may get error with grep. So remove ' - ' in pattern and repeat your search. grep "search.this" /path/to/file Hope that helps.
Read more

Uploading files to FTP/SFTP using CURL

By Raaz -
Hello, Today I am writing below article which can help you to upload files to SFTP/FTP by using CURL. Ok why we need that ? Let me tell explain!! How we login into SFTP/FTP ? [root@virt03 test]# sftp 192.168.56.110 Connecting to 192.168.56.110... root@192.168.56.110's password: sftp> ls anaconda-ks.cfg       nodes                 post-install          post-install.log sftp> exit and uploading files with put command. Its a lengthy way. So recently I have gone through few articles and with some R&D I have modified it as script and command-line argument support.So you can call the script with filename as argument.  So Lets do this!!! Command 1 : This is for uploading a single to SFTP/FTP by using CURL.  SFTP curl -k  -u virt03:virt03 -T file4  sftp://192.168.56.110/home/virt03/ Syntax :  curl -k  -u username:password -T filename sftp://IP_Addreess OR Hostname:/path/to/upload  FTP curl -k  -u virt03:virt03 -T f
Read more

How to find outgoing IP in Linux ?

By -
Here is a small snippet to find outgoing IP of your Linux box. 1. To find the outgoing internal IP only         # ip route get 8.8.8.8 | head -1 | gawk '{ print $7 }' 2. To find the outgoing internal IP along with interface.   # ip route get 8.8.8.8 | head -1 | gawk '{ print $5,$7 }' 3. In-case if your system is connected to Internet, to find the outgoing external IP   # curl ifconfig.me Note:   In case if your system having public IP assigned then there is no difference between outgoing internal IP and external IP (both are same).
Read more