Working with text
1. Read a CSV file line by line and extract specific filed, and store it in a variable.
file_name="data.csv"
while read line
do
field1=$(echo -e $line | gawk -F, '{ print $1}')
field2=$(echo -e $line | gawk -F, '{ print $2}')
done < $file_name
2. Find and repace a string in a file.
sed -i 's/foo/bar/g' file.txt
Arithematic Operations
1. Increment a variable
count=$((count+1))
2. Decrement a variable
count=$((count-1))
Working with date and time
1. Get current date in yy-mm-dd format.
current_date="`date +%y-%m-%d`"
echo $current_date
2. Get backup time stamp in yymmdd-hhmmss format and store it in a variable.
bkptime=("`date +%y%m%d-%H%M%S`")
3. Get yesterday's date in yyyy-mm-dd format.
yesterday_date="`date --date=yesterday +%Y/%m/%d`"
echo $yesterday_date
3. Get 10 days old date in yyyy-mm-dd format.
day=("`date --date=\"10 day ago\" +%Y-%m-%d`")
OR
daysold=10
day=("`date --date=\"$daysold day ago\" +%Y-%m-%d`")
4. Get last day of the month
To get last day of current month:
date -d "-$(date +%d) days 1 month" +%Y-%m-%d
To get last day of last month:
date -d "-$(date +%d) days 0 month" +%Y-%m-%d
To get last day of last before month
date -d "-$(date +%d) days -1 month" +%Y-%m-%d
To get last day of next month
date -d "-$(date +%d) days 2 month" +%Y-%m-%d
To get last day of next to next month
date -d "-$(date +%d) days 3 month" +%Y-%m-%d
If conditions
1. Check a vriable is having some value (it is non-zero)
if [ ! -z "$var" ]; then
echo "Variable value is $val"
else
echo "Variable has no value"
fi
2. Check if file exist
if [ -e "$file.csv" ]; then
echo "File exist"
fi
3. Check if doesn't file exist
if [ ! -e "$file.csv" ]; then
echo "File exist"
fi
4. Check if previously executed command is successful or not. If not successful exit from loop.
ssconvert "file.xls" "file.csv"
if [ "$?" == "0" ]; then
echo "File converted successfully.."
else
exit
fi
Working with files
1. Covert EXCEL(.xls) file to CSV file.
ssconvert "file.xls" "file.csv"
2. Compress a directory using 7zip.
/usr/bin/7z a reportdir.7z reportdir > /dev/null
3. Upload a directory and its content to an FTP location
ftp_server="ftp.mydomain.com"
ftp_bkp_username="myusername"
ftp_bkp_password="mypassword"
FTP Over TLS
lftp -c "open -e \"set ftps:initial-prot \"\"; set ftp:ssl-force true; set ftp:ssl-protect-data true; mirror -R mydir; \" -u \"$ftp_bkp_username\",\"$ftp_bkp_password\" ftp://$ftp_server/mypath/"
FTP (without TLS)
ncftpput –R -u $ftp_bkp_username -p $ftp_bkp_password $ftp_server mypath mydir
4. Upload a file to an FTP location.
ftp_server="ftp.mydomain.com"
ftp_bkp_username="myusername"
ftp_bkp_password="mypassword"
FTP Over TLS
lftp -c "open -e \"set ftps:initial-prot \"\"; set ftp:ssl-force true; set ftp:ssl-protect-data true; put myfile; \" -u \"$ftp_bkp_username\",\"$ftp_bkp_password\" ftp://$ftp_server/mypath/"
FTP (without TLS)
ncftpput -u $ftp_bkp_username -p $ftp_bkp_password $ftp_server mypath myfile
5. Create an empty directory in an FTP location over TLS.
ftp_server="ftp.mydomain.com"
ftp_bkp_username="myusername"
ftp_bkp_password="mypassword"
lftp -c "open -e \"set ftps:initial-prot \"\"; set ftp:ssl-force true; set ftp:ssl-protect-data true; put myemptydir; \" -u \"$ftp_bkp_username\",\"$ftp_bkp_password\" ftp://$ftp_server/mypath/"
Working with user input
1. Read input from user.
echo -n "Enter date for processing data (yyyy/mm/dd) : "
read given_date
echo "You have entered $given_date"
Working with MySQL statements
1. Change a MySQL environment variable.
db.cnf
[client]
user = root
password = mypassword
host = localhost
mysql --defaults-extra-file=db.cnf -e "set global general_log='ON';"
2. Get the pids of sleeping MySQL connections and kill them whose sleep time is more than 100 sec.
sleeptime=100
/usr/bin/mysql -N -h $host -p$password -u $username -e "select id from information_schema.processlist where command='Sleep' and time > $sleeptime;" > kill_ids
while read id
do
/usr/bin/mysqladmin -h $host -p$password -u $username kill $id
done < kill_ids
rm -f kill_ids
3. Various options used while taking MySQL DB backup
/usr/bin/mysqldump -h $host --no-data --routines -p$password -u $username $database > $database-Schema_only.sql
/usr/bin/mysqldump -h $host -p$password -u $username $database --no-create-db --no-create-info --ignore-table=$database.cdrdetails --ignore-table=$database.clientsummary > $database-without_CDR_Data_tables.sql
--no-data à Do not write any table row information i.e structure only
--routines à Dump stored routines (procedures and functions)
--no-create-db à Do not write CREATE DATABASE statements
--ignore-table=db_name.tbl_name à Do not dump given table
--skip-add-drop-table à Do not add a DROP TABLE statement before each CREATE TABLE statement
Sending Mail
1. Send mail with body text stored in a file.
message.txt
Hi
Some text here
Thanks & Regards
xxxxxxxxxxxxx
mail -s "Log Rotation Failed" abc@xyz.com < message.txt
2. Send mail with attachment.
mail -a Report.txt -s "Log Rotation Failed" abc@xyz.com < message.txt
3. Send a mail with Cc
mail -a Report.txt -s "Log Rotation Failed" -c id@domain.com abc@xyz.com < message.txt
3. Send a mail with Cc
mail -a Report.txt -s "Log Rotation Failed" -c id@domain.com abc@xyz.com < message.txt
Working across multiple scripts
- There will be situations, where multiple scripts dependent on same data. Such data can be stored in one script, and can be exported to multiple scripts. This will help us in managing the data centrally that changes time to time.
00_CONFIG.sh
#!/bin/bash
host="192.168.10.23"
export host
database=mydbname
export database
port=3306
export port
serverids="1 2 3 4 5 6 7 8 9 10"
export serverids
recenthours="3 2 1"
export recenthours
mysql_username="myuserid"
export mysql_username
mysql_password="mypassword"
export mysql_password
01_DAILY DATA_VERIFY.sh
#!/bin/bash
source 00_CONFIG.sh
for id in `echo $serverids`
do
..................
..................
done
02_DAILY DATA_BACKUP.sh
#!/bin/bash
source 00_CONFIG.sh
for id in `echo $serverids`
do
..................
..................
done
03_DAILY DATA_DELETE.sh
#!/bin/bash
source 00_CONFIG.sh
for id in `echo $serverids`
do
..................
..................
done
Important Note: The above example will work when you run scripts manually in shell. In case if you want to schedule the jobs using crontab, you need to declare those variables (CONFIG.sh) in cron environment. In this case it looks like this.
# crontab -e
host="192.168.10.23"
host="192.168.10.23"
database=mydbname
port=3306
serverids="1 2 3 4 5 6 7 8 9 10"
recenthours="3 2 1"
mysql_username="myuserid"
mysql_password="mypassword"
30 5 * * * /mypath_to_scripts/01_DAILY DATA_VERIFY.sh
59 23 * * * /mypath_to_scripts/01_DAILY DATA_BACKUP.sh
59 23 * * * /mypath_to_scripts/01_DAILY DATA_BACKUP.sh
