How To - Shell Scripting - Bash - Basics - 2 - AWK command usage

Hello Everyone,

In this post I will share what I have learned on awk.

A little bit introuduction of awk

AWK is a domain-specific language designed for text processing and
typically used as a data extraction and reporting tool. Like sed and
grep, it is a filter, and is a standard feature of most Unix-like
operating systems

awk has more variety of usages and all that information is well documented at
https://www.gnu.org/software/gawk/manual/gawk.html

All the code [more code in Gitlab repo than in blog, cause I am too lazy to reproduce my work :( ], along with sample files I am using in this blog post are commited to Github: https://github.com/rajagennu/awk_tutorial

AWK IF-Else

File: answers.txt

a,1,1
b,3,4
c,5,2
d,6,1
e,3,3
f,3,7

awk  -F  ','  '{if($2==$3){print $1","$2","$3} else {print "No Duplicates"}}' answers.txt

output

a,1,1
No Duplicates
No Duplicates
No Duplicates
e,3,3
No Duplicates

AWK While

First lets understand what NF in awk. as per documentation ‘NF is a predefined variable whose value is the number of fields in the current record. awk automatically updates the value of NF each time it reads a record. No matter how many fields there are, the last field in a record can be represented by $NF’

File: top10CEO.txt

Rich Lesser, Boston Consulting Group,99%
Shantanu Narayen, Adobe,99%
Peter Pisters, MD Anderson Cancer Center,99%
Gary C. Kelly, Southwest Airlines,98%
Alfred F. Kelly, Jr. Visa Inc.,97%
Satya Nadella, Microsoft,97%
Charles C. Butt, H.E.B.,97%
Ed Bastian, Delta Air Lines,97%
Paul Cormier, Red Hat,97%
Horacio D. Rozanski, Booz Allen Hamilton,97%

awk  -F',' '{i=0; while(i<=NF) { print i ":"$i; i++;}}' top10CEO.txt

0:Rich Lesser, Boston Consulting Group,99%
1:Rich Lesser
2: Boston Consulting Group
3:99%
0:Shantanu Narayen, Adobe,99%
1:Shantanu Narayen
2: Adobe
3:99%
0:Peter Pisters, MD Anderson Cancer Center,99%
1:Peter Pisters
2: MD Anderson Cancer Center
3:99%
0:Gary C. Kelly, Southwest Airlines,98%
1:Gary C. Kelly
2: Southwest Airlines
3:98%
0:Alfred F. Kelly, Jr. Visa Inc.,97%
1:Alfred F. Kelly
2: Jr. Visa Inc.
3:97%
0:Satya Nadella, Microsoft,97%
1:Satya Nadella
2: Microsoft
3:97%
0:Charles C. Butt, H.E.B.,97%
1:Charles C. Butt
2: H.E.B.
3:97%
0:Ed Bastian, Delta Air Lines,97%
1:Ed Bastian
2: Delta Air Lines
3:97%
0:Paul Cormier, Red Hat,97%
1:Paul Cormier
2: Red Hat
3:97%
0:Horacio D. Rozanski, Booz Allen Hamilton,97%
1:Horacio D. Rozanski
2: Booz Allen Hamilton
3:97%

AWK for loop

awk '{for (i = 1; i <= 3; i++) print $i}' top10CEO.txt

AWK Selectors

Selectors used for deciding whether a particular awk action should be executed or not.
For example display only CEOs with their name starting ‘S’

awk  -F','  '$1 ~ /^S/ {print $0}' top10CEO.txt

Shantanu Narayen, Adobe,99%
Satya Nadella, Microsoft,97%

relational expressions

awk  -F','  '$3 > "98%" {print $0}' top10CEO.txt

Rich Lesser, Boston Consulting Group,99%
Shantanu Narayen, Adobe,99%
Peter Pisters, MD Anderson Cancer Center,99%

Range patterns

awk  -F','  '/Peter Pisters/,/Satya Nadella/ {print $1 $3}' top10CEO.txt

Peter Pisters99%
Gary C. Kelly98%
Alfred F. Kelly97%
Satya Nadella97%

BEGIN…END

Special expression patterns include BEGIN and END which denote program initialization and end. The BEGIN pattern matches the beginning of the input, before the first record is processed. The END pattern matches the end of the input, after the last record has been processed.

awk  -F','  'BEGIN { print "starting list of top 10 CEOs" }; {print $1 $2 $3} END{print "end list of top 10 CEOs"}' top10CEO.txt

starting list of top 10 CEOs
Rich Lesser Boston Consulting Group99%
Shantanu Narayen Adobe99%
Peter Pisters MD Anderson Cancer Center99%
Gary C. Kelly Southwest Airlines98%
Alfred F. Kelly Jr. Visa Inc.97%
Satya Nadella Microsoft97%
Charles C. Butt H.E.B.97%
Ed Bastian Delta Air Lines97%
Paul Cormier Red Hat97%
Horacio D. Rozanski Booz Allen Hamilton97%
end list of top 10 CEOs

AWK ‘&&’ ‘||’ ‘!’

AWK supports && || !
I would like to see the CEO with score greater than 97% and starting with letter S

awk  -F','  '$3 > "97%" && $1 ~/^S/ { print $1}' top10CEO.txt

Shantanu Narayen

AWK variables

• $0 -> Print full line
• $1, $2… -> Field 1 and File 2…
• NR-> Number of row, usually print the current row number

echo  -e  "Hello\nGoodMorning"  |  awk  '{print NR"\t" $0}'
1   Hello
2   GoodMorning

• NF-> Number of fields, when you call NF it will print number of fields, and when you call $NF it will print last field, so if you have a use case like you would like to get last field, then $NF is the best case

echo  -e  "123,459,905\n456,544,345"  |  awk  -F','  '{print $NF}'
echo  -e  "123,459,905\n456,544,345"  |  awk  -F','  '{print NF}'

905
345
3
3

Dont mess with IFS and RS unless you know what you are doing.

AWK Length

awk  -F','  '{print "number of chars in line " NR "=" length($0)}' top10CEO.txt

number of chars in line 1=40
number of chars in line 2=27
number of chars in line 3=44
number of chars in line 4=37
number of chars in line 5=34
number of chars in line 6=28
number of chars in line 7=27
number of chars in line 8=31
number of chars in line 9=25
number of chars in line 10=44

Bottom Line:
AWK is a great text/file processing/manipulation with so many use cases. I just added what I have learned, if you like this post, please subscribe to my blog and add a star on my gitrepo : https://github.com/rajagennu/awk_tutorial

Thakn you.

How To - Shell Scripting - Bash - Basics - 1 - Functions

Hello Everyone,

I will try publishing a series of articles on shell scripting. We are going to use bash which is default shell in almost all famous linux & unit operating systems.

Functions

The theory behind creating a function is simple, if you have a piece of code and you are going to use over and over, then instead of rewriting the code, you will make it as a function.

Function Syntax:

function function_name {
# commands
}

In bash we have one more syntax for definiting functions.

function_name() { 
# commands
}

So lets get to the point, where functions are for better use. For example, lets say you are printing a format message for the commands you have executed and your code is like below

uptime_output=$(uptime)
free_ram=$(free -m)
cpu_core=$(nproc)

echo "Command : uptime output $uptime_output"
echo "Command : Free RAM $free_ram"
echo "Command : CPU Cores $cpu_core"

Here the line echo "Command : uptime output $uptime_output" has a pattern and its repeating for all the commands that we have executed. Now we got 3 commands, assume we have 10 commands, then without functions, we have to write this for 10 times.

Instead if we go with functions, our code looks more cleaner way and easy to manage in future.

function format_message {
  first_arg=$1
  shift # first arg is string, $2 onwards we have output, to set cursor to read from $2, we are using shift
  echo "Command $first_arg output: " "$@"
}
format_message "uptime: " "$(uptime)"
format_message "Free RAM: " "$(free -m)"
format_message "Number of CPU Cores: " "$(nproc)"

So now no matter, how many commands, you have you dont have to write that format message everytime.

But, wait a minute, this is still can be optimized, so all you are doing, doing something repetitively, we can use a for loop in shell scripting

cmds=('uptime' 'free -m' 'nproc') # Bash Array
function format_output {
  first_arg=$1
  shift
  echo -e "Execution output of $first_arg\n$@"
}

for cmd in "${cmds[@]}"; # iterating over bash array
do
format_output "$cmd" "$($cmd)"
done

and that’s it.

You only have to update the cmds variable with new command and your code behaves in the same way for every command.

vagrant@vagrant-ubuntu-trusty-64:~$ bash for_my_script.sh
Execution output of uptime
 05:04:49 up  1:18,  1 user,  load average: 0.06, 0.02, 0.00
Execution output of free -m
              total        used        free      shared  buff/cache   available
Mem:            488         106          43           1         338         357
Swap:             0           0           0
Execution output of nproc
1
vagrant@vagrant-ubuntu-trusty-64:~$

This post has covered

• How to define functions in bash
• How to pass command line arguments functions
• How to manipulate command line arguments
• How to define array and iterate over array in bash

Hope it helps.
Thank you.

How To: Logging with Javascript using Date class

Hello,

If you are trying to print a neat date format in Javascript using Date() class, then you can use below snippet.

Its a ES6 based function with template literals.

  
  const date = () => {
  const d = new Date();
  return `${d.getFullYear()}-${d.getMonth()}-${d.getDate()}_${d.getHours()}-${d.getMinutes()}-${d.getSeconds()}_${d.getMilliseconds()} => `;
};

 

If you are looking for ES5 syntax, you can use below one

 
  var getDate = function() {
    var d = new Date();
    return d.getFullYear()+"-"+d.getMonth()+"-"+d.getDate()+"_"+d.getHours()+"-"+d.getMinutes()+"-"+d.getSeconds()+"_"+d.getMilliseconds()+" =>";
  };
  
  

Hope it helps.

Thank you.

How to resolve VERR_NEM_VM_CREATE_FAILED in Windows ?

Hello Everyone,
So I tried to install VirtuaBox VM in Windows 10 and I was getting VERR_NEM_VM_CREATE_FAILED error. After trying many solutions and searching a log in internet, Finally issue got resolved with below command.

1. Open cmd prompt as Administrator.
2. Shut down all programs. You will have to shut down, unplug, and restart your host.
3. Ensure that none of these things are running:

• DeviceGuard
• CredentialGuard
• Windows Defender’s Core Isolation

4. Find the Command Prompt icon, right click it and choose Run As Administrator.
5. Enter this command:

   bcdedit /set hypervisorlaunchtype off  

Then run below command

DISM /Online /Disable-Feature:Microsoft-Hyper-V

5. Enter this command to shutdown your system

shutdown -s -t 2  

6. When the computer turns off, unplug it for 20 seconds. Then plug it in again and boot up Windows 10.

After following these instructions, my issue got resoved and I am able to bootup again.

Credit Goes to : virtualbox.org • View topic - VERR_NEM_VM_CREATE_FAILED: What do I do?

Hope it helps.
Thank you.

ServiceNow: How to get current scope name ?

Hello Everyone

If you want to know what is your application scope name, then you can use below API

  gs.info ( gs.getCurrentScopeName() );
  
  

Hope it helps.

Thank you

How to: Bash PID of last execute command

Hello Everyone

In shell scripting if you want to get PID of last executed command, then you can use BASH `$!` meta variable.

For example lets say if you have below command

# nohup sha256sum 1g.bin &

If you want to find that job PID which is running in the background, use below command

  # echo $!

For Example

[x:/tmp]# nohup sha256sum 1g.bin &
[1] 14866
[x:/tmp]# nohup: ignoring input and appending output to `nohup.out'

[x:/tmp]# echo $!
14866
[x:/tmp]# pgrep sha256sum
14866

Hope it helps.

Thank you.