Python: JSON module to read, write and manipulate JSON

Hello Everyone,

Youtube Video available Here

In this blog post I am going to explain you about how to write, read and manipulate JSON data using python.

In Python we have support for json by default. By using

import py

We can load all the libraries needed to work with JSON data.

Loading JSON data into python

We can use json.load() method to load the JSON object from external sources like database, files etc.

I have created a file locally with below content

file:sample.json

[
	{
		"python": {
		"type": "interpreter",
		"mode": "dynamic"
			},
		"c++" :{
		"type":  "compiler",
		"mode": "static"
			},
		"javascript" : {
			"type":  "interpreter",
			"mode": "dynamic"
			}
	}
]

And to load this JSON into python program use load method as below

In [2]: import json

In [3]: with open("sample.json", "r") as file:
...:     json_data = json.load(file)
...: 

And you can access specific object and attribute and their value as below

In [5]: json_data
Out[5]: 
[{'python': {'type': 'interpreter', 'mode': 'dynamic'},
'c++': {'type': 'compiler', 'mode': 'static'},
'javascript': {'type': 'interpreter', 'mode': 'dynamic'}}]

In [6]: json_data[0]
Out[6]: 
{'python': {'type': 'interpreter', 'mode': 'dynamic'},
'c++': {'type': 'compiler', 'mode': 'static'},
'javascript': {'type': 'interpreter', 'mode': 'dynamic'}}

In [7]: json_data[0]["python"]
Out[7]: {'type': 'interpreter', 'mode': 'dynamic'}

In [8]: json_data[0]["python"]["type"]
Out[8]: 'interpreter'

You can add new JSON data as well to existing object.

This code created entirely new object structure

In [9]: json_data.append( { "c" : {"type" : "interpreter", "mode": "static" } } );

In [10]: json_data
Out[10]: 
[{'python': {'type': 'interpreter', 'mode': 'dynamic'},
'c++': {'type': 'compiler', 'mode': 'static'},
'javascript': {'type': 'interpreter', 'mode': 'dynamic'}},
{'c': {'type': 'interpreter', 'mode': 'static'}}]

and this code created another JSON object in the same nested structure, observe the output you will figure it out, I am lazy to spoon feed :D

In [11]: json_data[0]["c"] = {"type" : "interpreter", "mode": "static" }

In [12]: json_data
Out[12]: 
[{'python': {'type': 'interpreter', 'mode': 'dynamic'},
'c++': {'type': 'compiler', 'mode': 'static'},
'javascript': {'type': 'interpreter', 'mode': 'dynamic'},
'c': {'type': 'interpreter', 'mode': 'static'}},
{'c': {'type': 'interpreter', 'mode': 'static'}}]

And now if you want to write this updated JSON object into a file you can use dump method of json module.

Writing JSON data from python

In [15]: with open("sample.json", "w") as file:
...:     json.dump(json_data, file , indent=4)

output file

In [16]: cat sample.json
[
	{
	"python": {
		"type": "interpreter",
		"mode": "dynamic"
		},
	"c++": {
		"type": "compiler",
		"mode": "static"
		},
	"javascript": {
		"type": "interpreter",
		"mode": "dynamic"
		},
	"c": {
		"type": "interpreter",
		"mode": "static"
		}
	},
	{
	"c": {
		"type": "interpreter",
		"mode": "static"
		}
	}
]

Hope it helps.

Thank you.

How to: Nginx Webserver Tutorial

Hello Everyone,
I am starting a new Nginx series, planning to publish complete end to end tutorial.

Setup, Installation, Starting Service and Accessing Initial webpage.

Setup

• Docker based Ubuntu container with custom network and port forwarding from 80 of the container to 8080 of the host machine, command is as below

docker run --net bridge -it -p 8080:80 --name ubuntu-nginx ubuntu

if you would like to give static IP, you can also give that with --ip 172.18.0.4 argument after --net bridge argument in the above command.

Installation

apt update
apt install nginx -y

NOTE : If DNS Resolution not working inside container, execute below command

echo "nameserver 8.8.8.8" >> /etc/resolv.conf

and then try again above installation steps.

Service

Once installed you can start the service with

service nginx start
# verify same
service nginx status

and access the service on locahost as we have added port forward while creating the container

__$ curl -I localhost:8080
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Wed, 03 Nov 2021 18:19:26 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Wed, 03 Nov 2021 17:49:52 GMT
Connection: keep-alive
ETag: "6182cbc0-264"
Accept-Ranges: bytes

or in your favorite browser you can access URL localhost:8080

This concludes a very simple setup of nginx server without any content init.

Basic Nginx Configuration

Quick look at nginx.conf file

Nginx configuration file located at /etc/nginx/nginx.conf you can verify with below command

root@6787c6e55a5d:/# ls /etc/nginx/nginx.conf 
/etc/nginx/nginx.conf
root@6787c6e55a5d:/# 

if you are unable to finx nginx.conf file after installation, you can use below find command

root@6787c6e55a5d:/# find / -name nginx.conf -type f -print 2>/dev/null
/etc/nginx/nginx.conf
root@6787c6e55a5d:/# 

Lets look at first couple of line of nginx.conf

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

• user www-data -> The line saying that the nginx worker process will be started by user www-data, so just to verify that look at the output of ps -ef | grep nginx

root@6787c6e55a5d:/# ps -ef | grep nginx
root          28       1  0 07:39 ?        00:00:00 nginx: master process /usr/sbin/nginx
www-data      29      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      30      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      31      28  0 07:39 ?        00:00:00 nginx: worker process

• worker_processes auto; : worker_processes indicates number of workers available in nginx to serve requests, and it can be either integer or auto, when you mention auto as your value, if you have 4 CPU cores, then nginx will span 4 workers automatically, if you have 10 CPUs, 10 workers. For example I have 16 CPUs in my laptop, so I am supposed to have 16 workers, as my configuration is auto.

root@6787c6e55a5d:/# nproc
16
root@6787c6e55a5d:/# 

root@6787c6e55a5d:/# ps -ef | grep "[n]ginx"
root          28       1  0 07:39 ?        00:00:00 nginx: master process /usr/sbin/nginx
www-data      29      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      30      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      31      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      32      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      34      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      35      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      36      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      37      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      38      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      39      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      40      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      41      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      42      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      43      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      44      28  0 07:39 ?        00:00:00 nginx: worker process
www-data      45      28  0 07:39 ?        00:00:00 nginx: worker process
root@6787c6e55a5d:/# ps -ef | grep "[n]ginx" | wc -l
17
root@6787c6e55a5d:/# 

you are seeing 17 lines which includes line 1 which is main process.

• If I change the worker_process to any integer like 1, then I should be having inly 1 worker thread.

root@6787c6e55a5d:/# ps -ef | grep nginx
root        2718       1  0 08:00 ?        00:00:00 nginx: master process /usr/sbin/nginx
www-data    2719    2718  0 08:00 ?        00:00:00 nginx: worker process
www-data    2721    2718  0 08:00 ?        00:00:00 nginx: worker process
root        2723       1  0 08:00 pts/0    00:00:00 grep --color=auto nginx
root@6787c6e55a5d:/# grep worker_process /etc/nginx/nginx.conf 
worker_processes 2;
root@6787c6e55a5d:/# 

• pid /run/nginx.pid; : this line includes the PID of nginx master process

root@6787c6e55a5d:/# cat /run/nginx.pid 
2718
root@6787c6e55a5d:/# 

• include /etc/nginx/modules-enabled/*.conf; : This is the include directive, which will be used to include configuration files from a specific directory and as you can see regex also supported here.

• The remaining lines of the nginx.conf are self explanatory, but I am still covering some of them below. As per initial installation, you would see only basic configuration only.

• We can have more configuration files and couple of them are here as follows, sort of important ones.
  |Standard Name|Description |
  |-------------|---------------------------------------------------------|
  |nginx.conf |as you know this is the main config file |
  |mime.types |A list of file extensions and their associated MIME types|
  |fastcgi.conf |Fast CGI-Related configuration |
  |proxy.conf |Proxy related configuration |
  |sites.conf |virtual host related configuration |

• Its not mandatory to use this predefined config filenames, you can copy all those sections and paste into nginx.conf and it will work without any issues.

• Its just matter of maintenance when your nginx inventory growing larger.

• if you have done changes to your configuration file, it is always recommended to verify the syntax of config file with command nginx -t

root@6787c6e55a5d:/# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
root@6787c6e55a5d:/# 

• When you install a module, you would define its respective configuration in it directive blocks. From the nginx.conf file

events {
        worker_connections 768;
        # multi_accept on;
}

from the above line, the understanding we need that, events module enabled and with direcive blocks, we are specifying configuration for events modules.

• But you might think what is the meaning of worker_connections and is it somehow related to worker_procesess means, Yes.
• if you defined your worker_processess as 1 and worker_connections as 768, that means, our nginx server can serve 768 clients at a time, if worker_processes as 2 then as per above worker_connection configuration 768*2 = 1536 connections it can server at a time. so this is the relation between these two.
• Variables will start with $ in nginx, they are similar to bash variables. But not all directives supports variables. for example log_format directive supports variables but not error_log directive.
• for blank space, semicolor, (), {}, better enclosing them as strings either single quote or double quote.

Base Module Directives

Base modules are like preinstalled and already enabled modules in nginx. These modules are available by default and help nginx with basic functionality.

- core Module: Essential features, directives such as process management and security.
- Events Module: lets you configure the inner mechanisms of the networking capabilities. 
- Configuration Module: enables the inclusion mechanism. 

Nginx process architecture

• when nginx service started, with the user who launched this process, a master process will be started, and this master process will spawn worker process number in nginx.conf file.

Core Module Directives

• name: daemon
  • if you set it off then nginx will not start in backgroud, it always stays in foreground, this is the best option for debugging. By default its always on unless you explicitly set it to daemon off

  root@88c06ed24ab3:/# grep daemon /etc/nginx/nginx.conf                                     
daemon off                                                                                 
root@88c06ed24ab3:/# nginx                                                                 
nginx: [emerg] directive "daemon" is not terminated by ";" in /etc/nginx/nginx.conf:6      
root@88c06ed24ab3:/# nginx -t                                                              
nginx: [emerg] directive "daemon" is not terminated by ";" in /etc/nginx/nginx.conf:6      
nginx: configuration file /etc/nginx/nginx.conf test failed                                
root@88c06ed24ab3:/# vim /etc/nginx/nginx.conf                                             
root@88c06ed24ab3:/# nginx -t                                                              
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok                           
nginx: configuration file /etc/nginx/nginx.conf test is successful                         
root@88c06ed24ab3:/# nginx    # here as the daemon off, process running in the foreground.                                                             
^Croot@88c06ed24ab3:/# vim /etc/nginx/nginx.conf                                           
root@88c06ed24ab3:/# grep daemon /etc/nginx/nginx.conf                                     
daemon on;                                                                                 
root@88c06ed24ab3:/# nginx    # here as the daemon on, process running in the background.                                                              
root@88c06ed24ab3:/#                                                                       
root@88c06ed24ab3:/#                                                                       

Motivation: Invest in yourself

 By looking at this post, you might think like, save some money -> learn something kind of post. No but its not that. 

I just saw a video stating that, every day in an account named "you" there are 86400 ( 24 * 60 * 60 ) seconds are crediting. And I am saying that you invest them wisely. 

Invest them in happiness, Invest them in health, Invest them in good sleep, Invest them in your career to move forward, to learn something new.

Dont invest those valuable seconds in watching youTube while you are supposed to work.

Don't invest those valuable seconds in watching Netflix while you are supposed to sleep.

Don't invest those valuable seconds by sitting lazy while you are supposed to have a morning/evening walk or a workout.

Its all investment and good investment grows( increases) the value(your life, family ) and bad investment crush you down to your knees. 

Thanks for reading.

Raaz.

How To - Shell Scripting - Bash - Basics - 2 - AWK command usage

Hello Everyone,

In this post I will share what I have learned on awk.

A little bit introuduction of awk

AWK is a domain-specific language designed for text processing and
typically used as a data extraction and reporting tool. Like sed and
grep, it is a filter, and is a standard feature of most Unix-like
operating systems

awk has more variety of usages and all that information is well documented at
https://www.gnu.org/software/gawk/manual/gawk.html

All the code [more code in Gitlab repo than in blog, cause I am too lazy to reproduce my work :( ], along with sample files I am using in this blog post are commited to Github: https://github.com/rajagennu/awk_tutorial

AWK IF-Else

File: answers.txt

a,1,1
b,3,4
c,5,2
d,6,1
e,3,3
f,3,7

awk  -F  ','  '{if($2==$3){print $1","$2","$3} else {print "No Duplicates"}}' answers.txt

output

a,1,1
No Duplicates
No Duplicates
No Duplicates
e,3,3
No Duplicates

AWK While

First lets understand what NF in awk. as per documentation ‘NF is a predefined variable whose value is the number of fields in the current record. awk automatically updates the value of NF each time it reads a record. No matter how many fields there are, the last field in a record can be represented by $NF’

File: top10CEO.txt

Rich Lesser, Boston Consulting Group,99%
Shantanu Narayen, Adobe,99%
Peter Pisters, MD Anderson Cancer Center,99%
Gary C. Kelly, Southwest Airlines,98%
Alfred F. Kelly, Jr. Visa Inc.,97%
Satya Nadella, Microsoft,97%
Charles C. Butt, H.E.B.,97%
Ed Bastian, Delta Air Lines,97%
Paul Cormier, Red Hat,97%
Horacio D. Rozanski, Booz Allen Hamilton,97%

awk  -F',' '{i=0; while(i<=NF) { print i ":"$i; i++;}}' top10CEO.txt

0:Rich Lesser, Boston Consulting Group,99%
1:Rich Lesser
2: Boston Consulting Group
3:99%
0:Shantanu Narayen, Adobe,99%
1:Shantanu Narayen
2: Adobe
3:99%
0:Peter Pisters, MD Anderson Cancer Center,99%
1:Peter Pisters
2: MD Anderson Cancer Center
3:99%
0:Gary C. Kelly, Southwest Airlines,98%
1:Gary C. Kelly
2: Southwest Airlines
3:98%
0:Alfred F. Kelly, Jr. Visa Inc.,97%
1:Alfred F. Kelly
2: Jr. Visa Inc.
3:97%
0:Satya Nadella, Microsoft,97%
1:Satya Nadella
2: Microsoft
3:97%
0:Charles C. Butt, H.E.B.,97%
1:Charles C. Butt
2: H.E.B.
3:97%
0:Ed Bastian, Delta Air Lines,97%
1:Ed Bastian
2: Delta Air Lines
3:97%
0:Paul Cormier, Red Hat,97%
1:Paul Cormier
2: Red Hat
3:97%
0:Horacio D. Rozanski, Booz Allen Hamilton,97%
1:Horacio D. Rozanski
2: Booz Allen Hamilton
3:97%

AWK for loop

awk '{for (i = 1; i <= 3; i++) print $i}' top10CEO.txt

AWK Selectors

Selectors used for deciding whether a particular awk action should be executed or not.
For example display only CEOs with their name starting ‘S’

awk  -F','  '$1 ~ /^S/ {print $0}' top10CEO.txt

Shantanu Narayen, Adobe,99%
Satya Nadella, Microsoft,97%

relational expressions

awk  -F','  '$3 > "98%" {print $0}' top10CEO.txt

Rich Lesser, Boston Consulting Group,99%
Shantanu Narayen, Adobe,99%
Peter Pisters, MD Anderson Cancer Center,99%

Range patterns

awk  -F','  '/Peter Pisters/,/Satya Nadella/ {print $1 $3}' top10CEO.txt

Peter Pisters99%
Gary C. Kelly98%
Alfred F. Kelly97%
Satya Nadella97%

BEGIN…END

Special expression patterns include BEGIN and END which denote program initialization and end. The BEGIN pattern matches the beginning of the input, before the first record is processed. The END pattern matches the end of the input, after the last record has been processed.

awk  -F','  'BEGIN { print "starting list of top 10 CEOs" }; {print $1 $2 $3} END{print "end list of top 10 CEOs"}' top10CEO.txt

starting list of top 10 CEOs
Rich Lesser Boston Consulting Group99%
Shantanu Narayen Adobe99%
Peter Pisters MD Anderson Cancer Center99%
Gary C. Kelly Southwest Airlines98%
Alfred F. Kelly Jr. Visa Inc.97%
Satya Nadella Microsoft97%
Charles C. Butt H.E.B.97%
Ed Bastian Delta Air Lines97%
Paul Cormier Red Hat97%
Horacio D. Rozanski Booz Allen Hamilton97%
end list of top 10 CEOs

AWK ‘&&’ ‘||’ ‘!’

AWK supports && || !
I would like to see the CEO with score greater than 97% and starting with letter S

awk  -F','  '$3 > "97%" && $1 ~/^S/ { print $1}' top10CEO.txt

Shantanu Narayen

AWK variables

• $0 -> Print full line
• $1, $2… -> Field 1 and File 2…
• NR-> Number of row, usually print the current row number

echo  -e  "Hello\nGoodMorning"  |  awk  '{print NR"\t" $0}'
1   Hello
2   GoodMorning

• NF-> Number of fields, when you call NF it will print number of fields, and when you call $NF it will print last field, so if you have a use case like you would like to get last field, then $NF is the best case

echo  -e  "123,459,905\n456,544,345"  |  awk  -F','  '{print $NF}'
echo  -e  "123,459,905\n456,544,345"  |  awk  -F','  '{print NF}'

905
345
3
3

Dont mess with IFS and RS unless you know what you are doing.

AWK Length

awk  -F','  '{print "number of chars in line " NR "=" length($0)}' top10CEO.txt

number of chars in line 1=40
number of chars in line 2=27
number of chars in line 3=44
number of chars in line 4=37
number of chars in line 5=34
number of chars in line 6=28
number of chars in line 7=27
number of chars in line 8=31
number of chars in line 9=25
number of chars in line 10=44

Bottom Line:
AWK is a great text/file processing/manipulation with so many use cases. I just added what I have learned, if you like this post, please subscribe to my blog and add a star on my gitrepo : https://github.com/rajagennu/awk_tutorial

Thakn you.

How To - Shell Scripting - Bash - Basics - 1 - Functions

Hello Everyone,

I will try publishing a series of articles on shell scripting. We are going to use bash which is default shell in almost all famous linux & unit operating systems.

Functions

The theory behind creating a function is simple, if you have a piece of code and you are going to use over and over, then instead of rewriting the code, you will make it as a function.

Function Syntax:

function function_name {
# commands
}

In bash we have one more syntax for definiting functions.

function_name() { 
# commands
}

So lets get to the point, where functions are for better use. For example, lets say you are printing a format message for the commands you have executed and your code is like below

uptime_output=$(uptime)
free_ram=$(free -m)
cpu_core=$(nproc)

echo "Command : uptime output $uptime_output"
echo "Command : Free RAM $free_ram"
echo "Command : CPU Cores $cpu_core"

Here the line echo "Command : uptime output $uptime_output" has a pattern and its repeating for all the commands that we have executed. Now we got 3 commands, assume we have 10 commands, then without functions, we have to write this for 10 times.

Instead if we go with functions, our code looks more cleaner way and easy to manage in future.

function format_message {
  first_arg=$1
  shift # first arg is string, $2 onwards we have output, to set cursor to read from $2, we are using shift
  echo "Command $first_arg output: " "$@"
}
format_message "uptime: " "$(uptime)"
format_message "Free RAM: " "$(free -m)"
format_message "Number of CPU Cores: " "$(nproc)"

So now no matter, how many commands, you have you dont have to write that format message everytime.

But, wait a minute, this is still can be optimized, so all you are doing, doing something repetitively, we can use a for loop in shell scripting

cmds=('uptime' 'free -m' 'nproc') # Bash Array
function format_output {
  first_arg=$1
  shift
  echo -e "Execution output of $first_arg\n$@"
}

for cmd in "${cmds[@]}"; # iterating over bash array
do
format_output "$cmd" "$($cmd)"
done

and that’s it.

You only have to update the cmds variable with new command and your code behaves in the same way for every command.

vagrant@vagrant-ubuntu-trusty-64:~$ bash for_my_script.sh
Execution output of uptime
 05:04:49 up  1:18,  1 user,  load average: 0.06, 0.02, 0.00
Execution output of free -m
              total        used        free      shared  buff/cache   available
Mem:            488         106          43           1         338         357
Swap:             0           0           0
Execution output of nproc
1
vagrant@vagrant-ubuntu-trusty-64:~$

This post has covered

• How to define functions in bash
• How to pass command line arguments functions
• How to manipulate command line arguments
• How to define array and iterate over array in bash

Hope it helps.
Thank you.

How To: Logging with Javascript using Date class

Hello,

If you are trying to print a neat date format in Javascript using Date() class, then you can use below snippet.

Its a ES6 based function with template literals.

  
  const date = () => {
  const d = new Date();
  return `${d.getFullYear()}-${d.getMonth()}-${d.getDate()}_${d.getHours()}-${d.getMinutes()}-${d.getSeconds()}_${d.getMilliseconds()} => `;
};

 

If you are looking for ES5 syntax, you can use below one

 
  var getDate = function() {
    var d = new Date();
    return d.getFullYear()+"-"+d.getMonth()+"-"+d.getDate()+"_"+d.getHours()+"-"+d.getMinutes()+"-"+d.getSeconds()+"_"+d.getMilliseconds()+" =>";
  };
  
  

Hope it helps.

Thank you.

How to resolve VERR_NEM_VM_CREATE_FAILED in Windows ?

Hello Everyone,
So I tried to install VirtuaBox VM in Windows 10 and I was getting VERR_NEM_VM_CREATE_FAILED error. After trying many solutions and searching a log in internet, Finally issue got resolved with below command.

1. Open cmd prompt as Administrator.
2. Shut down all programs. You will have to shut down, unplug, and restart your host.
3. Ensure that none of these things are running:

• DeviceGuard
• CredentialGuard
• Windows Defender’s Core Isolation

4. Find the Command Prompt icon, right click it and choose Run As Administrator.
5. Enter this command:

   bcdedit /set hypervisorlaunchtype off  

Then run below command

DISM /Online /Disable-Feature:Microsoft-Hyper-V

5. Enter this command to shutdown your system

shutdown -s -t 2  

6. When the computer turns off, unplug it for 20 seconds. Then plug it in again and boot up Windows 10.

After following these instructions, my issue got resoved and I am able to bootup again.

Credit Goes to : virtualbox.org • View topic - VERR_NEM_VM_CREATE_FAILED: What do I do?

Hope it helps.
Thank you.