Bhoj Karki

Converting Multiple DateTime formate into Single Uniform format using Python and Pandas

2024-08-06T12:00:00.000Z

Problem Statement:

Imagine you are receiving data for an ETL pipeline from multiple sources, each with a different datetime format for when the items are created. Through the ETL process, this data is collected into a single Excel file where you must convert the datetime values to a standard format before inserting them into the data warehouse.

Assume you have datetime values in five different formats:

'2024-02/05',
'02/01/2024',
'02/16-2024 10:01:35',
'02 22 2024'

Your output should be a standard format such as:

2024-02-22 00:00:00
2024-02-16 10:01:35
2024-02-05 00:00:00
2024-02-01 00:00:00

Solution:

We can use python pandas to convert the list of dates into data frame and then use apply function to apply parse date function. The parse date function uses python datetime.strptime() function.

We can use Python Pandas to convert the list of dates into a DataFrame and then use the apply function to apply theparse_datefunction. Theparse_datefunction utilizes Python’sdatetime.strptime() function.

About the datetime.strptime() class method: It parses a string into a datetime object given a corresponding format.

Syntax:

strptime(date_string, format)

Code:

import pandas as pd
from datetime import datetime


# Step 1: given dates from questions as input
given_date =[
        '2024-02/05',
        '02/01/2024',
        '02/16-2024 10:01:35',
        '02 22 2024']

given_date_df = pd.DataFrame({'date': given_date})

# Step 2: Define a function to parse and standardize the date format
def parse_date(date_str:str):
    date_formats = [
        '%Y-%m/%d',
        '%m/%d/%Y',
        '%m/%d-%Y %H:%M:%S',
        '%m %d %Y'
    ]
    for fmt in date_formats:
        try:
            return datetime.strptime(date_str, fmt)
        except ValueError:
            pass
    raise ValueError(f'Invalid format {date_str}')


given_date_df['parsed_date'] = given_date_df['date'].apply(parse_date)

print(given_date_df)

Output:

                  date         parsed_date
0           2024-02/05 2024-02-05 00:00:00
1           02/01/2024 2024-02-01 00:00:00
2  02/16-2024 10:01:35 2024-02-16 10:01:35
3           02 22 2024 2024-02-22 00:00:00

Sorting the given values in latest order

Now if you want to sort the values in descending order to get the most recent ones in the top you can do following:

# Step 4: Sorting the dataframe by the parsed date in descending order
sorted_date_df = given_date_df.sort_values(by='parsed_date', ascending=False).reset_index(drop=True)

# Step 5: Display the sorted dataframe
sorted_date_df[['date', 'parsed_date']]

Output:

date	parsed_date
0	02 22 2024	2024-02-22 00:00:00
1	02/16-2024 10:01:35	2024-02-16 10:01:35
2	2024-02/05	2024-02-05 00:00:00
3	02/01/2024	2024-02-01 00:00:00

Optimizing SQL Query

2024-02-16T12:00:00.000Z

We have used:

PostgreSQL@14 and pgAdmin4

Assumption of audience

It is assumed that you(the audience) have a moderate understanding of SQL language.

Introduction and Background

Understanding how to craft optimized SQL queries is essential for a proficient query developer. This article delves into various strategies for composing efficient queries, with a focus on their application in PostgreSQL databases. To illustrate these techniques, we will utilize two tables in our demonstration.

Table 1:  department (
	department_id  serial primary key,
	department_name varchar(255) not null 
);

Table 2: employee_detail (
	emp_id  serial PRIMARY KEY, 
	emp_name VARCHAR(255) NOT NULL, 
	dob  DATE not NULL,
	department_id int, 
	foreign key(department_id) references department(department_id) on delete set NULL
);

The department table has 5 rows and the employee_detail has 1000101(one million one hundred and one) records. I added large records to see the effect on execution time. For demo purposes, we will be using pgAdmin 4 which also gives execution time.

Tip 1: Writing Sargable Query

Sargable: A Sargable query short for “Search ARGument ABLE” is a query that can utilize the indexes in the database to optimize the query process. The Sargable query utilizes the argument that can take advantage of indexes(if it exists) to make the query faster.

Non-Sargable: A non-sargable query can not utilize the indexes even if it is present. The most common way to make a query non-sargable is to put the column inside a function in the where clause.

Let’s look at some of the examples:

Example1: Original query (Non-Sargable):

select * from employee_detail where extract('Year' from dob)=2000;

Performance:Total query runtime: 861 msec and 970098 rows affected.

Improved query (Sargable):

SELECT * FROM employee_detail where dob >= '01-01-2000' and dob < '01-01-2001';

Performance: Total query runtime: 541 msec and 970098 rows affected.

Example2: Original query (Non-Sargable):

select * from employee_detail where DATE_PART('year', current_date) - DATE_PART('year', dob) >= 30;

Performance: Total query runtime: 206 msec. 10001 rows affected.

Improved query (Sargable):

select * from employee_detail  where dob < current_date - INTERVAL '30 year';

Performance: Total query runtime: 97 msec. 10001 rows affected.

Example3: Original query (Non-Sargable):

select * from employee_detail where coalesce(emp_name, 'Name22')='Name22';

Performance: Total query runtime: 93 msec. 10001 rows affected.

Improved query (Sargable):

select * from employee_detail where emp_name='Name22' or emp_name is null;

Performance: Total query runtime: 63 msec. 10001 rows affected.

Tip2: Using Indexes to improve performance.

The purpose of the indexes is to increase the query performance. Let’s look at a query before and after an index has been added.

Original Query:

select emp_name, dob from employee_detail where emp_name like 'Name%';

Before index performance:

Performance: Total query runtime: 354 msec. 1000101 rows affected.

After index performance:

Performance: Total query runtime: 337 msec.1000101 rows affected. Here due to indexing we can see that the performance has improved.

Tip 3: Use explicit column name in place of *

If you are extracting all the columns then use * in the select statement, however, if you want to pull only certain column use column names.

Pulling all the columns (* is better):

Select * from employee_detail;

Performance: Total query runtime: 513 msec. 1000101 rows affected.

select emp_id, emp_name, dob, department_id from employee_detail;

Performance: Total query runtime: 571 msec. 1000101 rows affected.

Here emp_id, emp_name, dob, and department_id are all the columns and we can see the * operation in this case performs better.

Pulling only some columns (column names is better)

select emp_name, dob from employee_detail;

Performance: Total query runtime: 364 msec. 1000101 rows affected.

Here if we pull only emp_name and dob column we can see that the query time is much less compared to * operation

Tip 4: Use WHERE statement in place of HAVING:

In group by statement we are to use the HAVING clause for filtering, however, we can also use where before group by to first filter the data and then use group by for grouping.

Having said that, there are some conditions where HAVING is absolutely necessary, in those cases only using HAVING. For instance, the filter is by aggregating certain column values.

Original Query:

select count(emp_id) as emp_count, emp_name, dob from employee_detail 
group by dob, emp_name
having emp_name like 'Name%';

Performance: Total query runtime: 169 msec.101 rows affected.

Improved Query:

select count(emp_id) as emp_count, emp_name, dob from employee_detail 
where emp_name like 'Name%'
group by dob, emp_name;

Performance: Total query runtime: 140 msec. 101 rows affected.

Tip 5: Using join properly

Tip 5.1: Joining Big table to smaller one

While joining tables, putting the bigger table before the join statement makes the performance faster. Here, employee_detail is a larger table (one million + records) than department table(5 records).

Joining in pattern - big table - small table

select emp_name, dob, department_name from employee_detail 
inner join department 
on employee_detail.department_id = department.department_id

Performance: This below query took 478 msec; 1000101 rows affected.

Joining in pattern - small table - big table

select emp_name, dob, department_name from department
inner join employee_detail  
on employee_detail.department_id = department.department_id

Performance: Total query runtime: 494 msec.1000101 rows affected.

Both the above query do the same thing and only the order here is changed. If we know which one is a big table and which one is smaller, putting a big table in front of the join statement can improve performance.

Tip 5.2 Avoid using multiple OR statements in a join statement

If you have two or more OR statements in the ON operator, using a single operator for joining and making multiple queries which are joined by union all would provide better performance.

Example:

select column1, column2, columnN table1 t1
inner join table2 t2
on t1.column1=t2.column1 OR t1.column2 = t2.column2

This can be broken down into

select  column1, column2, columnN table1 t1 
inner join table2 t2
on t1.column1=t2.column1
UNION ALL
select column1, column2, columnN table1 t1 
inner join table2 t2
on t1.column2 = t2.column2

Tip 6: Replacing multiple OR statement with IN

If you are using multiple OR statements to filter out data, comparing those values using IN statements is much more efficient.

Original Query:

select emp_id from employee_detail where emp_name = 'Name11' or emp_name = 'Name22' or emp_name='Name33';

Performance: Total query runtime: 94 msec. 30003 rows affected.

Improved Query:

select emp_id from employee_detail where emp_name in ('Name11', 'Name22','Name33');

Performance: Total query runtime: 91 msec. 30003 rows affected.

Tip 7: Using UNION ALL Over UNION

The union all and union both are used to combine results from two or more queries. The UNION ALL combines data as it is, without filtering for distinct elements. The UNION combines data and also filters for unique values so that the combined result will have only distinct items.

Both of these are doing separate tasks one is just combining and another is combining as well as filtering for distinct items. We can not compare them if they are providing different outputs. However, if in case we know that 2 result sets are unique to each other then using UNION ALL in place of UNION will improve query performance.

Let’s check by combining result from same table:

select emp_name, dob from employee_detail
UNION ALL
select emp_name, dob from employee_detail

Performance: Total query runtime: 649 msec.2000202 rows affected.

select emp_name, dob from employee_detail
UNION
select emp_name, dob from employee_detail

Performance: Total query runtime: 1 secs 24 msec.101 rows affected.

Here we can see that the outputs are completely different numbers of rows, however, the key point is since the UNION is also performing filtering operations it takes more time.

Tip 8: Using LIMIT to extract data

Most of the time, we see data in parts, not all at once. It’s like flipping through pages. Only when there’s not too much data does it show up all together. That’s when using LIMIT comes in handy, saving us time by grabbing just the right amount we need.

Original Query:

select emp_name, dob from employee_detail;

Performance: Total query runtime: 364 msec. 1000101 rows affected.

Improved Query:

select emp_name, dob from employee_detail LIMIT 50;

Performance: Total query runtime: 65 msec. 50 rows affected.

Conclusion

In this article, we looked at some of the ways to make SQL query better. These are some techniques, there are still more techniques based on optimizing subqueries, improving through partition and sharding, and utilizing stored procedures for optimizing SQL queries which we will examine in future articles.

Implementing github hook to run jenkins job on a local machine

2020-08-31T12:00:00.000Z

Background

In this article you will see how you can implement github hook and run your jenkin’s job running on you local computer. As you already know in order for github hook to trigger you need a valid internet ip address or domain name. You can not use localhost:8080( default port for jenkin’s job) on github hoook.

What is jenkins and github hook?

In the world of continuous integration and continuous delivery(CI/CD) git and jenkins are important tools. Jenkins is a tool to automate jobs and run or build your program based on various configuration such as run it periodically on fix time, or run it using github hook or be it triggered using a callback url from remote machine, etc. This makes jenkins suitable for building your program and testing it. Git helps to integrate our code and thus helps various developer to collaborate on a single project. This fulfills the CI part of CI/CD portion. Jenkins helps in pulling your code from version control system and build them using Jenkin’s jobs.

How to set up jenkins?

Setting up jenkins is simple you need to have java installed, and you can download jenkins from jenkins website(Downlaod Jenkins ). Jenkins supports windows, Linux, and Mac, you can download for your operating system. Once you download and install it it automatically runs in port 8080 i.e localhost:8080. After installation you need to get the secrete key from mention location when you type localhost:8080 and enter it in the box after that you need to create a user and login via that user.

Creating a job to be triggered:

To create a job follow these steps:

login to the jenkins web interface using localhost:8080/
create jenkins job using create new item
select free style porject for creting job and save it.

add a description and save if for now. You will have to come back later to change these configuration.

After this steps let’s setup github to support hook trigger. Follow these steps to create github.

create a github repository (you will need to have github account)
Let’s add a index.html file in it.
make a clone of it in your local machine and link it to remote repository.

For now this is all we have to do. Let’s come to the important stuff. How are we going to link github to our localhost:8080 as our jenkins is hosted in local machine. Welcome to the world of ngrok. The ngrok is an awesome tool it let’s you expose you local machine port to the internet through it’s server.

Setup ngrok in local machine.

Follow these steps to setup

download ngrok for your machine. Download ngrok
Create an ngrok account you need an api to identify your machine and run ngrok properly.
Login to ngrok and get the secrete key
Go to the directory where ngrok is download and unzip it and then run these commands(if you are in linux you have to unzip ngrok)
1. unzip /path/to/ngrok.zip (For unzipping ngrok, after that go inside the unzip foloder)
2. ./ngrok authtoken < your secrete key here >
3. ./ngrok help
4. ./ngrok http 8080

After this you will get an ngrok url link which maps to your localhost:8080 port. Some thing like this https://a5131490f792.ngrok.io -> http://localhost:8080
This links updates everytime you run it. If you run another time you will get a different link like this
You can now use this ngrok link( https://a5131490f792.ngrok.io ) and add it in github for web hooks.

Finishing github part.

You have added a repo and pushed to the github now got to settings and click on webhooks section. Once you are there you click on add web hook and link the ngrook in you github webhooks like this

After that click on webhooks and it will be saved. By default github will tirggers this on every push request to the github. Now back to jenkins, let’s configure githubhooks

Finishing jenkins part

Follow these steps

go to your job previously created
go to configuration
Go to Source code manage management and link your github repot to jenkins. Let’s see my java program which is a public repo being attacted to jenkins
Go to Build triggers and check GitHub hook trigger for GITScm polling
When ever there is push to the repo this hook will be triggered and your reposetory code will be downloaded.

That’s it ! Now everytime you make push to your github repo this jenkins job will be automatically triggered and pull code from github. You can add scripts to build you project and many more configuration.

Conclusion

In conclusion, in this article you saw how to link your jenkins running on local machine to be triggered using github hook.

Design pattern in python- part4 Behavioral Pattern

2020-08-22T12:00:00.000Z

Behavioral Pattern

Behavioral desing pattern is concerned communication between objects. Lets look at some of the examples of Behavioral pattern in this article.

Chain of responsibility desgin pattern:

The chain of responsibility pattern is used to achieve loose coupling in software where a specified request from the client is passed through a chain of objects included in it. It helps in building a chain of objects. The request enters from one end and moves from one object to another.Let’s look at an example below:

    class ReportFormat(object):
        PDF = 0
        TEXT = 1


    class Report(object):
        def __init__(self, format_):
            self.title = 'Monthly report'
            self.text = ['Things are going', 'really, really well.']
            self.format_ = format_


    class Handler(object):
        def __init__(self):
            self.nextHandler = None

        def handle(self, request):
            self.nextHandler.handle(request)


    class PDFHandler(Handler):

        def handle(self, request):
            if request.format_ == ReportFormat.PDF:
                self.output_report(request.title, request.text)
            else:
                super(PDFHandler, self).handle(request)

        def output_report(self, title, text):
            print('<html>')
            print(' <head>')
            print(' <title>%s</title>' % title)
            print(' </head>')
            print(' <body>')
            for line in text:
                print(' <p>%s</p>' % line)
            print(' </body>')
            print('</html>')


    class TextHandler(Handler):

        def handle(self, request):
            if request.format_ == ReportFormat.TEXT:
                self.output_report(request.title, request.text)
            else:
                super(TextHandler, self).handle(request)

        def output_report(self, title, text):
            print(5 * '*' + title + 5 * '*')
            for line in text:
                print(line)


    class ErrorHandler(Handler):
        def handle(self, request):
            print("Invalid request")


    if __name__ == '__main__':
        report = Report(ReportFormat.TEXT)
        pdf_handler = PDFHandler()
        text_handler = TextHandler()

        pdf_handler.nextHandler = text_handler
        text_handler.nextHandler = ErrorHandler()
        pdf_handler.handle(report)

Output:

    *****Monthly report*****
    Things are going
    really, really well.

Observer desgin pattern:

In this pattern, objects are represented as observers that wait for an event to trigger.Let’s look at an example below:

    import threading
    import time


    class Downloader(threading.Thread):

        def run(self):
            print("downloading...")
            for i in range(1, 5):
                self.i = i
                time.sleep(2)
                print("fun fun")
            return "Hellow world"


    class Worker(threading.Thread):
        def run(self):
            # print("worker")
            for i in range(1, 5):
                print("working running ... {}{} \n".format(i, t.i))
                time.sleep(1)
                t.join()

            print("done")


    t = Downloader()
    t.start()

    print("----------sleep---------")
    time.sleep(1)

    t1 = Worker()
    t1.start()

    t2 = Worker()
    t2.start()

    t3 = Worker()
    t3.start()

Output:

    ----------sleep---------
    downloading...
    working running ... 11 

    working running ... 11 

    working running ... 11 

    fun fun
    fun fun
    fun fun
    fun fun
    working running ... 24 
    working running ... 24 


    working running ... 24 

    working running ... 34 
    working running ... 34 


    working running ... 34 

    working running ... 44 

    working running ... 44 

    working running ... 44 

    done
    done
    done

Conclusion

In conclusion, it is always a good idea to know about software design patter because it let’s us find solution to common problem in efficient way.

There are some criticism of design pattens as well. As an avid reader we should be aware of both the positive and negative aspect of design pattern. As quoted in Wikepedia one criticism is layed out as " A primary criticism of Design Patterns is that its patterns are simply workarounds for missing features in C++, replacing elegant abstract features with lengthy concrete patterns, essentially becoming a “human compiler” or “generating by hand the expansions of some macro”" There are other criticism as well but as a developer it’s more fruitful to know about these design pattern concepts as they are commonly used in day to day life.

You can also search for more patter implementation in the web as this article has only some major ones.

Design pattern in python- part3 Structural Pattern

2020-08-22T12:00:00.000Z

Structural Pattern

Structural desing pattern is concerned with class and object composition. They use inheritance to compose interfaces and declare ways to compose object to get the desired new functionality. Lets look at some of the examples of Structural pattern in this article.

Adaptor desgin pattern:

This pattern allows incompatible interfaces to work together by wrapping the interfaces around an already existing class. This can be verified via following example:

    class EuropeanSocketInterface:
        def voltage(self): pass

        def live(self): pass

        def neutral(self): pass

        def earth(self): pass


    # adaptee

    class Socket(EuropeanSocketInterface):
        def voltage(self):
            return 230

        def live(self):
            return 1

        def neutral(self):
            return -1

        def earth(self):
            return 0

    class USASocketInterface:
        def voltage(self): pass

        def live(self): pass

        def neutral(self): pass

    # this is the adaptor pattern

    class Adaptor(USASocketInterface):
        def __init__(self, interface):
            self.__interface = interface

        def voltage(self):
            return 110

        def live(self):
            return self.__interface.live()

        def neutral(self):
            return self.__interface.neutral()

    # client
    class ClientElectricKettle:
        def __init__(self, obj):
            self.__power = obj

        def boil(self):
            if self.__power.voltage() > 110:
                print("Too hot")
            elif self.__power.live() == 1 and self.__power.neutral() == -1:
                print("Coffee time")
            else:
                print("No power")


    if __name__ == "__main__":

        socket = Socket()
        adaptor = Adaptor(socket)

        client = ClientElectricKettle(adaptor)
        client.boil()

        print("end")

Output:

    Coffee time
    end

Facade desgin pattern:

This pattern provides simplified interfaces to a large body of interfaces.Let’s look at an example below:

    class _IgnitionSystem(object):

        @staticmethod
        def produce_spark():
            return True


    class _Engine(object):

        def __init__(self):
            self.revs_per_minute = 0

        def turnon(self):
            self.revs_per_minute = 2000

        def turnoff(self):
            self.revs_per_minute = 0


    class _FuelTank(object):

        def __init__(self, level=30):
            self._level = level

        @property
        def level(self):
            return self._level

        @level.setter
        def level(self, level):
            self._level = level


    class _DashBoardLight(object):

        def __init__(self, is_on=False):
            self._is_on = is_on

        def __str__(self):
            return self.__class__.__name__

        @property
        def is_on(self):
            return self._is_on

        @is_on.setter
        def is_on(self, status):
            self._is_on = status

        def status_check(self):
            if self._is_on:
                print("{}: ON".format(str(self)))
            else:
                print("{}: OFF".format(str(self)))


    class _HandBrakeLight(_DashBoardLight):
        pass


    class _FogLampLight(_DashBoardLight):
        pass


    class _Dashboard(object):

        def __init__(self):
            self.lights = {"handbreak": _HandBrakeLight(), "fog": _FogLampLight()}

        def show(self):
            for light in self.lights.values():
                light.status_check()


    # Facade
    class Car(object):

        def __init__(self):
            self.ignition_system = _IgnitionSystem()
            self.engine = _Engine()
            self.fuel_tank = _FuelTank()
            self.dashboard = _Dashboard()

        @property
        def km_per_litre(self):
            return 17.0

        def consume_fuel(self, km):
            litres = min(self.fuel_tank.level, km / self.km_per_litre)
            self.fuel_tank.level -= litres

        def start(self):
            print("\nStarting...")
            self.dashboard.show()
            if self.ignition_system.produce_spark():
                self.engine.turnon()
            else:
                print("Can't start. Faulty ignition system")

        def has_enough_fuel(self, km, km_per_litre):
            litres_needed = km / km_per_litre
            if self.fuel_tank.level > litres_needed:
                return True
            else:
                return False

        def drive(self, km=100):
            print("\n")
            if self.engine.revs_per_minute > 0:
                while self.has_enough_fuel(km, self.km_per_litre):
                    self.consume_fuel(km)
                    print("Drove {}km".format(km))
                    print("{:.2f}l of fuel still left".format(self.fuel_tank.level))
            else:
                print("Can't drive. The Engine is turned off!")

        def park(self):
            print("\nParking...")
            self.dashboard.lights["handbreak"].is_on = True
            self.dashboard.show()
            self.engine.turnoff()

        def switch_fog_lights(self, status):
            print("\nSwitching {} fog lights...".format(status))
            boolean = True if status == "ON" else False
            self.dashboard.lights["fog"].is_on = boolean
            self.dashboard.show()

        def fill_up_tank(self):
            print("\nFuel tank filled up!")
            self.fuel_tank.level = 100


    # the main function is the Client
    def main():
        car = Car()
        car.start()
        car.drive()
        car.switch_fog_lights("ON")
        car.switch_fog_lights("OFF")
        car.park()

        car.fill_up_tank()
        car.drive()
        car.start()
        car.drive()

    if __name__ == "__main__":
        main()

Output:

    Starting...
    _HandBrakeLight: OFF
    _FogLampLight: OFF


    Drove 100km
    24.12l of fuel still left
    Drove 100km
    18.24l of fuel still left
    Drove 100km
    12.35l of fuel still left
    Drove 100km
    6.47l of fuel still left
    Drove 100km
    0.59l of fuel still left

    Switching ON fog lights...
    _HandBrakeLight: OFF
    _FogLampLight: ON

    Switching OFF fog lights...
    _HandBrakeLight: OFF
    _FogLampLight: OFF

    Parking...
    _HandBrakeLight: ON
    _FogLampLight: OFF

    Fuel tank filled up!


    Can't drive. The Engine is turned off!

    Starting...
    _HandBrakeLight: ON
    _FogLampLight: OFF


    Drove 100km
    94.12l of fuel still left
    Drove 100km
    88.24l of fuel still left
    Drove 100km
    82.35l of fuel still left
    Drove 100km
    76.47l of fuel still left
    Drove 100km
    70.59l of fuel still left
    Drove 100km
    64.71l of fuel still left
    Drove 100km
    58.82l of fuel still left
    Drove 100km
    52.94l of fuel still left
    Drove 100km
    47.06l of fuel still left
    Drove 100km
    41.18l of fuel still left
    Drove 100km
    35.29l of fuel still left
    Drove 100km
    29.41l of fuel still left
    Drove 100km
    23.53l of fuel still left
    Drove 100km
    17.65l of fuel still left
    Drove 100km
    11.76l of fuel still left
    Drove 100km
    5.88l of fuel still left
    Drove 100km
    0.00l of fuel still left

Decorator desgin pattern:

This pattern dynamically adds/overrides existing method to add new functionality without changing it’s calling style. Let’s look at an example below. In this example we have two files

decorator.py
implement.py

The decorator.py file has the decorator design pattern code and the implement.py file uses decorator.py method to get Cost, Ingredient, Sales tax for various items.

Code in decorator.py

    import six
    from abc import ABCMeta


    @six.add_metaclass(ABCMeta)
    class AbstractCoffee:
        def get_ingridents(self):
            pass

        def get_price(self):
            pass

        def get_taxes(self):
            return 0.1 * self.get_price()


    class ConcreteCoffee(AbstractCoffee):
        def get_ingridents(self):
            return 'coffee'

        def get_price(self):
            return 100


    @six.add_metaclass(ABCMeta)
    class AbstractCoffeeDecorator(AbstractCoffee):
        def __init__(self, decorated_coffee):
            self.decorated_coffee = decorated_coffee

        def get_ingridents(self):
            return self.decorated_coffee.get_ingridents()

        def get_price(self):
            return self.decorated_coffee.get_price()


    class Suger(AbstractCoffeeDecorator):
        def get_ingridents(self):
            return self.decorated_coffee.get_ingridents() + " ,suger"

        def get_price(self):
            return self.decorated_coffee.get_price() + 25


    class Milk(AbstractCoffeeDecorator):
        def get_ingridents(self):
            return self.decorated_coffee.get_ingridents() + " ,milk"

        def get_price(self):
            return self.decorated_coffee.get_price() + 75


    class Vanilla(AbstractCoffeeDecorator):
        def get_ingridents(self):
            return self.decorated_coffee.get_ingridents() + " ,vanilla"

        def get_price(self):
            return self.decorated_coffee.get_price() + 200

Code in implement.py

    import decorator

    my_coffee = decorator.ConcreteCoffee()
    print(
        'Ingredients :', my_coffee.get_ingridents(),
        ', Cost :', my_coffee.get_price(),
        ', Sales Tax :',my_coffee.get_taxes()
    )

    my_coffee = decorator.Milk(my_coffee)
    print(
        'Ingredients :', my_coffee.get_ingridents(),
        ', Cost :', my_coffee.get_price(),
        ', Sales Tax :',my_coffee.get_taxes()
    )
    my_coffee = decorator.Vanilla(my_coffee)
    print(
        'Ingredients :', my_coffee.get_ingridents(),
        ', Cost :', my_coffee.get_price(),
        ', Sales Tax :',my_coffee.get_taxes()
    )
    my_coffee = decorator.Suger(my_coffee)
    print(
        'Ingredients :', my_coffee.get_ingridents(),
        ', Cost :', my_coffee.get_price(),
        ', Sales Tax :',my_coffee.get_taxes()
    )

Now, let’s run implement.py and see the output Output:

    Ingredients : coffee , Cost : 100 , Sales Tax : 10.0
    Ingredients : coffee ,milk , Cost : 175 , Sales Tax : 17.5
    Ingredients : coffee ,milk ,vanilla , Cost : 375 , Sales Tax : 37.5
    Ingredients : coffee ,milk ,vanilla ,suger , Cost : 400 , Sales Tax : 40.0

Proxy desgin pattern:

This pattern provides a placeholder for another object to control access, reduce cost, and reduce complexity. Let’s look at an example below.

    class Image:
        def __init__(self, filename):
            self._filename = filename

        def load_image(self):
            print('Loading...', self._filename)

        def display_image(self):
            print('display...', self._filename)


    class Proxy:
        def __init__(self, subject):
            self._subject = subject
            self._proxystate = None


    class ImageProxy(Proxy):

        def display_image(self):
            if self._proxystate is None:
                self._subject.load_image()
                self._proxystate = 1
            print("Displaying ...",self._subject._filename)


    img1 = ImageProxy(Image("IMAGE_SUB_1"))
    img2 = ImageProxy(Image("IMAGE_SUB_2"))

    img1.display_image()
    img1.display_image()

    img2.display_image()
    img2.display_image()

    img1.display_image()
    img1.display_image()

Output:

    Loading... IMAGE_SUB_1
    Displaying ... IMAGE_SUB_1
    Displaying ... IMAGE_SUB_1
    Loading... IMAGE_SUB_2
    Displaying ... IMAGE_SUB_2
    Displaying ... IMAGE_SUB_2
    Displaying ... IMAGE_SUB_1
    Displaying ... IMAGE_SUB_1

In the next lession we will look at some behaviour design patterns.

Design pattern in python- part2 Creational Pattern

2020-08-20T12:00:00.000Z

Creational Pattern

Lets look at some of the examples of crational pattern in this article.

Singleton desgin pattern:

This pattern restricts the creation of instance to one instance. This can be verified via following example:


    class Singleton:
        __instance = None

        @staticmethod
        def get_instance():
            if Singleton.__instance is None:
                Singleton()
            return Singleton.__instance

        def __init__(self):
            # print(self)
            # print("type = ", type(self))
            if Singleton.__instance is not None:
                raise Exception("Singleton class !Can't create multiple objects!")

            else:
                Singleton.__instance = self


    s = Singleton()
    print(s)

    s = Singleton.get_instance()
    print(s)

    s = Singleton.get_instance()
    print(s)

    s = Singleton.get_instance()
    print(s)

Output:

<__main__.Singleton object at 0x7f78d1920c50>
<__main__.Singleton object at 0x7f78d1920c50>
<__main__.Singleton object at 0x7f78d1920c50>
<__main__.Singleton object at 0x7f78d1920c50>

Builder design pattern:

Builder design pattern helps to create complex from simple object by separating construction and representation. Following example illustrates builder design pattern using python:


    class Builder:
        """
        This is like an abstract class
        """

        def get_wheel(self): pass

        def get_body(self): pass

        def get_engine(self): pass


    class JeepBuilder(Builder):

        def get_wheel(self):
            wheel = Wheel()
            wheel.size = 34
            return wheel

        def get_engine(self):
            eng = Engine()
            eng.horsepower = 770
            return eng

        def get_body(self):
            body = Body()
            body.shape = "Range Rover"
            return body


    class Director:
        __builder = None

        def __init__(self, b):
            self.__builder = b

        def get_car(self):
            car = Car()

            car.set_body(self.__builder.get_body())
            # car.set_wheel(self.__builder.get_wheel())
            car.set_engine(self.__builder.get_engine())

            # set wheel
            i = 0
            while i < 4:
                wheel = self.__builder.get_wheel()
                car.set_wheel(wheel)
                i += 1

            return car


    # This one is our main product
    class Car:
        _body = None
        _engine = None
        _wheels = list()

        def set_body(self, body):
            self._body = body

        def set_engine(self, eng):
            self._engine = eng

        def set_wheel(self, wh):
            self._wheels.append(wh)

        def get_specification(self):
            print(f"body shape = {self._body.shape}")
            print(f"engine horsepower= {self._engine.horsepower}")
            print(f"wheel size = {self._wheels[0].size}")


    class Wheel:
        size = None


    class Body:
        shape = None  # e.g SUV


    class Engine:
        horsepower = None


    if __name__ == "__main__":
        print("jeep")
        builder = JeepBuilder()
        director = Director(builder)

        jeep = director.get_car()  # return an instance
        jeep.get_specification()
        print("thanks")

Output:

    jeep
    body shape = Range Rover
    engine horsepower= 770
    wheel size = 34
    thanks

Factory design patter:

This pattern helps to create object without giving underlying details to client and communicating to those created object via common interfact. Following example illustrates factory design pattern:

    class Button(object):
        html = "<button></button>"

        def get_html(self):
            return self.html


    class Image(Button):
        html = "<img></img>"


    class Input(Button):
        html = "<input></input>"


    class Card(Button):
        html = "<obj></obj>"


    class ButtonFactory:

        def create_button(self, typ):
            # Note we need to capitalize() because we are passing values in lower case
            return globals()[typ.capitalize()]()


    btn = ButtonFactory()
    input_list = ['image', 'input', 'card']
    for b in input_list:
        t = btn.create_button(b).get_html()
        print(t)

Output:

    <img></img>
    <input></input>
    <obj></obj>

Prototype design pattern:

Prototype design pattern helps to hide the complexity of the instances created by the class via cloning an object. Following example show prototype desing pattern:


    import copy


    class Prototype:
        _type = None
        _value = None

        def get_type(self):
            return self._type

        def get_value(self):
            return self._value

        def clone(self):
            pass


    class Type1(Prototype):
        def __init__(self, value):
            self._type = "type1"
            self._value = value

        def clone(self):
            return copy.copy(self)


    class Type2(Prototype):
        def __init__(self, value):
            self._type = "type2"
            self._value = value

        def clone(self):
            return copy.copy(self)


    class ObjectFactory:
        __type1value1 = None
        __type1value2 = None
        __type2value1 = None
        __type2value2 = None

        @staticmethod
        def initialize():
            ObjectFactory.__type1value1 = Type1(1)
            ObjectFactory.__type1value2 = Type1(2)
            ObjectFactory.__type2value1 = Type2(1)
            ObjectFactory.__type2value2 = Type2(2)

        @staticmethod
        def get_type1_value1():
            return ObjectFactory.__type1value1.clone()

        @staticmethod
        def get_type1_value2():
            return ObjectFactory.__type1value2.clone()

        @staticmethod
        def get_type2_value1():
            return ObjectFactory.__type2value1.clone()

        @staticmethod
        def get_type2_value2():
            return ObjectFactory.__type2value2.clone()


    def main():
        print("starting main")

        # initialize and call method
        ObjectFactory.initialize()

        obj = ObjectFactory.get_type1_value1()
        print(f"type= {obj.get_type()}, value = {obj.get_value()}")

        obj = ObjectFactory.get_type1_value2()
        print(f"type= {obj.get_type()}, value = {obj.get_value()}")

        obj = ObjectFactory.get_type2_value1()
        print(f"type= {obj.get_type()}, value = {obj.get_value()}")

        obj = ObjectFactory.get_type2_value2()
        print(f"type= {obj.get_type()}, value = {obj.get_value()}")

        print("thanks")


    if __name__ == "__main__":
        main()

starting main
type= type1, value = 1
type= type1, value = 2
type= type2, value = 1
type= type2, value = 2
thanks

In the next lession we will look at some structural design pattern.

Design pattern in python- part1 Introduction

2020-08-19T12:00:00.000Z

We have used:

Python 3.6+

Assumption of audience

It is assumed that you(the audience) have a moderate understanding of programming language. Experience in python will be added advantage.

Design Patterns

Design patterns are everywhere. When we talk about design pattern the history starts with 4 people, name as gang of four, starts to write software development concepts that are constantly reused in software development. Design pattern are solution to general software engineering problem. They help to solve problem and make product better. They are not finished products in themself but rather a concepts to solve certain software development problems. This artilce present the design pattern in series of parts and tries to implement them using python programming language. The design pattern can be divided into 3 category and further sub category as shown below:

Creational design pattern
Structural design pattern
Behaviour design pattern

Creational design pattern: It is the design pattern that deals with the creation of an object. This category has following patterns:

Design pattern name	Description
Factory method	It creates and object without specifying exact class to create
Abstract Factory	It groups object factory that have common theme
BUilder	It constructs complex objects by separating construction and representation
Singleton	It lets only one instance to be created from a class
Prototype	It creates an object by cloning an existing object

Structural design pattern: It is the design pattern that deals with creation of class structure such as inheritance and composition. It concerns class and object composition.This category has following patterns:

Design pattern name	Description
Adaptor	It helps to interface between two incompatible classes with an adaptor class
Bridge	It decouples an abstraction from its implementation so that the two can vary independently.
Composite	It zero or more similar objects so that they can be manipulated as single object.
Decorator	It adds feature to already existing method without changing it’s calling structure
Facade	It provides simplified interface for larger body of code.
Flyweight	It reduces the cost of creating and manipulating a large number of similar objects.
proxy	It provides a placeholder for another object to control access and reduce complexity

Behaviour design pattern: It is the design pattern that deals with suitable interaction between objects, how to privide lose coupling and how to provide flexibility to extend them in future.This category has following patterns:

Design pattern name	Description
Chain of responsibility	It assigns command to chain of processing objects
Command	It creates objects which encapsulates actions and parameter
Intrepreter	It implements a specialized language
Iterator	It accesses the elements of an object sequentially without exposing its underlying representation.
Mediator	It allows loose copuling between classes by being mediator who know about detail of both classes
Memento	It provides the ability to restore an object to its previous state (undo).
Observer	It implements publisher/subscriber pattern to allow the observer object to see change in events
State	It allows an object to change it’s behavior when internal state changes
Strategy	It allows one of a family of algorithms to be selected on-the-fly at runtime.
Template method	implements an algorithm as an abstract class, allowing its subclasses to provide concrete behavior.
Visitor	It separates an algorithm from an object structure by moving the hierarchy of methods into one object.

Django class based view vs function based view

2019-12-26T12:00:00.000Z

We have used:

Python 3.6+
django 3.0+

Assumption of audience

It is assumed that you(the audience) have a moderate understanding of django and python and you can create models and view.

Description

When programming in django we come across a topic of Class Based View(CBV) and Function Based View(FBV). In this article we will discuss about how to use them and which one to use in what scenerio. The point of this article is to give Django developer a scenerio when to use FBV over CBV and vice versa.

Table of content

Models Creation:
Function Based view
- List view
- Detail View
Class based view
- List view
- Detail view
Comparision of CBV and FBV
Conclusion

Models Creation

To explain this concept lets create a model in django. In this article we will create a simple weather data storage system.

    # models.py
    from django.db import models

    # Create your models here.
    class WeatherData(models.Model):
        location = models.CharField(max_length=30)
        temperature = models.FloatField(blank=True, null=True)
        wind_speed = models.FloatField(blank=True, null=True, help_text="in km/hr")  # in km/hr
        humidity = models.FloatField(default=0)  # form 0-100%
        rain = models.CharField(max_length=2, choices=(
            ('NR', "No Rain"),
            ('LR', 'Less/Few Rain'),
            ('MR', 'Moderate Rain'),
            ('HR', 'Heavy Rain'),
        ))
        created_date = models.DateTimeField(auto_now=True)

        def __str__(self):
            return "{0}_{1}".format(self.location, self.temperature)

Lets add some data in the models List of images added

Function Based View(FBV)

In the begining of django development there was only function based view, later developers started noticing some feature were redundent and tried to extend view using various generic view. However, it was consistently hard to extend function based view. Therefore, later they introduced class based views. This example has 2 function based views 2 for listing data and another for displaying individual item.

FBV List View

We can list item in function based view as shown below. Since this is a list view it is always good practice to paginate data and present them in small chunks with links to move to new data.

Here is our views.py with function FBV_list_veiw

def FBV_list_view(request):
    # you need to order it before pagination else paginator throws warning
    data = WeatherData.objects.all().order_by('-created_date')
    page = request.GET.get('page', 1)

    paginator = Paginator(data, 5)
    try:
        weather_data = paginator.page(page)
    except PageNotAnInteger:
        weather_data = paginator.page(1)
    except EmptyPage:
        weather_data = paginator.page(paginator.num_pages)

    return render(request, 'cbv_vs_fbv_demo/list.html', {
        'weather_data': weather_data
    })

This view can be rendered like this in urls.py

from django.urls import path, include
from .views import FBV_list_view

urlpatterns = [
    path('', FBV_list_view, name="fbv_list_view"),
]

The templates looks like this in list.html


<table>
  <thead>
    <tr>
      <th>Location</th>
      <th>Temperature</th>
      <th>Wind Speed</th>
      <th>Humidity</th>
      <th>Rain</th>
    </tr>
  </thead>
  <tbody>
    {% for data in weather_data %}
      <tr>
        <td>{{ data.location }}</td>
        <td>{{ data.temperature }}</td>
        <td>{{ data.wind_speed }}</td>
        <td>{{ data.humidity }}</td>
        <td>{{ data.rain }}</td>
      </tr>
    {% endfor %}
  </tbody>
</table>

{% if weather_data.has_other_pages %}
  <ul class="pagination">
    {% if weather_data.has_previous %}
      <li><a href="?page={{ weather_data.previous_page_number }}">&laquo;</a></li>
    {% else %}
      <li class="disabled"><span>&laquo;</span></li>
    {% endif %}
    {% for i in weather_data.paginator.page_range %}
      {% if weather_data.number == i %}
        <li class="active"><span>{{ i }} <span class="sr-only">(current)</span></span></li>
      {% else %}
        <li><a href="?page={{ i }}">{{ i }}</a></li>
      {% endif %}
    {% endfor %}
    {% if weather_data.has_next %}
      <li><a href="?page={{ weather_data.next_page_number }}">&raquo;</a></li>
    {% else %}
      <li class="disabled"><span>&raquo;</span></li>
    {% endif %}
  </ul>
{% endif %}

FBV Detail View

For detail view we just pull the data based on id or slug. Here in this tutorial we are going to use it using id. Detail view function looks like this.

def FBV_detail_view(request, id):
    data = WeatherData.objects.get(id=id)

    return render(request, 'cbv_vs_fbv_demo/detail.html', {
        'data': data
    })

We will add path like this in urls.py

#... 
urlpatterns = [
    #...
    path('detail/<int:id>/', FBV_detail_view, name="FBV_detail_view"),
]

Class Based View(CBV)

For class based view we will use generic classes and render the same thing. Class based view makes it much easier to write our code in less amount of time.

CBV List View

Following is the code for doing the same thing as list view did in function based view. In veiws.py we will make our view class.

#... others as it is 
from django.views.generic.list import ListView
from django.views.generic.detail import DetailView

class CBV_list_view(ListView):
    """
    Class based view for listing data
    """
    model = WeatherData
    paginate_by = 5
    context_object_name = "weather_data"
    template_name = "cbv_vs_fbv_demo/list_cbv.html"

We can link this in url using following code snippet

#....others as it is
path('list/', CBV_list_view.as_view(), name="cbv_list_view"),

The main point to notice here is .as_view() method call. In function based view we call functions directly but in class based view we call as_view() method. We can also say urls simply have methods call in them.

CBV Detail View

Detail view has following code in it


class CBV_detail_view(DetailView):
    model = WeatherData
    context_object_name = "data"
    template_name = "cbv_vs_fbv_demo/detail.html"

urls looks like this

#....others as it is
path('cbv_detail/<int:pk>/', CBV_detail_view.as_view(), name="CBV_detail_view"),

Comparision of CBV and FBV

In the begining of the django there was only function based views later they introduced the concept of class based view to simply some repetative task. Lets compare advantages and disadvantages in CBV and FBV

Advantages of CBV:

The code becomes concise and saves time.
We can use already built in generic views, this makes it easy to extend the class
We use separate get and post method to handle various request.

Disadvantages of CBV:

It hides a lot of implementation detail which make it harder to read and maintain.
It difficult for beginners to wrap head around in the first place. Once they understand the flow it becomes easier.
For using decorators we need to import separte method called method_decorator(syntax @method_decorator(…))

Advantages of FBV:

The code is more explicit and make it easy to read and understand.
Easy to use decorators (like login_required and so on)
Easy to understand for beginners

Disadvantages of FBV:

It uses if else statement for various request like GET and POST
It difficult to extend commonly used functionality for instance pagination.

Conclusion

When should we use which.

In my opinion both FBV and CBV completes the job. I would recommend using CBV in most of the cases as it is simple to extend and add functionality. However, if we need to write code that needs explicit explanation(like implementing an complex algorithm) FBV might be right approach. Also task like serving pdf files, or other resources can take FBV.

I hope you like this article please feel free to contact me if you have any question.

Deploying django app on VPS using nginx,gunicorn, supervisor

2019-12-25T12:00:00.000Z

We have used:

python version: python3

machine: linux

django version: django-2.x

Project Structure

Lets assume that you have a django project and the folder structure looks like this:

Figure: Django app folder structure

About structure

mysite/ is our django actual project

config/ folder has config.json file which contains all our configuration file.

mysite/settings/ folder contains main setting of our application. Setting is seprated in base.py, development.py and production.py. If we want to run in development mode we need to pass the settings file. base.py is common settings file and development.py is for development mode. For running app in development we do: python3 manage.py runserver --settings=mysite.settings.development.

Requiremnts/ folder has our requirement files base.txt-- all pacakges in our app development.txt – packages only used in development(e.g django-debug-toolbar) production.txt – package used in production environment but not in development.

mysite/ contains main djago apps.

mysite_apps/ contains all our apps(note this folder is a python package)

Requirements

nginx: Nginx is used as a server to serve static files in our project to the user. gunicorn: This is an used as an application server to serve our application. In development mode we can do python3 manage.py runserver and it will run the app. If we want to run the app using gunicorn we need to configure gunicorn in our application. We will see the configuration later. supervisor: This tool is used to run gunicorn. Setting supervisor will help our application to restart in case of any problem. Supervisor is configured to handle running of gunicorn.

What is the flow of application:

user request —> nginx server—> gunicorn(served by supervisor)—>our django app

Steps:

Install requirements
Setup virtualenvironment for our application
Clone the project from github or other version control system
Install requirements of our application(i.e. packages used by our app)
Setting up postgres database
Run migration and test application by running it
Configure gunicorn
Configure nginx
Configure supervisord (or simply supervisor)
Test everything is running fine

1. Install requirements:

Here we assume that we are using python3 for our application and we are deploying on linux machine. We need to install python3 using pip, virtualenv. Enough talk lets dive into it: Log into the vps machine using the credentials given by your vps provide. Once you are in we go through following steps: First lets update our machine and install python3 and virtualenv:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python3-pip
sudo pip3 install virtualenv

2. Setup virtualenvironment for our application:

Lets create a virutal environment and start the virtualenv


virtualenv /Projects/venv
source /Projects/venv/bin/activate

3. Clone the project from github or other version control system

Now we clone our project from github:


git clone http://github.com/[path to your project]/mysite.git

4. Install requirements of our application(i.e. packages used by our app)

Once the app is downloaded and virtualenv is running lets install the requirements. All our project dependent packages are in base.txt file and we use production.txt file to install from base.txt and also other packages from production.txt. We go to folder where manage.py is and run following command:

pip install -r requirements/production.txt

5. Setting Up postgres database

first go to postgres as:

sudo su - postgres

Then create database and user and set password in user:

postgres=# CREATE DATABASE db_name;
postgres=# CREATE USER db_user;
postgres=# ALTER USER db_user PASSWORD 'myPassword';

Grant permission:

postgres=# GRANT ALL PRIVILEGES ON DATABASE db_name TO db_user;

After granting permission log out from psql using ‘\q’ symbolizing quite. Then change the postgres user to root user as:

postgres@server:~$ su - root

6. Run migration and test application by running it

cd mysite
./manage.py migrate --settings=mysite.settings.production
./manage.py runserver --settings=mysite.settings.production

After testing everything stop the server.

7. Configure gunicorn

Lets install gunicorn:

pip install gunicorn

After installing gunicorn lets run our application using gunicorn for testing

DJANGO_SETTINGS_MODULE=mysite.settings.prduction  gunicorn mysite.wsgi

Note: This is similar to saying ./manage.py runserver --settings=mysite.settings.production in development.

We can also pass workers to gunicorn using workers parameter as:

DJANGO_SETTINGS_MODULE=mysite.settings.prduction  gunicorn --workers 3 mysite.wsgi

Once we verify everything is working fine we move on to configuring nginx.

8. Configure nginx

Lets install nginx:

sudo apt-get install nginx

After installation lets configure nginx file in /etc/nginx/sites-available/ Lets create a file name mysite in /etc/nginx/sites-available/mysite and add following configuation:

server{
	listen 80;
	server_name ipaddress(e.g.x.x.x.x);
	
	location /static {
		alias /var/www/html/Projects_static_root;
		expires 30d;
	}
	
	location /media {
	alias /var/www/html/Projects_media_root;
	expires 30d;
	}

	location / {
		include proxy_params;
		proxy_pass http://127.0.0.1:8000;
	}
	
	}

Here in place ipaddress(e.g.x.x.x.x) use the ipaddress provided by your host provider.

Now, check if our configuration file was correctly written and start our nginx:

sudo nginx -t
sudo service nginx restart

9. Configure supervisord (or simply supervisor)

Installation:

sudo apt-get install supervisor

Now, let’s create mysite.conf in /etc/supervisor/conf.d/ folder:

sudo vim /etc/supervisor/conf.d/mysite.conf

and configure our project:

[program:mysite]
command=/root/Projects/venv/bin/gunicorn --workers 3 --bind 127.0.0.1:8000 mysite.wsgi:application
environment=DJANGO_SETTINGS_MODULE=mysite.settings.production
directory=/root/Projects/mysite
autostart=true
autorestart=true
stderr_logfile=/var/log/mysite.err.log
stdout_logfile=/var/log/mysite.out.log
user=root
group=www-data
programs:mysite

Now lets start our app uing supervisor

service supervisor stop
service supervisor restart

Now our app should be running on the ipaddress configured by you check for further details.