Micro-services

Wed, 12 Oct 2016 11:34:11 -0400

Micro-services design is really helping alleviate some problems we experienced with the monolithic model. Implementing micro-services is perhaps one of the greatest ways to improve the producitivity of a software engineering team.

Specifically,

You deploy only the component that was changed (as opposed to the whole application), this keeps the deployments and the tests managable.
If multiple team members are working on the application, they need to wait until everyone is done with development, testing and ready to move forward. However, with microservices-services model, whenever each piece of functionality is ready it could be deployed.
Each micro-service runs in its own process space, so if we want to scale for any reason, we can scale the particular micro-service that needs more resources instead of scaling the entire application.
When one micro-service needs to communicate with other other micro-service, it does using a lightweight protocol such as http.

This post examines, how to deploy microservices created using python and django on nginx and uwsgi.

Before going into the details of a microservices design, lets understand and examine how a typical monolithic application looks like.

Monolithic Design

However, with breaking the functionality into logically isolated micro-services, this is how the design would like.

Microservices Design

Now, lets examine how the configuration would like for a python and django application that runs on nginx on a typical linux server.

Because the application code is spread across multiple repos (or subrepos) grouped by logically independent code, this will be the typical organization of the application directories on the server.

[ec2-user@ip-172-31-34-107 www]$ pwd
/opt/www
[ec2-user@ip-172-31-34-107 www]$ ls -lrt
total 8
drwxr-xr-x. 5 root root 4096 Oct 12 14:09 microservice1
drwxr-xr-x. 7 root root 4096 Oct 12 19:00 microservice2
drwxr-xr-x. 5 root root 4096 Oct 12 14:09 microservice3
drwxr-xr-x. 7 root root 4096 Oct 12 19:00 microservice4

Nginx, which is typically deployed a front-end gateway or a reverse proxy will have the this configuration.

[ec2-user@ip-172-31-34-107 serviceA]$ cat /etc/nginx/conf.d/service.conf 
upstream django1 {
    server unix:///opt/www/service1/uwsgi.sock; # for a file socket
}

upstream django2 {
    server unix:///opt/www/service2/uwsgi.sock; # for a file socket
}

upstream django3 {
    server unix:///opt/www/service3/uwsgi.sock; # for a file socket
}

upstream django4 {
    server unix:///opt/www/service4/uwsgi.sock; # for a file socket
}

server {
    # the port your site will be served on
    listen      80;
    # the domain name it will serve for
    server_name localhost;
    charset     utf-8;

    # max upload size
    client_max_body_size 75M;   # adjust to taste

    location /api/service1/ {
	    uwsgi_pass  django1; 
	    include     /etc/nginx/uwsgi_params; 
    }
 
   location /api/service2/ {
            uwsgi_pass  django2;
            include     /etc/nginx/uwsgi_params; 
    }

    location /api/service3/ {
	    uwsgi_pass  django3; 
	    include     /etc/nginx/uwsgi_params; 
    }
 
   location /api/service4/ {
            uwsgi_pass  django4;
            include     /etc/nginx/uwsgi_params; 
}

Multiple uWSGI process have to be created that would process the request for each microservice.

/usr/bin/uwsgi --socket=/opt/www/service1/uwsgi.sock --module=microservice-test.wsgi --master=true --chdir=/opt/www/service1

/usr/bin/uwsgi --socket=/opt/www/service2/uwsgi.sock --module=microservice-test.wsgi --master=true --chdir=/opt/www/service2

/usr/bin/uwsgi --socket=/opt/www/service3/uwsgi.sock --module=microservice-test.wsgi --master=true --chdir=/opt/www/service3

/usr/bin/uwsgi --socket=/opt/www/service4/uwsgi.sock --module=microservice-test.wsgi --master=true --chdir=/opt/www/service4

High Availability for MongoDB

Sat, 01 Aug 2015 11:34:11 -0400

MongoDB is a no-sql database that is very easy to setup and administer.

High availability is achieved using replica sets that provide automatic fail-over. It is important to note that fail-over is automatic, in that it does not require manual interaction.

In this post we examine how replica sets are configured and how automatic failover is achieved. This assumes that mongodb is already installed.

In order to understand replica sets it is important to understand the mongod process. mongod is the process that listens for incoming requests on a certain port and process those requests. If there is only one mongod process involved, then it is a single point of failure, however to incorporate failover using replica sets we create multiple mongod process and designate them as primary, secondary and arbiter.

Primary -> primary member handles all write operations. It is the current master instance.
Secondary -> it is a replica set member that replicates the contents of the master database.
Arbiter -> the purpose of arbiter is to solely vote in elections. Arbiter does not replicate data.

Create 3 mongod instances, once each for primary, secondary and arbiter

cd /data
mkdir db1
mkdir db2
mkdir db3
mongod --dbpath ./db1 --port 30000 --replSet "demo" --fork --logpath /var/log/mongodb.log
mongod --dbpath ./db2 --port 40000 --replSet "demo" --fork --logpath /var/log/mongodb.log
mongod --dbpath ./db3 --port 50000 --replSet "demo" --fork --logpath /var/log/mongodb.log

At this points, 3 instances are created. It could be verified.

ps -ef|grep -i 'mongod'

root      4760     1  0 04:13 ?    00:00:00 mongod --dbpath ./db1 --port 30000 --replSet demo --fork --logpath /var/log/mongodb.log
root      4892     1  0 04:14 ?    00:00:00 mongod --dbpath ./db3 --port 50000 --replSet demo --fork --logpath /var/log/mongodb.log
root     20490     1  0 01:56 ?    00:01:01 mongod --dbpath ./db2 --port 40000 --replSet demo --fork --logpath /var/log/mongodb.log

Now, lets define which instance is primary, secondary and arbiter.

Connect to one instance.

mongo --port 30000
db.getMongo()

At this point, we are in mongo shell, now we create a javascript that a json object that defines the primary, secondary and arbiter.Assigning a priority of 10 means it is primary. After it is defined then rs.initiate is run

var demoConfig = {
	"_id" : "demo",
	"members" : [
			{
				"_id" : 0,
				"host" : "192.200.15.72: 30000",
				"priority" : 10
			},
			{
				"_id" : 1,
				"host" : "192.200.15.72: 40000"
			},
			{
				"_id" : 2,
				"host" : "192.200.15.72: 50000",
				"arbiterOnly" : true
			}
		]
}
rs.initiate(demoConfig)

At this point, replica set is initiated. Now will create a collection in primary and verify it was replicated in secondary.

use test
db.phonecountrycodes.save({_id:1, country:'USA', code:001})
db.phonecountrycodes.find()

Now exit and connect to secondary.

mongo --port 40000

Check the data in created collections, it will give an error initially because secondary needs to be set as slave.

db.phonecountrycodes.find()
db.setSlaveOk()
db.phonecountrycodes.find()

To test the automatic failover, from another window kill the primary and watch the prompt changes to primary of the open mongo shell. Through the use of polling algorithm arbiter detects primary is down and makes secondary as primary.

ps -ef|grep -i 'mongod'
kill xxxxx

If you can run a python script, I suggest to perform this test using python. It will help you understand, how application reacts to automatic fail over.

Install pymongo and run this program.

from pymongo import *
from time import *

while (True):
    sleep(3)
    client = MongoClient(['192.200.15.72:30000','192.200.15.72:40000', '192.200.15.72:50000'], replicaSet='demo')
    db = client['test']
    code = db.phonecountrycodes.find_one()
    print client.primary
    client.close()

While this program is running, in another window, kill the mongod primary process and restart it after a few seconds.

kill xxxxx
mongod --dbpath ./db1 --port 30000 --replSet "demo" --fork --logpath /var/log/mongodb.log

Clearly, you can see the primary output of the python program alternate between 192.200.15.72:30000 and 192.200.15.72:40000 very seamlessly.

/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python /Users/rakesh.varma/PycharmProjects/dtv_test/pymongotest.py
('192.200.15.72', 30000)
('192.200.15.72', 30000)
('192.200.15.72', 30000)
('192.200.15.72', 30000)
('192.200.15.72', 30000)
('192.200.15.72', 30000)
('192.200.15.72', 40000)
('192.200.15.72', 40000)
('192.200.15.72', 40000)
('192.200.15.72', 40000)
('192.200.15.72', 40000)
('192.200.15.72', 40000)
('192.200.15.72', 30000)
('192.200.15.72', 30000)
('192.200.15.72', 30000)

No disruption is observed by the application.

Java vs Python Performance

Sat, 04 Jul 2015 11:34:11 -0400

It is well known that Python programs tend to run slower compared to Java, C#.net, C++. This post attempts to demonstrate the fact. I have taken a classic CPU bound problem, generate nth prime number, implemented a python solution and java solution based on the same algorithm.

Java implementation source code is available at this link

Python implementation source code is available at this link

Java Single Thread Test took 127 milli seconds.

Python Single Thread Test took 4.8 seconds.

The difference is quite stark, python performs 35 times worse. I’ve learnt that to perform performance benchmarking of different languages, need to try them on CPU bound problems. Think of CPU bound problems as one that have heavy looping, sorting, floating point operations and they have no IO (databases, file reads, network, etc). If you program contains IO operations, then it is not usually suitable for a good benchmark test because IO time will consume a significant execution time and IO execution time(database call or a web service call) is usually the same regardless of the python or java implementation.

Conclusion

It is very hard to beat the complied implementations of these CPU bound, iterative numerical calculations.
Python most selling features that evolve around developer productivity. But this kind of a large scale iterative mathematical operations come with a cost. The python solution was run on the cPython, the default implementation of python. However, using cython, pypy or even numpy would offer significant performance benefits.
I have include a test for multi-threading also. Since python uses GIL (global interpretor lock), the multi-threaded implementation actually perform worse than the single threaded implementation. However, for Java it is not the case.

Python Global Interpreter Lock

Thu, 28 May 2015 11:34:11 -0400

Python uses GIL (Global interpreter lock). GIL ensures that python process can only process 1 instruction at a time, regardless the number of cores that CPU is using. This has been the behavior of the default implementation of python, cPython.

However, this could be overcome by using alternative implementations such as cython and by using mutliprocessing module (instead of threading).

To understand the impact of GIL, I ran this program on a multi-core CPU laptop. This is a classic CPU bound problem, generate the nth prime. Think of CPU bound problem as ones that has heavy looping, floating point operations and no IO (databases, disk operations, network, etc).

The full source code is available at this link

import math
import threading, time
from functools import wraps

def timefn(fn):
    @wraps(fn)
    def measure_time(*args, **kwargs):
        t1 = time.time()
        result = fn(*args, **kwargs)
        t2 = time.time()
        print ("@timefn:" + fn.func_name + " took " + str(t2 - t1) + " seconds")
        return result
    return measure_time

def is_prime(num):
    for j in range(2,int(math.sqrt(num)+1)):
        if (num % j) == 0:
            return False
    return True

def prime(nth):
    i = 0
    num = 2
    while i < nth:
        if is_prime(num):
            i += 1
            if i == nth:
                print('The ' + str(nth) + 'th prime number is: ' + str(num))
        num += 1

@timefn
def singleThreadtest():
    for count in range(12):
        prime(10000)


@timefn
def multiThreadtest():
    threads = []
    for count in range(12):
        t = threading.Thread(target = prime, args=[10000])
        threads.append(t)
        t.start()

    for t in threads:
        t.join()

if __name__ == "__main__":
    singleThreadtest()
    multiThreadtest()

@timefn:singleThreadtest took 4.8399810791 seconds.

@timefn:multiThreadtest took 5.75388002396 seconds.

Although, generally speaking, it is a bit suprising to hear that mutli-threaded version performs slower than the single-threaded version, however in python GIL world, it should not be much of a surprise. Because, only single core of CPU is used and only 1 instruction is executed at a given time. So, implementing threading in this case creates overhead of managing threads and introduces waits for shared resources.

Conclusion

If your program is a CPU bound, think twice about implementing threading. At the least, experiment using cython or multi-processing modules.
If your program is IO-bound, then multi-threading will generate real speeds. You can go thru my post title “Oracle AWR Using Python”. I’ve tested multi-threading there and performance benefits are exponential.

Oracle AWR generation using Python

Sat, 23 May 2015 11:34:11 -0400

I had to implement a job that involved saving the Oracle performance stats to CSV files on a disk. It was an interesting problem to say the least because speed was a consideration and obviously any technology could be choosen. But I choose python, mainly because python (along with most common libraries) comes automatically shipped on most commercial linux servers.

The full souce code is available [at this link] (https://bitbucket.org/varmarakesh/fullstack/src/0c08ed6ea9e8a7c02ba702775725f60e37dddde4/python/awr/?at=master)

#Oracle AWR

AWR (automatic workload repository) collects and stores all runtime stats generated in Oracle database. It is an effective tool used by database community to troubleshoot and address performance problems. In short all the data related to application sql’s, connections, resource usages, etc is stored in Oracle and is accessible thru a number of views (DBA_HIST_). Since this data tends to be historical, it is grouped by a certain interval and it is known as snapshot.

This gives the list of snapshot Id’s that would group all the stats generated today. Usually the interval time would be every 30 min or 1 hour.

SELECT 
	distinct SNAP_ID, 
	to_char(END_INTERVAL_TIME, 'hh24:mi_dd_mon_yy') as "END_INTERVAL_TIME" 
FROM 
	DBA_HIST_SNAPSHOT 
WHERE 
	TRUNC(END_INTERVAL_TIME) = TRUNC(SYSDATE) 
ORDER BY 
	SNAP_ID

Once there is the list of snapshot Id’s, we can get the runtime stats for all the sql’s in the specific snapshot using this query like this.

SELECT
                     D.NAME,
                     SS.SNAP_ID,
                     SS.INSTANCE_NUMBER,
                     SS.SQL_ID,
                     ST.SQL_TEXT,
                     MODULE,
                     ACTION,
                     PARSING_SCHEMA_NAME,
                     FETCHES_TOTAL,
                     SORTS_TOTAL,
                     EXECUTIONS_TOTAL,
                     LOADS_TOTAL,
                     PARSE_CALLS_TOTAL,
                     DISK_READS_TOTAL,
                     BUFFER_GETS_TOTAL,
                     ROWS_PROCESSED_TOTAL,
                     CPU_TIME_TOTAL,
                     IOWAIT_TOTAL,
                     ELAPSED_TIME_TOTAL
                    FROM
                     DBA_HIST_SQLSTAT SS,
                     DBA_HIST_SNAPSHOT SP,
                     DBA_HIST_SQLTEXT ST,
                     V$DATABASE D
                    WHERE
                     SS.SNAP_ID = SP.SNAP_ID AND
                     SS.DBID = SP.DBID AND
                     SS.DBID = D.DBID AND
                     SS.INSTANCE_NUMBER = SP.INSTANCE_NUMBER AND
                     SS.DBID = ST.DBID AND
                     SS.SQL_ID = ST.SQL_ID AND
                     SS.SNAP_ID = :SNAP_ID

#Save Results to a CSV File

So, basically need to get the details snapshot details iteratively and save the results to a local CSV File. I used the csv module in python for it.

import csv
conn = getOracleConnection()
        cursor = conn.cursor()
        cursor.execute(q_awr, [snap['snap_id']])
        with open(path, "wb") as csv_file:
            csv_writer = csv.writer(csv_file)
            csv_writer.writerow([i[0] for i in cursor.description]) # write headers
            csv_writer.writerows(cursor)
        cursor.close()

#Multi-threading

So, the need is issue Oracle queries multiple times (once every snapshot) and save the results in multiple CSV files (once file per snapshot). This is a classic IO bound problem, because, the program issues a database call, waits for the response, save the data to disk, then makes another call and so on. So, I implemented threading and you can in the end that the performance gain using threading is pretty amazing.

So, get a list of snapshots, then spawn a thread for each snapshot. Each thread spawn invokes a method called save_awr to get the snapshot details and saves the results to a file.

def generate_awr_multithreads():
    snapshots = get_snapshots()
    threads = []
    for requestcount in range(snapshots.__len__()):
        t = threading.Thread(target = save_awr, args=[snapshots[requestcount]])
        threads.append(t)
        t.start()

    for t in threads:
        t.join()

This program saves the results to file system.

$ ls -lrt AWR*.csv
-rw-r--r--  1 rakesh.varma  staff    86K May 25 12:08 AWR_DLINKD_5074_10:00_25_may_15.csv
-rw-r--r--  1 rakesh.varma  staff    85K May 25 12:08 AWR_DLINKD_5075_11:00_25_may_15.csv
-rw-r--r--  1 rakesh.varma  staff    93K May 25 12:08 AWR_DLINKD_5076_12:00_25_may_15.csv
-rw-r--r--  1 rakesh.varma  staff    81K May 25 12:09 AWR_DLINKD_5064_00:00_25_may_15.csv
-rw-r--r--  1 rakesh.varma  staff    78K May 25 12:09 AWR_DLINKD_5065_01:00_25_may_15.csv
-rw-r--r--  1 rakesh.varma  staff    85K May 25 12:10 AWR_DLINKD_5066_02:00_25_may_15.csv
-rw-r--r--  1 rakesh.varma  staff    81K May 25 12:10 AWR_DLINKD_5067_03:00_25_may_15.csv
-rw-r--r--  1 rakesh.varma  staff   196K May 25 12:11 AWR_DLINKD_5068_04:00_25_may_15.csv
-rw-r--r--  1 rakesh.varma  staff    87K May 25 12:11 AWR_DLINKD_5069_05:00_25_may_15.csv
-rw-r--r--  1 rakesh.varma  staff   111K May 25 12:12 AWR_DLINKD_5070_06:00_25_may_15.csv
-rw-r--r--  1 rakesh.varma  staff    86K May 25 12:12 AWR_DLINKD_5071_07:00_25_may_15.csv
-rw-r--r--  1 rakesh.varma  staff    95K May 25 12:13 AWR_DLINKD_5072_08:00_25_may_15.csv
-rw-r--r--  1 rakesh.varma  staff    56K May 25 12:13 AWR_DLINKD_5073_09:00_25_may_15.csv

If multi-threading could not be implemented, this is how a single threaded version would like.

#Single Thread

def generate_awr_singlethread():
    snapshots = get_snapshots()
    for requestcount in range(snapshots.__len__()):
        save_awr(snap = snapshots[requestcount])

#Conclusion

Elapsed time differences between multi-threading and single threaded versions are very stark because this is a classic example of an IO bound problem. Oracle connections and saving the AWR data to disk are two IO bound tasks. Saving the AWR data is very IO intensive given the volume of data. These are the results when I ran this script against my database.

@timefn:generate_awr_multithreads took 58.2649428844 seconds

@timefn:generate_awr_singlethread took 361.62610507 seconds

Please check the source at https://bitbucket.org/varmarakesh/fullstack/src and let me know the feedback. Anybody that has an Oracle client installed, the setup is quite minimal and it will work without issues.

Oracle Cascade Delete - Always better than explicit child table deletes

Sat, 16 May 2015 11:34:11 -0400

Throughout all applications, there are many areas where we enforce basic referential integrity, but various applications handle differently the scenarios when it comes to deleting the child records in the event of the parent record delete. Oracle, like many other databases, offers referential integrity with on delete cascade option. When child tables are created with foreign keys that have the ‘on delete cascade’ option, then deletion of parent record will automatically delete the records in all child tables. The other option to delete the child table records is to, explicitly delete the records in all child tables. This article discusses the POC of both approaches. Oracle run time stats have been gathered for both approaches. If you want to try please use sqlplus.

POC Sample (Cascade Delete vs manual deletion of child table records)

parent = parent
children = child1, child2, child2.
Tests conducted on exadata, version = Oracle enterprise version, 11.2.0.4.0 - 64bit

No On Cascade Delete Option

Without using ‘on delete cascade’ option, the application needs to perform a explicit delete of all child tables.

SET AUTOTRACE ON
SET TRANSACTION NAME 'POC_NO_CASCADE_DELETE'
DELETE FROM CHILD1 WHERE ID = 100;
DELETE FROM CHILD2 WHERE ID = 100;
DELETE FROM CHILD3 WHERE ID = 100;
SELECT USED_UBLK FROM V$TRANSACTION WHERE NAME='POC_NO_CASCADE_DELETE';
ROLLBACK;

On Cascade Delete Option

Using this deletion option, need to specify ‘on delete cascade’ when creating the constraint for each child table.

ALTER TABLE CHILD1 ADD CONSTRAINT FK01_CHILD1 FOREIGN KEY (ID) REFERENCES PARENT (ID) ON DELETE CASCADE ENABLE;
ALTER TABLE CHILD2 ADD CONSTRAINT FK01_CHILD2 FOREIGN KEY (ID) REFERENCES PARENT (ID) ON DELETE CASCADE ENABLE;
ALTER TABLE CHILD3 ADD CONSTRAINT FK01_CHILD3 FOREIGN KEY (ID) REFERENCES PARENT (ID) ON DELETE CASCADE ENABLE;
SET TRANSACTION NAME 'POC_CASCADE_DELETE'
SET AUTOTRACE ON;
DELETE FROM PARENT WHERE ID = 100;
SELECT USED_UBLK FROM V$TRANSACTION WHERE NAME='POC_CASCADE_DELETE';

Once the constraints with ‘on delete cascade’ option is in place, deleting the parent record while automatically remove the child record.

Oracle Stat	No Cascade Delete	Cascade Delete
Total Recursive Calls	426	363
Total db block gets	19	19
Total redo	3224	2956
Total undo blocks	3	1

Results Summary

There is no performance penalty using the ‘On delete cascade’. Oracle stats perform well on all areas with ‘Cascade Delete’ option.
There is a significant improvement of undo tablespace usage when using the ‘Cascade Deletion’ option. Oracle is using at least 1 block for undo associated with each DML statement. However, when deletes are implicit using cascade deletes, the undo usage for all deletes is just 1 undo block. This is an important observation because when benchmarking the results of deletes, undo and redo are perhaps the most important stats.
Using ‘On Delete Cascade’ , the application code (data models) is very easy to maintain. The purge scripts are also easier to maintain. We don’t need to track the order of the child table deletes. Importantly, when table model change or new child tables are created, the application code and purge scripts do not need to updated.

Create blog in bitbucket using jekyll

Sat, 16 May 2015 11:34:11 -0400

Jekyll is a great way to quickly build a blog site. I was looking for easier ways to setup a blog on bitbucket and stumbled upon Jekyll.

In this post I documented some steps to quickly create a blog (along with some posts) site and deploy it to bitbucket.

Install jekyll and create a new project

$ gem install jeykyll
$ jeykyll jeykyll_blog
$ cd jeykyll_blog
$ jekyll serve

jekyll serve will generate all the files in the _site directory.

Open http://127.0.0.1:4000/ in a browser. This will render the blog.

.
├── _config.yml
├── _includes
│   ├── footer.html
│   ├── head.html
│   └── header.html
├── _layouts
│   ├── default.html
│   ├── page.html
│   └── post.html
├── _posts
│   └── 2015-05-16-jekyll-setup.markdown
├── _sass
│   ├── _base.scss
│   ├── _layout.scss
│   └── _syntax-highlighting.scss
├── _site
│   ├── about
│   │   └── index.html
│   ├── css
│   │   └── main.css
│   ├── feed.xml
│   ├── index.html
│   └── jekyll
│       └── update
│           └── 2015
│               └── 05
│                   └── 16
│                       └── jekyll-setup.html
├── about.md
├── css
│   └── main.scss
├── feed.xml
└── index.html

This is the directory structure of the jeykll. config.yml => has the common settings that are used throughout the site. posts => posts directory the individual posts. The jekyll will generate the post htmls that will be created in the jeykll/update directory.

update about.md, posts.

You can see the updates by refreshing the http://127.0.0.1:4000/

Now that we have the blog running successfullly locally, we need to deploy it to the bitbucket.

create a bitbucket repository with the following format, username.bitbucket.org add all the files from _site folder to this repo locally and then push it to remote master. This are the complete commands.

$ cd jekyll_blog/
$ jekyll build
$ cp -r _site/ ~/bitbucket/username.bitbucket.org/
$ cd ~/bitbucket/username.bitbucket.org
$ git config --global user.name "username"
$ git config --global user.email "user.name@gmail.com"
$ git remote set-url origin https://bitbucket.org/username/username.bitbucket.org
$ git config -l
$ git add --all
$ git commit -m "Jekyll post creation."
$ git push origin master

Now, open username.bitbucket.org and you should see your blog.