Production Server: openIMIS With Gunicorn And Nginx On A Linux Server.

1 Understanding terminology: Gunicorn and Nginx
- 1.1 1. Why Use Gunicorn, Django, and Nginx Together?
- 1.2 2. Production Status
- 1.3 3. Benefits of Using Gunicorn with Django and Nginx
  - 1.3.1 a. Performance
  - 1.3.2 b. Scalability
  - 1.3.3 c. Security
  - 1.3.4 d. Static File Handling
  - 1.3.5 e. Flexibility
2 Estimation of load and requirement of server configuration
3 Step 1 - Installing python and nginx
4 Step 2 - Creating a python virtual environment
5 Step 3 - Installing Django and gunicorn
6 Step 4 - Cloning up openIMIS project
- 6.1 Step 4.1 - Install necessary modules requirements
- 6.2 Step 4.2 - setup necessary environment variables and database configuration
7 Step 5 - Configuring gunicorn
8 Step 6 - Configuring Nginx as a reverse proxy
9 Step 7 - Configure frontend
10 Step 8 - Test the deployment
11 Troubleshooting guide for daemon service

This guide will let user to deploy openIMIS in any VPS service (AWS/Digital Ocean/ Google cloud / Azure etc)

Strategy:

Understanding terminology: Gunicorn and Nginx

Using Gunicorn with Django and Nginx is a production-grade setup and is widely used in the industry to deploy Django applications. This combination is robust, scalable, and well-documented. Below is an overview of the production status, benefits, and considerations for using this stack:

1. Why Use Gunicorn, Django, and Nginx Together?

Gunicorn: A production-ready WSGI (Web Server Gateway Interface) server for serving Django applications. It handles multiple requests concurrently and is designed for performance.
Django: A high-level Python web framework for building web applications.
Nginx: A high-performance web server and reverse proxy. It serves static files, handles SSL termination, and routes requests to Gunicorn.

This setup separates responsibilities:

Gunicorn handles the Django application logic.
Nginx handles static files, load balancing, and acts as a reverse proxy.

2. Production Status

Widely Adopted: This setup is used by many companies and organizations to deploy Django applications in production.
Scalable: Nginx and Gunicorn can handle high traffic loads when configured properly.
Secure: Nginx provides features like SSL termination, rate limiting, and request filtering, which enhance security.
Well-Documented: Both Gunicorn and Nginx have extensive documentation and community support.

3. Benefits of Using Gunicorn with Django and Nginx

a. Performance

Gunicorn uses a pre-fork worker model, which allows it to handle multiple requests concurrently.
Nginx is highly efficient at serving static files and handling a large number of simultaneous connections.

b. Scalability

You can scale horizontally by adding more servers or vertically by increasing resources (e.g., CPU, RAM).
Nginx can act as a load balancer to distribute traffic across multiple Gunicorn instances.

c. Security

Nginx provides SSL/TLS termination, protecting your application with HTTPS.
It can also filter malicious requests and prevent DDoS attacks.

d. Static File Handling

Nginx is optimized for serving static files (e.g., CSS, JavaScript, images), reducing the load on Gunicorn and Django.

e. Flexibility

You can customize Nginx and Gunicorn configurations to suit your application's needs.

Estimation of load and requirement of server configuration

Calculating the production capability of a server in terms of how many parallel or concurrent users it can handle depends on several factors, including:

Application Complexity: The resource usage of your Django application (e.g., CPU, memory, database queries).
Traffic Patterns: The number of requests per second, request size, and response time.
Gunicorn Configuration: The number of workers and threads.
Database and Caching: Whether you're using a database, caching (e.g., Redis), and how optimized they are.
Static File Serving: Whether static files are served efficiently (e.g., via Nginx or a CDN).

Below, are the estimates for the number of concurrent users each server configuration can handle, assuming a typical Django application with moderate complexity.

1. General Assumptions

Gunicorn Workers: Each worker can handle one request at a time. The number of workers is typically calculated as (2 * CPU cores) + 1.
Memory Usage: Each worker consumes ~50-100 MB of RAM (depending on the application).
Concurrent Users: Each user makes a request that takes ~200-500 ms to process (including database queries and rendering).
Static Files: Served by Nginx, so they don’t consume Gunicorn resources.

Source for estimation of the workers based on server configuration

Django performance guide

2. Server Configurations and Estimates

The server configuration depends on the number of CPU cores, RAM, and the type of workload (e.g., CPU-bound, memory-bound, or I/O-bound). Here’s a general guideline:

a. Gunicorn Workers

The number of Gunicorn workers is calculated using the formula:

Copy

number_of_workers = (2 * CPU_cores) + 1

This formula ensures that the server utilizes all available CPU cores efficiently while leaving some overhead for the operating system and other processes.

gunicorn memory usage and cpu usage

b. Memory Usage

Each Gunicorn worker typically consumes average 60-80 MB of RAM memory utilize by worker, depending on the size and complexity of the Django application. The total memory usage should not exceed the available RAM.

c. Database and Caching

Use a separate database server or a managed database service (e.g., AWS RDS, PostgreSQL).
Use Redis or Memcached for caching frequently accessed data.

d. Static Files

Serve static files using Nginx or a CDN (e.g., Cloudflare, AWS CloudFront).

Step 1 - Installing python and nginx

sudo apt update
sudo apt install python3-pip python3-dev nginx

This will install python, pip and nginx server

Step 2 - Creating a python virtual environment

sudo pip3 install virtualenv

This will install a virtual environment package in python. Let's create a project directory to host our openIMIS and create a virtual environment.

mkdir ~/openimis
cd ~/openimis
virtualenv env

A virtual environment named env will be created. Let's activate this virtual environment:

source venv/bin/activate

Step 3 - Installing Django and gunicorn

pip install django gunicorn

Step 4 - Cloning up openIMIS project

install git before you start cloning the openIMIS project <openimis-be_py>

Install Git - guide

git clone https://github.com/openimis/openimis-be_py.git

Step 4.1 - Install necessary modules requirements

pip install requirements.txt

Step 4.2 - setup necessary environment variables and database configuration

Add your IP address or domain to the ALLOWED_HOSTS variable in settings.py.

Configure openIMIS database settings: In this guide postges database shall be used

copy .env.example to .env

change the database configuration in .env file

DB_ENGINE = postgres database

DB_HOST = host / ip address where your openIMIS database is located

DB_PORT= database port default port is 5432

DB_NAME = database name

DB_USER = database username

# Database engine, should be changed for mssql
DB_ENGINE=django.db.backends.postgresql
# Database host
DB_HOST=pg-openimis-sunilparajuli2002-1955.l.aivencloud.com
# Database port
DB_PORT=12345
# Database name
DB_NAME=openimis
# Database username
DB_USER=12345
# Database user password
DB_PASSWORD=abcdefghijkl
# Site root that will prefix all exposed endpoints. It's required when working with openIMIS frontend
SITE_ROOT=api
# Should the debug be on (i.e. debug information will be displayed)
DEBUG=True
# Log level to be used. Remove for default.
DJANGO_LOG_LEVEL=WARNING
# Log handler to be used, reffer to openIMIS/openIMIS/settings.py. Remove for default.
DJANGO_LOG_HANDLER=debug-log
# Photo path root used in insuree module. Only used if InsureeConfig value not specified. Comment out for default.
PHOTO_ROOT_PATH=<photo path>
# Should the database be migrated before start (entrypoint.sh - docker setup). Will be migrated anyway if $SITE_ROOT=api. Comment out for False
DJANGO_MIGRATE=True
# Should the modules be searched for scheduled tasks. Comment out for false
# SCHEDULER_AUTOSTART=True
#for testing purpose, this can be set to 
#DB_TEST_NAME=test_imis

Run migrations using below command :

/openimis-be_py/openIMIS# python manage.py makemigrations
/openimis-be_py/openIMIS# python manage.py migrate

Let's test this openIMIS project by running the following commands:

sudo ufw allow 8000

This opens port 8000 by allowing it over the firewall. Let's start our openIMIS Django development server to test the setup so far:

/openimis-be_py/openIMIS# python manage.py runserver 0.0.0.0:8000

Step 5 - Configuring gunicorn

Lets test gunicorn's ability to serve our application by firing the following commands:

gunicorn --bind 0.0.0.0:<port> openIMIS.wsgi

Let's create a system socket file for gunicorn now:

sudo vim /etc/systemd/system/gunicorn.socket

Insert the code below, this will create a service for gunicorn, which will run even the server restarts

Description:

[Unit]
Description=gunicorn socket

[Socket]
ListenStream=/run/gunicorn.sock

[Install]
WantedBy=sockets.target

Next, we will create a service file for gunicorn

sudo vim /etc/systemd/system/gunicorn.service

Paste below code, remember to setup your path , it is just an example:

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target

[Service]
User=root
Group=www-data
WorkingDirectory=/var/www/html/openimis/openimis-be_py/openIMIS
ExecStart=/var/www/html/venv/bin/gunicorn \
          --access-logfile - \
          --workers 3 --timeout 300 \
          --bind unix:/run/gunicorn.sock \
          openIMIS.wsgi:application

[Install]
WantedBy=multi-user.target

Lets now start and enable the gunicorn socket

sudo systemctl enable gunicorn.socket
sudo systemctl start gunicorn.socket

Step 6 - Configuring Nginx as a reverse proxy

install nginx webserver which will host frontend and backend of openIMIS

Install Nginx tutorial - link

Create a configuration file for Nginx using the following command

install nginx and create your site name - example : frontend

sudo nano /etc/nginx/sites-available/frontend

Paste the below contents inside the file created

server {
    listen 80;
    server_name demoimis.tinker.com.np www.demoimis.tinker.com.np; # Change this to your domain name or server IP

    location / {
        root /var/www/html/frontend/build;
        index index.html index.htm;
        try_files $uri $uri/ /index.html;
    }

    location /api/ {
        proxy_pass http://unix:/run/gunicorn.sock;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Activate the configuration using the following command:

sudo ln -s /etc/nginx/sites-available/frontend /etc/nginx/sites-enabled/

Restart nginx and allow the changes to take place.

sudo systemctl restart nginx

Step 7 - Configure frontend

Go to the frontend directory of openIMIS <openimis-fe_js> and run following command

yarn build

this will generate build directory within the project directory

copy the build folder to VPS server and lets give it a name “frontend/build”. The location can be set to /var/www/html

configure the nginx to be set in port 80 to serve it as static file as stated above nginx configuration:

Make sure you configure proxy pass variable in package.json in frontend, in this configuration to set it to the django server

server {
    listen 80;
    server_name demoimis.tinker.com.np www.demoimis.tinker.com.np; # Change this to your domain name or server IP

    location / {
        root /var/www/html/frontend/build;
        index index.html index.htm;
        try_files $uri $uri/ /index.html;
    }

Step 8 - Test the deployment

You should configure A name and DNS address to be able to host it in domain name

open the domain name you have set under server_name

Example:

Troubleshooting guide for daemon service

If the server fails to start or changes are made using pull / push from git then, gunicorn daemon service might needs to restart.

check if gunicorn service status

sudo service gunicorn status

restart the daemon service of gunicorn

sudo systemctl daemon-reload
sudo service gunicorn reload

openIMIS