Avil Page (Posts about devops)

Build/Deploy Full Stack Web Apps with only Config

Anand Reddy Pandikunta — Sat, 28 Mar 2026 06:17:29 GMT

The best code is the code you don't have to write

Config First Tools

Any tool that provides first class support for configuration files can be considered a Config First Tool. These tools allow developers to setup/build/deploy applications by writing configuration files instead of writing code.

In DevOps world, tools like Terraform, Ansible, and Kubernetes are examples of Config First Tools. These kinds of tools are not that popular in the web development world.

In this article, lets see how we can build a full stack web application using only configuration files.

Todo - Web App

A simple backend API that allows users to manage todo lists. A simple frontend that uses the backend API. A simple script to deploy the backend and frontend applications.

Backend API

PocketBase is a backend API that provides a simple way to create and manage databases, authentication, and file storage. It is a great tool for building simple backend APIs without writing any code.

$ brew install pocketbase
$ pocketbase serve

We can create a collection called Todo with required fields and pocketbase will automatically generate the API for us.

Frontend

Lowdefy is a config first tool that allows us to build web applications using yaml config.

$ npm install -g lowdefy
$ lowdefy init
$ lowdefy dev

For basic crud apps, instead of us writing the boilerplate config, we can ask an AI agent to write the config for us.

Deployment

For deployment, we can use Ansible to deploy both the backend and frontend applications. Here also we can just AI agent to write the playbook for us.

$ pip install ansible
$ ansible-playbook deploy.yml

OliveTin

Even though we can use ansible to deploy the applications, it is not that convenient to run ansible playbooks on mobile. OliveTin is a config first tool that allows us to create web UIs for our scripts.

  - title: "Deploy ToDo App"
    popupOnStart: execution-dialog
    shell: ansible-playbook deploy.yml
    timeout: 6000
    icon: "🤖"

We can create a simple web UI for our deployment script with just 5 lines.

Conclusion

For quick prototyping and simple applications, config first tools can be a great choice as they allow us to build applications without writing any code. They also allow us to focus on the application logic rather than the implementation details.

With the rise of AI, it is much easy to maintain 500 lines of config than 5000 lines of code.

In Praise of OliveTin

Anand Reddy Pandikunta — Fri, 30 Jan 2026 20:13:11 GMT

In a world of complex internal tools, OliveTin bridges the gap between raw CLI and usable web UI with just yaml config.

Server Admin Panel

I used admin panels like CockPit, Ajenti, etc to provide simple web UI for non-developers to provide access to services. Developing custom widgets is time consuming and requires programming skills as well.

With OliveTin, I can provide web UI directly with just yaml config. This saves a lot of time and effort. It also provides a clean UI as it starts from scratch.

Backend Driven UI

For existing python scripts, I often use tools like Streamlit, NiceGUI, etc to create web UIs. Organizing the UI components for multiple scripts is time consuming.

OliveTin can provide web UI directly python scripts and they can be organized in groups with just yaml config.

Mobile Friendly

To do a simple deployment on mobile, I need to open an app that supports ssh, ssh into the server, navigate to a directory, run a script to deploy.

OliveTin provides a "Deploy" button on mobile browser which is way convenient.

Authentication

OliveTin provides authentication(local users, OAuth2, JWT, etc), authorization(ACLs) & accounting(logs) out of the box. With just yaml config, we can secure the web UI.

Conclusion

There are hundreds of other features provided by OliveTin like scheduling, file uploads, webhooks, etc.

If you ever want to provide a simple web UI for scripts with low code or no code, give OliveTin a try!

Remote Access to k3d/k3s Kubernetes cluster

Anand Reddy Pandikunta — Wed, 17 Dec 2025 03:10:39 GMT

Introduction

We learnt how to deploy kubernetes cluster anywhere with a single k3d command. By default, k3d cluster is accessible only from the host machine.

Remote Access

Create new cluster with

$ k3d cluster create cloud-k8s \
  --api-port 6443:6443@loadbalancer \
  --k3s-arg "--tls-san=<remote-ip>@server:*"

--api-port 6443:6443@loadbalancer maps port 6443 on the host machine to the cluster's API server load balancer. This can be changed (e.g., 8080:6443) if 6443 is taken on the host.
--k3s-arg "--tls-san=..." adds the host's public IP to the certificate, preventing SSL errors later.
Ensure remote server firewall allows incoming traffic on the chosen port (e.g., 6443 or 8080). To verify, run telnet <remote-ip> 6443 from local machine.

On the remote server, get kubeconfig:

$ k3d kubeconfig get cloud-k8s
---
apiVersion: v1
clusters:
- cluster:
    server: https://0.0.0.0:6443
  name: k3d-rk
   ... # truncated for brevity

Copy this kubeconfig, replace 0.0.0.0 with the remote server's public IP.

Open Free Lens, paste this config and connect to the remote k3d cluster.

Conclusion

You can use your favorite Kubernetes tools (kubectl, FreeLens, k9s, etc.) to manage the remote k8s cluster.

Migrating from Nginx to Caddy

Anand Reddy Pandikunta — Fri, 30 May 2025 15:51:21 GMT

When deploying web apps in production, a web server is essential for serving static files, handling reverse proxying, and managing SSL/TLS certificates.

I have been using Nginx for a long time, but I have recently switched to Caddy as it provides automatic HTTPS by default. In addition, Caddy has a simpler configuration syntax and is easier to set up for basic use cases.

Nginx vs Caddy

To illustrate the differences, here are some examples of how to configure a simple web server in both Nginx and Caddy.

Nginx Configuration

server {
    listen 80;
    server_name avilpage.com;

    location / {
        root /var/www/html;
        index index.html index.htm;
    }

    location /api {
        proxy_pass http://localhost:3000;
    }
}

With Nginx, you need to manually handle SSL/TLS certificates, which can be cumbersome. You typically use tools like Certbot to obtain and renew certificates.

Caddy Configuration

avilpage.com {
    root * /var/www/html
    file_server

    reverse_proxy /api localhost:3000
}

Caddy automatically manages SSL/TLS certificates for you, so you don't need to worry about obtaining or renewing them. It also provides default redirection from HTTP to HTTPS, making it easier to secure your site.

Conclusion

Caddy's automatic HTTPS and simpler configuration syntax make it a compelling choice for web servers, especially for those who want to get started quickly without dealing with the complexities of SSL/TLS management.

Free DockerHub Alternative - ECR Public Gallery

Anand Reddy Pandikunta — Sun, 09 Feb 2025 16:08:34 GMT

DockerHub started rate limiting¹ anonymous docker pulls. When testing out a new CI/CD setup, I hit the rate limit and had to wait for an hour to pull the image. This was a good time to look for alternatives.

AWS ECR Public Gallery² is a good alternative to DockerHub as of today(2025 Feb). It is free and does not have rate limits even for anonymous users.

Once we find the required image from the gallery, we can simply change the image name in the docker pull command to pull the image from ECR Gallery.

docker pull public.ecr.aws/ubuntu/ubuntu

In Dockerfile, we can use the image from ECR Gallery as follows:

FROM public.ecr.aws/ubuntu/ubuntu

That is a quick way to avoid DockerHub rate limits.

How (and when) to use systemd timer instead of cronjob

Anand Reddy Pandikunta — Mon, 05 Aug 2024 07:37:50 GMT

Introduction

* * * * * bash demo.sh

Just a single line of code is sufficient to schedule a cron job. However, there are some scenarios where I find systemd timer more useful than cronjob.

How to use systemd timer

We need to create a service file(contains the script to be run) and a timer(contains the schedule).

# demo.service
[Unit]
Description=Demo service

[Service]
ExecStart=bash demo.sh

# demo.timer
[Unit]
Description=Run myscript.service every 1 minutes

[Timer]
OnBootSec=1min
OnUnitActiveSec=1min

[Install]
WantedBy=multi-user.target

We can copy these files to /etc/systemd/system/ and enable the timer.

$ sudo cp demo.service demo.timer /etc/systemd/system/

$ sudo systemctl daemon-reload

$ sudo systemctl enable --now demo.timer

We can use systemctl to see when the task is executed last and when it will be executed next.

$ sudo systemctl list-timers --all

Use Cases

Singleton - In the above example, lets say demo.sh takes ~10 minutes to run. With cron job, in ten minutes we will have 10 instances of demo.sh running. This is not ideal. With systemd timer, it will ensure only one instance of demo.sh is running at a time.
On demand runs - If we want to test out the script/job, systemd allows us to immediately run it with usual systemctl start demo without needing to run the script manually.
Timer - With cron, we can run tasks upto a minute precision. Timer can run tasks till second level precision.

[Timer]
OnCalendar=*-*-* 15:30:15

In addition to that, we can run tasks based on system events. For example, we can run a script 15 minutes from reboot.

[Timer]
OnBootSec=15min

Conclusion

Systemd timer is a powerful tool that can replace cronjob in many scenarios. It provides more control and flexibility over cronjob. However, cronjob is still a good choice for simple scheduling tasks.

Mastering Kraken2 - Part 2 - Performance Optimisation

Anand Reddy Pandikunta — Sun, 28 Jul 2024 05:21:30 GMT

Mastering Kraken2

Part 1 - Initial Runs

Part 2 - Classification Performance Optimisation (this post)

Part 3 - Build custom database indices

Part 4 - Build FDA-ARGOS index

Part 5 - Regular vs Fast Builds (upcoming)

Part 6 - Benchmarking (upcoming)

Introduction

In the previous post, we learned how to set up kraken2¹, download pre-built indices, and run kraken2. In this post, we will learn various ways to speed up the classification process.

Increasing RAM

Kraken2 standard database is ~80GB in size. It is recommended to have at least db size RAM to run kraken2 efficiently². Let's use 128GB RAM machine and run kraken2 with ERR10359977³ sample.

$ time kraken2 --db k2_standard --report report.txt ERR10359977.fastq.gz > output.txt
Loading database information... done.
95064 sequences (14.35 Mbp) processed in 2.142s (2662.9 Kseq/m, 402.02 Mbp/m).
  94816 sequences classified (99.74%)
  248 sequences unclassified (0.26%)
kraken2 --db k2_standard --report report.txt ERR10359977.fastq.gz >   1.68s user 152.19s system 35% cpu 7:17.55 total

Now the time taken has come down from 40 minutes to 7 minutes. The classification speed has also increased from 0.19 Mbp/m to 402.02 Mbp/m.

The previous sample had only a few reads, and the speed is not a good indicator. Let's run kraken2 with a larger sample.

$ time kraken2 --db k2_standard --report report.txt --paired SRR6915097_1.fastq.gz SRR6915097_2.fastq.gz > output.txt
Loading database information... done.
Processed 14980000 sequences (2972330207 bp) ...
17121245 sequences (3397.15 Mbp) processed in 797.424s (1288.2 Kseq/m, 255.61 Mbp/m).
  9826671 sequences classified (57.39%)
  7294574 sequences unclassified (42.61%)
kraken2 --db k2_standard --report report.txt --paired > output.txt  526.39s user 308.24s system 68% cpu 20:23.86 total

This took almost 20 minutes to classify ~3 Gbp of data. Out of 20 minutes, 13 minutes was spent in classification. The remaining time in loading the db into memory.

Let's use k2_plusPF⁴ db, which is twice the size of k2_standard and run kraken2.

$ time kraken2 --db k2_plusfp --report report.txt --paired SRR6915097_1.fastq.gz SRR6915097_2.fastq.gz > output.txt
Loading database information...done.
17121245 sequences (3397.15 Mbp) processed in 755.290s (1360.1 Kseq/m, 269.87 Mbp/m).
  9903824 sequences classified (57.85%)
  7217421 sequences unclassified (42.15%)
kraken2 --db k2_plusfp/ --report report.txt --paired SRR6915097_1.fastq.gz  >   509.71s user 509.51s system 55% cpu 30:35.49 total

This took ~30 minutes to complete, but the classification took only 13 minutes similar to k2_standard. The remaining time was spent in loading the db into memory.

Preloading db into RAM

We can use vmtouch⁵ to preload db into RAM. kraken2 provides --memory-mapping option to use preloaded db.

$ vmtouch -vt k2_standard/hash.k2d k2_standard/opts.k2d k2_standard/taxo.k2d
           Files: 3
     Directories: 0
   Touched Pages: 20382075 (77G)
         Elapsed: 434.77 seconds

When Linux requires RAM, it will incrementally evict the db from memory. To prevent this, we can copy the db to shared memory (/dev/shm) and then use vmtouch to preload the db.

$ cp -r k2_standard /dev/shm

$ vmtouch -t /dev/shm/*.k2d

Now, let's run kraken2 with --memory-mapping option.

$ time kraken2 --db k2_standard --report report.txt --memory-mapping --paired SRR6915097_1.fastq.gz SRR6915097_2.fastq.gz > output.txt
Loading database information... done.
17121245 sequences (3397.15 Mbp) processed in 532.486s (1929.2 Kseq/m, 382.79 Mbp/m).
  9826671 sequences classified (57.39%)
  7294574 sequences unclassified (42.61%)
  kraken2 --db k2_standard --report report.txt --paired SRR6915097_1.fastq.gz   >  424.20s user 11.76s system 81% cpu 8:54.98 total

Now the classification took only ~10 minutes.

Multi threading

kraken2 supports multiple threads. I am using a machine with 40 threads.

$ time kraken2 --db k2_standard --report report.txt --paired SRR6915097_1.fastq.gz SRR6915097_2.fastq.gz --memory-mapping --threads 32 > output.txt
Loading database information... done.
17121245 sequences (3397.15 Mbp) processed in 71.675s (14332.5 Kseq/m, 2843.81 Mbp/m).
  9826671 sequences classified (57.39%)
  7294574 sequences unclassified (42.61%)
kraken2 --db k2_standard --report report.txt --paired SRR6915097_1.fastq.gz      556.58s user 22.85s system 762% cpu 1:16.02 total

With 32 threads, the classification took only 1 minute. Beyond 32 threads, the classification time did not decrease significantly.

Optimising input files

So far we have used gzipped input files. Let's use unzipped input files and run kraken2.

$ gunzip SRR6915097_1.fastq.gz
$ gunzip SRR6915097_2.fastq.gz

$ time kraken2 --db k2_standard --report report.txt --paired SRR6915097_1.fastq SRR6915097_2.fastq --memory-mapping --threads 30 > output.txt
Loading database information... done.
17121245 sequences (3397.15 Mbp) processed in 34.809s (29512.0 Kseq/m, 5855.68 Mbp/m).
  9826671 sequences classified (57.39%)
  7294574 sequences unclassified (42.61%)
kraken2 --db k2_standard --report report.txt --paired SRR6915097_1.fastq    30   565.03s user 17.12s system 1530% cpu 38.047 total

Now the classification time has come down to 40 seconds.

Since the input fastq files are paired, interleaving the files also takes time. Lets interleave the files and run kraken2.

To interleave the files, lets use seqfu tool.

$ conda install -y -c conda-forge -c bioconda "seqfu>1.10"

$ seqfu interleave -1 SRR6915097_1.fastq.gz -2 SRR6915097_2.fastq.gz > SRR6915097.fastq

$ time kraken2 --db k2_standard --report report.txt --memory-mapping SRR6915097.fq --threads 32 > output.txt
Loading database information... done.
34242490 sequences (3397.15 Mbp) processed in 20.199s (101714.1 Kseq/m, 10090.91 Mbp/m).
  17983321 sequences classified (52.52%)
  16259169 sequences unclassified (47.48%)
kraken2 --db k2_standard --report report.txt --memory-mapping SRR6915097.fq  32  618.96s user 18.24s system 2653% cpu 24.013 total

Now the classification time has come down to 24 seconds.

Conclusion

In terms of classification speed, we have come a long way from 0.1 Mbp/m to 1200 Mbp/m. In the next post, we will learn how to optimise the creation of custom indices.

Headlamp - k8s Lens open source alternative

Anand Reddy Pandikunta — Sun, 23 Jun 2024 20:18:02 GMT

Since Lens is not open source, I tried out monokle, octant, k9s, and headlamp¹. Among them, headlamp UI & features are closest to Lens.

Headlamp

Headlamp is CNCF sandbox project that provides cross-platform desktop application to manage Kubernetes clusters. It auto-detects clusters and provides cluster wide resource usage by default.

It can also be installed inside the cluster and can be accessed using a web browser. This is useful when we want to access the cluster from a mobile device.

$ helm repo add headlamp https://headlamp-k8s.github.io/headlamp/

$ helm install headlamp headlamp/headlamp

Lets port-forward the service & copy the token to access it.

$ kubectl create token headlamp

# we can do this via headlamp UI as well
$ kubectl port-forward service/headlamp 8080:80

Now, we can access the headlamp UI at http://localhost:8080.

Conclusion

If you are looking for an open source alternative to Lens, headlamp is a good choice. It provides a similar UI & features as Lens, and it is accessible via mobile devices as well.

https://headlamp.dev/ ↩

macOS - Log & track historical CPU, RAM usage

Anand Reddy Pandikunta — Fri, 31 May 2024 20:18:02 GMT

In macOS, we can use inbuilt Activity Monitor or third party apps like Stats to check the live CPU/RAM usage. But, we can't track the historical CPU & memory usage. sar, atop can track the historical CPU & memory usage. But, they are not available for macOS.

Netdata

Netdata¹ is an open source observability tool that can monitor CPU, RAM, network, disk usage. It can also track the historical data.

Unfortunately, it is not stable on macOS. I tried installing it on multiple macbooks, but it didn't work. I raised an issue² on their GitHub repository and the team mentioned that macOS is a low priority for them.

Glances

Glances³ is a cross-platform monitoring tool that can monitor CPU, RAM, network, disk usage. It can also track the historical data.

We can install it using Brew or pip.

$ brew install glances

$ pip install glances

Once it is installed, we can monitor the resource usage using the below command.

$ glances

Glances can log historical data to a file using the below command.

$ glances --export-csv /tmp/glances.csv

In addition to that, it can log data to services like influxdb, prometheus, etc.

Let's install influxdb and export stats to it.

$ brew install influxdb
$ brew services start influxdb
$ influx setup

$ python -m pip install influxdb-client

$ cat glances.conf
[influxdb]
host=localhost
port=8086
protocol=http
org=avilpage
bucket=glances
token=secret_token

$ glances --export-influxdb -C glances.conf

We can view stats in the influxdb from Data Explorer web UI at http://localhost:8086.

Glances provides a prebuilt Grafana dashboard⁴ that we can import to visualize the stats.

From Grafana -> Dashboard -> Import, we can import the dashboard using the above URL.

Conclusion

In addition to InfluxDB, Glances can export data to ~20 services. So far, it is the best tool to log, track and view historical CPU, RAM, network and disk usage in macOS. The same method works for Linux and Windows as well.

Timestamp to Relative Time - Kibana Scripted fields

Anand Reddy Pandikunta — Wed, 10 Apr 2024 08:45:33 GMT

When browsing logs in Kibana, there will be a timestamp stamp field on the left for all the docs. It is difficult to read & comprehend the timestamp in the logs. It would be better if we can convert the timestamp to a human-readable relative time like 5 minutes ago, 1 hour ago, etc.

Kibana Scripted Fields

Kibana provides a feature called scripted fields to create new fields in the index pattern. We can use this feature to convert the timestamp to a relative time.

Go to Stack Management -> Index Patterns -> Create index pattern -> Select the index pattern -> Scripted fields, click on Add scripted field, add the below script.

long now = new Date().getTime();

long timestamp = doc['@timestamp'].value.toInstant().toEpochMilli();
long diff = now - timestamp;
if (diff > 7200000) {
  return Math.round(diff / 3600000) + " hours ago";
} else if (diff > 3600000) {
  return Math.round(diff / 3600000) + " hour ago";
} else if (diff > 120000) {
  return Math.round(diff / 60000) + " minutes ago";
} else if (diff > 60000) {
  return (Math.round(diff / 60000) + " minute ago");
} else {
  return Math.round(diff / 1000) + " seconds ago";
}

Once the field is saved, we can go back to Discover and see the new field in the logs. We can toggle the visibility of the Relative Time field to see the relative time.

Conclusion

Instead of looking at the timestamp and calculating the relative time in our head, we can use relative time in Kibana . This will make it easier to read & comprehend the logs.

Remotely Access Raspberry Pi or Any Device

Anand Reddy Pandikunta — Mon, 25 Sep 2023 01:49:54 GMT

I recently started using Raspberry Pi and I wanted to access it when I am outside of home as well. After trying out few solutions, I stumbled upon Tailscale¹.

Tailscale is a mesh VPN that makes it easy to connect out devices, wherever they are. It is free for personal use and supports all major platforms like Linux, Windows, Mac, Android, iOS, etc.

Installation

I installed tailscale on Raspberry Pi using the following command.

$ curl -fsSL https://tailscale.com/install.sh | sh

Setup

Once the installation is done, I run tailscale up to start the daemon. This opened a browser window and asked me to log in with email address. After I logged in, I can see all the devices in the tailscale dashboard.

tailscale has CLI tool as well and status can be viewed with the following command.

$ tailscale status
100.81.13.75   m1                    avilpage@  macOS   -
100.12.12.92   rpi1.tailscale.ts.net avilpage@  linux   offline

I also set up a cron job to start tailscale daemon on boot.

$ crontab -e
@reboot tailscale up

Access

Now I can access the device from anywhere using the tailscale IP address. For example, if the IP address is 100.34.2.23. I can ssh into the device using the following command.

$ ssh pi@100.81.12.92

It also provides DNS names for each device. For example, I can ssh into the device using the following command as well.

$ ssh pi@raspberry3.tailscale.net

Conclusion

Tailscale is a great tool to access devices remotely. It is easy to set up and works well with Raspberry Pi, Mac & Linux as well.

https://en.wikipedia.org/wiki/Tailscale ↩

Record Resource Usage of Single Process

Anand Reddy Pandikunta — Fri, 14 Apr 2023 00:48:37 GMT

Introduction

On Linux & Mac, we can use an inbuilt top command line tool to monitor the resource usage of a single process in real time.

# On Linux, for a given pid
$ top -p 1234

# On Mac, for a given pid
$ top -pid 1234

In this article, we will see how to record and plot resource usage of a single process using top and a Python package called psrecord¹.

Record Resource Usage

In some cases, we need to record the resource usage of a process to use it later. For example, we can use this data to find out the peak resource usage of a process. For this, we can use top to log resource usage into a text file.

# On Linux, for a given pid
$ top -p 1234 -b -d 1 > top.log

# On Mac, for a given pid
$ top -l 0 -s 1 -pid 32515 | awk 'NR%13==0; fflush(stdout)' > top.log

Once we have the log file, we can view the raw data or we can plot the resource usage by using tools like gnuplot or matplotlib.

Instead of using top command, we can use psrecord to record the resource usage of a process. psrecord is a Python package that can be installed all using pip.

$ python -m pip install psrecord

Once installed, we can use psrecord to record the resource usage of a process.

# record resource usage of a process with pid 1234
$ psrecord 1234 --log top.log

# start and record resource usage of a process
$ psrecord python script.py --plot graph.png

We can view the raw data in the log file.

# view raw data
$ head top.log
$ head a.txt
# Elapsed time   CPU (%)     Real (MB)   Virtual (MB)
       0.000        0.000        5.000   399461.438
       0.000       93.700        5.000   399461.438
       0.000       96.300        5.000   399461.438
       0.000       91.900        5.000   399461.438

Here is the generated graph.

Conclusion

In this article, we have seen how to record and plot resource usage of a single process using top(inbuilt tool), psrecord(3rd party package).

https://pypi.org/project/psrecord/ ↩

Setup Kubernetes Anywhere with Single Command

Anand Reddy Pandikunta — Fri, 03 Mar 2023 21:25:27 GMT

Introduction

In an earlier article, we have seen how to set up Kubernetes on M1 Mac. That involved spinning up a VM and installing Kubernetes¹ on it. In this article, we will see how to set up Kubernetes directly on Docker so that we can use the same set-up on any operating system.

Prerequisites

Ensure you have Docker installed on your system. If you are on a Mac or Windows, you can install Docker Desktop².

k3s/k3d

k3s³ is a lightweight Kubernetes distribution by Rancher. It is a single binary that can be run on any Linux machine. But it doesn't work on Mac or Windows.

k3d⁴ is a wrapper around k3s that allows you to run k3s on Docker. It is a great option for running Kubernetes on your local machine.

Installation

k3d can be installed using the following command:

$ brew install k3d  # mac
$ chocolatey install k3d  # windows
$ curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash # linux

Once it is installed, we can create a cluster using the following command:

$ k3d cluster create demo

This will launch a cluster with a single node. We can also setup a multi-node cluster using the following command:

$ k3d cluster create demo --servers 3 --agents 2

We can verify the cluster is up and running using the following command:

$ kubectl get nodes

We can also use GUI tools like Lens to manage and navigate the cluster. In the above video we have used Lens to create a Jenkins deployment as well.

Conclusion

In this article, we have seen how to set up Kubernetes on Docker. This is a great option for running Kubernetes on your local machine. We can also use this to run production setup for small applications.

Speed Up AMD64(Intel) VMs on ARM(M1 Mac) Host

Anand Reddy Pandikunta — Thu, 20 Oct 2022 17:08:56 GMT

Introduction

From 2020, Apple has transitioned from Intel to ARM based Apple Silicon M1. If we run uname -mp on these devices, we can see the CPU architecture details.

$ uname -mp
arm64 arm

Let's run the same command on a device using Intel x86 processor.

$ uname -mp
x86_64 x86_64

Many popular docker images¹ doesn't have ARM64 support yet. When setting up a dev environment in M1 Mac, there are high chances that we stumble on these containers if we are using plain docker or ARM64 VM. So, there is a need to spin up x86_64 VMs.

In this article, lets see how the performance affects when running a cross architecture containers and how to speed it up.

Setup

Lima² can run foreign architecture(x6_64) VMs on Mac. Let's install lima, start a AMD64 VM & ARM64 VM and install k3s[^k3s] in them. k3s will run multiple process in the background and let's see how resource consumption varies in these VMs.

$ brew install lima

$ limactl start linux_arm64
$ limactl start linux_amd64

When starting a VM, we can edit arch parameter in the configuration. Once VM starts, we can see the details by running limactl list.

$ limactl list
NAME                ARCH
linux_amd64         x86_64
linux_arm64         aarch64

Lets login to each VM & install k3s.

$ limactl shell linux_arm64

$ curl -sfL https://get.k3s.io | sh -

$ limactl shell linux_amd64

$ curl -sfL https://get.k3s.io | sh -

If we look at resource consumption on the host machine, x86_84 VM is using way more resources than ARM64 VM. This is because of the emulation layer that is running on top of the VM.

We can login to individual VMs, run top to see the load average as well.

Fast Mode

lima provides fast-mode option for cross architecture VMs which will speed up the performance.

For that, we need to log in to VMs and install emulators.

$ sudo systemctl start containerd
$ sudo nerdctl run --privileged --rm tonistiigi/binfmt --install all

After that we can restart the VMs and monitor the resource consumption. On an average, we can see that the resource consumption is reduced by 50%.

Conclusion

In this article, we saw how to run cross architecture VMs on M1 Mac and how to speed up the performance. We can use this technique to run cross-architecture containers on Linux as well.

Local Kubernetes Cluster with K3s on MacBook M1

Anand Reddy Pandikunta — Tue, 11 Oct 2022 02:38:36 GMT

Update: This article is outdated. I wrote an updated article on setting up k8s anywhere with single command.

Introduction

Kubernetes(k8s)¹ is an open-source system for managing large scale containerized applications. K3s² is lightweight K8s in a single binary file. However, K3s won't work directly on Macbook as it needs systemd/OpenRC.

$ curl -sfL https://get.k3s.io | sh -

[ERROR]  Can not find systemd or openrc to use as a process supervisor for k3s

To setup k8s/k3s on Mac, we need to setup a Linux layer on top of Mac. An easy way to spin up Linux VMs on Macbook M1(Apple Silicon) is to use multipass³. In this article, lets see how to setup K3s on Mac using multipass

K3s Setup

Install multipass with brew by running the following command.

$ brew install --cask multipass

Once it is installed, spin up a new VM by specifying memory and disk space.

$ multipass launch --name k3s --mem 4G --disk 40G

Once VM is launched, we can see VM details.

$ multipass info k3s
Name:           k3s
State:          Running
IPv4:           192.168.64.4
                10.42.0.0
                10.42.0.1
Release:        Ubuntu 22.04.1 LTS
Image hash:     78b5ca0da456 (Ubuntu 22.04 LTS)
Load:           1.34 2.10 1.70
Disk usage:     3.7G out of 38.6G
Memory usage:   1.2G out of 3.8G
Mounts:         /Users/chillaranand/test/k8s => ~/k8s
                    UID map: 503:default
                    GID map: 20:default

We can even mount Mac directories on the VM.

$ multipass mount ~/test/k8s k3s:~/k8s

This will be useful when we are making changes on host directories and want to apply changes on the cluster which is inside VM.

Now, we can install k3s by running the install script inside the VM.

$ multipass shell k3s

ubuntu@k3s:~$ curl -sfL https://get.k3s.io | sh -

This will setup a k3s cluster on the VM. We can use kubectl and deploy applications on this cluster.

By default, k3s config file will be located at /etc/rancher/k3s/k3s.yaml. With this config file, we can use Lens⁴ to manage k8s cluster.

Lets find out IP of the VM & k8s token so that we can spin up a new VM and add it to this cluster.

# get token & ip of k3s
$ multipass exec k3s sudo cat /var/lib/rancher/k3s/server/node-token
$ multipass info k3s | grep -i ip

$ multipass launch --name k3s-worker --mem 2G --disk 20G

$ multipass shell k3s-worker

ubuntu@k3s-worker:~$ curl -sfL https://get.k3s.io | K3S_URL=https://192.168.64.4:6443 K3S_TOKEN="hs48af...947fh4::server:3tfkwjd...4jed73" sh -

We can verify if the node is added correctly from k3s VM.

ubuntu@k3s:~$ kubectl get nodes
NAME         STATUS   ROLES                  AGE     VERSION
k3s          Ready    control-plane,master   15h     v1.24.6+k3s1
k3s-worker   Ready    <none>                 7m15s   v1.24.6+k3s1

Once we are done with experimenting k3s, we can delete the VMs.

$ multipass delete k3s k3s-worker
$ multipass purge

Conclusion

multipass is a great tool to spin up Linux VMs on Mac with single command. K3s is better tool to setup k8s cluster locally for development and testing.

Even though we have mentioned this tutorial is meant for Mac M1, it should work fine on any Linux distribution as well.

Make Python Docker Builds Slim & Fast

Anand Reddy Pandikunta — Sat, 31 Oct 2020 12:53:13 GMT

Introduction

When using Docker, if the build is taking time or the build image is huge, it will waste system resources as well as our time. In this article, let's see how to reduce build time as well as image size when using Docker for Python projects.

Project

Let us take a hello world application written in flask.

import flask


app = flask.Flask(__name__)


@app.route('/')
def home():
    return 'hello world - v1.0.0'

Let's create a requirements.txt file to list out python packages required for the project.

flask==1.1.2
pandas==1.1.2

Pandas binary wheel size is ~10MB. It is included in requirements to see how python packages affect docker image size.

Here is our Dockerfile to run the flask application.

FROM python:3.7

ADD . /app

WORKDIR /app

RUN python -m pip install -r requirements.txt

EXPOSE 5000

ENTRYPOINT [ "python" ]

CMD [ "-m" "flask" "run" ]

Let's use the following commands to measure the image size & build time with/without cache.

$ docker build . -t flask:0.0 --pull --no-cache
[+] Building 45.3s (9/9) FINISHED

$ touch app.py  # modify app.py file

$ docker build . -t flask:0.1
[+] Building 15.3s (9/9) FINISHED

$ docker images | grep flask
flask               0.1     06d3e985f12e    1.01GB

With the current docker, here are the results.

1. Install requirements first

FROM python:3.7

WORKDIR /app

ADD ./requirements.txt /app/requirements.txt

RUN python -m pip install -r requirements.txt

ADD . /app

EXPOSE 5000

ENTRYPOINT [ "python" ]

CMD [ "-m" "flask" "run" ]

Let us modify the docker file to install requirements first and then add code to the docker image.

Now, build without cache took almost the same time. With cache, the build is completed in a second. Since docker caches step by step, it has cached python package installation step and thereby reducing the build time.

2. Disable Cache

FROM python:3.7

WORKDIR /app

ADD ./requirements.txt /app/requirements.txt

RUN python -m pip install -r requirements.txt --no-cache

ADD . /app

EXPOSE 5000

ENTRYPOINT [ "python" ]

CMD [ "-m" "flask" "run" ]

By default, pip will cache the downloaded packages. Since we don't need a cache inside docker, let's disable pip cache by passing --no-cache argument.

This reduced the docker image size by ~20MB. In real-world projects, where there are a good number of dependencies, the overall image size will be reduced a lot.

3. Use slim variant

Till now, we have been using defacto Python variant. It has a large number of common debian packages. There is a slim variant that doesn't contain all these common packages⁴. Since we don't need all these debian packages, let's use slim variant.

FROM python:3.7-slim

...

This reduced the docker image size by ~750 MB without affecting the build time.

4. Build from source

Python packages can be installed via wheels (.whl files) for a faster and smoother installation. We can also install them via source code. If we look at Pandas project files on PyPi¹, it provides both wheels as well as tar zip source files. Pip will prefer wheels over source code the installation process will be much smoother.

To reduce Docker image size, we can build from source instead of using the wheel. This will increase build time as the python package will take some time to compile while building.

Here build size is reduced by ~20MB but the build has increased to 15 minutes.

5. Use Alpine

Earlier we have used, python slim variant the base image. However, there is Alpine variant which is much smaller than slim. One caveat with using alpine is Python wheels won't work with this image².

We have to build all packages from source. For example, packages like TensorFlow provide only wheels for installation. To install this on Alpine, we have to install from the source which will take additional effort to figure out dependencies and install.

Using Alpine will reduce the image size by ~70 MB but it is not recomended to use Alpine as wheels won't work with this image.

All the docker files used in the article are available on github³.

Conclusion

We have started with a docker build of 1.01 GB and reduced it to 0.13 GB. We have also optimized build times using the docker caching mechanism. We can use appropriate steps to optimize build for size or speed or both.

How To Deploy Mirth Connect To Kubernetes

Anand Reddy Pandikunta — Wed, 30 Sep 2020 17:55:46 GMT

Introduction

NextGen Connect(previously Mirth Connect) is widely used integration engine for information exchange in health-care domain. In this article, let us see how to deploy Mirth Connect to a Kubernetes cluster.

Deployment To k8s

From version 3.8, NextGen has started providing official docker images for Connect¹. By default, Connect docker exposes 8080, 8443 ports. We can start a Connect instance locally, by running the following command.

$docker run -p 8080:8080 -p 8443:8443 nextgenhealthcare/connect

We can use this docker image and create a k8s deployment to start a container.

---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: mirth-connect
  namespace: default
spec:
  template:
    spec:
      containers:
      - name: mirth-connect
        image: docker.io/nextgenhealthcare/connect
        ports:
        - name: http
          containerPort: 8080
        - name: https
          containerPort: 8443
        - name: hl7-test
          containerPort: 9001
        env:
          - name: DATABASE
            value: postgres
          - name: DATABASE_URL
            value: jdbc:postgresql://avilpage.com:5432/mirth_db

This deployment file can be applied on a cluster using kubectl.

$ kubectl apply -f connect-deployment.yaml

To access this container, we can create a service to expose this deployment to public.

---
apiVersion: v1
kind: Service
metadata:
  name: mirth-connect
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:ap-south-1:foo
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    external-dns.alpha.kubernetes.io/hostname: connect.avilpage.com
spec:
  type: LoadBalancer
  selector:
    app: mirth-connect
  ports:
    - name: http
      port: 80
      targetPort: 8080
      protocol: TCP
    - name: https
      port: 443
      targetPort: 8443
      protocol: TCP
    - name: hl7-test
      port: 9001
      targetPort: 9001
      protocol: TCP

This will create a load balancer in AWS through which we can access mirth connect instance. If an ingress controller is present in the cluster, we can use it directly instead of using a seperate load balancer for this service.

Once Mirth Connect is up & running, we might have to create HL7 channels running on various ports. In the above configuration files, we have exposed 9001 HL7 port for testing of channel. Once we configure Mirth Channels, we need to expose appropriate ports in deployment as well as service similiar to this.

Conclusion

Earlier, there were no official docker images for Mirth Connect and it was bit diffucult to dockerize Mirth Connect and deploy it. With the release of official Docker images, deploying Mirth Connect to k8s or any other container orchestration platform has become much easier.

https://hub.docker.com/r/nextgenhealthcare/connect/ ↩

Tips On Improving kubectl Productivity

Anand Reddy Pandikunta — Sun, 31 May 2020 13:12:00 GMT

kubectl is CLI tool to control Kubernetes clusters. As we start using kubectl to interact with mutliple clusters, we end up running lengthy commands and even running multiple commands for simple tasks like running a shell in a container.

In this article, lets learn few tips to improve our productivity when using kubectl.

Aliases

Aliases in general improve the productivity when using a shell.

kubectl provides shortcuts for commands. For example,

# instead of running full command
$ kubectl get services

# we can use short hand version
$ kubectl get svc

It also provides completion for commands.

# enable completion for zsh
$ source <(kubectl completion zsh)

# type `kubectl ` and hit `<TAB>` will show possible options
$ kubectl
annotate       attach         cluster-info
api-resources  auth           completion
api-versions   autoscale      config
apply          certificate    convert

# type `kubectl g`, and hit `<TAB>` will show possible options
$ kubectl get

Even though completions are helpful, setting up up aliases for most commanly used commands will save a lot of time.

alias k='kubectl'

alias kdp='kubectl describe pod'
alias kgp='kubectl get pods'
alias kgpa='kubectl get pods --all-namespaces'
alias ket='kubectl exec -it'
alias wkgp='watch -n1 kubectl getp pods'

alias kga='kubectl get all'
alias kgaa='kubectl get all --all-namespaces'

alias kaf='kubectl apply -f'

alias kcgc='kubectl config get-contexts'
alias kccc='kubectl config current-context'

If you don't write your own aliases, there is kubectl-aliases which provides exhuastive list of aliases. We can source this file in rc file and start using them.

Use Functions

Even though aliases help us to run lengthy commands with an alias, there are times where we have to run multiple commands to get things done for a single task.

For example, to view kubenetes dashboard, we have to get the token, start proxy server and then open the url in browser. We can write a simple function as shown below to do all of that.

kp() {
    kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}') | grep 'token:' | awk '{print $2}' | pbcopy
    open http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
    kubectl proxy
}

Now from the shell, when we run kp, it function will copy the token to clipboard, open kubernetes dashboard in browser and will start the proxy server.

Use Labels

To describe a pod or tail logs from a pod, we can use pod names.

$ kubectl get pods
NAME                             READY   STATUS
hello-world-79d794c659-tpfv2     1/1     Running


$ kubectl describe pod hello-world-79d794c659-tpfv2

$ kubectl logs -f pod/hello-world-79d794c659-tpfv2

When the app gets updated, the name of pod also updates. If your shell has auto completion feature, it will autocomplete to previous name.

So, instead of using pod name, we can use pod labels as mentioned below.

$ kubectl describe pod -l=hello-world

$ kubectl logs -f -l=pod/hello-world

We run the command once and next time shell will show autocomplete and we can use that directly.

Kubectl Tools

k8s has a good ecosystem and the following packages are aimed to make certain k8s tasks easier.

kubectl-debug - Debug pod by a new container with all troubleshooting tools pre-installed.

kube-forwarder - Easy to use port forwarding manager.

stern - Multi pod and container log tailing.

kubectx - Quick way to switch between clusters and namespaces.

kubebox - Terminal and Web console for Kubernetes.

k9s - Interactive terminal UI.

kui - Hybrid CLI/UI tool for k8s.

click - Interactive controller for k8s.

lens - Stand alone corss platform k8s IDE.

Conclusion

In this article we have seen some useful methods as well as some tools to improve productivity with kubectl. If you spend a lot of time interacting with kubernetes cluster, it is important to notice your workflows and find better tools or ways to improve productivity.

Continuous Deployment To Kubernetes With Skaffold

Anand Reddy Pandikunta — Thu, 30 Apr 2020 11:15:36 GMT

In this article, let us see how to set up a continuous deployment pipeline to Kubernetes in CircleCI using Skaffold.

Prerequisites

You should have a kubernetes cluster in a cloud environment or in your local machine. Check your cluster status with the following commands.

$ kubectl cluster-info
$ kubectl config get-contexts

You should know how to manually deploy your application to kubernetes.

# push latest docker image to container registry
$ docker push chillaranand/library

# deploy latest image to k8s
$ kubectl apply -f app/deployment.yaml
$ kubectl apply -f app/service.yaml

Skaffold

Skaffold is a CLI tool to facilitate continuous development and deployment workflows for Kubernetes applications.

Skaffold binaries are available for all platforms. Download the binary file for your OS and move it to bin folder.

$ curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-darwin-amd64
$ chmod +x skaffold
$ sudo mv skaffold /usr/local/bin

Inside your project root, run init command to generate a config file. If your project has k8s manifests, it will detect them and include it in the configuration file.

$ skaffold init
Configuration skaffold.yaml was written

$ cat skaffold.yaml
apiVersion: skaffold/v2beta1
kind: Config
metadata:
  name: library
build:
  artifacts:
  - image: docker.io/chillaranand/library
deploy:
  kubectl:
    manifests:
    - kubernetes/deployment.yaml
    - kubernetes/service.yaml

To deploy latest changes to your cluster, run

$ skaffold run

This will build the docker image, push to registry and will apply the manifests in the clusters. Now, k8s will pull the latest image from the registry and create a new deployment.

CircleCI Workflow

version: 2.1

orbs:
  aws-cli: circleci/aws-cli@0.1.19
  kubernetes: circleci/kubernetes@0.11.0

commands:
  kubernetes-deploy:

    steps:
      - setup_remote_docker

      - aws-cli/setup:
          profile-name: default

      - kubernetes/install-kubectl:
          kubectl-version: v1.15.10

      - checkout

      - run:
          name: container registry log in
          command: |
            sudo $(aws ecr get-login --region ap-south-1 --no-include-email)

      - run:
          name: install skaffold
          command: |
            curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64
            chmod +x skaffold
            sudo mv skaffold /usr/local/bin

      - run:
          name: update kube config to connect to the required cluster
          command: |
            aws eks --region ap-south-1 update-kubeconfig --name demo-cluster

      - run:
          name: deploy to k8s
          command: |
            skaffold run

CircleCI orbs are shareable packages to speed up CI setup. Here we are using aws-cli, kubernetes orbs to easily install/setup them inside the CI environment.

Since CircleCI builds run in a docker container, to run docker commands inside container, we have to specify setup_remote_docker key so that a separate environment is created for it.

Remaining steps are self explainatory.

Conclusion

Here we have seen how to set up CD to kubernetes in CircleCI. If we want to set up this another CI like Jenkins or Travis, instead of using orbs, we have to use system package mangers like apt-get to install them. All others steps will remain same.

Versioning & Retrieving Files From AWS S3 With Boto

Anand Reddy Pandikunta — Wed, 24 Jul 2019 15:51:21 GMT

Introduction

Amazon S3 (Amazon Simple Storage Service) is an object storage service offered by Amazon Web Services. For S3 buckets, if versioning is enabled, users can preserve, retrieve, and restore every version of the object stored in the bucket.

In this article, we will understand how to enable versioning for a bucket and retrieve all versions of an object from AWS web interface as well as Python boto library.

Versioning of Bucket

Bucket versioning can be changed with a toggle button from the AWS web console in the bucket properties.

We can do the same with Python boto3 library.

import boto3


bucket_name = 'avilpage'

s3 = boto3.resource('s3')
versioning = s3.BucketVersioning(bucket_name)

# check status
print(versioning.status)

# enable versioning
versioning.enable()

# disable versioning
versioning.suspend()

Retrieving Objects

Once versioning is enabled, we can store multiple versions of an object by uploading an object multiple times with the same key.

We can write a simple script to generate a text file with a random text and upload it to S3.

import random
import string

import boto3

file_name = 'test.txt'
key = file_name
s3 = boto3.client('s3')

with open(file_name, 'w') as fh:
    data = ''.join(random.choice(string.ascii_letters) for _ in range(10))
    fh.write(data)

s3.upload_file(key, bucket_name, file_name)

If this script is executed multiple times, the same file gets overridden with a different version id with the same key in the bucket.

We can see all the versions of the file from the bucket by selecting the file and then clicking drop-down at Latest version.

We can write a script to retrieve and show contents of all the versions of the test.txt file with the following script.

import boto3


bucket_name = 'avilpage'
s3_client = boto3.client('s3')

versions = s3_client.list_object_versions(Bucket=bucket_name)

for version in versions:
    version_id = versions['Versions'][0]['VersionId']
    file_key = versions['Versions'][0]['Key']

    response = s3.get_object(
        Bucket=bucket_name,
        Key=file_key,
        VersionId=version_id,
    )
    data = response['Body'].read()
    print(data)

Conclusion

Object versioning is useful to protect data from unintended overwrites. In this article, we learnt how to change bucket versioning, upload multiple versions of same file and retrieving all versions of the file using AWS web console as well as boto3.

How To Install Private Python Packages With Pip

Anand Reddy Pandikunta — Thu, 31 Jan 2019 06:42:12 GMT

Introduction

To distribute python code, we need to package it and host it somewhere, so that users can install and use it. If the code is public, it can be published to PyPi or any public repository, so that anyone can access it. If the code is private, we need to provide proper authentication mechanism before allowing users to access it.

In this article, we will see how to use pip to install Python packages hosted on GitLab, GitHub, Bitbucket or any other services.

Packaging

To package python project, we need to create setup.py file which is build script for setuptools. Below is a sample setup file to create a package named library.

import setuptools


with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="library",
    version="0.0.1",
    author="chillaranand",
    author_email="foo@avilpage.com",
    description="A simple python package",
    long_description=long_description,
    url="https://github.com/chillaranand/library",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
)

Python provides detailed packaging documentation on structuring and building the package.

Installation

Once module(s) is packaged and pushed to hosting service, it can be installed with pip.

# using https
$ pip install git+https://github.com/chillaranand/library.git

# using ssh
pip install git+ssh://git@github.com/chillaranand/library.git

This usually requires authentication with usersname/password or ssh key. This setup works for developement machines. To use it in CI/CD pipelines or as a dependency, we can use tokens to simplify installation.

$ export GITHUB_TOKEN=foobar

$ pip install git+https://$GITHUB_TOKEN@github.com/chillaranand/library.git

Conclusion

In this article, we have seen how to package python code and install private packages with pip. This makes it easy to manage dependencies or install packages on multiple machines.

Setup Continous Deployment For Python Chalice

Anand Reddy Pandikunta — Tue, 30 Oct 2018 15:51:21 GMT

Outline

Chalice is a microframework developed by Amazon for quickly creating and deploying serverless applications in Python.

In this article, we will see how to setup continous deployment with GitHub and AWS CodePipeline.

CD Setup

Chalice provides cli command deploy to deploy from local system.

Chalice also provides cli command generate-pipeline command to generate CloudFormation template. This template is useful to automatically generate several resources required for AWS pipeline.

This by default uses CodeCommit repository for hosting code. We can use GitHub repo as a source instead of CodeCommit.

Chalice by default provides a build file to package code and push it to S3. In the deploy step, it uses this artifact to deploy the code.

We can use a custom buildpsec file to directly deploy the code from build step.

version: 0.1

phases:
  install:
    commands:
      - echo Entering the install phase
      - echo Installing dependencies
      - sudo pip install --upgrade awscli
      - aws --version
      - sudo pip install chalice
      - sudo pip install -r requirements.txt

  build:
    commands:
      - echo entered the build phase
      - echo Build started on `date`
      - chalice deploy --stage staging

This buildspec file install requirements and deploys chalice app to staging. We can add one more build step to deploy it production after manual intervention.

Conclusion

We have seen how to setup continous deployment for chalice application with GitHub and AWS CodePipeline.

How To Deploy Django Channels To Production

Anand Reddy Pandikunta — Fri, 18 May 2018 15:51:21 GMT

In this article, we will see how to deploy django channels to production and how we can scale it to handle more load. We will be using nginx as proxy server, daphne as ASGI server, gunicorn as WSGI server and redis for channel back-end.

Daphne can serve HTTP requests as well as WebSocket requests. For stability and performance, we will use uwsgi/gunicorn to serve HTTP requests and daphne to serve websocket requests.

We will be using systemd to create and manage processes instead of depending on third party process managers like supervisor or circus. We will be using ansible for managing deployments. If you don't want to use ansible, you can just replace template variables in the following files with actual values.

Nginx Setup

Nginx will be routing requests to WSGI server and ASGI server based on URL. Here is nginx configuration for server.

server {
    listen {{ server_name }}:80;
    server_name {{ server_name }} www.{{ server_name }};

    return 301 https://avilpage.com$request_uri;
}


server {
    listen {{ server_name }}:443 ssl;
    server_name {{ server_name }} www.{{ server_name }};

    ssl_certificate     /root/certs/avilpage.com.chain.crt;
    ssl_certificate_key /root/certs/avilpage.com.key;

    access_log /var/log/nginx/avilpage.com.access.log;
    error_log /var/log/nginx/avilpage.com.error.log;

    location / {
            proxy_pass http://0.0.0.0:8000;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_redirect off;
    }

    location /ws/ {
            proxy_pass http://0.0.0.0:9000;
            proxy_http_version 1.1;

            proxy_read_timeout 86400;
            proxy_redirect     off;

            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host $server_name;
    }

    location /static {
        alias {{ project_root }}/static;
    }

    location  /favicon.ico {
        alias {{ project_root }}//static/img/favicon.ico;
    }

    location  /robots.txt {
        alias {{ project_root }}/static/txt/robots.txt;
    }

}

WSGI Server Setup

We will use gunicorn for wsgi server. We can run gunicorn with

$ gunicorn avilpage.wsgi --bind 0.0.0.0:8000 --log-level error --log-file=- --settings avilpage.production_settings

We can create a systemd unit file to make it as a service.

[Unit]
Description=gunicorn
After=network.target


[Service]
PIDFile=/run/gunicorn/pid
User=root
Group=root
WorkingDirectory={{ project_root }}
Environment="DJANGO_SETTINGS_MODULE={{ project_name }}.production_settings"
ExecStart={{ venv_bin }}/gunicorn {{ project_name}}.wsgi --bind 0.0.0.0:8000 --log-level error --log-file=- --workers 5 --preload


ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
Restart=on-abort
PrivateTmp=true


[Install]
WantedBy=multi-user.target

Whenever server restarts, systemd will automatically start gunicorn service. We can also restart gunicorn manually with

$ sudo service gunicorn restart

ASGI Server Setup

We will use daphne for ASGI server and it can be started with

$ daphne avilpage.asgi:application --bind 0.0.0.0 --port 9000 --verbosity 1

We can create a systemd unit file like the previous one to create a service.

[Unit]
Description=daphne daemon
After=network.target


[Service]
PIDFile=/run/daphne/pid
User=root
Group=root
WorkingDirectory={{ project_root }}
Environment="DJANGO_SETTINGS_MODULE={{ project_name }}.production_settings"
ExecStart={{ venv_bin }}/daphne --bind 0.0.0.0 --port 9000 --verbosity 0 {{project_name}}.asgi:application
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
Restart=on-abort
PrivateTmp=true


[Install]
WantedBy=multi-user.target

Deployment

Here is an ansible playbook which is used to deploy these config files to our server. To run the playbook on server avilpage.com, execute

$ ansible-playbook -i avilpage.com, django_setup.yml

Scaling

Now that we have deployed channels to production, we can do performance test to see how our server performs under load.

For WebSockets, we can use Thor to run performance test.

thor -C 100 -A 1000 wss://avilpage.com/ws/books/

Our server is able to handle 100 requests per second with a latency of 800ms. This is good enough for low traffic website.

To improve performance, we can use unix sockets instead of rip/port for gunicorn and daphne. Also, daphne has support for multiprocessing using shared file descriptors. Unfortunately, it doesn't work as expected. As mentioned here, we can use systemd templates and spawn multiple daphne process.

An alternate way is to use uvicorn to start multiple workers. Install uvicorn using pip

$ pip install uvicorn

Start uvicorn ASGI server with

$ uvicorn avilpage.asgi --log-level critical --workers 4

This will spin up 4 workers which should be able to handle more load. If this performance is not sufficient, we have to setup a load balancer and spin up multiple servers(just like scaling any other web application).