Avil Page (Posts about DevOps)

Autoscale k8s pods with queue size (KEDA)

Anand Reddy Pandikunta — Thu, 27 Mar 2025 06:07:52 GMT

Introduction

Kubernetes(k8s) is a popular container orchestration tool, and it provides Horizontal Pod Autoscaler(HPA) to scale pods based on CPU and Memory usage.

To scale pods based on the queue size we can use KEDA.

Setup

For this demo, we will use PostgreSQL table to store tasks and KEDA to scale the pods based on the queue size.

As written earlier, we can use k3d and setup k8s cluster with a single command anywhere.

Once the cluster is up, we can set up a simple shell script to produce and consume tasks from the PostgreSQL table.

I am adding only relevant snippets here. You can find the complete code in the GitHub repo.

PostgreSQL deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-deployment
spec:
  template:
    spec:
      containers:
        - name: postgres
          image: postgres:13

Producer shell script:

#!/bin/bash

psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" <<-EOSQL
  CREATE TABLE IF NOT EXISTS tasks (
      id SERIAL PRIMARY KEY,
      description TEXT NOT NULL,
      status TEXT NOT NULL
  );
EOSQL

# Continuously insert tasks
while true; do
    psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" <<-EOSQL
    INSERT INTO tasks (description, status)
    VALUES ('$DESCRIPTION', '$STATUS');
done

Producer dockerfile:

FROM postgres:13

RUN apt-get update && apt-get install -y bash curl

COPY producer.sh /producer.sh

RUN chmod +x /producer.sh

CMD ["/producer.sh"]

Producer deployment:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: task-producer
          image: task-producer
          imagePullPolicy: Never

Consumer shell script:

#!/bin/bash

TASK=$(psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" -t <<-EOSQL
  SELECT id, description, status FROM tasks
  WHERE status != 'Completed'
  ORDER BY id
  LIMIT 1;
EOSQL

TASK_ID=$(echo "$TASK" | awk '{print $1}')

psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" <<-EOSQL
  UPDATE tasks
  SET status = 'Completed'
  WHERE id = $TASK_ID;
EOSQL

Consumer dockerfile:

FROM postgres:13

RUN apt-get update && apt-get install -y bash

COPY consumer.sh /consumer.sh

RUN chmod +x /consumer.sh

CMD ["/consumer.sh"]

Consumer deployment:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: task-consumer
          image: task-consumer
          imagePullPolicy: Never

Instead of pushing these images to any container registry, we can directly load the images into cluster using k3d image.

k3d image import task-producer --cluster demo-cluster
k3d image import task-consumer --cluster demo-cluster

After that, we can deploy postgres, consumer and producer using the above deployment files.

Once the deployment is done, we can monitor the tasks from pod logs.

Lets install KEDA in the cluster and setup a scaled-object to scale the consumer pods based on number of pending tasks.

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda

ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: task-consumer-scaler
  namespace: default
spec:
  scaleTargetRef:
    name: task-consumer  
  pollingInterval: 1    
  cooldownPeriod: 1     
  minReplicaCount: 0    
  maxReplicaCount: 50   
  triggers:
    - type: postgresql
      metadata:
        host: postgres-service  
        query: "SELECT COUNT(*) FROM tasks WHERE status != 'Completed';"
        targetQueryValue: "10"

With this, KEDA will scale the consumer pods based on the number of tasks in the PostgreSQL table. The targetQueryValue is set to 10, which means if there are more than 10 tasks in the table, KEDA will scale up the consumer pods.

In the cluster, after a while, a bunch of tasks will be created in the PostgreSQL table. Now, we can set the status of all tasks to empty so that we can see auto-scaling in action.

kubectl exec -it task-producer-<pod-id> -- psql -h "$DB_HOST" -U "$DB_USER" -d "$DB_NAME" <<-EOSQL
  UPDATE tasks
  SET status = '';
EOSQL

On k8s dashboard, we can see the consumer pods scaling up and down based on the number of tasks in the PostgreSQL table.

Conclusion

KEDA is a powerful tool to scale k8s pods based on custom metrics. In this post, we have used PostgreSQL but we can use KEDA with variety of other data stores as well. Full code for this post is available here.

Free DockerHub Alternative - ECR Public Gallery

Anand Reddy Pandikunta — Sun, 09 Feb 2025 16:08:34 GMT

DockerHub started rate limiting¹ anonymous docker pulls. When testing out a new CI/CD setup, I hit the rate limit and had to wait for an hour to pull the image. This was a good time to look for alternatives.

AWS ECR Public Gallery² is a good alternative to DockerHub as of today(2025 Feb). It is free and does not have rate limits even for anonymous users.

Once we find the required image from the gallery, we can simply change the image name in the docker pull command to pull the image from ECR Gallery.

docker pull public.ecr.aws/ubuntu/ubuntu

In Dockerfile, we can use the image from ECR Gallery as follows:

FROM public.ecr.aws/ubuntu/ubuntu

That is a quick way to avoid DockerHub rate limits.

Install Cockpit on Remote Linux VM

Anand Reddy Pandikunta — Mon, 30 Dec 2024 22:54:07 GMT

Introduction

Cockpit is an easy to use web-based interface(like a cPanel) for managing Linux servers. When we want to provide access to non-developers or people who are new to linux, it is a good idea to get them started with Cockpit. It provides a user-friendly interface to manage services, containers, storage, logs, and more.

Setup

Let's create a new Ubuntu VM and install Cockpit on it.

sudo apt update
. /etc/os-release
sudo apt install -t ${VERSION_CODENAME}-backports cockpit

Once the installation is complete, we can get the public ip of the VM and access the Cockpit web interface running on port 9090.

It will be difficult to remember the public ip of the VM. So, let's create a DNS record for the VM. Let's add an A record in DNS settings to point cockpit.avilpage.com to the public ip of the VM.

Reverse Proxy

Let's set up a reverse proxy to access the Cockpit web interface using a subdomain.

sudo apt install caddy

Add the below configuration to /etc/caddy/Caddyfile.

cockpit.avilpage.com {
    reverse_proxy localhost:9090
}

We need Origins to Cockpit configuration at /etc/cockpit/cockpit.conf to allow requests from the subdomain.

[WebService]
Origins = https://cockpit.avilpage.com

Restart both services and open https://cockpit.avilpage.com in browser.

sudo systemctl restart cockpit
sudo systemctl restart caddy

Conclusion

Cockpit web UI is a great tool to manage Linux servers even for non-developers. Users can browse/manage logs, services, etc. It also provides a terminal to run commands on the server

Headlamp - k8s Lens open source alternative

Anand Reddy Pandikunta — Sun, 23 Jun 2024 20:18:02 GMT

Since Lens is not open source, I tried out monokle, octant, k9s, and headlamp¹. Among them, headlamp UI & features are closest to Lens.

Headlamp

Headlamp is CNCF sandbox project that provides cross-platform desktop application to manage Kubernetes clusters. It auto-detects clusters and provides cluster wide resource usage by default.

It can also be installed inside the cluster and can be accessed using a web browser. This is useful when we want to access the cluster from a mobile device.

$ helm repo add headlamp https://headlamp-k8s.github.io/headlamp/

$ helm install headlamp headlamp/headlamp

Lets port-forward the service & copy the token to access it.

$ kubectl create token headlamp

# we can do this via headlamp UI as well
$ kubectl port-forward service/headlamp 8080:80

Now, we can access the headlamp UI at http://localhost:8080.

Conclusion

If you are looking for an open source alternative to Lens, headlamp is a good choice. It provides a similar UI & features as Lens, and it is accessible via mobile devices as well.

https://headlamp.dev/ ↩

macOS - Log & track historical CPU, RAM usage

Anand Reddy Pandikunta — Fri, 31 May 2024 20:18:02 GMT

In macOS, we can use inbuilt Activity Monitor or third party apps like Stats to check the live CPU/RAM usage. But, we can't track the historical CPU & memory usage. sar, atop can track the historical CPU & memory usage. But, they are not available for macOS.

Netdata

Netdata¹ is an open source observability tool that can monitor CPU, RAM, network, disk usage. It can also track the historical data.

Unfortunately, it is not stable on macOS. I tried installing it on multiple macbooks, but it didn't work. I raised an issue² on their GitHub repository and the team mentioned that macOS is a low priority for them.

Glances

Glances³ is a cross-platform monitoring tool that can monitor CPU, RAM, network, disk usage. It can also track the historical data.

We can install it using Brew or pip.

$ brew install glances

$ pip install glances

Once it is installed, we can monitor the resource usage using the below command.

$ glances

Glances can log historical data to a file using the below command.

$ glances --export-csv /tmp/glances.csv

In addition to that, it can log data to services like influxdb, prometheus, etc.

Let's install influxdb and export stats to it.

$ brew install influxdb
$ brew services start influxdb
$ influx setup

$ python -m pip install influxdb-client

$ cat glances.conf
[influxdb]
host=localhost
port=8086
protocol=http
org=avilpage
bucket=glances
token=secret_token

$ glances --export-influxdb -C glances.conf

We can view stats in the influxdb from Data Explorer web UI at http://localhost:8086.

Glances provides a prebuilt Grafana dashboard⁴ that we can import to visualize the stats.

From Grafana -> Dashboard -> Import, we can import the dashboard using the above URL.

Conclusion

In addition to InfluxDB, Glances can export data to ~20 services. So far, it is the best tool to log, track and view historical CPU, RAM, network and disk usage in macOS. The same method works for Linux and Windows as well.

Record Resource Usage of Single Process

Anand Reddy Pandikunta — Fri, 14 Apr 2023 00:48:37 GMT

Introduction

On Linux & Mac, we can use an inbuilt top command line tool to monitor the resource usage of a single process in real time.

# On Linux, for a given pid
$ top -p 1234

# On Mac, for a given pid
$ top -pid 1234

In this article, we will see how to record and plot resource usage of a single process using top and a Python package called psrecord¹.

Record Resource Usage

In some cases, we need to record the resource usage of a process to use it later. For example, we can use this data to find out the peak resource usage of a process. For this, we can use top to log resource usage into a text file.

# On Linux, for a given pid
$ top -p 1234 -b -d 1 > top.log

# On Mac, for a given pid
$ top -l 0 -s 1 -pid 32515 | awk 'NR%13==0; fflush(stdout)' > top.log

Once we have the log file, we can view the raw data or we can plot the resource usage by using tools like gnuplot or matplotlib.

Instead of using top command, we can use psrecord to record the resource usage of a process. psrecord is a Python package that can be installed all using pip.

$ python -m pip install psrecord

Once installed, we can use psrecord to record the resource usage of a process.

# record resource usage of a process with pid 1234
$ psrecord 1234 --log top.log

# start and record resource usage of a process
$ psrecord python script.py --plot graph.png

We can view the raw data in the log file.

# view raw data
$ head top.log
$ head a.txt
# Elapsed time   CPU (%)     Real (MB)   Virtual (MB)
       0.000        0.000        5.000   399461.438
       0.000       93.700        5.000   399461.438
       0.000       96.300        5.000   399461.438
       0.000       91.900        5.000   399461.438

Here is the generated graph.

Conclusion

In this article, we have seen how to record and plot resource usage of a single process using top(inbuilt tool), psrecord(3rd party package).

https://pypi.org/project/psrecord/ ↩