Avil Page (Posts about python)

Python UV in 100 seconds

Anand Reddy Pandikunta — Wed, 30 Apr 2025 17:46:24 GMT

UV is a fast Python package manager that replaces tools like pip, pyenv, virtualenv, etc.

Install uv

# On macOS and Linux.
curl -LsSf https://astral.sh/uv/install.sh | sh
# On Windows.
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Installing Python

uv python list

uv python install 3.13

This makes it easy to install and manage multiple versions of Python on your system without needing of pyenv.

Using UV with existing requirements.txt

uv venv --seed -p 3.13

source .venv/bin/activate

uv pip install -r requirements.txt

This is way faster than using any other tool to create virtual environments.

Managing virtual environments

uv init -p 3.13 --name demo

uv add pandas

This creates a new virtual environment named demo with Python 3.13 and adds the pandas package to it.

Runs scripts with inline dependencies

# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "requests",
# ]
# ///
import requests

print(requests.get("https://avilpage.com"))

uv run script.py

uvx

Similar to pipx, it can install and run Python applications in isolated environments.

uv tool install glances

uv tool run glances

# shortcut
uvx glances

Conclusion

UV speeds up Python development by providing a fast package manager, virtual environment management, and inline dependency management. It is a great alternative to traditional tools like pip, pyenv, and virtualenv.

macOS - Log & track historical CPU, RAM usage

Anand Reddy Pandikunta — Fri, 31 May 2024 20:18:02 GMT

In macOS, we can use inbuilt Activity Monitor or third party apps like Stats to check the live CPU/RAM usage. But, we can't track the historical CPU & memory usage. sar, atop can track the historical CPU & memory usage. But, they are not available for macOS.

Netdata

Netdata¹ is an open source observability tool that can monitor CPU, RAM, network, disk usage. It can also track the historical data.

Unfortunately, it is not stable on macOS. I tried installing it on multiple macbooks, but it didn't work. I raised an issue² on their GitHub repository and the team mentioned that macOS is a low priority for them.

Glances

Glances³ is a cross-platform monitoring tool that can monitor CPU, RAM, network, disk usage. It can also track the historical data.

We can install it using Brew or pip.

$ brew install glances

$ pip install glances

Once it is installed, we can monitor the resource usage using the below command.

$ glances

Glances can log historical data to a file using the below command.

$ glances --export-csv /tmp/glances.csv

In addition to that, it can log data to services like influxdb, prometheus, etc.

Let's install influxdb and export stats to it.

$ brew install influxdb
$ brew services start influxdb
$ influx setup

$ python -m pip install influxdb-client

$ cat glances.conf
[influxdb]
host=localhost
port=8086
protocol=http
org=avilpage
bucket=glances
token=secret_token

$ glances --export-influxdb -C glances.conf

We can view stats in the influxdb from Data Explorer web UI at http://localhost:8086.

Glances provides a prebuilt Grafana dashboard⁴ that we can import to visualize the stats.

From Grafana -> Dashboard -> Import, we can import the dashboard using the above URL.

Conclusion

In addition to InfluxDB, Glances can export data to ~20 services. So far, it is the best tool to log, track and view historical CPU, RAM, network and disk usage in macOS. The same method works for Linux and Windows as well.

Screen Time Alerts from Activity Watch

Anand Reddy Pandikunta — Wed, 01 May 2024 01:29:27 GMT

Introduction

Activity Watch¹ is a cross-platform open-source time-tracking tool that helps us to track time spent on applications and websites.

At the moment, Activity Watch doesn't have any feature to show screen time alerts. In this post, we will see how to show screen time alerts using Activity Watch.

Python Script

Activity Watch provides an API to interact with the Activity Watch server. We can use the API to get the screen time data and show alerts.

import json
import os
from datetime import datetime

import requests


def get_nonafk_events(timeperiods=None):
    headers = {"Content-type": "application/json", "charset": "utf-8"}
    query = """afk_events = query_bucket(find_bucket('aw-watcher-afk_'));
window_events = query_bucket(find_bucket('aw-watcher-window_'));
window_events = filter_period_intersect(window_events, filter_keyvals(afk_events, 'status', ['not-afk']));
RETURN = merge_events_by_keys(window_events, ['app', 'title']);""".split("\n")
    data = {
        "timeperiods": timeperiods,
        "query": query,
    }
    r = requests.post(
        "http://localhost:5600/api/0/query/",
        data=bytes(json.dumps(data), "utf-8"),
        headers=headers,
        params={},
    )
    return json.loads(r.text)[0]


def main():
    now = datetime.now()
    timeperiods = [
        "/".join([now.replace(hour=0, minute=0, second=0).isoformat(), now.isoformat()])
    ]
    events = get_nonafk_events(timeperiods)

    total_time_secs = sum([event["duration"] for event in events])
    total_time_mins = total_time_secs / 60
    print(f"Total time: {total_time_mins} seconds")
    hours, minutes = divmod(total_time_mins, 60)
    minutes = round(minutes)
    print(f"Screen Time: {hours} hours {minutes} minutes")

    # show mac notification
    os.system(f"osascript -e 'display notification \"{hours} hours {minutes} minutes\" with title \"Screen TIme\"'")


if __name__ == "__main__":
    main()

This script² will show the screen time alerts using the Activity Watch API. We can run this script using the below command.

$ python screen_time_alerts.py

We can set up a cron job to run this script every hour to show screen time alerts.

$ crontab -e
0 * * * * python screen_time_alerts.py

We can also modify the script to show alerts only when the screen time exceeds a certain limit.

Conclusion

Since Actvity Watch is open-source and provides an API, we can extend its functionality to show screen time alerts. We can also use the API to create custom reports and dashboards.

Setup FTP server on Mac OS X

Anand Reddy Pandikunta — Tue, 30 Apr 2024 18:04:14 GMT

On Linux & Mac OS X, Python comes pre-installed. On Windows, we can install it from Windows store or from https://python.org website.

We can verify the Python version using the below command.

$ python --version
Python 3.11.6

We can use the pyftpdlib library to create an FTP server. We can install the library using the below command.

$ python -m pip install pyftpdlib
[I 11:28:21] concurrency model: async
[I 11:28:21] masquerade (NAT) address: None
[I 11:28:21] passive ports: None
[I 11:28:21] >>> starting FTP server on :::2121, pid=99951 <<<

Now, we can start the FTP server using the below command.

$ python -m pyftpdlib

It will start the FTP server on port 2121. We can connect to the FTP server using the below command.

$ ftp localhost 2121

Cross Platform File Explorer in 50 lines of code

Anand Reddy Pandikunta — Sat, 27 Jan 2024 12:29:05 GMT

In an earlier post, I wrote about why I need a "line count" column in file explorer and how I wrote a Lua script to see it in xplr file manager.

xplr has only terminal interface. It is hard for non-developers to use it. I wanted a small team to use this feature so that it will save several hours of their time. So I decided to write a cross-platform GUI app.

GUI app

Since I am familiar with PySimpleGUI, I decided to write a simple file explorer using it.

As seen in the above screenshot, the file explorer has a "Line Count" column. It is a simple Python script with ~50 lines of code.

The project is open source and source code is available at github.com/AvilPage/LCFileExplorer.

Cross Platform

A new user can't directly run this Python script on his machine unless Python is already installed. Even if Python is installed, he has to install the required packages and run it. This requires technical expertise.

To make it easy for non-tech users to run this program, I decided to use PyInstaller to create a single executable file for each platform.

I created a GitHub action to build the executable files for Windows, Linux, and macOS. The action is triggered on every push to the master branch. This will generate .exe file for Windows, .AppImage file for Linux, and .dmg file for macOS. The executable files are uploaded to the artifacts.

Conclusion

It is easy to create a cross-platform GUI app using Python and PySimpleGUI. It is also easy to distribute the apps built with Python using pyinstaller.

Running tests in parallel with pytest & xdist

Anand Reddy Pandikunta — Sat, 30 Dec 2023 14:46:31 GMT

When tests are taking too long to run, an easy way to speed them up is to run them in parallel.

When using pytest as test runner, pytest-xdist & pytest-parallel plugins makes it easy to run tests concurrently or in parallel.

pytest-parallel works better if tests are independent of each other. If tests are dependent on each other, pytest-xdist is a better choice.

If there are parameterised tests, pytest-xdist will fail as the order of the tests is not guaranteed.

$ pytest -n auto tests/

Different tests were collected between gw0 and gw1. The difference is: ...

To fix this, we have to make sure that the parameterised tests are executed in the same order on all workers. It can be achieved by sorting the parameterised tests by their name.

Alternatively, we can use pytest-randomly plugin to order the tests.

Rearrange CSV columns alphabetically from CLI

Anand Reddy Pandikunta — Fri, 04 Aug 2023 01:49:54 GMT

We can use tools like KDiff3 to compare two CSV files. But, it is difficult to identify the diff when the columns are not in the same order.

For example, look at the below output of 2 simple csv files.

Even though it highlights the diff, it is difficult to identify the diff because the columns are not in the same order. Here is the same diff after rearranging the columns alphabetically.

Now, it is easy to identify the diff.

Rearrange CSV columns alphabetically

We can write a simple python script using Pandas¹ as follows.

#! /usr/bin/env python3

"""
re-arrange columns in alphabetical order
"""
import sys

import pandas as pd


def colsort(df):
    cols = list(df.columns)
    cols.sort()
    return df[cols]


def main():
    input_file = sys.argv[1]
    try:
        output_file = sys.argv[2]
    except IndexError:
        output_file = input_file
    df = pd.read_csv(input_file)
    df = colsort(df)
    df.to_csv(output_file, index=False)


if __name__ == '__main__':
    main()

We can use this script as follows.

$ python3 rearrange_csv_columns.py input.csv output.csv

Instead of writing a script by ourselves, we can use miller² tool. Miller can perform various operations on CSV files. We can use sort-within-records to sort the columns.

$ mlr --csv sort-within-records -f input.csv > output.csv

Conclusion

We can use miller to sort the columns in a CSV file. This will help us to identify the diff easily when comparing two CSV files.

Train LLMs with Custom Dataset on Laptop

Anand Reddy Pandikunta — Thu, 06 Jul 2023 23:06:42 GMT

Problem Statement

I want to train a Large Language Model(LLM)¹ with some private documents and query various details.

Journey

There are open-source available LLMs like Vicuna, LLaMa, etc which can be trained on custom data. However, training these models on custom data is not a trivial task.

After trying out various methods, I ended up using privateGPT² which is quite easy to train on custom documents. There is no need to format or clean up the data as privateGPT can directly consume documents in many formats like txt, html, epub, pdf, etc.

Training

First, let's clone the repo, install requirements.txt and download the default model.

$ git clone https://github.com/imartinez/privateGPT
$ cd privateGPT
$ pip3 install -r requirements.txt
$ wget https://gpt4all.io/models/ggml-gpt4all-j-v1.3-groovy.bin

$ cp example.env .env
$ cat .env
MODEL_TYPE=GPT4All
MODEL_PATH=ggml-gpt4all-j-v1.3-groovy.bin

I have sourced all documents and kept them in a folder called docs. Let's ingest(train) the data.

$ cp ~/docs/* source_documents

$ python ingest.py

This will take a while depending on the number of documents we have. Once the ingestion is done, we can start querying the model.

$ python privateGPT.py
Enter a query: Summarise about Gaaliveedu

The default GPT4All-J v1.3-groovy³ model doesn't provide good results. We can easily swap it with LlamaCpp⁴. Lets download the model and convert it.

$ git clone https://huggingface.co/openlm-research/open_llama_13b

$ git clone https://github.com/ggerganov/llama.cpp.git
$ cd llama.cpp
$ python convert.py ../open_llama_13b
Wrote ../open_llama_13b/ggml-model-f16.bin

We can now update the .env file to use the new model and start querying again.

$ cat .env
MODEL_TYPE=LlamaCpp
MODEL_PATH=/path/to/ggml-model-f16.bin

$ python privateGPT.py
Enter a query: Summarise about Gaaliveedu

Conclusion

This makes it easy to build domain-specific LLMs and use them for various tasks. I have used this to build a chatbot for my internal docs and it is working well.

Remote Debug Docker Container with PyCharm

Anand Reddy Pandikunta — Sun, 11 Jun 2023 15:36:04 GMT

Problem Statement

How to debug a Python application running inside a Docker container that is launched by a third-party process using PyCharm?

Solution

Install the pydevd-pycharm package in the Docker image.

RUN pip install 'pydevd-pycharm~=222.4554.11'

Add the following lines to the Python script that you want to debug.

import pydevd_pycharm
pydevd_pycharm.settrace('host.docker.internal', port=12345, stdoutToServer=True, stderrToServer=True)

Create a new Python Remote Debug configuration in PyCharm with the following settings.

Run the Remote Debug configuration in PyCharm.
Run the Docker container with the following command or let a shell script or another package run the container.

$ docker build . -t flask_web
$ docker run --rm flask_web

Explanation

The pydevd-pycharm package is a Python debugger that can be used to debug a Python application running inside a Docker container. The pydevd_pycharm.settrace() function is used to connect the debugger to the PyCharm IDE. The host.docker.internal is the hostname of the host machine from inside the Docker container. The port is the port number that is used to connect to the PyCharm IDE. The stdoutToServer and stderrToServer are used to redirect the standard output and standard error to the PyCharm IDE.

Gotchas

You might face the following error depending on the version of the pydevd-pycharm package.

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/flask/cli.py", line 218, in locate_app
    __import__(module_name)
  File "/app/app.py", line 5, in <module>
    import pydevd_pycharm
  File "/usr/local/lib/python3.10/site-packages/pydevd_pycharm.py", line 3, in <module>
    from pydevd import settrace
  File "/usr/local/lib/python3.10/site-packages/pydevd.py", line 41, in <module>
    from _pydevd_bundle import pydevd_utils
  File "/usr/local/lib/python3.10/site-packages/_pydevd_bundle/pydevd_utils.py", line 24, in <module>
    from _pydevd_asyncio_util.pydevd_asyncio_utils import eval_async_expression_in_context
ModuleNotFoundError: No module named '_pydevd_asyncio_util'

There seems to be an issue with all 223.*.* versions. The solution is to use the 222.*.* version.

You might face ConnectionRefused error when running the docker container.

  File "/usr/local/lib/python3.10/site-packages/pydevd.py", line 1758, in _locked_settrace
    debugger.connect(host, port)  # Note: connect can raise error.
  File "/usr/local/lib/python3.10/site-packages/pydevd.py", line 660, in connect
    s = start_client(host, port)
  File "/usr/local/lib/python3.10/site-packages/_pydevd_bundle/pydevd_comm.py", line 463, in start_client
    s.connect((host, port))
ConnectionRefusedError: [Errno 111] Connection refused

Ensure that you have started the Remote Debug configuration in PyCharm before running the docker container.

Reducing System Load With ChatGPT

Anand Reddy Pandikunta — Sat, 01 Apr 2023 02:25:49 GMT

Problem Statement

I am using M1 Macbook Air for Python development purposes. Since M1 uses ARM architecture, many Python packages don't have wheels for ARM64/aarch64. confluent-kafka-python is one of them.

I had to run AMD64 docker container to use confluent-kafka-python. Since it is a cross-architecture container, its CPU usage is too high and performance was too slow.

Solution

To reduce system load, I decided to build aarch64 wheels for confluent-kafka-python. I looked at open issues on GitHub and asked maintainers how to build aarch64 wheels. There was no response¹ from them.

As a workaround, I asked ChatGPT² on how to build confluent-kafka-python aarch64 wheels in a docker container.

This initial suggestion didn't work as confluent-kafka-python depends on librdkafka which is a C library. I had to build librdkafka from source for aarch64 and then build confluent-kafka-python from source.

To build librdkafka from the source, I again asked ChatGPT. After making minor changes to the snippet suggested by ChatGPT, I was able to build librdkafka from the source for aarch64.

Here is the final snippet:

FROM ubuntu:22.04

ARG DEBIAN_FRONTEND=noninteractive

RUN apt update && apt install -y \
  wget git curl g++ make postgresql-client \
  nano less shared-mime-info openjdk-17-jre-headless \
  libpq-dev vim tzdata python3 python3-dev

RUN apt install -y python3-pip
RUN python3 -m pip install setuptools

WORKDIR /
RUN git clone https://github.com/confluentinc/confluent-kafka-python
WORKDIR confluent-kafka-python

COPY . /app
WORKDIR /app
RUN ./configure --arch=aarch64 --prefix=/usr
RUN make
RUN make install

WORKDIR /confluent-kafka-python
RUN python3 setup.py install

Conclusion

By running native containers, I was able to reduce the system load by ~50%. With ChatGPT, it is easy to build/tweak programs in languages & environments that we are not familiar with.