Twilio Integration With Frappe Framework

Frappe1 is a low code framework for rapidly developing web applications. Twilio2 is a SAAS platform for SMS, Video etc with APIs.

In this post, lets see how to setup Twilio Integration with Frappe.

Sending SMS

Frappe has inbuilt SMS manager3 where users can confgiure SMS gateway and send SMS to mobiles directly.

To send out messages/SMS with Twilio, we just need to configure Twilio API keys in SMS settings.

First, create an account in Twilio and collect the following information from the account.

  • Twilio account SID

  • Gateway URL

  • Auth Token

  • "From" Phone number

These details need to be added to SMS settings in the following format.

For authorization parameter, we need to enter base64 encoded value of account_sid:auth_token.

Once these values are set, we can go to SMS Center and send out dummy messages to ensure all settings are configured properly.

Twilio App

If we want to manage incoming/outgoing voice calls or send messages via WhatsApp, we need to install twilio-integration4 app. We can install the app on our site by running the following commands.

bench get-app https://github.com/frappe/twilio-integration.git
bench --site example.com install-app twilio_integration

Once the app is installed, we can go to Twilio Settings and configure the keys as shown below.

After that we can setup Voice Call Settings to manage incoming/outgoing calls.

To send messages via Whatsapp, we can set the channel as Whatsapp in Notification doctype.

This is how we can send SMS, Whatsapp messages & manage calls via Twilio using Frappe Framework.

Why DMART is not in FNO category?

FNO Segment

There are 4000+ companies1 that are traded in NSE/BSE. Of these, 198 stocks are included in FNO segement2. For these stocks, Futures & Options(FNO) contracts will be available and these stocks won't have any fixed ciruit limits.

FNO Reviews

On 11th April 2018, SEBI released a circular(SEBI/HO/MRD/DP/CIR/P/2018/67) on a framework3 for reviewing stocks in derivatives segement. Based on this framework, a stock has to meet the below criteria be added in FNO segement.

  • The stock shall be chosen from amongst the top 500 stocks in terms of average daily market capitalization and average daily traded value in the previous six months on a rolling basis.

  • The stock’s median quarter-sigma order size over the last six months, on a rolling basis, shall not be less than ₹25 Lakh.

  • The market wide position limit(MWPL) in the stock shall not be less than ₹500 crore on a rolling basis.

  • Average daily delivery value in the cash market shallnot be less than ₹10 crore in the previous six months on a rolling basis.

If a stock is in FNO category and fails to meet this criteria, it will be removed from FNO segment.

In 2021 alone, SEBI released 6 circulars thrugh which 32 new stocks are added in the FNO segment.

DMART

DMART has met the above FNO criteria long back. Even after it has met the criteria, there were more than 6 reviews456 and suprisingly DMART was not added to the FNO segement.

There is a long discussion on Zerodha TradingQ&A forum7 on why DMART was not added to FNO category but no one could give any explaination.

After that, I have sent an email to SEBI & NSE seeking clarification for the same. It has been more than 6 months and they haven't responded yet. I have reached out to few people in the trading community privately to get clarification on the same. But nobody I know has any clue on this.

I am still looking for an answer. If you can shed some light, please send out a message to me. I would like to have a quick chat with you regarding the same.

Hoping to solve the mystery soon.

Using Frappe Framework As REST API Generator

Introduction

When a company plans to build a mobile application and/or a web application, they need to create REST APIs to store data. This is a common requirement for CRUD applications.

In the Python ecosystem, there are several projects like Django Rest Framework, Flask-RESTful, FastApi which does the heavy lifting of implementing REST APIs. In other ecosystems, there are similar projects/frameworks for this job.

REST API Generators

By using the above mentioned frameworks, developers can build REST APIs at a faster rate. Still, developers have to develop and maintain code for these APIs.

To avoid even this work, there are REST API generators like postgrest1, prest which can instantly generate REST APIs by inspecting database schema. With these, developers just have to design the DB schema and these tools take care of generating APIs without writing a single line of code.

In this post, let us see how Frappe framework can be used as a REST API generator and what are the advantages of using Frappe.

Frappe Framework

Frappe framework is meta data driven low code, web framework written in Python and Javascript.

Web UI

Frappe framework provides web UI to create models(called doctypes in Frappe) and it provides REST API2 for all the models out of the box.

There is no need to write manual SQL queries to manage schema. With some training even non-developers can even manage models in Frappe.

Roles & Permissions

With traditional API generators, managing roles & permissions involves additional development and maintenance costs. Frappes comes with an authentication system and it has support for role based permissions out of the box. From the web UI, users can manage roles & permissions.

Hooks

Even though REST API generators give API out of the box, there will be scenarios where custom business logic needs to be hooked in for various events. In such scenarios, developers end up using an alternate framework/tool to manage hooks and business logic.

Frappe provides server scripts by which arbitrary python code can be executed dynamically based on model events. There is no need to set up another framework for these things.

Utilities

Frappe framework comes with a lot of utilities like Job Queues, Schedulers, Admin interface, socket.io etc. As the project grows and the need evolves, Frappe has all the common utilities that are required for a web application development.

Conclusion

When a company wants to build a solution to a problem, it should focus most of the time in solving that problem instead of wasting their time on building CRUD interfaces or REST APIs.

Frappe framework was designed to rapidly build web applications with low code. If you need a REST API generator and some additional functionality for the REST APIs, Frappe framework fits the bill and reduces a lot of development time.

Adding Fuzzy Search To Frappe Framework

Introduction

As software developers, we use fuzzy search a lot especially when using Emacs or any other editors/IDEs. For example to search a file called patient_history.js, in the editor, we can just type pah and editor will narrow it down.

This is quite handy as we can open any file with just few characters.

FF is a low code, open source, web framework in Python and Javascript. All sites built with FF will have a global search bar(aka awesome bar) as shown below. Here, we can search for doctypes, reports, pages etc.

To open Patient History, we have to type almost the entire text in search bar. If we type pah like we have typed in the editor, it won't show any results.

Instead, we can add fuzzy search here so that we can search for any item with just a frew key strokes.

Fuzzy Search

There are many third party packages which implement fuzzy search in programming languages. However we can't use any of these fuzzy search packages directly. These editors internally use a scoring algorithm to rank the results and display results based on score.

It internally considers many factors mentioned below for scoring.

  • Matched letters

  • CamelCase letters

  • snake_case letters

  • Consecutive matching letters

We can come up with a scoring mechanism for these factors and based on the matches, we can rank the results. I have implemented a custom fuzzy search alogirthm based on the above factors but it was slow and results were not good in some cases.

Then I stumbled up this fts_fuzzy_match implementation. This is a reverse engineered implementation of sublime text fuzzy search and it has a detailed scoring mechanism as well. It assigns negative ranking to mismatched letters and bonus points for consecutive matches.

This works well and is as effective as most IDEs search. Now that there is a solid fuzzy search, all we need to do is hook this up in FF.

FF internally has a fuzzy search function and we can directly hook it up here as shown here.

After that, we can search for anything in just few key strokes. For example to open patient history, we can just type pah and it will show results like this.

Conclusion

Fuzzy search in editors/IDEs is quite handy and when we bring to other places like FF or any other search bar, it improves search experience a lot.

Using Frappe Framework As An Integration Engine

Introduction

In healthcare orgainsations, data exchange between systems is complicated and has interopolabilty issues. Integration engines are widely used in healthcare industry for bi-directional data transfer.

In this article, let us look at the advantages of using interface engines and how Frappe Framework can be used as an interface engine.

Integration Engines

In a traditional agile development approach, building a new interface might take weeks/months. With an integration engine, a new interface can be replicated in a matter of hours with little or no scripting at all.

Creating a REST API, listening to a webhook, transforming a data between channels, broadcasting a message, sending/receiving HL7 messages or any other commonly performed task can be implemented in integration engine without much effort.

Due to this integration engines like Mirth Connect are widely used in healthcare.

The above diagram shows usage of integration engine in a healthcare orgainsation.

Frappe Framework

Frappe Framework is a low code web application framework with batteries included. Even though Frappe is lablled as a framework, it can be used as an integration engine as well.

It provides REST APIs out of the box for all the models(called doctypes in Frappe). Users can create custom APIs using server scripts and it has support for webhooks as well.

Users can schedule custom cron jobs, setup emails, enable data transformations and other tasks without much coding.

One feature Frappe Framework lacks when compared to integration engines is sending/receiving HL7 messages on ad-hoc ports. This feature is not available out of the box. Users need to develop a custom app or use any 3rd party app for it.

Frappe Healthcare is an HIS(Healthcare Information System) built on top of Frappe/ERPNext. If a hospital is using Frappe Healthcare, then there won't be a need to use integration engine as Frappe framework can take care of most of these things.

The above diagram shows usage of Frappe Healthcare as HIS in a healthcare orgainsation.

Conclusion

In healthcare, integration engines are used extensively to tackle data exchange between systems. Due to its low-code development and batteries included approach, even though Frappe is a web application framework, it can be used as an integration engine as well.

On Resuming Writing Challenge

Photo by Kaboompics on Pexels

In 2018, I decided to write at least one blog post per month throughout the year. Even though I tried to write posts every month, I couldn't publish anything in few months.

In 2019, I went a step ahead and made a legal(?) agreement with a friend. I paid him 1,00,000 rupees and told him that he could keep the money as a reward if I failed to write a blog post every month.

This agreement kept me on my toes. I didn't miss writing a single month in 2019. I stayed awake on the last days of the month to finish and publish the post before midnight.

In 2020, I took up the challenge again and I was able to write at least one post every month.

In 2021, I didn't take up the challenge. I wrote just three posts in the entire year.

In the two years when I took the challenge, even though I wrote a few mediocre articles, I wrote a few good articles. In the other two years when I didn't take the challenge, my writing quality and quantity declined.

Due to this, I decided to take up the writing challenge again this year.

Instead of limiting the 1,00,000 reward to my friend, I decided to extend it to all the readers.

The first person who calls out that there is no new blog post in a month will get the 1,00,000 reward. The next three people will get a small gift as a token of appreciation.

I will try my best to write at least one post every month. Let's wait till the end of the year and see how it goes.

A Typo Got Me $100 Bug Bounty

Introduction

On a lazy evening, while on a call with a friend, I made a typo while entering the url. Instead of typing http://app-00421.on-aptible.com, I typed http://app-00412.on-aptible.com1.

In this article, lets see how this typing mistake got me a bug bounty.

Vulnerability

A bug bounty program2 is a deal offered by companies by which individuals can receive recognition and compensation for reporting bugs, security exploits and vulnerabilities.

Aptible provides HIPAA3 compliant PAAS platform so that healthcare companies can deploy their apps without compliance hassle.

After deploying an application on aptible, users can create an endpoint for public access. For this purpose, atpible generates domain names in sequential order.

Due to this, a set of publicly exposed servers will have incremental domain names. A lot of companies use these sequentially generated domain names for staging & testing purposes. In general, many companies don't bother about implementing security best practices on non-production servers.

When I was trying to access a demo site at http://app-00421.on-aptible.com, I made a typo and visited http://app-00412.on-aptible.com. This site was a staging site of some other company without any authentication. The company's source code, AWS keys and a lot of sensitive information was publicly accessible.

I quickly sent an email to that company regarding this issue and they took their site offline. As per Aptible disclosure policy4, this bug is out of scope. However I sent an email to their team regarding the severity of the issue. Since sequential domain names are generating additional target surface for attackers, I suggested to move to random urls.

For this disclosure, they have provided a bounty of 100$ and Aptible decided to move away from sequential domain names.

Lesser Know Useful Utilities For Mac

Introduction

When using Mac, there are few utilities which come in handy for day to day operations and also aid in productivity.

Here are some of the useful but lesser know utilities for mac.


iGlance

iGlance is a system monitor tool that shows all the stats right from the menu bar itself.


Debokee Tools

Wondering which network your Mac connected to? If you use multiple wireless networks, then Debokee Tools can show the connected wireless network name directly in the menu bar.


Espanso

Espanso is a text expanding tool that improves productivity across the system. We can set up shortcuts for frequently typed things like email, phone number etc so that we don't have to type them again and again.


Karabiner-Elements

Karabiner Elements allows users to customize keyboard via simple modifications, complex modifications, function key modifications etc.


Flycut

Flycut is a simple clipboard manager, stores history. When you want to copy/paste frequently, this comes in handy.


CheatSheet

Ever wondered what are the keybindings when using any application? With CheatSheet, we can just hold key bit longer, and it will show all the available shortcuts in the application.


Bandwidth+

Bandwidth+ tracks network usage on Mac. If there are multiple networks, it gives detailed information about the network consumed on all the networks.


Grand Perspective

If Mac is running low on disk space, Grand Perspective shows a graphical view of the disk usage. It will be much easier to pinpoint large files that are consuming the disk and then clean them up.


Conclusion

These are some useful utilities for day to day usage. In the upcoming articles, lets learn about useful command line utilities that improve productivity on a daily basis.

Mastering PACS/DICOM #2 - Setup Orthanc Demo Server

This is a series of articles on mastering Dicom. In the earlier article, we have learnt how PACS/DICOM simplifies the clinical work flow.

In this article, lets setup a dicom server so that we have a server to play around with Dicom files.

Orthanc Server

There are several Dicom servers like Orthanc, Dicoogle etc. Orthanc is a lightweight open source dicom server and is widely used by many Health care organisations.

Sébastien Jodogne, original author of Orthanc maintains docker images. We can use these images to run Orthanc server locally.

Ensure docker is installed on the machine and then run the following command to start Orthanc server.

$ docker run -p 4242:4242 -p 8042:8042 --rm \
    jodogne/orthanc-python

Once the server is started, we can visit http://localhost:8042 and explore Orthanc server.

Heroku Deployment

Heroku is PAAS platform which supports docker deployments. Lets deploy Orthac server to Heroku for testing.

By default, Orthanc server runs on 8042 port as defined in the config file. Heroku dynamically assigns port for the deployed process.

We can write a shell script which will read port number from environment variable, replace it in Orthanc configuration file and then start Orthanc server.

#! /bin/sh

set -x

echo $PORT

sed 's/ : 8042/ : '$PORT'/g' -i /etc/orthanc/orthanc.json

Orthanc /etc/orthanc/

We can use this shell script as entry point in docker as follows.

FROM jodogne/orthanc-python

EXPOSE $PORT

WORKDIR /app
ADD . /app

ENTRYPOINT [ "./run.sh" ]

We can create a new app in heroku and we can deploy this container.

$ heroku apps:create orthanc-demo

$ heroku container:push web
$ heroku container:release web

Once the deployment is completed, we can access our app from the endpoint provided by heroku. Here is a orthanc demo server running on heroku.

Conclusion

In this article, we have learnt how to setup Orthanc server and deployed it to Heroku. In the next article, let dig deeper into dicom protocol by upload/accessing dicom files to the server.

Minimum Viable Testing - Get Maximum Stability With Minimum Effort

Introduction

Even though Test Driven Development(TDD)1 saves time & money in the long run, there are many excuses why developers don't test the software. In this article, lets look at Minimum Viable Testing(aka Risk-Based Testing)2 and how it helps to achieve maximum stability with minimum effort.

Minimum Viable Testing

Pareto principle states that 80% of consequences come from 20% of the causes. In software proucts, 80% of the users use 20% of the features. A bug in these 20% features is likely to cause higher impact than the rest. It makes sense to prioritize testing of these features than the rest.

Assessing the importance of a feature or risk of a bug depends on the product that we are testing. For example, in a project a paid feature gets more importance than free feature.

In TDD, we start with writing tests and then writing code. Compared to TDD, MVT consumes less time. When it comes to testing, there are unit tests, integration tests, snapshot tests, ui tests and so on.

When getting started with testing, it is important to have integration tests in place to make sure if something is working. Also the cost of integration tests is much cheaper compared to unit tests.

Most of the SAAS products have a web/mobile application and an API server to handle requests for the front end applications. Having UI tests for the applications and integration tests for APIs for the most crucial functionality should cover the ground. This will make sure any new code that is being pushed doesnt break the core functionality.

Conclusion

Even though RBT helps with building a test suite quicker that TDD, it should be seen as an alternate option to TDD. We should see RBT as a starting point for testing from which we can take next step towards achieving full stability for the product.