Refactoring Django With FST


Django developers decided to drop Python 2 compatability in Django 2.0. There are serveral things that should be refactored/removed.

For example, in Python 2, programmers has to explicitly specify the class & instance when invoking super.

class Foo:
    def __init__(self):
        super(Foo, self).__init__()

In Python 3, super can be invoked without arguments and it will choose right class & instance automatically.

class Foo:
    def __init__(self):
        super().__init__()

For this refactoring, a simple sed search/replace should suffice. But, there are several hacks in codebase where super calls the grandparent instead of the parent. So, sed won't work in such cases. Also it is very hard to refactor them manually and much harder for reviewers to review it as there are 1364 super calls in code base.

→ grep -rI "super(" | wc -l
   1364

Changes has to be scripted. I wrote a simple script to replace super calls by class names. This worked only for 50% of the cases. It failed to capture classes which had comments on top of them, classes with decorators and nested classes.

To handle all these cases, a normal python script gets more complicated and there is no guarantee that it can handle all edge cases. So, I experimented with AST(Abstract Syntax Trees).

Python has ast module to convert code to AST but it can't convert AST back to code. There are 3rd party packages like astor which can do this.

# this is a comment
def foo():

   print(
    "hello world"
)

Converting above code to AST and then converting back gives this

def foo():
    print('hello world')

Code to AST is a lossy transformation as they cannot preserve empty lines, comments and code formatting.

ast_to_code(code_to_ast(source_code)) != source_code

For lossless transformation, FST(Full Syntax Trees) packages like Baron or RedBaron should be used.

fst_to_code(code_to_fst(source_code)) == source_code

With RedBaron FST, just locate super calls, find nearest class node, check class name with super and replace accordingly. It took less than 10 lines of code.

RedBaron has good documentation with relveant examples and its API is similar to BeautifulSoup. Next time when writing code that modifies code consider using RedBaron.

Thanks to Tim Graham & Aymeric Augustin for reviewing the patch.



I am Chillar Anand. I daydream a lot and write about the things that interest me here. You can read more about this blog here.

See all articles

RSS Feed for the blog

Edit this page