Advanced Python: Using decorators, argparse, and inspect to build Fastarg, a command line argument parser library

03.13.2022

I would like to announce the initial release of Fastarg, an open source, lightweight argparse wrapper for building CLI tools.

https://github.com/travisluong/fastarg

https://pypi.org/project/fastarg/

There are no dependencies required to run Fastarg, as it is built on Python’s standard argparse library.

Fastarg follows a nearly identical API to Typer, another popular library for building CLI applications. Typer was the primary inspiration for Fastarg.

Fastarg was developed out of pure curiosity. I wanted to understand the magic of the @ decorator syntax that was so prevalent in many of the frameworks I used regularly.

This all led me down the rabbit hole of metaprogramming, introspection, and inevitably building the first prototype of Fastarg.

In this article, I will go over some of the lessons I learned in that journey. But first, I must give the disclaimer. Use the library and the information provided here at your own risk.

Decorators

The first concept I had to wrap my head around was decorators. In Python, functions are first class objects, which means they can be passed as parameters into other functions and wrapped with additional logic.

For example, a decorator can be created without using the @ syntax:

def deco(func):
    def inner():
        print("running inner()")
        func()
    return inner

def target():
    print('running target()')

target = deco(target)

target()

The above code snippet is essentially the same as this:

def deco(func):
    def inner():
        print("running inner()")
        func()
    return inner

@deco
def target():
    print('running target()')

target()

The @deco above the target function is syntactic sugar for target = deco(target). Both code snippets will return the same output:

running inner()
running target()

When we pass target into deco, we are wrapping the target function with the inner function of deco and returning that as a new function. That also creates a closure, but that’s outside the scope of this article.

argparse

argparse is a standard Python library used for building user friendly command line interfaces, however the interface of argparse itself is quite verbose. Here is a sample from the argparse docs:

import argparse

# create the top-level parser
parser = argparse.ArgumentParser(prog='PROG')
parser.add_argument('--foo', action='store_true', help='foo help')
subparsers = parser.add_subparsers(help='sub-command help')
# create the parser for the "a" command
parser_a = subparsers.add_parser('a', help='a help')
parser_a.add_argument('bar', type=int, help='bar help')
# create the parser for the "b" command
parser_b = subparsers.add_parser('b', help='b help')
parser_b.add_argument('--baz', choices='XYZ', help='baz help')
subparsers_b = parser_b.add_subparsers(help='parser b sub commands')

parser_c = subparsers_b.add_parser('c', help='c help')
parser_c.add_argument('qux', type=str, help='qux help')

# parse some argument lists
args = parser.parse_args()
print(args)
args = parser.parse_args()
print(args)

While this provides a lot out of the box, such as help text, positional argument parsing, and optional argument parsing, it isn’t so readable. Once you start building non-trivial CLI applications, this can get unwieldy extremely fast.

However, it does provide the parsing logic and help text generation, which it does exceptionally well. This is a step up from manually parsing arguments with sys.argv. For that reason, I have used argparse as the foundation for Fastarg.

Introspection

It is much easier to understand the purpose of a tool when you understand why you need a tool in the first place. So first, I will start with the why.

The Typer library introduced to me the idea of using function parameters to parse command line arguments. For example:

import typer

app = typer.Typer()


@app.command()
def hello(name: str):
    typer.echo(f"Hello {name}")


@app.command()
def goodbye(name: str, formal: bool = False):
    if formal:
        typer.echo(f"Goodbye Ms. {name}. Have a good day.")
    else:
        typer.echo(f"Bye {name}!")


if __name__ == "__main__":
    app()

The script could then be executed like so:

$ python main.py hello Camila
$ python main.py goodbye --formal Camila

The hello subcommand takes one positional argument of name. The goodbye subcommand takes a positional argument of name and an optional argument of formal.

It has always perplexed me as to how it was able to convert the function parameters into argument parsing logic. After a bit of research, I came across another standard Python module called inspect. This module allows you to inspect live objects, such as functions and classes. It contains a very useful method called signature, which allows you to see the signature of a method.

For example, I can now see exactly the name and type of each of my function parameters:

from inspect import signature

def cli(func):
    sig = signature(func)
    for name, param in sig.parameters.items():
        print(param.kind, ':', name, '=', param.default)
        annotation = param.annotation
        print(annotation)

@cli
def target(foo: str, bar: int = 1):
    print(f"foo: {foo} bar: {bar}")

If you execute the above script, you will see this output:

POSITIONAL_OR_KEYWORD : foo = <class 'inspect._empty'>
<class 'str'>
POSITIONAL_OR_KEYWORD : bar = 1
<class 'int'>

Perhaps you can already predict where I am headed with this. Now that you are able to introspect into the details of a function, you can pass that information into argparse to set up your argument parsing logic.

The argparse decorator prototype

The code block below is the original prototype of Fastarg. It was developed using the concepts in this article. The general idea is being able to use decorators to enable CLI argument parsing using a concise function parameter syntax.

import sys
import inspect
import argparse
import functools
from inspect import signature

parser = argparse.ArgumentParser(prog="prog", description="cli tool")
subparsers = parser.add_subparsers(help="subparser help")
commands = []

def cli(func):
    global commands
    commands.append(func.__name__)

    sig = signature(func)
    parser_a = subparsers.add_parser(func.__name__, help=func.__doc__)
    for name, param in sig.parameters.items():
        print(param.kind, ':', name, '=', param.default)
        annotation = param.annotation
        if annotation is bool:
            action = argparse.BooleanOptionalAction
        else:
            action = None
        
        if param.default is inspect._empty:
            arg_name = name
            parser_a.add_argument(arg_name, type=annotation, help=f"type: {annotation.__name__}", default=param.default, action=action)
        else:
            arg_name = '--' + name
            parser_a.add_argument(arg_name, type=annotation, help=f"type: {annotation.__name__}", default=param.default, action=action)

    def wrapped(*args, **kwargs):
        args = parser.parse_args()
        ka = dict(args._get_kwargs())
        func(**ka)
    return wrapped

@cli
def hello(name: str):
    print("hello " + name)

@cli
def goodbye(name: str, formal: bool = False):
    if formal:
        print(f"Goodbye Ms. {name}. Have a good day.")
    else:
        print(f"Bye {name}!")

if __name__ == "__main__":
    if len(sys.argv) > 1 and sys.argv[1] != '-h':
        a = sys.argv[1]
        print(a)
        print(commands)
        command = a
        locals()[command]()
    else:
        args = parser.parse_args()

Nested subcommands

We have achieved a prototype, but we’re still lacking many key features, such as nested subcommands and the ability to easily separate them into modules.

For example, we want the ability to do this:

main.py

import fastarg
import commands.todo as todo
import commands.user as user

app = fastarg.Fastarg(description="productivity app", prog="todo")

@app.command()
def hello_world(name: str):
    """hello world"""
    print("hello " + name)

app.add_fastarg(todo.app, name="todo")
app.add_fastarg(user.app, name="user")

if __name__ == "__main__":
    app.run()

commands/address.py

import fastarg

app = fastarg.Fastarg(description="address", help="manage addresses")

@app.command()
def create_address(
    user_id: int, 
    address: str, 
    city: str = fastarg.Option("", help="city (e.g. Seattle)"), 
    state: str = fastarg.Option("", help="state (e.g. WA)"), 
    zip: str = fastarg.Option("", help="zip")
    ):
    """create address for user"""
    print(f"creating address for user {user_id}")
    print(f"{address} {city} {state} {zip}")

commands/todo.py

import fastarg

app = fastarg.Fastarg(description="to do", help="manage todos")

@app.command()
def create_todo(title: str, completed: bool = False):
    """create a todo"""
    print(f"create todo: {title} - {completed}")

@app.command()
def update_todo(
    id: int = fastarg.Argument(help="the primary key of todo"), 
    completed: bool = fastarg.Option(False, help="completed status")
    ):
    """update a todo"""
    print(f"update todo: {id} - {completed}")

commands/user.py

import fastarg
import commands.address as address

app = fastarg.Fastarg(description="user", help="manage users")

@app.command()
def create_user(email: str, password: str, gold: float):
    """create a user"""
    print(f"creating {email}/{password} with {gold} gold")

@app.command()
def delete_user(email: str):
    """delete a user"""
    print(f"deleting user {email}")

app.add_fastarg(address.app, name="address")

The ability to easily create arbitrarily large trees of subcommands is what made Typer such a great library for building CLI tools. I wanted to replicate that functionality for Fastarg.

Data structures

Tree

Achieving the nested subcommand feature requires storing each Fastarg object into a tree-like structure.

The tree structure from the above code would look something like this:

  • Fastarg(prog=”todo”)
    • commands
      • hello_world
    • fastargs
      • Fastarg(name=”todo”)
        • commands
          • create_todo
          • update_todo
      • Fastarg(name=”user”)
        • commands
          • create_user
          • update_user
        • fastargs
          • Fastarg(name=”address”)
            • commands:
              • create_address

Queue

When we parse the arguments of a command which looks like this:

python3 main.py user address create_address 123 "456 main st" --city bellevue --state wa --zip 98004

We must store the arguments in a queue and search the entire tree for the correct subcommand to invoke. For example, we first search for user. We search the root Fastarg to find a Fastarg object named user and then recursively search both its Fastarg objects and commands for the next argument which is address. We continue traversing the tree until we find a subcommand that matches. In this case, it is the create_address subcommand, which is invoked.

Here is the recursive code snippet taken straight from the source code of Fastarg:

def run(self):
    # recursively parse all child fastargs

    # if root fastarg, then generate the root parser object
    self.parser = argparse.ArgumentParser(prog=self.prog, description=self.description)

    # after root is generated, traverse tree of fastargs
    self.traverse_fastargs(self)

    # finally, parse the arguments
    args = self.parser.parse_args()

    argqueue = sys.argv[1:]

    # traverse tree of fastargs for the subparser or command to invoke
    self.search_to_invoke(self, argqueue, args)

def search_to_invoke(self, fastarg, argqueue, commandargs):
    arg = None
    if len(argqueue) > 0:
        arg = argqueue.pop(0)

    if not arg:
        return

    # search fastargs for name of current sys argv
    for cfastarg in fastarg.fastargs:
        if cfastarg.name == arg:
            # if match, recurse on the fastarg with same name
            self.search_to_invoke(cfastarg, argqueue, commandargs)
            return


    # if no match, search commands for current sys argv
    for command in fastarg.commands:
        if command.get_name() == arg:
            # if found, invoke the function
            ka = dict(commandargs._get_kwargs())
            command.function(**ka)

Unpacking argument list

The double star aka double splat aka **-operator is used in the above code to pass a dictionary of key value pairs into the stored function as keyword arguments.

We get the arguments from argparse from this line ka = dict(commandargs._get_kwargs()) and pass them into the function using command.function(**ka).

The command is referencing a Fastarg Command object. Check out the source code for more context.

Help Text

Arguments and Options

One of Typer’s other great features I wanted to copy was the ability to add help text to arguments via a default parameter.

For example:

@app.command()
def update_todo(
    id: int = fastarg.Argument(help="the primary key of todo"), 
    completed: bool = fastarg.Option(False, help="completed status")
    ):
    """update a todo"""
    print(f"update todo: {id} - {completed}")

Running this:

$ python main.py todo update_todo -h

Should give us this help text:

usage: todo todo update_todo [-h] [--completed | --no-completed] id

positional arguments:
  id                    [int] the primary key of todo

optional arguments:
  -h, --help            show this help message and exit
  --completed, --no-completed
                        [bool] completed status (default: False)

Subparser descriptions

Running this:

$ python main.py todo -h

Should show this:

usage: todo todo [-h] {create_todo,update_todo} ...

positional arguments:
  {create_todo,update_todo}
    create_todo         create a todo
    update_todo         update a todo

optional arguments:
  -h, --help            show this help message and exit

The implementation for adding help text to arguments and subparsers was fairly straightforward as it involved defining an Argument class and Option class. Then we store the help text into these instances and pass them into the right argparse parameters during the traverse_fastargs method call which constructs the entire Fastarg tree.

Python Packaging

The last step in building the library was packaging it and publishing it to PyPI. I recommend following the official tutorial as-is before attempting to package your own library. Making sure the module is properly exported can be tricky.

Here is a tip:

src/fastarg/__init__.py

from .main import Fastarg
from .main import Argument
from .main import Option

In __init__.py import the classes from the main module src/fastarg/main.py. This will make the Fastarg classes available from the fastarg package. Yes, I did look at the Typer source code to figure this part out. I suppose that is the beauty of open source.

Once a package is uploaded to PyPI, anyone will be able to install it into any Python project.

pip install fastarg

Conclusion

It was a lot of fun working on this open-source project. It was a great learning opportunity.

If you need a lightweight argument parser with virtually no other dependencies, then give Fastarg a try.

Check out the completed project on GitHub and PyPI:

https://github.com/travisluong/fastarg

https://pypi.org/project/fastarg/

Hope you found this content useful.

Thanks for reading.

References

  1. Packaging python projects. https://packaging.python.org/en/latest/tutorials/packaging-projects/.
  2. Code structure. https://docs.python-guide.org/writing/structure/.
  3. Typer. https://github.com/tiangolo/typer.
  4. Python decorators. https://realpython.com/primer-on-python-decorators/.
  5. Fluent Python: Clear, Concise, and Effective Programming. https://github.com/fluentpython.
  6. argparse. https://docs.python.org/3/library/argparse.html.
  7. Python inspect. https://docs.python.org/3/library/inspect.html.
  8. Unpacking argument lists. https://docs.python.org/3/tutorial/controlflow.html#unpacking-argument-lists