Drafts

Adding a --pdb flag to your applications can make them much easier for those using it to debug your application, especially if your applicatoin is a cli application where the user has much fewer options to start this for themselves. To add a pdb flag --pdb to your applications you will need to wrap your function call in a try/except, and start a post_mortem debugger. I give credit to this stack overflow post for helping me figure this out.

import pdb, traceback, sys


def bombs():
    a = []
    print(a[0])


if __name__ == "__main__":
    if "--pdb" in sys.argv:
        try:
            bombs()
        except:
            extype, value, tb = sys.exc_info()
            traceback.print_exc()
            pdb.post_mortem(tb)
    else:
        bombs()

Using –pdb #

python yourfile.py --pdb

Converting markdown posts to pdf on ubuntu takes a few packages from the standard repos. I had to go through a few stack overflow posts, and nothing seemed to have all the fonts and packages that I needed to convert markdown, but this is what ended up working for me.

Installing all the packages #

sudo apt install \
  pandoc \
  texlive-latex-base \
  texlive-fonts-recommended \
  texlive-extra-utils \
  texlive-latex-extra \
  texlive-xetex

Using pandoc to convert markdown to a pdf #

# older versions of pandoc, I needed this one on ubuntu 18.04
pandoc pages/til/convert-markdown-pdf-linux.md -o convert-markdown-pdf.pdf --latex-engine=xelatex
# newer versions of pandoc, I needed this one on ubuntu 21.04
pandoc pages/til/convert-markdown-pdf-linux.md -o convert-markdown-pdf.pdf --pdf-engine=xelatex

Python comes with an enum module for creating enums. You can make your own enum by inheriting importing and inheriting from Enum.

from enum import Enum


class LifeCycle(Enum):
    configure = 1
    glob = 2
    pre_render = 3
    render = 4
    post_render = 5
    save = 6

auto incrementing #

Enum values can be auto incremented by importing auto, and calling auto() as their value.

from enum import Enum, auto


class LifeCycle(Enum):
    configure = auto()
    glob = auto()
    pre_render = auto()
    render = auto()
    post_render = auto()
    save = auto()

using the enum #

Enum’s are accessed directy under the class itself, and have primarily two methods underneath each thing you make, .name and .value.

Lifecycle.glob
Lifecycle.glob.value
Lifecycle.glob.name

I recently paired up with another dev running windows with Ubuntu running in wsl, and we had a bit of a stuggle to get our project off the ground because they were missing com system dependencies to get going.

Straight in the terminal #

Open up a terminal and get your required system dependencies using the apt package manager and the standard ubuntu repos.

sudo apt update
sudo apt upgrade
sudo apt install \
      python3-dev \
      python3-pip \
      python3-venv \
      python3-virtualenv
pip install pipx

Using an Ansible-Playbook #

I like running things like this through an ansible-playbook as it give me some extra control and repeatability next time I have a new machine to setup.

- hosts: localhost
  gather_facts: true
  become: true
  become_user: "{{ lookup('env', 'USER') }}"

  pre_tasks:
    - name: update repositories
      apt: update_cache=yes
      become_user: root
      changed_when: False
  vars:
    user: "{{ ansible_user_id }}"
  tasks:
    - name: Install System Packages 1 (terminal)
      become_user: root
      apt:
        name:
          - build-essential
          - python3-dev
          - python3-pip
          - python3-venv
          - python3-virtualenv
    - name: check is pipx installed
      shell: command -v pipx
      register: pipx_exists
      ignore_errors: yes

    - name: pipx
      when: pipx_exists is failed
      pip:
        name: pipx
      tags:
        - pipx

video clip #

Here is a clip of me getting pipx running on ubuntu 21.10, and running a few of my favorite pipx commands.

Stow is an incredible way to manage your dotfiles. It works by managing symlinks between your dotfiles directory and the rest of the system. You can then make your dotfiles directory a git repo and have it version controlled. In my honest opinion, when I was trying to get started the docs straight into deep detail of things I frankly don’t really care about and jumped right over how to use it.

When using stow its easiest to keep your dotfiles directory (you may name it what you want) in your home directory, with application directories inside of it.

Then each application directory should reflet the same diretory structure as you want in your home directory.

zsh #

Here is a simple example with my zshrc.

mkdir ~/dotfiles
cd ~/dotfiles
mkdir zsh
mv ~/.zshrc zsh
stow --simulate zsh

You can pass in the –simulate if you wish, it will tell you if there are going to be any more errors or not, but it wont give much more than that.

WARNING: in simulation mode so not modifying filesystem.

Once your ready you can stow your zsh application.

stow zsh

nvim #

A slightly more complicated example is neovim since its diretory structure does not put configuration files directly in your home directory, but rather at a deeper level.

mkdir ~/dotfiles/nvim/.config/nvim/ -p
cd ~/dotfiles
mv ~/.config/nvim/ ~/dotfiles/nvim/.config/nvim/
stow zsh

!notice how the nvim directory inside of dotfiles is structured like it would be in your $HOME directory.

The copier answers file is a key component to making your templates re-runnable. Let’s look at the example for my setup.py.

❯ tree ~/.copier-templates/setup.py
/home/walkers/.copier-templates/setup.py
├── [[ _copier_conf.answers_file ]].tmpl
├── copier.yml
├── setup.cfg
└── setup.py.tmpl

0 directories, 4 files

Inside of my [[ _copier_conf.answers_file ]].tmpl file is this, a message not to muck around with it, and the ansers in yaml form. The first line is just a helper for the blog post.

# ~/.copier-templates/setup.py/\[\[\ _copier_conf.answers_file\ \]\].tmpl
# Changes here will be overwritten by Copier; NEVER EDIT MANUALLY
[[_copier_answers|to_nice_yaml]]

Inside my copier.yml I have setup my _answers_file to point to a special file. This is because this is not a whole projet template, but one just for a single file.

# copier.yml
# ...
_answers_file: .setup-py-copier-answers.yml

Once I change the _answers_file I was incredibly stuck

Run it #

I’m making a library of personal copier templates in my ~/.copier-templates directory and I am going to run it from there.

copier copy ~/.copier-templates/setup.py

Results #

After rendering the template we have the following content in our .setup.setup-py-copier-answers.yml file. This will allow us to update quick if we ever change our template.

# .setup-py-copier-answers.yml
# Changes here will be overwritten by Copier; NEVER EDIT MANUALLY
_src_path: /home/walkers/.copier-templates/setup.py
author_github: waylonwalker
author_name: Waylon Walker
description: awesomeness
framework: null
keywords: null
package_name: my-package

Update it #

This is where I was most stuck, primarily becuase -a <answers_file> must come exactly after the base command copier. This felt a bit odd to and not where I expected it so it.

copier -a .setup-py-copier-answers.yml update

Stop asking all these damn questions #

So the defaults are now changed to our previous results, but it keeps asking for them. To stop asking we can simply add a -f flag.

copier -fa .setup-py-copier-answers.yml update

Once you have made your sick looking cli apps with rich, eventually you are going to want to add some keybindings to them. Currently Textual, also written by @willmcgugan, does this extremely well. Fair Warning it is in super beta mode and expected to change a bunch. So take it easy with hopping on the train so fast.

Get the things #

Install them from the command line.

pip install textual
pip install rich

Import make a .py file and import them in it.

from textual.app import App
from textual.widget import Widget
from rich.panel import Panel

Make what you have a widget #

If you return your rich renderable out of class that inherits from textual.widget.Widget, you can then dock this inside of an app class inheriting from textual.app.App.

class MyWidget(Widget):
    def render(self):
        my_renderable = Panel("press q to quit")
        return my_renderable

class MyApp(App):
    async def on_mount(self) -> None:
        await self.view.dock(MyWidget(), edge="top")
        await self.bind("q", "quit")

run it #

You’ve made a TUI (text user interface). Run the classmethod run to display the it in its full screen glory.

MyApp.run(log="textual.log")

Final result #

At this point It probably does not look much different, but it can be interactive by binding keys to any method on your app that starts with the word action_, this includes the built-in actions such as action_quit.

from textual.app import App
from textual.widget import Widget
from rich.panel import Panel


class MyWidget(Widget):
    def render(self):
        my_renderable = Panel("press q to quit")
        return my_renderable


class MyApp(App):
    async def on_mount(self) -> None:
        await self.view.dock(MyWidget(), edge="top")
        await self.bind("q", "quit")


if __name__ == "__main__":
    MyApp.run(log="textual.log")

I’ve been stuck many times looking at a vim buffer with little question marks at the beginning of each line and trying to get rid of them. for so long I didn’t know what they were so trying to get rid of them was impossible.

It turns out they are tabs, and you can get rid of the little leading question marks with this substitution command.

:%s/\t/    /g

tmux popups can be sized how you like based on the % width of the terminal on creation by using the flags (h, w, x, y) for height, width, and position.

# normal popup
tmux popup figlet "Hello"
# fullscreen popup
tmux popup -h 100% -w 100% figlet "Hello"
# 75% centered popup
tmux popup -h 100% -w 75% figlet "Hello"
# 75% popup on left side
tmux popup -h 100% -w 75% -x 0% figlet "Hello"

example running these commands

I was completely stuck for awhile. copier was not replacing my template variables. I found out that adding all these _endops fixed it. Now It will support all of these types of variable wrappers

# copier.yml
_templates_suffix: .jinja
_envops:
  block_end_string: "%}"
  block_start_string: "{%"
  comment_end_string: "#}"
  comment_start_string: "{#"
  keep_trailing_newline: true
  variable_end_string: "}}"
  variable_start_string: "{{"

!RTFM: Later I read the docs and realized that copier defaults to using [[ and ]] for its templates unlike other tools like cookiecutter.

I’ve been looking for a templating tool for awhile that works well with single files. My go to templating tool cookiecutter does not work for single files, it needs to put files into a directory underneath of it.

template variables #

By default copier uses double square brackets for its variables. variables in files, directory_names, or file_names will be substituted for their value once you render them.

# hello-py/hello.py.tmpl
print('hello-[[name]]')

note! by default copier will not inject variables into your template-strings unless you use a .tmpl suffix.

Before running copier we need to tell copier what variables to ask for, we do this with a copier.yml file.

# copier.yml
name:
  default: my_name
  type: str
  help: What is your name

installing copier #

I prefer to install cli tools that I need globally with pipx, this always gives me access to the tool without worrying about dependency conflicts, bloating my system site-packages, or managing a separate virtual environment for it myself.

pipx install copier

running copier #

When running copier copy we pass in the directory of the template, and the directory that we want to render the template into.

copier copy hello-py .

note! the directory ‘.’ is often referred to in cli programs to represent the current working directory that we are calling the command from.

results #

The resulting files will have your variables injected into them if you have setup your template and copier.yml up correctly.

print('hello-you')

I just installed a brand new Ubuntu 21.10 Impish Indri, and wanted a kedro project to play with so I did what any good kedroid would do, I went to my command line and ran

pipx run kedro new --starter spaceflights

But what I got back was not what I expected!

Fatal error from pip prevented installation. Full pip output in file:
    /home/walkers/.local/pipx/logs/cmd_2022-01-01_20.42.16_pip_errors.log

Some possibly relevant errors from pip install:
    ERROR: Could not find a version that satisfies the requirement kedro (from versions: none)
    ERROR: No matching distribution found for kedro
Error installing kedro.

This is weird, why cant I run kedro new with pipx? Lets try pip.

pip install kedro

Same issue.

ERROR: Could not find a version that satisfies the requirement kedro (from versions: none)
ERROR: No matching distribution found for kedro

What is Kedro

Curious what kedro is? Check out this article.

What’s up #

wrong python version

The issue is that kedro only runs on up to python 3.8, and on Ubuntu 21.10 when you apt install python3 you get python 3.9 and the standard repos don’t have an old enough version to run kedro.

How to fix this? #

Theres a couple of ways you can fix this? They all involve installing a distribution that does not come from the standard repo.

Where Can I get the right version #

• Anaconda
• Python.org
• deadsnakes
• pyenv
• miniconda

I have two articles that can help you #

How to Install miniconda on linux (from the command line only)

Using miniconda

conda create -n myenv python=3.8

My first impressions with pyenv

Using pyenv

pyenv install 3.8.12

Pluggy makes it so easy to allow users to modify the behavior of a framework without thier specific feature needing to be implemented in the framework itself.

I’ve really been loving the workflow of frameworks built with pluggy. The first one that many python devs have experience with is pytest. I’ve never created a pytest plugin, and honestly at the time I looked into how they were made was a long time ago and it went over my head. I use a data pipelining framework called kedro, and have build many plugins for it.

Making a plugin #

super easy to do

As long as the framework document the hooks that are available and what it passes to them it’s so easy to make a plugin. Its just importing the hook_impl, making a class with a function that represents one of the hooks, and decorating it.

from framework import hook_impl

class LowerHook:
    @hook_impl
    def start(pluggy_example):
        pluggy_example.message = pluggy_example.message.lower()

installing pluggy #

Installing pluggy is just like most python applications, install python, make your virtual environment, and pip install it.

pip install pluggy

Making a plugin driven framework #

much less easy

At the time I started playing with pluggy, their docs were less complete, or I was just plain blind, but this was a huge part of the docs that were missing for me that now actually appear to be there. But to get some more examples out there, here is my version.

import pluggy

# These don't need to match
HOOK_NAMESPACE = "pluggy_example"
PROJECT_NAME = "pluggy_example"

hook_spec = pluggy.HookspecMarker(HOOK_NAMESPACE)
hook_impl = pluggy.HookimplMarker(HOOK_NAMESPACE)


class PluggyExampleSpecs:
    """
    This is where we spec out our frameworks hooks, I like to refer to them as
    the lifecycle.  Each of these functions is a hook that we are exposing to
    our users, with the kwargs that we expect to pass them.
    """
    @hook_spec
    def start(self, pluggy_example: PluggyExample) -> None:
        """
        The first hook that runs.
        """
        pass

    @hook_spec
    def stop(self, pluggy_example: PluggyExample) -> None:
        """
        The last hook that runs.
        """
        pass


class PluggyExample:
    """
    This may not need to be a class, but I wanted a container where all the
    hooks had access to the message.  This made sense to me to do as a class.
    """

    def __init__(self, message="", hooks=None) -> None:
        """
        Setup the plugin manager and register all the hooks.
        """
        self._pm = pluggy.PluginManager(PROJECT_NAME)
        self._pm.add_hookspecs(PluggyExampleSpecs)
        self.message = message
        self.hooks = hooks
        if hooks:
            self._register_hooks()

    def _register_hooks(self) -> None:
        for hook in self.hooks:
            self._pm.register(hook)

    def run(self):
        """
        Run the hooks in the documented order, and pass in any kwargs the hook
        needs access to.  Here I am storing the message within this same class.
        """
        self._pm.hook.start(pluggy_example=self)
        self._pm.hook.stop(pluggy_example=self)
        return self.message


class DefaultHook:
    """
    These are some hooks that run by default, maybe these are created by the
    framework author.
    """
    @hook_impl
    def start(pluggy_example):
        pluggy_example.message = pluggy_example.message.upper()

    @hook_impl
    def stop(pluggy_example):
        print(pluggy_example.message)


if __name__ == "__main__":
    """
    The user of this framework can apply the hook in their own code without
    changing the behavior of the framework, but the library has
    implemented it's own default hooks.
    """
    pe = PluggyExample(
        message="hello world",
        hooks=[
            DefaultHook,
        ],
    )
    pe.run()

Modifying behavior #

as a user of PluggyExample

Now Lets pretent the user of this library likes everything about it, except, they don’t like all the shouting. They can either search for a plugin on Google, github, or pypi and find one, or make it themself. the magic here is that they do not need to have the package maintainer patch the core library itself.

class LowerHook:
    """
    This is a new hook that a plugin author has created to modify the behavior
    of the framework to lowercase the message.
    """
    @hook_impl
    def start(pluggy_example):
        pluggy_example.message = pluggy_example.message.lower()

from pluggy_example import PluggyExample
pe = PluggyExample(
    message="hello world",
    hooks=[
        DefaultHook,
        LowerHook
    ],
)
pe.run()

Running Pluggy Example #

Here is a short clip of me running the pluggy example in it’s default state, then adding the LowerHook, and running a second time.

One of the most useful skills you can acquire to make you faster at almost any job that uses a computer is getting good at finding text in your current working diretory and identifying the files that its in. I often use the silver searcher ag or ripgrep rg to find files in large directories quickly. Both have a sane set of defaults that ignore hidden and gitignored files, but getting them to list only the filenames and not the matched was not trivial to me.

I’ve searched throught he help/man pages many times looking for these flags and they always seem to evade me.

ag #

Passing the flag -l to ag will get it to list only the filepath, and not the match. Here I gave it a --md as well to only return markdown filetypes. ag supports a number of filetypes in a very similar way.

ag nvim --md -l

rg #

Giving rg the --files-with-matches flag will yield you a similar set of results, giving only the filepaths themselves and not the match statement. Also passing in the -g "*.md" will similarly yield only results from markdown files.

rg --files-with-matches you -g "*.md"

pyenv provides an easy way to install almost any version of python from a large list of distributions. I have simply been using the version of python from the os package manager for awhile, but recently I bumped my home system to Ubuntu 21.10 impish, and it is only 3.9+ while the libraries I needed were only compatable with up to 3.8.

I needed to install an older version of python on ubuntu

I’ve been wanting to check out pyenv for awhile now, but without a burning need to do so.

installing #

Based on the Readme it looked like I needed to install using homebrew,so this is what I did, but I later realized that there is a pyenv-installer repo that may have saved me this need.

Installing Homebrew on Linux

List out install candidates #

You can list all of the available versions to install with pyenv install --list. It does reccomend updating pyenv if you suspect that it is missing one. At the time of writing this comes out to 532 different versions!

pyenv install --list

Let’s install the latest 3.8 patch #

Installing a version is as easy as pyenv install 3.8.12. This will install it, but not make it active anywhere.

pyenv install 3.8.12

let’s use python 3.8.12 while in this directory #

Running pyenv local will set the version of python that we wish to use while in this directory and any directory underneath of it while using the pyenv command.

pyenv local python3.8.12

.python-version file #

This creates a .python-version files in the directory I ran it in, that contains simply the version number.

3.8.12

using with pipx #

I immediately ran into the same issue I was having before when trying to run pipx, as pipx was running my system python. I had to install pipx in the python3.8 environment to get it to use it.

pyenv exec pip install pipx
pyenv exec pipx run kedro new

python is still the system python #

When I open a terminal and call python its still my system python that I installed and set with update-alternatives. I am not sure if this is expected or based on how I had installed the system python previously, but it’s what happened on my system.

update-alternatives --query python

Name: python
Link: /home/walkers/.local/bin/python
Status: auto
Best: /usr/bin/python3
Value: /usr/bin/python3

making a virtual environment #

To make a virtual environment, I simply ran pyenv exec python in place of where I would normally run python and it worked for me. There is a whole package to get pyenv and venv to play nicely together, so I suspect that there is more to it, but this worked well for me and I was happy.

pyenv exec python -m venv .venv --prompt $(basename $PWD)

Now when my virtual environment is active it points to the python in that virtual environment, and is the version of python that was used to create the environment.

Links #

https://github.com/pyenv/pyenv#installation

Installing brew on linux proved quite easy and got pyenv running for me within 4 commands.

I had never used homebrew before, honestly I thought it was a mac only thing for years. Today I wanted to try out pyenv, and the reccommended way to install was using homebrew. I am not yet sure if I want either in my normal workflow, so for now I am just going to pop open a new terminal and install homebrew and see how it goes.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
echo 'eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"' >> /home/walkers/.zprofile
eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"

That was it, now homebrew is working. Starting a new shell and running the command to install pyenv worked.

brew install pyenv

Links #

• homebrew

When I first moved to vim from and ide like vscode or sublime text one of my very first issues was trying to preview my website at localhost:8000. There had always just been a button there to do it in all of my other editors, not vim. There are not many buttons for anything in vim. While there is probably a plugin that can run a webserver for me in vim, it’s not necessary, we just need the command line we are already in.

running a separate process #

You will need a way to run another process alongside vim, here are a couple ideas to get you going that are not the focus here.style

• use background jobs
  • c-z to send a job to the background
  • fg to bring it back
• use a second terminal
• use a second tab
• use tmux and run it in a separate split/window
• use an embeded nvim terminal

running a development webserver from the command line #

Python already exists on most linux systems by default, and most are now on python3. If you are on windows typing python will take you directly to the windows store to install it, or you can also use wsl.

# python3
python -m http.server

# running on port 5000
python -m http.server --directory markout 5000

# for the low chance you are on python2
python -m SimpleHTTPServer

# running on port 5000
python -m SimpleHTTPServer 5000
python -m SimpleHTTPServer --directory markout 5000

using nodejs #

If you are a web developer it’s likely that you need nodejs and npm on your system anyways and may want to use one of the servers from npm. I’ll admit with these not being tied to the long term support of a language they are much more feature rich with things like compression out of the box. In my opinion they are nice things that you would want out of a production server, but may not be necessary for development.

installing npx #

# if you don't alredy have npx
npm i -g npx

npx is a handy tool that lets you run command line applications straight from npm without installing them. It pulls the latest version every time you want to run, then executes it without it being installed.

running the http-server with npx #

npx http-server

# running on port 5000
npx http-server -p 5000
npx http-server markout -p 5000

Many command line tools can output a list of files, this is quite powerful. I often want to search for something, then open it from a fuzzy picker. This can be done with fzf in the terminal, but often I am already in vim and I want to open it inside my current session.

Telescope #

how to pass a custom command to telescope

Telescope is the fuzzy file finder I use every day inside of neovim. Its pretty fantastic and easy to extent like this. This first example I am only passing in files from the current working directory by using ls.

:Telescope find_files find_command=ls

This brings up a normal Telescope picker with results from the ls command.

More arguments #

how to pass a muli-argument command to telescope

Adding more arguments can be done by comma separating them as shown in the example below. This command will run the silver-searcher, search for all occurences of nvim inside of a markdown file, and return only the filepaths so Telescope can pick from them.

:Telescope find_files find_command=ag,nvim,--md,-l