Drafts

pytest has open sourced their amazing plugin framework, it allows library authors to give their users a way to modify the libaries behavior without needing to submit a change that may not make sense to the entire library. Previous Experience # My experience so far as a plugin user, and plugin author has been great. Building and using plugins are incredibly intuitive. I wanted to dive a bit deeper and see how they are implemented inside of a library and its a bit of a mind bend the first time you try to do it. Plugins vs. Hooks # A hook is a single function that has a specific place that it is ran by the PluginManager. A Plugin is a collection of one or more hooks. Layers # library author plugin author end user Using a plugin # For a plugin to be registered is must be registered by the PluginManager which is implemented by the library author. It is the job of the library author to determine what plugins are actively registered or disabled. There are two common ways that I have seen that…

There are so many tutorials out there for creating OG images with or, but I have yet to see many using python. PIL # Template # Text # Overlays #

There are various ways to configure python tools, config files, code, or environment variables. Let’s look at a few projects that allow users to configure them through the use of config files and how they do it. Motivation # This will not include how they are implemented, I’ve looked at a few and its not simple. This will focus on where config is placed and the order in which duplicates are resolved. The motivation of this article is to serve as a bit of a reference guide for those who may want to create their own package that needs configuration. Flake8 # Global # User settings can exist in the users file to configure how flake8 runs on their machine. Per-Project # Only One project config file will be considered, but allows for several options. These files all use the format and must have a section header to be consideered. Selection of the config file can also be overridden by the cli option. An extra config file may be selected as. It will be read in last and take highest precedence…

Docs # https://kedro.readthedocs.io/en/stable/04_kedro_project_setup/01_dependencies.html?highlight=install pip-tools # pip-compile # requirements # requirements.in requirements.txt

Generating an api for a blog is much simpler than one might expect with python. Markdown # Frontmatter # Fill in the blanks # fix missing data Fast #

Overrated underrated command line applications Ship More Ship Less - get feedback and apply feedback, build the right things kedro YouTube series # Get started on daily kedro shorts, these are indended to be short clips that people can watch a playlist and learn about kedro concepts at their own pace. This is meant to be low barrier to entry for me to create. Like the tmux series, I hope to make a sub three minute video within one or two takes, no edits, all straight from obs. written # how I kedro 8/16 what is kedro 8/17 kedro new 8/18 three ways to create a kedro environment 8/19 git init 8/20 kedro install 8/21 kedro pipeline create 8/22 staging # making your first nodes in kedro 8/23 kedro run 8/24 kedro catalog create 8/25 add new dependencies 8/26 listing things from the kedro cli 8/27 Storing catalog entries 8/30 Comprehensive nodes modular piplines 8/29 kedro pipeline package kedro pipeline pull parameters pipeline registry find-kedro lambdas in kedro nodes built in pipeline fi…

?docstring??sourcecode %run %debug %autoreload %history autoformat %reset!shell commands?docstring # Stop going to google everytime your stuck and stay in your workflow. The ipython is a superhero for productivity and staying on task. Note This does jump you into a pager, a j,k or up, down to navigate, q to quit.??sourcecode # Docstring not enough for you use case. I often run into cases where the docstring is not clear enough and I need to see the implementation for myself to see what a function does. %run # I turned my nose up at this one, prior to seeing the famous I don’t like notebooks by Joel Grus. My first snobby reaction was that developing modules and using autoreload was superior. I have since realized there is a place for, and it can cut down on some keystrokes to import, setup, and run even when developing in modules. %debug # ipython comes with a post-mortem debugger, and it can be a lifesaver. If we have a long running function that runs into an error it can be a complete…

Lint/Format/Doc black flake8 interrogate mypy Pipeline Assertions pipeline constructs pipeline as expected nodes pipeline has minimum nodes test minimum tags test alternate tags Catalog Assertions test catalog follows naming structure Node Tests test function does the correct operations on test data Great Expectations

Dynamically generate kedro pipelines with yaml or script Inspiration dag-factory

simple landing page https://swyx.io joel on software recent reading lists More from waylon just above footer 4x2 grid link strategy latest post next/prev similar tags search in nav tag stickers simple cards? bookmarks? nav style stinks single post template flat routes no need to /blog /notes post types 🌳 full 🌱 budding 🖊 Note 💻 hot tip usage of tags MDX stories slides ⚠ ❌ ✔ kedro viz charts inlink component https://joshwcomeau.com/ auto-card oneline links meta posts about uses how site is built how to search stories TODO # review package.json update package.json Done # ahrefs fix canonical urls fix broken inlinks convert to one post template

Nesting loops inside of each other in python makes for much harder code to understand, it takes more brain power to understand, and is thus more error prone than if its avoidable. One issue with this complexity is that toy examples may make sense, but most real example will grow and become more deeply nested over time. Avoiding this complexity from the start can help simplify the project in the future. setup # Lets take a pretty simple example where we are using a ficticious library to get some sales data for our transportation company. The api allows us to fetch teh sales data for one class of vehicle and one region at a time. ❌ Nesting Loops # We have setup to fetch our data with two lists that represent the vehicles and regions that we want to analyze. We know that we need to make a call to for every vehicle and region so nesting loops is a very common first solution to jump to. itertools.product # Python provides us with the beautiful itertools module that allows us to prepare our…

Learn Kedro in 5 days Day 0 Setup # vm install python editor Day 1 # kedro new kedro viz Day 2 # catalog filter catalog load data fsspec Day 3 # pipeline nodes Day 4 # filter pipeline run partial pipeline Day 5 # kedro docker GitHub Actions Advanced Kedro # hooks custom datasets modular pipelines

Upcoming posts to the blog. Have an idea edit this post, submit a PR and we will talk. 🧠 # Kedro run changed How I manage Environments My Data Workflow. Daily Schedule desk keeb Material Shell Why blog search with fuse.js Testing a blog with ahrefs matrix testing in github actions Think like a Senior Dev Editor # tmux vim shortcuts gitui fzf devinsideyou Core # gracefully adopt kedro catalog - in progress pipeline kedro Silent Logger Templated config loader params/env 10 reasons you shouldnt use kedro 10 reasons to use filter viz Steel-toes env Why framework How I write pipelines when I write pipelines pipeline node templates Convert a notebooks to pipelines Testing Pipelines professional python cookiecutter flake8 black mypy pre-commit click pytest git parametrize environment variables My top pandas methods Actions How I cut my gatsby deployment in half VSCODE update plugins I use uses python # What does if name == “ main ”: do? ternary operators check if file exists calling external…