Published

With the realease of came a new module in the project template. Here are some notes that I learned while playing with this new module. migrating to # create a file create a function in that mirrors the register_pipelines method from your module do not bring the decorator remove register_pipelines method on your class You should now have something that looks like this in your. pipeline_registry only works in Conflict Resolution # What happens If I register pipelines in both places I was not able to find any official documentation on how conflict resolution worked so I stepped into a project and added to both my and file. I noticed that it would pick up pipelines from both modules, but pipelines from always take precedence. The entire duplicate pipeline will be over written by the one from. kedro automatically merges pipelines from both hooks.py takes precedence Ready to update # In my experience there were no issues upgrading from to. I would reccomend only having one so decide to migra…

People who are quick to toss team members under a bus are not well trusted or highly thought of and it will lead to some toxic team dynamics. Building Steam # While collaborating on any project there are going to be decisions made that aren’t necessarily your favorite, during a summer internship my mentor made some decisions that I was not on board with, but I accepted his wisdom and moved forward with little push back. Full Spead Ahead # During a review, leadership showed interest in the option that I wanted to go towards. I was quick to jump up and say I told you so right then and there and pitch reasons why my idea was so much better. The Aftermath # I walked out so proud that leadership recognized my idea, but I will never forget the look on my teammate’s face and what he told me after the meeting. “Wow quick to throw me under the bus in front of everyone 😢”. I ~ my teammate Realizing what Happened # It was at this moment that I threw my mentor, a good friend in front of a bus that…

I create this blog with one person in mind, me. There are others like me # This is not completely selfish, as there are likely many others out there that think similarly to me. Everyone comes from different backgrounds and varying levels of experience. In no way do you need to be an expert to create content others will benefit from. Accurate # I am as accurate as possible. I don’t know everything, and If I waited for that to happen I would never post, or write at such a high level no one else (including me) would ever want to read. Generally I know that I am directionally accurate for most of the article. In fact there is often a part of the article that I don’t feel the best about. It may have been some code that could have been better. I try to point these things out. Often when I am and point out where I know there is improvement to be had I get the most welcoming comments, “that was just fine how you did that”, or “I tried that myself and saw a better abstraction. Create the articl…

How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It’s only a total of 35 lines of python, 8 in and 27 in. 📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner. Minimal Kedro Pipeline # I have everything for this post hosted in this gihub repo, you can fork it, clone it, or just follow along. Installation # Caveats # This repo represents the minimal amount of structure to build a kedro pipeline that can be shared across projects. Its installable, and drops right into your or modules. It is not a runnable pipeline. At this point I think the config loader requires to have a logging config file. This is a sharable pipeline that can be used across many different projects. Usage # Implemantation # This builds on another post that I made about creating the minimal python package. I am not sure if it should be called a package, it’s a module, but what do you call it after you build…

This is a post that may be a work in progress for awhile, Its a collections of thoughts on managing my blog, but could be translated into anythiung that is just a collection of markdown. Listing things # posts tags draft posts data # frontmatter filepath content template html render content # Markdown.Markdown support extentsions frontmatter cleaning. # provide ways to hook in or clean up the frontmatter Markata.Markata methods # load render save Markata.Post methods # load render save Markata plugins # before_load before_post_load after_load after_post_load before_save before_post_save after_save after_post_save Markata plugins # cleanse_frontmatter html_feed json_feed rss_feed save_posts CLI #