Posts tagged: kubernetes Archive

kubernetes is beautiful r/kubernetes

2026-03-24T02:29:40Z

This is a fantastic progression through kuberentes concepts. From running a pod, to making it resiliant, holding secrets, accepting traffic, and autoscaling.

dot dockerenv

2026-03-18T10:07:16Z

Today I learned that docker creates an empty /.dockerenv file to indicate that you are running in a docker container. Other runtimes like podman commonly use /run/.containerenv. kubernetes uses neither of these, the most common way to detect if you are running in kubernetes is to check for the presence of the KUBERNETES_SERVICE_HOST environment variable. There will also be a directory at /var/run/secrets/kubernetes.io/serviceaccount that contains the service account credentials if you are running in kubernetes.

diff kubernetes manifest with cluster

2026-02-05T09:37:39Z

Like a dufus this morning I did a hard reset on a git repo for getting I was working on a manifest for. You see I generally use argo, but occasionally I have no idea what I am doing or want yet and I start raw doggin it, fully aware that I’m going to just nuke this namespace before getting it into a proper argocd.

I was overjoyed when I found out that you can diff your manifests with live production using the kubectl diff command. It uses standard diff so you can bring all your fancy diff viewers you like.

# regular manifest
kubectl diff -f k8s/shots -n shot
# kustomize
kubectl diff -k k8s -n go-waylonwalker-com
# using a fancy diff viewer
kubectl diff -f k8s/shots -n shot | delta
# using an even fancier diff viewer
# pinkies out for this one
kubectl diff -f k8s/shots -n shot | delta --diff-so-fancy

Now I can get those changes back that I thought I lost, and apply updates with confidence knowing what is about to change.

AI, DevOps, and Kubernetes: Kelsey Hightower on What’s Next - ...

2025-12-13T10:07:15Z

Kelsey has a really good lightbulb moment here about platform engineering.

“if you had to do all the deployments for the entire company what questions would you ask of the development team?”

That’s your api, your platform, this is your product as a platform engineer. It’s not images, docker, terraform, hcl, yaml, kubernetes, It’s building out the right api for your company to deploy its products effectively.

https://www.youtube.com/watch?v=HdUbTyvrfKo&t=429s

timestamped

The Right Reasons To Run Kubernetes In Your Homelab

2025-12-10T09:48:14Z

Running kubernetes in your homelab is a fantastic way to learn, explore, express yourself, and run services that you use.

The Right Reasons To Run Kubernetes In Your Homelab #

There are not many

You want to learn kubernetes
You like kubernetes
You want to learn to scale

There are also The Wrong Reasons To Run Kubernetes In Your Homelab

You want to learn kubernetes #

Homelabbing is a such a great way to learn new skills, deploy real apps that you use. Create new custom apps for your specific use cases that no one else has. You should absolutely run kubernetes in your homelab if you want to learn it.

I would recommend to start locally, pull up kind, minikube, or k3d and start from your local machine before putting it on a server.

When you decide you are ready for a server, you probably don’t need any crazy hardware. You can probably run on some old retired Dell Optiplex or an old desktop someone is throwing out as it no longer runs windows.

You like kubernetes #

Hell Yeah Brother, 100% no better reason to run kubernetes at home than because you enjoy it. I’m with you. There’s nothing quite like having git ops kick in and deploy new services, updates, watching deployments rollover with zero downtime. Watching your cluster heal itself when a node goes down. Never ssh-ing in to do deployments. Still owning your entire hardware.

You want to learn to scale #

This is probably a stretch reason, maybe not a good one, there are probably better ways, but here we go.

Don’t claim that you need scale in your homelab, you don’t. But it sure is fun to run a cluster of nodes, and load balancing services that run across them. Solving these hard problems to scale across machines is hard. There’s no way around it, there’s a lot to think about. Doing so in a low stakes environment that you have skin in the game is a great way to learn.

I run kubernetes in my homelab #

I run it and I really like it

What flavor of autism did you guys get, I got the kind where I run kubernetes in my basement.

Here are some things I really like about it, and Yes I know you can achieve most of these without kubernetes.

I don’t have to ssh, hardly ever.
I can see everything I’m running, and its defined in a manifest
k9s is amazing, and I use it all the time.
- shell into running pods
- restart deployments
- scale deployments
- trigger cronjobs
- watch logs
- I can scale our minecraft server to 0 in seconds if we are in a different season of life
ArgoCD is amazing
- I ❤️ gitops
Ingress just works
Longhorn
- makes snapshots and backups easy
- makes multi node easy
zero-downtime deployments
self healing health checks

I mostly do very simple things, deployments with a container, a volume and ingress. Probably things that you could easily run on fly.io. Theres nothing really fancy. I just like how easy this setup works for me.

minecraft server memory

2025-12-10T08:46:36Z

I learned to today that setting MEMORY on your minecraft server causes the JVM to egregiously allocate all of that memory. Not setting it causes slow downs and potential crashes, but setting INIT_MEMORY and MAX_MEMORY gives us the best of both worlds. It is allowed to use more, but does not gobble it all up on startup.

In this economy we need to save all the memory we can!

Here is a non-working snippet for a minecraft server deployment in kubernetes.

      containers:
        - name: dungeon
          image: itzg/minecraft-server
          env:
            - name: EULA
              value: "true"
            - name: INIT_MEMORY
              value: "512M"
            - name: MAX_MEMORY
              value: "3G"

and in docker compose

  dungeon:
    image: itzg/minecraft-server
    environment:
      EULA: "true"
      INIT_MEMORY: "512M"
      MAX_MEMORY: "3G"

check your kubeconfig expire time

2025-12-08T20:59:27Z

Today I learned an important lesson that you should periodically check on your kubeconfigs expiration date. It’s easy to do. You can ask for the client-certificate-data from your kubeconfig, decode it, and use openssl to get the expiration date.

kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' \
  | base64 -d 2>/dev/null \
  | openssl x509 -noout -dates

Note

This will only work for the first user, if you have more than one user or context defined in your kubeconfig you will need to adjust.

The Wrong Reasons To Run Kubernetes In Your Homelab

2025-12-06T09:46:47Z

Running kubernetes in your homelab is complex, time consuming, there are almost no docs to help you (homelab focused docs for things you want to install), and nothing is copy paste. You have to make everything happen yourself.

The Wrong Reasons To Run Kubernetes In Your Homelab #

I run compose and think kubernetes is the next logical step
Techno Tim runs it
I heard it’s what cool kids do
Kubernetes BTW
Talos Linux looks cool
I found a cool helm chart on GitHub
I need scale

There are also The Right Reasons To Run Kubernetes In Your Homelab.

I run compose and think kubernetes is the next logical step #

No it’s not. It’s much different than running docker, compose, swarm. It’s meant for scale, it’s complex, it’s made for enterprise, not your local development or your homelab. It can do these things, it can do them quite well, but it’s not the target audience.

Techno Tim runs it #

I heard it’s what cool kids do

You need to rethink who the cool kids are, touch some grass. Tim also does it for his job, he likes it, he knows it, he wants to lean on it and learn more.

Kubernetes BTW #

Kubernetes does not make you look cool, it makes you look like you are trying to over optimize and over engineer your life. It’s not worth it, in fact nothing in life is worth worrying about what everyone else thinks of you.

Talos Linux looks cool #

Talos is an S tier OS wherever you deploy it. It is a secure, minimal, kubernetes first OS. They also have some really great people working there putting Talos in some really cool places like backpack or Apple Power Mac

I found a cool helm chart on GitHub #

No you didn’t. Everything in homelab is compose first. A few things have a k8s option, but almost nothing is k8s first.

I need scale #

No. You’re homelab does not need scale. If you think it does, you have some real shit hardware, some bad optimizations, or somehow you have a startup you need to launch cause you got more users than most.

k3s system-upgrade minor by minor

2025-12-05T09:25:39Z

The k3s system-upgrade controller is a fantastic tool for upgrading k3s automatically. It has done a fantastic job for me every time I’ve used it. Today I ran it on a cluster that needed to upgrade several minors and I learned that the controller does not pick up on changes to the channel url if you change from minor to minor.

The solution I came up with was to name the plan with the version it supports. Then on each patch upgrade, change both the plan name and the channel. I use gitops with argocd, it automcatically cleaned up old plans, created new plans, and the system-upgrade-controller picked up the plan and started applying immediately.

# Server plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan-v1.33 # <- This is important if you want to change the channel name
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: In
      values:
      - "true"
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  channel: https://update.k3s.io/v1-release/channels/v1.33
---
# Agent plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: agent-plan-v1.33 # <- This is important if you want to change the channel name
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: DoesNotExist
  prepare:
    args:
    - prepare
    - server-plan
    image: rancher/k3s-upgrade
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  channel: https://update.k3s.io/v1-release/channels/v1.33

I’d love to see a better way if you have a way to upgrade through minors, or manually control the minor of your cluster let me know.

K8s Diagram Builder - Free Visual Kubernetes Architecture Desi...

2025-12-01T20:10:55Z

K8s Diagram Builder - Visual Kubernetes YAML Generator

Free Kubernetes diagram builder with drag-and-drop design. Auto-generate production-ready YAML for Ingress, Services, Deployments, ConfigMaps, Secrets & more. No signup required.

K8s Diagram Builder · k8sdiagram.fun

This looks like great prototyping tool for k8s. I too often ask ai to get me going with the things I need. I’ve used k8s long enough that I can generally remember all the things I need, roughly where they go, would probably forget a few things and need to iterate, but I cannot remember exactly what goes where and need examples at a minimum. I need to give this a go from desktop and see if it will work for me. Right now looking through mobile looks promising.

PETaflop cluster - Justin Garrison

2025-11-12T23:44:41Z

PETaflop cluster

AI is a pain in the back.

Justin Garrison · justingarrison.com

Justin makes the coolest kubernetes clusters wishing I could see it in the flesh at Kubecon.

Performance Difference between RWX and RWO volumes · longhorn/...

2025-08-16T00:13:56Z

Performance Difference between RWX and RWO volumes · longhorn longhorn · Discussion #6964

Hey all, because of some internal testing I made a couple of experiments on our Cluster related to performance of RWX and RWO volumes. Because this might be of interest to some people I thought I s...

GitHub · github.com

Interesting longhorn storage performance test, author does highlight right away that this is a simulation and not a REAL test. I did not fully understand the storage semantics before reading through this.

RWO - Always presents a filesystem ext4 or xfs
RWX/ROX - Always presents a network share nfs to the pod.

This is an important distinction for applications that use sqlite or a tool on top of sqlite such as diskcache. With sqlite it is not recomended to run over nfs due to missing required file locking mechanisms.

Longhorn storage still provides a lot of benefits to these applications as the storage is automatically replicated, if the node that your application is running on goes offline a new pod will start on an existing node. If you have planned downtime, you can cordon and drain a node. Since the data is available in another location you will be able to start a new pod on anther node. barring your PodDisruptionBudget settings, taints, and affinity, this may happen automatically.

Should I kubernetes My Homelab

2025-08-11T09:28:16Z

Yes

Ok we should probably dive deeper into this, but good chance if you are here and have made it this far you it would probably be a fine choice. The choice is quite time and skill dependant.

Time #

First thing up, if you like copy pasting thing into your homelab, changing a few config options, but mostly running it as the docs instructed, kubernetes is not for you. The homelab/self hosting space is heavily reliant on docker compose, 90% of the things you want to run will likely have a docker command, and likely a docker compose example that you can copy paste and get running right away. Maybe 5% of projects have something for kubernetes, you Will have to do it yourself.

Kubernetes is very DIY in the self hosting space, and not very plug and play.

Skill #

Meet Gor | Just Fucking use kubernetes

2025-08-02T19:53:45Z

External Link

meetgor.com

Sometimes, all you need is a mindset shift, a blocker in your mind that holds you back from doing certain things. And for me, I have consumed enough tutorials and posts about Kubernetes, that I need to put to use and create. I have been stuck in the learning cycle, lets push to prod with kubernetes.

This hurts. I know others with this learning style that need to see the full picture before actually doing something with new tech. The way I first got into kubernetes I was looking for the easy route and somehow k8s came up several times as a suggested route Looking for a Heroku replacement, What I found was shocking!, So I dove in head first with k3s and kompose. What I found was that it was not all that hard once you start to see how the pieces fit together, no amount of reading tutorials would have gotten me there.

Does anyone care if you use simple yet fragile bash scripts or heavy weight Kubernetes cluster for just clicking buttons and creating and updating rows in a database? No!

You know what, let’s fucking use Kubernetes.

Let’s Gooo. Use what is right for you and stop parroting kubernets is hard, heavy, for big companies, maybe actually try it first.

Quoting Meet Gor | Kubernetes isn't for you

2025-08-02T19:44:18Z

External Link

meetgor.com

If you want to use it for the purpose of learning it, please do use it.

Kubernetes as usual is a tool like others, you can’t use one tool everywhere. Where bash scripts work, they just work, where they don’t they fall apart too, kubernetes works like a charm.

Use your grug brains a little and choose wisely! In the end, who the hell cares if you use kubernetes or bash scripts to scale if your users are happy?

Well Said!

Kubernetes Isn't for You

2025-08-02T19:40:42Z

Kubernetes Isn

Kubernetes isn

sliplane.io

This post feels like it was written by someone who has never tried kubernetes, someone who reads twitter, listens to t3.gg and thePrimeagen (who cant even container let alone kubernetes). If you cant run linux, use bash, build your own docker images, run docker comfortably. If infra is not your thing kubernetes is probably not for you.

Kubernetes Was Built for Google

Just like how react was built for facebook to solve facebook problems with many teams contributing effectively to the same interactive interfaces. Turns out that react is actually a pretty good product if you have a highly interactive page, and if this is your bread and butter, you can make overly heavy static sites with too much build very effectively. It works and runs much of the internet now.

We are getting serious. We need serious tools. Big companies use Kubernetes. We should too. It feels more professional. It sounds like we know what we are doing.

If anyone uses these reasons to pitch kubernetes to me they don’t belong in a position to make any sort of decision. The first one could be a heading with maybe something under it.

But Kubernetes should not be your first infrastructure step. It should be a response to real pain, not an emotional milestone

As with anything, it depends! Keeping with the react example, if you have a team with these skills its a solid choice, maybe its overkill, but you got this skills to start here then go for it.

If you have never given something a real shot then don’t be writing articles shitting on the tech. Actually if you do not have a deep expertise in it you probably should not be writing articles shitting on other tech. If you are the CEO of an alternative, you definitely should not be writing articles shitting on your competition. just build the biggest fucking building in town .

If you are kube curios give kind and kompose a try, you will be surprised at how quickly you can get something up and running in kubernetes. You might be surprised at how easy it is to remotely manage, add gitops workflows with argocd. Give k9s a try and you can see all of your nodes, services, ingress, pvcs, EVERYTHING you have deployed and its status in one easy to use TUI.

I avoided kubernetes for a long time because articles like this told me to and I never gave it a fair try.

Check out just fucking use kubernetes for a satirical opposite take.

just fucking use kubernetes

2025-07-01T12:46:33Z

You want to run containers?

JUST FUCKING USE KUBERNETES.

Obvious satire

If you don’t like harsh language this is not the post for you. Obviously ripping off motherfuckingwebsite.

Seealso

Some more serious posts

“But it’s complicated!” #

Shut up. Close twitter and fucking do something. Life is complicated. You know what else is complicated? Email. DNS. Life. Kubernetes is the least painful way to orchestrate containers at scale. Docker Compose is for your laptop.

Swarm is dead.
Nomad is just sad.
Systemd units? Get out of here.

“But my app is small!” #

SO IS YOUR AMBITION. You could write a bunch of bash scripts and hope they work on prod. You could SSH into servers and handcraft your infra like it’s 2011. Or you could just fucking use Kubernetes and sleep at night.

I can just throw my script in crontab #

Tell that to your boss when your cronjob failed 16 times in the last week without anyone noticing. kubernete makes it fucking simple, want retry ask for it. Hanging script, activeDeadlineSeconds that bitch. Connecting to six other services in your shitty ass infrastructure this shit retries automatically.

“I don’t need autoscaling!” #

Cool. Tell that to your boss when the CEO tweets your link and the site goes down harder than your last date.

“But YAML is ugly!” #

So is your Terraform, your Ansible, your Prometheus config, your custom CI/CD scripts written in Bash, and the spaghetti you called a monolith before you went microservices and made it worse.

“Kubernetes is too heavy!” #

Compared to what?

Your handcrafted, artisanal, single-node LXC setup running on an Intel NUC from 2014?

Heard of k3s? k0s? No. These fuckers will have you running kubernetes running on your grandpas goddamn gateway 2000 right next to AOL messenger without splooging out the the ashes of his Marlboro Reds.

SSH? #

You don’t need no goddam ssh to install your 200lb gorilla editor so you can hand edit your init files and carefully contruct your init system. This is kubernetes, you use the fucking api, all you need is a connection and a kubeconfig. This motherfucker runs containers so you can keep your bitch-ass editor where it belongs, off the fucking host machine!

“What if it’s overkill?” #

What if YOU are underkill?

“How do I do zero-downtime deploys?” #

Probes my dude, you fucking probe your shit. Rolling out a new deployment kubernets won’t cut over to your broke ass release if that shit don’t pass. No more writing janky scripts that SSH into prod and run git pull while praying to the CI/CD gods.

“What if I still fuck it up - How do I roll back?” #

k9s is your best friend, pop that shit open find your broke ass deployment, jump owner to the replicaset and roll that bitch back to the working shit.

I need to scale #

This shit is built in, add a goddamn replica or 6 for fuck sake, need autoscaling use the HPA. This aint your granpas hand fucking crafted pet server, its fucking cattle. Load balancing just fucking happens, don’t think about it just use it, and it will work for your six goddamn friends that actually use your shit.

I want gitops #

Let me introduce you to argocd, this fucker uses helm so one fucking command and your ass is on the beach while your dev team deploys their own shit.

USE KUBERNETES #

It fucking works.
Everyone else is using it.
There are like 500 open-source projects built just to make it easier.
It runs on your laptop, your server, the cloud, and inside your dreams.
It will make your resume better.

Not convinced? #

Here’s your alternative stack:

A bash script that restarts Docker when it dies.
A Makefile that deploys via SCP.
A cron job that prays to the log gods.
A wiki page explaining how to debug your hand-rolled bullshit.
You. Crying.

So yeah #

Save yourself. Save your team. Save your infrastructure.

JUST #

FUCKING #

USE #

KUBERNETES #

(or don’t, and become a DevOps cautionary tale)

Seealso

kubernetes node labels

2025-05-28T18:49:19Z

If you need to target a specific k8s node in the cluster, you can use labels. You want to treat your nodes as much like cattle as you can, but sometimes budgets get in the way. You might be like me and just run any free hardware you can get in your cluster, or you might have some large storage or gpu needs that you can’t afford to put on every node in the cluster.

kubectl get nodes --show-labels

# add the bigpool label
kubectl label node k8s-1 bigpool=true
kubectl get nodes --show-labels

# remove the bigpool label
kubectl label node k8s-1 bigpool-

To use the label in a pod set spec.nodeSelector to the label that you applied.

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  containers:
  - name: busybox
    image: busybox
  nodeSelector:
    bigpool: "true"

k8s-monitoring-helm/charts/k8s-monitoring/docs/examples/privat...

2025-05-24T00:58:59Z

k8s-monitoring-helm/charts/k8s-monitoring/docs/examples/private-image-registries/globally/values.yaml at main · grafana/k8s-monitoring-helm

Contribute to grafana/k8s-monitoring-helm development by creating an account on GitHub.

GitHub · github.com

k8s-monitoring requires setting imageregistry and pullsecrets twice

global:
  image:
    registry: my.registry.com
    pullSecrets:
      - name: my-registry-creds
  imageRegistry: my.registry.com
  imagePullSecrets:
    - name: my-registry-creds

Chris Biscardi's Digital Garden

2025-05-12T18:05:33Z

External Link

christopherbiscardi.com

Interesting take on kubernetes from a front end perspective. All valid arguments to me, and really the answer to any do you need to any specific implementation of tech is probably no. We got along just fine before k8s ever existed and you still can, but its really nice in a lot of cases. If your skills lean toward backend or infrastructure I encourage you to give it a try.

k8s distros #

There are a lot of beginner friendly k8s distros that you can setup with relative ease, kind and k0s are great for single node, If you want multi-node k3s is what I generally use. If you want a very lightweight OS that you only interact with through an api, and has a very small attack surface talos is an amazing product.

When else might you want k8s #

Internal, on-prem, self hosted. If you are trying to avoid the cloud for cost, rules, regulations, red tape, kubernetes is a great option to manage your container workflows yourself without needing to have a cloud budget, get approvals and sign offs on running workflows in a public cloud.

grafana dashboards as k8s configmap

2025-05-06T20:24:35Z

I’m trying to learn proper logs, monitoring, otel, and grafana. Today I imported a bunch of pre-made k8s dashboards and made a few of my own for specific apps, and it made me want to know how I can turn my own custom dashboards into infrastructure as code. Turns out grafana makes it pretty easy to do this, if you have the grafana dashboard sidecar running. It will pick up any ConfigMap with the grafana_dashboard label and import it.

Go to Dashboards -> Pick a Dashboard -> Export -> JSON.

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-dashboard
  namespace: meta
  labels:
    grafana_dashboard: "1"
data:
  my-dashboard.json: |
    {
      "annotations": {
        "list": [
      ...
      "uid": "fel2uhjhepg5ce",
      "version": 3
    }

Diun

2025-04-07T18:33:20Z

Diun

Receive notifications when a Docker image is updated on a Docker registry

crazymax.dev

Diun, looks like a very interesting tool to monitor for image updates, it does not make any change, it only makes notifications. This feels like an easy start to getting image updates started with low effort, keep git ops, but requires manual updates. I see this as a tool that would be a great start and pair well with automated image updaters to ensure they are working as expected.

Keel

2025-04-07T18:30:49Z

Keel

Kubernetes Operator to automate Helm, DaemonSet, StatefulSet & Deployment updates

keel.sh

Keel looks interesting, I might give it a try as a simple image updater. I’m unsure if it fits my gitops patterns though. I like to keep everything defined in git, I don’t like drift outside of that so Keel might not be the thing I want.

Changing k8s Storage Class - Migration Job

2025-04-04T09:57:50Z

I’m setting up longhorn in my homelab, and I ran into an issue where I initially setup some pvcs under longhorn, and later realized that to get longhorn to snapshot and backup I needed to hand edit volumes after the fact or change storage class. I’m all in on gitops so option 1 was not an option. So changing storageclass it is.

Now the issue is that you CANNOT mutate storageclass on a provisioned pvc, it is an immutable attribute.

Migration Job #

This migration job will create a new pvc with the new storageclass and move the data from the old pvc to the new pvc.

Existing Pods

 This migration job will not work if you have a pod using the old pvc.  You

will need to shutdown the pod and delete it.

# old pvc with longhorn storageclass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: site-pvc-longhorn
  namespace: waylonwalker-com
spec:
  storageClassName: longhorn-backup
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
# new pvc with longhorn-backup storageclass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: site-pvc-longhorn-backup
  namespace: waylonwalker-com
spec:
  storageClassName: longhorn-backup
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
# migration job to move the data to the new pvc
apiVersion: batch/v1
kind: Job
metadata:
  name: pvc-migration
  namespace: waylonwalker-com
spec:
  template:
    spec:
      containers:
      - name: pvc-migration
        image: ubuntu:22.04
        command: ["/bin/bash", "-c"]
        args:
          - |
            echo "Starting migration..."
            cd /data
            cp -av source/. destination/
            echo "Migration complete!"
        volumeMounts:
        - name: source-vol
          mountPath: /data/source
        - name: dest-vol
          mountPath: /data/destination
      restartPolicy: Never
      volumes:
      - name: source-vol
        persistentVolumeClaim:
          claimName: site-pvc-longhorn
      - name: dest-vol
        persistentVolumeClaim:
          claimName: site-pvc-longhorn-backup

Apply the manifests and wait for the job to complete.

kubectl apply -f pvc-migration.yaml

Cleanup #

I had chatgpt create me a script to help me find what is using the pvc so that it can be deleted.

#!/bin/bash

NAMESPACE="waylonwalker-com"
PVC_NAME="site-pvc-longhorn-new"

echo "⏳ Checking if PVC exists..."
kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" || {
  echo "✅ PVC already deleted."
  exit 0
}

echo "🔍 Describe PVC..."
kubectl describe pvc "$PVC_NAME" -n "$NAMESPACE"

echo -e "\n🔗 Checking if any pod is using this PVC..."
kubectl get pods -n "$NAMESPACE" -o json | jq -r \
  --arg PVC "$PVC_NAME" \
  '.items[] | select(.spec.volumes[].persistentVolumeClaim.claimName == $PVC) | .metadata.name'

echo -e "\n🧹 Checking finalizers..."
kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o json | jq '.metadata.finalizers'

echo -e "\n🔎 Checking associated VolumeAttachment..."
PV_NAME=$(kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.volumeName}')
echo "🔗 PVC is bound to PV: $PV_NAME"

kubectl get volumeattachment -A -o json | jq \
  --arg PV "$PV_NAME" \
  '.items[] | select(.spec.source.persistentVolumeName == $PV) | {name: .metadata.name, node: .spec.nodeName, attached: .status.attached}'

echo -e "\n🚀 Done."

I had still had cronjob pods completed, so I had to delete them first.

🔗 Checking if any pod is using this PVC...
pvc-migration-ndv92
waylonwalker-com-cronjob-29057840-8s92p
waylonwalker-com-cronjob-29057850-4rvm9
waylonwalker-com-cronjob-29057860-6g89j

kubectl delete pod pvc-migration-ndv92 -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057840-8s92p -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057850-4rvm9 -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057860-6g89j -n waylonwalker-com

AI workloads on Talos Linux - Sidero Labs

2025-02-15T01:27:54Z

AI workloads on Talos Linux

Companies are exploring how to run GPU accelerated workloads on Kubernetes.

Sidero Labs · siderolabs.com

cool article for setting up talos linux with an nvidia gpu. What a wild world it we are living in where these devices that started out being only for hardcore gamers are becoming commonplace in servers and slowly entering the homelab space.

Why is Kubernetes everywhere? | Kelsey Hightower - YouTube

2025-01-29T22:28:21Z

Kelsey says several times in this interview, you don’t need kubernetes. If you are running one node you don’t need kubernetes. My question though is, would you use kubernetes? Ya I get it if you are a web developer, data scientist, backend dev, but if you are looking to bee a whole ass engineer, or infrastructure engineer, you know kubernetes, Should you use kubernetes on single node?

poolers.postgresql.cnpg.io CRD metadata.annotations Too long ·...

2025-01-21T23:06:21Z

poolers.postgresql.cnpg.io CRD metadata.annotations Too long · Issue #325 · cloudnative-pg/charts

Unable to deploy helm chart using ArgoCD. Getting following error Failed sync attempt to : one or more objects failed to apply, reason: CustomResourceDefinition.apiextensions.k8s.io "poolers.postgr...

GitHub · github.com

I’ve never seen or needed to use a serversideapply in kubernetes before, but I ran into this same issue in my k3s homelab while installing cloudnative-pg.

You can do it with argo

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
syncPolicy:
syncOptions:
- ServerSideApply=true

and you can do it with kubectl

kubectl apply --server-side --force-conflicts -f cnpg-1.25.0.yaml

Keycloak

2025-01-19T11:34:32Z

Keycloak

Keycloak - the open source identity and access management solution. Add single-sign-on and authentication to applications and secure services with minimum effort.

Keycloak · keycloak.org

Keycloak looks like an interesting way to setup sso. It’s part of the cncf so it’s got a good backing. I want something better for argo workflows and this might be it. I’m curious what else I can tie into it.

GitHub - ngalaiko/tree-sitter-go-template: Golang template gra...

2025-01-07T00:49:13Z

GitHub - ngalaiko/tree-sitter-go-template: Golang template grammar for tree-sitter

Golang template grammar for tree-sitter. Contribute to ngalaiko/tree-sitter-go-template development by creating an account on GitHub.

GitHub · github.com

This setup fixed my nvim syntax highlighting in helm templates.

slow nfs performance

2025-01-02T20:23:10Z

I’m running a two node k3s cluster at home, I thought I could simply mount an nfs share on each worker node, and essentially have the same storage accross all nodes. I’m already learning why this is not reccommended.

Slow #

I’ve been running some cronjobs and argo workflows on the second node for awhile, these are things that run in the background and I don’t care if they take a bit longer to keep my master node freed up for more critical work.

I just started trying to build this site in a cronjob, It was taking 20 minutes to build, and something I noticed was that markata was taking minutes to run glob ( search for files ), normally this happens in a few ms and I never notice this step.

I just moved into the master node and the results were wild at ~30x faster

Permissions #

I have seen where you can get diffent permissions on the nfs share based on user id. Since I’m homelabbing here I only have one user per machine. As you step into enterprise level VMs with tighter controls and dozens of users for all the different services that might run on it.

I’ve ran into maybe one issue where I was root in one place and not another, other than that it’s been fine.

And it only got better #

As the cache was warm subsequent runs only got better.

I just checked again and we are now 80x faster

Conclusion #

I don’t have the answers yet, it might be my network, it might be nfs settings, it might be ext4 filesystem. I have some things to try, but what I do know is that it is not as easy as I thought it would be just to have the same file system mounted on both ends and share data between nodes.

I might even go with a complete alternative and use minio as a storage backend, and sync the files in on each run, this will add some latency to do the sync each time though.

Changelog on X: "🗣️@dhh on Kubernetes' migration pitch: “Oh, w...

2024-11-25T08:30:44Z

@changelog) on X — 🗣️@dhh on Kubernetes' migration pitch:

“Oh, we’re on Amazon? Flip the switch, George! Let’s move over to GCP.”

Vs reality:

"Flip the switch, George! We’ll start having the meetings about the meetings about how we can move out four years from now."" loading=“lazy”>

Changelog (@changelog) on X

🗣️@dhh on Kubernetes' migration pitch:

“Oh, we’re on Amazon? Flip the switch, George! Let’s move over to GCP.”

Vs reality:

"Flip the switch, George! We’ll start having the meetin…

X (formerly Twitter) · x.com

switching cloud providers, theres no easy way. K8s was supposed to get us there, haha, the deep integrations with each vendor just keep locking us in

Inside Argo: Automating the Future - YouTube

2024-11-22T08:29:00Z

This is a really amazing documentary of argocd. I got into k8s pretty late in the game. Which is pretty typical for me. As I went to use k8s for the first time i was using workflows, then cd. both of these tools had a level of polish that made them seem like they had been there forever and not quite as young as they actually are.

I thought it was interesting how they focused on how the name must be two syllables or less, start with a or b, logo needs to be cutesy funny and recognizable seemed interesting, but puts them at the top of lists and makes them look like they’ve been there forever.

Cluster API book

2024-11-05T10:17:37Z

Provider List - The Cluster API Book

cluster-api.sigs.k8s.io

Cluster API book

k3s config after first install

2024-09-15T16:57:41Z

After first setting up a new k3s instance your kubeconfig file will be located in /etc/rancher/k3s/k3s.yaml.

You cans use it from here by setting $KUBECONFIG to that file.

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Or you can copy it to ~/.kube/config

cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

If you have installed k3s on a remote server and need the config on your local machine then you will need to modify the server address to reflect the remote server.

scp user@<server-ip>:/etc/rancher/k3s/k3s.yaml ~/.kube/config

Warning

only do this if you don’t already have a ~/.kube/config file, otherwise copy it to a new file and set your $KUBECONFIG env variable to use it.

Now you will need to open that file and change the server address, making sure to keep the port number.

apiVersion: v1
clusters:
  - cluster:
      certificate-authority-data: ****
      server: https://<server-ip>:6443
    name: default

k8s kustomize diff

2024-07-06T09:42:42Z

I’ve started leaning in on kubernetes kustomize to customize my manifests per deployment per environment. Today I learned that it comes with a diff command.

kubectl diff -k k8s/overlays/local

You can enable color diffs by using an external diff provider like colordiff.

export KUBECTL_EXTERNAL_DIFF="colordiff -N -u"

You might need to install colordiff if you don’t already have it.

sudo pacman -S colordiff

sudo apt install colordiff

Now I can try out kustomize changes and see the change with kustomize diff.

kubectl dash k

2024-07-05T20:15:11Z

Kubernetes ships with a feature called kustomize that allows you to customize your manifests in a declarative way. It’s a bit like helm, but easier to use. I found this useful.

kustomization.yaml #

In order to use kustomize you need to have a kustomization.yaml, kustomization.yml, or Kustomization file in the directory you are applying.

kubectl apply -k .
error: unable to find one of 'kustomization.yaml', 'kustomization.yml' or 'Kustomization' in directory '/tank/git/kustomize-playground'

The kustomization.yaml file cannot be empty.

❯ kubectl apply -k pod
error: kustomization.yaml is empty

lets make an mvp #

mkdir pod
touch pod/pod.yaml
touch pod/kustomization.yaml

# pod.yaml

# kustomization.yaml

Overlays #

Overlays must point to yaml or a directory with a kustomization.yaml file, and cannot reside inside the same directory causing a circular reference.

Layout #

k8s
├── base
│   ├── deployment.yaml
│   └── kustomization.yaml
├── overlays
│   ├── local
│   │   └── kustomization.yaml
│   ├── dev
│   │   └── kustomization.yaml
│   └── prod
│       ├── special-prod.yaml
│       └── kustomization.yaml

base #

Place all of the common k8s manifests here in the base directory along with kustomization.yaml. You can split them up into separate files or just one file, I’m keeping it to one for simiplicity.

base - kustomization.yaml #

In the base kustomization.yaml file add all of the manifests that you want to use as a resource.

resources:
  - deployment.yaml

overlays #

Now for each separate environment that you want to have create a directory in overlays with a kustomization.yaml file, and any special manifests that only apply to a particular environment.

overlays - kustomization.yaml #

resoures #

resources:
  - ../../base
# any other specific resources, maybe special ones for prod here
  - sealed-dotenv.yaml

images #

images:
  - name: my-image
    newName: docker.io/myrepo/myimage
    newTag: 1.0.0

Kustomize: The Best Way to Manage Your Kubernetes Configs - Yo...

2024-07-03T18:31:47Z

Great intro into kustomize. This helped me get started with kustomize.

kind cluster

2024-07-02T08:01:20Z

kind{.hoverlink} is a very useful tool to quickly standup and teardown kubernetes clusters. I use it to run clusters locally. Generally they are short lived clusters for trying, testing, and learning about kubernetes.

Kind is Kubernetes in Docker, its very fast to get a new cluster up and running. Other than checking a box in docker desktop it is the easiest way currently to get a cluster up and running. I’ve used docker desktop for k8s before I really developed on k8s and it was buggy at the time and sometimes started and sometimes didn’t, when it didnt I had no idea how to fix it. I’d suggest kind as the best option to get a cluster up and running locally.

Not Production #

If you are looking for a production ready cluster this is not it. I really like k3s{.hoverlink}. At the time that I chose k3s it was the most lightweight option that easily supported multi-node clusters.

Starting a kind cluster #

The first step, and maybe only one that you need is to create a cluster and give it a name. This command will edit your $KUBECONFIG file, and set the kind cluster as your default cluster to interact with.

kind create cluster --name <CLUSTER_NAME>

Using podman as a backend #

I use podman as my docker engine, kind works with docker and podman, but docker by default, in order to switch to podman you need to set an environment variable.

export KIND_EXPERIMENTAL_PROVIDER=podman

This will tell kind to use podman as the backend provider instead of docker.

Loading images #

If your images are not publically available from a registry, you can load them in kind using the kind load docker-image command.

kind load docker-image $REPOSITORY:$TAG --name <CLUSTER_NAME>

Note

 the CLUSTER_NAME is the name that you gave kind when you started the kind cluster.

Argocd #

Argocd is a great way to setup gitops workflows in kubernetes. compared to just hand-rolling kubectl apply, argo holds the state and is able to not only apply new and change, but cleanup removed things.

setting up a kind cluster with argocd installed

You can stand up argocd in kind for learning argo or getting a nice visual. But often when I use kind its overkill. The cluster is not long lived, I don’t care if things are not cleaned up, and I want to quickly apply changes without a commit and push to a git repo.

Install sealed-secreats via manifest

2024-07-02T07:54:01Z

Yesterday I realized that I have overlooked the default installation method of the sealed secrets controller for kubernetes kubeseal this whole time an jumped straight to the helm section. I spun up a quick kind cluster and had it up quickly. I can’t say this is any better or worse than helm as I have never needed to customize the install. According to the docs you can customize it with [[ kustomize ]] or helm.

# option if you don't have a cluster try with kind

kind create cluster

curl -L https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.27.0/controller.yaml > controller.yaml

kubectl apply -f controller.yaml

Where DOESN’T curl run with curl BDFL Daniel Stenberg (Changel...

2024-06-26T04:39:38Z

Where DOESN’T curl run with curl BDFL Daniel Stenberg (Changelog & Friends #49)

Daniel Stenberg shares his guiding principles for BDFL'ing curl, gives us his perspective on the state of the internet, talks financial independence, ensuring curl won't be the next XZ & more!

Changelog · changelog.com

Love how Daniel just brushed off the number of netflix tv installs as small beans for curl installs. overall great episode on open source, owning a major component to technology with such a small team.

Releases · stackrox/kube-linter

2024-06-24T22:09:37Z

GitHub - stackrox/kube-linter: KubeLinter is a static analysis tool that checks Kubernetes YAML files and Helm charts to ensure the applications represented in them adhere to best practices.

KubeLinter is a static analysis tool that checks Kubernetes YAML files and Helm charts to ensure the applications represented in them adhere to best practices. - stackrox/kube-linter

GitHub · github.com

A linter for linting kubernetes manifests and help charts.

argoproj/argo-events: Event-driven Automation Framework for Ku...

2024-06-09T19:30:26Z

GitHub - argoproj/argo-events: Event-driven Automation Framework for Kubernetes

Event-driven Automation Framework for Kubernetes. Contribute to argoproj/argo-events development by creating an account on GitHub.

GitHub · github.com

Argo events is an event driven automation framework for kubernetes that can create kubernetes objects among other things based on events. I’ve been using native kubernetes cronjobs to kick off jobs based on a cron trigger.

For instance I am running reader.waylonwalker.com every hour, to rebuild the site and re-deploy it. It takes about two minutes to fetch every rss feed, so this is a nice application of a job compared to a web server fetching the feeds live. Now my posts may be up to an hour stale but they load fast.

Argo events takes event drien architecture to the next level allowing to be triggered by many more things, and do many more things than creating a cron job. I’m definitely thinking about dropping this in my homelab.

eraser-dev/eraser: 🧹 Cleaning up images from Kubernetes nodes

2024-05-25T06:26:09Z

GitHub - eraser-dev/eraser: 🧹 Cleaning up images from Kubernetes nodes

🧹 Cleaning up images from Kubernetes nodes. Contribute to eraser-dev/eraser development by creating an account on GitHub.

GitHub · github.com

This is kinda sick, its a tool to clean up container images in a k8s cluster.

How to Restart All Pods in a Kubernetes Namespace | Boot.dev

2024-04-26T02:59:56Z

How to Restart All Pods in a Kubernetes Namespace | Boot.dev

Where I work, we use a repo-per-namespace setup and so it often happens that I want to restart all pods and deployments in a single Kubernetes namespace. Maybe I want to see the startup logs, or ma...

Boot.dev · blog.boot.dev

As of kubernetes 1.15 there is an easy way to restart all pods in a deployment.

kubectl -n {NAMESPACE} rollout restart deploy

Thanks Lane give him a follow @wagslane

argocd automated sync

2024-04-20T00:36:47Z

![[none]]

---

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kanboard
  namespace: argocd
spec:
  project: default
  destination:
    namespace: kanboard
    server: 'https://kubernetes.default.svc'
  source:
    path: kanboard
    repoURL: 'https://github.com/waylonwalker/homelab-argo'
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true

Manual Upgrades | K3s

2024-04-19T17:51:03Z

Manual Upgrades | K3s

You can upgrade K3s by using the installation script, or by manually installing the binary of the desired version.

docs.k3s.io

You can give k3s an install channel to install stable, latest, or specific versions like 1.26. This is handy to make sure that you install the same version on all of your workers.

curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=latest <EXISTING_K3S_ENV> sh -s - <EXISTING_K3S_ARGS>

scheduling cron jobs in kubernetes

2024-04-08T16:56:46Z

For my reader app I am using cronjobs to schedule my a new build and upload to cloudflare pages every hour. In this example I have built a docker image docker.io/waylonwalker/reader-waylonwalker-com and pushed it to dockerhub. It uses a CLOUDFLARE_API_TOKEN secret to access cloudflare, and the entrypoint itself does the build and upload.

apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: null
  name: reader
  namespace: reader

---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: reader-cronjob
  namespace: reader
spec:
  schedule: "0 * * * *"
  successfulJobsHistoryLimit: 6
  failedJobsHistoryLimit: 6
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: reader-container
              image: docker.io/waylonwalker/reader-waylonwalker-com:latest
              env:
                - name: CLOUDFLARE_API_TOKEN
                  valueFrom:
                    secretKeyRef:
                      name: cloudflare-secret
                      key: cloudflare-secret
          restartPolicy: OnFailure

Sealed Secrets

2024-03-28T06:07:06Z

External Link

sealed-secrets.netlify.app

kubeseal is a pretty simple to get started with way to manage secrets such that they can be stored in a git repo and be picked up by your continuous delivery service.

Sealed Secrets provides declarative Kubernetes Secret Management in a secure way. Since the Sealed Secrets are encrypted, they can be safely stored in a code repository. This enables an easy to implement GitOps flow that is very popular among the OSS community.

kubernetes kubeseal

2024-03-27T20:02:57Z

In my homelab kubernetes cluster I am using kubeseal to encrypt secrets. I have been using it successfully for a few months now wtih great success. It allows me to commit all of my secrets manifests to git with out risk of leaking secrets.

You see kubeseal encrypts your secrets with a private key only stored in your cluster, so only the cluster itself can decrypt them using the kubeseal controller.

KubeSeal #

https://sealed-secrets.netlify.app/

installation #

Installation happens in two steps. You need the kubernetes controller and the client side cli to create a sealed secret.

For a more complete instruction see the [docs#installation](https://github.com/bitnami-labs/sealed-secrets?tab=readme-ov-file#installation]

installation - controller #

Warning

 **context**

Make sure that you are in the right context before running any kubectl commands.

kubectl config current-context

sealed-secrets is installed using the helm package manager. To install sealed-secrets run the following command.

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets -n kube-system --set-string fullnameOverride=sealed-secrets-controller sealed-secrets/sealed-secrets

installation - client #

For the client you can check your OS package manager, brew, or the github-releases. For me I found it in the main arch repos.

paru -S kubeseal
# or
sudo pacman -S kubeseal
# or
brew install kubeseal

Note

 You will need to install kubeseal on every device that you will want to

create sealed secrets on.

Example #

Most of these commands come straight from the docs. From my experience I have always specified the namespace, my projects go per namespace and I don’t have any reason that other namepsaces should see the secret, and if they do I deploy another secret in that namespace.

# Create a json/yaml-encoded Secret somehow:
# (note use of `--dry-run` - this is just a local file!)
echo -n bar | kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o yaml -n thenamespace > mysecret.yaml

note that the key of the secret is foo and the value is bar

results

apiVersion: v1
data:
  foo: YmFy
kind: Secret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: thenamespace

Note

 The data is base64 encoded.

echo -n bar | base64
# YmFy

# This is the important bit:
kubeseal -f mysecret.yaml -w mysealedsecret.yaml

# At this point mysealedsecret.json is safe to upload to Github,
# post on Twitter, etc.

# Eventually:
kubectl create -f mysealedsecret.yaml -n thenamespace
# sealedsecret.bitnami.com/mysecret created

# Profit!
kubectl get secret mysecret
kubectl get secret mysecret -n thenamespace
# NAME       TYPE     DATA   AGE
# mysecret   Opaque   1      27s

cat mysealedsecret.yaml | kubeseal --validate

echo -n bar | kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o yaml \
  | kubeseal -o yaml -n thenamespace > mysealedsecret.yaml
echo -n baz | kubectl create secret generic mysecret --dry-run=client --from-file=bar=/dev/stdin -o yaml \
  | kubeseal -o yaml -n thenamespace --merge-into mysealedsecret.yaml

Results

---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: thenamespace
spec:
  encryptedData:
    bar: AgBLkamltcLfH1dC1JxQ3qd8lJ8aBZF2ARoq3uo055hnzXOy8g2T5liTx5UPvyPV8yyWqABU8eOnwjNhDtzSATvYeBB3fGkucdOZziWEoiNsWTR9ZtFEkod7Ya6uGkzZOJwi3IkrHFVIT9oWZQUxxJZ6vFhPiFcx9Dorr8TNSzG4KOug25+PhWPPiHDgSup5N3CkWCZaYOF7dbZRVSA4nGP1fZxjFByHP4AsdjLCHptyVbkpLRKeiXTkLxfLX4K+JLZGM41S1On5bSP56mCfv1daTJx619kDXkRLw9l21Ot283/L0NMNAiw781AefYMVoO3aHmYgcT6wAtsQAKje9fyL7DQRHt8a5NZOWukp/P6XjdXRz/nfQasQlbSTrRkDpplKIM5/WdPcBoKi+yyoOL0rZ8x1X7YzUI3BggZmzWyEPD01BK1YAHGZnYIZbbCy1JSm8JCBvP+xWMg+i0Z/DCD8nclAhH1GX2Q7/NrNHF//589AJfuriymd2+mk7uaLA4RRsY0l5QeZD6HVAqSv5jWsVQQtSftWmI9vn9oL/Pno7sEUjSDpXPfF4nnsULhxsPEe2DFAMm1kZAjdF06ueF4/x2Fdy80ZQNyycaDx2CWm4z3b14A75WGyOXl2wJZQqxrFCz8el4hD2nH3zQFEzd6AIh49myqVAGuu2qGlYP4p94LJghVa+mQjztLD/2ZUZjY+anQ=
    foo: AgAducXW6iUCY900cPDdmRfuj7tKnh2hY4C1+2hFoAtjyvjepsKNWsiPJ81t8anaMfFPat4ta060l5VtTrceFE8oS/rViz1tvNWGPBjL+GwL/QjkGl6H8ju87vKERKQn5qw7B/V5j24VM8AxOd+/vJNt/IeRIHLubvFft4hyMq4b0xmIxaemBSTxchQX/5364T3VJH2kHaqpqd+JJgQnTbiTQe/XnyZokDX8GSxw4rAbJUJSRUtY9DB9ZDu2zC5VngX+GJjbwHGbv9EKs8LShJIPrD8xHqrDmlSXGkkP01D4A6268Qoi3x5S0H5aqDgtrgBiWsNkzdKwjfrTNx7pKecOi41lyFdffHOGUew4aPPMqjzWR2TEms9WNNQXwnBdDHKMkFsisocF52BolEkcjF8g/u5h2Af92abMu6k16VybJrB7TV1set5A9W7rqG1iXI4+1W6XQfFnpja8xL/zJBvZcyHgeYMNaxa3C3s6PANhPzAUVaXV9eedAkptGJLN13IZj4LujpoAxRKo6bEdydv/5P23R3fx5PgTOpVI7riECAOIg2PThFsEoVCUwStmKCvIx1I2+YixIlv/OiaUWo4lrI/3ve5WGp4ZnuiJPk34JoYAlRbR6+sX14d8Ek6viq/pJUUIfVpNIkNMboUL4u+KpT47eyQ/mWih/KFduQyX9II/vQ+/IJGzEEHIipxAhdmV+K4=
  template:
    metadata:
      creationTimestamp: null
      name: mysecret
      namespace: thenamespace

backing up your sealing key #

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >main.key

converting .env files to a secret #

Working with web applications .env is a common way to store credentials. Let’s look at how we can convert these to secrets.

kubectl create secret generic mysecret --from-env-file=.env -n thenamespace --dry-run=client -o yaml > mysecret.yaml

Now you have a secret that looks like this.

apiVersion: v1
data:
  foo: YmFy
kind: Secret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: thenamespace

Seal it up just like before.

kubeseal -f mysecret.yaml -w mysealedsecret.yaml

Using the secrets #

I typically use the secrets in the container spec.

containers:
  - name: myapp
    envFrom:
      - secretRef:
          name: mysecret
    # You can still have other env vars
    env:
      - name: foo
        value: bar

Sometimes I want to mount the secret as a volume.

containers:
  - name: myapp
    volumeMounts:
      - name: mysecret
        mountPath: /mysecret
volumes:
  - name: mysecret
    secret:
      secretName: mysecret

Image Pull Secrets #

I also need to use imagePullSecrets. Let’s walk through the whole process. Starting with the secret.

kubectl create secret docker-registry regcred --docker-server=myprivateregistry.example.com --docker-username=foo --docker-password=bar --dry-run=client -o yaml

Generates the following secret.

apiVersion: v1
data:
  .dockerconfigjson: eyJhdXRocyI6eyJteXByaXZhdGVyZWdpc3RyeS5leGFtcGxlLmNvbSI6eyJ1c2VybmFtZSI6ImZvbyIsInBhc3N3b3JkIjoiYmFyIiwiYXV0aCI6IlptOXZPbUpoY2c9PSJ9fX0=
kind: Secret
metadata:
  creationTimestamp: null
  name: regcred
type: kubernetes.io/dockerconfigjson

---

the secret

Now we we can seal that secret.

kubeseal -f regcred.yaml -w regcred-sealed.yaml

And that gives us the following sealed secret that we can deploy into our cluster.

---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: regcred
  namespace: default
spec:
  encryptedData:
    .dockerconfigjson: AgATYVEywkyYEaoErQbJo6xEZfOnRn1ydNTLkO3Jt/NF/UH+0o9lHpDecRpN0XnVu8xJUcdjkgD9q2XkwP8e6qQDS2mMPTiTNIN+8gbJx97WrD1YQDT0lXBuoyi9I/iwlXxx6MgH/6GY6CeGTz5SRlvoU0Xhlt4d11s7/xapdE8QMLsAReqPEv8oZHEyAxDRrjXX0V+tO8dV+G+GXjUDMBBceLael9rvGzSKIwDVXACVqQhLkB6FoP98M+yyBE46RBNnSnS0ShQM5PprL24HKpRZ43x4RM53KBrQ7R/MxeshafY+B6vUvrolmVox4sud8xngMOcjTO28LLOrck5V8ZiDabhN7ajHEf03IESr1o/ADGf5k9988Vv1txJtsZW0K2mpRu0D7/BLVL9KzbZ5ywULqIoD/Ur2GIGnZqMAKOq4laGp/GJtMKLrhmEvekT397wC/Gf/xdDKVhHf2p4ocsPu7LKFuS5H/Auel/Q5grdn8L5wwrO4VWRv3eJroKh/Hux7Qd7f64O7qdi0XthDocf+gmtjys+Gy72M7tyf8f/O+3oKbS4CWQVTj4ZThMc9znrFnHqt2q/7pAyytTQCpk51wlzOsNvOhCueJM/jmeahaL0LuBrqngqISpnd65sgVzBcZpwK9i2Fckyt0DrZLH+NoIuvaqNhzlF+OMbAft/ylWWKCH4WUP+FKG+1LXM7ud7AA3MMbGSBxHL0/WK/INa7MB56xZKMqqyvvLLQHFTQUROJjkgkzsumdOgwZTRgIFnAZ4+vOX3/1Rtt3mAs3vdoJhL4GuKUYCnEHt908eKkWEVEs7eMk5SdSRtIsbaXO2s0dtADwg==
  template:
    metadata:
      creationTimestamp: null
      name: regcred
      namespace: default
    type: kubernetes.io/dockerconfigjson

Now that we have our sealed registry secret, we can deploy it into our cluster.

kubectl apply -f regcred-sealed.yaml

Now we can use it to pull images from our private registry.

containers:
  imagePullSecrets:
    - name: regcred

Full example #

Here is a full deployment example using all the secrets we have created.

regcred
mounting a secret
envFrom secret

---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    service: myservice
  name: myservice
  namespace: mynamespace
spec:
  replicas: 1
  selector:
    matchLabels:
      service: myservice
  strategy:
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        service: myservice
    spec:
      containers:
        - envFrom:
            - secretRef:
                name: mysecret
          env:
            - name: foo
              value: bar
          image: private-registry.io/myimage:1.0.0
          name: myimage
          ports:
            - containerPort: 5000
              protocol: TCP
          resources: {}
          volumeMounts:
            - mountPath: /mysecret
              name: mysecret
      restartPolicy: Always
      volumes:
        - name: mysecret
          secret:
            secretName: mysecret
      imagePullSecrets:
        - name: regcred

Downside #

Now the main downside I see with kubeseal is that it does not provide a way to store your secrets in a way that you can access outside of your cluster. So you need to make sure that you have another solution in place to store your secrets so that you still have them if you ever were to take the cluster down or move from k8s to something else.

Overall the likelyhood of you loosing a production cluster is pretty low, so maybe it’s ok to just trust it depending on what the secrets are. Especially for things you control and can rotate anyways its fine.

kubernetes 6 months in

2024-03-26T13:20:24Z

I stumbled into kubernetes December 2023 when I was looking for a better way to self host applications. I was looking for something that didn’t require logging into a server and building and deploying like a cave man. I wanted a smoother experience than docker compose was giving me.

https://waylonwalker.com/looking-for-a-heroku-replacement/

This post turned into a list of tools that I have adopted into my k8s workflow, and plan to keep. enjoy.

Kompose #

Kompose is a great tool for gettting going and converting your docker-compose to kubernetes manifests or helm templates. It was a great tool for me to get started with, but I was afraid that it was hindering me learning more and just blindly using its output so I have tried to use it less and less. I’m now not solely leaning on it, but using it to get out quick POCs with low friction.

Kompose really helped me go 0 to 60 and get right into kubernetes with my existing docker compose files and very little change. I found it a great way to get my feet wet and learn.

Argocd #

https://argoproj.github.io/cd

Argocd is the first tool I am going to reach for in any kubernetes deployment that I work on. The number one reason is that it keeps track of all of everything internally, and if you remove something it cleans it up. Very quickly if you are not diligent you will end up with stuff laying around not sure if you need it, or where it came from.

For instance if I create a new deployment and don’t include the namespace, If I just change the yaml with the new namespace without taking it down first, that original one will be left behind and I will be stuck manually cleaning it up.

Argo Rollouts #

https://argoproj.github.io/rollouts

I’m looking into rollouts, and have played with it, but argocd really does most of what I need with low friction. I thought I needed rollouts to prevent downtime, but argo tends to do a good job of cutting over to a new pod only after its up and ready. This solves most of my problems, and I can pretty quickly roll back with low friction. If I needed to manage a high availabilty application Id look closer.

KubeSeal #

https://sealed-secrets.netlify.app/

KubeSeal is a fantastic low friction pair to argocd. The biggest benefit to kubeseal is that you can safely commit secrets straight to git. It uses a one way encryption method that is very secure and you can only decode the secrets if you gain access to the keys.

kubernetes kubeseal

Configure Liveness, Readiness and Startup Probes | Kubernetes

2024-03-15T19:38:02Z

Configure Liveness, Readiness and Startup Probes

This page shows how to configure liveness, readiness and startup probes for containers. For more information about probes, see Liveness, Readiness and Startup Probes. Before you beginYou need to ha...

Kubernetes · kubernetes.io

What is the difference between health, liveness, readiness, and startup? This article does a great job at a full writeup description of how it works in kubernetes, here is my TLDR.

health 200 OK - I’m still responding to requests
health ERR - something happened and I cant respond to requests
liveness 200 OK - I’m ready for more work
liveness ERR - I’m still responding to requests, and i’m already working send requests to another pod, or scale up

Z-pages #

These probes are commonly deployed at /healthz and /livez endpoints.

Why the z?

z is a convention that comes from google for meta endpoints to reduce conflict with actual endpoints, and can be deployed to any application.

Uptime Kuma

2023-11-11T08:46:12Z

Uptime Kuma

A self-hosted monitoring tool

uptime.kuma.pet

Uptime kuma is a fantastic self hosted monitoring tool. One docker run command and you are up and running. Once you are in you have full control over checking status of urls, frequency, allowed timeouts, and a HUGE list of notification providers

docker run -d --restart=always -p 3001:3001 -v uptime-kuma:/app/data --name uptime-kuma louislam/uptime-kuma:1

I deployed it in my homelab today.

Kubernetes Secrets in 5 Minutes! - YouTube

2023-10-30T21:47:19Z

I am converting my docker compose env secrets over to k8s secrets. This guide was clear and to the point how I can replicate this exact workflow.

First set the secret, the easiest way is to use kubectl wtih –from-literal because it automatically base64 encodes for you.

kubectl create secret generic minio-access-key --from-literal=ACCESS_KEY=7FkTV**** -n shot

If you don’t use the --from-literal you will have to base64 encode it.

echo "7FkTV****" | openssl base64

Once you have your secret deployed, you have to update the container spec in your deployment manifest to get the valueFrom secretKeyRef.

    spec:
      containers:
        - env:
            - name: ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: ACCESS_KEY
                  name: minio-access-key
            - name: SECRET_KEY
              valueFrom:
                secretKeyRef:
                  key: SECRET_KEY
                  name: minio-secret-key
          image: registry.wayl.one/shot-scraper-api
          name: shot-wayl-one
          ports:
            - containerPort: 5000
              protocol: TCP
          resources: {}
      restartPolicy: Always

Waylon Walker 🐍 on X: "Which is more complicated" / X

2023-10-30T18:25:53Z

waylon walker (@_WaylonWalker) on X

Which is more complicated

X (formerly Twitter) · twitter.com

Wow, shocked at these results. All this time I’ve been told and believed that k8s is incredibly hard, and you need a $1M problem before you think about it because it will take a $1M team to maintain it. So far my experience has been good, and I definitely do not have a $1M problem in my homelab.

Creating k8s jobs with python

2023-10-28T05:52:24Z

External Link

thoughts.waylonwalker.com

I was looking to add running kubernetes jobs to a python cli I am creating, and I found this solution, mostly thanks to ollama run mistral:7b-instruct-q4_K_M and my loose understanding of what the yaml syntax is supposed to look like for a kubernetes job. This will let me create a job in the cluster, choose the image that runs, the command that is called, and how long until the job expires and is cleaned up. While the job still exists I can go in and look at the logs, but once its ttl has expired they are gone.

from kubernetes import client, config

# Load the default kubeconfig
config.load_kube_config()

# Define the API client for batch jobs
api_instance = client.BatchV1Api()

# Create a new job object
job = client.V1Job(
    api_version="batch/v1",
    kind="Job",
    metadata=client.V1ObjectMeta(name="myjob"),
    spec=client.V1JobSpec(
        ttl_seconds_after_finished=100,
        template=client.V1PodTemplateSpec(
            metadata=client.V1ObjectMeta(labels={"app": "myjob"}),
            spec=client.V1PodSpec(
                containers=[
                    client.V1Container(
                        name="myjobcontainer",
                        image="busybox",
                        command=["ls", "/"],
                    ),
                ],
                restart_policy="Never",
            ),
        ),
        backoff_limit=1,
    ),
)

# Call the Kubernetes API to create the job
api_instance.create_namespaced_job(namespace="default", body=job)

mkimuram/k8sviz: Generate Kubernetes architecture diagrams fro...

2023-10-23T02:07:26Z

GitHub - mkimuram/k8sviz: Generate Kubernetes architecture diagrams from the actual state in a namespace

Generate Kubernetes architecture diagrams from the actual state in a namespace - mkimuram/k8sviz

GitHub · github.com

This is a sick kubernetes architecture diagran generation tool.

Here is an example

installation #

$ curl -LO https://raw.githubusercontent.com/mkimuram/k8sviz/master/k8sviz.sh
$ chmod u+x k8sviz.sh

Usage #

./k8sviz.sh --kubeconfig ~/.config/kube/falcon-k3s.yaml -t png -o k8sviz.png

Translate a Docker Compose File to Kubernetes Resources | Kube...

2023-10-22T07:04:18Z

Translate a Docker Compose File to Kubernetes Resources

What

Kubernetes · kubernetes.io

kompose is a sick cli to convert docker-compose.yml to kubernetes manifest.

# install

curl -L https://github.com/kubernetes/kompose/releases/download/v1.26.0/kompose-linux-amd64 -o kompose

kompose convert
kompose convert -o deployment.yaml

Kubernetes Persistent Volumes with Deployment and StatefulSet

2023-10-21T06:47:58Z

Kubernetes Persistent Volumes with Deployment and StatefulSet

How to use Kubernetes persistent volumes with deployment and stateful set and also when you should use one or another.

Alen Komljen · akomljen.com

Example of how to add a pvc to a deployment.

Inspect a Kubernetes PersistentVolumeClaim | Frank Sauerburger

2023-10-21T06:34:08Z

I was curious to see what was going on inside of my minio object storage. Great technique here by Frank to create an inspector pod, then you can do as you wish with the data.

I created the manifest as pvc-inspector.yml

apiVersion: v1
kind: Pod
metadata:
  name: pvc-inspector
spec:
  containers:
  - image: busybox
    name: pvc-inspector
    command: ["tail"]
    args: ["-f", "/dev/null"]
    volumeMounts:
    - mountPath: /pvc
      name: pvc-mount
  volumes:
  - name: pvc-mount
    persistentVolumeClaim:
      claimName: pvc-name

Then used it like this.

# create pvc-inspector pod
kubectl apply -f pvc-inspector.yml
# exec into inspector
kubectl exec -it pvc-inspector -- sh
# explore data
ls /pvc
# cleanup
kubectl delete -f pvc-inspector.yml

Can't create Secret in Kubernetes: illegal base64 data at inpu...

2023-10-21T05:49:39Z

External Link

stackoverflow.com

In order to use k8s secrets manifest you first need to encode the data values.

echo -n 'mega_secret_key' | openssl base64

Can I access k3s using just kubectl (no sudo and no k3s comman...

2023-10-21T03:52:00Z

Reddit - Please wait for verification

reddit.com

Right after installing k3s you are going to need to use sudo to use any kubectl command. The reason for this is that the default config is owned by root. To get around this you will need to make your own config and set the KUBECONFIG environment variable

To do this I used sudo one last time to copy the k3s.yaml file into my own directory and take ownership of it.

sudo cp /etc/rancher/k3s/k3s.yaml /home/waylon/.config/kube

sudo chown -R waylon:waylon ~/.config/kube

export KUBECONFIG=~/.config/kube/k3s.yaml

Quick-Start Guide | K3s

2023-10-21T02:32:03Z

Quick-Start Guide | K3s

This guide will help you quickly launch a cluster with default options. Make sure your nodes meet the requirements before proceeding.

docs.k3s.io

I recently spun up k3s in my homelab. I’m trying to offload some work off of my free tier fly.io app in order to keep it free tier without crashing.

# install and start k3s
curl -sfL https://get.k3s.io | sh -
# check to see if your nodes are started
sudo kubectl get nodes

My main hiccup so far was the machine I am running on runs zfs on root, and it would not start the master node. Rather than figuring out how to make zfs play nice I just pointed k3s to a drive that is not zfs.

# manuallly
sudo k3s server -d /mnt/vault/.rancher/k3s
# without editing systemd service
sudo ln -s /mnt/vault/.rancher/k3s /var/lib/rancher/k3s