Posts tagged: tech

The Faulted Disk: harbor Replacement Writeup

2026-10-06T21:30:00Z

The sequel to panicking-led-to-losing-my-desktop — this time the monitoring actually caught the disk dying, and nothing was lost.

What happened #

harbor is my replica pool — a 10.9T mirror (2x 12TB) that receives syncoid snapshots from tank. One side of the mirror, a Seagate Exos ST12000NM0127 (serial ZJV4QFLB, /dev/sdb), went FAULTED with 14 read + 22 checksum errors.

mirror-0                              DEGRADED     0     0     0
  ata-ST12000VN0008-2PH103_ZTM0NFDW  ONLINE       0     0     0
  ata-ST12000NM0127_ZJV4QFLB         FAULTED     14     0    22  too many errors

The IronWolf mirror side carried the pool — No known data errors. ZFS redundancy did exactly its job.

The difference from last time #

Last failure: no monitoring, found out by accident months later, desktop died.

This failure: SigNoz + node-exporter’s ZFS collector → node_zfs_zpool_state{state="degraded"} → alert rule → Gotify → my phone. The gotify notification fired before I knew anything was wrong.

Diagnosis #

Before zpool clear or replace — check SMART. smartctl-exporter already scrapes all disks into SigNoz, so I didn’t even need sudo:

Reallocated_Sector_Ct raw = 3024 and counting
Offline_Uncorrectable value/worst still 100 but raw errors climbing
SMART overall: still PASS (SMART’s overall bit is conservative until threshold)

3k+ remapped sectors is a platter going bad — not a cable blip. Verdict: replace, don’t clear.

The swap (the annoying part) #

Hot-plug is never as smooth as it should be:

zpool offline harbor <old> to stop writes to the dead drive — actually skipped; drive fell off the bus on its own
New IronWolf ZTN1CQ07 in hand → plugged into ghost’s SATA → didn’t enumerate
Forced rescan (echo "- - -" | sudo tee /sys/class/scsi_host/host*/scan) → nothing
Moved it to the old drive’s port → nothing
Sabrent USB dock on aurora → dock enumerated as 0B device, no disk behind it → reseated + replugged → drive spun up and appeared: /dev/sdd, 10.9T, old ZFS partition table on it (used drive — part1/part9 layout)
Back to ghost, direct SATA → enumerated as sdb

Lesson: “spins up but doesn’t enumerate” on direct SATA + dock-shows-0B = seating/power problem, not DOA. The drive was fine all along.

The replace #

sudo zpool replace harbor \
  ata-ST12000NM0127_ZJV4QFLB \
  ata-ST12000VN0008-2PH103_ZTN1CQ07

Resilver: 8.80T in 18h38m, 0 errors (~154M/s). Pool stayed online and usable the whole time.

One gotcha: after resilver, zpool status showed errors: 1 data errors — but zpool status -v showed an empty error list. The corrupted data was already repaired; the counter was just stale. sudo zpool clear harbor → clean.

The monitoring that made this possible #

Wired during this session (all now in SigNoz):

node_zfs_zpool_state — pool health (node-exporter zfs collector)
smartctl-exporter — SMART attributes incl. reallocated sectors (the early-death signal)
zfs-metrics.sh textfile bridge — sanoid --monitor-* exit codes, zpool error counters, scrub ages, resilver %, syncoid last-success
Alert rules → Gotify → phone

The replication alerting even proved itself live: syncoid failed twice during the resilver window and I got paged on both. Turned out to be send-time contention — self-healed once resilver finished — but the notification path works end to end.

Remaining homework #

Buy the replacement spare (the shelf’s empty now)
10Fold datasets are garbage + have zero snapshots — destroy
Every scheduled job emits cron_last_success_epoch now — if a job dies silently again, I’ll know

Same failure as April, opposite outcome. Redundancy did the protection; monitoring did the detection. You need both.

Co-authored with Devin, who ran the monitoring stack, the SMART diagnosis, and the alerting loop.

It Will Be Hard To Do More

2026-05-22T06:23:20Z

Recently I was given a raise at work after I broached the subject with my manager. I wasn’t sure what the outcome would be but I have learned two things in nearly a decade in corporate America:

No one will fight for you harder than yourself
The worst they can say is no

The Raise #

So I talked with my manager about my workload and compensation, and to my surprise he took me seriously. He went up the chain with some accomplishments of the year that I wrote up for him (it’s his job to advocate for me, but it’s certainly my job to give him the ammunition to take into battle). Before much time had passed, a big wig put some time on my calendar. I had a good relationship with my (then) manager, and even with the big wig, so there was always some amount of personal connection amidst the money-talk.

The Reorg #

Cue a managerial change and reorganization of the team. The dust settled, and my manager wasn’t my manager and the big wig wasn’t a big wig over my team anymore. The details aren’t that relevant, but the outcome was that my now manager is someone I don’t know personally — back to square one of professional-relationship-building.

The Conversation #

Due to the timing of it all, when my raise became official it was this new manager who signed off on it. It was their happy-responsibility to have a quick chat to tell me it was official. But that conversation was the most awkward of the three.

Does this raise of X% close the gap with the other competing opportunities you mentioned to (previous manager)? Because it’ll be hard to do more than this.

???

Hard to do more?

I was taken aback initially. It felt almost threatening.

I really wasn’t sure how to respond. I think I was grateful and cordial, talked about how I really do believe in the mission of our work and I’m thankful that I can receive a raise. This lady is nice but she doesn’t know what my “other opportunities” are, so there’s no honest answer for me to give. I wanted to be sarcastic, crack a joke about record profits, make light of her describing me as a “critical player in enterprise deliverables,” but I’m not trying to make enemies.

The Reframe #

So I called a friend to vent, of course, and he gave me some good advice.

It was a threat.

But not in the way I took it at first. My friend has been at the same company longer than me, and he expounded on her words: “it will be hard to do more.” I felt like she was annoyed with me for fighting for a raise, but my friend said she was probably just trying to give me a realistic view of my position in the company. “It will be hard to do more” means there are processes in place at such a large enterprise. Rewarding start-up-like behavior and contribution just isn’t well-supported. It’s much easier to let someone quit who’s been with the company a decade and replace them for 20% more than it is to just pay them 20% more. If someone (like me) isn’t satisfied, that’s not the only thing in the conversation - time at the company, time in position, time under specific manager, etc. all are hurdles to more compensation, and I’m referring to salary anyways, equity feels like it’s out of my grasp. But if I’m gone and market conditions dictate that my position and responsibilities are worth 20% more than what I was being paid, that’s what the position will be posted at. My previous manager and the big wig were the types to say policy-be-damned when it would get in the way - it might be how I got this raise in the first place. But no shade to my new manager, they’re just doing their job.

In the end, they were just being honest, and I should appreciate that.

Panicking Led to Losing My Desktop

2026-05-13T08:24:00Z

False Sense of Security #

I thought I had backups handled… can you imagine how the rest of this post is going to go with that intro?

To be fair, I do have backups figured out on my NAS - simple ZFS + sanoid/syncoid + replica pool + off-site backup with simple restore pathways. However, my desktop has been another story entirely. My desktop OS didn’t support ZFS when I started checking it out, and I spent weeks thinking through how I would backup my HOME directory and projects mostly. I landed on a solution that I did validate once, but it fell off my radar and lo’ and behold that was problematic…

So that backup was based on restic for my home directory, but it was lazy. I verified it one time but I had built it with ai, thought I understood the restic repo part, and then promptly moved on with my life never buttoning it all up. That home directory backup got too big for where I was going to end up restoring it. My desktop system was installed on a 4 TB NVMe drive and due to the circumstances spawning this blog post I was gonna have to drop to a 500 GB boot drive with some extra disks as the storage layer. Overall it looked like:

A 4 TB SSD that was going bad - old OS
A 500 GB SSD, that was going to be my new operating system boot disk
A 2 TB SSD that was originally going to be this external storage volume anyways but I never set it up because the version of Aurora I was running didn’t have ZFS, I was married to the idea of using ZFS, so I never ended up taking advantage of the space. However it was moot to me because my boot drive was 4 TB, high quality drive, so I was “just sure” I didn’t need it.
AND a 4 TB rust disk as well, which was already a ZFS pool, left over from a previous desktop configuration, and admittedly I had forgotten it was even in the system.

The Storm #

If it wasn’t clear the problem is that my super-nice high-speed 4TB NVMe drive was going bad, like really bad. Eventually my OS stopped booting, it was even difficult to live-boot from any other ISO due to, I think ultimately, that disk causing such extreme latency in the start-up processes that they just failed. So I quickly found myself with little-to-no access to my primary desktop’s data…

Where It Went Wrong #

What I did is I live booted into an Ubuntu server environment (which took blood sweat and tears to successfully get into), mounted my home directory from the 4 TB SSD, and tried to continue my restic backup to my NAS, like an idiot. But at the same time I also tried to prune it by only backing up a few projects because I was getting worried about time. This was the first primary mistake - trying to muck with my backup script under duress.

Then over the course of the whole thing it ended up taking over a week to solve this when it could’ve been 2-3 days. So say it with me kids - “Don’t make decisions under duress”

Climbing Out #

I downloaded opencode and had it help me write the right excludes syntax in my restic backup script and got it back up going. That went ok but opencode agents had no historical context for why anything was the way it was, and frankly an agent would’ve been misled thinking the backup solution was much more solid than it was due to how I documented it.

Agents also miss things… in my chat sessions it knew about the other 2 available disks on the desktop system, I could have done a fresh backup to the 4 TB spinning rust disk no problem: install zfs, mount the pool, change target of restic, run full… that would’ve been beautifully simple. But instead I trimmed it down and backed not-everything up to the NAS over the network, and to a different backup target nonetheless… SMH.

As I started to consider which OS I was going to go with next I failed to install Pop_OS! or Ubuntu onto the new disc… Then I tried Omarchy and the install script just looped. So, I reinstalled Aurora onto the new 500 GB disk and then quickly realized I don’t have Firefox tabs, my SSH keys are in that restic backup, my ssh config, api keys in hidden files…. Everything is in that restic backup… The backup that’s too big to restore to my new boot drive.

But you know what I have? That 2 terabyte disk mounted just fine as a ZFS dataset. And I could mount the 4 TB rust disk with zfs as well because this version of Aurora has zfs working flawlessly!

Hindsight #

What I should’ve done is so simple… While in that ubuntu live environment I should’ve just either updated restic to be a local backup to the 4 TB rust disk, or rsync’d my home directory to it plain and simple… I got all in my head about not backing up python venvs, node_modules, etc. that I didn’t think to just basically carbon copy it all to a healthy disk and then prune it later. Then I could’ve synced everything back over that I needed to the new Desktop’s $HOME and then scheduled the rsync or restic again to that locally mounted disk.

The Detail I Left Out #

The keen reader might stop to think… why not just mount the old 4TB disk and copy what you need to your new desktop? And that’s a prudent question… However, in order to get anything installed I had to physically remove the 4TB SSD from the motherboard, which was basically a full PC tear-down. From there I was able to at least boot in and out of iso’s like you’d otherwise expect, and I have a USB/NVMe adapter so I planned to mount the old drive and copy things over from there… But sadly… it won’t mount. it’s dead-dead and it appears that anything I didn’t save in my days-long-panicked-state is just. gone.

I feel pretty stupid to have not taken advantage of the 2 available disks local to the machine, to have naively copied stuff over and dealt with the organization later once my OS was back up. I tried to be smart and efficient and ended up wasting so much time and losing quite a lot of “stuff”… ideas, blog posts that I never committed, etc.

Current Status #

So a few lessons…

untested backups are not backups
false backups might be worse than none, although I did at least save a few things so maybe the jury is out here
making decisions while stressed out will lead to missing obviously better pathways… slow down, talk it out

As for my current status - I’m working on Desktop Setup 2026 and recovering what I can from my haphazard’d rsyncs in the live ubuntu env I got into. I’m also setting up a new Linux laptop at work at the same time so maybe I’ll hve some workflow changes to write about in the future. For now, it’s nice to be forced to accept that not every idea was that important, the good stuff will come back around, and ultimately computers and shit are just things, they’re not life.

The Relief of Earning

2026-05-08T05:43:39Z

The Setup #

Work has been crazy for months, and I haven’t written too much about it, a few notes here and there but nothing substantial. The craziness is sourced in a pretty cliched-scenario where we have lofty goals and important things to build but the foundation on which we are building, like architectural decisions, is basically wet sand. So as I’ve been trying to find my footing in the mud I feel like I end up spinning my wheels moreso than making any progress. The chaos led me eventually to approach my manager about my compensation - I make a nice living but for the stress I’d been feeling I could certainly go make more. My manager successfully fought for me up the chain and I was given a slightly bigger carrot, but that’s not the point of this post. I started to feel even more anxiety after I was told about the raise because, just like with AI and agentic coding, now I was sure the expectations would be even higher. I didn’t think anything would actually come from me asking my boss about a raise, but now that it did I was afraid the ceiling of expectations was just going up up up.

The Carrot and the Anxiety #

That’s where the relief of earning comes in… I mentioned this to my project manager, and we have candid conversations regularly, and he told me something that I think some people might have been offended by but I was genuinely relieved to hear it. He said he’s glad I got a raise, and that he told our boss to not really pursue it for me until I had actually delivered something. So even though I feel like I’m spinning my wheels a lot, my direct leadership sees real value - and the relief is that I can rest in confirmation that the carrot is a response to work I already did, not a quiet elevation of expectations.

Fixing Firefox Launcher on Kubuntu 22

2026-04-11T10:32:50Z

Problem: Firefox installed but missing from KDE/Plasma app menu
Root cause: No .desktop launcher file — Firefox only worked from terminal
Solution: Created desktop entry with standard KDE fields


[Desktop Entry]
Version=1.0
Name=Firefox
GenericName=Web Browser
Comment=Web Browser
Exec=firefox %u
Terminal=false
Icon=firefox
Type=Application
Categories=Network;WebBrowser;
MimeType=x-scheme-handler/http;x-scheme-handler/https;
StartupNotify=true
StartupWMClass=firefox

Location: ~/.local/share/applications/firefox.desktop (user-local, no sudo needed)
Discovery step: Checked /usr/share/applications/ — empty for Firefox
Applied fix: Ran kbuildsycoca5 to rebuild Plasma’s menu cache
Result: Firefox now appears in app menu (Network category) and Alt+F2 autocomplete

Data Loading is a Huge Deal

2026-04-08T07:40:14Z

I’ve been thinking about the work I am doing and have to do in my role at Cat, in Cat Autonomy, building Forge (see Forge Ahead). I feel like I have little revelations almost every day now, not that it means I’m writing something amazing and producing it really fast but there’s just a whole suite of problems that different technologies solve at different levels and the more I become aware of the problems that exist, the more the existence of some solutions makes sense.

The Problem Perspective

I don’t know if this is a real thinking technique or if I’m onto something novel(doubt) but I think a lot in terms of problems - “what problem needs solving?” and that’s how I’ve come to prioritize my work, it’s only been very recently that I’ve realized I do this and I think I should highlight it’s very important to document the problem, otherwise every day you might try to solve a different problem but be working on the same code

Problem Space

Kedro solves a lot of these problems, so when making rada in Reman the problem space was already more contained and narrow, Forge’s problem-space is much more vast

The one problem I’m really fixated on right now is data loading, the problem I need to solve is accessing data from a wide-variety of scripts/tools, without a framework or standard method/library of accessing data in the first place. I’ve seen lots of projects on GitHub claim to make data loading easier and I didn’t quite understand what problem they were solving… one example to name is Data Load Tool. I’ve seen similar ones I don’t have the name for right now that claim to make it easy/fast to load data from s3, or ways to make s3 and a database both abstract in the user-experience for loading data. But I hadn’t really understood why these tools existed. I have a lot of experience with Kedro and their DataCatalog which provides a python object over a set of yamlfiles that makes it pretty simple to load and save data in a way that isolates I/O from the business logic. But what I didn’t realize at the time how powerful that catalog was, the power of standard patterns and shared libraries. Now that I don’t have it available to me, I’m quite aware of the absence.

In my new role something I’m realizing is that for all the developers my team now supports, there isn’t a canonical way to access data. When I was in Reman and working with kedro, the DataCatalog was the access pattern and so when I was developing a platform I never really had to think about it - it was an established pattern that I treated as a constraint and then built processes around it. I’ve been battling some mental block for weeks on Forge because of the lack of that canonical pattern, and as I’ve talked with other engineers it seems like the baseline assumption is that data is just available on a filesystem, but everyone’s code loads data in different ways. On my small Reman team, with common patterns to build on, it was easy to make things cloud-native or shim in some devops to improve people’s lives. But when everyone’s doing their own thing, and everyone’s “own thing” is very much built-on some rigid tribal patterns then it’s hard to really move fast cause everyone isn’t already moving in the same direction.

That made me realize that the first problem Forge needed to solve was in providing a way for engineers to have filesystem-native data access in the Cloud, where we are S3-first in our storage philosophy. I didn’t need to figure out a way for everyone to name a dataset, define the dataset in the first place, and give a nice my_dataset.load that worked in python, bash, cpp, and who knows what else…. I reframed the problem from “how do engineers load up the data” to “how do engineers have access to the data”. The requirements of the Cat Autonomy group was pretty simple: POSIX-compliant storage.

My pathway to solving this problem is initially underway, I can’t imagine it’ll be too difficult to setup for FSx instances for teams and give them an api to run a Batch Job with the FSx mounted. From there, their code can load data from /mnt/fsx/<whatever> just like they otherwise could be doing locally. Or maybe FSx will let us setup mounts to very flexible mount points and their local scripts will “just work” :shrugs:. I don’t know the exact shape, but after realizing the loading data is a big deal, I’m thankful I have a narrower problem to solve first.

S3 Files

Literally yesterday, AWS launched “S3 Files” offering an NFS filesystem service over buckets. I’m not sure if NFS is going to be a viable filesystem protocol for all of our use cases, but looks like we’re not the only people who need the filesystem access patterns over S3.

A Future Problem - Canonical Reference

Another high-value thing Forge needs to solve is “what is data”. The data formats we have are not super simple, it’s not just a set of SQL tables. We have files that relate to each other based on hard-filepath patterns, and those patterns are full of tribal knowledge and distributed processes. So a simple question like “How do I use forge to access my data” is hard. In Reman a data scientist would ask “how do I use rada to access my data?” and the answer is “We use Kedro, and Kedro solved that problem for us via the Catalog” but without Kedro, without 100% being in python (devs are also in embedded systems, cpp code, and more), without even consistent practices in the existing “how do I access my data” workflows, it’s really impossible to systemetize and codify it. It is my next challenge to tackle though…

Agents can miss obvious things too

2026-04-03T14:39:51Z

I am working with MiniMax M.2 to vibe up some webapp ideas for Nexus and I just spent many minutes not having the app render correctly because the agent mounted the wrong directory into the docker compose stack. I noticed it about 5 seconds after opening the file, but wasted nearly a half hour thinking it was a docker issue in distrobox…

Danger

Clankers are still just clankers man

The Irony of 10x

2026-03-26T06:00:50Z

Opener #

Agentic coding has been an exciting change to me over the last couple of months specifically. I’ve been using AI tools for a few years now but something really shifted with Opus 4.5 as well as the tools/harnesses getting better and more useful around the same time. I’ve been influenced by people like Simon Willison and Steve Yegge who have been on the forefront of agentic coding and vibe engineering since the dawn of ChatGPT and in my small circles of work I’m definitely on the bloodiest bleeding edge of the adoption of these practices. Are they the future? I don’t know - I tried to maintain a skeptical posture but until the bubble pops it’s looking like this is at least a direction the future of my line of work is going.

Danger

The tough part is mixing the new world of agentic coding with developers on wildly different points in the spectrum of adoption and maturity.

Here’s a short anecdote about what I mean…

In a new project that I’m on, I’ve refactored/reimplemented a lot of legacy code, producing about 140k lines of code, configuration, and docs in about 2 months. It’s an insane amount of “product” and I’ve done it entirely with agents. But I didn’t do it with a handful of vanilla chat sessions like “Hey Copilot, reimplement this API, no mistakes”. I’ve been building out my own process for using LLMs effectively. I have no actual idea besides my own experience if what I’m building is useful, but it feels pretty good - agents for planning, building, reviewing, testing, etc. My “harness” includes specialized agents and opinionated development workflows to try to ensure that code is never one-shotted into production.

The catch though is that I’m working with a handful of developers and they are not early-adopters or aggressive experimenters with these new agentic tools. Most of them are still in Steve Yegge’s stage 2 or 3 of AI coding as he’s outlined in this medium article introducing GasTown. They open a chat session, and say yes or no to Opus. I’ve been in-between stages 6 and 7 for a while now - managing multiple agentic sessions that themselves run specialized subagents primarily for context management, although I don’t feel quite ready for full blown stage 8 (agents running agents running agents). So this isn’t me saying that I’m “better” than the guys I’m working with, just the project we’re on together is now being touched by people on wildly different ends of the agentic coding spectrum, and the project itself is an experiment to me of living in the “New World”.

Finally the anecdote - I’m getting PRs from guys in stages 2 and 3, thousands of lines of code and config, that I know they are not necessarily experts in, but they’re producing that code with fairly vanilla practices. How do I verify it? I know my own prompts, I know what agents generated my code, I know the direction and prodding I gave as they’re building, but all I know in a PR review from someone else is the diff - not how they tested it (unless of course there’s tests, and there must be, but tests are only as good as the tests are…), not the steering they gave, not the manual UAT validation they did, or even the full intent… And ultimately I own the code they merge because chances are I outlast their contract with the team. It’s kind of a scary thing to review and accept… do I trust my agentic practices to validate their work? Cause I certainly don’t trust myself to validate it perfectly.

a note about contractors

I’m not implying anything about anyone who works contract, I think the facts is that contractors are typically dis-incentivized to really “own” something - an unfortunate consequence of the world.

This is about me, not everyone #

I want to make sure to reiterate the context of this post and my thoughts - it’s really about me. I’m not saying anything here is true for anyone else, I’m not making predictions about tomorrow, and I’m not a prominent FOSS developer or a high-profile ex-FAANG engineer. I’m not talking about every project under the sun. I work at a big company, not a fast-paced startup. I work on internal tooling, nothing that should even be seen outside our network. I’m also a self-taught developer, not a trained software engineer.

As far as AI goes, lately I’m mostly all in on Github Copilot CLI with a sprinkling of Opencode. I’m not using Claude Code or Codex which seem to have their own communities around plugins and usage.

And since we’re so focused on me right now, the important thing to keep as the backdrop for this post is my temperament and make-up. I’m definitely made a certain kind of way, not different than every single person necessarily, but different enough, from enough people, to not blend in with the 9-5 guy who can log out and not think about work until the morning (I’m not saying that’s a strength either, my boundary problems are for another time).

I feel an aggressive burden to solve problems and own those solutions, call it white-knighting or a savior complex if you want, but I’ve got enough of a reputation at Caterpillar now (and anywhere else I’ve worked) to become a go-to person for more than I think is necessarily appropriate. And with that burden I’ll bring results - if a problem hooks into my brain it. will. be. solved. I probably won’t do the best most clean-code solution right out of the gate, but I’ll do whatever I can to find a solution.

Who I was before agents #

That is who I was before agents made code easy to produce. My first boss at Caterpillar used the phrase “tenacious learner” to describe me in several reviews. I kind of rejected the description because I basically refused to believe that I was really any different, any harder of a worker, than my peers.

But with almost a decade of experience in the corporate world, and some adult perspective on my life, I think it’s accurate… I am a harder worker, to my detriment sometimes, than a fair number of people I’ve worked with**.

I’m not brilliant but I can focus for a long time in the right circumstances. The gift of perseverance (or the curse of not being able to let something go, depending on how you look at it) has led to blessing in my life in both reward and skill.

\*\*

There’s nothing wrong with it either, Cat’s somewhat noticed that work in my EOY reviews and I’m certainly not against “just doing your job”.

And then AI came along and with another set of the right circumstances catalyzed a new way to work.

Leaning into “Agentic Engineering” #

I jumped onto experimenting with AI coding tools as soon as they became available, but mostly I just tried vibe coding rather than using tools for real engineering work. I vibed up an API at one of my jobs that went into production way too early, with far too little validation, and it was scary to support it from then on out. I also did the meme, vibe-coded a TODO app, and threw that puppy into the internet without locking down my API endpoints… That was before agents were quite as useful as they can be now, but that experience along with a handful of other stepping stones (like learning some real actual fundamentals about security) began to give me confidence in using the AI as a tool, like my IDE is a tool, for producing solutions that take the form of code.

Note

As one-shot apps got better and better, and as I learned about scoping work more appropriately for agentic tools my confidence in them grew.

There’s quite a difference between “Claude make me a todo app, no mistakes” and scoping out a solution in natural language, with some technical guardrails, and having agents tackle the implementation methodically.

What actually changed #

What’s actually changed for me is quite a lot… I haven’t opened my IDE to seriously write code for months now. I’ve oscillated between Opencode and Copilot CLI, leaning moreso into Copilot since it’s an approved tool at work and as of mid-February is quite good. Mentally I’m approaching problems with a little more thought on the front-end than before because prior to agents I would think as I implemented. At the scale of work that I do, this was really fine - working on CLI utilities to solve simple problems, developing an iterative testing cycle for each problem that allowed me to move fast, and once I found a groove I was cooking. But now I don’t even need to find it, I open Opencode or Copilot CLI with my Planner agent, describe what I want to happen and have Opus or GPT scope out a plan for me. Usually there’s some back and forth on feature scoping, then I review a markdown file it produces, and once it looks decent enough to me I say “go” and it goes.

That works a lot better than I even care to admit because at the same time as I’ve been leaning harder into agents, I’ve been building my own harness of sorts - not a replacement for Copilot CLI or a competitor to Opencode, but moreso an opinionated workflow spine that I force agents into to give strict gates to the SDLC (software development lifecycle).

Problem Solving Workflows

Plan and implement is fine for a lot of things, and I do think it’s only getting better. My harness, mentioned a few times around here before, called Nexus, is a set of agents and rules that I want the code I’ll be responsible for to go through before it lands in production. That cycle isn’t too complex, and there’s only about 10,000 similar tools to Nexus on Github trending right now. I’ve thought about dropping my idea and picking up something more popular, like superpowers but at the moment I’m continuing to develop on and lean into my own idea here.

Mini post on Nexus

I keep saying a blog post is coming, but the high level of Nexus is that it’s a task tracker with a CLI that agents use to advance a ticket through a plan -> build -> test -> review -> verify -> merge lifecycle that is almost exactly how I would otherwise have solved a problem by hand. I think it needs work, I need to be harder on TDD methodlogies with agents, and work on verification gating a bit more (shoutout to showboat by Simon Willison) but overall it’s a system of thinking that I already participate in so I’m doing my best to farm out specific parts of my workflow to agents rather than trying to one-shot enterprise problems and solutions.

Who's doing the thinking?

I’ve noticed that as I’ve developed Nexus out though, I lean on the agents for more and more of my own thinking, and am trusting my problem solving process moreso than my actual problem solving abilities.

Hidden costs #

The cost of this increase in speed is a lack of familiarity - and the fallout of lack of familiarity is hard to express. There’s also many facets to it. For me, the first facet is that Nexus helps me move fast, but as I’ve leaned into it for more and more of the planning, I’m less and less familiar with the state of the code. I find myself asking my reviewer agents in fresh sessions often to explain it to me, and thankfully they’re usually consistent, but nonetheless I’m still not intimately familiar with the code. And on Nexus it’s not a big deal, that’s low stakes, it’s just me and my workflow.

I’m using Nexus + Copilot at work and that feels like higher stakes… I have my agents explain the status of our project and although they’re also somewhat consistent the thing that’s scary is that other people are working on that repo with me, and that’s where another layer of complexity manifests itself. If it’s just me and my Clankers, let’s go all day long, rebuild, ask questions, etc… but I have other developers I rub shoulders with now, and if they ask me a question what am I going to say? “Hold on, let me prompt my agent for you” - it’s LMGTFY on steroids. And the burden becomes if I feel like I can own and support what those other developers push into the repo.

Murky responsibility boundaries #

Why do I own their work? Well for the third time, this post is pretty self-centered and all about me, and my situation is that the other developers I work with presently are all contractors. Their work agreement with Cat could end at any second, for practically any reason. The incentive structure isn’t there for these guys who technically work for an agency… Their bonuses aren’t bigger (or even exist) if Cat performs well, there’s no extra vacation days in it for em (aye, contractors don’t get vacation days anyways), and not that it’s a problem, moreso just the nature of the world we’re in - but they’re basically mercenaries out to the highest bidder and I happen to know of multiple times where Cat lost a good person to a higher bidder.

So this isn’t really me trying to be negative about contractors at all, I’m here for a pay-check as well but Cat at least gives me SOME incentive to work hard with the goal of compensation regardless of how altruistic I feel in my own circumstances.

Incentive

Better ratings mean marginally better end-of-year salary increases, and I’ve received some other awards that certainly give me pause about jumping ship to another long-term place even when things can be crazy at Cat.

It’s more than just the contrator-ownership dilema, I’ve dug myself quite a hole over the last 8-10 years, gaining a reputation that I think many would appreciate, but for me only lately increases the stress. I don’t need to parrot every accolade I’ve ever received, that’s not the point, but to make the point as clear as I can - I have a lot of respect from quite a few people at Caterpillar. I’m blessed to have that reputation, and it’s not like I haven’t worked hard for it - but people talk about me in a way so flattering I feel like the main character in a fictional story sometimes.

In a fictional story I can check out the ending, hit up spark notes, or ask AI how it ends… but there isn’t an “end” in my real world scenario, there’s only tomorrow and I feel the pressure of not knowing what tomorrow holds now more than ever.

Danger

Being noticed is starting to feel more costly than rewarding…

Financial irony #

What’s the cost? It’s hard to get specific without writing a novel but here’s the TLDR - because I’ve been pretty good at what I do I’ve been able to do this type of work outside my normal 9-5 responsibilities and with that extra work has been some pretty great financial benefits. However with Cat changes, responsibility increases, and now owning code that others (and their clankers) write, the extra time I gained for myself is eaten-up and has been reclaimed by the mega-corp… “Exceeding expectations” every year just meant the bar is raised, the expectations are higher, the time-commitment requirement is higher, and as I’ve had to meet the requirements of both the new world and the curse of being noticed, I’ve lost the time for the extra work… For years I’ve realized the benefit of my own skills and drive, but the irony of agents (and a handful of other things) is that with the dramatic increase in expectations, not only on me but on those I work with and therefore their output, I don’t get to realize the benefits of my own gifts anymore.

Meaning and fatigue #

I feel very torn because the work I’ve been called into with Cat is good, I said in Cat Autonomy 2.0 that autonomy will save people’s lives. I love getting to participate in that mission, it’s the primary reason I didn’t jump ship to try to maintain the levels, and type, of work I was doing before… But in a few short months the mission is being drowned out by expectations and requirements that are so high I’m losing the grip on my own life.

Open questions #

That leads me to questions that I can’t answer, the question I ask daily now of “What about tomorrow?”. What will agents do for us tomorrow, what problems will be solved, what bugs will I create (by agents of course because I’ve never written a bug by hand in my whole life \s). If I stopped using agents would people still be impressed? Would it even matter?

Fin #

I’m certainly not anti-AI, it’s typing all my code. I’m not anti-collaboration, although I do wish I could work alone with just my clanker-army to worry about. I’m not sure what I am anymore though… AI has changed how I work, what I work on, and who I work with… Everything has changed in such a short period of time and like the ending of this post, it’s pretty jarring.

Danger

Death comes to us all - James Acaster.

Thanks for reading.

Remote Terraform State Requires Working Traefik… DUH!

2026-03-23T08:12:25Z

I’m working on some spring cleaning in my homelab and backed myself into a hilarious corner yesterday. I use Open Tofu for any of my Terraform needs now, and although I don’t manage a ton with terraform, I do manage all my cloudflare stuff with it. I decided I wanted to use my own minio instance as the s3 remote state backend for my workspaces so I could rely on my typical NAS data backup/retention workflow for the buckets in case anything went wrong, as opposed to a local state file that I’m not taking a lot of precautions with. Well during my Spring Cleaning I was working towards replacing ingress into my home network with Cloudflare tunnels and in the midst of that update I took down traefik, no matter a simple ’tofu apply’ should get me right back to working order…

╷
│ Error: Error inspecting states in the "s3" backend:
│     operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: , HostID: , api error NotFound: Not Found

Hilarious problem with thankfully an easy fix… downloading the state file from Minio wasn’t a big deal since the container was still running without issue, and placing the state file in the folder to use as the local state solution for the interim went totally smooth, but this highlights the set of interdependencies I’m creating for myself and as I take the next few days/weeks to do some spring cleaning I’m hoping I can separate out the external ingress from internal with a bit more clear boundaries so that I never lock myself out of a workflow I only execute on my LAN in the first place!

To Live In A World Without AI

2026-03-22T14:50:38Z

I’m finding lately that I wish we could go back to pre-ChatGPT… A world without a code-gen easy button, where “easy” was LSP autocomplete, where tools were at my fingertips rather than remote inference endpoints, and where I and everyone I worked with, was paid and judged based on what we could produce with our own 2 hands, even if ctrl + c and ctrl + v was sometimes a common set of keys… it sure beat answering “yes” to a robot every 38 seconds.

paynepride dot com outage on vacation

2026-03-16T13:21:31Z

The day after I leave for vacation I start getting SSL errors on every homelab service I host for myself and others. The culprit was my Cloudflare API token expiring. It was easy to find the 403s in the logs for Traefik (thank goodness for Tailscale getting me into the lab from afar). The solution was to rotate the API token, replace the value in Traefik’s .env file, and hit it with the “just deploy” button. Now I don’t know why this expired - the key looks like it has no expiration to me - and I’m too tired from the beach to dig in further. Until next time, I expect this error to come back March 16 2027 I suppose.

My Thoughts on Beads

2026-03-03T05:00:47Z

Steve Yegge is a pretty well-known individual in the tech field, having been around for a long time at some of the larger companies. He’s making quite a splash in the agentic coding world as well. I’ve appreciated Steve’s posts and projects lately and wanted to put some thoughts on one called beads.

Beads #

Beads is an issue tracker with links - issues relate to and block each other, but agents use beads to keep track of information and dependencies without storing it in their context 100% of the time. It seems like a very popular and useful tool - but I am not using it, and that’s what I wanted to capture… why not?

The answer for me is about where the organization layer is for the developer. Beads exists in a single repo - it’s a system-wide CLI but you ‘bd init’ in a git repo, and beads uses the .git/ folder, worktrees, dolt, and some git hooks to operate within that git repo. Outside the repo, it takes another tool to tie together all the beads databases you might have.

For me, I’m hardly “in” a git repo anymore. My workflow is that when I have something to work on, I create a “workspace” (self-defined concept) which is just a folder on my filesystem where I check-out git worktrees from any of the repos related to the work I’m doing. Sometimes it’s 1 worktree from 6 repos, sometimes it’s 6 worktrees from 1 repo for parallel work…

So because I like to organize myself in this way, beads is already “out” for me. That’s the main reason - I don’t have any real technical issues with beads or any criticism, it just is designed for a workflow that is not how I work.

This is why I’m building Nexus, something I hope to be able to put out there “soon”. It won’t be as general-purpose as beads, but my goal with it is to be plug-and-play for any agentic harness (copilot cli, claude code, opencode, etc.). It’s a challenge thinking about it as a personal tool but also as a tool to share someday, but agentic coding is making it possible to make some cool shareable stuff and I’m excited for my own workflow-task-manager to mature and at least become something useful to me (it already is, but building the plane in the air makes it kind of hard to enjoy the plane).

Credit #

banner image from ChatGPT

Forge Ahead

2026-02-17T05:14:15Z

Yesterday’s Reflection - Contentment and Work has a second-part this morning. As I was wrapping up a project I didn’t realize the closed-off-ness of leaving… I handed in some notes, and mid-message to someone the clock struck midnight and I was locked out. It’s fitting to be honest, and now in the wake of yesterday’s contentment post, experiencing some more loss than I expected, today we forge ahead.

My main focus for work now is a project I will refer to as forge. It is what I will bring to Caterpillar Autonomy but I can only build it thanks to the gifts and experience God has given me. The thing I want to put on paper is a short list of experiences I think God in his sovereignty, gave me over the last few years and what they provide for me for Forge.

Cat Reman Platform #

One of the first things that comes to mind is another project I was sad to lose: a platform I started for an analytics team in Cat Reman. That began as a simple python cli to automate some developer operations that was otherwise a dozen clicks through the AWS console. Using [boto3] and some simple python I gave that team the start of some real velocity gains. Eventually that grew into a larger kubernetes-based service where the data science operations: code quality, deployment, updates, webapps, etc. were all handled by our platform. It wasn’t perfect, it wasn’t self-contained, it was a set of things kind of glued together with systems and scripts, but it worked, it works today and is under fantastic ownership.

What I learned just from starting that cli was to be passionate about solving problems. No one asked me to make it, but it needed made, and the team is in such a better place for me having started it.

Kedro and OpenShift #

Another short project I was able to participate in a few years ago was leading a data-syncronization task into a fiery horrific crash that lasted weeks - neigh months longer - than was necessary or appropriate… The details aren’t relevant - I was the lead dev in a new place tasks with syncing up data between 2 applications. Ultimately, could’ve been a python script but I over-designed a kedro-based solution because of some requirements I misunderstood. Part of that misunderstanding was not knowing how to get the requirements I needed, but they weren’t provided in full, I didn’t know any better, so for weeks we were a corporate meme trying to use python to update a database that no one on our team understood if we were allowed to write to…. It was very confusing.

But what I learned was a lot about gathering requirements, questioning assumptions, and the importance of understanding your constraints as fully as possible as early as possible.

A Real Platform #

Then one of the biggest blessings I see for Forge, is the experience I’ve gotten recently with AWS at scale… Not 1 or 2 services created with Medium.com copy pasta tutorials from docs… but experience working in a large project across many accounts, supporting several teams, using a wide variety of technologies from Terraform to Kubernetes.

On this project I learned about practical system design and gained a lot of confidence in supporting systems that are complicated… I’m not the smartest guy in the world but I’m no dummy, and even while lacking fluency in the system I was supporting on this project I learned a lot of troubleshooting skills and gained confidence in my ability to troubleshoot complex systems.

I’ll need this for Forge - which will break certainly, but I’ll be there to fix it and I’m sure I can because I’ve done these other things.

Fin #

The Lord has been with me through these projects - he has certainly blessed me with a skillset and personality that lends itself to being really useful in the Tech world I’ve landed in (which is a whole ’nother story of God’s sovereignty and provision). I am very grateful for the swath of experience I’ve gotten over the last 8 years or so, and this season I’m in of change is rocking me a little more than I anticipated but by God’s grace I think I see the purpose, or at least a purpose, and I pray I am making the choices for work he wants me to make.

Kubernetes External Secrets Operator

2026-02-10T07:48:34Z

I wanted to put a short demo together of using External Secrets Operator (ESO) to expose secrets from a vault (like Hashicorp Vault, AWS Secrets Manager, etc) to services running in kubernetes

Demo code is here in this github repo

This post is a high level overview of the components, see the repo for the full example.

Setup #

docker for containerized development
kind for setting up a quick cluster
kubectl for accessing the cluster
helm for installing ArgoCD and ESO
and then justfile is there to wrap the commands to easier execution

Step 0 - Vault #

for the demo we’ll setup Hashicorp Vault in docker compose to easily bring it up and down
and the init-script is in the repo - it uses curl to make some secrets in vault that we’ll reference later

services:
  vault:
    image: hashicorp/vault:1.18
    container_name: vault
    ports:
      - "58200:8200
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: root
      VAULT_DEV_LISTEN_ADDRESS: 0.0.0.0:8200
    volumes:
      - vault-data:/vault/file
    cap_add:
      - IPC_LOCK
    command: server -dev -dev-root-token-id=root

volumes:
  vault-data:

bringing up the vault instance is a simple docker compose up (use the just recipes which some curl commands for checking status etc.)

curl -s http://localhost:58200/v1/sys/health | jq .  # or just vault-status
{
  "initialized": true,
  "sealed": false,
  "standby": false,
  "performance_standby": false,
  "replication_performance_mode": "disabled",
  "replication_dr_mode": "disabled",
  "server_time_utc": 1770893657,
  "version": "1.18.5",
  "enterprise": false,
  "cluster_name": "vault-cluster-acfb9930",
  "cluster_id": "4d9162f4-e501-371b-7f94-bd60052b40a3",
  "echo_duration_ms": 0,
  "clock_skew_ms": 0,
  "replication_primary_canary_age_ms": 0
}

Step 1 - App #

We need an app that requires secrets
app code in repo, essentially it’s a python webserver to show the vault values (obviously this would expose real secrets so it’s just a demo)
Below is one of the endpoints in the vibe-coded app, just to illustrate that we’re going to give secrets to the app as environment variables (or as mounted files!)
In the repo, the app is included and there’s a just build which builds the docker image
There is also just deploy which handles loading the image into kind’s image cache


# example route from demo-app - see git repo
@app.get("/env", response_class=HTMLResponse)
def show_env():
    # ESO brings Vault secrets into environment variables
    env_vars = dict(os.environ)

    # Sort by category, then by key
    sorted_items = sorted(env_vars.items(), key=lambda x: (classify_env(x[0]), x[0]))

    cards = "".join(create_card(k, v) for k, v in sorted_items)

    secret_count = sum(1 for k in env_vars if k in SECRET_KEYS)
    config_count = sum(1 for k in env_vars if k in CONFIG_KEYS)
    system_count = len(env_vars) - secret_count - config_count

    html = HTML_TEMPLATE.format(
        cards=cards,
        secret_count=secret_count,
        config_count=config_count,
        system_count=system_count,
    )

    return HTMLResponse(content=html)

def read_mounted_files(directory: str) -> dict:
    """Read all files from a mounted directory."""
    files_data = {}
    if os.path.exists(directory) and os.path.isdir(directory):
        for filename in os.listdir(directory):
            filepath = os.path.join(directory, filename)
            if os.path.isfile(filepath):
                try:
                    with open(filepath, "r") as f:
                        files_data[filename] = f.read().strip()
                except Exception as e:
                    files_data[filename] = f"<Error reading file: {e}>"
    return files_data

Step 2 - Cluster #

use kind to bring up a cluster
this will start a few docker containers to act as your control-plane and workers

# kind-config.yml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: eso-demo
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30080
        hostPort: 58080
        protocol: TCP
  - role: worker

kind create cluster --config kind-config.yaml --name eso-demo

Step 3 - External Secrets Operator #

installed with helm from the official helm chart
NOTE: this is the Operator, not the secrets… this is the thing which goes to the secrets backend and creates kubernetes secrets

helm repo add external-secrets https://charts.external-secrets.io 2>/dev/null || true
helm repo update
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --wait

In the repo this is mostly just eso-install

Step 3.5 - Secretstore #

You need a clustersecretstore to be the place that ESO puts secrets

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://10.10.0.1:58200"
      path: "secret"
      version: "v2"
      auth:
        tokenSecretRef:
          name: vault-token
          key: token
          namespace: external-secrets

Step 4 - Secrets #

Secrets go in the clustersecretstore
- in this example it’s called ‘vault-backend’
In the demo we can just kubectl apply -f <manifest> to deploy the secret to the cluster
In practice this should be handled by something more mature than raw-doggin kubectl commands

# manifests/external-secrets.yml
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: demo-app-secrets
  namespace: default
spec:
  refreshInterval: "10s"
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault-backend
  target:
    name: demo-app-secrets
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_PASSWORD
      remoteRef:
        key: secret/data/demo-app/secrets
        property: database_password
    - secretKey: API_KEY
      remoteRef:
        key: secret/data/demo-app/secrets
        property: api_key

Step 4.1 - Files #

ESO supports mounting files to containers as well through special ExternalSecret resources
One of the example seecrets is a TLS certificate

# manifests/external-secrets-files.yml
---
# File-based ExternalSecret for TLS certificates
# These will be mounted as files in /etc/secrets/
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: demo-app-tls-files
  namespace: default
spec:
  refreshInterval: "10s"
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault-backend
  target:
    name: demo-app-tls-files
    creationPolicy: Owner
    # Template to ensure proper file formatting
    template:
      type: Opaque
      data:
        tls.crt: "{{ .tls_crt }}"
        tls.key: "{{ .tls_key }}"
  data:
    - secretKey: tls_crt
      remoteRef:
        key: secret/data/demo-app/tls-files
        property: tls.crt
    - secretKey: tls_key
      remoteRef:
        key: secret/data/demo-app/tls-files
        property: tls.key

Notice how there’s a spec.target.template which templates out the file contents from the secret contents

Step 5 - Helm Chart #

This isn’t about setting up a helm chart so I’m not going to explain a lot but the working example is simple, not secure, and in the repo
The helm chart renders manifests - I’ve paired one down and added comments to the relevant things
The thing to just take note of is the reference of the secrets in the envFrom section

# deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    meta.helm.sh/release-name: demo-app
    meta.helm.sh/release-namespace: default
  name: demo-app
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: demo-app
        app.kubernetes.io/name: demo-app
    spec:
      containers:
        - envFrom:
            - secretRef:
                name: demo-app-secrets # name of example secret from section 4
            - secretRef:
                name: demo-app-config # another example in the repo
          image: demo-app:latest # the image you built and loaded into kind - simple 'just' recipe in the repo
          imagePullPolicy: Never
          name: demo-app
          volumeMounts:
            - mountPath: /etc/secrets
              name: secrets-volume
              readOnly: true
            - mountPath: /etc/config
              name: configs-volume
              readOnly: true
      volumes:
        - name: secrets-volume
          secret:
            defaultMode: 420
            secretName: demo-app-tls-files # example secret file from section 4.1
        - name: configs-volume
          secret:
            defaultMode: 420
            secretName: demo-app-config-files

Step 5.1 - Deploy #

We can deploy the demo-app from the git repo to the cluster
For a local demo a few things happen
- local image build
- loading that image into kind (kind doesn’t have access to your host’s docker image cache, so images need to be loaded into the cluster cache)
just deploy takes care of this for you, read the recipe in the repo if you’re interested in more there, the focus of this post and example are to briefly show how to use ESO though

Step 6 - Profit #

The example app just displays things that are mounted in - totally vibe-coded to illustrate the secrets mounting, not the appropriate way to leak secrets.

Cat Autonomy 2.0

2026-02-09T06:07:12Z

I wrote about my new role New Job - Caterpillar Autonomy a bit a couple weeks ago during an insanely busy time - having just started the role and wrapping up what ended up being side-work, paired with some sleep-deprived ADHD hyperfocus on my new responsibilities and Nexus, that post is less of an update and more of a mind-dump. This post is meant to be a calmer update about my upcoming time with Caterpillar Autonomy.

It starts, like all good stories, with Caterpillar’s RTO (return to office) mandate and how it has affected a lot of people negatively (in my opinion). I haven’t seen one positive remark about it fromanyone making less than $250k a year + Bonus + Equity if you catch my drift…

I know of people who really did lose their jobs at the end of 2025 for refusing to move. My story with RTO is covered elsewhere, I’d been blessed with remote work and was even secure in a remote position in the face of Caterpillar being willing to fire me After Exceeding Expectations For 4 Years.

But time passes and life happens, and the sovereign Lord brought me to this current circumstance: where I’m fully committed to Caterpillar Autonomy, and it feels weird to say it. I’m going to skip the RTO details, if you know you know. Where I’m at right now is on the other side (or very nearly on the other side) of a choice I didn’t feel freedom to make because it came from a conviction.

The manager of the group I’m in told me a story about working for another mining company, where he was managing mines in the Congo. These mines can be in the most remote of remote places on earth - in cultures where human life is definitely treated differently than in the first-world midwest USA. Operators of these huge mining trucks in these parts of the world can be smoking meth in the cabs - which obviously leads to unsafe operation. Safety of the mines themselves can be a second-thought as well, they collapse and then the people inside have to be excavated out. Sean told me he had come back to work after a weekend just to be told, as if it was no big deal, that “Motombu died” and to find another operator. Very crass attitude towards human life…

I’m still pretty frustrated at Caterpillar for how they’ve handled RTO (and corporitisms like “Caterpillar family” and “we bleed Cat yellow” are distracting nonsense), and part of me wants to leave still over the personal offence, but Caterpillar will make the safest mining equipment, without a doubt. And given that Cat will make the safest equipment then the story I was told and the mission I couldn’t help but believe in is that rolling out good autonomy software and solutions to these remote mines will save lives.

Warning

I’m obviously not like superman or anything, but something feels better about enabling good autonomy rather than supporting an online sales platform, or working for a bank…

I was invited (coerced?) into that mission - and my role in it is enabling the Autonomy group to manage data for their software stack. I get to help these engineers be experts in the things they’re experts in by offloading some operational overhead (data movement, pipelineing, infrastructure and devops, etc.) so that they can produce the best autonomy software on the planet.

Learning How To Agent

2026-02-03T05:50:33Z

Introduction & Background #

I’ve been using AI tools for codegen for a few years now, but not super heavily. I either use the in-line copilot stuff, which is like LSP on seteroids, or I have gone all the way to the other side of full on vibecoding with Windsurf. That’s been fun enough, but not super fulfilling and at the end of that I either have a simple webapp that does what I want but I don’t understand, or else a half-broken thing I tried to understand but couldn’t prompt in the right direction.

One Caveat

My one caveat with Windsurf, which is a full IDE that I don’t have a lot of comfortable navigation in due to all the Cascade keybindings clobbaring my own keymaps - is that with a HEAVY spec, it’s done will with the Claude models at implementing a real idea I have. It’s closer to vibe-engineering than vibe-coding, but that’s my only instance, the rest of my Windsurf usage is “make me a cool app - no mistakes”

This year it feels like some tools exploded and I mentioned this in New Job - Caterpillar Autonomy. I’ve been using opencode a lot over the last few weeks and have iterated many times already on a system of work that I’m trying to lean into for improving my efficiency.

The Problem Space #

My desire for a great Developer Experience is years old now, shout out to ThePrimeagen for his FEM course and Waylon Walker for being a constant source of encouragement to be the best developer I can be. I sometimes (often) get tunnel-visioned on developer-productivity initiatives and lose the forest through the trees when ironing out a workflow - generally to find out I way over complicated the solution OR worse, started solving a problem I don’t even have.

First Attempt: Local Progress Tracker #

Well that’s where I’ve been for a few weeks… I’m building Nexus, my second-brain at work to collaborate with agents on the truckload of stuff I’m expected to get done. For a while now I’ve had a “working-notes” repo, which is basically a blog, built with markata and navigated via markdown-lsp, where I do like what I do here - take daily notes, track projects and status, and it gets built into a nice little website I can reference with my boss.

Now that we have copilot in full-swing, I’m trying to integrate agents a bit more. Opencode has made this so nice - so many tools and modes of interaction, highly customizable interface but also an amazing default experience… So I was trying to lean into using agents and subagents for more work I started down the path of building out a progress tracker. I started with a simple “skill” that told the agent to put some info into a sqlite file, and even create the file and schema based on the work. This worked fine, but was specific to each repo (and actually each worktree I was in) and it was hard for me to get visibility into all the work my fleet of agents was doing. Now that’s actually problem 1 - I don’t have a fleet of agents, I had some terminal sessions going with opencode, but I got it in my head that I was going to have an army of Claude’s on my computer, constantly and autonomously knocking out tickets and I needed to know who was doing what, where, when, and why..

Nexus V1 & V2: The Over-Engineering Phase #

So, I dropped the local “progress tracker” and jumped into a huge FastAPI project that I called Nexus. It was a python cli + api, with a server for centralized management. It presented a kanban board, had policy gates on “plans” being approved before work could start on an Epic (and therefore any child tickets). It has worktree tracking and automation, etc. It had a lot… it didn’t all work, and it was hard to build the autonomous system… I wanted agentic feedback loops where voidshaper and I made epic plans for Epics (see the pun?) and then once the Plan was approved star-commander comes on the scene and makes tickets or checks tickets, depending on what’s already ready to go it farmed out the work to starsmith (and variants for complexity) to build and then automatically calls in recon-officer and qa-engineer and at the end of it gatekeeper came in and approved or denied the changes. If denied - automatically start the loop again, tracking the work in Nexus, if approved - rebase and merge the branch, clean up the worktree, update the ticket, close it, and get working on the next thing that opens up. Ticket dependencies were in there, tracking stale agent sessions, clever routing of tasks to smaller models where appropriate…. you can see that I went too far too fast too hard.

Nexus went through 2 or 3 iterations of this feature set. I was building it with several constraints in mind - specifically at work. 1. rate-limiting on large models (so not just using opus for everything), 2. good stewardship (not burning down cities for docstrings - farm out that work to haiku or a -mini model), 3. Copilot support - even though I like to use opencode, I wanted to supper copilot cli and in vscode as a means to share this with my coworkers who were mostly using vscode, 4. agent sessions were dying so I needed some kind of liveness probe for manual intervention in sessions where maybe the vpn died and the agent lost network… stuff like this was on my mind - but you know what wasn’t? actually doing some work… I was solving problems I don’t have, but were fun to think about.

The Reality Check #

So, what are my problems? I boiled them down to a few things…

I want to manage my workstreams in workspaces - folders on my computer where I put git worktrees
I want varying levels of ai automation in my workflows… some things could be handled by an agent fully autonmously, but most things I’m at least paired up, reviewing diffs still, doing research while working, etc.
In those varying levels of automation, I wanted the same amount of task tracking to a centralized location

And after 2 weeks of ADHD-driven hyper focus, and many iterations on Nexus, then Nexus V2…. what happened is that Nexus V2 died, and we say long live Nexus V3!

Nexus V3: The Pragmatic Pivot #

Nexus V3 is the same idea, but different approach… I’m leaning into opencode tooling specifically, dropping my care to support copilot given the lack of features. I’m using a few opencode plugins (opencode-notify and opencode-background-agents and a few commands. Instead of building my own tracker, I’m using kanboard because it’s basically a feature-complete agile/sprint/kanban board that I use at home, has a simple plugin ecosystem for light customization, and solves practically every status-tracking problem I tried to build from the ground up initially - ticket dependencies/linkages, actions to change ticket colors for a simple intuitive UI based on state, easy columns and tagging configuration, a simple API and there’s even an mcp server.

So, I’ve given up on the full automation for now, although opencode-pilot looks VERY PROMISING for this in the future. Today though, through some simple commands to give to agents for updating kanboard, I manually put them in worktrees, and they get to work - the tracking and human-in-the-loop model is going well for the work I need done.

Lessons Learned #

I learned and relearned plenty of lessons on this over the last 2 weeks… Data models matter more than almost anything, well-defined workflows are required if you want agents to help you iterate, and not everything has to be a product… That last one’s personal, but every time I have an idea I think is good, I’m sure it’ll be something to share, but a good lesson for me is to just build the things I need for me, and eventually maybe it can be cleaned up to share, but when I start building something with anyone other than me in mind, I’m in for a long hard journey

Current State & Future #

So what’s the summary? I don’t think I know how to agent super well yet - but I’m trying to get better to stay on the forefront of my co-workers who I see using AI in simple and sometimes scary ways

Copilot x sudo

Do not give copilot sudo on your CI server and say “fix my problem”… are you retarded???

I am not going hardcore with ralph wiggum or gastown - although those inspired Nexus v1 and v2, but I am dialing in my agentic workflow with some simple specialized agents, farming out work to subagents for context management, planning ahead of time to put appropriate context in a ticket, and getting close to having agents check out tickets, make worktrees, and do simple work by themselves (this was working in Nexus V2 but only intermittenly).

Someday I’ll open-source Nexus and share the configuration and workflow, for now it’s private as I’m actually using it to build out what I want rather than trying to recreate gastown with a cool space theme.

New Job - Caterpillar Autonomy

2026-01-26T10:24:54Z

In I'm back from the dead I mentioned my new role that started this year - it’s a return to Caterpillar Autonomy. I built some data pipelines and junior-grade infrastructure 5 years ago but left over burn-out at the prospect of a job with smaller scoped work and more technical guidance. That was a good move, and in God’s sovereignty he’s brought me back. Now I’m looking at code, thankfully not that I wrote, but that the guy after me wrote - it accomplished a job, like my original work did, but also like what I built back then - there’s better ways today. The amount of things in front of me is absolutely daunting, and the people on the team have been seeded with a very high opinion of me… I don’t know how much is valid and how much is hype, but I’m gonna try to live up to it.

One way I’m striving to do that is to maximize my efficiency with the AI coding tools available to me. We have Github Copilot and I’ve picked up a lot over the course of the last few years (and especially more recently) in using agents, planning out work, documenting plans for agents to follow, splitting up work via worktrees, etc. I feel pretty good about where I’m going and someday I might even release Nexus. Nexus is serving as my second-brain at work, the hub where I collaborate with my fleet of agents on the work that must get done. Details to come on this, but the thing that matters is that in ironing out workflow I’ve moved from prompt + chat to really managing a long-lived stream of work. AI has been changing the world, and there’s been developer hype for years now. But “make me a cool app, no mistakes” isn’t going to cut it. Even “make an app but generate a plan first” isn’t going to cut it… Developers have to adopt a higher level role, a systems-oriented and architecture-driven worldview must become primary in order to keep up. Code-gen and syntax writing are not what developers were ever paid to do, although many believed so. Our jobs have been to solve problems and deliver code that solves the problem. The code is cheap now, but solving problems still is a human task.

In the Autonomy group - there’s problems all up and down the stack. My focus is on infrastructure and developer operations - it’s become my bread and butter over the last several years. I’m excited to help the team grow, and I’m excited to grow personally/technically as I lean into the agentic workflow to produce code that I’m actually proud of, that has my name on the commit, and that solves real problems.

Example - Local Development #

One of the first things I’m tackling is a developer-pain-point of working on their laptop. I’ve been in this space for years, mostly with python programmer who are writing data science code. They don’t know about virtual environments, checking $PATH, assuming bad state in their terminal session, how to configure VS Code, etc. Often they just want to write some scripts and somehow test it. The solution I see most often is for devs to write code in JupyterLab/Notebooks in AWS or some environment close to their data - this is fine I guess, but it’s not developing good pipelines, and it’s tedious as hell. In my last job I helped set developers up with workflows that allowed them to run their IDE of choice locally (getting all the goodies of syntax highlighting, LSP, etc) and a CLI that took their code and ran it in the cloud, right next to data, in the same way that prod runs. It was a hit. After that I introduced some tools to help them manage python environments - we had strict templating requirements in our projects, so making tools to automate those things wasn’t too hard - it’s much easier than trying to make something flexible for every use case. The opinions made the automation and tooling easy to make and distribute.

Well I’m up against a similar task now, but oh so much worse… Larger team, larger environment sprawl, larger infrastructure mismanagement, the whole gambit. And I’m here for it… Here’s the first problem I’m addressing - local development for Airflow DAGs that run in an Airflow deployment on Kubernetes. The deployment itself is a little odd, Airflow is an orchestrator, all the pipelines run in external AWS Batch jobs - so a DAG hits the Batch API to run the code. The design there is actually nice, but how are devs testing code?

Oh that’s easy… they SSH into the prod server, which is a 5 year old desktop THAT I BUILT WITH A CO-WORKER BEFORE I WAS IN AUTONOMY THE FIRST TIME… hold on, WHAT!? Yes, it’s true.. so they SSH into the prod server, run some bash scripts in their userspace that setup airflow and a few db utilities in a docker compose stack, authenticate with AWS themselves from that server, and then they run DAGs against real data to test it… I am beyond shook.

Here’s what I’ve put together - a bootstrap process (I like just + PEP 723 python scripts as opposed to bash, but to each their own, and bash of course has its place) that spins up a kind cluster on their laptop, the process pulls some private images and loads them into the kind cluster, it installs airflow from a helm chart (the same helm chart we’ll use in dev and prod… no more docker compose over here, kubectl over there), and everything just. comes. up. No SSH into ancient server, no touching cloud infra (they get MinIO and a DB container to emulate S3 and RDS in our AWS accounts), no sweat on testing DAGs. They stage some data in MinIO (just open minio handles the port-forward and opens the browser), then just open airflow (their DAGs are hot-reloaded via hostPath mounting), and they can run DAGs locally until they’re satisfied with the results.

It’s taken me about a week of split-focused effort (I mentioned Nexus and I’ve been co-building and dogfooding that at the same time) but I’m proud of that local setup now. I am a bit shocked they’ve dealt with a brittle, hacky, often broken development workflow for the last 4 years or so, but that development I suppose.

The Point #

This post wasn’t meant to be me glazing myself for awesome local development practices, I am simply excited about this new chapter. I love solving problems, and I love owning those solutions. There’s a whole mess of things to address, my mind is buzzing, and I feel like God has blessed me with renewed passion (again see I'm back from the dead). I’ve given up some pay and some freedom to take this role, but I think it’ll pay dividends.

Life happens to all of us - this role change affected a lot for me, it’s been hard to digest some of those changes, but the work is good, the development is fun, the new world of using agents to fly through things you’re fluent in is exciting, and I’m here to help a team that desperately needs it to improve their lives and the work we do for Cat Autonomy.

I’m still pissed at Caterpillar Executives for RTO ruining parts of my life, but in God’s sovereignty their idiocy has led to non-trivial blessing, so I can say “Praise the Lord”

Increase inotify limit in your CI workers

2025-12-30T06:11:49Z

Today I tripped over a CI failure that I had to think about for a while.

I build zensical static sites in CI on my Forgejo instance. These builds had been working fine, then suddenly started failing with no code changes. Naturally, I assumed something upstream broke — maybe a new uv release, maybe zensical.

I pinned versions.
I tested older versions.
Same failure every time.

That ruled out regressions and pushed me toward the environment.

I pulled the runner and worker images locally and built the sites just fine… But that doesn’t perfectly emulate the CI setup - my forgejo runner relies on docker-in-docker and so we aren’t just running a container on a host, we have this middle layer to consider… I wasn’t sure how to really test this out locally so I succomed to AI and here’s where Jipity got me in about 5 minutes…

The failure #

The builds blew up with:

thread 'zrx/monitor' panicked at .../zensical-watch/src/agent/monitor.rs:154:49:
called `Result::unwrap()` on an `Err` value:
Error { kind: Io(Os { code: 24, message: "Too many open files" }) }

At first glance it means nothing to me but Jipity says this screams ulimit.

So following the AI overlords I checked:

ulimit -n was already very high

/proc/sys/fs/inotify/max_user_watches was also very high

PID limits were not constrained

Everything looked fine according to Jipity.

Yet the panic persisted.

The real culprit #

The actual limit being hit was:

/proc/sys/fs/inotify/max_user_instances

In my Forgejo runner container, it was set to 128.

That turns out to be far too low for zensical.

Here’s what ChatGPT said:

Even during a normal zensical build, the tool spins up its watch subsystem, which creates many inotify instances. Once it crosses the kernel limit, inotify_init() fails with EMFILE, and the process panics because the error is unwrapped.

The fix #

Raising the inotify instance limit fixed it immediately:

echo 1024 > /proc/sys/fs/inotify/max_user_instances
RUST_BACKTRACE=full uvx zensical build --clean

After that, the build succeeded consistently.

Prettier Docker ‘ps’ Command

2025-12-29T05:33:05Z

The docker ps command is very useful, but I hate reading the output. Turns out, you can make it prettier:

docker ps

–format “table” is implied with the command.


❯ docker ps --format "table" | grep can
9b716a7d1ab0   postgres:17                                               "docker-entrypoint.s…"   13 hours ago    Up 13 hours (healthy)            0.0.0.0:5432->5432/tcp, [::]:5432->5432/tcp                                                cannalyzer-db-1
f7708ea0c112   adminer                                                   "entrypoint.sh docke…"   13 hours ago    Up 13 hours                      0.0.0.0:8081->8080/tcp, [::]:8081->8080/tcp                                                cannalyzer-adminer-1
93cd67719ed4   frontend:latest                                           "/docker-entrypoint.…"   13 hours ago    Up 13 hours                      0.0.0.0:5173->80/tcp, [::]:5173->80/tcp                                                    cannalyzer-frontend-1
f517625fca98   traefik:3.0                                               "/entrypoint.sh --pr…"   13 hours ago    Up 13 hours                      0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:8090->8080/tcp, [::]:8090->8080/tcp           cannalyzer-proxy-1
f4daa036216e   schickling/mailcatcher                                    "sh -c 'mailcatcher …"   13 hours ago    Up 13 hours                      0.0.0.0:1025->1025/tcp, [::]:1025->1025/tcp, 0.0.0.0:1080->1080/tcp, [::]:1080->1080/tcp   cannalyzer-mailcatcher-1

But you can pass a template string to the --format option, like so:

docker ps --format "table {{.Names}}"


✗ docker ps --format "table {{.Names}}" | grep can
cannalyzer-db-1
cannalyzer-adminer-1
cannalyzer-frontend-1
cannalyzer-proxy-1
cannalyzer-mailcatcher-1

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"


❯ docker ps --format "table {{.Names}}\t{{.Ports}}" | grep can
cannalyzer-db-1             0.0.0.0:5432->5432/tcp, [::]:5432->5432/tcp
cannalyzer-adminer-1        0.0.0.0:8081->8080/tcp, [::]:8081->8080/tcp
cannalyzer-frontend-1       0.0.0.0:5173->80/tcp, [::]:5173->80/tcp
cannalyzer-proxy-1          0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:8090->8080/tcp, [::]:8090->8080/tcp
cannalyzer-mailcatcher-1    0.0.0.0:1025->1025/tcp, [::]:1025->1025/tcp, 0.0.0.0:1080->1080/tcp, [::]:1080->1080/tcp

Picture #

I noticed my template isn’t folding the codeblocks in a way that actually makes this post look like I’m lying!

Setup A Cloudflare Tunnel With Terraform

2025-12-12T21:06:59Z

I am cooking up some stuff at home and want to put it on the interwebs, but I don’t want it on the same infra as my homelab. Now… I only have a server or 2, so to some degree it will be, but networking-wise I didn’t want to funnel extra traffic through my reverse proxy.

So, I’d heard about Cloudflare Tunnels - they sound like P2P VPN to me, but I know there’s layers of the networking stack I’m blatantly ignoring. “What the tunnel is” isn’t much the point - I’m here to show you how to set one up and get yourself a fancy https://app.mydomain.com for your web app running kind of wherever you want

Example Repo linked at the bottom

Requirements #

Terraform or open-tofu. I currently use open-tofu but either would be fine. brew install open-tofu is a simple way to get going
Cloudflare account with a domain
API token with permissions:
- Account:Cloudflare Tunnel:Edit
- Zone:DNS:Edit
cloudflared (the example repo runs cloudflared in a docker compose stack)

Tunnel #

The module is simple and has just a few resources:

❯ tofu state list
module.tunnel.cloudflare_record.tunnel
module.tunnel.cloudflare_tunnel_config.this
module.tunnel.cloudflare_zero_trust_tunnel_cloudflared.this
module.tunnel.random_id.tunnel_secret

We see there will be the a DNS record that tofu references by the key “tunnel”. There is a tunnel configuration resource, the tunnel resource itself, and finally the associated secret required for the cloudflared daemon that will run alongside your webapp.

To get started you’ll need to fill out the example terraform.tfvars file with your info:

# Copy to terraform.tfvars and fill in values
# DO NOT commit terraform.tfvars to git

cloudflare_api_token  = "your-api-token-here"
cloudflare_account_id = "your-account-id"
cloudflare_zone_id    = "your-zone-id"
domain                = "example.com"
subdomain             = "app"
tunnel_name           = "my-tunnel"
origin_service        = "http://localhost:8000"

You can grab your account id and zone id from Cloudflare’s dashboard for your domain. It’s near the bottom of the Overview page

Then I presume you have a domain already, but if not hop over to namecheap to snag one and then register it with cloudflare so they can manage your DNS. I have terraform for this as well, a future blog post will combine this with a fuller terraform’d cloudflare setup for simple domain use cases

Once you fill those out, hit it with the tofu init and tofu plan to see what’s up

NOTE: terraform.tfvars is automatically sourced by terraform/tofu, you can name the file differently and then pass -var-file=myvars.tfvars to the commands

The initial plan should look something like this:


OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

  # module.tunnel.cloudflare_record.tunnel will be created
  + resource "cloudflare_record" "tunnel" {
      + allow_overwrite = false
      + comment         = "Managed by Terraform - soonish-tunnel"
      + content         = (known after apply)
      + created_on      = (known after apply)
      + hostname        = (known after apply)
      + id              = (known after apply)
      + metadata        = (known after apply)
      + modified_on     = (known after apply)
      + name            = "app"
      + proxiable       = (known after apply)
      + proxied         = true
      + ttl             = (known after apply)
      + type            = "CNAME"
      + value           = (known after apply)
      + zone_id         = "<REDACTED>"
    }

  # module.tunnel.cloudflare_tunnel_config.this will be created
  + resource "cloudflare_tunnel_config" "this" {
      + account_id = "<REDACTED>"
      + id         = (known after apply)
      + tunnel_id  = (known after apply)

      + config {
          + ingress_rule {
              + hostname = "app.notifiq.net"
              + service  = "http://localhost:8000"
            }
          + ingress_rule {
              + service = "http_status:404"
            }
        }
    }

  # module.tunnel.cloudflare_zero_trust_tunnel_cloudflared.this will be created
  + resource "cloudflare_zero_trust_tunnel_cloudflared" "this" {
      + account_id   = "<REDACTED>"
      + cname        = (known after apply)
      + id           = (known after apply)
      + name         = "soonish-tunnel"
      + secret       = (sensitive value)
      + tunnel_token = (sensitive value)
    }

  # module.tunnel.random_id.tunnel_secret will be created
  + resource "random_id" "tunnel_secret" {
      + b64_std     = (known after apply)
      + b64_url     = (known after apply)
      + byte_length = 32
      + dec         = (known after apply)
      + hex         = (known after apply)
      + id          = (known after apply)
    }

Plan: 4 to add, 0 to change, 0 to destroy.

As long as that looks good you to, then we tofu apply next (type yes when asked or pass -auto-approve)

Afterwards tofu state list should show you the 4 resources, and if you go to your cloudflare zone’s dashboard you should see the CNAME associated with the tunnel address

Daemon #

Run the compose stack or the binary itself. Get the token from terraform state with tofu output -raw tunnel_token.

TUNNEL_TOKEN=$(tofu output -raw tunnel_token) docker compose up -d will do you nicely

Enjoy your tunnel!

example repo