Building my first agent workflow: A skill issue...

I thought I understood it until I spent over a month building an agent to manage our CRM that made several bad updates, including repeating one update 44 times. This was the first clear warning. After restarting the agent, I added ~750 lines of code to compensate for one edge case after another, cut most of it, added some back, and still watched a single thread generate 10 proposals for the same relationship. Eventually, I turned the agent off and looked for a better solution.

I spent all of this time building a real system. It handled sensitive emails, routed them to the right deal, proposed CRM changes, and asked for human approval via Slack. The work itself was useful, but the implementation was not.

My Problem

Part of my job as an investor is keeping our CRM current so we can manage relationships and run fundraising processes efficiently. The work is time-consuming and repetitive, especially when we have several live processes at once. I had used the CRM API heavily during an earlier migration, so I knew much of it could be automated.

At the time, I also wanted an excuse to work with the new tools that I was reading about on TechCrunch and on X – agents, hosting environments, and evaluation harnesses were tools that I wanted to gain experience with on a real project. I have a computer science degree, so all of the buzzwords are familiar, but this was my first time setting up an agent to be run against live input. My mistake was letting my interest in the tools determine the architecture before I had proved what the project required.

Starting Off: The First Three Weeks Produced Nothing

Prior to building, I spent three weeks working through internal compliance, data-governance, and budget questions. Given the sensitive nature of the data processed (email conversations), my internal team (mainly my compliance officer) was rightfully concerned about security.

I mapped the data flow, researched vendors I wanted to use, reviewed their security practices and histories, and explained the system to nontechnical colleagues. I also had to make the case for the additional spend. None of this was part of building an agent, but it forced me to map out and understand the system I wanted to create before I gave it access to real data.

The First Iteration

Thankfully, setting up the infrastructure was easier than navigating the approval process. DigitalOcean, Cloudflare, and Claude made it possible for me to get the first version running even though cloud deployment was new to me.

The first version followed a simple order. Microsoft Graph sent an email to the system. Deterministic address matching decided whether the message might matter. A router selected the right deal skill, which contained that deal's taxonomy. A proposer recommended CRM updates. Hardcoded rules blocked writes to fields the system was not allowed to touch, and then Slack messaged me to approve or decline each proposal.

Email -> Router -> Deal skill -> Proposer -> Allowlist -> Slack approval -> CRM

This initial version worked, but its judgment was unreliable, causing several errors in the first day. The router chose the incorrect deal too often. The proposer wrote too much or not enough, moved stages too easily, and sometimes repeated the same update several times. It was also a headache being pinged by Slack and Outlook for the same event. The final straw on the first iteration was when the system missed several relevant outbound emails and reported on an internal discussion that was irrelevant to the CRM.

Building an Eval

To start minimizing these errors, I built a simple HTML page and labeled ~500 real emails, acting as both the router and the proposer agent. I used 284 examples for development and held back 116 for testing. The first score was pretty terrible: 58% routing accuracy and ~70% proposal accuracy.

This was the first useful result of my project so far. I no longer had a vague sense that the agent was unreliable. I knew what was failing, had clear examples of the failure, and could now start testing changes to understand exactly what would help.

My next step was A/B testing two router designs. One used a cheaper model to decide whether an email mattered (gpt-5.4-mini), then passed relevant messages to a more expensive classifier (gpt-5.5) to determine which deal an email belonged to. The other fused both decisions into one call on the more capable model (gpt-5.5). The fused router scored six points higher on the held-out set. The split design was cheaper, but not enough to justify the loss of accuracy.

For the proposer, I ran a three-model test across GPT-5.4-mini, GPT-5.5, and Claude Opus 4.8. I graded multiple batches by hand and selected the best result per dollar rather than the model with the highest raw score.

These changes raised accuracy from 58% to 92% on the held-out set, good enough for me to turn the system back on and test it against my live inbox.

System Issues

I got carried away by the accuracy improvement and turned off manual approval for a day. A few hours later, it had made some incorrect writes based on outbound email. Then, after a restart, it replayed 44 stale auto-approved proposals. I turned manual approval back on within 24 hours.

What was frustrating was that the router eval had not really captured the things that broke. The model could identify an email correctly and still do the wrong thing with it. I had not tested how the system behaved after a restart, or the difference between our team sending an email and an investor actually engaging with us.

After that, I made my biggest mistake of the project: adding deterministic fixes one at a time. My post-model logic grew to ~750 lines of code, mostly responsible for handling errors or edge cases that had once gone wrong when the bot was live. Eventually, I could no longer explain the system or why each line was written, which was a sign for me to cut back.

I cut it back to the things that actually had to be deterministic and added a golden set so I could see what I broke before trying it live again. I overdid that cut too. I removed response confirmation and deal tagging, which should not have been left to model judgment in the first place. The eval caught that and I added both back.

A few days later, another email thread matched several deals and generated ten proposals for the same relationship. This was my last straw at the time, so I paused development and turned off the bot. I had stopped fixing individual bugs and started creating a bigger pile of code around them.

What I actually needed

When fundraising kicked off again for a new process, I searched for a new solution. I did not restart my agent, but I did use the old logs, decision files, and evaluation results for a much smaller solution: a simple skill file. Instead of running continuously, this skill runs on demand. It reviews my inbox and calendar activity, proposes changes to the CRM in one batch, requires my review of all changes, writes only approved updates, and generates an output for me to share with my team.

I typically run the skill once per day, and I can do so from my phone via a GitHub Action. It requires less infrastructure: no server and no webhook listener waiting for unbounded input. Despite being less impressive on a technical diagram, it is much more useful on a Thursday morning.

Most of the work I did on the original system carried over. The evaluation harness, routing logic, entity resolution, and approval boundaries all informed the development of the skill. Luckily, the time spent developing and debugging was not wasted. It helped me figure out which parts earned a spot in my current solution.

Lessons Learned

The bitter lesson in AI is usually framed as a warning against replacing general methods with piles of handcrafted logic. This project forced me to understand a more practical version of it. When my agent met the live inbox, I ended up building deterministic code to compensate for errors that grew faster than I could comprehend what was necessary. Eventually, I could no longer tell which rules were actually necessary and which were just patches for one bad email.

There are still things I would keep in code, like preventing writes to fields the agent is not allowed to touch or preventing a status from moving backward. But I did not need a server listening all day, trying to handle every weird email as it arrived. For this workflow, it is better to have the agent propose a batch of changes, review them myself, and then approve what gets written.

I made an even greater mistake before I even ran my ideas past compliance. I let my desire to test new tools dictate the solution that I built. I did not start by defining what I actually needed: a quick way to scan my inbox, keep the CRM current, and update my team on each opportunity. Instead, I spent time optimizing a workflow before I had established that it needed to run continuously in the first place.

Ultimately, this project reminded me of the value of staying close to the work. A fully automated update process would save me time and mental bandwidth, but it would also remove much of the context I pick up while maintaining the CRM directly. Over time, that would make me less useful to my teammates and less engaged in the work itself. The skill-based approach I'm running now offers a better tradeoff, allowing me to stay on top of all of the opportunities without having to spend hours each week manually updating the CRM.

I’ve been consistently amazed by the progress of LLMs and other AI tooling. Each major release can drastically change one of my workstreams in ways I couldn’t imagine even six months ago. Working on this project helped me realize that the point is not to automate every task simply because the tools make it possible. The better question is whether automation actually improves the way I work.