I Spent 3 Months Building My Agent. Here Are the 5 Failures That Almost Killed It.

I spent three months building a personal agent and learned something uncomfortable very quickly: the interesting part is not getting the agent to do something once. The interesting part is keeping it alive after the first week.

Everyone loves the demo phase. The inbox gets summarized. A task gets routed. A draft appears. It all feels inevitable. Then the real world shows up: providers block you, tools fight each other, models improvise at the worst possible moment, and the "obvious" automation path turns into a week of infrastructure damage.

I hit five failures that mattered enough to reshape the whole system. None of them were edge cases. They were architectural lessons I had not respected yet.

1. Email Broke First

I assumed email would be the boring layer. That was my first mistake.

I started with Gmail because it felt like the safest default. Three weeks later, the account was gone. No warning worth acting on. No real appeal path. Just termination. An autonomous system reading and sending email looked enough like abuse that Google's stack treated it exactly that way.

I moved to Microsoft's side and ran into a different version of the same problem: repeated human-authentication checkpoints that an agent simply cannot clear while you're asleep.

The eventual fix was not clever. It was GMX plus Thunderbird MCP. In other words: a quieter provider and a proxy layer that made agent behavior look more like normal desktop behavior.

Lesson: infrastructure built for humans is not automatically infrastructure that tolerates agents. For agent systems, boring often beats premium.

2. Diagram Generation Was Mostly Fantasy

I wanted the agent to read architecture notes and generate clean Mermaid diagrams automatically. It sounded like a small feature. It wasn't.

The syntax was often fine. The diagrams were not. Boxes overlapped, flows made no visual sense, and the output kept drifting toward something that looked technical without actually being useful.

The reason is simple in hindsight: text models do not really see what they produce. They generate something statistically similar to a diagram, but they do not experience the layout failure the way a human does.

The only setup that started working was a two-step loop: generate the diagram, render it, then let a vision-capable model critique the visual output before iterating.

Lesson: if a human would judge the result visually, a text-only generation path is incomplete by design.

3. Self-Deployment Was a Trap

At one point I thought, why not let the assistant deploy and configure itself remotely? It sounded elegant. It was not elegant.

It was four straight days of permission problems, dependency problems, SSH problems, package-manager problems, and environment-specific nonsense stacked on top of each other. Every blocker led directly to the next blocker.

That was the moment I had to admit something I should have admitted earlier: some forms of automation are so meta that they stop being leverage. They become a full-time hobby.

The healthier split was obvious once I stopped romanticizing autonomy. Human-managed DevOps handles provisioning and privileged changes. The agent handles monitoring, reporting, and operational follow-up.

Lesson: not every system boundary should be crossed by the agent. The control surface has to be designed, not assumed.

4. The IDE Should Not Be a Second Brain

I also tried giving the agent deeper control over the development environment itself: open files, coordinate with the IDE, work around Copilot, manage edits live.

What actually happened was a quiet war over the same surface area. The agent changed something, the IDE reformatted it, autocomplete jumped in, the agent reacted to a state that had already changed, and the whole thing became harder to trust than a normal patch flow.

That setup taught me a useful boundary: the IDE should be a tool of the agent, not a peer reasoning system competing for the same file.

The better model was simpler. Let the agent submit explicit patches, run commands through a controlled terminal path, and observe structured diffs. One actor at a time. Fewer surprises.

Lesson: if two systems are editing the same surface without a conflict protocol, you do not have automation. You have coordination debt.

5. Cheap Models Can Be Very Expensive

I started with a model choice that looked sensible on paper: fast, cheap, easy to call, good enough for a lot of tasks.

Then I watched it ignore constraints, invent workflows, generate code when explicitly told not to, and generally behave like a system that was optimized for responsiveness, not obedience.

That was expensive in the worst way. Not billing expensive. Trust expensive.

Once I switched to a multi-model routing setup, the whole system improved. Lightweight tasks went to cheaper models. More delicate reasoning and code-generation tasks went to models that were slower but far more reliable.

Lesson: model choice is not a branding decision. It is an architectural routing problem. The cheapest model in your stack can become the most expensive if it keeps creating rework.

What Survived

What came out of all this was not a perfect agent. It was a better architecture.

The winning pattern was modular and slightly less magical than I imagined at the start: separate transport from orchestration, separate deployment from operation, separate model routing from task logic, and treat failures as signals the system can learn from rather than as one-off annoyances.

That shift matters. Once you stop asking, "How do I make the agent do everything?" and start asking, "Which responsibilities should this system own directly, and which should it interface with carefully?" the design gets better fast.

That is the real lesson from these five failures. The system improved the moment I stopped designing for an idealized autonomous future and started designing for the messy reality the agent actually had to survive.