Your Agent Does Not Need Inbox Access. It Needs Email Intelligence.
github.com/ghassan-ai-projects/email-intelligence-platform — Python, SQLite, MCP, MIT licensed.
A few weeks ago, I gave my agents access to email.
Not the demo version where an agent reads one message and writes a summary. I wanted the useful version: follow a thread across weeks, find decisions, surface unanswered messages, extract action items, prepare a reply, and remember what happened the next time the agent runs.
I started where everyone starts: provider APIs.
Gmail gave me OAuth, refresh tokens, scopes, MIME payloads, and an integration shaped for a hosted application. Microsoft Graph gave me a different version of the same problem. Both APIs are capable. Neither was the abstraction I wanted for a local agent.
Then I tried a simpler bridge: GMX, Thunderbird, and a community MCP server. It worked. The agent could reach the mailbox.
That is when I realized access was the wrong goal.
A Mailbox API Gives You Messages, Not Understanding
Raw access creates a loop that looks productive but wastes work:
- The agent searches the mailbox.
- It pulls several long messages into context.
- It reconstructs the thread and decides what matters.
- The session ends.
- The next agent does the same work again.
The problem was not Gmail. It was not Thunderbird. It was not MCP.
The root cause was the level of abstraction. I was exposing a mailbox when the agent needed a knowledge system.
An email is not only a subject and a body. It may contain a decision, a deadline, a commitment, an unanswered question, an attachment, a relationship, and a piece of context that will matter three months later. If those stay trapped inside raw messages, every agent pays to rediscover them.
So I stopped building a better mailbox connector and built mailintel: a local-first email intelligence platform that turns mail into durable, queryable state.
The Architecture: Maildir First, Intelligence Second
The system has a deliberately boring data path:
IMAP → mbsync → Maildir → ingest + guardrails
↓
SQLite + FTS5 + sqlite-vec
↓
enrichment + knowledge tables
↓
MCP tools
↓
agentsmbsync handles provider-specific IMAP synchronization. Gmail, GMX, or another IMAP provider all become the same local Maildir tree. That tree is the source of truth. The intelligence layer never needs provider OAuth logic and never owns the mailbox.
Ingestion parses messages, extracts plain text, rebuilds conversations, records attachment metadata, sanitizes hidden control characters, and indexes searchable content. Everything lands in one SQLite file.
That choice matters. There is no hosted inbox copy, no Docker requirement, and no database server to operate. Backup is a file copy. Inspection is a SQL query. If the enrichment provider is unavailable, the local mailbox and index still work.
Email Is an Attack Surface
The first prototype treated email content as ordinary tool output. That was a mistake.
Email is third-party input delivered directly into an agent's reasoning loop. Anyone who can send you a message can attempt to place instructions in front of the model. A newsletter can consume a large context window. A quoted reply can hide text the human never noticed. A malicious message can ask the agent to reveal data or send it somewhere else.
So the security boundary starts at ingestion, before enrichment and before normal agent use. The current scanner runs eight detectors:
- Prompt injection — direct overrides, jailbreak language, and instruction probing
- Encoding anomalies — suspicious Base64, hex, URL-encoded, and nested payloads
- Size abuse — configurable byte and character limits to control oversized input
- Unicode attacks — zero-width characters, bidirectional overrides, homoglyphs, and unusual emoji density
- Structural anomalies — suspicious content types, attachment extensions, and abnormal headers
- Reply-chain manipulation — injection patterns hidden inside quoted history
- Repetition attacks — repeated lines, n-grams, and padding intended to dominate context
- Exfiltration requests — attempts to make the agent disclose secrets, files, or credentials
Each detector produces a score and warnings. The final score combines the strongest signal with half the mean of all triggered signals, then persists the verdict and its reasons with the email.
This does not make untrusted content safe. No regex chain can make that promise. It creates defense in depth: sanitization at ingest, a hardened enrichment prompt, explicit untrusted-content notices on MCP results, and constrained side effects if an instruction gets through.
The Important Boundary Is Read Versus Act
Reading an unsafe email is bad. Letting it trigger an unsafe action is worse.
Outgoing mail is disabled by default. When enabled, recipients can be restricted with allowlist patterns, local attachments must come from approved directories, and a per-hour send cap stops a runaway loop. Every MCP call is recorded with its arguments, caller, timestamp, and outcome.
The preferred workflow is draft-first:
create_draft → inspect or revise → send_draftCreating a draft has no external side effect. Sending it crosses a separate boundary and runs the same recipient, attachment, and rate-limit checks as direct sending.
That separation is more important than another prompt telling the model to "be careful." Prompts are policy. Boundaries are enforcement.
From Search to Durable Inbox Memory
Once an email is indexed, the platform runs one structured enrichment call. It extracts a summary, importance, sentiment, action items, entities, and facts. A separate embedding makes meaning-based retrieval possible.
The agent no longer has to ask, "Which emails mention Acme?" It can ask:
- What decisions did we make about Acme?
- Which commitments are still open?
- Who is waiting for my reply?
- Find discussions related to latency, even if they never used that word.
- What changed in this thread since yesterday?
The result is not a chat transcript pretending to be memory. Facts, action items, tags, notes, drafts, and events are explicit records. Agents can complete a task, add a note, change importance, or tag a message. The next run sees that state instead of rediscovering it.
An append-only event feed makes the loop reactive. Agents persist a cursor, ask for events since that cursor, process only the delta, and write their state back. No full mailbox scan on every run.
Local First Does Not Mean Local Only
The storage and control plane stay local. The intelligence providers are pluggable.
For enrichment, mailintel supports OpenAI-compatible endpoints such as DeepSeek, Ollama, Groq, OpenRouter, and OpenAI, plus Anthropic's native API. Embeddings can use Voyage or a local OpenAI-compatible endpoint such as Ollama with nomic-embed-text.
You can keep the entire path local, or make an explicit tradeoff and use hosted models. The architecture does not force that decision on you.
The MCP server uses stdio by default. A Streamable HTTP transport is also available for remote agents, bound to localhost unless deliberately configured otherwise, with optional shared-token authentication. TLS and stronger authentication belong at the reverse proxy rather than hidden inside a small local service.
What Is Open Source Today
The initial release is intentionally small in operational footprint and broad in agent capability:
- Maildir ingestion, conversation threading, FTS5, and semantic search
- Structured summaries, facts, decisions, entities, and action items
- Knowledge-level MCP tools instead of raw mailbox primitives
- Append-only events and agent write-back
- Draft-first replies and guarded SMTP sending
- Prompt-injection scanning, output notices, rate caps, and audit logs
- Resumable enrichment jobs with retries
- A test suite covering storage, search, security, MCP, agent loops, and transport
The project is MIT licensed because local infrastructure should be easy to inspect, modify, and embed into a larger agent stack. Security code especially benefits from scrutiny. I do not want the trust boundary around my inbox to be a black box.
The Lesson
I started by trying to connect an agent to email.
I ended up separating four concerns that mailbox integrations usually collapse: synchronization, knowledge extraction, security, and action.
That separation changed the system. Providers became replaceable sync endpoints. Email became durable state. Untrusted content became visible risk. Sending became a guarded boundary instead of another tool call.
Access lets an agent read your inbox. Intelligence lets it understand the work inside it without treating every message as trusted instruction.
That is the layer I wanted. So I built it — and now I am making it open source.
Read more technical writing and case-study notes from the archive.
Read More Articles