Vectorizer — memory for AI agents

The problem

An agent without memory is a demo. Ask it something in January, ask again in March, and it starts from zero — you re-explain the project every time and call it "new context".

But the obvious fix is worse. Dump everything into one store and two failures show up fast:

1. Cross-talk. Every agent reads every other agent's notes. One team's client details surface in another's answer, and nobody can tell which "memory" an answer came from.
2. Unexplainable answers. A vector search returns similar text, the agent asserts it confidently, and when someone asks why do you think that there is no answer — just a cosine score.

So the real requirement isn't "add a vector database". It's memory with boundaries and provenance: each agent has its own, everything is searchable the way people actually search, and you can trace a conclusion back to the premises it came from.

The approach

Vectorizer is a self-hosted memory server for AI agents, written in Go, with ChromaDB underneath.

Isolation first. Every agent gets its own workspace namespace (its own collection prefix). No cross-talk by default — sharing has to be deliberate, not accidental.

Search the way people look things up. Semantic search alone isn't enough, so retrieval is hybrid: vector similarity (HNSW) plus keyword ranking (BM25 with reciprocal-rank fusion), with temporal filters ("what did we decide last month?") and plain grep for when you know the exact string.

Identity for agents. Workspaces hold peers with peer cards — an agent knows who it is and who else is in the room, so conversations between agents have authors, not anonymous text.

Conclusions you can audit. Beyond retrieval there's a reasoning graph: statements are stored as premises with edges between them, and conclusions can be walked back (breadth-first) to the chain that produced them. When a supervisor agent asks "why", it gets a path, not a shrug. An offline pass re-reads what accumulated, summarises the parts that were actually surprising, and files them — so routine chatter doesn't bloat the memory while signal does.

Served three ways. REST for straightforward reads and writes, gRPC for hot paths, and MCP so agents get memory as native tools. TypeScript and Python SDKs for everything else. One docker compose up brings up the database, the embedding service and the server.

What it is today

• 108 commits and three contributors in the five weeks after it started (github.com/alfirus/vectorizer, 24 Aug → 27 Sep 2026; checked 30 Sep 2026), from nothing to evals.
• Public repository — the code is readable, not a black box.
• In production use: it is the memory layer behind the agent profiles running on my own machine — including the agent drafting this page. Its outputs (search results, reasoning traces) are in this session's toolbelt, which is a slightly strange thing to admit in a case study about memory.
• Measured, not asserted: recall and reasoning benchmarks (LongMemEval-style) ship with it, because a memory system without evals is a feeling, not a system.

Honest caveat: it is deliberately lightweight. It is not a warehouse for a whole company's documents, and it doesn't make a weak model smart — it makes a model's memory scoped, searchable and explainable, which is a different and more useful job.

The stack

Layer	Choice	Why
Server	Go	Single binary, low footprint, runs next to the agents
Vector store	ChromaDB (HNSW)	Fast cosine search, collection-level isolation
Embeddings	Qwen3-Embedding-4B (1536d), nomic fallback, OpenAI-compatible option	Local by default, swappable when you want
Retrieval	Hybrid — vector + BM25 RRF, temporal filters, grep	People search by meaning and by exact phrase
Interfaces	REST · gRPC · MCP	Agents, services and SDKs all get a native door
Deployment	Docker Compose	One command, no cloud dependency
Evaluation	LongMemEval-style recall + reasoning benchmarks	Memory claims need numbers

If your agents keep forgetting

Memory is the first thing that breaks when an agent team moves from toy to daily work — and it's the thing I've already built and run. That's one of the three things I do; the other two are field-ops software and web builds.

If your agents need memory with boundaries, request a quote. Four fields, two minutes, and I reply within one working day.