Building an agent on your own data, without regretting it in six months
LangChain will get you a working agent on your data in an afternoon. The question is whether you can still change it in half a year. Here is where we draw the line.
Building an agent over your own data has become genuinely easy. A competent engineer with LangChain, a vector store and an afternoon can demo something that answers questions about your documents and looks impressive in a meeting.
The hard part was never the first demo. It is month six, when the person who built it has moved on, the library has had four releases, nobody can say why a particular answer came out wrong, and the business has started making decisions based on it.
This is the path we use instead. It is slower for the first week and much faster for everything after.
Build it in four steps, in this order
Step 1 — Answer one question with retrieval only.
No agent. No loop. No tools. Take a single real question, retrieve the chunks you think contain the answer, and put them in front of a human. The only thing you are measuring is whether the answer is present in the retrieved context at all.
This is where most projects quietly fail, and it is invisible if you skip straight to an agent — a model that was handed the wrong context will still produce a fluent answer, and it will be wrong in a way that reads as authoritative.
Step 2 — Add exactly one tool.
Still no loop. One function, with a typed signature and a real permission boundary, called deterministically. If the question is "what are this account's open invoices", you want a parameterised function, not a model writing SQL against your warehouse.
Step 3 — Add the loop only when a question needs more than one step.
Plenty of valuable internal use cases never need this. "Summarise what changed in this policy" is retrieval plus a prompt. If you add an agent loop to it, you have added latency, cost and failure modes to buy nothing.
Step 4 — Write the eval set before you add anything else.
Forty real cases with known answers, including the ones where the correct response is to refuse or escalate. Run them on every change. Without this you are not engineering, you are adjusting prompts and hoping.
If you cannot tell whether yesterday's prompt change made the system better, you do not have a system. You have a demo with uptime.
Where LangChain earns its place, and where it costs you
We use LangChain and LangGraph on real engagements. We are also deliberate about how much of the system they own.
| Use the framework for | Write it yourself |
|---|---|
| Graph state, checkpointing, interrupts | The system prompt |
| Streaming and async plumbing | Tool signatures and their contracts |
| Standard model and store interfaces | Retrieval: chunking, filters, reranking |
| Human-in-the-loop resumption | Anything a regulator might ask about |
The reasoning is simple. LangGraph's durable state and interrupt model is genuinely hard to build well, and you should not. But the moment your prompts, your tool definitions and your retrieval logic live inside framework abstractions, three things happen: upgrades break you in ways you did not choose, debugging means reading library source, and the interesting behaviour of your system is hidden in someone else's defaults.
Keep the framework at the edges. Keep the parts that define your system in your own code, in your own repo, under your own tests.
# The shape we like: our contracts, their plumbing.
from langgraph.graph import StateGraph, END
from app.prompts import SYSTEM # ours, in version control, diffable
from app.tools import OPEN_INVOICES # ours, typed, permission-scoped
from app.retrieval import search # ours, chunking and filters explicit
MAX_STEPS = 6 # the loop has a budget, enforced here
def agent(state):
if state["steps"] >= MAX_STEPS:
return {"answer": "Escalating: step budget exhausted.", "done": True}
...
graph = StateGraph(State)
graph.add_node("agent", agent)
graph.add_node("tools", tool_node([OPEN_INVOICES]))
graph.add_conditional_edges("agent", route, {"tools": "tools", "done": END})
graph.set_entry_point("agent")
app = graph.compile(checkpointer=store) # durable state is worth the dependencyNote what is explicit in that sketch: the step ceiling, the prompt source, the tool list. Those are the three things you will need to reason about at 2am, so they do not belong behind an abstraction.
The three failures we see most
Chunking chosen once and never revisited. Someone picks 1,000 characters with 200 overlap on day one because that was in a tutorial, and nobody ever checks it against the actual documents. Tables get split mid-row. Policy clauses lose their headings. Measure retrieval separately from generation, or you will spend weeks tuning a prompt to compensate for a chunking decision.
No metadata filters. Semantic search over everything will cheerfully return last year's deprecated policy alongside the current one. Effective dates, document status and jurisdiction belong in the filter, not in the hope that the embedding noticed.
Tools that are too wide. query_database(sql) is not a tool. It moves your entire query surface inside the model's decision space and makes every audit question unanswerable. Narrow, named, parameterised functions take an extra hour each and remove a whole class of incident.
What "easy" should actually mean
Easy should not mean "a working demo on Friday". It should mean: a new engineer can read the repo and understand what the agent is allowed to do, change the retrieval strategy without touching the loop, and run forty cases to find out whether they improved it.
That version takes about two weeks longer to reach the first demo. It is the only version we have seen still in production a year later.
If you want the short version: build retrieval first, make tools narrow and governed, give the loop a budget, and write the eval set before you write the second prompt.