|
Debugging Agents in Different Environments - Live Workshop (Sponsored)
 |
Your agent returns something odd. Was it the prompt, a tool call that timed out, or a response your code could not parse? Without traces, you are guessing.
In this hands-on workshop, Serge from Sentry instruments three agents with Sentry Agent Tracing: a chatbot in an ecommerce store, a custom Slack agent, and a GitHub Action that reviews PRs. You will see how to catch bad tool calls and unexpected output, plus how to track token spend and performance across every agent you run.
Imagine a scenario where an employee is stranded at an airport after a cancelled flight. Before booking a hotel, they ask the company’s AI assistant a simple question: “Can I expense a hotel if my flight gets cancelled?”
The company has thousands of documents covering travel, expenses, insurance, employee benefits, and regional policies. Somewhere inside them, a paragraph states that accommodation costs caused by involuntary travel disruption are reimbursable, subject to certain conditions.
For an LLM chatbot, finding that piece of information is harder than it appears. The question mentions a “cancelled flight,” while the policy refers to “involuntary travel disruption.” Other documents discuss hotels but apply to different countries. An older policy may be present that contains a reimbursement limit that has since changed.
In such a case, the LLM must find information that matches the question’s meaning, belongs to the correct policy, and remains valid today. Only then can it write a useful answer that solves the user’s problem.
This is known as the retrieval problem behind many LLM applications. In this article, we are going to look at how LLMs can find a needle in a haystack. Here’s what we will cover:
An LLM needs evidence to answer questions
How documents are turned into searchable passages
How LLMs find the meaning behind different words
How close is close enough
Why searching every passage is too expensive
Following connections to the the right neighborhood
How much searching is enough
What happens when the answer changes
The LLM Needs Evidence to Answer Questions
A language model does not automatically know what is inside a company’s private documents. An application must provide that information, either by including documents directly in its input or by retrieving relevant passages when a question arrives.
 |
For a large collection, retrieval offers a way to select a manageable amount of information. An embedding model converts the question into a numerical representation. A search system then uses that representation to find promising passages. The application supplies the original text to the LLM.
The model can then use those passages to explain the policy and cite the exact sources where it got the information from. This pattern is called retrieval-augmented generation, or RAG. A vector database supports the retrieval part by storing and searching numerical representations of content.
 |
This division of responsibilities is really useful. If the application retrieves an outdated policy, even a capable language model will produce an outdated answer. If it retrieves a general hotel-booking rule while missing the cancellation exception, the answer may sound reasonably correct while overlooking the important condition.
Reliable answers therefore depend on what happens before generation. This means we have to make the document collection searchable at the right level of detail.
How Documents are Turned into Searchable Passages
A travel handbook can cover flights, accommodation, meals, approvals, and insurance. However, treating the entire handbook as a single searchable item produces a broad representation that can obscure a specific rule.
Instead, the application divides documents into smaller units called chunks. One chunk might describe hotel reimbursement, while another explains approval requirements. Search can then identify a particular passage instead of merely identifying the handbook that contains it.