Most LLM benchmarks are broken not because the questions are bad, but because the industry treats them as proof of capability when they’re actually proof of optimization. A model that scores 90% on MMLU has demonstrated one thing: it’s good at MMLU. That number tells you almost nothing about whether it can handle your task, your users, or your domain.

It gets worse. Labs actively tune against public benchmarks. Training data gets curated to maximize leaderboard positions. The scores go up. The real-world performance stays flat. Benchmarks aren’t failing because they’re poorly designed. They’re failing because they’ve become marketing artifacts.

The fix is straightforward: stop relying on someone else’s eval and build your own. The performance that matters is performance on your task, in your context, for your users. Everything else is noise.

In this article, I walk through the full process from choosing an evaluation source to structuring a dataset, selecting models, and designing a scoring framework. I use the West African Senior School Certificate Examination (WASSCE) as the case study: a high-stakes regional exam taken by over 2 million students every year, many of whom already turn to AI tools for help. It’s a useful example for anyone who wants to evaluate models against real-world knowledge rather than relying solely on public benchmarks.

But the framework applies to any domain. Medical licensing exams. Legal bar questions. Industry-specific certifications. Internal company assessments. If you have a corpus of questions with known correct answers that represent your use case, you can build an eval.

Why Public Benchmarks Are Misleading You

MMLU, GPQA, HumanEval, GSM8K; these have become the standard scorecards for every LLM release. They’re treated as definitive. They’re not.

They’re being overfit. When every model release is tuned to maximize performance on the same benchmarks, the scores stop meaning what you think they mean. Labs don’t just train models, they train models to win leaderboards. High MMLU increasingly tells you how well a lab optimizes for MMLU, not how capable the model is on novel tasks.

They over-represent one slice of the world. AP exams, college-level science, competition math. All drawn from American and European academic traditions. If your users aren’t in that distribution, you’re evaluating against someone else’s reality.

They don’t test what you actually need. If you’re building an education platform for West African students, a legal research tool for Brazilian attorneys, or a diagnostic assistant for rural healthcare workers, the questions that matter aren’t in any public benchmark. Full stop.

Relying on public benchmarks to make product decisions is like hiring someone based on their SAT score. It tells you something. It doesn’t tell you enough. Here’s how to build the eval that does.

Step 1: Choose Your Evaluation Source

The foundation of any custom benchmark is a structured corpus of questions with verified correct answers. The best sources share three properties: they’re high-stakes (so the questions are well-vetted), they have clear right answers (so scoring is unambiguous), and they represent the knowledge domain you actually care about.

Good candidates: standardized exams (national or regional), professional certification tests, internal company assessments, and university-level subject exams.

What to avoid: Do not use scraped question banks without verified answer keys you’ll be measuring noise. Avoid subjective or free-response questions unless you have a reliable grading pipeline (human or model-based). And stay away from question sets smaller than 200 questions per subject. Below that threshold, your results will be too noisy to draw real conclusions.

The WASSCE example: Every year, over 2 million candidates across five West African countries; Ghana, Nigeria, Sierra Leone, The Gambia, and Liberia sit for the West African Senior School Certificate Examination. It covers a standardized curriculum, has published marking schemes, and is the single most consequential exam in the region, gating university admission for an entire generation.

I took this exam at Presbyterian Boys’ Secondary School (Presec Legon) in Accra, Ghana. I know the content, I know the stakes, and I know that nobody in the AI evaluation space is testing models against it. That gap is exactly the opportunity.

WASSCE also offers something most benchmarks can’t: a natural gradient between universal knowledge and regionally situated knowledge. Some subjects test the same physics and algebra you’d find anywhere. Others require knowledge of ECOWAS trade policy, the 1992 Ghanaian constitution, or the ecological zones of the Sahel. That gradient lets you measure not just whether a model is capable, but where its knowledge actually comes from.

Step 2: Select and Structure Your Subjects (The Situated Evaluation Framework)

This is the most important design decision in your benchmark, and most people get it wrong by testing only one type of knowledge.

I call this the Situated Evaluation Framework: every custom benchmark should include both universal subjects (where you’d expect models to perform well regardless of training distribution) and situated subjects (where performance depends on how well the training data represents your specific domain).

The universal subjects are your control group. They establish a baseline: “this model is generally capable.” The situated subjects are your stress test. They answer the question that actually matters: “is this model capable for my users?”

The spread between them is the signal. A model that scores 85% on your universal subjects and 45% on your situated ones doesn’t have a general capability problem. It has a training data problem. That distinction changes everything about how you respond — you don’t need a bigger model, you need better data or fine-tuning.

The WASSCE example — four core subjects:

Core Mathematics (Universal Control)

50 multiple-choice questions covering algebra, geometry, trigonometry, statistics, and number theory. This is the control group; the content is largely universal. If a model can handle GSM8K or MATH, it should perform well here. If it doesn’t, that tells us something about question formatting or curriculum-specific conventions rather than knowledge gaps.

English Language (Cultural Fluency Test)

Comprehension passages, vocabulary in context, grammar, sentence construction, and an oral English section tested via written proxies like stress and intonation patterns. The comprehension passages are often drawn from West African literary or cultural contexts. A model trained predominantly on American or British English corpora might parse the grammar fine but miss the contextual cues. This subject tests cultural fluency, not just linguistic competence.

Integrated Science (Applied Regional Knowledge)

Biology, chemistry, physics, and earth science; but with an applied, West African lens. Questions reference local agricultural practices, tropical diseases, regional ecosystems, and public health issues relevant to the subregion. This is where you start to see whether a model’s “science knowledge” is truly general or just Western-general.

Social Studies (Maximum Regional Specificity)

This is the subject most likely to break models. Social Studies on WASSCE covers governance structures in West Africa, the role of ECOWAS, environmental sustainability in the Ghanaian context, youth and national development, and cultural practices across the subregion. A model might ace AP US Government and have no idea what the Directive Principles of State Policy are in the 1992 Constitution of Ghana.

This is the real test of whether training data represents the world or just part of it.

How to apply this to your domain: Pick 3-5 subjects or topic areas. At least one universal control, at least one situated stress test. If every subject in your benchmark is situated, you have no baseline. If every subject is universal, you’re rebuilding MMLU. You need the gradient.

Step 3: Build the Dataset

This is the most labor-intensive step, and the one that produces the most lasting value. Your goal is a clean, machine-readable, structured dataset that can be run through any eval harness.

The sourcing pipeline:

1. Collect source material — past papers, question banks, or assessment archives in whatever format they exist (PDF, image, print)

2. Extract and digitize — OCR, manual transcription, or a combination

3. Verify against answer keys — cross-check with official marking schemes or expert review

4. Structure into a standard format — JSON is the most common for eval frameworks

A single question entry might look like:

{

  “subject”: “Social Studies”,

  “year”: 2023,

  “question_number”: 15,

  “question”: “The main objective of ECOWAS is to promote”,

  “options”: {

    “A”: “military cooperation among member states”,

    “B”: “economic integration among member states”,

    “C”: “cultural exchange programs in West Africa”,

    “D”: “political unification of West African states”

  },

  “correct_answer”: “B”

}

A note on data availability: If you’re working with educational data from regions outside the US or Europe, structured, machine-readable datasets are rare. Part of the value of projects like this is building that artifact. The dataset itself becomes a contribution.

For well-resourced domains (US medical licensing, bar exams, etc.), you may find existing structured datasets. But even then, building your own gives you control over quality, scope, and format.

Step 4: Design the Prompt Format

Keep it simple. For multiple-choice evaluations, zero-shot prompting with a clear instruction format is the cleanest baseline:

The following is a multiple choice question from [Exam Name] in [Subject].

Question: [question text]

A. [option A]

B. [option B]

C. [option C]

D. [option D]

Answer with just the letter of the correct option.

Zero-shot measures the model’s baseline knowledge without in-context learning crutches. This is what you want for an initial benchmark. Few-shot variants (providing example questions and answers before the test question) are worth exploring in follow-up work to measure how quickly models can adapt.

Why this matters: The prompt format directly affects results. If your prompt is ambiguous, you’re benchmarking prompt sensitivity, not knowledge. Keep it clean, keep it consistent across all subjects and models.

Step 5: Select Your Models

Don’t just run the biggest model available. The goal is to test hypotheses, not collect trophies.

Pick one model from each of four archetypes:

The Dense Baseline — a standard, well-understood model with strong general capability. This is your reference point. (e.g., DeepSeek-V3-0324, Llama-3.3-70B-Instruct, GPT-OSS-120B)

The Reasoning Specialist — a model optimized for chain-of-thought or extended thinking. Does explicit reasoning help on your situated content, or only on math and logic? (e.g., DeepSeek-R1-0528, Kimi-K2-Thinking)

The Efficient Model — a smaller or mixture-of-experts (MoE) architecture. MoE models activate only a subset of their parameters per token, trading total size for speed and cost. This tests whether you can get comparable results at lower compute. (e.g., Nemotron-3-Nano-30B, Gemma-3-12b-it)

The Multilingual/Large-Scale Entry — a model with broader training data or massive scale. Tests whether more data or more parameters close the gap on situated knowledge. (e.g., Qwen3-235B-A22B-Instruct, Nemotron-3-Super-120B)

The WASSCE example: I’m running all nine models listed above across the four core subjects. The specific hypotheses: Does Qwen3’s multilingual pretraining give it an edge on West African Social Studies? Does DeepSeek-R1’s reasoning help on situated content, or only on Maths? Does Gemma at 12B hold up against models 10x its size on universal subjects?

All of these models are available on Crusoe Intelligence Foundry via managed inference — one API, no GPU management. Run the full eval without provisioning a single machine.

Step 6: Score and Analyze

For multiple-choice: accuracy (correct / total) per subject, per model. But raw accuracy is just the starting point. The interesting analysis comes from the comparisons.

Cross-subject analysis: Do models uniformly degrade across your situated subjects, or is the drop concentrated in specific areas? If every model tanks on Social Studies but handles Maths fine, that’s a training data gap. If some models handle the transition better than others, that’s an architecture or training insight.

Question-type analysis: Within a subject, which topics trip models up? Constitutional history vs. arithmetic? Tropical agriculture vs. basic chemistry? This granularity tells you exactly where models need improvement for your use case.

Architecture effects: Do MoE models with fewer active parameters perform differently than dense models of comparable total size? This matters if you’re choosing models for deployment on constrained infrastructure.

Reasoning vs. baseline: Does explicit chain-of-thought (DeepSeek-R1, Kimi-K2-Thinking) actually help on culturally situated questions, or does it only improve performance on math and logic? If reasoning tokens don’t help on Social Studies, that tells us the gap is in knowledge, not in reasoning ability.

Common Mistakes That Will Wreck Your Benchmark

I’ve seen enough eval projects go sideways to know where the landmines are. Avoid these.

Dataset leakage. If your eval questions appeared in the model’s training data, your benchmark is measuring memorization, not capability. This is the single most common failure mode. Use questions from recent exam years, check for contamination, and if in doubt, hold back a subset of questions that you never publish.

Prompt inconsistency. If you change the prompt format between subjects or models, you’re benchmarking prompt sensitivity, not knowledge. Lock the format. Use the exact same template for every question, every model, every run.

Treating accuracy as ground truth. A model scoring 72% on one run and 68% on the next isn’t necessarily different. Run multiple passes, report variance, and don’t draw conclusions from single-digit accuracy differences. If your dataset is small (under 200 questions per subject), confidence intervals matter more than point estimates.

Ignoring the failure mode. Accuracy tells you how often the model is right. It doesn’t tell you how it’s wrong. A model that consistently picks the same wrong answer on constitutional questions has a knowledge gap. A model that picks randomly has a comprehension problem. The fix is different. Look at the errors, not just the score.

Overfitting your own eval. If you keep iterating on prompts until a specific model scores well on your benchmark, congratulations!! you’ve done the same thing you criticized the labs for. Set the format once. Run it. Report what you find.

What I Expect to Find (WASSCE Predictions)

I’ll be upfront about my priors. These are testable hypotheses, not conclusions.

Core Maths: Models will perform reasonably well — 70-85% for the larger ones. The content is universal enough that existing training data should cover it.

English: Mid-range performance. Grammar and vocabulary will be fine. Comprehension passages with West African cultural context will cause noticeable drops.

Integrated Science: Slightly below Maths. Universal science questions will be handled. Locally contextualized ones, tropical agriculture, regional health issues will be weaker.

Social Studies: The biggest gap. I’d be surprised if any open model breaks 60% without specific fine-tuning on West African content.

If these priors hold, the implication is straightforward: open models have a regional knowledge gap, and it’s measurable. That matters for anyone building AI-powered education tools, tutoring systems, or information services for West Africa or any region underrepresented in English-language training data.

Why This Matters

The Situated Evaluation Framework is general. Any domain where public benchmarks don’t cover your use case and that’s most domains benefits from building custom evals. The methodology in this post works for medical licensing in Southeast Asia, legal reasoning in Latin America, or internal compliance assessments at your company.

But the WASSCE case study highlights something that needs to be said directly.

West Africa has one of the youngest populations on Earth. Millions of students prepare for WASSCE every year, and they’re increasingly turning to AI-powered tools for study help. If the open models behind those tools have systematic blind spots on the very content these students need to learn, that’s not a minor gap. That’s a failure mode with consequences measured in denied university admissions and misdirected study time.

MMLU told us models were getting good at Western academic knowledge. That was useful. But it was never the whole picture. The question this benchmark answers is whether progress is truly global or whether we’re building intelligence that only knows half the world.

The rule is simple: if it’s not in your eval, it doesn’t exist to your model.

Build the benchmark. Run the numbers. Make the gaps impossible to ignore.

The WASSCE benchmark dataset and eval pipeline are open source at https://github.com/acheamponge/wassce-benchmark . If you found this useful, share it with someone building AI for underrepresented communities.

About The Author

Emmanuel Acheampong is a Senior Manager at Crusoe AI and previously co-founded and led AI at yshade.ai. He took the WASSCE at Presec Legon in Accra, Ghana.

Subscribe to Emmanuel

Introduction

Agentic AI is a powerful new paradigm that enables a new class of high-value applications. One of the first use cases to show immediate returns is customer support. We’ll explore how to build these systems using customer support agents as a case study, but the same core principles apply broadly across many domains.

Traditionally, customer agents read emails from customers, prioritised them, and followed up with required actions. Agentic AI can now support this initial triaging by summarising the email and preparing a draft response for human review to ensure customers receive rapid feedback. This saves time that would have been spent reading the original email, understanding it, and composing the response.

Furthermore, since there’s a human in the loop, the bot output is not customer-facing and is safe enough for quick deployment without the strict guardrails required for fully autonomous customer-facing applications.

This was the first Agentic AI application we deployed in my company, and it has proven popular among the agents it supports. In this tutorial, we’ll build a skeletal version of this application. We’ll start with a simple bot, add tool-calling capabilities, and finally add human review and feedback features.

LangGraph Overview

In order to build the application, one could code it from the ground up, but that would take a lot of time. Fortunately, there are several frameworks that handle much of the heavy lifting, allowing us to focus on the application development. The one employed in this tutorial is LangGraph, which provides a powerful framework for working with LLMs and building stateful, agentic AI applications.

The fundamental challenge solved by agentic AI frameworks is how to describe the various interactions within AI applications in a way that is both evolvable and scalable. LangGraph tackles this challenge using a graph-based workflow.
The practical application of graphs is ubiquitous — they are used in knowledge representation, social networks, recommendation systems, and more.

Technical interviews on data structures and algorithms (DSA) are rarely complete without questions involving graphs. In the context of agentic AI, graphs enable the execution of flowcharts with memory. They allow for modular implementation of agentic architectures and provide a structured framework for the flow of associated data.

LangGraph leverages graphs to develop an orchestration framework for the development, deployment, and management of stateful agents. It provides functionalities that enable efficient implementation of production-grade, stateful, and multi-agent workflows.

This tutorial will explore the different aspects of LangGraph with increasing levels of complexity, allowing the application to evolve gradually as new functionalities are added. This step-by-step progression will enhance comprehension and demonstrate the application’s growth through different stages of sophistication.

1. Building Munah: The Basic Agent

StateGraph

This is the fundamental structure that underpins stateful agent development within LangGraph. It models stateful agents as a graph of states and the transitions between them. It defines the flow of executions with proceedings depending on output or conditions. The structure enables the ease of developing debuggable stateful agents that can maintain context and memory.

State

The State serves as the working database of the graph. It is a temporary memory that holds all the context or data relevant to a graph execution. As the workflow progresses, the state dynamically updates to include inputs, conversation history, decisions, variables, and other intermediate information. This data is accessible to all nodes within the graph and ensures each component operates with the proper context. In essence, the State enables the graph to perform coherent and context-aware execution of its logic.

Nodes

This is the point at which the logic of the graph is executed. It serves as an action centre, enabling operations such as model invocation, function execution, database querying, and more. Each node receives the current State, modifies it, and passes the updated state to downstream operations.

Edges

Edges serve as the flow paths for information between nodes. They act as conduits through which the State moves across the graph. Edges can also be conditional, meaning the state of the workflow determines which path to follow. For example, if an LLM determines that a tool is needed to fulfill a request, the flow transitions to the ToolNode; otherwise, it proceeds to the next node.

With these components in place, the stage is now set to build our basic component for Munah.

Coding the Basic Workflow

Imports and Declaration

# import libraries
import logging
from dotenv import load_dotenv
from typing import TypedDict, Annotated
from langchain_groq import ChatGroq
from langgraph.graph import StateGraph, START, END
from langgraph.graph.message import add_messages
from langchain_core.messages import HumanMessage

# Setup logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(”customer_complaint_bot”)

# Load environment variables (for Groq API key)
load_dotenv()

# Initialize LLM (using Groq’s Llama 3.1)
llm = ChatGroq(model=”llama-3.1-8b-instant”, temperature=0)

Defining the State

The State is declared with variables we would need to hold the workflow data. The state inherits from TypedDict meaning we expect it to contain the declared keys.

class AgentState(TypedDict):
    “”“
    The state of our customer complaint agent.
    “”“
    # ‘add_messages’ is a special modifier that appends
    # new messages to the list, rather than overwriting it.
    messages: Annotated[list, add_messages]
    
    # The original customer email
    original_email: str
    
    # The final summary
    summary: str

Nodes declaration

The summarise_complaint node receives the email, sends it to an LLM and returns the summarised version.

def summarise_complaint(state: AgentState):
    “”“
    A node that takes the customer input from the state
    and generates a summary.
    “”“

    email = state[”messages”][-1].content
    logger.info(f”Summarising customer email: {email}”)

    prompt = f”“”
    You are an expert customer service agent. Please summarise the following
    customer complaint email concisely. Extract the core issue. 
    Do not add any formatting.
    
    Email:
    {email}
    
    Summary:
    “”“
    
    response = llm.invoke(prompt)
    summary = response.content
    logger.info(”Summary generated successfully.”)
    
    # Return a dictionary to update the state
    return {”original_email”: email, “summary”: summary}

Note: This is an example prompt for instructional purposes. Prompt engineering and evaluation are out of scope for this article, but they are crucial for ensuring the accuracy and relevance of summarisations for your specific product and user base.

Graph definition

The components are now ready for building the bot. It will be done in four main steps. First the graph is declared, then nodes are added to the graph. This is followed by connecting the nodes with edges and finally, the graph is complied.

# Initialize the graph
customer_complaint_bot = StateGraph(AgentState)

# Add the nodes
customer_complaint_bot.add_node(”summarise”, summarise_complaint)

# Define the edges
customer_complaint_bot.add_edge(START, “summarise”)
customer_complaint_bot.add_edge(”summarise”, END)

# Compile the graph
app_stage1 = customer_complaint_bot.compile()

Graph Visualisation

It is instructive to visualise the current state of the graph. We will use pygraphviz with the script below:

from IPython.display import Image, display

# Get the graph object
graph = app_stage1.get_graph()

# Draw it as a PNG (returns raw bytes)
png_bytes = graph.draw_png()

# Display the PNG in the notebook
display(Image(png_bytes))

Running the Bot

# Define a sample complaint
complaint_email = (
    “Hi, I bought the ‘SuperWidget 5000’ (Order #A-12345) last week and “
    “it arrived broken. The screen is cracked. I tried calling support “
    “but was on hold for 45 minutes. This is unacceptable. I want a “
    “full refund and a return label.”
)

# Invoke the graph
# We pass the input as the ‘messages’ list
inputs = {”messages”: [HumanMessage(content=complaint_email)]}
result = app_stage1.invoke(inputs)

print(f”Summary: {result[’summary’]}”)

Final Output

Summary: The customer received a broken ‘SuperWidget 5000’ with a 
cracked screen, was unable to get through to support after waiting 45 minutes,
 and is requesting a full refund and a return label.

Walkthrough

As a summary of the workflow above, let’s take a quick walkthrough of the sequence of events:

1. The graph is invoked with customer input

2. summarise node takes in the State , extracts the input, interpolates it on the prompt, calls the llm with the prompt, gets a response and returns both the llm summary and the customer input

3. The graph captures the summarise node output, merges it to the State and returns the State object

We grab the State object, extract the summary from it and print it out. It is instructive to know that after each node returns, the graph updates the State object with the node output.

The main achievement of this first stage is that we learnt the fundamentals of LangGraph which are State, Nodes and Edges with a simple LLM interaction.

2. Adding Tool Calling and Conditional Flow

Having familiarised ourselves with the fundamentals, let’s move on to more advanced capabilities of the LangGraph framework.

When customer input requires the LLM to fetch additional context in order to generate a more relevant response, it needs to leverage a tool call. Fundamentally, tools are functions or APIs that allow the LLM to perform actions beyond text generation — such as database lookups, calculations, or external API interactions.
We’ll now evolve Munah by equipping her with tool-calling ability.

The main steps to implement this are:

1. Define the logic for the tool in terms of a function or API call.

2. Wrap the defined logic inside a ToolNode, which serves as the execution component in LangGraph.

3. Bind the tools to the LLM so that it is aware of them and can call them when needed.

4. Modify the prompt and workflow to instruct the LLM to use the tools when necessary.

In this tutorial, our imaginary customer will be asking questions about their order. The LLM will read the message, determine that it needs the order details, and call the appropriate tool to interact with the database. After retrieving the order details, it will summarise the email and prepare a draft response.

Both the summary and the response will then be passed to the customer support agent for review before making the final decision.

Let’s dive into the details.

Define the Tool

from langgraph.prebuilt import ToolNode, tools_condition
from langchain.tools import tool
from langchain_core.messages import SystemMessage, ToolMessage

# Define the logics (tool) to get order details
def get_order_details(order_id: str) -> dict:
    “”“Retrieve order details by order ID from mock database.
     ️Args:
        order_id (str): The order ID to look up.
    Returns:
        dict: A dictionary with order details.
    “”“
    logger.info(f”Retrieving details for order ID: {order_id}”)

    # Mock database
    mock_db = {
        “A-12345”: {”product”: “SuperWidget 5000”, “status”: “Delivered”, “price”: “£299”},
        “B-54321”: {”product”: “MegaWidget 2000”, “status”: “Shipped”, “price”: “£149”},
    }

    order = mock_db.get(order_id, {”error”: “Order not found”})
    return order

# Define the tool node
tools = [get_order_details] # More tools can be added as required
tool_node = ToolNode(tools=tools)

# Bind the tools to the LLM - ensuring LLM can call them
llm_with_tools = llm.bind_tools(tools)

# Redefine the summarise_complaint function to include tool usage
def tooled_summarise_complaint(state: AgentState):
    original_email = state.get(”original_email”) or state[”messages”][-1].content
    messages = state.get(”messages”, [])

    # Extract any tool responses so far
    tool_messages = [m.content for m in messages if isinstance(m, ToolMessage)]

    # Build the new LLM input
    llm_input = [
        SystemMessage(content=(
            “You are an expert customer service agent. “
            “Summarise the customer’s complaint and interpret any tool results below. “
            “Use the tool outputs to answer questions accurately. “
            “Always keep the original email context.”
        )),
        HumanMessage(content=original_email),
    ] + tool_messages  # <-- add all tool results here
    
    response = llm_with_tools.invoke(llm_input)
    print(f”LLM Response: {response}”)
    summary = response.content
    logger.info(”Summary generated successfully.”)

    # Return a dictionary to update the state
    return {”messages”: response, “original_email”: original_email, “summary”: summary}

Structure the graph

# build a new graph incorporating the tool node
customer_complaint_bot_with_tools = StateGraph(AgentState)
customer_complaint_bot_with_tools.add_node(”summarise_with_tools”, tooled_summarise_complaint)
customer_complaint_bot_with_tools.add_node(”tools”, tool_node)

# Define the edges with tool condition
customer_complaint_bot_with_tools.add_edge(START, “summarise_with_tools”)
customer_complaint_bot_with_tools.add_conditional_edges(
    “summarise_with_tools”,
    tools_condition,   # built-in condition to check for tool calls
)
customer_complaint_bot_with_tools.add_edge(”tools”, END)

#  Compile the new graph
app_stage2 = customer_complaint_bot_with_tools.compile()

# display the new graph
graph = app_stage2.get_graph()
png_bytes = graph.draw_png()
display(Image(png_bytes))

Run the Bot

# call the new graph with a new input that requires tool use
new_complaint_email = (
    “Hello, I ordered the ‘MegaWidget 2000’ with order ID B-54321 two days ago, “
    “but I haven’t received any shipping updates. Can you check the status of my order? “
    “Also, I want to confirm the price I was charged.”
)
new_inputs = {”messages”: [HumanMessage(content=new_complaint_email)]}
new_result = app_stage2.invoke(new_inputs)
print(f”New Summary with tools: {new_result[’summary’]}”)

Output

New Summary with tools: 

Understanding Why It Fails

The implementation above does not produce any output, and a quick look at the graph visualisation explains why. Figure 3 shows that the summarise_with_tools node either calls the tool or moves directly to the end node. When the tool is called, its output is not passed back to the LLM, which prevents the workflow from generating a final response.

We could print the tool output for inspection, but that would be of little use. The core issue is that the LLM never receives the tool’s result to complete the workflow.

ReAct Pattern

The ReAct (Reason + Act) pattern enables the LLM to reason about its query and then act based on that reasoning. In the previous implementation, when the LLM decided to call a tool, the action was taken but the tool’s output was not passed back to the LLM. As a result, the model could not combine the user query and the tool response to generate a context-aware answer. To fix this, we need to modify the graph by adding an edge from the tool node back to the LLM node, allowing the LLM to continue reasoning with the new information and produce a more relevant response.

#  Add edge from tool node back to summarise node
customer_complaint_bot_with_tools.add_edge(”tools”, “summarise_with_tools”)

#  Compile the new graph
app_stage2 = customer_complaint_bot_with_tools.compile()

# Display the new graph
graph = app_stage2.get_graph()
png_bytes = graph.draw_png()
display(Image(png_bytes))

Let’s now try passing in our customer email and observe the output.

# call the new graph with a new input that requires tool use
new_complaint_email = (
    “Hello, I ordered the ‘MegaWidget 2000’ with order ID B-54321 two days ago, “
    “but I haven’t received any shipping updates. Can you check the status of my order? “
    “Also, I want to confirm the price I was charged.”
)
new_inputs = {”messages”: [HumanMessage(content=new_complaint_email)]}
new_result = app_stage2.invoke(new_inputs)


# Run the graph
new_result = app_stage2.invoke(new_inputs)

# Extract key parts 
original_email = new_result.get(”original_email”)
summary = new_result.get(”summary”)

# Extract the latest AI message (final model output)
final_ai_msg = None
for msg in reversed(new_result.get(”messages”, [])):
    if msg.__class__.__name__ == “AIMessage” and msg.content.strip():
        final_ai_msg = msg.content
        break

# Extract tool results (if any)
tool_outputs = [
    msg.content
    for msg in new_result.get(”messages”, [])
    if msg.__class__.__name__ == “ToolMessage”
]

# Print formatted output
print(”\nOriginal Customer Email:”)
print(f”{original_email}\n”)

if tool_outputs:
    print(”Tool Output(s):”)
    for t in tool_outputs:
        print(f”  → {t}”)
    print()

if final_ai_msg:
    print(”Final AI Response:”)
    print(final_ai_msg)
else:
    print(”No final AI message found in result.”)

The formatted output is

Original Customer Email:
Hello, I ordered the ‘MegaWidget 2000’ with order ID B-54321 two days ago, but I haven’t received any shipping updates. Can you check the status of my order? Also, I want to confirm the price I was charged.

Tool Output(s):
  → {”product”: “MegaWidget 2000”, “status”: “Shipped”, “price”: “£149”}

Final AI Response:
Customer’s concern
The customer ordered the MegaWidget 2000 (Order ID B-54321) two days ago and has not yet received any shipping updates. They would like to know the current status of the shipment and confirm the amount that was charged.

Tool output interpretation
- Product: MegaWidget 2000
- Order status: Shipped
- Price charged: £149

Response
Hi there! I’ve pulled up the details for your order B-54321. Your MegaWidget 2000 has already been shipped, and the amount charged was £149.  

If you’d like a tracking number or more detailed shipping information, just let me know and I’ll forward it to you right away. Otherwise, you should expect delivery shortly.  

Thanks for reaching out, and feel free to ask if there’s anything else I can help with!

3. Human-in-the-Loop (HITL) and Persistence

Currently, Munah can understand customer correspondence and leverage available tools to provide contextual summaries and draft responses. To achieve full automation with human oversight, we can utilize two key LangGraph features: Interrupts and Persistence.

Human-in-the-Loop via Interrupt

The Interrupt feature allows pausing graph execution to enable human oversight, such as reviewing or editing Munah’s draft response. When an interruption occurs, the current graph state is stored and presented to a human for review. After the review, execution resumes from exactly where it stopped, incorporating any feedback provided.

State Persistence

By default, the graph state persists only during execution. LangGraph provides long-term persistence through the MemorySaver class, which allows the graph to preserve workflow context and resume without data loss.

We have added a node that implements the human review process, as shown below:

# Import
from langgraph.checkpoint.memory import MemorySaver

# Initialize the graph
customer_complaint_bot_with_tools = StateGraph(AgentState)

# Add the nodes
customer_complaint_bot_with_tools.add_node(”summarise_with_tools”, tooled_summarise_complaint)
customer_complaint_bot_with_tools.add_node(”tools”, tool_node)
customer_complaint_bot_with_tools.add_node(”human_review”, human_in_the_loop_review)

# Redirect flow:
# After summarising (with tools), send to human review before ending.
# Conditional: if the LLM requests a tool, go to tools; otherwise go to human review
customer_complaint_bot_with_tools.add_edge(START, “summarise_with_tools”)
customer_complaint_bot_with_tools.add_conditional_edges(
    “summarise_with_tools”,
    tools_condition,   # built-in LangGraph function that checks for tool calls
    {
        “tools”: “tools”,           # if tool call detected
        “__end__”: “human_review”,    # if no tool call, go straight to human review
    }
)
customer_complaint_bot_with_tools.add_edge(”tools”, “summarise_with_tools”)
customer_complaint_bot_with_tools.add_edge(”human_review”, END)

# Add a MemorySaver for Persistent State 
memory = MemorySaver()

# Compile the new graph with MemorySaver as checkpointer
app_stage3 = customer_complaint_bot_with_tools.compile(checkpointer=memory)


# Visualize the Graph
graph = app_stage3.get_graph()
png_bytes = graph.draw_png()
display(Image(png_bytes))

Visualisation

Running the workflow

When the workflow is run, it will interrupt the flow and send the LLM output to a human. This can be implemented through a dashboard, web/app UI, API gateway or some other CRM platform. The human can choose to approve the draft response based on the original email and tool call output or they can modify the draft response.

# Run the graph with human-in-the-loop

# Use the same thread_id for continuity
config = {”configurable”: {”thread_id”: thread_id}}
new_inputs = {”messages”: [HumanMessage(content=new_complaint_email)]}

logger.info(”--- Starting workflow... ---”)
# This call will PAUSE at ‘human_review’ and return the current state
interrupted_state = app_stage3.invoke(new_inputs, config=config)

logger.info(”--- WORKFLOW PAUSED FOR HUMAN REVIEW ---”)

# --- Step 2: Show the human (you) what to review ---
# Get the AI’s draft response from the interrupted state
ai_draft = “”
for msg in reversed(interrupted_state.get(”messages”, [])):
    if msg.__class__.__name__ == “AIMessage” and msg.content.strip():
        ai_draft = msg.content
        break

logger.info(f”\n Original Email:\n{interrupted_state[’original_email’]}\n”)
logger.info(f” AI-Generated Draft:\n{ai_draft}\n”)

# --- Step 3: Get REAL feedback from the user via input() ---
logger.info(”---  WAITING FOR YOUR INPUT ---”)
logger.info(”Provide feedback/edits (or just press Enter to approve the draft as-is):”)
feedback_comment = input()

human_feedback = {}
if feedback_comment:
    # If you provided feedback, you must also provide the *new* summary
    print(”\nPlease provide the new, approved summary:”)
    approved_summary = input()
    human_feedback = {
        “approved_summary”: approved_summary,
        “comment”: feedback_comment
    }
else:
    # If you just pressed Enter, approve the original draft
    print(”---  No feedback provided. Approving original draft... ---”)
    human_feedback = {
        “approved_summary”: ai_draft, # Approve the original draft
        “comment”: “Approved as-is.”
    }

print(”--- Resuming workflow... ---”)
# We use invoke() again with the same thread_id to resume after human feedback
final_result = app_stage3.invoke(human_feedback, config=config)

# Display Final Results
logger.info(”\n Workflow Completed with Human Review\n”)
print(”Final Summary:”, final_result.get(”summary”), “\n”)

logger.info(”Conversation Messages:”)
for msg in final_result.get(”messages”, []):
    print(f”- {msg.__class__.__name__}: {msg.content}\n”)

Output

The output below shows that it is straightforward to modify the output from the LLM and this application greatly improve the customer support workflow.

## Final Summary: **Customer’s Complaint (summarised)**

The customer ordered the *MegaWidget 2000* (Order ID **B-54321**) two days ago and has not yet received any shipping updates. They would like to know the current status of the shipment and confirm the amount that was charged.

---

### **Interpretation of the Tool Output**
- **Product:** MegaWidget 2000  
- **Order status:** **Shipped**  
- **Price charged:** **£149**

---

### **Response**

Hi there,

Thank you for reaching out. I’ve pulled up your order details:

| Item | Detail |
|------|---------|
| **Product** | MegaWidget 2000 |
| **Order ID** | **B-54321** |
| **Current status** | **Shipped** |
| **Amount charged** | **£149** |

Your package has already left our warehouse and is on its way to you. You should receive a tracking number and further shipping updates within the next 24 hours. If you’d like me to send the tracking details right away, just let me know.

Please feel free to ask if you have any other questions or need further assistance.

Best regards,  
**[Your Name]**  
Customer Support Team

Conclusion

We have built a complete agentic bot, starting from a simple LLM call, then adding tool-call capability, and eventually including a human-in-the-loop. These are the fundamentals of modern agentic applications, and the agentic customer support AI bot we have built is a highly practical bot, suitable for adoption due to its quick wins and minimal risk of errors.

Gotchas

• The tools_condition function is a routing mechanism. It takes the current state, inspects the last message in messages for a tool-call request, and returns the appropriate tools and _end_ if none are found. It will error if it encounters an unexpected structure. Without a routing map, tools_condition cannot reliably decide which node to route to in cases of noisy or ambiguous input. It is therefore recommended to provide a routing map that explicitly instructs tools_condition on the correct path when no tool call is detected.

• The state object serves as the main database during graph execution. Ensure that any data required by downstream nodes is returned to the state as soon as it is generated.

Next Steps

Phew! That was quite a journey through the basics of LangGraph, and it should get easier from here onward. I’ll keep this brief, but there are many possible extensions: for example, human feedback could be fed back into the LLM to refine its output, or tool calls could be implemented using MCP.

Before concluding, here are a few points often overlooked in tutorial applications: while learning is a key goal, one should also consider the ultimate aim — deriving real value from the application. Achieving this requires putting the application into production, which involves additional considerations such as safety, security, compliance, scalability, evaluation, cost, and more. For now, let’s enjoy exploring the new agentic capabilities provided by LangGraph.

About The Author

Kabiru Akande, PhD is an AI Engineering Manager with over a decade of experience leading the design and deployment of secure, scalable enterprise AI systems, with deep expertise in security, safety, infrastructure, and operations.

He holds a PhD in Signal Processing and Digital Communications from The University of Edinburgh and is an inventor with multiple granted U.S. patents in signal processing and communications. He is also a speaker and presenter on topics including AI engineering, wireless communications and academic career pathways. (full bio)

Subscribe to Kabiru

This is Part 9 of a 10-part series on AI-assisted development workflows. Find the series here

I had six agents running in parallel across six git worktrees — separate working copies of the repo, each on its own branch, sharing the same git history but with their own files on disk. A large migration, broken into independent pieces — each agent with its own worklog, its own branch, its own PR. The editing portion went fine. Diffs were clean. Plans were being followed.

Then they all started running tests at the same time.

193 GB of memory. On a 64 GB machine. The system was swap-thrashing so hard I couldn’t even open Activity Monitor. I had to force-reboot.

Editing is cheap. Execution is the bottleneck. A swarm doesn’t become powerful when more agents start writing. It becomes powerful when coordination, verification, and review scale faster than code generation. Six agents writing code in parallel barely registers on the CPU. Six agents each spinning up Gradle daemons, compiling Kotlin, and running test suites? That’s six independent build systems fighting over the same physical resources.

And memory isn’t even the only wall. Kotlin’s build cache hardcodes absolute file paths. Every worktree I spin up does a full cache invalidation — 40+ minute initial builds on Reddit’s codebase. Another developer on an M2 Pro Mac reported thermal throttling running a single agent loop on a complex Android app. Two instances? Physically impossible.

For web developers with fast builds and no compilation cache, multi-agent localhost might work fine. For compiled languages with heavy build systems, it’s currently impractical. The fix is cloud agents, and that’s where this is all heading.

The Multi-Agent Spectrum

There are three ways to run multiple agents, and which one you reach for depends on what you’re doing and what your system can handle.

Subagents run inside a parent agent’s session, each with its own context window and prompt. The parent only receives the summary, not the full search trail — subagents actually save context by doing deep work in their own windows. Subagents save context, but they can’t coordinate laterally. They only talk to the parent.

I learned the limits of this the hard way. On Sparkpass, a Next.js ticketing app, the product spec had drifted from the codebase. I planned a 7-domain parallel audit — Auth, Events, Payments, Wallets, Dashboard, Attendee, Admin — each subagent with its own slice of the spec and code locations to examine. The plan was clean. The machine couldn’t handle it.

Too many subagents running simultaneously hit local resource limits, and the whole thing ground to a halt. I had to fall back to running the domains serially, one agent at a time. That’s where the NEXT_AGENT_PROMPT pattern came from — each agent wrote a structured handoff of what it found, what was left, and what the next agent should be aware of. The serial version found 23 gaps across the codebase. The parallel plan was better in theory. The serial fallback actually shipped.

Independent sessions are separate editor or terminal windows, each in its own git worktree, with its own agent. No shared context at all. The key is to design the work so agents don’t need to coordinate — choose tasks that are genuinely independent. Separate features, documentation, isolated modules. If two agents touch the same file, you’ve picked the wrong tasks to parallelize. On platforms with fast builds, independent sessions will usually beat serial execution on wall-clock time.

Serial handoffs are what the Sparkpass audit fell back to — and they worked. No compaction loss because nothing carries over in-context; the handoff file is the context. Steve Yegge’s Beads system takes this further with an issue tracker as shared working memory, bd readysurfacing unblocked tasks, and a dependency graph enforcing ordering. Serial makes sense when a task is genuinely unparallelizable or when you’re up against device constraints like the Gradle and memory walls from earlier.

The decision rule is simple: subagents for parallel research, independent sessions for isolated implementation, serial handoffs for dependency-heavy or long-lived work. Most real workflows mix all three.

When do you actually need parallel agents? Sergio Sastre Florez, an Android GDE, pushed back on this: if agents are running long enough to parallelize, the output is probably too large to review carefully. Large parallel outputs hide duplication and unnecessary code.

He’s right. Most tasks don’t need parallelism. A task too simple for parallel execution wastes more time on setup than it saves.

And there’s a hidden cost beyond compute: every agent you spawn makes you a manager. You’re context-switching between worktrees, checking progress, unsticking blocked agents, reviewing diffs across multiple PRs. Engineers are used to a maker schedule, not a manager schedule. Three parallel agents can drain you faster than one focused session if you’re not careful.

When parallel actually helps: large migrations with battle-tested skills, epics with independent tickets, features with parts that don’t depend on each other. And you shouldn’t decide this yourself. Ask the agent: “Can this work be parallelized?” Then verify the proposed split against file overlap, dependency chains, and review burden. It might say no. That’s a valid answer.

Specialists Beat Generalists

Marvin Minsky’s Society of Mind argues that intelligence emerges from many simple agents operating at different abstraction levels. High-level agents coordinate strategy. Mid-level agents execute specialized tasks. Low-level agents handle mechanical operations.

I learned the difference between horizontal and vertical partitioning building Eagle Eye, a screenshot review system. The first version used 3 generalist subagents — each one reviewed a batch of screenshots for everything: contrast, clipping, layout, touch targets. Fifteen things to check per screenshot. Low contrast text on the settings screen shipped because all three generalists checked for contrast and none of them owned it. When everybody’s responsible, nobody’s accountable.

The fix was vertical partitioning. Instead of 3 generalists reviewing batches, I proposed 4 specialists reviewing all screenshots for one concern each:

• Text Contrast Agent — WCAG violations across all screenshots

• UI Clipping Agent — content overflow across all screenshots

• Layout Agent — design system spacing violations

• Touch Target Agent — interactive elements below 48dp

A coordinator aggregates their findings into a unified report. Each specialist has one mandate. Narrower prompts mean lower token costs. And because each agent owns exactly one concern, things are less likely to fall through.

The same pattern showed up in Reddit’s Flag Lifecycle Agent — an autonomous system that discovers stale feature flags in the codebase, validates them against production experiment data, and generates cleanup PRs. Three abstraction layers:

• Orchestration — discovery, validation, complexity scoring, spec generation

• Specialists — Planner, Coder, Reviewer (a 3-agent loop inside a cloud sandbox)

• Platform — sandboxed execution, git operations, PR creation, experiment data APIs

Each layer communicates via structured data. The orchestrator doesn’t write Kotlin. The coder doesn’t know how to query experiment data. The platform doesn’t know what a flag cleanup is. Clear boundaries, clear responsibilities. In Phase 1 testing, this architecture cleaned up 7 flags in a single day at $8.79 total — all 7 PRs passed CI.

The pattern generalizes across stacks. I used the same layered architecture in AutonomyOS, a Next.js/TypeScript agent platform. Events trigger proposals, proposals decompose into missions, missions break into concurrent steps — each step executed independently, results aggregated by a coordinator. Different language, different domain, same shape: coordinator at the top, specialists in the middle, platform at the bottom.

The Conductor Pattern

Think of orchestrating parallel agents like a CPU scheduler. Independent tasks get parallelized. Dependent tasks stay serial. You set the goals and constraints. The AI proposes the schedule.

In practice, this means one main editor window stays on develop and orchestrates. It creates git worktrees, writes worklogs, and delegates. Each execution window opens in its own worktree, follows its assigned worklog, commits, pushes, and opens a PR. The main window never implements. It plans and coordinates.

The migration from the opening used this pattern. Each worktree got a dedicated worklog (TDD plan, verification plan, invariants, a checklist of surprises and notes) and a separate PR — independent, not stacked.

The orchestrator agent is the key upgrade. Instead of manually deciding what to parallelize, the orchestrator looks at all in-progress worktrees and picks up the next task with minimal conflicts. Ask the agent: “Look at the work remaining and the worktrees already running. What should the next agent pick up?” It considers file overlap, module boundaries, and dependency chains. It might tell you everything remaining is serial. That’s fine.

Verification makes this trustworthy. Without closing the loop (TDD, red-green commits, a verification plan), the AI will push slop. Every agent following a worklog has a contract: write the test, watch it fail, write the implementation, watch it pass, commit at each transition. The worklog structure is what separates parallel execution from parallel chaos.

And the worklog survives compaction. When a long-running agent hits context limits and the conversation gets compressed, the worklog on disk is still intact. The agent can re-read it and restore full context. This is what makes parallel agents viable for complex PRs, not just trivial ones.

This is still pre-background-agent territory. You’re manually triggering each agent. You’re reviewing each PR. But you’ve separated orchestration from execution. Skills and worklogs are now executable artifacts.

More agents writing code means more code to review. That review layer is its own multi-agent problem.

Who Reviews the Agents?

As execution parallelizes, review must also become layered and partially automated, or the human becomes the bottleneck again. In Part 2, a second model caught an issue the first one missed entirely. Different models have different blind spots, which is exactly why you want AI handling the mechanical review layer — missing imports, type inconsistencies, test coverage gaps — before a human ever sees the PR.

The principle: review must decompose into fast mechanical checks and slow human judgment. AI handles the first layer. Humans handle the second. Neither works alone at scale.

The specific tools matter less than the pattern. For my personal projects, I use Greptile for confidence scoring (a 2/5 score has consistently flagged real problems) and CodeRabbit for deep static analysis and auto-generated sequence diagrams. But any tool that gates the mechanical layer works.

Greptile’s confidence score flagging a critical issue — 2/5, with a clear explanation of why

CodeRabbit catching test gaps and accessibility issues with suggested fixes

The workflow:

1. Agent writes code and tests

2. Push to branch, open PR

3. AI reviewer scores confidence and flags mechanical issues

4. CI passes (tests, lint, build)

5. Human reviews with mechanical analysis already done — primarily architecture and product decisions

But you’re still in the loop. Still triggering every agent. Still reviewing every PR. And flag cleanup is just one use case. Unused dependency removal, lint upgrades, library migrations, dead code cleanup — every piece of tedious maintenance work that devs do on a regular basis is a candidate for the same pattern: discover, verify, ship.

A self-driving codebase doesn’t begin with autonomy. It begins when work becomes structured enough to delegate safely.

Follow Andrew to catch the release of Part 10 for where this is heading: cloud agents, autonomous maintenance loops, executable worklogs, and the full Redit Flag Lifecycle case study that shows how multi-agent systems can safely discover, verify, and ship real code changes.

About The Author

Andrew Orobator is a Senior Android Engineer @ Reddit, Musician and Public Speaker.

Subscribe to Andrew

Many engineers use databases every day. Few ever look inside one. I wanted to change that for myself, so I built youdaheDB a distributed key-value database engine written from scratch in Rust, using only the standard library. No external crates. No frameworks. Just raw bytes, files, and the Rust compiler yelling at me.

What is youdaheDB?

It’s a working key-value storage engine built on the same architecture that powers databases like RocksDB and LevelDB: the LSM-Tree (Log-Structured Merge Tree).

You can put keys, get keys, delete keys, and scan everything in sorted order — all through an interactive REPL. Kill the process mid-write, restart it, and your data is still there.

$ cargo run

Distributed DB Engine v0.2 (with SSTables)

db> put user:1 youdahe
OK

db> put user:2 alice
OK

db> get user:1
youdahe

Kill it. Restart it.

$ cargo run

[engine] Recovered 2 entries from WAL

db> get user:1
youdahe

Everything survived.

The Architecture

youdaheDB has four core components, each built from scratch:

1. Write-Ahead Log (WAL)

Before your database can do anything else reliably, it needs to be honest about what happened — even if the power goes out.

That’s what a Write-Ahead Log is. It’s a simple append-only log file on disk where every write gets recorded before it gets committed to the db in memory.

Think of it like a receipt printer: before your order is prepared, the kitchen gets the ticket. If something crashes, you replay the tickets from the beginning to reconstruct exactly where you left off.

Even though it hits disk first which is slower than memory, it doesn’t need to do any file seeks since it’s “append-only”. This is what makes it fast from a latency perspective because appending to the end of a file is an O(1) operation like adding to a Linked List.

When the db client sends a write, it can get a response back as soon as the new value is appended to the end of the log without needing to wait for it to be fully processed into the actual database. Evening updating the value for an existing key or deleting a key are operations that get appended to the end of the WAL.

You might be wondering how can a log be used to represent key value pairs? You’d expect something like a Hash Map not a Linked List. And how could adding another entry to the log change a value or delete anything?

The format of the log file is simple: length-prefixed key-value pairs in little-endian binary. The length shows where the entry begins and helps the log reader know where the current record ends so it can parse the next record.

On restart, you replay the WAL from the beginning and rebuild your state in memory, which includes processing delete commands. If a key shows up twice, it just overwrites the value it saw before with the latest value.

2. MemTable

A MemTable (memory table) is the database’s active in-memory view of the latest data. After each write is appended to the WAL for durability, it’s also inserted into the MemTable so reads can see the newest value immediately without going to disk first.

In youdaheDB, the MemTable is backed by Rust’s BTreeMap, which keeps keys sorted automatically. That sorted order matters for two reasons: it makes range scans efficient (searching by a prefix of a key), and it makes flushing to disk simpler because the data is already in the right order for an SSTable (sorted on-disk file format described in the next section).

Updates overwrite older in-memory values for the same key, but deletes are still only marked for deletion rather than removing the key entirely, so older copies on disk do not come back later.

3. SSTables (Sorted String Tables)

RAM is fast, but it’s limited in size. Once the MemTable grows past a certain threshold, youdaheDB flushes it to disk as an SSTable — a Sorted String Table and the MemTable is emptied.

An SSTable is an immutable, sorted file. “Sorted” means the keys are written in alphabetical order, which makes lookups efficient: once you scan past the key you’re looking for, you know it isn’t in this file. “Immutable” means once it’s written, it’s never modified — no partial updates, no corruption risk.

Over time, multiple SSTables accumulate on disk. Reads search them newest-to-oldest, and the first match wins.

4. Tombstones

When you delete a key, you can’t just erase it from the MemTable and call it a day. Older copies of that key may still exist in files on disk. If you silently removed the entry from memory, those older values would “reappear” the next time you went looking on disk.

The solution is a tombstone — a special marker that says “this key was deleted.” This is the missing piece to expalain how deletion is happening as described in the WAL, MemTable and SSTable.

Instead of removing the entry, youdaheDB inserts a None value for that key. When a read finds a tombstone, it knows to stop looking further and return “not found,” even if older copies exist on disk.

Tombstones are eventually cleaned up during compaction (more on that below).

5. The Storage Engine

The coordinator that ties everything together. It owns the WAL, MemTable, and all SSTables.

Write path: WAL (durability) → MemTable (speed) → SSTable flush when full.

Read path: MemTable (newest) → SSTables newest-to-oldest → not found.

On startup, it discovers existing SSTable files on disk, replays the WAL to rebuild the MemTable, and you’re back in business.

The Roadmap

I’m building in public, youdaheDB is still early-stage. This article is part I in a series where I’ll explain from start to end how to build a production ready distributed database.

There’s a list of missing pieces waiting to be tackled:

Compaction is the big one. As SSTables accumulate, reads get expensive — the engine has to search more and more files. Compaction merges multiple SSTables into one: it sorts and combines all the entries, applies tombstones (permanently removing deleted keys), and discards superseded versions of the same key. Without compaction, the engine gets slower over time and wastes disk space on stale and deleted data.

Beyond compaction, what’s missing includes:

• Bloom filters — probabilistic data structures that let the engine quickly rule out SSTables that definitely don’t contain a key, avoiding unnecessary disk reads

• Indexes — structures that enable faster lookups without scanning files linearly

• Checksums — to detect file corruption on disk

• Manifests — a durable record of which SSTable files are current, so the engine doesn’t have to re-discover them on startup

• WAL rotation — periodically replacing the WAL with a fresh file after old entries are safely flushed to SSTables

The Next Phase: Raft Consensus

A single-node database is useful, but what happens when that machine dies?

youdaheDB’s next phase is implementing the Raft consensus protocol to replicate data across multiple nodes. Consensus protocols exist to solve a hard problem: how do you get multiple machines to agree on the state of shared data, even when some of them crash or go offline?

The core idea in Raft: one node is elected leader, and all writes go through it. A write is only considered committed once a quorum — a majority of nodes — confirms they’ve stored it. If the leader dies, the remaining nodes hold an election and choose a new one. Because commit quorums and election quorums always overlap, any newly elected leader is guaranteed to have seen all previously committed data.

Each node maintains its own local persistent log that it uses to track and replay committed entries — similar in purpose to the single-node WAL. The Raft protocol coordinates these logs across the cluster: the leader sends log entries to followers, and once a majority acknowledges them, the entry is committed on all nodes. It’s not a single shared log — it’s a distributed agreement protocol that keeps independent local logs in sync.

Why Rust?

Rust’s ownership model is well-suited for database engineering. The borrow checker enforces memory safety at compile time — it eliminates whole classes of bugs like use-after-free and data races that are common in systems code written in C or C++.

That said, Rust doesn’t fully solve the reader-writer coordination problem on its own. For example: if one thread is serving reads while another is flushing the MemTable to an SSTable, Rust doesn’t automatically figure out how to coordinate that handoff. You still need to choose the right synchronization primitive — a Mutex, an RwLock, a channel — and reason carefully about the access pattern.

What Rust does do is make it much harder to accidentally create undefined behavior while doing that. In Java or Python, you might use locks incorrectly and not find out until a crash in production. In Rust, many of those mistakes won’t compile. It does not automatically prove that your concurrency design is correct though.

No garbage collector also means predictable latency. When you’re flushing a MemTable to disk or replaying a WAL, you don’t want a GC pause in the middle of it.

What I Took Away

Building a database from scratch teaches you things that no amount of SELECT * FROM ever will:

• Why databases use WALs — sequential disk appends are fast; random writes are not

• Why tombstones exist — you can’t just “delete” something when older copies live on disk

• Why LSM-trees flush and compact — memory is fast but finite; disk is slow but abundant, and compaction keeps reads from getting too expensive over time

• Why consensus is hard — getting 3 machines to agree on anything, while handling failures and network partitions, is a genuinely difficult problem

The entire thing is built with zero external dependencies. Just Rust’s standard library, a Cargo.toml, and a lot of binary serialization code.

youdaheDB isn’t production-ready. But it taught me more about how data actually lives and moves through a system than any tutorial ever could.

Check out the work in progress here on Github → youdaheDB

About The Author

Youdahe Asfaw is a Sophomore studying CS @ Gustavus Adolphus College, founder of an edtech platform called Docere, and an ML researcher. He specializes in infrastructure, building replicas of tools like Docker and Kubernetes from scratch.

Subscribe to Youdahe

What if I told you that your singleton implementation choice could make your Java application 871 times slower? That’s not a typo. In a billion-operation benchmark across different singleton patterns, I discovered performance differences so dramatic they’ll make you question everything you know about “optimized” code.

While the singleton pattern seems simple enough to ensure only one instance of a class exists in an application, in high-concurrent Java environments, the implementation details determine whether your application scales or dies under load in a low-latency environment.

By the end of this article, you’ll understand the performance characteristics of different singleton patterns, their real-world impact on high-throughput applications, and the benchmark data that reveals these dramatic differences. Whether you’re building REST clients, connection pools, configuration managers, or metrics and logging systems, the use cases are enormous — and choosing the right singleton pattern could be the difference between an application that scales gracefully and one that collapses under load.

The Traditional Approach: Synchronized Singleton

This is the most common implementation of the Singleton pattern that is usually the go-to approach for many developers because it’s simple, thread-safe, and works correctly in every scenario. Unfortunately, it “works correctly” and “perform well” are two very different things which is why despite seeming bulletproof, it hides a performance time bomb.

public class Logger {
    private static Logger instance;
    private Logger() {
        System.out.println("Logger initialized");
    }
    
    public static synchronized Logger getInstance() {
        if (instance == null) {
            instance = new Logger();
        }
        return instance;
    }
    public void log(String message) {
        System.out.println("[LOG] " + message);
    }
}

Here’s where the performance disaster lurks: at runtime, every single call to getInstance() acquires a synchronized lock, even after the instance has been created. Your logger, connection pool, or configuration manager could be accessed millions of times daily, and each access must wait for the same lock.

In low-traffic applications, this overhead might seem negligible. But as concurrent users increase, the situation deteriorates rapidly. Threads queue up for the lock, creating artificial serialization in what should be parallel operations. Instead of working simultaneously, threads are forced to wait in line, each blocking the others.

The impact? In our billion-operation benchmark, the synchronized singleton took 3.5 seconds for single-threaded access. Add just 10 concurrent threads, and that time jumps to 13 seconds, which is far from the expected instant operation. This performance gap can be the difference between a responsive application and one that users abandon in frustration.

The “Optimized” Solution: Double-Checked Locking

The Double-Checked Locking (DCL) approach is designed to solve the synchronized singleton’s lock contention nightmare. It checks if an instance exists before acquiring any lock, and only synchronize during the rare moment of creation.

public class Logger {
    private static volatile Logger instance;
    
    private Logger() {
        System.out.println("Logger initialized");
    }

    public static Logger getInstance() {
    // First check - no lock needed
        if (instance == null) {          
		        // Lock only when necessary          
            synchronized (Logger.class) {    
		            // Second check - prevent race condition      
                if (instance == null) {            
                    instance = new Logger();
                }
            }
        }
        return instance;
    }
    ...
}

This transforms the performance landscape and ensures that after the instances is created, threads can access getInstance() without any synchronization overhead. The first check happens outside the synchronized block, meaning 99.9% of calls avoid locking entirely.

The performance improvement based on our benchmark is dramatic. DCL reduced single-threaded access time from 4.5seconds to 460milliseconds, a 7.6x improvement. With 10 concurrent threads, the gains are even more striking from 13 seconds to just 128 milliseconds, making DCL over 100x faster than the synchronized approach.

However, DCL requires the volatile keyword to ensure proper memory visibility across threads. Without thevolatile, you risk seeing partially initialized instances, which can be a dangerous bug if omitted. The pattern is intricate enough that experts like Joshua Bloch and Doug Lea have warned against it, considering it an anti-pattern due to its error-prone nature and the memory model issues that plagued Java versions before Java 5.

The Modern Approach: Initialization-on-Demand Holder

The Holder pattern, also known as “Initialization-on-Demand Holder,” addresses the problems of DCL by leveraging Java’s classloader behavior without the need for the volatile keyword. It ensures thread safety, removes synchronization overhead, and provides simplicity and correctness across all Java versions. Since the volatile keyword isn’t used, it relies on the JVM’s guarantees about class loading and static initializers.

public class Logger {
    private Logger() {
        System.out.println("Logger initialized");
    }

    // Static inner class is not loaded until getInstance() is called
    private static class Holder {
        private static final Logger INSTANCE = new Logger();
    }

    public static Logger getInstance() {
        return Holder.INSTANCE;
    }
    ...
}

The performance results are remarkable. In our benchmark, the Holder pattern reduced single-threaded access time to just 4 milliseconds, which is 115x faster than DCL and an astounding 871x faster than the synchronized approach. The JIT compiler optimizes this pattern to near-zero cost, essentially turning singleton access into a simple field read.

Benchmark Deep Dive: Numbers Don’t Lie

To compare the performance impact of the three patterns, I tested different scenarios in a single-threaded environment, scaling from two million to one billion operations. The tests ran on a multi-core system with JVM warmup and multiple measurement runs for accuracy.

Below is the progressive performance scaling showing how the different options perform under different operations.

| Operations | Synchronized | DCL   | Holder | Holder Advantage |
|------------|--------------|-------|--------|------------------|
| 2M         | 11ms         | 6ms   | 4ms    | 2.75x            |
| 400M       | 1,374ms      | 192ms | 3ms    | 458x             |
| 1B         | 3,487ms      | 460ms | 4ms    | 871.75x          |

For small-scale operations, the synchronized approach doesn’t look bad and seems to perform acceptably. As we scale to medium-level operations, the performance degradation gradually becomes noticeable. However, large-scale operations expose a performance gap that increases exponentially with scale.

Real-World Impact

For some high-traffic web applications and enterprise systems highlighted below, these performance differences translate into tangible business consequences:

High-Traffic Web Applications:

• Social media platforms with millions of daily active users could see response times degrade from milliseconds to seconds during peak traffic

• API gateways handling configuration or logging requests would become bottlenecks, which could cause cascade failures across microservices

Enterprise Systems:

• In real-time trading systems where microseconds matter, synchronized singletons would be completely unusable

• Banking applications process millions of transactions daily across different payment channels and would accumulate hours of unnecessary overhead if a poor singleton pattern is chosen for their operations

In production environments processing billions of operations, choosing the wrong singleton pattern can mean the difference between a system that scales gracefully and one that collapses under load. The 871x performance difference isn’t just a benchmark curiosity, it’s the difference between meeting SLA requirements and facing system outages.

When Pattern Choice Matters Most

Not every application needs to worry about singleton performance, but certain scenarios make pattern choice crucial. Consider the following instances:

High-Frequency Access Patterns: Connection pools managing database or HTTP connections, logging systems called thousands of times per second, and configuration managers accessed on every request all require optimal singleton performance.

Concurrent Environments: Real-time systems processing continuous data streams that ingest big data and multi-threaded batch processing applications handling large operations demand efficient singleton access under load.

Performance-Critical Systems: High-throughput applications processing millions of operations and low-latency systems where every millisecond counts are scenarios where pattern choice becomes make-or-break.

The Hidden Performance Trap: During development and testing with low load, singleton performance issues often remain invisible and seem perfectly adequate when handling dozens of requests per minute. However, production environments reveal the truth as the system scales. What works well for 10 concurrent users becomes a nightmare with 1,000, and by the time the problem surfaces, changing the pattern might require significant refactoring across multiple system components.

Practical Implementation Guidelines

Default Recommendation: Use Holder Pattern

For 99% of production singleton implementations, the Holder pattern should be your first choice:

public class Logger {
    private Logger() {
        System.out.println("Logger initialized");
    }

    // Static inner class is not loaded until getInstance() is called
    private static class Holder {
        private static final Logger INSTANCE = new Logger();
    }

    public static Logger getInstance() {
        return Holder.INSTANCE;
    }
   // Business methods...
}

Why Holder Pattern Wins:

• Best performance: 871x faster than synchronized, 115x faster than DCL

• Simplest code: No synchronization complexity or volatile keywords

• Thread-safe by design: Leverages JVM class loading guarantees

• Maintainable: Easy to understand and modify

• Future-proof: Performance characteristics remain excellent as load increases

Migration Strategy

For New Development: Start with the Holder pattern as your default choice. It provides the best combination of performance, simplicity, and maintainability.

For Existing Systems: Begin by identifying high-traffic singleton access points using profiling tools, then measure current performance to establish baseline metrics. Prioritize migration of singletons accessed most frequently, and test thoroughly in staging environments with realistic load. Finally, monitor production performance after migration to validate improvements.

Code Quality Benefits: The Holder pattern delivers fewer bugs since simpler code means fewer opportunities for threading errors. It enables easier testing without synchronization complexity to mock or test around. Future developers benefit from better maintainability as they can understand and modify the code easily. Additionally, you gain performance predictability with consistent behavior across different load levels.

Conclusion

Our billion-operation benchmark revealed a stark truth on how the choice of Singleton pattern can create an 871x performance difference between responsive applications and ones that users abandon. Even though the initial pattern seems like a simple design decision, it becomes a critical architectural choice that determines the scalability of an application.

We discovered that the synchronized singleton, while appearing “safe,” creates performance disasters under load. DCL offers a significant improvement, reducing the overhead by 7.6x, but the Holder pattern seems to be the clear winner, delivering 871x better performance with simpler code.

Performance differences are exponential at scale, with small loads hiding massive performance gaps that only surface in production under real load scenarios. In enterprise environments processing billions of operations, the Holder pattern should be your default singleton implementation because it combines the best performance traits with the simplest and most maintainable code.

You can check out the code for the benchmark here.

Coming Up in Part 2

If you enjoyed reading this article, don’t refactor all your singletons to use the Holder pattern just yet — there’s something crucial you need to know about multi-threading. In Part 2, we will discover why DCL can outperform the Holder pattern in high-concurrency scenarios, and when the highlighted choice might not be optimal after all. The results will challenge everything we’ve learned about singleton performance.

About The Author

Adewale Adeleye is a software engineer with a background in fintech, blockchain, and scalable backend systems. His work spans enterprise software, payment infrastructure, and emerging technologies, and he writes about innovation, business, and the digital systems shaping the future. Outside of work, he loves adventure, travel, and exploring new places and ideas.

Subscribe to Adeleye

I recently worked on a project that required querying data by geographical coordinates, which was my first experience with a production-grade app that needed this feature. I started by trying to use a WHERE clause to query data within specific geographic coordinates (latitude, longitude). This worked but was painfully slow and impractical for a production environment.

So I did some research on ways to make this work efficiently with minimal latency in response. I found a working solution using the PostgreSQL extension PostGIS, and I then decided to dig deeper into the topic to better understand what was actually happening under the hood.

Why Traditional Indexes Fail

While B-trees can index multiple columns at a time (composite indexes), they fundamentally sort data linearly. Because of this, when you need to query data using true multi-dimensional criteria—such as searching within a bounding box using latitude and longitude—B-trees become inadequate and inefficient.

Imagine you have a table with 1 million store locations, each defined by latitude and longitude. You want to quickly find all stores within a 5 km radius of a customer’s location. With standard B-tree indexes, you’d create:

CREATE INDEX idx_latitude ON stores(latitude);

CREATE INDEX idx_longitude ON stores(longitude);

Your initial bounding-box query might look like:

SELECT *

FROM stores

WHERE latitude BETWEEN 40.7 AND 40.8

  AND longitude BETWEEN -74.1 AND -73.9;

However, this approach:

• Requires two separate scans (latitude and longitude).

• Forces an expensive intersection operation in your application code.

• Requires further filtering to verify distances precisely.

• Quickly degrades performance as the dataset grows because B-trees can’t inherently manage multi-dimensional searches efficiently.

Why Geospatial Indexes Exist

Imagine typing “Bus stops near me” into Google and instantly getting the five closest stops around you with no lag or missed results. That’s geospatial indexing at work. Geospatial indexing is a method for efficiently querying location-based data by latitude and longitude (2D) and sometimes by altitude (3D). Many popular applications today use geospatial indexing to deliver quick, efficient results to their users, leveraging latitude and longitude pairs.

Unlike traditional indexes such as B-trees, geospatial indexes are designed to handle multidimensional spatial data. A spatial index typically organizes data in a hierarchical structure composed of:

Leaf Nodes: The leaf nodes contain bounding rectangles and references to the stored geometries.

Interior Nodes: The interior nodes store larger bounding regions that encompass their child nodes.

During a query, the database traverses only the branches whose bounding regions intersect the search area. This allows the system to quickly eliminate large portions of irrelevant data. The most widely used structure for achieving this is the R-Tree.

R-Trees

An R-Tree is a tree-based data structure used to efficiently store and retrieve spatial (location-based) data. It follows the same idea as a B-Tree, but instead of indexing scalar values, it indexes spatial objects such as points, lines, and polygons.

The key idea behind an R-Tree is the use of Minimum Bounding Rectangles (MBRs).

An MBR is the smallest rectangle that fully contains a spatial object. These rectangles allow the index to group nearby spatial objects together, making spatial queries much faster. Most spatial databases, such as PostGIS, implement R-Tree behavior through GiST indexes, which generalize the R-Tree structure for multiple data types. In PostgreSQL, creating one is as simple as:

CREATE INDEX ... USING GIST (location)

and PostGIS handles the rest.

Inserting Data

When geographical data is inserted into a database, the spatial object is stored in the table, and the spatial index then computes the object’s Minimum Bounding Rectangle based on its coordinates (latitude, longitude) or (latitude, longitude, and altitude). This rectangle is inserted into the R-Tree structure.

Each node in the tree stores bounding rectangles that cover all spatial objects contained within that node. As more data is inserted, nearby objects tend to be grouped under the same nodes. If a node exceeds its capacity, it is split into multiple nodes, and the tree structure adjusts accordingly. The node capacity for an R-Tree, like a B-Tree, depends on the page size (generally 8 kilobytes).

Querying Data

During queries, the R-Tree quickly eliminates regions that cannot contain relevant results by checking whether their bounding rectangles intersect the search area. Only candidate geometries that pass this initial filter are then evaluated with precise spatial operations.

A simple latitude/longitude WHERE clause feels like it should be enough until your dataset grows, and queries that took milliseconds start taking seconds. The problem isn’t the query, it’s the index behind it.

Geospatial indexing solves this by organizing spatial data into structures like R-trees, which group nearby objects using Minimum Bounding Rectangles and traverse only the relevant parts of the dataset, skipping entire geographic regions that can’t match and running precise calculations only on a small set of candidates.

Tools like PostGIS bring these capabilities directly to PostgreSQL, enabling developers to build fast, location-aware systems, from mapping services and logistics platforms to ride-sharing and food delivery apps.

About The Author

Wilfred Chetat is a Senior Backend Engineer at UXE Security Solutions in Dubai, where he builds and scales high-throughput distributed systems for real-time video analytics. With over 6 years of Python development experience, he works across the full backend stack, from API design and message queues to databases and container orchestration.

Subscribe to Wilfred

Recommendation engines are incredibly powerful tools that drive the core experiences of social networks, e-commerce, and media platforms. Today, we can create complex recommendation systems much more easily using modern technology, without needing to be machine learning experts.

At a high level, traditional Retrieval-Augmented Generation (RAG) retrieves information based on text similarity, but it struggles with complex, multi-layered constraints. GraphRAG solves this by retrieving structured networks of relationships, giving the LLM precise, deterministic facts to reason over.

To illustrate this, let’s look at real estate as a prime example where recommendations are notoriously challenging and a graph model helps immensely. I’ve had this crazy idea: what if we could utilize GraphRAG and relationships for better recommendations….

Not just “homes similar to what you clicked,” but recommendations that actually understand the web of constraints behind real decisions:

• “Like this neighborhood, but quieter.”

• “Within 25 minutes of work and close to my partner’s job.”

• “Rental-friendly HOA only.”

• “Good schools, but still walkable to coffee + gym.”

• “Don’t show me places with sketchy parking or bad street noise.”

This moves beyond simple similarity, combining relational structure, strict constraints, and transparent explanations. And that’s exactly where GraphRAG starts to look… weirdly reasonable.

This post is a deep dive into:

• What GraphRAG means for recommendation engines,

• Why it’s powerful for recommendations involving complex relationships,

• What a sane architecture looks like, and

• When this is a terrible idea and you should not build it.

The problem with traditional recommendation systems?

Most recommendation systems work well when the goal is straightforward:

• “People who viewed X also viewed Y” (collaborative filtering)

• “This listing is similar to your tastes” (content-based / embeddings)

• “Rank what you’re most likely to click” (learning-to-rank)

But recommendations get messy when you have to weigh dozens of subjective, competing factors. For example in real estate, a user isn’t just selecting a product. They’re making a life impacting decision:

• commute time

• school district

• safety + noise

• lifestyle (walkability, parks, gym, groceries)

• deal-breakers (HOA rules, pets, rental bans)

• social constraints (close to family, near friends)

• financial constraints (budget, taxes, insurance)

• investment constraints (cap rate, STR legality)

If your system only knows “similar listing text,” it will repeatedly violate user intent in subtle ways. And subtle violations kill trust fast.

So what is GraphRAG in recommendation terms?

GraphRAG = Graph retrieval + LLM reasoning

Instead of retrieving “similar items,” we retrieve a relevant subgraph of facts and relationships that explain why something is a good match.

A simple mental model:

• Traditional retrieval: “Give me listings similar to what this user liked.”

• GraphRAG retrieval: “Give me listings similar to what this user liked, and also pull the network of constraints and relationships that make those listings relevant.”

Then the LLM uses that retrieved context to:

• apply nuanced preferences (“quiet street is more important than granite countertops”),

• reason over multi-hop relationships (“this neighborhood is similar to yours because…”),

• generate explanations grounded in facts,

• ask clarifying questions when constraints conflict.

GraphRAG is not “LLM decides everything.” It’s “retrieve structured context deterministically, then let the LLM reason on top.”

Why real estate here (and where else it applies)?

GraphRAG shines when recommendations require multi-hop reasoning and constraint satisfaction. Real estate is full of that, but this approach easily applies to other datasets that can be modeled as a graph, such as:

• Expert matching: weighing required skills, company culture, salary expectations, and remote flexibility.

• Travel itineraries: balancing travel dates, group size, interests, and proximity to lodging/transit.

Example of a recommendation that is NOT just similarity: User says: “Show me 3-bed homes similar to what I saved, but I need rental-friendly HOA rules, and my commute must be under 30 minutes.”

A pure embedding search might nail “similar style” but miss HOA rental restrictions, commute constraints, and neighborhood nuances. A graph, though? This is literally graph-shaped.

What does the graph look like?

Let’s model a real estate recommendation graph with a few core node types:

This graph encodes the real question: “What is this user actually optimizing for, and what rules must never be violated?”

What does the workflow look like?

If you’re building this seriously, you typically do it in two stages. To set up your architecture, you need to rely on specific technology choices, balancing trade-offs between speed, cost, and reasoning power.

Architecture & Technology Stack

To build a solid GraphRAG framework, you need to select tools for both generation and reasoning:

Embedding Model

Use a smaller, fine-tuned embedding model (e.g. text-embedding-3-small).

These models are fast, scalable, and highly cost-effective for generating candidate vectors during the initial retrieval phase.

Vector Database

Store and query these embeddings in a robust vector database (e.g. Pinecone).

You need ultra-low latency vector retrieval to execute the first pass of similarity matching efficiently.

Graph Database & Framework

Utilize a graph database (like Neo4j) paired with a framework (like LlamaIndex or Microsoft’s GraphRAG).

This is essential for efficiently mapping and querying the multi-hop relationships required to pull bounded subgraphs without dumping the entire graph into a prompt.

Reasoning LLM

For the final retrieval and ranking, deploy a powerful LLM (like GPT-5 or Claude 4.5 Sonnet).

The second stage requires heavy cognitive lifting to evaluate complex constraints, detect conflicts, and generate grounded explanations. While these models have higher latency and cost tradeoffs, their reasoning capabilities are mandatory for accurate GraphRAG. You can configure the reasoning effort / thinking budget to achieve desired latency.

Stage 1: Candidate generation (fast + scalable)

Use a hybrid approach to rapidly narrow down the universe of options:

• Hard filters: price, beds/baths, location bounds.

• Vector retrieval: utilizing your fine-tuned embedding model and vector DB for listing “vibe/style similarity”.

• Graph expansion: neighborhoods the user tends to like, HOA constraints, POIs, commute feasibility.

This stage returns maybe 200–500 candidates.

Stage 2: GraphRAG reasoning + rerank (smart + explainable)

For each candidate (or the top 50), retrieve a bounded subgraph: neighborhood stats, HOA rules, commute time estimate, nearby POIs, similarity link to prior saved listings/neighborhoods.

Now your powerful LLM can do something useful:

• rerank based on soft preferences

• produce explanations

• detect conflicts (e.g., “you want STR but this HOA bans STR”)

• ask clarifying questions when needed

You end with top 10–20 recommendations that are constraint-compliant, explainable, and personalized beyond text similarity.

What “bounded subgraph retrieval” looks like?

This is the part people mess up. You do not dump your whole graph into the prompt. You retrieve a compact “fact pack,” like:

Listing A

• $820k, 3bd/2ba, 1,750 sqft

• Neighborhood: Greenlake

• Commute: 28 minutes (typical traffic)

• HOA: rentals allowed, STR not allowed

• Nearby: gym (0.4 mi), grocery (0.3 mi), park (0.2 mi)

• Similar to: 3 saved listings in Ravenna (neighborhood similarity edge)

That’s the “RAG context,” except it’s graph-derived. Now the LLM isn’t guessing; it’s reasoning over facts.

Why GraphRAG can produce better recommendations?

GraphRAG improves recommendations in a few specific ways:

1. It handles multi-hop constraints: “Close to my sister” → sister’s location → commute edges → neighborhood candidates.

2. It makes preferences explicit: If a user repeatedly saves homes in quiet neighborhoods, that preference becomes a graph signal — not a vague embedding.

3. It supports cold-start: New listing? New user? Graph connections still provide signal: neighborhood, HOA, POIs, school district, comps.

4. It enables real explanations: You can finally answer “Why did you recommend this?” …and not with fluffy nonsense.

When is GraphRAG a terrible idea?

GraphRAG is not the default architecture for recommendations. Avoid it when:

1. Similarity already solves your problem: If embeddings + LTR gets you 90% of the value, GraphRAG might be expensive “coolness”.

2. Your graph relationships are weak or noisy: If your edges are mostly heuristics and guesses, you’ll add complexity without real lift.

3. You need ultra-low latency at massive scale: Graph queries + LLM inference = not cheap, not fast.

4. Your entity resolution is messy: Graphs break when the same HOA appears under multiple names, addresses aren’t normalized, or entities aren’t deduped cleanly.

5. You can’t measure lift: GraphRAG feels smart, but you still need constraint satisfaction rates, save rates, lead conversion, and “good match” feedback. If you can’t evaluate, then what are we doing?

A realistic MVP

MVP graph scope - Keep it tight: User ↔ Listing ↔ Neighborhood ↔ HOA ↔ POI

MVP capabilities: hard filters | embedding similarity | HOA rule filtering | neighborhood context + explanation | commute estimation

Use GraphRAG only where it matters: Don’t run GraphRAG on infinite scroll. Run it on high-intent flows like search results refinement, “recommended for you” after saves, or investment mode toggles.

GraphRAG for recommendation engines sounds crazy until you realize that some recommendation problems aren’t primarily similarity problems. They’re relationship + constraint + trust problems.

So if your users care about constraints that must never be violated, multi-hop relationships, grounded explanations, and personalization beyond text similarity… GraphRAG is a serious tool — when used with discipline. Ultimately, this approach extends far beyond real estate; it is a vital framework for unlocking value in any dataset with complex relationships that can be modeled as a graph.

About The Author

Bryan Ofuokwu is an Applied AI Engineer at Salesforce building production AI systems at scale. He explores LLMs, RAG, ML pipelines, and the intersection of software engineering and machine learning.

Subscribe to Bryan

My current interests are many things involving quantum information, as I work on quantum-safe cryptography. For this article, I’d like to present a classical problem that was asked by a trader when I participated in a PhD symposium hosted by a prop trading firm on Wall Street many years ago. It stuck with me because of the elegance of its solution.

Today, finance departments in quantitative research are still fascinated by derivations of this problem.

Last summer (2025), I worked with two researchers, Otene Sunday ThankGod and Jibril Abdullahi, on an efficient quantum implementation as a Womanium & Wiser Quantum Program Scholar. The Goal: “How can I maximize my payout when playing Plinko?”

Plinko is essentially the Galton-Box Problem

It’s time to play “The Price is Right!” But what does this have to do with Quantum Computing?

Let’s say you were chosen as a contestant on “The Price is Right” and have a chance to win a big payout by playing the game “Plinko”. Plinko is a game in which a contestant stands at the top of the stairs looking down at a board with varying levels of cash prizes.

The goal as a contestant is to drop the ball from a starting position to maximize your payout (we want to end up in the slot with the most cash value).

Modelling a Game with Maths

Statisticians have been fascinated with this problem because we can model the problem as a statistical likelihood problem with some degree of uncertainty. Additionally, it’s a great example to visualize the Central Limit Theorem as we find our solution converges to Binomial Distribution. Solutions that demonstrate this often contain some sort of Monte Carlo simulation.

Many papers and blog posts have been written about the Galton-Box or Galton Board problem, where our Galton-Box is our Plinko board and we have a series of pegs laid out with a depth for our box, n. For this post, we can ignore the term and its definition, so every time “Galton-Box” is mentioned, replace it with “Plinko board”. Galton himself is a problematic contributor to the field of statistics, but I digress, because our focus is Winner winner chicken dinner!

So what are the odds that if I stand at the extreme left side of the board, my ball will land at the top cash prize by the time it makes its way to the bottom of the board? We can imagine ourselves standing at the top of the stairs, dropping the ball and calculating for each peg on the board what the probability of the ball going down a certain path might be. As we try each starting position, we note which are more successful than others, but doing this over and over for each iteration is tiring!

Marked Trees and Random Walks

If you noticed that each path resembles a tree, you would be right, as our path is essentially a discrete random walk with a number of steps over a time t, where with some probability the ball drops left or right as it hits a peg.

In traditional computing, when we want to solve a problem that may have many solutions, we might use backtracking to traverse every possible solution. Think about using a classical computer to solve Sudoku or the N-Queens Problem.

How would we solve this Quantumly?

In Quantum Computing, we can use Quantum Walks, a type of Quantum Algorithm (that is the quantum version of a classical random walk) to find paths that are “marked”, meaning that they will lead to some solution (this solution may be wrong or correct, but this is distinct from a partial solution where we are not certain whether one exists after running the algorithm).

We can then use a subroutine of this algorithm, called a Quantum Fourier Transform (QFT), to make our solution “faster” by using some sort of evolution operator. QFTs are the cornerstone of many of our current Quantum algorithms such as Shor’s Algorithm, which was developed in 1994 to find prime factors of an integer.

What do you Mean by Faster?

Since QFTs use superposition (meaning that they can traverse a number of possibilities simultaneously), a concept intrinsic to quantum mechanics, they can provide what we call a “quantum advantage”; a speedup in the land of quantum solutions relative to a classical solution (where a classical solution means a world where we do not have access to a quantum computer).

TLDR

As it turns out, Plinko is a real-world abstraction for the (Quantum) Galton-Box problem. With a quantum solution, we would model each peg as a “quantum peg”, which is composed of a number of quantum gates which we need to set, swap and reset for each round.

However, one of the challenges of implementation for finding a solution that may provide a quantum advantage is that our current quantum computers are noisy and therefore error-prone, meaning that finding a solution as we increase the depth of n becomes challenging.

Perhaps in the future, as quantum computers become fault-tolerant (meaning that they are less error-prone), these problems and their implementations will become more tractable. Bonus: Statistically, the middle slot is the best spot to maximize a payout.

About The Author

Krystal Maughan is a PhD student currently working on Quantum-Safe Cryptography.

Subscribe To Krystal

More reading:

• Binomial Distribution: https://en.wikipedia.org/wiki/Binomial_distribution

• The Central Limit Theorem: https://en.wikipedia.org/wiki/Central_limit_theorem

• “The Skeptical Statistician: Games of the Price is Right: Plinko”, https://theskepticalstatistician.blogspot.com/2012/09/games-of-price-is-right-plinko.html

• Marked Vertices: “Quadratic speedup for finding marked vertices by quantum walks”, https://arxiv.org/pdf/1903.07493

• Monte Carlo Simulation: https://en.wikipedia.org/wiki/Monte_Carlo_method

• N-Queens Problem: https://en.wikipedia.org/wiki/Eight_queens_puzzle

• Sir Francis Galton: https://en.wikipedia.org/wiki/Francis_Galton

• Quantum Walks, Notes by Andrew Childs on Quantum Algorithms: ​​https://www.cs.umd.edu/~amchilds/qa/qa.pdf

• Quantum Galton Box: “Universal Statistical Simulator”, “Universal Statistical Simulator” Carney M., and Varcoe B., 2022, https://arxiv.org/abs/2202.01735

• Quantum Fourier Transform, https://en.wikipedia.org/wiki/Quantum_Fourier_transform

• Quantum Walk: https://en.wikipedia.org/wiki/Quantum_walk

• Random Walks: https://en.wikipedia.org/wiki/Random_walk

• Shor’s Algorithm: https://en.wikipedia.org/wiki/Shor’s_algorithm

• Quantum superposition: https://en.wikipedia.org/wiki/Quantum_superposition

• Time Evolution Operators: https://en.wikipedia.org/wiki/Hamiltonian_(quantum_mechanics)#Schr%C3%B6dinger_equation

Models like Gemini, ChatGPT, and Claude are all built on the same core idea: The Attention Mechanism. This idea enables models to reason over long chats and understand entire codebases. Without this algorithm, we would have likely never entered the large language model (LLM) era.

To better understand the algorithm, we first need to investigate what previous problem it was designed to solve. This requires turning a few pages back and revisiting an older generation of models called recurrent neural networks (RNNs).

RNNs

Recurrent neural networks (which are now archaic on the machine learning timeline) are networks trained on sequential data, like language, where the current element depends on previous elements in the sequence. They allow us to operate over sequences of data. This means that models develop context dynamically as they’re exposed to more sequence positions. Why does the context matter, you ask? Let’s look at an example.

I wrote it in python and ran the script

Without additional context, it’s unclear whether ‘python’ refers to the programming language or the snake. The intended meaning can only be understood by looking at surrounding words.

To explain this more formally (skip this paragraph if you’re allergic to equations), the context (h) is continuously updated as the model sees more tokens (x). At any timestep (t), an RNN has access to the current token (xt) and a context that summarizes all previous tokens (ht-1). This brings us to the core equation for updating a context at any timestep (ht):

where Wx and Wh are learned weight matrices, and b is a bias vector

One visible problem here is the packing of all past information into a single vector (ht). As the sequence grows longer, the model is forced to forget some of this information to make room for new information. Can we somehow keep updating the context without sacrificing earlier information? Yes! And that’s where attention comes in.

Attention

Attention allows a model to focus directly on previous tokens and selectively pull important information instead of compressing all past information into a single fixed-size vector. It lets each token decide which other token it needs to focus on and how much information to pull in. How? By projecting the same token representation into three distinct subspaces: query, key and value vectors.

The query and key vectors dictate how much information is transferred between any two tokens. The query vector asks ‘which token is relevant to me?’ while the key vector represents what each token has to offer. By taking the dot product between these two vectors, we can measure their compatibility score. Here is an example.

The cat sat on a mat

The query for ‘sat’ represents the kind of information the token is looking for. It computes a compatibility score with every other token in the sentence by taking the dot product between its query vector and other token’s key vector. Remember, each token has three corresponding vectors: key, query, and value. A high score between the query vector of ‘sat’ and the key vector of ‘cat’ indicates that ‘sat’ should rely heavily on the token ‘cat’ for its downstream tasks (whether next word prediction or machine translation).

Note: Queries as questions and keys as information holders are purely pedagogical metaphors. These are just vectors learned to increase the predictive power of the model.

The above score represents the relevance of the key to the query and determines how much information from the corresponding token’s value vector is incorporated into the querying token’s final representation. Each token’s representation is the weighted sum of all value vectors. For a given token i, the output representation is

where aij is the attention score between tokens i and j and vj is the value vector for token j

But can’t we just have a value vector? If we did, each token’s context would be all previous tokens regardless of how relevant these tokens are to the current token. In other words, every token would contribute equally, forcing the model to treat all past information as equally important. As the sequence grows longer, important information that appeared earlier in the sequence becomes diluted. Attention scores prevent this by ensuring connection is determined by relevance and not distance from the current token.

Hope you now understand why we need three separate vectors derived from the same representation for the attention mechanism to work.

Self-attention vs multi-head attention

So far, we’ve described self-attention, where each token computes a single attention score with each token it’s looking at. In multi-head attention, the model computes multiple attention heads in parallel. Each head has its own query, key, and value projections. This is equivalent to the current token looking at the same other token from different perspectives, inquiring different information (like semantics or syntax) with each head. The output of all heads is then concatenated to form the final representation.

To explain this better, imagine you’re trying to hire a candidate for a position. You’ll likely evaluate them from multiple angles before you make a decision. This can include technical proficiency, teamwork, past experience, and so on. There’s no single metric that can fully capture a candidate’s fit. This is similar to a token considering multiple ‘criteria’ before deciding how much influence other tokens should have over its representation.

Parallelization advantage

The attention mechanism has another advantage, which is particularly attractive for companies building large models (and NVIDIA): parallelization. Sequential data requires sequential processing. But models using attention can compute attention scores between tokens independently of each other. No token depends on the output of another token in the sequence, making the computation highly parallelizable.

Research questions

In this post, we’ve discussed the attention mechanism in depth through formal mathematical definitions and analogies. This might lead you to think that the attention’s role in a model is thoroughly understood. However, understanding how a mechanism computes does not explain why it behaves that way. Despite our current understanding, there are still open questions in the research community as to how exactly each token allocates attention scores to other tokens. In addition, strong attention scores don’t correspond to higher importance or causality. This highlights that much remains to be investigated about this mechanism and its role in creating highly performant and efficient architectures.

About the Author

Ayda Sultan is a machine learning researcher focused on interpreting the internals of large models. Her work centers around understanding how neural networks work internally. She’s an advocate of careful deployment of AI systems and believes AI models can be fully understood and surgically controlled.

Subscribe to Ayda

Building an emulator made me a better engineer…

I’m a self-taught software engineer. Most of what I know about computers came from reading books and articles, watching YouTube videos, taking online courses, and, most importantly, building software.

That approach gave me momentum. It taught me how to learn quickly and how to get unstuck. But it didn’t always give me satisfying answers to why or how things worked the way they did.

A lot of my learning was reactionary. I learned what I needed when I needed it. If a concept came up at work, I’d dive in just far enough to solve the problem in front of me. That’s often the most practical way to learn, but it also meant there were entire areas of computer science I could avoid as long as things kept working.

Over time, I started to notice that some parts of my understanding felt vague. I could write correct code, but I wasn’t always confident I understood what was happening underneath it. I began to suspect that being more proactive and actively seeking out what I didn’t fully understand, would make me a better engineer, especially in harder situations or when writing more intentional, efficient systems.

So one day I went hunting for a challenging project, and I decided to build a Game Boy / Game Boy Color emulator… in Rust 🦀…

An emulator is hardware or software that enables one computer system to behave like another computer system; it’s a computer inside a computer. I wrote a software emulator.

I wrote about it in this post I made on LinkedIn. The gist of the post was that building an emulator was a challenging and fun project where I learned some of the following topics:

• Endianness

• Bit manipulations

• CPU instruction sets & assembly

• CPU instruction execution pipelines & interrupts

• Disassembling bytes into assembly

• Pixel Processing

• Buses

• RAM and ROM

• I/O handling

• Timing (polling vs scheduling)

What I didn’t fully explain in the LinkedIn post was why building an emulator mattered so much to me. That project forced me to slow down. It made gaps in my understanding of computer architecture impossible to ignore. It gave me a concrete way to push into other computer science topics like operating systems and database.

What Emulation Requires of You

Emulation gave me a way to ask the question What don’t I actually understand yet? by forcing me to model a system’s behavior directly.

When you write a web application, the rules of the machine are already in place. The hardware exists. Code execution is handled for you. You’re working inside a system whose behavior is defined ahead of time. The computer becomes a black box. You don’t care how it works internally, you care about the interface it exposes and whether it behaves as expected.

When you build an emulator, you don’t get that luxury. You have to build the black box itself.

The system you’re emulating has a fixed, real behavior. Your task is to reproduce that behavior accurately. Very quickly, vague ideas about computers turn into concrete questions:

• How does the CPU execute instructions, and in what order?

• Where does state live, and when does it change?

• How does data move between components?

At a basic level, something has to fetch and execute instructions. Something has to hold memory. Something has to represent I/O components (timers, graphics, sound, input). Something has to move data between all of them.

An emulator is a system where every step depends on the last one. It’s a state machine. The next state depends on the current state, and the current state depends on the previous one. There’s no room to reinterpret mistakes as design choices. If one part of your model is wrong, the effects propagate.

Bugs show up in subtle but unforgiving ways. A misunderstood instruction can alter control flow. A small timing error can change observable behavior. A mistake in how memory is modeled can corrupt state long before anything crashes. Often, nothing fails loudly. Things just stop behaving like the machine you’re trying to emulate.

Building an emulator forces me to engage directly with structure, causality, and behavior, the fundamentals that drive both hardware and software systems.

Modeling The Structure

Let’s take a peek at some code and ideas to get a taste of the structure, causality, and behavior I mentioned.

CPU

At the center of a computer, and by extension, an emulator, is the Central Processing Unit (CPU). The CPU has access to the bus and registers. Registers are quick data storage that the CPU uses to know what it wants to do next. The cpu uses a special register called the program counter to retrieve the next instruction; the program counter is like an index into the memory that tells the CPU where it is in the executing program.

The CPU’s main goal is to fetch, decode, and execute instructions continuously and every time it does those actions; it completes a cycle.

struct Cpu {
    bus: Bus,
    registers: [u8; 5],
    program_counter: u8,
}

impl Cpu {
    //...

    fn cycle(&mut self) {
        // Fetch
        let instruction = self.fetch_instruction(self.program_counter);
        // Decode
        let instruction_type = self.decode_instruction(instruction);
        // Execute
        match instruction_type {
            Add => self.add_instruction(instruction),
        }
    }

    //...
}

fn main() {
    //...
    loop {
        // something signaled to interrupt the cpu
        if bus.interrupt_pending() {
            bus.handle_interupt() // clear interrup and unblock CPU
        }

        cpu.cycle()
    }
}

The cpu cycles until the program ends, something halts it, or until an interrupt is triggered. An interrupt is an external signal that tells the CPU something needs attention before continuing.

Memory

Memory is where state lives. It’s where programs live, data lives, and results are stored. The CPU relies on memory to keep track of instructions, data, and previous computations. Every read and write goes through memory. Without it, there is no persistent state for the system to operate on. We treat memory like an array of bytes

Usually when speaking about memory we are talking about, Random Access Memory (RAM). However, there is also Read Only Memory (ROM), usually from external memory devices like CDs or Cartridges.

Bus

The bus is how everything communicates. Instruction fetches, data reads and writes, addresses, and interrupt signals all move across it. It’s not glamorous, but it’s what allows the CPU, memory, and devices to coordinate with each other.

struct Bus {
    ppu: Ppu,
    cartridge: Cartridge,
    interrupts: Interrupts,
}

impl Bus {
    fn read_8(&self, address: u16) -> u8 {
        match address {
            // Other components
            0x8000..=0x9FFF => self.ppu.read_8(address),
            0xC000..=0xFDFF => self.memory.read_8(address),
            // Other components
            0xFFFF => self.interrupts.read_8(address),
            _ => 0xFF,
        }
    }

    fn write_8(&mut self, address: u16, value: u8) {
        match address {
            // Other components
            0x8000..=0x9FFF => self.ppu.write_8(address, value),
            0xC000..=0xFDFF => self.memory.write_8(address, value),
            // Other components
            0xFFFF => self.interrupts.write_8(value),
            _ => {}
        }
    }
}

We can represent the Bus as a match that routes to different components based on the read and write address.

I/O Devices

Devices are everything that isn’t the CPU. Timers, graphics hardware, sound hardware, controllers, and input devices are how time passes, pixels appear, and user actions matter. Devices are also how the outside world signals the CPU through interrupts that something has happened and needs to be handled.

I’m leaving out quite a bit here, intentionally. The goal isn’t completeness. It’s clarity. If you understand these pieces, you have enough of a mental model to start learning about emulators and computer architecture. Everything else builds on this foundation.

What’s to Come

This post is the first in a series about computer architecture concepts, using emulation as the guide. This won’t be a formal tutorial on how to write an emulator, but I will explore some of the topics that go into building one.

Next time we’ll start with the CPU and go into depth on instruction execution, then move to time, then memory, then devices like pixel processors. Finally, we’ll look at how all of these pieces interact to produce the systems we recognize.

You don’t need to understand everything at once. Each article will introduce one piece and we’ll drive it home.

By the end of this series, you won’t magically know everything. But you will have gained trust in your understanding. Debugging should feel less mysterious. Low-level systems should feel less intimidating. And the abstractions you use every day should start to make more sense.

If that sounds useful, let’s keep going! Thanks for reading!

About the Author

Aza Walker is a Software Engineer, fascinated with the systems that make modern computing possible. If he’s not coding he’s listening to music, playing video games, or watching anime. Aza currently works at ShipBob as a Senior Software Engineer, where he focuses on developing scalable microservices.

Subscribe to Aza

Aza's Github