Published on

Prompt and Circumstance

Authors
  • avatar
    Name
    Benjamin Lee
    Twitter

Changing a chatbot's instructions is the easiest edit in software. Knowing whether you have improved anything is the hard part.

Changing a prompt feels like the safest thing a programmer can do. It is a string, not a binary: no compiler to appease, no types to satisfy, no unit tests flashing red. A few words can be altered on a Friday afternoon and shipped before the pub. And therein lies the danger. That same innocuous-looking string governs how a large language model behaves—its tone, its format, the depth of its reasoning, the inventive ways in which it fails—and the consequences stay hidden until real users collide with them.

The only dependable way to learn whether a prompt change helps or hurts is to try it on live traffic, exactly as one would test any other user-facing change. That means an A/B test. As Traceloop, an observability firm, argues in its production guide, nothing else accounts for the messiness of real query distributions, real user behaviour and real downstream effects. Offline hunches are no substitute.

Send in the canary

The workhorse pattern is the canary rollout: deploy the new prompt beside the old one, send a sliver of traffic to the newcomer and measure the two side by side.

import random
from langchain_anthropic import ChatAnthropic
from langchain_core.prompts import ChatPromptTemplate

CONTROL_PROMPT = """You are a helpful assistant. Answer concisely."""
VARIANT_PROMPT = """You are a helpful assistant. Think step by step before answering. Be concise."""

def get_prompt_variant(user_id: str, traffic_split: float = 0.1) -> tuple[str, str]:
    # Deterministic assignment by user_id — same user always gets same variant
    import hashlib
    hash_val = int(hashlib.md5(user_id.encode()).hexdigest(), 16) / (2**128)
    if hash_val < traffic_split:
        return VARIANT_PROMPT, "variant"
    return CONTROL_PROMPT, "control"

def run_with_tracking(user_id: str, user_input: str) -> str:
    prompt_text, variant_name = get_prompt_variant(user_id)
    prompt = ChatPromptTemplate.from_messages([
        ("system", prompt_text),
        ("human", "{input}"),
    ])
    chain = prompt | ChatAnthropic(model="claude-sonnet-4-6")
    response = chain.invoke({"input": user_input})

    # Log for analysis
    log_experiment_event(
        variant=variant_name,
        user_id=user_id,
        input_tokens=response.usage_metadata["input_tokens"],
        output_tokens=response.usage_metadata["output_tokens"],
        response=response.content,
    )
    return response.content

The detail that matters is hashing the user's identifier rather than flipping a coin on every request. Pin each user to one variant and the noise falls, and per-user analysis becomes possible. A user who is bounced between two personalities mid-conversation will tell you nothing except that your product feels broken.

What to measure

Ordinary software experiments chase a single number—click-through rate, say, or conversions. Language models are less obliging, and demand a blend of the automatic, the human and the operational. Galileo, an evaluation outfit, lays out the full menu; the essentials sort into three buckets.

Operational metrics are cheap and automatic:

  • Latency (p50, p95, p99)
  • Cost per request (input + output tokens × model price)
  • Error rate and refusal rate

Quality metrics take rather more effort:

  • Relevance: does the response address the question actually asked?
  • Faithfulness: are the factual claims grounded in the context provided?
  • Format compliance: does the output match the structure you expect?

The awkward one is quality, and the current practical answer is to set a machine to judge the machine. An LLM-as-judge—a separate, more capable model scoring responses against a rubric—scales where human reviewers cannot. The discipline is consistency: freeze the judge model and its scoring prompt for the life of the experiment, or you will be measuring your ruler rather than your results.

from langchain_anthropic import ChatAnthropic

judge = ChatAnthropic(model="claude-opus-4-7")

JUDGE_PROMPT = """Rate the following response on Relevance (1-5) and Conciseness (1-5).

User question: {question}
Response: {response}

Return JSON: {{"relevance": <score>, "conciseness": <score>, "reasoning": "<one sentence>"}}"""

def score_response(question: str, response: str) -> dict:
    result = judge.invoke(JUDGE_PROMPT.format(question=question, response=response))
    import json
    return json.loads(result.content)

The tooling

Rolling your own plumbing is instructive once and tedious thereafter. Langfuse, an open-source observability tool, is built for the job: it versions prompts, splits traffic between variants and gathers the per-variant numbers—latency, cost, evaluation scores—into one dashboard. The integration is a single decorator.

from langfuse.decorators import langfuse_context, observe
from langfuse import Langfuse

langfuse = Langfuse()

@observe()
def run_agent(user_input: str, user_id: str):
    # Langfuse fetches the active prompt variant for this session
    prompt = langfuse.get_prompt("my-agent-prompt", label="production")
    langfuse_context.update_current_trace(
        user_id=user_id,
        tags=[f"prompt-version:{prompt.version}"],
    )
    # ... run model ...

From there the variants can be compared across any metric you have logged, sliced by date range, model version or user cohort.

Do not call it too soon

The commonest mistake is impatience. Model outputs vary wildly, and a sample of 50 queries tells you almost nothing beyond how eager you are to declare victory. A few rules of thumb impose some rigour:

  • Run each variant over at least 500–1,000 requests before drawing conclusions.
  • Use a two-sample t-test or a Mann-Whitney U test for continuous metrics such as latency and scores.
  • Apply a Bonferroni correction when testing several metrics at once, or false positives will oblige you with a "win" that is really just chance.

Where traffic is thin, offline evaluation is the sensible fallback: assemble 200–500 real queries, run both variants against the lot and score the results. It is less rigorous than a live test and far quicker, which on most days is the trade a team will happily make.

When recommendations are the point

Agents that drive recommendation or ranking systems raise the stakes. Research from LFAI & Data shows that generative recommenders can fold a multi-stage ranking pipeline into a single model call—but the prompt engineering is anything but trivial, because you are now juggling diversity, novelty and fairness alongside plain relevance.

For these systems, test against business metrics—click-through rate, conversion, session depth—not merely against quality scores. A judge model can tell you whether a response is relevant; only a real user can tell you whether it is useful. The two are not the same, and confusing them is how good demos become disappointing products.

Knowing when to ship

Call the experiment when four things are true:

  1. The sample is large enough (see above).
  2. The primary quality metric has improved by a margin that clears the noise.
  3. Latency and cost have not quietly regressed.
  4. The gain holds across user segments, rather than riding on a single cohort.

A prompt that wins on quality while adding 300 milliseconds of latency or 40% to the token bill is not a free lunch; it is a deliberate trade, and one worth writing down before it is made. The string was easy to change. Everything after that is the actual work.

Sources:

Discuss on Twitter