Prompt Engineering Patterns That Actually Work

How to split system and user roles, structure prompts with XML, choose few-shot vs chain-of-thought, and force reliable structured output.

Prompt Engineering·12 min

Most prompt failures aren't about clever wording. They're about structure: putting the wrong content in the wrong channel, leaving the model to infer boundaries it can't see, or asking it to do something a schema should be doing instead. This piece walks through the patterns that hold up once a prompt goes from a one-off experiment to something running in production against real, messy input.

System prompts vs. user messages

The API gives you two channels: a top-level system parameter and the messages array. They're easy to blur together, but they serve different jobs, and mixing them up is one of the most common sources of drift in multi-turn applications.

The system prompt is where standing instructions live: the assistant's role, tone, output-format defaults, tool-use policy, safety constraints — anything that should hold true for the entire conversation. It's set once and, in practice, persists as long as your client keeps sending it. The user message carries the per-turn task: the specific question, the document to summarize, the code to review right now. A useful test is whether the instruction would still be true on turn 50 of the same conversation. If yes, it belongs in the system prompt. If it's only true for this one exchange, it belongs in the user message.

A pattern worth watching for: a support bot that rebuilds its entire system prompt on every call, stuffing in that turn's ticket subject and the customer's latest message alongside the role and tone instructions. Every call regenerates a slightly different system prompt, the message history grows without a stable prefix, and later turns can drift away from the original tone rules because they're competing with per-turn noise baked into the same channel. The fix is to keep the system prompt fixed — role, tone, policy — and move the ticket subject and message into the user turn where it actually belongs.

This isn't just a cleanliness issue. A system prompt that stays byte-for-byte identical across calls is exactly what prompt caching wants: it's the natural home for long, reusable content like role descriptions and static reference material. Regenerate it slightly differently every turn and you lose the cached prefix, paying to reprocess the same tokens repeatedly. If you need to change standing behavior mid-conversation, don't bury a new rule inside a user turn hoping it sticks — it will compete with the original system prompt and fade over subsequent turns. Update the actual system prompt on the next request instead.

Structuring prompts with XML

Once a prompt mixes several kinds of content — instructions, background context, a reference document, examples, the live question — the model has to infer where one section ends and the next begins. Wrapping each component in tags removes that ambiguity. There's no reserved tag vocabulary; <context>, <background>, and <reference_material> are all fine choices as long as you're consistent and every opening tag has a matching close.

This matters for complexity, not length. A two-sentence prompt with one instruction gets nothing from tags. A prompt with a long reference document, a set of formatting rules, and three worked examples benefits a lot — without tags the model can conflate example text with the live input, or mistake part of the reference material for an instruction. Put the actual task last, wrapped in something like <task>, so the model has an unambiguous signal for where supporting material ends and the live request begins.

<role>
You are a support-ticket triage assistant for a B2B SaaS product.
</role>

<policy>
- Severity is one of: low, medium, high, critical.
- Never invent a severity not backed by the ticket text.
- Summaries are one sentence, no more than 20 words.
</policy>

<reference_material>
{{ escalation_matrix }}
</reference_material>

<task>
Triage the following ticket and return severity, category, and a
one-sentence summary.

Ticket: "Users in the EU region can't complete checkout since 9am UTC,
support queue is backing up fast."
</task>

Nesting is fine and often useful — an <examples> block containing several <example> children, each with its own <input> and <output>, is a reliable structure that scales well as you add more examples.

Few-shot vs. chain-of-thought

These solve different problems, and reaching for the wrong one is a common source of wasted iteration.

Few-shot prompting means including two to five worked input/output pairs that demonstrate the exact pattern you want, instead of describing that pattern in prose. It's the right tool when the desired output has a precise shape that's hard to fully specify in words — a particular JSON key ordering, a specific tone and length, a citation convention. If you have clear, well-written prose instructions and the model still comes back with slightly wrong field names or ordering, the fix usually isn't more detailed instructions — it's two or three examples. Precise patterns are something a model can copy from examples far more reliably than it can infer from a description.

Example selection matters more than example count. Two is the floor for establishing a pattern at all; past five, returns diminish fast and you're mostly spending tokens. Cover the edge cases you actually care about, not the same easy case repeated with different wording — the model generalizes from whatever the examples actually demonstrate, including edge-case handling if you show it, or the absence of it if you don't.

<examples>
  <example>
    <input>Can't log in after resetting my password twice.</input>
    <output>severity: medium; category: auth; summary: Login blocked after password reset.</output>
  </example>
  <example>
    <input>All checkout requests failing across every region since this morning.</input>
    <output>severity: critical; category: payments; summary: Checkout down globally, revenue-impacting.</output>
  </example>
  <example>
    <input>The dashboard export button is slightly misaligned on mobile Safari.</input>
    <output>severity: low; category: ui; summary: Minor mobile layout issue, no functional impact.</output>
  </example>
</examples>

Chain-of-thought (CoT) asks the model to work through reasoning step by step before producing a final answer. It reliably improves accuracy on tasks with multiple dependent reasoning steps — multi-step arithmetic, logic problems with several interacting constraints, debugging where the cause has to be traced through several layers. It does nothing useful for a simple lookup, a clean classification into a handful of categories, or a direct extraction task — there's no chain of dependent steps to walk through, so you pay the latency and token cost for no accuracy gain. If a workload is running slow and expensive because every prompt includes "think step by step" ahead of a single-label classification, removing the CoT instruction is usually the fix, not tuning it further.

The most reliable way to structure CoT is a clearly delimited reasoning block followed by a clearly delimited final answer, so downstream code can strip the reasoning and use only the answer:

Work through the compliance check step by step inside <thinking>
tags, evaluating each of the five policy clauses in order. Then give
your verdict inside <answer> tags as a single word: compliant or
violation.

Note that prompted CoT (asking the model to reason in the prompt text) is a different mechanism from a model's built-in extended-thinking mode, which you enable through API parameters rather than prompt text. They can be used independently. Also, latency tolerance changes the calculus: for large offline batches — scoring a backlog of tickets overnight — per-request latency barely matters, so CoT is easier to justify on multi-step tasks than it would be in a live, user-facing chat. And sometimes the better fix for a task that seems to need in-line reasoning isn't CoT at all but splitting it into separate calls — one that extracts relevant facts, another that reasons over just those facts — particularly when the intermediate output benefits from being independently inspected, validated, or cached.

Guaranteeing structured output

Getting reliable structured output out of a model is a layered problem, and each layer catches a different failure mode.

Prefill is the cheapest layer. You supply the start of the assistant's response yourself as an assistant-role message with partial content, and the model continues from there. Prefilling with a literal { leaves the model no room for a conversational preamble like "Sure, here's the JSON you requested:" before the actual object. It eliminates a specific failure mode — preamble text, markdown fences around JSON — but it's a nudge, not a schema. A prefilled response can still continue into malformed JSON, wrong field names, or the wrong types.

Asking nicely vs. actually constraining. Putting "respond only with valid JSON matching this schema" in the prompt relies entirely on the model choosing to comply, and it can fail under edge-case input. If production code is throwing occasional JSON parse errors despite a "return only valid JSON" instruction, rewording the instruction more forcefully rarely fixes it — the instruction was never a guarantee. Actually constraining output means using a mechanism where the response is generated to conform to a schema: a tool/function-call definition with a strict input schema is the common way to do this, even when you have no intention of "executing" anything — the tool-call mechanism is being used purely to get schema-conformant output.

Validation and retry. Even with prefill and a strict schema, responses can still fail validation on business rules a schema can't express — a discount percent out of range, an end date before a start date, a value that's syntactically valid JSON but semantically wrong. Handle this as a loop: generate, validate in code, and on failure, send a new request that includes the specific validator error and asks for a correction — not a generic "please try again." A retry with no information about what was wrong tends to reproduce the same mistake, because nothing the model has to act on has changed.

response = call_claude(prompt, prefill="{")
result = json.loads(response)

errors = validate(result, schema)  # e.g. discount_percent not in 0..50
if errors:
    retry_prompt = (
        "Your previous response had a validation error: "
        + errors[0]
        + ". Correct the value and return the full corrected object."
    )
    response = call_claude(
        prompt,
        history=[assistant_turn(response), user_turn(retry_prompt)],
        prefill="{",
    )

Append the failed attempt and the specific error as new turns rather than silently discarding it and starting over — the model reasoning over its own prior attempt alongside the correction converges faster than a cold retry. Cap retries at two or three and define a fallback path — log for human review, fall back to a safe default — rather than retrying indefinitely. And keep in mind these layers are complementary, not competing: prefill and schema constraints lower how often you need a retry in the first place; the retry loop exists to catch the tail of failures that get through anyway, especially on unusual inputs no amount of upfront example curation covers.

One more distinction worth keeping straight: a validation/retry loop reacts to an objective, programmatic failure. It's not the same as a second-pass review call that checks qualities a schema can't check at all — tone, reasoning quality, completeness. Both are legitimate, and a single pipeline can use both: validate structurally with a retry loop, then send passing output through a separate review call for the qualitative checks.

Common mistakes in production

A handful of patterns account for most of the "why is this prompt unreliable" problems that show up once a prompt is running against real traffic.

Negative-only instructions. "Don't include a preamble," "don't use markdown," "don't be too long" all leave the model guessing at the alternative, which produces inconsistent output. Pair or replace every prohibition with a concrete positive instruction: not "don't use markdown" but "respond in plain prose sentences, no headers or bullet lists."
Overloaded prompts. Asking one call to summarize, translate, extract structured data, and critique tone all at once tends to produce weaker results on each sub-task than focused, separate calls would. This is different from a genuinely multi-step task where the steps depend on each other — the anti-pattern is specifically bundling unrelated work purely to save a call.
Self-review in one call. Asking a single call to generate an answer and then check its own work in the same turn tends to rubber-stamp the generation, since the same reasoning that produced a flawed answer is what's evaluating it. Use a separate call for review, ideally with a differently framed prompt or an explicitly adversarial framing like "find problems with this output."
Vague success criteria. "Write a good summary" forces the model to guess at length, audience, and what to include or exclude. State them directly.
Unstructured context burying the task. A long block of reference material with no tags separating it from the actual instructions makes it easy for the model to lose track of what's actually being asked — this is the same problem XML structuring solves, just unaddressed.
Treating these as independent. They compound. A prompt with vague success criteria, only negative framing, and several unrelated tasks bundled together is common, and fixing just one of the three rarely moves reliability much — when a prompt is described as "unreliable," it's worth checking it against the full list rather than assuming a single cause.

None of these patterns require a bigger model or a longer prompt to fix. They require putting content in the right channel, making boundaries explicit instead of implicit, and pushing correctness checks into code wherever a schema or a validator can do the job more reliably than a hopeful instruction.