Chain-of-Thought: When It Helps

When asking Claude to reason step by step improves accuracy, and when it just adds latency and cost for no benefit.

Prompt Engineering·Lesson 3 of 6·7 min

Chain-of-thought (CoT) prompting asks the model to work through its reasoning step by step before producing a final answer, instead of jumping straight to the answer. The exam tests whether you can identify when this is worth the cost, because it is not free.

The tradeoff

CoT reliably improves accuracy on tasks that require multiple dependent reasoning steps: multi-step arithmetic, logic problems with several constraints, tasks that require weighing multiple pieces of evidence before concluding, or debugging where the cause has to be traced through several layers. Working through the steps explicitly reduces the chance the model skips a step or jumps to a plausible-sounding but wrong conclusion.

The cost is real: reasoning text adds tokens, which adds latency and adds spend. For a simple lookup, a classification into one of a few clear categories, or a direct extraction task, there is no chain of dependent steps to walk through, so CoT adds cost without adding accuracy. Reserve it for genuinely multi-step problems, not routine tasks that just happen to sound complicated.

How to structure it

The most reliable pattern is to ask for reasoning in a clearly delimited block (for example inside <thinking> tags) followed by a clearly delimited final answer (for example inside <answer> tags), so downstream code can strip the reasoning and use only the final answer. Asking for reasoning "somewhere in the response" without structure makes it harder to parse out the actual answer programmatically.

Note also that Claude models with extended thinking are a distinct mechanism from prompted chain-of-thought — extended thinking is a model-level feature you enable via API parameters, while prompted CoT is a prompting technique you can use with any model, including ones without a thinking mode. The exam may ask you to distinguish "ask the model to think step by step in the prompt" from "enable a dedicated reasoning mode" as two different tools for a similar goal.

Exam trap A scenario describes a simple sentiment classification task (positive/negative/neutral) that is running slowly and expensively because every prompt includes "think step by step before answering." The correct fix is to remove the CoT instruction, since single-label classification from clear text is not a multi-step reasoning task and gains nothing from it. A wrong answer choice will suggest keeping CoT but "optimizing" it in some other way — the actual issue is that CoT is the wrong tool for this task at all, not that it's misconfigured.

A finance team asks Claude to determine whether a transaction violates a multi-clause compliance policy, where the policy has five interacting conditions (amount thresholds, counterparty type, jurisdiction, prior flags, and an exception clause). A direct answer without reasoning is frequently wrong because the model conflates two of the conditions or misses the exception clause. Asking for step-by-step reasoning through each clause before stating a verdict raises accuracy substantially, because this is exactly the multi-step, multi-constraint reasoning case CoT is built for, and the added latency is acceptable since this runs asynchronously, not in a live user-facing chat.

Batch workloads and latency tolerance

The latency cost of chain-of-thought matters most in interactive, user-facing contexts where someone is waiting on a response. For large, non-interactive workloads — classifying a backlog of 50,000 support tickets, scoring a dataset overnight — latency per request is far less important than throughput and cost, and this is also the scenario where the async Batches API is the right infrastructure choice regardless of whether CoT is used, since it is built for large non-interactive jobs where waiting for results is acceptable. A scenario that combines "large offline batch job" with "task requires multi-step reasoning" can reasonably use CoT precisely because the usual latency objection doesn't apply.

CoT is not a substitute for a better task decomposition

Sometimes a task that appears to need chain-of-thought reasoning inside a single call is actually better solved by splitting it into separate calls — for example, a call that extracts relevant facts, followed by a call that reasons over just those facts. This is different from prompted CoT, which keeps everything in one call and asks the model to narrate its reasoning inline. Both can raise accuracy on genuinely multi-step problems; which one is preferable depends on whether the intermediate steps benefit from being independently inspectable, validated, or cached, in which case splitting into separate calls is usually the stronger design.

Try it

Take a task you currently run with "think step by step" prepended and check whether it is truly multi-step. If it is a single-step classification or lookup, strip the CoT instruction, run 10 identical inputs both ways, and compare both accuracy and response latency/token count between the two versions. Then take a genuinely multi-step task you run without CoT today, add a structured "reason inside <thinking> tags, then answer inside <answer> tags" instruction, and check whether accuracy improves enough to justify the added latency.