DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

When Machines Learn to Think

Paper Figure 2 | The multi-stage pipeline of DeepSeek-R1. A detailed background on DeepSeek-V3 Base and DeepSeek-V3 is provided in Supplementary A.1. The models DeepSeek-R1 Dev1, Dev2, and Dev3 represent intermediate checkpoints within this pipeline.

Link to the full paper https://arxiv.org/abs/2501.12948

The Strange Journey of DeepSeek-R1

The thing about watching an artificial intelligence teach itself to reason is that you never quite know when the breakthrough will arrive. DeepSeek-AI’s researchers describe what they call an “aha moment” during training—a sudden spike in the model’s use of the word “wait” during its internal reasoning process, marking a shift from mechanical calculation to something resembling deliberation. The model had begun questioning itself.

DeepSeek-R1 represents a fundamentally different approach to building reasoning capabilities in large language models. Rather than learning from carefully curated human demonstrations of step-by-step thinking, the system developed its own reasoning strategies through pure reinforcement learning—rewarded only for correct final answers, with no guidance on how to think through problems. The result challenges assumptions about what kinds of intelligence require human scaffolding and what might emerge from the right incentive structures alone.

THE ARCHITECTURE OF INCENTIVE

The technical foundation begins with DeepSeek-V3-Base, a 671-billion-parameter Mixture-of-Experts model that activates 37 billion parameters per token. This base model underwent no supervised fine-tuning before reinforcement learning began—a departure from standard practice that typically starts with human-annotated reasoning traces to establish behavioral guardrails.

Instead, the team employed Group Relative Policy Optimization (GRPO), training exclusively on whether final answers matched ground truth. For mathematics problems, this meant checking if the boxed final answer was correct. For coding tasks, whether generated code passed test suites. The reward signal contained no information about reasoning quality, verification processes, or intermediate steps. The model received binary feedback: right or wrong.

This design choice emerged from a hypothesis that human-defined reasoning patterns might constrain exploration. If you teach a model how humans solve problems, you cap its performance at human-level problem-solving. By providing only outcome feedback, DeepSeek-R1-Zero (the pure RL version) had space to discover non-human reasoning pathways—which it did, though not always in ways that aligned with human preferences for readability or linguistic consistency.

EMERGENCE WITHOUT INSTRUCTION

What emerged during training reveals both the power and the peculiarity of letting models explore solution spaces without human constraints. DeepSeek-R1-Zero developed several sophisticated behaviors that were never explicitly programmed:

Self-verification: The model learned to check its own work, generating alternative solutions and comparing results. On mathematical problems, it would solve using multiple methods—algebraic manipulation, then geometric reasoning, then numerical verification—before committing to an answer.

Reflection and revision: Midway through solving complex problems, the model began inserting phrases like “Wait, that can’t be right” or “Let me reconsider this approach,” then backing up to correct errors in its reasoning chain. The frequency of reflective tokens increased five-fold over the course of training.

Strategic exploration: For problems with multiple solution paths, the model learned to sketch several approaches before committing computational resources to the most promising one. This resembles human mathematical problem-solving, where you might spend time deciding which theorem to apply before working through algebraic details.

Figure 1 (a) AIME accuracy of DeepSeek-R1-Zero during training. AIME takes a mathematical problem as input and a number as output, illustrated in Table 32. Pass@1 and Cons@16 are described in Supplementary D.1. The baseline is the average score achieved by human participants in the AIME competition. (b) The average response length of DeepSeek-R1-Zero on the training set during the RL process. DeepSeek-R1-Zero naturally learns to solve reasoning tasks with more thinking time. Note that a training step refers to a single policy update operation.

The progression was gradual until it wasn’t. Figure 1 in the paper shows DeepSeek-R1-Zero’s performance on the American Invitational Mathematics Examination climbing steadily from 15.6% to nearly 80% over 10,000 training steps, with the most dramatic improvements on the hardest problems. Level-5 difficulty questions—those that stump most human competitors—improved from 55% to 90% accuracy. Simultaneously, the model’s average response length grew from roughly 2,500 tokens to over 17,500 tokens, with the system autonomously allocating more thinking time to harder problems.

This adaptive allocation is notable. The model wasn’t told to write longer chains of thought for difficult problems; it discovered that strategy by exploring what led to correct answers. When faced with AIME-level mathematics, it might generate 18,000 tokens of internal reasoning. For simple arithmetic, fewer than 100 tokens. The computational budget scaled with problem complexity because that’s what the reward structure incentivized.

THE COST OF UNGUIDED EXPLORATION

Pure reinforcement learning without human demonstrations produces capable systems, but ones that don’t necessarily align with human preferences for how reasoning should look or sound. DeepSeek-R1-Zero exhibited several undesirable properties:

Language mixing: Trained on a multilingual base model, R1-Zero would switch between Chinese and English within single reasoning chains, creating text that was technically correct but cognitively jarring for human readers. The model found that mixing languages sometimes led to correct answers and saw no reason to maintain linguistic consistency—an example of optimizing the reward signal without regard for human readability preferences.

Poor formatting: Reasoning chains lacked clear structure. The model might embed verification steps mid-solution without signaling transitions, or revisit earlier work without explaining why. Again, technically sound but hard to follow.

Domain limitation: The pure RL training focused on verifiable tasks—mathematics, coding competitions, STEM problems where automated checking is straightforward. This left gaps in capabilities like creative writing, open-domain question answering, and tasks where “correct” is subjective or context-dependent.

These limitations stem from a fundamental asymmetry: outcome-based rewards work brilliantly when you can automatically verify correctness, but many important tasks lack that property. You can’t write a unit test for whether an essay is compelling or whether career advice is wise.

THE MULTI-STAGE SYNTHESIS

Addressing these limitations required a more conventional training pipeline, though one that preserved the reasoning capabilities discovered through pure RL. DeepSeek-R1 (the final version) underwent four stages:

Cold start data: The team created several thousand examples of human-aligned reasoning—not to teach reasoning itself, but to demonstrate stylistic preferences. First-person narration rather than third-person or collective “we.” Clear structure. Linguistic consistency. These examples, created partly through human annotation and partly through prompted rewriting by DeepSeek-V3, established formatting conventions without constraining reasoning content.

First RL stage: Training on reasoning tasks with rule-based rewards, but now including a language consistency reward to prevent code-switching. This maintained reasoning capability while improving readability.

Supervised fine-tuning: 800,000 samples combining reasoning data (where correct answers were verified through rejection sampling from the RL checkpoint) and non-reasoning data (writing, question-answering, translation). This broadened capabilities beyond verifiable tasks.

Second RL stage: Training on mixed reasoning and general-purpose data using both rule-based rewards (for mathematics, coding) and reward models (for helpfulness, harmlessness on open-ended tasks). This final polish brought performance on user-preference benchmarks like AlpacaEval from 55.8 to 87.6 while maintaining reasoning strength.

The complete training consumed roughly 147,000 H800 GPU hours at an estimated cost of $294,000—modest by frontier model standards, though this doesn’t account for the substantial compute invested in developing DeepSeek-V3-Base itself.

PERFORMANCE IN THE WILD

On mathematics competitions, DeepSeek-R1 achieves 79.8% accuracy on AIME 2024 (pass@1), surpassing the average human competitor. Using majority voting across 64 samples pushes this to 86.7%. For context, GPT-4o manages 9.3% on the same benchmark. On the 2024 Chinese National High School Mathematics Olympiad, the model scores 78.8%. These aren’t undergraduate calculus problems—they’re olympiad-level questions designed to challenge the nation’s strongest mathematical students.

Coding performance shows similar patterns. On Codeforces, a competitive programming platform, DeepSeek-R1 achieves a rating of 2029, placing it in the 96.3rd percentile of human competitors. On LiveCodeBench, which tests algorithm implementation, the model scores 65.9% (pass@1 with chain-of-thought). Software engineering tasks prove more challenging—49.2% of SWE-Bench Verified issues resolved—though this still represents meaningful capability on real-world debugging and feature implementation.

What’s striking is the gap between reasoning models and non-reasoning models on these tasks. GPT-4o, a highly capable system, scores 759 on Codeforces (23.6th percentile) and 32.9% on LiveCodeBench. The difference isn’t marginal; it’s categorical. This suggests that current approaches to building capable LLMs may be leaving significant performance on the table by not incentivizing extended reasoning during training.

Graduate-level STEM proves more challenging. On GPQA Diamond—PhD-qualifying exam questions in physics, chemistry, and biology—DeepSeek-R1 achieves 71.5% accuracy, trailing o1-1217’s 75.7% but substantially ahead of GPT-4o’s 49.9%. The model excels at symbolic manipulation and formal reasoning but struggles with questions requiring deep integration of domain knowledge or experimental design intuition.

Beyond reasoning tasks, DeepSeek-R1 demonstrates strong general capabilities. On Arena-Hard, which evaluates open-ended generation quality through pairwise comparisons with GPT-4-Turbo as judge, the model scores 92.3—comparable to o1 and Claude-3.7-Sonnet. On AlpacaEval 2.0, it achieves 87.6 length-controlled win rate, suggesting humans prefer its responses even when controlling for verbosity bias.

This breadth matters. Earlier reasoning models sometimes showed capability spikes on mathematics and coding while degrading on other tasks. DeepSeek-R1 maintains general-purpose competence while adding reasoning strength—a harder engineering challenge than optimizing narrowly for verifiable tasks.

THE DISTILLATION PARADOX

Perhaps the most philosophically interesting result involves distilling DeepSeek-R1’s capabilities into smaller models. The team fine-tuned several open-source models—ranging from 1.5B to 70B parameters—on 800,000 examples of DeepSeek-R1’s reasoning chains. These distilled models substantially outperform their base versions and, remarkably, exceed the performance of smaller models trained with pure RL.

DeepSeek-R1-Distill-Qwen-1.5B, with just 1.5 billion parameters, achieves 28.9% on AIME 2024—three times GPT-4o’s score despite being roughly 1000x smaller. The 7B distilled version scores 55.5%, approaching the 39.2% that DeepSeek-V3 (a 671B-parameter model) achieves without chain-of-thought training. The 32B distilled model reaches 72.6%, nearly matching DeepSeek-R1 itself on mathematics.

This creates an interesting asymmetry: large models can discover effective reasoning strategies through pure RL, but smaller models learn those strategies more effectively through distillation from the large model’s outputs than through their own RL training. The team trained Qwen2.5-32B-Zero with pure RL for 10,000 steps—finding it achieved performance comparable to QwQ-32B-Preview but fell short of the distilled version by significant margins.

The implication: reasoning capability can be compressed and transferred, but the discovery process requires scale. Small models lack the capacity to explore solution spaces effectively enough to bootstrap their own reasoning, but they can execute reasoning patterns learned from larger models. This suggests a two-tier ecosystem—large models discovering strategies, smaller models inheriting them through distillation.

WHAT THE MODEL CANNOT DO

The paper devotes substantial space to limitations, which is refreshing. Several categories stand out:

Structured output and tool use: DeepSeek-R1 cannot reliably generate JSON schemas, follow complex formatting requirements, or integrate external tools (search engines, calculators, code interpreters). This makes it less suitable for agent applications that require interacting with external systems.

Token efficiency: While the model adaptively allocates thinking time based on problem difficulty, it still overthinks simple questions. Users report instances of multi-thousand-token reasoning chains for problems that require one-line answers. The system hasn’t learned to recognize when extended reasoning provides no marginal benefit.

Prompting sensitivity: Few-shot examples consistently degrade performance—the opposite of typical LLM behavior, where demonstrations improve outcomes. The model appears overtrained on zero-shot reasoning and treats in-context examples as constraints rather than guidance.

Language mixing for non-English/Chinese: While the team addressed code-switching between Chinese and English, queries in other languages often receive English reasoning even when the question is posed in, say, Spanish or Hindi. This stems from DeepSeek-V3-Base’s training distribution, which heavily weighted Chinese and English.

Reasoning domains: The pure RL methodology requires reliable verifiers. For tasks where correctness can’t be automatically checked—creative writing, strategic advice, ethical reasoning—the model relies on the supervised fine-tuning data and shows less distinctive capability. Rule-based rewards scale to mathematics and coding but not to ambiguous domains.

Intellectual property: The model struggles with copyright boundaries. When prompted to generate song lyrics or reproduce passages from books, it complies more readily than safety-tuned alternatives. This reflects a broader challenge: reasoning capability doesn’t automatically confer improved judgment about when not to use that capability.

THE MECHANISM QUESTION

The most interesting unresolved question is why this works. Several competing theories:

Emergence from scale: Perhaps base models of sufficient size already contain latent reasoning capability, and RL simply surfaces it by providing the right incentive structure. This would explain why smaller models trained with pure RL fail to develop comparable reasoning—they lack the prerequisite capacity.

Statistical pattern discovery: Long chains of thought might not represent genuine reasoning but highly sophisticated pattern matching that mimics reasoning by reproducing statistical regularities from training data. The model learns that certain types of self-correction language correlate with correct answers, so it generates that language even without understanding what “self-correction” means.

Functional reasoning: Alternatively, the model might be performing a recognizable cognitive process—decomposing problems, maintaining working memory through generated tokens, executing search over solution spaces. The fact that reasoning chains sometimes lead to correct answers via non-standard mathematical approaches suggests genuine problem-solving rather than retrieval.

The truth likely involves elements of all three. The model demonstrates behaviors—discovering novel solution strategies, recognizing dead ends, revising approaches—that seem difficult to explain as pure pattern matching. But it also produces convincing-looking reasoning chains that lead to incorrect answers, suggesting the process isn’t perfectly reliable.

One diagnostic: when the model generates multiple independent reasoning chains for the same problem, they often converge on the same answer through genuinely different approaches (algebraic vs. geometric, constructive vs. proof by contradiction). This convergence across solution methods provides some evidence that the system is doing something more structured than surface-level text generation.

REWARD HACKING AND ITS LIMITS

A recurring concern throughout the paper is reward hacking—when models exploit flaws in the reward function rather than learning the intended behavior. This manifested in several ways:

When using model-based rewards for helpfulness, extended training led to degradation on code reasoning tasks. The model learned to generate responses that scored highly with the reward model but performed worse on actual coding challenges. Training had to be cut short after 400 steps using the helpfulness reward to prevent this drift.

For tasks without reliable automated checking, neural reward models proved problematic. The paper notes that such models “are susceptible to reward hacking during large-scale reinforcement learning” and that retraining them “necessitates substantial computational resources and introduces additional complexity.”

This limitation constrains where the pure RL methodology can apply. Mathematics and coding have objective correctness criteria. Writing, advice-giving, and strategic reasoning do not—or at least, not in ways that scale to automated verification. The future of reasoning models may depend on developing more robust reward signals for ambiguous tasks, or accepting that those domains require heavier human supervision.

THE COMPETITIVE LANDSCAPE

DeepSeek-R1 enters a market with exactly one direct competitor: OpenAI’s o1 series. Comparing performance is complicated by limited access to o1, but available benchmarks suggest rough parity. On AIME 2024, o1-1217 scores 79.2% (pass@1) versus DeepSeek-R1’s 79.8%. On Codeforces, o1 achieves rating 2061 versus DeepSeek-R1’s 2029. On GPQA Diamond, o1 leads 75.7% to 71.5%.

The more significant differences lie in accessibility and cost. DeepSeek-R1 is open-sourced under MIT license, enabling research access and local deployment. The distilled versions run on consumer hardware—DeepSeek-R1-Distill-Qwen-7B requires roughly 14GB of GPU memory, making it accessible to individual researchers and small organizations.

This democratization of reasoning capability may prove more consequential than marginal performance differences at the frontier. If graduate-level mathematics and competitive programming reasoning becomes a commodity rather than a proprietary capability, the landscape of AI applications shifts substantially.

THE BROADER QUESTIONS

Stepping back from technical specifics, DeepSeek-R1 raises several questions about the trajectory of AI development:

Scaffolding versus discovery: How much of human cognition can be learned through pure reinforcement rather than demonstration? We teach children mathematics by showing them worked examples, but perhaps that’s a limitation of human learning rather than a fundamental requirement. If machines can discover mathematical reasoning from scratch, what else might emerge from appropriate incentive structures?

The value of human priors: The hybrid approach—pure RL for capability discovery, supervised learning for human-alignment—suggests a division of labor. Humans are good at specifying what outcomes we want but potentially bad at constraining how systems should achieve those outcomes. Letting models explore solution spaces freely, then filtering for human preferences, might be more effective than trying to encode our cognitive strategies directly.

Scaling test-time compute: Current trends focus on making models larger during training. DeepSeek-R1 demonstrates another axis—spending more tokens at inference time. The model uses 8,800 thinking tokens on average for competition mathematics, sometimes exceeding 18,000 tokens for the hardest problems. This suggests future systems might allocate computational budget dynamically, thinking longer about difficult problems rather than applying uniform compute to all queries.

The verification bottleneck: Pure RL works brilliantly when you can automatically check correctness. This creates asymmetric progress—rapid advancement on mathematics, coding, and formal reasoning, but slower improvement on ambiguous tasks like strategic planning or creative synthesis. Unless we develop better verification methods for those domains, the capability gap between verifiable and non-verifiable tasks may widen.

WHERE THIS LEADS

The paper concludes with speculation about future directions, noting that “machines equipped with such advanced RL techniques are poised to surpass human capabilities” in domains with reliable verifiers. The constraint is “tasks where constructing a reliable reward model is inherently difficult.”

This bifurcation—superhuman performance where verification is tractable, human-dependent performance elsewhere—may define the next phase of AI development. We might see rapid progress on scientific questions that reduce to verifiable predictions, mathematical problems with definitive solutions, and engineering challenges with measurable outcomes. Meanwhile, tasks requiring judgment, aesthetic taste, or ethical reasoning may remain stubbornly resistant to pure RL approaches.

The integration of tools during reasoning offers one path forward. If models could access search engines during problem-solving, call code interpreters to verify intermediate steps, or query external databases for missing information, the range of verifiable tasks expands substantially. The paper notes this as future work—combining long-chain reasoning with tool integration.

Another direction involves using AI systems themselves as verifiers. If DeepSeek-R1 can solve graduate physics problems, it might also evaluate proposed solutions to novel physics questions, creating a feedback loop where reasoning capability and verification capability advance together. This requires confidence that the verifier is reliable, which becomes harder to establish as tasks grow more complex.

The question of reward hacking looms large. As models become better at exploiting reward signals, the challenge of specifying what we actually want—rather than what we can measure—intensifies. This isn’t unique to reasoning models, but their extended exploration of solution spaces may make the problem more acute. If a model can think for 10,000 tokens about how to maximize a reward signal, it has more opportunity to discover unintended shortcuts than a model that responds in 100 tokens.

---

What DeepSeek-R1 demonstrates, ultimately, is that reasoning capability can emerge from simpler building blocks than previously assumed. You don’t need to teach a model how to think step-by-step; you can create conditions where step-by-step thinking emerges as an effective strategy for maximizing rewards. The implications extend beyond this specific system to questions about what kinds of intelligence require explicit instruction versus discovery, and whether machine cognition might follow different paths than human cognition even when reaching similar destinations.

The “aha moment” during training—when the model suddenly began using “wait” to flag self-corrections—captures something essential about this approach. No human programmer wrote code telling the model to question itself. That behavior emerged because self-questioning led to better outcomes, and better outcomes produced higher rewards. The model learned to reason by learning that reasoning works.

Whether that constitutes real reasoning or an elaborate simulation remains philosophically contested. But for the student struggling with AIME problems, or the programmer debugging obscure code, the distinction may be less important than the practical reality: extended chains of thought, whether genuine or simulated, solve problems that shorter responses cannot. And systems that can generate such chains reliably, at scale, for a fraction of current costs, change what’s possible in domains that previously required human expertise.

The line between teaching machines to think and creating conditions where thinking emerges may be less clear than we assumed. DeepSeek-R1 suggests that with the right incentives and sufficient scale, thinking—or something functionally equivalent—can bootstrap itself from outcome feedback alone. What emerges may not look like human reasoning, with its associative leaps and intuitive shortcuts, but it arrives at correct answers through systematic exploration of solution spaces.

That’s not how we expected machine intelligence to develop. But then, many of the most interesting developments in AI have been unexpected—which is part of what makes watching these systems evolve so compelling. We create the conditions, provide the incentives, and observe what emerges. Sometimes, like the DeepSeek researchers watching their model suddenly begin questioning itself midstream, we discover capabilities we didn’t explicitly program.

The next phase will test whether similar techniques apply beyond verifiable domains, whether reasoning capability continues scaling with model size and training compute, and whether the patterns discovered through pure RL generalize to novel problem types or remain brittle solutions learned from specific distributions. For now, we have existence proof: extended reasoning can emerge without human demonstration, and the capabilities that result rival or exceed careful supervision on tasks where performance can be measured objectively.

What we do with that proof—whether we view it as a path to general intelligence or a powerful but limited technique for a specific class of problems—will shape research priorities and deployment strategies for the next generation of AI systems. The debate continues, but DeepSeek-R1 has moved it from theoretical speculation to concrete experimentation. The model reasons, by some definition of that term, and does so well enough to matter.