teaching_llm_applications

Reasoning models and GRPO

Reasoning Large Language Models (LLMs)

Reasoning Large Language Models (LLMs) learn to think by extending standard text generation into multi-step problem-solving using specialized training methods and test-time computation.

How Reasoning Models Learn

1. Pre-training (Foundational Knowledge)

Models start with standard next-token prediction on massive datasets of text, code, and mathematics. This builds a foundational understanding of:

2. Supervised Fine-Tuning (SFT) & Chain-of-Thought (CoT)

Models can be trained on data containing explicit step-by-step rationales, or chain-of-thought (CoT), that map a complex problem to a correct final answer.

This encourages the model to break a problem down into sequential reasoning steps rather than attempting a single-shot answer.

3. Reinforcement Learning (RL) on Verifiable Rewards

Reasoning models can generate long reasoning trajectories and receive rewards based on the correctness of their outputs.

For tasks such as mathematics, programming, and formal reasoning, the answer can often be checked automatically.

Pure RL: DeepSeek-R1-Zero

DeepSeek-R1-Zero demonstrated that reasoning behaviours can emerge when reinforcement learning is applied directly to a base model using primarily rule-based rewards.

The model does not initially require a large collection of human-written reasoning traces.

Reported emergent behaviours include:

Group Relative Policy Optimization (GRPO)

GRPO evaluates multiple generated reasoning paths for the same problem and compares their rewards relative to one another.

A simplified picture is:

                 Problem
                    │
          ┌─────────┼─────────┐
          ▼         ▼         ▼
       Path A     Path B     Path C
          │         │         │
        Score     Score      Score
          └─────────┼─────────┘
                    ▼
           Relative comparison
                    │
                    ▼
             Policy update

Rather than requiring an independent value model for every trajectory, GRPO can use the relative performance of a group of sampled solutions to guide learning.

4. Process and Outcome Supervision

Reasoning can be supervised at different levels.

Outcome Reward Models (ORMs)

An Outcome Reward Model evaluates the final answer.

Problem → Reasoning → Final Answer → Reward
                                      ↑
                                  Correct?

The main question is:

Did the model arrive at the correct answer?

Process Reward Models (PRMs)

A Process Reward Model evaluates individual reasoning steps.

Problem
   │
   ▼
Step 1 → ✓
   │
   ▼
Step 2 → ✓
   │
   ▼
Step 3 → ✗
   │
   ▼
Final Answer

This allows training to distinguish between a reasoning process that is correct throughout and one that happens to arrive at a correct answer despite flawed intermediate reasoning.

5. Knowledge Distillation

Large reasoning models can generate large datasets containing reasoning trajectories.

These datasets can then be used to train smaller student models.

        Large reasoning model
                 │
                 ▼
       Millions of reasoning
             examples
                 │
                 ▼
          Student model
                 │
                 ▼
       Smaller + cheaper model

The goal is to transfer some of the reasoning capabilities of a large, computationally expensive model into a smaller and more efficient model.

How Reasoning Models Differ During Inference

The major difference is that reasoning models can use additional computation at inference time, rather than doing all of their computation during training.

This is often called test-time compute or test-time scaling.

1. Internal Scratchpads / Thinking Tokens

Instead of immediately producing an answer, a reasoning model can generate a long internal reasoning trajectory.

It can use this additional computation to:

Conceptually:

Standard LLM

Prompt ───────────────► Answer


Reasoning LLM

Prompt
  │
  ▼
Think
  │
  ├──► Try approach A
  │
  ├──► Detect problem
  │
  ├──► Try approach B
  │
  ├──► Verify
  │
  └──► Correct answer

The internal reasoning tokens are sometimes described as a scratchpad.

Important: The internal reasoning process of modern reasoning models should not necessarily be equated with the model’s complete or literal “thought process.” What is exposed to a user may be a summary or selected representation rather than the model’s underlying computation.

2. Test-Time Scaling

A reasoning model can often be given a larger thinking budget at runtime.

For example:

Low effort
   ↓
Small compute budget
   ↓
Fast response


Medium effort
   ↓
More computation
   ↓
More extensive reasoning


High effort
   ↓
Large compute budget
   ↓
More extensive search / verification

This is particularly useful for problems involving:

The central idea is:

Instead of making the model substantially larger, give it more computation when solving a difficult problem.

The Overall Picture

Reasoning LLMs can therefore be viewed as combining several components:

                  PRE-TRAINING
                      │
                      ▼
             Foundational model
                      │
                      ▼
            Reasoning-oriented
              fine-tuning
                      │
          ┌───────────┴───────────┐
          ▼                       ▼
        SFT                      RL
     CoT examples        Verifiable rewards
          │                       │
          └───────────┬───────────┘
                      ▼
             Reasoning model
                      │
                      ▼
             TEST-TIME COMPUTE
                      │
          ┌───────────┼───────────┐
          ▼           ▼           ▼
       Search      Verify     Backtrack
          │           │           │
          └───────────┼───────────┘
                      ▼
                  Answer

The key conceptual shift is from:

“Predict the next token.”

towards:

“Use computation to search for a good solution before producing the answer.”

This does not mean that the underlying architecture necessarily changes dramatically. Much of the capability can arise from training the model to use its existing generative machinery as a reasoning process and allocating additional computation at inference time.

Notes

image

Reasoning based benchmarks

image

GRPO

image

How GRPO Works: A prompt is given. The student LLM generates multiple answers. A fixed teacher and evaluator score these answers, and a KL penalty is applied to keep the student close to the teacher. The use scores are then normalized into relative advantage scores that identify which answers were relatively good and bad. This advantage signal is used to update the student model, completing the training loop.

image

Chain of Thought (CoT)

wait, wait, wait, I see ….

image

models might think in some other space, which may not be human-like reasoning or may not be language-space !

PPO

image

image

image

image

Note that this may be leading to output length becoming bigger and bigger (see Increasing output length section below)

Concept 🧩 🚀 PPO is used to tune a model and align it to human preferences

Concept 🧩 🚀 GRPO is used to train a model to reason

Concept 🧩 🚀 GRPO does not have a value function

How do you open the black box and explain reasoning

image of Anthropic LLM Microscope

Increasing output length

image

Other adjustments

Recap

image

DeepSeek-R1 model

image

image

Talk on reasoning models

reasoning behaviour emerges naturally such as self-verification, backtracking.

Distillation

References

  1. IBM — Reasoning Models
  2. Sebastian Raschka — Understanding Reasoning LLMs
  3. Prompt Engineering Guide — Reasoning LLMs
  4. Cameron R. Wolfe — Demystifying Reasoning Models
  5. Maarten Grootendorst — A Visual Guide to Reasoning LLMs
  6. Reddit — How Do Reasoning Models Work?
  7. YouTube — Reasoning Models
  8. YouTube — Reasoning LLMs
  9. YouTube — GRPO and Reasoning
  10. YouTube — Reasoning Models
This site is open source. Improve this page.