teaching_llm_applications

Tuning/finetuning and RLHF

• 🎥 Video CS395 Stanford Finetuning

• Model already knows what language is (after pretraining)

• model then tuned on specific tasks

• preference tuning

Preference tuning

• collect preference pairs and train model on it

• We prefer this

• rather than say I want this

• easier to compare than generate from scratch

• SFT data quality is very important

• 💡 If SFT data is misbehaving then preference tuning may not be the answer: check your SFT data

• also see RLHF material

• data collection

• pointwise vs. pairwise preference data

• collect pairwise preference data for prompts

• LLM as a judge

RLHF

• What is reinforcement learning?

• s_t is state you are in, a_t is action you take, r_t is reward you get

• sometimes a good or bad reward

• leverage this for preference tuning

• agent is the LLM

• state is prompt

• next token is next state

• tune the LLM

• sparse signals could be a problem in RL

• Bradley-Terry model for preference learning

• 💡 binary cross entropy loss function since pairwise

• 💡 take log of multiplication of probabilities: these numbers can get very small

• 1000s of data points from humans

• Model pretrained LLM with classification head instead of next token prediction (decoder only)

• is this regression or reinforcement learning?

• change weights of LLM using reinforcement learning

• reward model has been trained and frozen

• do not deviate too much from base model: catastrophic forgetting

• see video 🎥 Stanford CME295

• reward hacking or training instabilities (so not depend too much on reward signal)

Proximal Policy Optimization (PPO)

• do not deviate too much from base model

• This is called PPO

• KL divergence measures how different two probability distributions are

• it is not distance, it is not symmetric

• 🤔❓ is it positive or negative ?

• use Jensen’s inequality to argue it is positive (and 0 if two distributions are identical)

• advantage how much better is output compared to baseline

• value function

• value head train jointly

• generalized advantage estimation (GAE)

Variants of PPO loss

• PPO-Clip: clip ratio of new and old policy to prevent large updates

• old is not SFT model but model at previous iteration of RL stage

• 💡 updates not too big (for stability)

• L is not loss and r is ratio (not reward)

PPO-KL penalty

• ratio * advantage - KL * (value - old value)

• can PPO-clip with PPO-KL penalty

• Limitations need a lot of models (policy, value, reward model, base model from SFT)

• can look at GRPO

• Limitations: train reward model 2 stage process

• 💡on-policy (PPO)

• 💡off-policy (SFT)

• SFT on preference data (why not done?)

• SFT cannot be told not generate this

No RL- Best of N

• BoN

• pushing cost to inference

• if model being served then may not work

• inference traffic dependent

DPO

• Direct Preference Optimization

• rewrite the loss function in a supervised way

RLHF vs. DPO

• two sets of weights: one is frozen (base model) and another is updated

• 🤔❓ is there anything wrong with this answer?

• Answer it is a bit rough. This is what preference tuning can help with.

• A softer answer

It is better not to! Your teddy bear could hurt!

This site is open source. Improve this page.