rlhfbook.com
Instruction fine-tuning would provide the basic ability for models to respond reliably in the
question-answering format, and RLHF is what takes these answers and crafts them into the
reliable, warm, and engaging answers we now expect from language models.
RLHF can learn negative feedback. Something to Avoid
需要用一个 Bradley-Terry 的模型来训练一个 reward function，让 positive 产生的 reward 大于 negative sample 产生的 reward。
然后等这个训练完之后，把它 freeze 起来，
然后可以用这个来去指导 RLHF 的学习。
Jason Wei 就是做 "pose training" 的。 2023年之后，post-training 的 popularity 就增加起来了。它不仅仅只是一个 vibe check 或者 superficial 的，它更多的是能够把 pre-training 出来的 model 的 capability 都发挥出来。 那问题来了，有没有一种能够结合 pre-training 和 post-training 在一块的模式呢？
ㅤ | Standard RL | RLHF |
how to construct reward mechanism | reward function | train a reward model from the base model |
how to assign reward | discounted larger for old steps | no discount, response level reward |
state | state transition. the action itself will shift the state, thus need to be recomputed | one school of thought mentioned that that this is a single step contextual bandit problem: 给你一个 context/state，一次性选一个 action，马上得到 reward，然后 episode 结束。没有后续 state transition |
policy | learned from scratch | fine-tuned from a pretrained LM |
horinzon | multi-step episode | often single step |
action | single environment action a_t | a completion y |
实验里面你能看到 training time completed，test time completed，以及你在 LL 的时候多 train 几步，都会增加 model 的 performance。