[ih] Fwd: History of AI and Internet

[Greg Skinner]

Forwarded for Barbara

> On Thursday, June 25, 2026 at 08:08:57 PM PDT, Barbara Denny <b_a_denny at yahoo.com> wrote:
> 
> 
> Hi 
> 
> I have heard things contrary to what i think you said regarding Reinforcement Learning and its applicability to today's LLMs
> 
> Here is what Gemini told me but i do know sometimes it makes big mistakes.  I still find it interesting to see what it put together. Too far out of my experience to tell you whether it produced a good response.
> 
> Yes, absolutely. In fact, Reinforcement Learning (RL) is the primary engine that transforms a raw, unpredictable language model into a helpful, conversational assistant (and, more recently, into advanced "reasoning" models).
> However, it is important to distinguish when this happens. An LLM doesn't actively run a reinforcement learning loop while it is typing out a response to you (inference). Instead, RL is used heavily during its post-training and alignment phases to shape how it behaves.
> Here is exactly how RL is used to produce the results you see.
> 1. The Alignment Phase (RLHF and RLAIF)
> When an LLM finishes its initial "pre-training," it is essentially just a massive text-completer. If you ask it, "How do I write an essay?", it might just respond with another question like, "How do I write a poem?" because it saw similar lists on the internet.
> To make it act like an assistant, developers use RL:
> RLHF (Reinforcement Learning from Human Feedback): The model generates several different responses to a prompt. Human annotators rank those responses from best to worst. A separate Reward Model is trained to understand what humans like (clarity, politeness, accuracy) and what they hate (hallucinations, toxicity). The core LLM is then optimized using an RL algorithm—traditionally PPO (Proximal Policy Optimization)—to maximize its "reward score."
> RLAIF (Reinforcement Learning from AI Feedback): Because human scaling is expensive, modern models are increasingly trained using a structured set of principles (a "constitution") where a more powerful AI acts as the judge, scoring and rewarding the trainee model's outputs.
> What RL teaches the model: Supervised learning teaches a model what a good answer looks like. Reinforcement learning teaches the model which of several plausible answers is the absolute best.
> 2. The Reasoning Revolution (RLVR)
> There has been a massive structural shift in how the newest generation of LLMs (like DeepSeek-R1 or OpenAI's "o" series) use RL.
> Instead of just optimizing for conversational style, models are now trained using Reinforcement Learning with Verifiable Rewards (RLVR) to solve complex math, logic, and coding problems.
> Instead of a human judging the output, the model is plugged into a sandbox or checker:
> The model is given a hard problem and tries to generate a "Chain of Thought" (thinking steps) to solve it.
> If it gets the answer right (e.g., the code compiles and passes tests, or the math equation balances), it receives a massive positive reward.
> If it fails, it receives a penalty.
> Through millions of these trial-and-error iterations, the model naturally learns how to double-check its own work, correct its own mistakes, and "think" systematically—entirely driven by an RL reward loop.
> The Modern RL Toolkit
> While PPO was the gold standard for a long time, managing multiple massive models simultaneously during training is incredibly expensive. Today, engineers utilize a variety of direct and architectural RL variants depending on the task:
> Method	How it Works	Primary Use Case
> PPO (Proximal Policy Optimization)	Uses a standalone reward model to nudge the LLM's token probabilities step-by-step.	Classic behavioral and safety alignment.
> DPO / GRPO (Direct / Group Relative Policy Optimization)	Math hacks that eliminate the need for a separate reward model, optimizing the LLM directly from paired choices.	High-efficiency alignment and scaling up reasoning capacity.
> Process-Level RL	Rewards the model for individual steps in its thinking process, not just the final answer.	Eliminating hallucinations in complex, multi-step logic.
> barbara
> 
> ----- Forwarded Message -----
> From: Brian E Carpenter via Internet-history <internet-history at elists.isoc.org>
> To: Greg Skinner <gregskinner0 at icloud.com>; "internet-history at elists.isoc.org" <internet-history at elists.isoc.org>
> Sent: Thursday, June 25, 2026 at 03:27:41 PM PDT
> Subject: Re: [ih] History of AI and Internet
> 
> Greg,
> 
> > Since some of you are or were professors, I’d be interested in your opinions about the required math background, as the book is designed for a course in reinforcement learning.
> 
> In my opinion, the book is a masterpiece, but it needs fluency in differential calculus and matrix algebra (which I lost many years ago). I think it would be quite hard to read cover-to-cover while skipping the maths. Also, if you're interested in applicability to large language models, it won't help you. A lot has happened since 2020.
> 
> Ananthaswamy's book that I mentioned recently relies on similar maths, but you can largely skip it without losing the thread.
> 
> Regards/Ngā mihi
>     Brian Carpenter
>