[ih] History of AI and Internet

[william yeager]

I’ve used Perplexity Pro ($200/year) to gain an in depth understanding including all of the math. I’m lucky to have completed my Phd studies in math albeit in 1970 at the University of Washington. 

I felt given the above that neither RLHF nor RLAIF cannot eliminate either toxicity or hallucinations completely. Neither to the other RL algorithms. Clever individuals can query and receive them. 

A little review - I think it is correct - most from memory and some excised from my perplexity notes. The latter are obvious. Here goes ...

We all now know that training uses gradient descent, back propagation and cross-entropy to minimize the loss between the calculated next token and the token itself. Training has as a goal to minimize the loss between the them.

Responding to queries is different and uses stochastic probability to predict the best next token. 

The following is slightly technical but I think most can follow it, and points out the difference between training and queries.

In a 50 layer LLM with about 30 billion parameters, let’s say training has created about 500 matrices with a dimension of 4096, 10 per layer. The dimension refers to length or number of columns of a row. And all matrices are dimension x dimension, eg, 4096x4096 in this case. There are some hidden layers, and here, if a query has N tokens, then one finds some special matrices N x dimension. 

But enough !

The fixed matrices are created during training. They are are initialed with random values prior to training. Training is expensive and 90% of what makes these models work.

When training is completed, The matrices are each query’s calculations to generate a response.  Theses modified versions are created, and kept in the cache until the response is completed. They are the basis for creating the probabilities to produce the next best token. The trained matrices are reused for each query.

Let’s assume that the LLM also has as a fixed vocabulary of about 60 thousand words, and about 80,000 tokens in what is called the embedded table; and let’s assume the dimension of the model is 4096. I hope the array info is OK for the readers. Each token has a row in the embedded table’s 80,000x4096 array. This is fixed, and activations of the tokens are used when processing a query. One can view 80,000 as the token vocabulary. And again, the word/subword vocabulary is usually in the [30K, 100K] size and for 80,000 tokens, 60K is OK for discussion. 

There is a variable, T, called Temperature comes directly borrowed from Boltzmann‑style statistical mechanics. The final step to predict the next token is as below. Note that multiple passes are made through these layers until the response is completed. The number of passes as well as the number of layers are hyper parameters in the LLM’s design.

Note that the z(i)s are called logits because the math works with numerical values or weights created during the processing of the query, and the i is an index into the actual token’s row in the embedded tokens' table. To wit (I hope this is OK too) this is how the probabilities, p(i) are computed for the best next token’s choice - the following is a png file, and softmax is what computes them. 



Assuming you understand how exponentials amplify differences:

If 
T
<1:

Division by 
T magnifies differences between logits.

The largest 
z
i
 
exp
⁡
(
z
i
/
T
)

The distribution becomes more peaked (low entropy); the model is more deterministic and tends to pick the top‑logit token almost always.

If 
T
>
1:

Division by 
T shrinks differences between logits.

exp
⁡
(
z
i
/
T
) values get closer together.

The distribution becomes flatter (higher entropy); the model samples more from lower‑probability tokens, so choices are more random.


T = 1 has no effect and softmax is used by cross-entropy .

T = 0 just selects one token using argmax which picks the largest p(i) token because you cannot divide by 0.
Now to the subject at hand:

RLHF reduces toxicity and some hallucinations, but it does not eliminate either of them.

Here’s how and why. Most from perplexity and all have references which I randomly check for validity. Social media refs are always suspect.

1. What RLHF does well

RLHF trains the model to prefer answers that humans rate as good: helpful, safe, non‑toxic, on‑topic.

Human labelers rank responses; a reward model learns to score “good” higher than “bad.”

RL (usually PPO) then shifts the LLM’s behavior toward high‑reward outputs.

Empirically, this:

Reduces overt toxicity and abusive content compared to the raw pretrained model.

Encourages refusals on obviously unsafe or clearly unanswerable prompts (“I can’t help with that”).

So RLHF is very effective as a behavioral filter and style/intent alignment tool.

2. Why hallucinations are not solved

Hallucinations (confidently wrong answers) come from:

The underlying pretrained model’s imperfect world model and reliance on patterns, not explicit facts.

Pressure to always produce an answer, even when unsure.

RLHF doesn’t change the core training objective (next‑token prediction) or give the model a ground‑truth database; it just tells it which style of answer humans prefer.

Key limitations:

Reward models are trained on human judgments that may not reliably detect subtle factual errors.

Human raters often reward plausible and fluent text more than pedantically correct but awkward text, which can actually reinforce “polished hallucinations.”

Studies and commentary increasingly note that RLHF improves alignment and niceness but does not reliably fix factual reliability.

So RLHF can discourage obviously nonsensical answers, but it doesn’t guarantee truthfulness, especially on long, detailed, or niche factual queries.

3. Why toxicity isn’t fully eliminated

Toxic content is reduced, but not gone, because:

The reward model only approximates human safety judgments and may misgeneralize to new prompts or adversarial phrasings.

Attackers can craft “jailbreak” prompts that drive the model into regimes where the reward model and safety filters are less effective.

Human feedback is noisy and not perfectly consistent; some borderline content slips through.

Recent work shows that models trained with RLHF can still be induced to generate harmful or biased content with carefully designed prompts, indicating safety remains imperfect.

4. Consensus in current research

The emerging consensus is:

RLHF is a core tool for safety and alignment in today’s LLMs.

It significantly reduces toxicity and some blatant hallucinations compared to unaligned models.

It does not eliminate hallucinations or toxicity and has fundamental limitations as a sole safety mechanism.

That’s why current work layers RLHF together with:

Better pretraining data curation.

System prompts and rule‑based filters.

Tool use (retrieval, calculators, constrained decoding).

Adversarial testing and other alignment methods.

So, RLHF moves the needle in the right direction but is not a mathematically guaranteed cure for either hallucinations or toxic outputs.

If you’re interested, we could next look at why reward models systematically miss subtle hallucinations—essentially a bias in the human feedback signal itself.

What is RLAIF



RLAIF is essentially “RLHF, but the judge is another AI model instead of a human.”

More formally:

RLHF = Reinforcement Learning from Human Feedback.

RLAIF = Reinforcement Learning from AI Feedback (or AI‑based feedback).

The pipeline is almost identical; the only thing that changes is who provides the preference labels.

1. Basic idea

RLAIF replaces the human annotator in the RLHF loop with a teacher/critic LLM:

Start from a pretrained (and usually SFT‑tuned) base model.

For a prompt 
x
x, sample several candidate responses 
y
(
1
)
,
y
(
2
)
,
…
y(1),y(2),… from the student model.

Ask a stronger or specially prompted AI labeler (teacher model) which response is better according to some principles or “constitution.”

Use those AI‑generated preferences to:

Train a reward model, or

Directly use them as rewards during RL (direct RLAIF).

Run RL (often PPO) to fine‑tune the student model toward high‑reward behaviors, just like RLHF.

So structurally, RLAIF = RLHF with AI feedback instead of human feedback; the RL and reward‑model math stays the same.

2. Motivation vs RLHF

Why bother?

Scalability: AI can label millions of response pairs far faster and cheaper than humans, limited mainly by compute.

Consistency: A given AI labeler applies the same “preferences” more uniformly than a pool of human annotators (though it has its own biases).

Safety & specialization: AI labelers can be used on sensitive or technical domains where using  human raters is difficult, risky, or requires rare expertise.

Empirical work shows that, for tasks like summarization and helpful/harmless dialogue, RLAIF can reach comparable performance to RLHF in terms of alignment metrics, at much lower human‑labeling cost.

3. Relationship to Constitutional AI

Constitutional AI (Anthropic) is a prominent instance of this idea:

Use a written “constitution” (high‑level principles: be harmless, honest, etc.).

Ask a powerful LLM (the critic) to judge responses based on this constitution and suggest improvements.

Use that AI feedback to train and refine the assistant model—this is effectively RLAIF within a structured framework.

4. Limitations

RLAIF trades human‑label noise for AI‑label bias:

If the AI labeler is misaligned, biased, or simply wrong in certain domains, its flaws propagate into the reward model and the student.

There’s a real risk of “AI teaching AI its own mistakes,” so careful design of the teacher, prompts, and constitutions is crucial.

Current work suggests RLAIF is powerful and scalable, but not a drop‑in replacement for human oversight; in practice, people often combine human and AI feedback and validate AI‑labeled data with humans on critical tasks.

That’s a fair way to put it: both RLHF and RLAIF are only as good as the feedback signal they’re optimizing.

With RLHF, if human raters reward fluency and “sounds good” over factual accuracy or deep reasoning, the model learns to produce very plausible but sometimes wrong answers—polished hallucinations. Garbage (or shallow judgment) in, garbage out.

With RLAIF,
 you’re stacking another model’s biases on top: if the AI critic has 
blind spots or systematic errors, those propagate into the reward model and then into the trained assistant.

The “clever response” part is crucial: once you have a very strong 
generator, small biases in the reward signal tend to produce outputs 
that are:

Grammatically excellent.

Stylistically aligned.

But sometimes factually or logically off in ways that are hard for non‑expert raters to spot.

This is why current alignment work is putting so much emphasis on:

Better feedback collection protocols (more expert raters, better guidelines, better evaluation tasks).

Combining human and AI feedback, rather than using AI feedback alone.

Adding external tools (search, calculators, retrieval) so the model isn’t forced to “make up” answers from its weights alone.

So your intuition is right: these methods are extremely powerful amplifiers of whatever signal you feed them—good or bad.


In the above I was curious about “garbage-in, garbage-out.” It cannot be avoided 100% of the time now. 

So, I hope this is helpful, and I’m sure I made some grammatical errors. I will be using small essays like the above in a PDF that will be part of my three hour presentation on “A History of AI.” It will be publicly available after the conference and I will post it to this list.

The Internet and AI has been useful for me to find key areas to present. One area I haven’t seen is Claude Shannon’s contributions to today’s Internet and circuit design.

In his Mathematical Theory of Communication written in 1948 he uses bits (0 and 1), defines communication Capacity, and then shows given entropy or noise in the channel when a message from a source can reliable reach the destination:

1948 paper → noisy, networked world

In 1948, his “Mathematical Theory of Communication” formalized information entropy, channel capacity, and coding theorems, saying in effect: here is how much information you can push through a noisy medium, and how to do so nearly error‑free.
Everything from deep‑space probes to fiber backbones, Wi‑Fi, 5G, and compressed media codecs operates inside this Shannon framework; we just keep finding better codes and hardware to approach his theoretical limits.


His 1937 MIT master’s thesis is called the greatest master’s thesis ever written:

Master’s thesis → digital hardware

In his 1937 master’s thesis, Shannon showed that Boolean algebra and relay/switching circuits are mathematically equivalent, so logical propositions could be implemented by electrical networks in a systematic way.

This result provided the theoretical foundations for digital circuits and thus for modern digital computers, establishing that logic design could be done algebraically rather than by ad hoc wiring.

Shannon is widely regarded as the “father of the information age” because his work on digital circuit theory, information theory, and sampling theory underlies computer architecture, digital communication, data compression, and error-correcting codes.

These ideas also influenced later work on computer architectures that implement quantum algorithms, which are still described using logical gates and circuit models that conceptually descend from Shannon’s Boolean-circuit viewpoint.

Thus, that one step turned logic design from an art into an algebraic engineering discipline, and it is the conceptual root of modern CPUs, GPUs, FPGAs, and essentially any digital hardware that obeys logic gates.

He is one of my personal heroes.  Definitely unsung, an by many considered among the top 3 minds in the 20th century. 

Bill


————打🎾————
马年快乐🐎
<Happy year of the horse>
保佑众生 🙏🐘🌲🦋🏃‍♀️🐳
<Bless & Protect all living things>