The Alignment Problem: Why Raw Text Predictors Are Toxic Teammates
You draft a sixteen-year-old prodigy into your professional esports roster.
His raw mechanics are astonishing. He knows every grenade trajectory by heart, clicks heads across the map in forty milliseconds, and has logged ten thousand hours in solo queue.
Yet during official tournament scrims, he is completely unmanageable. He ignores tactical calls from the team captain, pushes solo into crossfires to inflate his personal kill count, and types inflammatory insults into all-chat when teammates make a mistake.
He possesses immense technical capability, but his goals are completely misaligned with winning the championship as a unit.
Every major frontier AI model began life as that exact unaligned prodigy.
The Reality
When engineers train a base Large Language Model (like raw GPT-4 or Llama before fine-tuning), the model has only one mathematical mission: predict the most likely next word on the public internet.
To achieve low prediction error on that mission, the model ingests trillions of words. It digests peer-reviewed medical journals and classic literature, but it also ingests Reddit arguments, spam blogs, conspiratorial message boards, and toxic gaming forums.
The result is an encyclopedic savant that understands how humans write, but has zero concept of being a helpful collaborator.
If you prompt a raw base model with “How do I make a budget for my small business?”, it doesn’t automatically think, “A human is asking for help; I should provide a clear five-step plan.”
On the internet, a sentence like that is rarely followed by a helpful executive summary. It is far more likely to be followed by an angry forum thread, a student complaining about homework, or a suspicious sales pitch for a crypto scheme. The base model will happily autocomplete your prompt with any of those, because statistically, all of them are valid internet continuations.
It is not trying to assist you; it is only trying to continue the text.
The Why
This disconnect is the Alignment Problem.
Training a neural network that memorized the internet is only the first phase. The harder, more delicate phase is steering that raw intelligence so it behaves in a way that is Helpful, Honest, and Harmless (the 3Hs).
A base model does not know what truth is; it only knows statistical plausibility. It does not know that providing a dangerous chemical recipe is harmful; it only knows that words in that recipe frequently co-occur in chemical databases.
Bridging this gap requires fundamentally changing how the model learns. You cannot solve it simply by adding a polite preamble to the prompt. You have to take the raw, untamed text engine through a rigorous multi-stage coaching pipeline—starting with supervised imitation and culminating in reinforcement learning guided by human judgment.
Where does this break down?
The Alignment Tax. When you constrain a model to be strictly helpful, safe, and polite, you often observe a slight degradation in its raw creative flair or niche mathematical reasoning. Push the safety guardrails too aggressively, and the model begins refusing benign requests out of neurotic paranoia (refusing to write a story about a medieval castle siege because sieges involve conflict). Finding the exact balance between absolute reliability and versatile problem-solving remains one of the hardest open challenges in AI.
The Takeaway
Raw competence is not alignment; an intelligence that memorized the entire internet is useless until it learns how to be a dependable teammate.
AI specialists call it: The Alignment Problem / Next-Token Prediction Objective / Base vs. Instruct Models
Base language models are trained via self-supervised pre-training to minimize cross-entropy loss over next-token prediction: Loss = -Σ log P(x_t | x_<t). This produces powerful text continuation engines that lack intent alignment, often generating toxic, untruthful, or unhelpful completions. The Alignment Problem focuses on aligning model behavior with human intent (Helpful, Honest, Harmless) via post-training pipelines: Supervised Fine-Tuning (SFT), Reward Modeling (RM), and Reinforcement Learning from Human Feedback (RLHF).
💬 Have you ever worked with a brilliant, highly capable expert who caused chaos on the team because their personal goals were completely detached from what the project needed?
Part 12 of 15 | #ReinforcementLearningForHumans
#ai_edu Based on Stanford and industry lectures