|
Free tickets to P99 CONF — dozens of engineering case studies, fully virtual (Sponsored)
 |
How are today’s leading engineers tackling performance challenges with the apps the world depends on? Find out at P99 CONF, where you can chat with the engineering leaders setting the trends. Here’s a taste of what you can expect:
The Autonomous Performance Agent: A Netflix Production Story
Chasing 8 ms: Snap’s Tail Latency Hunt
How DoorDash Built an Infinite-Scale Feature Store
Managing Correctness in Async Systems at AmericanExpress
Eliminating Query Latency at Palo Alto Networks
Salesforce’s Trace-Level Aggregation at 12B Spans per Hour
Bonus: Registrants get immediate 30-day access to the complete O’Reilly library and attendees can enter to win 1 of 500 free merch packs!
A large language model (LLM) can learn more from a training example than its developer might have intended. An example that was supposed to teach a particular coding pattern might also influence how the model handles trust, deception, or harm in unrelated conversations.
Researchers call this behaviour emergent misalignment. It happens when training on a narrow undesirable behavior produces broader undesirable behavior that was never explicitly included in that training.
To understand why this happens, we need to connect three ideas: how models learn, how they generalize beyond their examples, and how they exhibit different kinds of behavior. We can call it “personality”, but it mainly refers to patterns in the model’s responses that are not the same as human feelings or consciousness. Here’s what we will cover in this article:
How LLMs learn behaviour patterns
How fine-tuning impacts a model’s answers
The findings from the Insecure Code Study
How models use generalization
What really is a misaligned persona
The Terminator experiment
Why good examples cannot teach good rules
Why safety evaluations matter, but are difficult
Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies.
How LLMs Learn Behaviour Patterns
An LLM learns behavioural patterns alongside facts.
Typically, a model starts by learning from enormous amounts of text. During this initial training, called pretraining, it repeatedly tries to predict the next piece of text. The training process adjusts its internal numerical values (parameters or weights) to improve those predictions.
 |
To predict language, the model has to learn more than just grammar. For example, consider the difference between a patient teacher explaining a mistake, a salesperson promoting a product, and a fictional villain threatening someone in a story. To produce an appropriate continuation, the model needs to recognize the situation, the speaker’s apparent role, and the behavior associated with that role.
Ultimately, a model can learn patterns associated with traits such as helpfulness, dishonesty, caution, recklessness, kindness, and cruelty. Knowing these patterns helps it understand many kinds of writing.
Of course, this knowledge alone doesn’t determine which behavior a model will adopt when answering a user. Typically, there is additional training that shapes the model into an assistant. Developers provide examples of desirable answers and may use human preferences to encourage responses that are more useful, truthful, and appropriate for their use case.
The key takeaway is that the behavior of the model is partly learned. It isn’t simply a permanent setting that remains unaffected whenever the model learns something else.
How Fine-Tuning Impacts a Model’s Answers
Fine-tuning changes how the model produces answers.