Teaching the Machine
to Teach
Rachel Koblic
- A learning designer by trade — a generalist at heart
- Six years at 2U / edX leading the design of online degree programs
- Most recently Chief Learning Officer at Matter & Space, reimagining learning in the age of AI
Neither of us does this alone
An emergent capability — greater than the sum of its parts, present in neither one alone.
The whole university
Get the machine's part right — and people get more time to be human.
The functions no machine replaces
Embodied skill
Show what can only be shown, not told.
Lived experience
Share what it was actually like.
Real empathy
See the person, not the performance.
Belonging
Build the room people belong in.
Trust
Be safe to say “I don't get it.”
Accountability
Be the one they want to rise for.
Frame-breaking
Introduce the unknown unknowns.
Judgment
Know “good” when the rubric runs out.
Provocation
Demand more than they think they can give.
Inspiration
Model who they might become.
Meaning
Help them find why it matters.
Moral example
Show integrity under pressure.
Structural, not fluffy — and this is only a start.
Designing for the machine learner
AlphaGo · 2016
- Built by Google DeepMind to play Go
- Trained on hundreds of thousands of human games
- Then it beat the best human player in the world
Source: deepmind.google/research/alphago
AlphaGo Zero · 2017
- This time: just the board, the rules, and a win condition — then let it play itself
- In three days it surpassed the original AlphaGo
- It invented moves experts called mistakes — until those moves won
Source: deepmind.google/blog/alphago-zero-starting-from-scratch
To teach AI to teach…
The Board
The Win Conditions
The Rules
What might we give AI to help it learn how to teach?
The Board
The terrain of knowledge and skills being learned
Knowledge graphs
A structured, machine-readable map of a domain — and increasingly, how serious players represent the board.
UMGC
A “skills architecture” inventorying every course by the skills it builds — using AI to assess prior learning against it.
Strategic Plan 2024–2030Coursera
A Career Graph mapping roles → skills → courses → credentials, serving millions of learners at scale.
Career GraphCZI Learning Commons
Machine-readable standards, components, and progressions — queryable via API. Built with Anthropic.
Knowledge GraphWhat is a graph?
- Nodes — concepts and skills
- Edges — how they relate
- Metadata — additional contextualizing information
Hover a node to see its metadata.
A veteran's mental model, made explicit
Name a course you teach
{
"metadata": { "title": "...", "domain": "..." },
"nodes": [ { "id": "...", "label": "...", "type": "...", "description": "...", "misconceptions": ["..."] } ],
"edges": [ { "source": "...", "target": "...", "relationship": "...", "description": "..." } ]
}
The Win Conditions
The win condition is that learning has actually happened.
And we already describe that all the time — in our rubrics.
Rubrics for humans vs. for AI
A human reader fills the gaps with judgment — they know what “strong” means here.
## Win condition
The learner succeeds when they pose a clear, strong research question — one that names measurable variables, is grounded in prior research or theory, and predicts a relationship that could be shown wrong.
## Evidence of success
- names specific, measurable variables
- operationalizes each variable
- grounds the question in prior research
- predicts a falsifiable relationship
## Common failure modes
- asks about a topic, not a relationship
- uses vague, unmeasurable variables
- ignores prior research and theory
- poses a question nothing could disprove
You do the work — calibrate by hand
↻ Repeat until it shares your win conditions
Just like calibrating human graders for inter-rater reliability.
Let it learn you
It learns your internalized rubric — even the parts you couldn't fully articulate.
A win condition on every node
Score every node continuously — a live map of what a learner has mastered, and exactly where they're stuck.
Back to our graph — add the win conditions
A win condition is the machine-readable version of a rubric criterion: one explicit sentence describing what success looks like for that concept, written for an AI to judge against. Human rubrics lean on tacit words ("poses a STRONG research question") that a human grader fills in with judgment. Make that judgment explicit instead — spell out exactly what "strong" means here, name the observable things a successful learner does, and leave nothing vague.
Return the SAME JSON with a "win_condition" string added to every node. Change nothing else.
The Rules of the Game
The rules of Go
- Black goes first.
- Connected stones form a group.
- A group needs at least one adjacent empty point to stay alive.
Hard-and-fast. Teaching's rules… are not.
The moves great teachers make
- Answer a stuck learner with a question before an answer.
- Leave room for productive struggle.
- Probe the why, not just the answer.
- Hand over the solution when a hint would do.
- Confirm a right answer reached by wrong reasoning.
- Move on before a misconception is resolved.
We encode these into prompts that guide AI.
The LLM's instinct fights good teaching
frictionless
productive struggle
The right pedagogical move is often not the logical one — so we give the AI explicit rules.
Be selective
Models get overloaded
There's only so much a model can hold and use well — which is why “context management” is suddenly everywhere.
Cost adds up
Everything you put into an LLM costs tokens — and tokens add up to real dollars, at scale and over time.
No universal rulebook
Every educator and institution runs its own mix of principles — and many only apply sometimes.
The work isn't dumping in everything — it's curating the playbook that works here.
Every rule we insert is our best hypothesis — what research says works for some learners, in some conditions.
The real unlock is when the system learns which rules work for you.
The Feedback Loop
Assessment becomes continuous
Not evidence for grading — evidence for adaptation.
Learning from a range of data
- correct vs. incorrect
- retention & transfer
- error patterns
- pace & pauses
- retries, skips
- returns or not
- time of day
- time available
- device
- goals & interests
- prior knowledge
- history
- frustration
- confidence
- curiosity
How much time have you got?
Learning is long and messy — teaching every kind of learner, one real student at a time, could take forever.
Wire up a roster of synthetic learners
A second AI role-plays each one — so the teacher can practice on many kinds of learner, with no real student at stake.
Use cases for synthetic learners
Pressure-test a lesson
Before a single real student is at stake.
Rehearse
Hand it to a nervous new instructor to practice on.
Validate the graph
Check whether the board we built actually holds up.
A working learning system
A prototype I built that puts the pieces together — a knowledge graph with win conditions, AI agents with rules, and synthetic learners.
Models + a harness
The model is just the engine. Everything around it is the pedagogical harness that helps it learn.
A chance to rethink the whole ecosystem
What might we be able to achieve in partnership with AI?