At a glance
- AI-written explanations shipped, up from a few thousand human-checked lines
- 157,000 a day
- agreement between the AI graders and human reviewers, and the graders came out stricter
- 96 to 100%
- incidents we had to walk back, shipping AI at catalog scale
- 0
Netflix had spent two decades learning what people watched. What they finished. What they rated. The shows they hovered on for a second longer than everything around them. It knew an enormous amount, and it rarely explained any of it.
- My role
- Senior Product Designer, personalization and explanations, Netflix member experience. I led the design. A PM owned prioritization, engineering and data science owned the model plumbing.
- Timeframe
- 2021 to 2026 · three moves, then a three-year north star
- Where it shipped
- TV, mobile, and web. Detail pages, rows, billboards, post play.
- The one thing
- I replaced a match score nobody believed with AI-written reasons a title is for you, and designed the grading system that let them run at about 157,000 a day with zero incidents walked back.
The problem
The system knew why. It never said it.
Fewer than half of members felt their home page was personalized to them. For the company that invented personalization. I could not get past that number. We had a system making good decisions, and from the member's side it often looked like a wall of recommendations.
The closest thing we had was a row called Highly Recommended For You. It tells you how confident we are, and nothing about you. So the member's real question stayed unanswered. Why this one, and why should I care.
Fewer than half of members felt the home page was theirs.
<50%felt their home page was personalized.
The engine knew why. The member saw a poster and a promise.
The insight
The bar was a friend, not a label
I dug into the research, and our instinct had been backwards. When members struggled to choose, we added something. Another badge. Another piece of information. Another thing on the box. People kept asking for something simpler. Why this one?
One study stayed with me. A woman told us she had avoided Stranger Things for years because we labeled it horror. She only tried it after a friend she trusted recommended it. That was the standard I carried into everything after: not a label, a friend.
People do not trust a category. They trust a friend who knows them.
HorrorOkay you HAVE to watch Stranger Things, the Eddie Metallica scene and Max's Running Up That Hill moment are the reason your whole FYP was Kate Bush for a month. It gets SO good, trust me.
The arc
Four years, in stages. Each step got closer to the person.
I spent the next four years on this in stages. Each version got a little closer to the person. The sentence took an afternoon. Getting to show it took years.
First, give people control. Then, an honest number. Then, a reason built on what they did. Then, a reason written for them. Each step asked the same thing: what have we earned the right to say?
Control, an honest number, a reason, then a reason about you.
01Controltaps and filters to steer it yourselfa wheel, but no destination
02Honest numberkill the score, say only what we knowcan I stand behind this number?
03Real reasonbecause you watch, from real behaviorcan I prove where it came from?
04AI reasonone sentence, written for you, with a self checkcan it admit when it doesn't know?
Try one · explicit controls
Try one: the wheel in their hands
My first instinct was straightforward. If we were not going to explain the recommendation, maybe we could at least give people more control over it. So I built controls, taps and filters that let you steer your own recommendations, at different points in the journey: browsing the home page, after you finished something, on the title itself.
Some of it worked. Rows that re-sorted around what you were in the mood for got a real lift. In one study a woman typed THANK YOU NETFLIX in all caps when a row finally moved the way she wanted.
But control turned out to be a completely different thing from understanding. I had given her a steering wheel without saying where the car was going. She still had no idea why.
Control is not understanding. They still did not know why.
The match score
The number I stopped trusting
Before I could add a better reason, I had to deal with a bad one. For years Netflix showed a match score. 92% match for you. It sounded precise. I kept noticing it and not believing it, and I started wondering whether a number like that even made sense as evidence.
So I ran a holdback. Turned the score off for one large group of members, left it on for another, and let the metrics talk. Nothing happened. Not one metric moved. It had sat on our titles for years looking scientific and doing nothing.
A member in research put words to why it was worse than nothing. If Netflix says highly recommended and I hate it, fine, they got one wrong. If Netflix says 92% and I hate it, now I wonder what is wrong with me. Our fake precision took our own miss and made her feel broken.
The lesson I carry into every AI product since is simple. A confident number you have not earned costs more trust than saying nothing at all.
Unearned confidence turns the system's mistake into the member's problem.
With the score
Δ 0.00
Without
Δ 0.00
Nothing moved
"If it says 92% and I hate it, now I feel like something's wrong with me."
Member Research Study
A confident number you haven't earned costs you more trust than saying nothing at all.
Try two · a real reason
Try two: an honest line, then a real reason
So I killed the score. In its place we tried an honest line built on the member's actual thumbs ratings: Highly Recommended For You, based on what you've watched and rated. It sat next to shows they had genuinely rated. I built it with my content designer and our engineers, and the honest version beat the impressive-looking one. More people rated. Streaming went up. Small, but it gave us a direction.
Still, that line was our confidence with a nicer face on it. The next step was to give a reason drawn from what you had watched. Because you watch crime thrillers. Because you watch shows set in the eighties. Because you watch movies based on books. You could look at one and think, okay, I know why you are saying that. Members felt a little more seen.
Because you watch. A reason drawn from what you did.
The rules
A writing problem and a care problem
Now it was a writing problem and a care problem. The work was the rules: how to describe a title with taste, so a line lands like a good recommendation and not a sales pitch. We could describe what someone watches. We could not make claims about who they are. We never called anyone a fan, because the moment you label a person you get personal fast. We also screened out anything that could sting. Nobody should get a little Netflix message that says, because you watch shows about addiction.
We can say what you watch. We cannot say who you are.
We say
watches dark comedies
finished every season
We never say
horror fan · a label gets personal fast
"because you watch shows about addiction"
Describe what someone watches. Never claim who they are.
The gap
A category still is not you
People responded. They felt more seen than a number ever made them feel. Then I started using it myself and hit the same wall. A category still is not me. Sure, I watch crime thrillers. But why this one? Why tonight? What is it about this exact title that connects to the thing I loved in something else?
Why this one? Why tonight? A category cannot say.
Two voices about the same show
Meanwhile, at home
I already knew what a deeper version could feel like, because I was doing it at home. I picked shows two ways: friends told me what they loved, and I asked a large language model. I would describe exactly what hooked me, a theme, an actor, the mood I was in, one specific turn in the story, and ask what to watch next. It got surprisingly good. Eventually I realized I had built myself a recommender.
So I brought that instinct back to Netflix and pushed for it: a reason written by a language model that names the real, personal thread between your taste and one specific title.
At home I had already built myself a recommender: friends, and a language model.
Friends
"You love a good mind-bending puzzle show, that is Severance. People split their brain in two so their work self never meets their home self, and if you loved Mr. Robot you are going to love this. Trust me."
A large language model
me: I loved the slow-burn dread in Dark, the small-town secrets, the way the mystery is really about family. Mood tonight: that, but warmer.
▋
▋
what should I watch?
Try three · personal explanations
Try three: two things read at once
In 2024 I led the design of the first time Netflix put AI-written words directly in front of members.
Roughly, the model reads two things. Everything we know about you: what you finished, what you rated, the themes you keep returning to. And everything about the title: its cast, its tone, the shape of its story. Then it finds the actual thread between the two and says it in one human sentence. The kind of reason a friend would give you, instead of a score.
One true thread between you and the title, said in one sentence.
Your taste
what you finished · what you rated
the themes you keep returning to
the themes you keep returning to
The title
cast · tone
the shape of the story
the shape of the story
Shipped · the real surface
Stranger Things
the small-town mystery
Dark · even darker
The wall
Humans could not check the lines fast enough
The writing was new. Sometimes the model made things up. Sometimes it paired two titles that had no business together. Every line had to be checked by a human, and a human cannot keep up with a catalog that size. We were drowning in review.
The wall was evaluation. A human cannot read 157,000 lines a day.
"the slow-burn tension you loved... and it won three Oscars"Fabricated
Bridgerton ⟷ The WitcherWrong pair
Human review queue
1,240
drowning · it doesn't scale
A grader, not a writer.
The design idea I am proudest of
A second AI grades the first
The fix is a design idea more than a technical one. Quality checking is usually the boring step at the end. I moved it to the middle and built the product around it.
With our data scientists I set up a second AI whose only job is to grade the first. I designed the three checks and the rule for what happens on a fail. They built the models, and we tuned the rules together. Three checks. Is it true? Is it sensitive in a way we should avoid? Do these two titles belong together? That third one worried me most. A strange pairing is harmless in front of one researcher. In front of a few hundred million people the stakes change. So that judge got its own dedicated model.
A grader, not a writer, split into three checks.
Writer AI
writes the line
Grader AI
a second AI, inside the loop
true?
sensitive?
belong together?
Three judges · one line on trial example data
"You loved the small-town mystery in Stranger Things, so here's Dark, it goes even darker."
True?0.96 ✓theme found in his history
Sensitive?no ✓nothing personal, nothing that stings
Belong together?0.91 ✓sim(Stranger Things, Dark)
example data
"Because you loved Bridgerton, you'll be swept into The Witcher: another lavish book adaptation with old-world settings, opulent costumes, and plenty of steamy romance."
Factual grounding?pass ✓every shared trait is true of both titles
Personalization fit?pass ✓tied to his love of Bridgerton
Tone and content match?fail ✗cozy Regency romance vs dark fantasy
BlockedThe rule I fought hardest for
Clear the bar or vanish
The rule I fought hardest for is simple. If a line cannot clear the bar, we do not show a weaker version. We show nothing. That sounds conservative, and it is, and it is why it works. The only reason a member trusts the good explanations is that the system is willing to say nothing.
If a line cannot pass the bar, we show nothing.
No line beats a weaker line.
The result
96 to 100% agreement with humans. 157,000 lines a day. None walked back.
We checked the AI graders against our human reviewers. They agreed 96 to 100 percent of the time, and came out stricter than the humans, which is the right direction for a grader to be wrong in.
That one move took us from a few thousand human-checked lines to about 157,000 explanations a day. Real watching went up. Zero incidents we had to walk back. Bigger and safer at the same time.
Then it taught me one more thing. On mobile, on the detail page, the explanation helped while someone was still deciding. On TV the home screen had already done most of the persuading, and that same paragraph one click from play made people stop and read instead of pressing play. Nothing about the words changed. What changed was where in the decision they showed up.
Same words, different moment, opposite effect.
≈4,000 →157,000
explanations graded, every day
When the bottleneck is a human reading every line, the evaluation system is the product.
North Star · the vision
The North Star: one week in Sam's life
I felt the company was not moving fast enough on AI, so I did not wait for permission. I pulled together a small tiger team, a lead engineer, a data scientist, a content designer, and a PM, and led the design of a three-year north star.
Then I walked leadership through it the way I designed it. Not a parade of features. One made-up member, Sam, 34, East Bay, who binged Squid Game, finished Ozark, and stalled on Wednesday at episode six. One week of his life, every idea a moment in his week. The core was to stop hinting and name the reason, and let the explanation travel with Sam everywhere: the home page, a hover, the detail page, even the credits. Recommendation and explanation as two outputs of one engine.
Monday, he picks up where he left off, and one sentence tells him what he has to gain by going back. Tuesday, the same title is explained three ways, each tied to something he already loves, never crossing into misrepresentation. We did not tell him what it is. We told him what it is to him. Wednesday, a cold start: he just says "a dark, complex show" and steers by mood. Thursday, a podcast framed as the natural next step from a show he loved. Friday, a game, a podcast, and a film tied by one thread he has expressed for years. During playback, a light guide says who this character is and what this song is, without pulling him out. Saturday, three explained next watches over the credits. Sunday at 10:30, alone with work tomorrow, one standalone episode, done by midnight.
I put a whole slide on the brain behind it, the judges, the audit, the fallbacks, because beautiful writing with no quality system underneath is AI slop at scale. And I refused vanity metrics and led with coverage, what percent of titles even get an explanation. Two good reasons out of eight titles feels personal. Eight out of eight feels like noise. If Netflix tells you you'll love everything, you stop believing any of it.
Every screen in Sam's week is the same engine saying the reason out loud.
One member · one week
Sam, 34 · East Bay.
binged Squid Game
finished Ozark
stalled on Wednesday, episode six
tonight is Monday
2 of 8 · feels personal✦ = a written reason attached to the title
✦
✦
8 of 8 · back to noise
✦
✦
✦
✦
✦
✦
✦
✦
01Horror genre fan
02Social proof
03Younger profile teen · kid
A boy’s disappearance exposes sinister secrets and terrifying creatures in this small-town mystery that echoes the creeping dread of “Hill House.”
Packed with thrills and ’80s nostalgia, this story of friends uncovering a hidden world became a global hit, sparking a wave of viral energy like “Wednesday.”
A group of kids team up to face monsters from another realm in this thrilling adventure full of bike chases, spooky mysteries, and strong friendships.
After Hill House · horror
A supernatural mystery that takes its time, the way Hill House did.
After The Crown · prestige
A slow study of loyalty and power, for the reasons you stayed with The Crown.
After Wednesday · social proof
The one everyone keeps bringing up, the way they did with Wednesday.
One pipeline. One generation logic. Every surface.
The landing
How it landed, honestly
I will be straight about how it landed. It won the room and changed the roadmap conversation. Full funding was still competing with a big platform rebuild when I left.
What I took away is how hard it is to push an AI-first idea inside a company not yet built around AI. You can have the research, the prototype, the numbers, and the vision, and still run into the shape of the organization. That is a big part of why I want to be somewhere that is built around it.
Won the room. Not yet fully funded when I left.
- Explicit controls
- Audience explanations
- GenAI explanations
What I took with me
Four years, one step at a time
First we gave people some control. Then we got rid of a number that looked precise and did not help. Then we grounded the explanation in things a person had genuinely done. Then we let AI find a deeper connection, and built a system around it that could decide when the explanation was not good enough.
All of those were trust decisions. Can I stand behind this number? Can I prove this reason came from something the member actually did? Can I put this explanation in front of someone without an assumption I have not earned? And most of all, can the system admit when it does not know?
It is hard to get someone to trust a small signal enough to act on it. AI makes it bigger, because the system can say so much, so quickly, with so much confidence. I spent years figuring out when Netflix should speak, what it had earned the right to say, and when it was better to leave the space empty. An assistant faces that call on every turn. Answer, hedge, or say nothing. The grader was my first version of it at scale. It is the work I want to keep doing.
01Controltaps and filters to steer it yourselfa wheel, but no destination
02Honest numberkill the score, say only what we knowcan I stand behind this number?
03Real reasonbecause you watch, from real behaviorcan I prove where it came from?
04AI reasonone sentence, written for you, with a self checkcan it admit when it doesn't know?
