Columbia UniversityCOMS W4995 Agentic EngineeringOctober 2, 2026

The End of Fake Learning

When finishing the work stopped proving you learned it

The End of Fake Learning▶ Play the slides

For your whole education, one assumption did a lot of quiet work: if you finished it, you learned it.

Teachers graded on it. Employers hired on it. And you judged yourself by it.

AI just broke that assumption. Anything you can finish, it can now finish for you. So here is a question most of us were never made to answer: what does it actually mean to have learned something? And why does learning so often feel backwards?

A guest talk for Columbia’s Agentic Engineering course · about 15 minutes to read

01What learning isThis slide ↗

The grade sees the result. Learning happens in the process.

Start with a thought experiment. Take any homework you have ever handed in. Now imagine you had copied it instead. Same answers. Same grade. Your transcript cannot tell the difference. But you learned nothing.

Why? Because the grade looks at the result, and learning happens in the process: being stuck on problem three for an hour, trying the wrong approach, figuring out why it failed. Copying keeps the result and deletes the process.

The resultwhat the grade seesThe processwhere the learning happens
Do the homeworkA gradeLearning
Copy itThe same gradeNothing
Same result. Same grade. No process, no learning.

For most of your education the two came together, because the only practical way to get the result was to go through the process. So we never had to ask what learning actually is. Teachers and grades answered that for us.

Now AI can produce the result without you going through the process. So you have to answer it yourself.

01What learning isThis slide ↗

Real learning is what you can still do: without the help, later, on a new problem.

Learning scientists have a classic distinction for this: performance versus learning.⁠1

Performance is how well you do right now, during practice or right after it, usually with help around you. Learning is what is left later, when the help is gone, time has passed, and the problem has changed.

  1. 01 · Take away the helpNo AI. No notes. No tutorial.
  2. 02 · WaitA week later, not right after.
  3. 03 · Change the problemNew inputs. A new constraint.
Doing well with the help, right after, on the same problem is performance.

So here is the definition I want you to leave with. You have really learned something if you can still do it without the help, a week later, on a problem you have not seen.

Everything else is performance. And fake learning is simply mistaking one for the other: treating “I made it” as “I learned it.”

Which raises a practical problem. If learning only shows up later, how do you know, while you are learning, whether it is happening?

02BackwardsThis slide ↗

Guess first: which group learned more? Which group felt it did?

Mostly, by feeling. So let’s test your feeling.

Harvard ran a randomized experiment in an introductory physics course.⁠2 Same content, same handouts. One group got a clear, polished lecture from a highly rated instructor. The other group worked through the problems in small groups, got stuck, and then got the explanation.

Group A

A clear, polished lecture from a highly rated instructor.

Group B

Work the problems in small groups. Get stuck. Then the explanation.

01Which group learned more, measured by a test?

02Which group felt it had learned more?

02BackwardsThis slide ↗

Your feeling of learning is often backwards.

Group B (worked it out) compared with Group A (polished lecture)

See the numbers
A · polished lectureB · worked it out
Learned (test score)LessMore · +0.46 SD
Felt they learned (self-rating)MoreLess · −0.56 SD

SD: standard deviations. Deslauriers et al., PNAS, 2019.⁠2

Make both guesses first, or .

Group B, the ones who worked it out, scored clearly higher on the test. And they felt they had learned less. Group A, the lecture, learned less, felt they had learned more, and said they wished every physics class were taught that way.

Smooth made people think they had learned. Struggle made them think they hadn’t. The truth was often the reverse. The researchers’ reading: the extra effort of working it out felt like not learning, when it was actually a sign that learning was happening.

We see the same thing in our own AI courses. We build in guided mistakes on purpose, and students get frustrated; some decide the course is bad. The smoothest sessions get the warmest reactions, and it is easy to walk out remembering the feeling and not the skill.

So why would struggle teach more?

02BackwardsThis slide ↗

The more you get wrong, the more you learn.

Because learning happens where your idea and reality don’t match.

  1. 01You guess.
  2. 02Reality disagrees.
  3. 03You ask why.
  4. 04Your idea changes.

You guess. Reality disagrees. You ask why. And the idea in your head actually changes. If you never guessed, there is nothing to disagree with, and nothing changes.

Psychologists have shown this again and again. Try to answer first, even if you get it wrong, then see the right answer, and it sticks better than if you had simply studied the answer.⁠3

And if your guess a minute ago was wrong, you will remember this result better than the people who got it right. That is the mechanism, working on you.

So: the more you get wrong, the more you learn. And smooth learning, with nothing to get wrong, often teaches nothing.

03When AI is rightThis slide ↗

AI is most dangerous when it’s right.

Now look at tutorials and AI through this lens.

  1. A polished lectureYou watch someone else think.
  2. A tutorialSomeone already made every mistake for you.
  3. An AI answerIt’s right before you’re even stuck.

SmootherLess chance to be wrong →

A tutorial is polished smoothness. Someone has already tried every step and filled every hole. You follow along, it works the first time, and it feels fast. But you never had a chance to be wrong. I once put it this way: what a tutorial teaches you keeps getting cheaper; what it lets you skip is the most valuable part.

AI is smoothness taken to the limit. The answer is on the screen before you have started to think. The code runs before you are stuck.

So the most dangerous moment is not when AI is wrong. It is when AI is right, and you think you have learned it too. That feeling of “I got it” makes you stop. You don’t try it again yourself. You don’t change the problem. The exact place where learning needed to happen gets skipped.

Here is what that looks like when someone measures it.

03When AI is rightThis slide ↗

The practice got better. The learning got worse.

A randomized experiment with nearly a thousand high-school math students in Turkey.⁠4 For several practice sessions, one group could ask GPT-4 anything.

Students who could ask GPT-4 for answers, compared with students who had no AI

Their practice scores went up 48 percent. Then came the exam, with no AI for anyone. The students who had practiced with GPT-4 scored 17 percent lower than students who never had it.

Better practice. Worse learning. In the same students, over the same weeks. And nobody could see it until the AI was taken away.

The writer Ted Chiang has the best image for this.⁠5 Using ChatGPT to write your essays is like bringing a forklift into the weight room. The weights move. The muscle doesn’t grow. And if you are learning AI, be careful: the tool you are learning with is exactly that forklift.

This is not only a school problem.

04Fake workThis slide ↗

Fake learning and fake work fool the same thing: your sense of progress.

I learned this at work, not in school.

At Tencent I led a team, and my main job was relaying: translating my boss’s strategy into actions for the team, then reporting the team’s results back up. I was good at it. My boss liked me, my team liked me, and I was comfortable. But I kept asking myself: what progress am I actually making? What value am I creating?

Yuzheng Sun, in a white T-shirt, arms crossed
Fake workVisible activity standing in for value that’s hard to measure.
Fake learningVisible results standing in for growth that’s hard to measure.

I call this fake work: visible activity standing in for value that is hard to measure. It is structural. When value is hard to measure, organizations watch progress reports, meetings and decks, so people start producing those. And the worst part is the feeling. A whole day of meetings, you are exhausted, and your brain tells you it was a productive day.

Fake learning and fake work fool the same thing: your sense of progress.

And that illusion is trained. In school, teachers and grades tell you what good is. At work, your boss and your KPIs do. After more than a decade of this, what we are best at is satisfying whoever is grading us. We were trained to be insensitive to value.

So people who come out of fake learning drift naturally into fake work. Not because they don’t try. It is the only kind of trying they were trained for.

05New gradesThis slide ↗

A new tool feels like progress. It expires in months.

In school, the scores were grades. For people learning AI, there are new scores: how many tools you have learned, and how many prototypes you have built.

Start with tools. Learn a new tool, set up an automation, finish a tutorial: the progress is visible. You can screenshot it, post it, put it on your résumé. But AI tools turn over every few months. Half a year later, many of the buttons and commands you learned are gone.

What you collecta new tool, an automation, a finished tutorialWhat stayshabits, methods, ways of thinking
Feels like progressRight awayHardly at all
You can show itScreenshot, post, résuméYou can’t
Half a year laterMany buttons are goneStill there
The tool feels most like progress, and expires fastest. The method feels like nothing, and stays with you.

What stays is invisible: habits, methods, ways of thinking. You split a fuzzy task into steps you can check. When the result is wrong, you look for the cause instead of switching tools and trying again. You cannot screenshot that. But change the tool, wait a year, change the problem, and it is still there. It passes the test we started with.

I have trained thousands of people at tech companies, and the pattern is consistent. With the same model, some people decide AI is unreliable, and others turn it into a dependable work system. The difference is not the tool. It is the method.

So why does everyone collect tools? A tool has a moment when you have “learned it.” A method has no such moment; it grows while you work. The tool feels like the most progress and expires fastest. The method feels like no progress and stays with you. Backwards again.

05New gradesThis slide ↗

A demo only has to be right at the demo. A product has to stay right.

Now prototypes. With a coding agent, you can build a convincing app in an afternoon. But a prototype is best at the smooth case: clear requirements, simple data, and one user, which is you.

A prototype · one afternoon with an agent

Clear requirements. Simple data. One user: you.

A product · other people

Inputs you never imagined. Data that breaks. And you’re not in the room.

A product faces other people. They type things you never imagined. The data breaks. The requirements change next month. And when they use it, you are not in the room. As I put it in an essay called “Don’t build”: a demo only has to be right at the moment you show it; a product has to stay right while reality keeps changing.

Every step from prototype to product is your idea hitting reality. You thought users would do this; they did that. Many of the right decisions simply do not exist until you have seen the wrong version. Every collision is a lesson.

That is why a prototype feels like such fast progress: it skips exactly the part where you would have learned.

So in the AI era, real learning lives on two roads: from a tool to a method, and from a prototype to a product.

06SpotterThis slide ↗

Same AI. Different design. Different learning.

So should you use less AI? No. Remember the math study. There was a third group.

A prediction first. Same GPT-4, but it only gives hints, never answers. Does the exam loss…

Two designs of GPT-4, compared with students who had no AI

See the numbers
GPT-4 as forkliftGPT-4 as spotter
How it helpedAnswers on requestHints, not answers
Practice (with the help)+48%+127%
Exam (no AI)−17%No significant loss

Bastani et al., PNAS, 2025.⁠4

Pick an answer above, or .

Teachers gave that GPT-4 the solutions and the common mistakes, and told it to give hints, not answers. On practice, that group improved even more: 127 percent. And on the exam, the loss essentially disappeared.

Same AI. Different design. The difference is whether students still got a chance to think first, and to be wrong.

That is the difference between a forklift and a spotter. A forklift lifts for you. A spotter stands behind you, lets you do the lift, and grabs the bar only when you are about to fail.

06SpotterThis slide ↗

Where it counts, choose the hard path.

Using AI as a spotter means choosing the hard path, at the moments that count.

Not all difficulty is worth it. Yan Wang, who spoke with me that day, calls things like configuring environments, requesting API tokens and debugging ports fake difficulty. They burn willpower, they give you the feeling of working hard, and they don’t make you any wiser. Hand those to AI without guilt. Keep the real difficulty: is this worth doing, how should it be broken down, and is the result actually good?

Fake difficulty · hand it to AI

Setting up environments, tokens, ports. Burns willpower. Grows nothing.

Real difficulty · keep it

Is this worth doing? How should it be split? Is the result actually good?

Then three moves

  1. Guess first. Before the agent runs, write down how you would do it and where you expect it to break. Then look at what it did. Being wrong is where the learning is.
  2. Take the help away. A day or two later, close the AI and do it again, or explain it to someone. Wherever you get stuck is what you haven’t learned yet.
  3. Change the problem. New data, a new constraint, a different user. See if you can still do it.

The first move gives you the chance to be wrong. The other two test whether you learned: without the help, later, on a new problem.

It will feel slower and harder. That is not falling behind. That is what learning feels like from the inside.

07What proves youThis slide ↗

When anyone can make beautiful work, the work stops proving anything.

Now the economics. Michael Spence won a Nobel prize for an idea you already understand: a signal only works when it is hard to fake.⁠6

What you can showUsed to signalNow costs
A polished essayYou can write.A prompt.
Code that passes the testsYou can code.An agent run.
A GitHub full of projectsYou can build.A weekend.

What will still prove you is what you can do once the help is gone. Expect interviewers to stop asking what you built, and start asking why you built it that way, what broke, and what you would change.

07What proves youThis slide ↗

What ends is a misunderstanding.

So what ends? Not grades, certificates or tutorials. They will stay, and they are still useful. What ends is a misunderstanding: mistaking “I made it” for “I learned it,” and mistaking smooth for progress.

For anyone who actually wants to learn, this is good news. A spotter who is always available, endlessly patient, and able to see where your thinking went wrong used to be almost impossible to find. Now it is on your laptop. The only question is whether you let it lift for you, or have it watch you lift.

You may have noticed that twice on this page, I asked you to guess before showing you the result. The guesses that were wrong are the ones you will remember.

Something you learned with AI this month: close the AI, change the problem. Can you still do it?

Learn real skills. Make real things.

Notes

  1. Soderstrom & Bjork, “Learning Versus Performance: An Integrative Review,” Perspectives on Psychological Science, 2015.
  2. Deslauriers et al., “Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom,” PNAS, 2019. Randomized, Harvard introductory physics.
  3. Kornell, Hays & Bjork, “Unsuccessful retrieval attempts enhance subsequent learning,” Journal of Experimental Psychology, 2009.
  4. Bastani et al., “Generative AI without guardrails can harm learning,” PNAS, 2025. Randomized, about 1,000 high-school math students in Turkey; changes are relative to students without AI. The tutor version used teacher-supplied solutions and common mistakes.
  5. Ted Chiang, “Why A.I. Isn’t Going to Make Art,” The New Yorker, 2024.
  6. Michael Spence, Nobel Prize in Economic Sciences, 2001, for the analysis of markets with asymmetric information (job-market signaling).

Further reading

Essays of mine the talk draws on, on Superlinear Academy.

About this talk

A guest session for Columbia University’s COMS W4995 Agentic Engineering, October 2, 2026. This page is the talk written out for reading; the slides are here, and there is a 中文版.

Still have a question?

Ask Lizheng answers from what I have said and written in public, with a source for every answer.

Ask Lizheng →