← All articles

Mikhail Polyakov

Ten minutes with AI, and people skip hard problems more often. Part 2: method, statistics and weak points of the study

Level: AdvancedReading time: 15 min

Topics: AI, Science, Education

Two glass bowls holding different numbers of pebbles, brass scales between them, a magnifying glass and a stack of blank cards on a wooden table

This is the second part. A plain retelling of the study, its main numbers and takeaways for work and learning are in the first part. Here I go through what those numbers rest on.

Short answer: in each of the three experiments, people were randomly split into a group with an AI assistant and a control group, and then two shares on the last three problems were compared: the share of correct solutions and the share of skips. The comparison was done with a t-test, and the size of the difference was measured with Cohen's d with a 95 percent confidence interval. The difference in the share of correct solutions is significant in all three experiments, ranging in size from 0.19 to 0.42 standard deviations. The difference in skips is significant in the first and third, but not in the second, the largest and best protected against distortions. By my estimate, the second experiment could reliably detect a difference in skips of roughly 0.23 standard deviations or more, while the observed one was 0.10, so the lack of significance does not prove the absence of an effect. The split of participants by how they used the AI rests on their own answers and does not prove cause, and some of the pairwise comparisons in it would not pass a strict correction for the number of comparisons.

The three experiments side by side

All numbers below are from the paper "AI Assistance Reduces Persistence and Hurts Independent Performance" (arXiv:2604.04721, version 5 of October 3, 2026). Participants in all three experiments are from the US, recruited through the online research platform Prolific.

Experiment 1 Experiment 2 Experiment 3
Problems fractions fractions reading comprehension, questions from open practice materials for the SAT
Recruited: with AI / control 191 / 163 349 / 318 104 / 97
Pay and duration $2.60, about 13 minutes $3.40, about 15 minutes $3.40, about 13 minutes
Pretest before AI none 3 one-step fraction problems 1 simple question
Problems with the assistant 12 11 5
Problems at the end without help 3 3 3
What the control group had in the side panel nothing solutions to the pretest problems, already shown earlier general tips on how to take tests
Remaining after exclusions: with AI / control 185 / 122 308 / 277 85 / 83

The assistant was always the GPT-5 model, which had been given each problem and its solution in advance. Wrong answers were not penalized, and pay did not depend on the number solved. After a wrong answer, the correct solution was shown on screen, in both groups. Everyone had a "skip" button. In the third experiment, an answer given in less than 5 seconds was also counted as a skip: according to the authors, the text cannot be read in that time.

What exactly was measured

For each participant, the authors took the last three problems, identical in both groups and solved without help, and calculated two shares: the share solved correctly and the share skipped. A detail follows from this that the paper does not mention separately: for each person, the share can only be one of four values, 0, 1/3, 2/3 or 1. Group averages such as 0.57 come from averaging these steps over hundreds of people.

Next, the groups were compared with a t-test for two independent samples. Its idea: the difference in means is divided by how much that difference could jump around by chance.

t = (x̄₁ − x̄₂) / (sₚ · √(1/n₁ + 1/n₂)), where sₚ = √(((n₁ − 1)·s₁² + (n₂ − 1)·s₂²) / (n₁ + n₂ − 2))

In words: x̄₁ and x̄₂ are the average shares in the two groups, s₁ and s₂ are the standard deviations (how much people within a group differ from each other), n₁ and n₂ are the group sizes, and sₚ is their pooled spread. The more people, the smaller the denominator and the more confidently a real difference can be told apart from a random one. In brackets after t the authors give the degrees of freedom, and everywhere it equals n₁ + n₂ − 2: in the first experiment 185 + 122 − 2 = 305. This corresponds to the ordinary Student's t-test with pooled spread; which exact variant of the test was used is not named in the text, this is my inference from the numbers.

The p-value answers the question: if there were actually no difference between the groups, how often would chance produce a difference at least as large as the observed one. The threshold in the paper is the usual one, 0.05.

But p does not say whether the difference is large. For that the authors give Cohen's d:

d = (x̄₁ − x̄₂) / sₚ

In words: how many standard deviations one group is, on average, away from the other. Let me check this on the first experiment using the numbers from the paper. The share of correct solutions is 0.57 with a deviation of 0.41 in the AI group and 0.73 with a deviation of 0.34 in the control group. The pooled spread:

sₚ = √((184 · 0.41² + 121 · 0.34²) / 305) ≈ 0.38

Then d ≈ (0.57 − 0.73) / 0.38 ≈ −0.42, exactly as in the paper. And t = d / √(1/185 + 1/122) ≈ −3.6 versus −3.64 in the paper: the small discrepancy comes from rounding the means to two digits.

A 95 percent confidence interval for d is built by a rule that, in 95% of repetitions of such an experiment, gives an interval covering the true size of the effect. Put simply: it is the range of effect values the data are consistent with. For one specific interval, you cannot say that the true value lies in it with 95% probability. An approximate formula for the standard error:

SE(d) ≈ √((n₁ + n₂) / (n₁ · n₂) + d² / (2 · (n₁ + n₂)))

For the first experiment SE ≈ 0.118, and the interval d ± 1.96 · SE gives roughly −0.65 to −0.19. In the paper it is −0.66 to −0.19, so the numbers agree up to rounding. If the interval does not cross zero, the difference is, as a rule, significant at the 0.05 level.

Results with full statistics

Shares, group averages, standard deviation in brackets.

Experiment and measure With AI Control t p d [95% interval]
1, correct solutions 0.57 (0.41) 0.73 (0.34) −3.64 < 0.001 −0.42 [−0.66; −0.19]
1, skips 0.20 (0.36) 0.11 (0.29) 2.16 0.031 0.25 [0.02; 0.48]
2, correct solutions 0.71 (0.36) 0.77 (0.32) −2.33 0.020 −0.19 [−0.36; −0.03]
2, skips 0.10 (0.26) 0.07 (0.23) 1.18 0.239 0.10 [−0.07; 0.26]
3, correct solutions 0.76 (0.34) 0.89 (0.25) −2.72 0.007 −0.42 [−0.73; −0.11]
3, skips 0.08 (0.21) 0.01 (0.08) 2.69 0.008 0.42 [0.11; 0.72]

Two observations. First: the effect on correct solutions in the second experiment is half the size of that in the first (−0.19 versus −0.42), although the problems are the same. Second: the intervals are wide. In the third experiment, the skip data are consistent with an effect of 0.11 and with an effect of 0.72 alike, which means they tell a weak effect from a strong one poorly.

Why a second experiment was needed

In the first experiment, exclusions were based on problems within the experiment: those who solved fewer than 3 of the 12 learning problems were excluded. The authors themselves call this a possible distortion. A weak participant in the AI group could get correct answers from the assistant and stay in the sample, while an equally weak participant in the control group dropped out. From the numbers in the table above, the AI group lost 6 people out of 191 and the control group 41 out of 163, this is my calculation, about 3% versus 25%. The authors write that the different exclusion rates could have inflated the gap.

The second experiment corrected this with two changes:

  1. A pretest before any AI. Three one-step problems; those who solved none were excluded. Exclusion rates became almost equal: 11.7% in the AI group and 12.9% in the control group.
  2. The same interface change. In the first experiment, the AI group's assistant panel disappeared before the test, while nothing changed for the control group. In the second, the control group also saw a side panel, with the already shown solutions to the pretest problems, and it also disappeared before the test. So the sudden disappearance of the panel no longer distinguished one group from the other.

It is in this better protected experiment that the effect on correct solutions shrank to −0.19 and the effect on skips became non-significant. The authors explain the lack of significance by people using the AI in different ways.

Why 0.10 may not have reached significance

There is also a simple statistical explanation, this is my calculation. The standard error of d in the second experiment with 308 and 277 participants:

SE ≈ √(1/308 + 1/277) ≈ 0.083

To detect an effect with 80% probability at a two-sided threshold of 0.05, an effect of roughly (1.96 + 0.84) · SE ≈ 0.23 standard deviations is needed. Here 1.96 corresponds to the 0.05 threshold and 0.84 to the required 80% probability of detection. The observed value was 0.10, half that. So such a sample could not reliably catch an effect of size 0.10, and the lack of significance here does not prove there is no effect. But the data do not support the conclusion "the effect is small" either: the interval from −0.07 to 0.26 is consistent both with no effect and with an effect the size of the first experiment's (0.25).

Breakdown by how people used the AI

At the end of the second experiment, participants in the AI group were asked how they had used the assistant. Groups: ready-made answers (189 people), hints and explanations (82), did not use it (37). Those who said they had not used it were checked against the chat logs: 83.8% of them did not message the assistant at all, and 94.1% sent fewer than three messages.

First, the authors applied a one-way analysis of variance (ANOVA) to four groups: three by usage style plus control. It tests whether the means differ for at least some pair of groups. The F statistic compares the spread between groups with the spread within groups:

F = (spread of means between groups / (k − 1)) / (spread within groups / (N − k))

Here k = 4 groups and N = 585 participants, hence the degrees of freedom (3, 581). The value η² shows what share of the total spread is explained by group membership.

What was compared F(3, 581) p η²
Correct solutions on the pretest 1.21 0.306 0.006
Skips on the pretest 0.56 0.639 0.003
Correct solutions on the test 7.89 < 0.001 0.039
Skips on the test 2.80 0.039 0.014

Before the AI, the analysis found no statistically significant differences between the groups on these two measures; after working with AI, it did. The absence of significant differences does not prove that the groups were equal: it only means that this analysis could not detect a difference. And even the significant difference in correct solutions explains 3.9% of the spread: the rest comes from differences between people within the groups.

Then came pairwise t-tests. Those who took ready-made answers (share correct 0.65) versus control (0.77): p < 0.001, d = −0.36. Versus those who took hints (0.76): p = 0.031, d = −0.29. Versus those who did not use it (0.89): p < 0.001, d = −0.66. On skips, those who took answers (0.13) versus control (0.07): p = 0.033, d = 0.20; versus those who took hints (0.05): p = 0.019, d = −0.31 (the sign of d in the paper depends on the order of comparison).

That is five pairwise comparisons on the test results. The section does not mention a correction for multiple comparisons. The more comparisons, the higher the chance of getting at least one "significant" result by chance. The simplest conservative correction, Bonferroni, divides the threshold by the number of comparisons: 0.05 / 5 = 0.01. With that threshold, the comparisons with p < 0.001 would pass, while those with p = 0.031, 0.033 and 0.019 would not. This is my recalculation, not the authors' conclusion. Even closer to the threshold is the analysis of change from pretest to test: the increase in the share of skips among those who took answers versus control gave p = 0.051, and versus those who took hints p = 0.047. The authors themselves call the first value marginal.

And the main limitation of this section, which the authors state directly: the data are comparative, the groups formed on their own, so the conclusions are not necessarily causal. For example, a person who got tired or lost interest could both take answers from the AI and then skip problems. Then both facts are explained by tiredness, not by the AI. The fact that no significant differences were found on the pretest does not confirm that the groups were comparable: three one-step problems measure far from everything, and the groups could differ in things that were not measured at all.

Checking for exclusion effects

In Appendix C.4, the authors checked whether the results depend on the exclusion rules.

  • Experiment 2, strong participants only. They kept those who solved all three pretest problems: 143 in the AI group and 140 in the control group. The gap in correct solutions remained, 0.78 versus 0.85 (p = 0.03). Skips 0.07 versus 0.05, not significant (p = 0.47).
  • Experiment 3, equalized groups. The two participants with the worst test results were removed from the AI group so that there were 83 people in each. Correct solutions 0.78 versus 0.89 (p = 0.018), skips 0.06 versus 0.01 (p = 0.019).

Removing the weakest from the AI group works against one's own hypothesis, so an effect that survives such a check looks more convincing. But these are two separate checks, not a systematic analysis of all possible exclusion rules.

Why persistence drops, in the authors' view

The authors propose two mechanisms, both as hypotheses that the experiment did not measure.

  1. A shifted reference point. When AI solves a problem in seconds, the sense of how long a problem should take changes. Working on your own starts to feel harder than before. The authors compare this to hedonic adaptation, getting used to a new level of comfort, and write that the mechanism reinforces itself: each task handed to the AI makes the next handoff more attractive.
  2. Loss of self-knowledge. Without working on their own, a person does not find out what they are capable of and is worse at judging whether to keep trying.

From this the authors conclude that hints instead of answers or limiting time with AI will help only at the margins. The real solution, in their view, is for AI systems themselves to be designed with people's long-term skills in mind, not only their satisfaction in the moment.

What to compare this work with

According to the review in the paper itself, earlier evidence of skill loss from AI was mostly correlational, from surveys and interviews, or based on small samples. Of the works the authors classify as randomized controlled trials, they name two. In the study by Bastani and co-authors from 2025, GPT-4 help during math practice worsened high school students' later results without help, while a version with an AI tutor that guides toward the solution partially softened the harm. In the work by Barcaui from 2025, a lasting drop in results was observed in undergraduates after a course taken with AI.

What is new in this work, according to the authors, is two things: the effect is shown on participants outside an educational setting and on two different types of tasks, and in addition to accuracy, willingness to keep going is measured, through the share of skips.

Weak points worth keeping in mind

The authors' limitations: short exposure (about 10 minutes in the abstract, a 10-15 minute session in the introduction); testing right after the AI was removed; an assistant that gives a full answer on the first request. To these I would add what can be seen from the method:

  • A test of three problems. Each participant's share takes four values, which is a coarse scale. One random mistake changes a person's share by a third.
  • Skipping as a measure of persistence. It shows a decision not to enter an answer, but not whether there was an attempt before the button was pressed. In the third experiment, answers faster than 5 seconds were also counted as skips, so the measure there is broader than in the first two.
  • Online participants for small pay. The motivation of a person solving fractions for $2.60 may differ from that of an employee or a student for whom the result matters.
  • One model and one setup. GPT-5 with a solution known in advance. How people would behave with an AI that makes mistakes or answers more slowly, the experiment did not test.
  • Spread of the effect. For correct solutions, d ranges from −0.19 to −0.42 depending on the experiment; the intervals overlap, but the exact size of the effect is poorly known.

How to read studies like this

These are my rules, not data from the paper.

  1. Look at d and the interval, not only at p. A small p says that such a difference is hard to get by chance alone if there is no real difference. Whether the difference is large, it does not say.
    • Who does it: the person making a decision based on the study, for example a manager choosing rules for how newcomers learn with AI.
    • How to check: find the effect size and interval in the paper and write down whether the end of the interval closest to zero is consistent with "almost nothing."
  2. Separate random assignment from self-report. A comparison of groups into which people were split at random can speak to cause. A comparison of groups formed by participants' own answers speaks only to association.
    • Who does it: the same reader.
    • How to check: for each conclusion, find in the method how the groups were formed.
  3. Check exclusions. If the groups lost different shares of participants, the result may depend on who remained.
    • Who does it: the same reader.
    • How to check: compare how many people were recruited and how many remained in each group.
  4. Compare the study's conditions with your own task. Who took part, what tasks, how long it lasted, how soon the test followed. If your situation is different, for example employees work with AI for months on complex work tasks, the study does not answer your question directly.
    • Who does it: the person carrying the conclusion over to their team or class.
    • How to check: write down the four conditions of the study next to the conditions of your task and mark whether each one matches.
    • If they do not match: treat the conclusion as a hypothesis. One way to test it yourself: split people in advance by lot into those who learn with ready-made answers and those who learn with hints, and then compare results on the same test problems without AI. In the study, only access to AI was assigned by lot, while people chose their usage style themselves; assigning the usage style by lot is my suggestion. A comparison of groups that chose their style themselves will not show cause.

Summary

The main result of the study holds up under scrutiny: in all three randomized experiments, the AI group did worse after the AI disappeared, and the authors checked two distortions they named themselves: different exclusion rates and a different interface change. The conclusion about skips is weaker: in the second, best protected experiment it is not significant, and by my estimate the sample could not reliably detect an effect of size 0.10, so this is not proof that there is no effect. The split into "those who took answers" and "those who took hints" is interesting, but it rests on self-report and on several comparisons with p around 0.02 and 0.03, without correction for their number. The answer to the main question, what happens after months of working with AI, requires long-term experiments, which the authors themselves write about.

I work on AI agents and automation. If you want to look at my projects or discuss your own task, visit my portfolio.

Sources