The Generative AI Learning Penalty: Evidence from Chinese Secondary Education

Using 30 months of panel data on 26,811 Chinese students in grades 7-12, we study how generative AI affects homework productivity and learning. The data combine monthly closed-book exams, high-school and college entrance exams, and homework scores and completion time across nine subjects. We exploit staggered AI adoption in a difference-in-differences design. AI adoption raises homework scores by 18% and reduces completion time by 30%, but lowers monthly exam scores by 20% within six months. High-stakes entrance-exam scores fall by 18 and 24%, with the full penalty emerging only after about two years. The losses are largest in social science subjects, followed by STEM and languages, and are especially large for junior students, high-achieving students, and boys. The learning losses are concentrated among roughly 80% of AI users whose behavior is consistent with homework outsourcing, as indicated by exceptionally short homework completion time coupled with high homework scores. AI users who maintain similar homework completion time as non-AI users experience small learning losses.

Edit: moving my comment up here

Just in case as it’s formatted a bit weirdly

X-Axis: Homework scores

Y-Axis: Exam scores

  • RFKJrsBrainworm@sh.itjust.works
    link
    fedilink
    arrow-up
    0
    ·
    9 hours ago

    If you don’t think the wealthy are well aware of this, for them , put knowledge behind a paywall… I know I would have probably fallen for the ease of asking instead of researching and learning.

    The next generation has lots of hurdles

  • Pruning9165@feddit.org
    link
    fedilink
    arrow-up
    0
    ·
    20 hours ago

    Meanwhile in the country where they made this study I saw a cram school signage boasting AI teachers (of I understood correctly)

      • YiddishMcSquidish@lemmy.today
        link
        fedilink
        English
        arrow-up
        0
        ·
        21 hours ago

        The axis still makes little sense. Why would they plot scores relative to scores? Thanks for checking out the article, my frustration isn’t with you it’s with their visualization choices.

        • Couldbealeotard@lemmy.world
          link
          fedilink
          English
          arrow-up
          0
          ·
          19 hours ago

          It’s comparing homework scores against exam scores

          The kids who scored highly on homework because of AI did poorly in exams. This shows that AI use is just doing the homework for the kids and the kids aren’t learning.

        • Alaknár@sopuli.xyz
          link
          fedilink
          arrow-up
          0
          ·
          19 hours ago

          Why would they plot scores relative to scores?

          You can use AI when doing homework (score is high), but you can’t during an exam (score plummets).

          What it shows is: the higher your homework score, the higher your exam score in non-AI using students, but with those using AI, that relation inverts - meaning that those who use AI to get stupid high scores in homework, just stop using their own brains and learning things.

  • melsaskca@lemmy.ca
    link
    fedilink
    arrow-up
    0
    ·
    1 day ago

    Yet another example that “tech bros” are selling snake oil to an unsuspecting customer base.

  • wheezy@lemmy.ml
    link
    fedilink
    English
    arrow-up
    0
    ·
    edit-2
    2 days ago

    Given OPs axis definitions.

    The max homework score for nonai is 115. But there are AI users scoring 115+? I’m confused why there are no better scores above 115 for nonai. Is that data insignificant? Are students with scores higher than 115 just being flagged as AI? It makes the data look unreliable.

    I’d also really like to know if the data range that isn’t comparable 115+ is at all meaningful data. Because that’s where there is a major dropoff. There is obviously a dropoff of significant before 115. But, just from looking at how the data is being presented with this massive dropoff but with zero data for nonai.

    I’d be interested in knowing what fraction of the total population is actually being represented in the 115+ dropoff. The way it’s presented it could literally be like 10 students.

    This is not a defense of AI use. But more a criticism of how the data is being presented.

    Also, Any student could be using AI as a resource similar to how I would use solutions manuals or previous tests to study back in my education. The good students aren’t gonna copy it verbatim and get flagged. Their going to get the right answer, and use it to learn the process so they can present their own work and actually learn the material.

    AI is trash. But this is nothing different than before AI when students would copy the solutions manuals blindly or their peers work. Those students have always existed. AI just makes it slightly more accessible. But, really only slightly. And, honestly, Chegg was just as easy to copy back in the day.

    Edit: This graph isn’t in the paper that OP linked. So, probably why it’s presented so badly. I’m assuming it’s from some click bait article then. But the paper clarifies that this is an actual survey of students. So it’s not just from flagged AI homework.

    The paper itself in section 5 supports what I hypothesized above. It’s not really about AI making students dumb. It’s about making it easier for students that already don’t want to learn to finish the homework. Same as the people that would copy solutions manuals.

    Quote from section 5.

    In this section, we show that the majority of AI students exhibit behavior consistent with homework outsourcing, spending little time completing homework assignments and experiencing learning losses. The remaining AI students spend as much time completing homework as non-AI students, receive higher homework scores, and have similar exam scores. In other words, they are learning as efficiently as the non-AI students.

    The paper doesn’t argue it from what I read. But I’d also argue their is some bias in the survey. The students scoring well in both exams and homework are less likely to consider their use of AI as meaningful enough to answer “used AI for homework”. It’s a problem with the survey. The survey is asking the students that actually learned the material to attribute their homework to AI alone in the same way the “copy and paste” AI users would.

    The survey would probably benefit from having No AI, Some AI, All AI as it’s responses. Even in anonymous surveys people make these own interpretations of the questions in their head. And a student that used AI as a resource to learn is very likely to just choose “No AI” when presented with questions of how they got their final answers to homework. Because in their head they are thinking of the students that copy and pasted AI responses and think “I’m not like that”.

    It’s likely why the double high scoring AI user sample is so low (paper says this). The survey is not allowing the response for this set of users to categorize themselves as using AI without feeling like they are like the copy and paste students. So those people are likely just self categorizing as “no ai” because the survey doesn’t allow them to distance themselves from the other population. Surveys are hard to write. Even ones where the users know it’s anonymous. Good students that use AI will see themselves as good students and the copy pasters as bad students. Human emotion plays a role and most people will not self categorize themselves further negatively than they feel they should be. They are more likely to select the imperfect category that is more positive.

    Bad students don’t care though. They’ll admit to AI use. They already don’t care enough to learn the material. They have no reason to lie. So they’ll select “used AI”.

    • AlfalFaFail@lemmy.ml
      link
      fedilink
      arrow-up
      0
      ·
      21 hours ago

      The max homework score

      The values aren’t a score. They normalized the average score to 100 for non-AI in both homework and exam. Then noted the deviation. People using AI to get very high scores (130% of the average) were tank the test (40% of the average) six months later.

  • yogurtwrong@lemmy.world
    link
    fedilink
    arrow-up
    0
    ·
    2 days ago

    Homeworks are useless, school bad, Einstein had bad grades (no he didn’t) crowd is very silent right now.

  • Not_mikey@lemmy.dbzer0.com
    link
    fedilink
    arrow-up
    0
    ·
    2 days ago

    Looks like millennials / elder Gen z are going to be the most educated generation in history. Older generations were taught when they knew less about the world and education, younger generations are screwed both by the pandemic and now AI.

    So Idiocracy was sort of right, only it’s not the eugenics “dumb people have more kids” reason they gave, it’s just us offloading our thinking to machines.

    • alphabethunter@lemmy.world
      link
      fedilink
      arrow-up
      0
      ·
      2 days ago

      EDIT: nevermind, I misread. I thought you were complaining about the one you posted… However, the image used by op is still a fairly obvious graph.

      Edit 2: the axis of the graph posted by op are also clearly defined, vertical is exam scores, horizontal is homework. The definition shows above and below the graph itself.

      It seems fairly obvious to me. It shows the clear mismatch between homework scores and test scores of AI users, where higher homework scores using AI correlate to lower test scores, clearly indicating that there is no learning, when compared to non-ai users, where higher homework scores directly translate to higher test scores as well.

      • Viceversa@lemmy.world
        link
        fedilink
        arrow-up
        0
        ·
        22 hours ago

        Y label is bold
        has the prefix “China*,”
        has additional row of text between the label and the diagram.
        Not even remotely near Y axis.

        X label is regular
        doesn’t have the prefix “China*,”
        but has the suffix “, average without Al=100”

        How is it clearly defined again?

      • brucethemoose@lemmy.world
        link
        fedilink
        arrow-up
        0
        ·
        2 days ago

        Yeah I get it.

        Looking at the OP graph again, I get it now, but initially I found the top label to be confusing. I thought it was part of the title, and that the Y axis was unlabeled.

      • brucethemoose@lemmy.world
        link
        fedilink
        arrow-up
        0
        ·
        2 days ago

        Side tangent, but Lemmy is the worst with sources.

        If I were admin, I’d make all users, new and extant, take an 8-minute course on clicking through reposts to finding original sources before they are allowed to submit posts.

        That’s my rant.

        • kingofras@lemmy.world
          link
          fedilink
          arrow-up
          0
          ·
          2 days ago

          I don’t think you can prevent this from being posted, though I agree that resource should be available. It’s just that our attention spans have evaporated, everything must be instant and with LLMs it’s only accelerating.

          This is about having a brief second of critical thinking and then not upvote or downvote the low effort post and upvote the first critical thinking comment.

          Another thing on Lemmy is that we have half a dozen “users” who don’t care about the content, they just automatically post high engagement content from other platforms and post it here (without declaring themselves as a bot account too). And the mindless scrollers (at least 60-70%) going by this post, just take the bait.

          Not having karma didn’t help much in hindsight.

    • JapaneseJudas@fedinsfw.app
      link
      fedilink
      English
      arrow-up
      0
      ·
      1 day ago

      I’ve seen the term “cognitive surrender” being thrown around for this and I think they’re two very nice terms to describe what’s happening to our society right now

  • queermunist she/her@lemmy.ml
    link
    fedilink
    arrow-up
    0
    ·
    2 days ago

    My takeaway: letting AI do your homework for you is essentially the same as letting AI study for you.

    Kind of obvious, but the data is still good to have.