StoryScope: Investigating idiosyncrasies in AI fiction
arxiv.org
As AI-generated fiction becomes increasingly prevalent, questions of authorship and originality are becoming central to how written work is evaluated. While most existing work in this space focuses on identifying surface-level signatures of AI writing, we ask instead whether AI-generated stories can be distinguished from human ones without relying on stylistic signals, focusing on discourse-level narrative choices such as character agency and chronological discontinuity. We propose StoryScope, a pipeline that automatically induces a fine-grained, interpretable feature space of discourse-level narrative features across 10 dimensions. We apply StoryScope to a parallel corpus of 10,272 writing prompts, each written by a human author and five LLMs, yielding 61,608 stories, each ~5,000 words, and 304 extracted features per story. Narrative features alone achieve 93.2% macro-F1 for human vs. AI detection and 68.4% macro-F1 for six-way authorship attribution, retaining over 97% of the performance of models that include stylistic cues. A compact set of 30 core narrative features captures much of this signal: AI stories over-explain themes and favor tidy, single-track plots while human stories frame protagonist' choices as more morally ambiguous and have increased temporal complexity. Per-model fingerprint features enable six-way attribution: for example, Claude produces notably flat event escalation, GPT over-indexes on dream sequences, and Gemini defaults to external character description. We find that AI-generated stories cluster in a shared region of narrative space, while human-authored stories exhibit greater diversity. More broadly, these results suggest that differences in underlying narrative construction, not just writing style, can be used to separate human-written original works from AI-generated fiction.

Abstract page for arXiv paper 2604.03136: StoryScope: Investigating idiosyncrasies in AI fiction

  • AnAmericanPotato@programming.devEnglish
    84·
    4 months ago

    93.2% macro-F1 for human vs. AI detection and 68.4% macro-F1 for six-way authorship attribution

    This isn’t what I’d call “reliable”.

    I’m also not impressed with their methodology, which is heavily based on Gemini.

    To generate mirrored AI stories, we reverse-engineer writing prompts from each human story by prompting Gemini 2.5 Flash (Gemini Team, 2025a) to infer the underlying premise

    That’s too many steps removed from anything you’d encounter in the wild. They’re not even testing against human-prompted output.

    And then they use Gemini again to analyze all the stories. Relying on proprietary cloud models for the core of your analysis is like building on sand.

      • Kevin@programming.devEnglish
        46·
        4 months ago

        Percentages are somewhat misleading here. That’s around 2 in 25, enough to falsely accuse a student or 2 in a fairly standard classroom for example.

        I remember somebody mentioning a quote along the lines of “A 95% success rate sounds good, until you become the 1 in 20 that it kills”

          • Holytimes@sh.itjust.worksEnglish
            10·
            4 months ago

            Getting your degree or being kicked out and banned from your college and still requires to pay back the loans. IS life or death.

            It literally can determine the quality of life of a person for the next 30-40 years. If not forever. This is NOT something to take lightly.

              • doben@lemmy.mlEnglish
                7·
                4 months ago

                Holy the ignorant self-centered anecdote. Too long, no relevance, do not recommend. I‘d probably give it a failing grade.

              • luciferofastora@feddit.orgEnglish
                1·
                4 months ago

                Okay so what I’m taking away from this is that you were lucky to find gainful employment without proof of a degree and didn’t have any debt to pay off? Because the context here is people having all the debt and none of the credentials required for jobs where they make enough to pay off that debt. If your card is declined, how you feel about it doesn’t matter.

      • Damarus@feddit.orgEnglish
        16·
        4 months ago

        That’s not good enough. Just a thought experiment: Only 7% of humans died because their work got flagged incorrectly

        • Allah@piefed.worldOPEnglish
          2·
          4 months ago

          this is just spherical cow fallacy combined with purity culture slop

          no methodology is perfect, it just needs to be practically valid, using ai != killing people for using ai

          unless you are for killing people who use AI, which wouldn’t surprise me one bit about Anti-AI horde

          • darkmarx@lemmy.worldEnglish
            28·
            4 months ago

            Tell that to the student who gets expelled because they’re writing was falsely identified as AI and then takes their own life.

            Tell that to the person falsely identified with facial recognition and is arrested and loses their job.

            Using AI to detect AI is as bad as using AI for any important decision.

            This isn’t hypothetical. It happens. People will lose their livelihood or die because of it. 7 in 100 is not accurate enough when any penalty is applied to detection.

  • Exec@pawb.socialEnglish
    23·
    4 months ago

    I wonder how does this stand to writers who have learnt English as a second language

  • palordrolap@fedia.io
    17·
    4 months ago

    … which will be fed into the next generation of LLMs in order to stop sounding quite so much like LLMs.

  • sunbeam60@feddit.ukEnglish
    12·
    4 months ago

    Here’s the thing: Read enough AI output and everyone can spot it. And that’s the seam that runs through it. The answer is partly right and it’s important to be honest about it. That’s the load bearing statement. It’s this. But it’s also that. And that’s doing a lot of the work here.

    It’s. So. Staccato. And it sounds like tech bro in a Patagonia gilet.

  • Wataba@sh.itjust.worksEnglish
    9·
    4 months ago

    Find me an option that works like GDocs keeping a running history of writing and editing, and I’ll consider it.

    I have receipts for the integrity of my writing. I dont need some shitty tool to validate it.

      • LastYearsIrritant@sopuli.xyzEnglish
        5·
        4 months ago

        It tracks the edits. If you generate a whole doc in one shot then it’s obviously AI.

        If you have a shitty draft, then a slightly less shitty draft, then… … … Then a quality end product, you can see how the human worked.

        AI can’t really replicate that.

        • ITGuyLevi@programming.devEnglish
          3·
          4 months ago

          I only ever wrote first drafts when they were forced upon me in school. There might be some edits, but my first draft is largely what I would submit because I hated to write as a kid and didn’t want to write the same thing 3-5 times by hand before submitting it. A small benefit from being around before computers were so common was learning how to write a paragraph ahead in my mind, it would help me catch anything I was about to forget to include, or to remove a trailing thought before I wrote it out.

          • TrackinDaKraken@lemmy.worldEnglish
            4·
            4 months ago

            I hated writing by hand, too.

            I had to write more starting in junior high (middle school). This was back in the late 70s. We had an old, non-electric typewriter at home, so, I took typing in seventh grade. For final edit, I’d cut the sheets up and use white out, and then take the taped up mess to a store with a copy machine. That’s how I got through high school English classes, too.

            Later, when I got into graphic design as a profession, I learned this was common practice, and called “paste up”.

          • bitfucker@programming.devEnglish
            3·
            4 months ago

            That’s not how story writing works. Or any long form written art works for that matter. Try to ask any writer if they ever “write in their head” and publish those as a book as-is

            • ITGuyLevi@programming.devEnglish
              3·
              4 months ago

              The biggest thing I “wrote in my head” was my capstone for my Bachelor’s, I managed a 93 and that is more than high enough for me. I’m not an author though so I doubt I could do a book.

    • commonmarmoset@reddthat.comEnglish
      2·
      4 months ago

      I love that docs receipt feature and agree with you. Great call.

      Forgive me being well-meaningly obnoxious, but make sure to get receipts outside of the Google ecosystem too. As a professional writer/editor myself, I’ve heard horror stories about folks losing access to an account, and therefore the receipts.

      I am also skeptical of these kinds of tools.

      • Holytimes@sh.itjust.worksEnglish
        2·
        4 months ago

        Never trust any cloud storage that you do not personally have access to cold back ups of.

        This is data storage 101!

    • zqps@sh.itjust.worksEnglish
      1·
      4 months ago

      Some moron is gonna vibe code a script that spoofs this on the surface. Yea it’ll be detectable, but not detected in most cases.

    • sbbq@lemmy.zipEnglish
      39·
      4 months ago

      I can’t even imagine what dickhead is saying there. Is he implying that the mean real people are bullying the sensitive AI babies?

      • AbouBenAdhem@lemmy.worldEnglish
        26·
        4 months ago

        Since Ben Garrison is the one human whose output most closely resembles AI, I imagine this is him having an existential crisis.

        • davidgro@lemmy.worldEnglish
          20·
          4 months ago

          You’re saying an actual human called that thing both “pen & ink” and a “typewrite”?

          … Also labeled the clearly working lamp as “power outage”

          … And doesn’t understand how USB works

          I don’t buy it. Even if Ben did real drawing in the past, he clearly isn’t now. Edit: Or as a reply pointed out, the signature could be fake too with Ben not involved.

          • AnAmericanPotato@programming.devEnglish
            11·
            4 months ago

            Is this real? Or is it a fake comic making fun of Ben Garrison? It’s hard to tell. I’d expect better even from him, which is saying a lot.

          • frongt@lemmy.zipEnglish
            8·
            4 months ago

            I dont think this is real. Definitely AI generated, very unlikely to be his actual work.

            • October@lemmy.worldEnglish
              2·
              4 months ago

              I agree, reverse image search isn’t coming up with anything for this one, but it works on other comics from his site. Seems to be an imitation.

          • Rose@slrpnk.netEnglish
            2·
            4 months ago

            You’re saying an actual human called that thing both “pen & ink” and a “typewrite”?

            It’s Ben Garrison! Dude can’t get an erection unless he labels every single goddamn thing. While being barely coherent. He’s been doing this for ages!

      • ThePowerOfGeek@lemmy.worldEnglish
        10·
        4 months ago

        Why did I think he had died? Probably getting him mixed up with Scott Adams.

        I agree with you, it’s a typical example of a Garrison cartoon. It’s so boringly obvious on a surface level that you can’t help but assume there’s some idiotic alt-right subtext.

      • Allah@piefed.worldOPEnglish
        3·
        4 months ago

        what? bro the joke is that ai writers are not real writers, it’s not that deep.

    • REDACTED@infosec.pubEnglish
      9·
      4 months ago

      I can’t tell if you posted this unironically, but in any case this is just bad AI slop

  • Zen_Shinobi@lemmy.worldEnglish
    6·
    4 months ago

    Just look for the em dash. Anytime someone says they have been using it before AI is a liar.

    Edit:being downvoted by all the AI users

    • finalarbiter@lemmy.dbzer0.comEnglish
      7·
      4 months ago

      I’m a liar for using basic English grammar rules like em dash and oxford comma? Lol, tell on yourself harder

      Lol even more at your edit. I’m as anti-LLM and generative AI as they come, are you just proud of your ignorance when it comes to grammar or something?

    • TrackinDaKraken@lemmy.worldEnglish
      4·
      4 months ago

      AI can be prompted not to use such things.

      It also mostly gets hands and feet right now, in images. It’s a constantly moving target, and it does get better at some things.

      One thing it still can’t get right, though, is a classroom full of chairs and desks. It just can’t figure out that chairs need backs, four legs and desks have different style legs, etc.: