Training a model to identify AI web content from structure alone

About Training a model to identify AI web content from structure alone

Word-level detectors identify unedited AI-generated text almost perfectly, but the literature documents their brittleness under rewording, and a word-level score neither characterizes a text nor identifies which AI model wrote it. We ask whether AI-generated text can be identified one level deeper, from structural signatures: how information is presented, in what order, with what evidence, and in what voice. We replicate StoryScope (Russell et al., 2026), which showed such patterns for AI-generated fiction, on commercial content: 2,250 pre-ChatGPT human blog posts from 268 company domains against 11,250 AI mirrors from five frontier models. A 203-feature instrument, applied by an LLM and validated in a human gold-annotation session (human-human kappa 0.939, human-model 0.951), detects AI posts from its 176 structural features alone at 97.0 macro-F1 on held-out companies, nearly unchanged (96.1) when every AI post is reworded by its own model. The signal characterizes and attributes: AI posts share a tidy, self-announcing shape, 68.6% are attributed to the correct source against a 16.7% chance rate, and human posts occupy rare structural configurations. All effects replicate StoryScope's, consistent in direction and at least as large in magnitude. We release pipeline, instrument, prompts, code, and aggregate artifacts.

In the maker’s words

Hey HN! We’re Vincent and Jochen from Sitefire ( https://sitefire.ai ). We have been working together for years, with backgrounds in RL/optimization at Stanford and software engineering from Technical University Munich (TUM). With Sitefire (YC W26), we help marketing teams get recommended by AI Search (ChatGPT, Google AI Overviews, AI Mode, Claude, etc.). Our software monitors prompts, sees which web pages get cited, and uses these insights to help marketing teams take action, e.g. create YouTube videos or write the right blog posts. This means we have a commercial stake in AI-generated web content. And for now, high-information, AI-generated content works great to get cited and recommended in AI Search. But after talking to hundreds of marketing teams, it became clear that everyone despises AI-generated content (“AI slop”). And yet, everyone still wants to leverage AI to create content.…

Where people found it

• Hacker NewsShow HN: Training a model to identify AI web content from structure alone74 points28 comments11 days ago
• Hacker NewsTracing a powerful GNSS interference source over Europe440 points238 comments2026-06-05
• Hacker NewsAll of human cooking compressed into 2 megabytes444 points174 comments2026-05-27
• Hacker NewsLLMs corrupt your documents when you delegate479 points201 comments2026-05-09
• Hacker NewsAll elementary functions from a single binary operator858 points298 comments2026-04-13
• Hacker NewsEmbarrassingly simple self-distillation improves code generation658 points201 comments2026-04-04
• Hacker NewsFrontier AI agents violate ethical constraints 30–50% of time, pressured by KPIs544 points366 comments2026-02-10
• Hacker NewsHow elites could shape mass preferences as AI reduces persuasion costs704 points661 comments2025-12-04
• Hacker NewsGLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models [pdf]417 points83 comments2025-08-12
• Hacker NewsMercury: Ultra-fast language models based on diffusion576 points242 comments2025-07-07
• Hacker NewsLossless LLM compression for efficient GPU inference via dynamic-length float411 points117 comments2025-04-25
• Hacker NewsPen and Paper Exercises in Machine Learning (2022)413 points58 comments2025-03-21
• Hacker NewsDeep Learning Is Not So Mysterious or Different485 points126 comments2025-03-17
• Hacker NewsThe FFT Strikes Back: An Efficient Alternative to Self-Attention456 points168 comments2025-02-26
• Hacker NewsDeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL1.4k points1056 comments2025-01-25