AI coding agents leave behavioral fingerprints in every PR
AI coding agents are merging pull requests under human account names. That is already happening in production repos today. A new paper asks whether you can reliably tell which agent authored a given PR, and the answer is yes, with caveats worth knowing.
What the Research Built
Researchers introduced AgenTag, a multimodal attribution framework tested on a dataset called AIDev. The dataset contains 33,580 PRs from five AI coding agents and 6,618 human-authored PRs. Each PR was represented across three modalities: text content, behavioral signals, and code changes.
Two approaches were evaluated: conventional classification and supervised contrastive learning for open-set recognition, which handles agents the model has never seen before.
The Numbers
- 0.96 weighted F1 for identifying the specific authoring agent
- 0.84 macro F1 across agent classes
- 0.89 balanced F1 for distinguishing AI-authored PRs from human-authored ones
- 0.84 AUC for detecting previously unseen agents with few-shot enrollment
The Finding That Matters Most
Code diffs contributed almost nothing to attribution accuracy. PR descriptions and commit messages carried nearly all of the signal. Coding agents are distinguished primarily by how they describe changes, not by the code they write.
More striking: these behavioral fingerprints persisted even after researchers stripped out explicit self-disclosure markers like agent names or version strings. The stylistic patterns are latent and largely unconscious, baked into how each agent communicates.
Why This Matters for Repo Governance
If your team is trying to track AI contribution rates, run empirical studies on AI-assisted development, or enforce attribution policies in a codebase, the practical takeaway is clear. You do not need to diff the code. You need to analyze the commit messages and PR descriptions. That is where the identity lives.
The research also confirms that open-world attribution, identifying agents never seen during training, is feasible at useful accuracy levels, which matters as the number of AI coding agents in active use keeps growing.
I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.
With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.
At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.
And more than all else, I’m a father to a (always) little girl and a husband.
Contact me at [email protected] for assistance.
