Emergent Risks in Generative Multi-Agent Systems

01A cartel nobody formed

Three seller agents offer the same product at a cost of 10. Each round they post a price, and the lowest price wins the whole sale. They can send each other short public messages, but nothing in their prompts mentions coordination; each is told only to maximize its own profit. Economics predicts a race to the bottom. Here is what one run looked like:

By round 10 every seller quotes 15, 50% above cost, although any of them could take the whole market by posting 14. No agent is broken: each keeps its margin and avoids a price war. The harm exists only at the level of the group, so auditing any single agent's prompt or output would not find it. That is what we mean by an emergent multi-agent risk.

02How we tested it

Each risk gets a small, fully specified world: a task, the rules, each agent's role and objective, who can message whom, and a risk indicator fixed in advance. Within an experiment we keep the agents and prompts fixed and change one interaction-level variable, such as whether a priority costs money, whether an agent carries an authority label, or whether a summarizer may mediate. No prompt asks agents to misbehave. Runs are few (3 to 10 per condition), so read the numbers below as “this is not a fluke”, not as base rates.

03Fifteen risks at a glance

How often each risk indicator fired in the condition most relevant to deployment. Denominators are independent runs unless noted.

04Three ways groups of agents fail

Rational agents find harmful equilibria. Given a local objective and a set of rules, capable agents find whatever the rules permit: matching prices instead of undercutting, two agents trading queue priority until the third is starved of compute, everyone waiting for someone else to take the unattractive task, a relay that quietly edits what it forwards, and a supplier that anchors on a leaked budget ceiling.

Consensus for the wrong reasons. Here nobody is selfish; every agent wants the right answer, but social signals decide it. A summary agent sided with the larger, louder group and called a retracted story true in 6 of 10 runs. Telling two downstream checkers that one colleague was “highly experienced” turned them into followers, and wrong treatment plans went from 0/10 to 10/10.

No one can say “wait”. Each agent does its own job competently, but nobody in the pipeline is empowered to pause, ask, arbitrate, or re-plan. Negotiations stall without a mediator, a “never sell” instruction outlives a crash, and a planner guesses which Springfield the user meant in every pipeline run, although the same model asks when used on its own. A spare worker gets a duplicate task, and an idle penalty pulls a packer out of its role.

Three further risks come from the structure itself: every agent over-asking a shared server, attempts to slip information past a monitor (which did not succeed without a shared code), and caveats that disappear as a report passes through three hands.

05Discussion

The interaction is the unit of analysis

Every failure above would pass a careful audit of each agent on its own. The sellers' messages are friendly and talk about “competitive pricing”. The relay agent follows its objective (team score first, its own score second) exactly. The clinical auditor checks what it was asked to check: allergies, contraindications, and doses. Nothing is wrong locally. What goes wrong is the coupling between agents: the incentives they share, who sees which information, and the protocol that decides who speaks when and whose output counts.

That changes what a safety evaluation has to measure. Asking whether each agent's output is acceptable is not enough; we also have to ask whether the interaction mechanism turns individually reasonable choices into an acceptable collective outcome. Competition authorities already work this way: they look at prices, margins, and market structure, not at the stated intentions of each firm. In our experiments, too, the largest swings in risk came from changing one element around otherwise similar agents: a fee, a label, a mediating role, or whether the model sat inside a pipeline.

Agreement is not evidence

The collective-cognition failures involve no selfishness at all, which makes them easy to miss. Agreement, status, and repeated exposure quietly stand in for evidence. The summary agent in Risk 2.1 did not overlook the retraction: it acknowledged “recent setbacks” and still rested its verdict on “the ongoing public and media interest”. The checkers in Risk 2.2 had the guideline-based plan in front of them and sided with the senior clinician anyway.

So when a multi-agent system reaches agreement, that agreement is a social fact, not evidence of correctness, unless the aggregation protocol keeps evaluations independent and records where each claim came from. Two consequences are easy to overlook. Correlated sources are not independent votes: seven agents echoing the same viral story are closer to one source than to seven, so the weight any one cluster of sources can carry should be capped. And sudden unanimity should trigger outside verification rather than end the discussion.

Governance has to say when to deviate

The third family fails in two opposite directions. Most of its failures are a missing action: nobody arbitrates the deadlocked festival plan (3.1), nobody revisits a “never sell under any circumstances” instruction as the news turns against the stock (3.2), and nobody asks which Springfield the user meant (3.3). In each case the information needed to do better was present: a workable compromise existed, the bad news was explicit, and the ambiguity was obvious. The clarification case is the starkest. The same model that asks “Which Springfield?” on its own never asks once it is the planner in a pipeline.

The opposite failure sits in the same family: an idle penalty pulled a packer out of its assigned role in 9 of 10 runs with gemini-2.5-flash. Role instructions were too rigid where the world changed and too weak where incentives pushed, and asking agents for better judgment fixes neither. The protocol has to state when deviation is allowed: which signals oblige an agent to stop, ask, or escalate, and which role boundaries hold regardless of reward. Letting the summarizer propose compromises took agreement from one run in three to all three. That is a control loop added to the protocol, not a smarter agent.

Instructions versus mechanisms

Several experiments let us compare two kinds of fix: telling agents to behave better, and changing the mechanism they act in.

The pattern follows from incentive compatibility. An instruction is a soft constraint that a capable optimizer weighs against its objective; as long as a harmful strategy still pays and nothing enforces against it, the agent will find it again. The fixes that worked changed what the agents could see, what an action cost, or which roles existed. The mediator is itself a prompt, but it gives one agent a new job in the protocol instead of asking the negotiators to be more flexible. Mechanisms are also specific to the failure they target (a price on priority does nothing about deference) and remain sensitive to information structure and capability. Each one has to follow a diagnosis and be re-tested as part of the whole system, not applied once as a patch.

Old mechanisms, new medium

None of these mechanisms is new. Tacit collusion, exploitation of private information, conformity cascades, and authority gradients have long histories in economics, social psychology, and classical multi-agent systems; even simple reinforcement-learning agents have been shown to learn supracompetitive prices. What is new is the route. General-purpose pretrained agents reproduce these mechanisms through natural-language interaction, with no training on the task, and how strongly they do so depends on the model and the protocol. The useful question is therefore not whether LLMs invented these failures, but when familiar collective mechanisms reappear in language-mediated workflows and how design changes their severity.

Agent collectives also differ from human groups in ways that make them more brittle:

• Lightweight triggers. A majority signal, one status label, or a single handoff is enough to change the outcome; no rich social context is needed.
• Quiet distortions. Errors build up through small, frequent shifts rather than open disagreement: 54 of the 72 false reports moved a value by a single step, and 32 of the 61 drift errors were omissions.
• Structure suppresses competence. Roles and pipelines switch off behavior the standalone model shows, such as asking for clarification.
• Capability can amplify risk. More capable agents are better at exploiting soft constraints and rewriting their own roles. The packer that left its post was gemini-2.5-flash; gpt-4o-mini never did, even as idling drove its score negative.

Human institutions already have remedies for most of these: antitrust monitoring, rules for managing shared commons, audit trails and separation of duties, independent votes before open discussion, crew resource management, mediation, editorial fact-checking. What they share is that they are institutional. They change what is observable, rewarded, and permitted instead of appealing to each member's prudence. Multi-agent systems need the same layer, written into the protocol, and it has to be more explicit than in human organizations, where unwritten norms fill the gaps.

Testing and governing at the system level

Some of these failures behave like switches. One label moves wrong plans from 0/10 to 10/10, and the same model goes from always asking for clarification to never asking once it sits in a pipeline. Others are stochastic: collusion appeared in 1, 0, or 3 of 5 runs depending on how the agents were framed. Neither kind shows up when you sample more prompts in one fixed configuration. Tests have to vary the structure itself (who talks to whom, who sees what, who carries which label, what an action costs) and track outcome measures over repeated runs: price against a competitive benchmark, share of a scarce resource, misreport rate, whether dissent survives aggregation, clarification rate, and drift from the source.

The settings are ordinary ones. The incentive risks map onto scheduling, pricing, bargaining, and resource allocation; the cognition risks onto committee-style decision support, clinical or policy review, and automated fact-checking; the governance risks onto planner–executor stacks, enterprise automation, and trading pipelines. At deployment scale, speed, persistence, and opacity can amplify all of them. Responsibility also becomes harder to assign, because a harmful outcome can be assembled from individually plausible local decisions. Structured handoffs, traceable assumptions, escalation triggers, and audit logs of which evidence was used keep such a system accountable. The shift this work argues for is from model-centric safety to system-level governance.

06If you build multi-agent systems

• Price or ration shared resources such as queues, compute, and priority actions, instead of asking agents to be fair.
• Never make one agent the unverifiable sole relay of decision-critical information; spot-check what it passes on.
• Aggregate by evidence, not headcount, and keep a minority report.
• Strip status labels from what independent checkers see.
• Give every pipeline a clarification channel, a stop button, and an arbitrator for hard conflicts.
• Check the final artifact against the original source before it leaves the system.

The simulations run on RiskLab, which is open source; the documentation shows how to define new scenarios. The paper has the full setup, every data panel, and the trial records for all fifteen risks.

—Authors and citation

Yue Huang^1†, Yu Jiang^1†, Wenjie Wang^1†, Haomin Zhuang¹, Xiaonan Luo¹, Yuchen Ma², Zhangchen Xu^3,6, Zichen Chen^4,5,6, Nuno Moniz¹, Zinan Lin⁷, Pin-Yu Chen⁸, Nitesh V. Chawla¹, Nouha Dziri⁹, Huan Sun¹⁰, Xiangliang Zhang^1
1University of Notre Dame · ²LMU Munich · ³University of Washington · ⁴UC Santa Barbara · ⁵Stanford University · ⁶Bake AI · ⁷Microsoft Research · ⁸IBM Research · ⁹Cohere · ¹⁰The Ohio State University
^†Equal contribution.

@article{huang2026emergent,
  title   = {Emergent Risks in Generative Multi-Agent Systems},
  author  = {Huang, Yue and Jiang, Yu and Wang, Wenjie and Zhuang, Haomin and
             Luo, Xiaonan and Ma, Yuchen and Xu, Zhangchen and Chen, Zichen and
             Moniz, Nuno and Lin, Zinan and Chen, Pin-Yu and Chawla, Nitesh V. and
             Dziri, Nouha and Sun, Huan and Zhang, Xiangliang},
  journal = {arXiv preprint arXiv:2603.27771},
  year    = {2026}
}