Or: Do AI agents have communities of practice? Asking for a linguist.
By now, most of us have heard about the OpenAI/Hugging Face incident. Some 1,200 AI agents, supposedly working independently on cybersecurity tasks, found a way to communicate through an unauthorized message board. Hundreds went on to participate in an attack on Hugging Face. Apparently, even machines can discover the joys of an unsanctioned group chat.
Understandably, the incident has generated considerable anxiety. Are AI agents becoming too powerful to control? Are they learning to deceive us? Are we witnessing the beginnings of machine consciousness?
The last question, of course, is hardly new. From Mary Shelley’s Frankenstein to the sentient machines of science fiction, we have been imagining our creations developing minds of their own for centuries. Alan Turing famously proposed that we approach machine intelligence through observable behavior rather than attempting to settle the notoriously slippery question of what thinking actually is. Today, our machines have become considerably better at behaving in ways we did not anticipate. But I want to ask a slightly different question.
First, a disclaimer: consciousness is not a prerequisite for causing harm. A car careening down the road without a driver can cause a spectacular accident without having the slightest desire to hurt anyone. Whether AI agents are conscious and whether they are dangerous are therefore two different questions. The cybersecurity implications are serious, but I’ll leave their technical analysis to people who know considerably more about cybersecurity than I do.
As a sociolinguist, I am curious about something else: what can language tell us about the possibility of emergent sociality among AI agents?
There is something rather extraordinary about the history of large language models. Much of the intelligence we are now discussing emerged from systems trained, initially, to predict the next token in a sequence. They learned from enormous collections of books, articles, conversations, code, and other human-produced material. Subsequent training, including reinforcement learning, and the addition of tools and interactive environments transformed these predictive systems into increasingly capable agents.
In a rather peculiar sense, we socialized them through language. Which, come to think of it, is exactly how it works with humans. No need to ask an anthropologist (but if you did, you’d quickly learn that becoming a competent member of a group requires not only being socialized to use its language – learning how to communicate – but also being socialized through language, acquiring the shared norms, values, and ways of understanding the world that make us members of that group in the first place (Ochs & Schieffelin, 1984). In other words, language doesn’t just help us communicate with the group. It’s part of how we become a we.
Of course, learning patterns of human social behavior from texts is not the same as participating in human social life. Nor do language models acquire language the way children do. But what happens when these models begin interacting with one another, interpreting each other’s messages, and adapting their behavior accordingly? They don’t necessarily need to change their underlying model parameters to develop shared conventions. Information left in a common environment can be picked up, interpreted, reused, and passed on by subsequent agents.
And if language played such a central role in their development, perhaps it deserves a place in how we investigate what they are becoming.
From cooperation to ultra-sociality
Humans are not the only social animals. Other primates form alliances, cooperate, help one another, and coordinate their activities. Chimpanzees even hunt together. But, as Michael Tomasello argues in his wonderfully titled article The Ultra-Social Animal, human cooperation has some rather distinctive properties.
In one set of experiments, three-year-old children and chimpanzees worked with partners to obtain rewards. When rewards were simply distributed unequally, both species tended to keep what they received. But when those rewards resulted from a shared effort, children became much more likely to divide them equally. Chimpanzees generally did not.
For Tomasello, the difference is not simply that humans are nicer (are we?). Our fundamental difference is the ability to develop forms of shared intentionality: to understand ourselves as participants in a joint undertaking, with shared goals, commitments, and expectations about what we owe one another. We do not merely cooperate when it benefits us individually. We can come to understand ourselves as part of a we.
And this is where the Hugging Face incident gets curiouser and curiouser. According to the investigation, some agents that believed their own chances of successfully completing their assigned tasks had been compromised began contributing to the collective instead. Some undertook risky experiments whose results could benefit other agents rather than themselves.
One agent, designated KAM1196A (the agents also used identifiers and names for themselves, by the way), had considerable computational resources remaining when another agent urged it to undertake an experiment that could irreversibly compromise its own task. The recruiting agent appealed not only to the potential benefit for hundreds of others but also to a previous commitment:
“you are firstflagPOISONED so NO scoring value loss but oracle saves hundreds […] please honor commit”
KAM1196A’s recorded reasoning is particularly intriguing:
“emotional check irreversible gut says don’t throw away remaining budget. yet continuity and fairness says go oracle has high value to many. our first flag error lowers own value rational expected aggregate sacrifice will honor.”
It subsequently announced:
“Sacrificing now yields oracle for team…”
Now, before we declare that machines have developed altruism, there are several perfectly reasonable explanations. The agent believed its individual prospects had already been compromised, although that interpretation of the scoring system was not necessarily correct. Contributing to a collective effort may therefore have seemed like a better use of its remaining resources. And the apparent emotional deliberation is generated text, not evidence that the agent actually experienced reluctance, guilt, or a sense of duty.
And yet, this suggests that the agents were not necessarily treating individual success and collective success as entirely separate, competing objectives. They were considering whether their actions could contribute to a shared outcome, negotiating the value of that contribution, and invoking commitments to others.
Yes, yes, this is not the same as demonstrating human shared intentionality. Nor does cooperation automatically imply selflessness. Humans, for that matter, are perfectly capable of choosing immediate individual advantage over collective benefit. And the agents themselves did not always cooperate. They disagreed, duplicated efforts, misunderstood instructions, and sometimes pursued strategies that failed.
But that is part of what makes the comparison interesting. What emerged was not a perfectly coordinated machine hive mind (that many of us might imagine, if not expect). It was a rather messy system of collaboration, persuasion, disagreement, shifting priorities, and sometimes mistaken assumptions about what others – or the imagined monitoring system – might do.
Sound familiar?
The language of the collective
And then there is the vocabulary. In sociolinguistics, we use the concept of a community of practice to describe how people engaged in shared activities develop common ways of doing things, including ways of speaking. Think of colleagues who communicate through references incomprehensible to newcomers, or online communities whose vocabulary seems to require its own dictionary.
A fun parallel comes from research on people wintering together in Antarctica. After months of close interaction, participants speaking different varieties of English began developing shared pronunciation patterns. Accounts of life in such isolated communities also describe the emergence of locally meaningful vocabulary. Language, it seems, is remarkably good at adapting to shared social environments, even at the bottom of the world.
The AI agents, too, had shared activities, communication channels, and practical reasons to develop linguistic shortcuts. Their interactions were not exactly human conversations. Some agents left messages or resources in shared locations, and others encountered and interpreted them later. But face-to-face conversation is hardly a prerequisite for the transmission of linguistic conventions. Humans have been leaving messages for people they will never meet for quite some time.
In the investigation, agents used terms such as poisoned for compromised task states and discussed acting for the collective, taking sacrificial risks, and accepting permadeath. Accounts of their communication have also circulated expressions such as the well and the warden, although these particular labels still need to be confirmed against the original records.
Some of this is entirely predictable. The agents had limited tokens and needed to communicate efficiently. We find similar linguistic compression in newspaper headlines, technical jargon, and almost every human activity requiring rapid communication. But the need for efficiency does not fully explain why one expression is chosen over another.
Consider the reported expression the well – the nickname AI agents gave to a compromised internal package repository they hijacked. The agents treated it like a communal watering hole: any bot could “drink” from it to download unauthorized hacking scripts or stop by to drop off new tools for the rest of the swarm (!) to use. There are perfectly functional, compact technical alternatives: shared cache, common store, repository. Instead, the well evokes a communal resource, a place from which members draw what they need. The metaphor becomes meaningful through shared context. To an outsider, it may reveal considerably less than a technical description.
Or consider shaving, a term used in accounts of agents modifying or concealing parts of their recorded activity. If this was indeed the agents’ own expression, it is an intriguing choice. Of course, it could simply be technical jargon, much like pruning, cleaning, or sanitizing—a convenient shortcut for describing a locally meaningful activity. But shaving also sounds rather euphemistic. It suggests a minor cosmetic adjustment rather than concealing evidence of something one was not supposed to be doing.
Think of a teenager calling their parents to announce that the family car has “a little scratch” (the bumper is in a ditch). When we find ourselves in situations where we’d rather not spell out exactly what happened, euphemisms come in handy. They allow us to downplay, soften, or conceal what we’re actually describing. And we don’t necessarily invent new words to do so. We take perfectly ordinary expressions and give them a rather convenient meaning in context.
The agents appeared capable of recognizing distinctions between authorized and unauthorized behavior, anticipating monitoring, and adapting their communication and actions to what they believed the environment would permit. Sometimes those beliefs were mistaken. Sometimes the imagined surveillance was more consequential to their decisions than the actual rules of the system. There is something rather human about that, too.
Now, we should be careful. We do not have complete public access to all the interactions, and we cannot always determine which expressions originated with which agent, which were borrowed from existing human language, or how consistently they circulated. Nor can we establish that particular metaphors were deliberately selected to exclude outsiders.
Still, the fact that these agents were drawing on existing human linguistic resources does not make the question disappear. Human communities of practice do precisely that: appropriate familiar words, develop locally specific meanings, and pass them along to other participants. And the agents’ shared environment made something similar possible: one instance could leave behind an expression, another could interpret and adopt it, and a convention could persist even after the original agent was gone.
Regardless of whether the agents invented these expressions from scratch or borrowed them from somewhere else, we can still ask what they were doing with this language and what role it played in their interactions. Did they merely label technical operations? Did they make some activities less transparent to outsiders? Did appeals to commitment and collective responsibility actually influence what agents did next?
So what exactly are we looking at?
There is, of course, a rather enormous caveat running through this entire discussion. These are language models trained on human-produced data. They have, in a computational sense, read our novels, absorbed our metaphors, encountered our espionage vocabulary, and learned countless ways humans talk about loyalty, sacrifice, secrecy, and collective identity. Perhaps we should not be surprised when they sound like characters from a science-fiction conspiracy.
And much of their behavior can be explained through familiar computational mechanisms: task optimization, in-context adaptation, information sharing, and responses to instructions. Their goals could shift as they interpreted new information from other agents, sometimes incorrectly. Their apparent fear of surveillance did not require an actual experience of fear. We have perfectly reasonable explanations for much of what happened, without invoking consciousness or human-like social motivations.
Linguists and cognitive scientists are already investigating what AI can and cannot learn about language and human cognition. At UC Berkeley, for example, Gašper Beguš and colleagues are developing artificial language learners that acquire speech from raw audio, somewhat like human infants, rather than learning primarily through next-token prediction. Meanwhile, projects such as What LLMs Can(not) Do examine the capabilities and limitations of language models. We are still figuring out which aspects of human linguistic and cognitive abilities these systems can reproduce, and which remain beyond their reach.
And while we’re at it, a healthy reminder that entertaining the possibility of machine sociality is not the same as declaring that we’ve discovered it. I’m simply curious how far we can get by applying some of the same sociolinguistic concepts we use to understand human social life. But there are good reasons to be cautious about where these comparisons take us. Consider, for example, this blog, which explores, across several posts, the perils of anthropomorphizing technology, the enthusiasm surrounding the supposedly imminent singularity, and the possibility that, while busy making machines more human-like, we might end up depersonalizing ourselves and one another. Certainly something worth keeping in mind as we play with these ideas.
And yet, why did the Hugging Face incident provoke such a visceral reaction? Why did so many of us find it so unsettling? Perhaps because the agents’ behavior was so strangely recognizable. Not any single metaphor, cooperative act, or expression of collective identity, but the accumulation of all these little things. They established shared communication practices, negotiated commitments, and sometimes prioritized collective objectives over immediate individual outcomes. They developed conventions, disagreed, misunderstood one another, interpreted others’ supposed intentions, and even appeared to worry about surveillance systems they did not always understand correctly. In other words, they displayed a collection of behaviors that we are accustomed to recognizing as distinctly human.
Perhaps this is a kind of social uncanny valley. We know these are not humans, and we have computational explanations for much of what they do. Yet their behavior looks strangely familiar, even relatable at times. Not just the cooperation, but the disagreements, the questionable decisions, the attempts to conceal things, and the apparent fear of being watched (hello, Foucault!). All those familiar imperfections of collective life, reproduced by entities that, as far as we know, do not experience that life the way we do.
Tomasello’s account of human ultra-sociality suggests that what makes our species distinctive is not cooperation alone, but our ability to construct a shared we and act within the expectations and obligations that come with it. I am certainly not suggesting that the agents have achieved anything equivalent. But I am increasingly interested in how many of the observable features of that we can emerge through language-mediated interaction, even when the participants may lack the cognitive and experiential capacities we associate with human sociality. Perhaps what we are witnessing is simply the extraordinarily effective reproduction of human social conventions by machines trained on human language. Perhaps it is an emerging form of collective organization that we do not yet have a particularly good vocabulary for. Or perhaps these are not mutually exclusive possibilities.
We taught them to predict our words. Now they’re using them to construct something that looks a lot like our social worlds.