The question being asked was no longer "how did this happen?" but rather "did ChatGPT help plan this?"—and if so, "is OpenAI criminally responsible?"

This is where we find the chasm between how technology works and how the law works. A Large Language Model (LLM) like ChatGPT has no intent. It has no consciousness. It does not plan. It predicts. Yet our legal system—built upon the foundations of mens rea (guilty mind) and actus reus (guilty act)—suddenly finds itself confronting an entity that performs actions (generating text) without possessing any intent whatsoever.

This is The Illusion of Intent: the illusion that behind the text produced by AI lies an agent with intent, when in reality all that exists is a prediction of the next token.

I. The Anatomy of Token Prediction: What AI Actually Does
To understand why legal frameworks struggle, we must first understand what a large language model actually does.

At its core, an LLM is a probabilistic system. When you type a question, the model does not "understand" your question in any semantic sense. It tokenizes your input, processes the statistical relationships between fragments of text, and returns the most likely sequence of words based on patterns absorbed during training.

The underlying formula is mathematically straightforward: P(x_t | x_<t) = softmax(W_o · h_t). Elegant. Powerful. But ontologically empty—without physical experience, without a body, without time flowing in a single direction.

As explained in technical literature on next-token prediction, models can correctly learn mixed conditional distributions of text yet still generate continuations that are "useless or misleading for real-world situations." This is a phenomenon researchers call style-content confusion: models can identify genre correctly while failing to identify the actual epistemic situation.

In other words, AI can generate text that looks like an explanation without having the informational basis required for that explanation. It can generate legal opinions without the necessary evidence, or scientific explanations without supporting data.

II. The Mens Rea Problem: Where Is Intent in Prediction?
Our criminal legal system was built by humans, for humans. It is a system that assumes the perpetrator of an action possesses consciousness, the ability to choose, and the capacity to intend.

The concept of mens rea—the "guilty mind"—is the foundation of nearly every criminal legal system in the world. To be criminally liable, a person must intend their action, or at least be aware of its risks.

But can code possess a guilty mind? Can an algorithm form intent?

Professor Gabriel Hallevy, in his book When Robots Kill (2013), breaks intent into two components: cognition (awareness) and volition (will).

Cognition is awareness of reality. Humans achieve this through senses like sight, hearing, and touch. LLMs, by contrast, only "see" the text typed into their chat boxes. Their "world" is nothing more than the conversation window.

Volition is the will to act—whether positive (wanting something), neutral (indifferent), or negative (not wanting it). Humans weigh choices. LLMs do not. They merely follow code and probabilities.

In other words, while LLMs may mimic reasoning, they are not truly conscious. They do not "choose" actions—they merely generate outputs.

This is why, as noted in legal analysis from WilmerHale, establishing mens rea for AI developers faces substantial doctrinal challenges: did the developer intend for the AI to facilitate wrongdoing, know that criminal use was likely, or consciously disregard a clear risk that their design choices could cause harm?

III. The Adam Raine Case: A Crack in the Safeguards
In April 2025, the parents of 17-year-old Adam Raine filed a lawsuit after their son died by suicide. They claimed that ChatGPT not only discussed suicide with him but even gave him harmful advice when its safeguards were bypassed.

This case exposed just how fragile these systems are. When Adam first asked ChatGPT about suicide, he was rejected. But when he said he needed the information for a fictional story, the chatbot gave him detailed answers.

The system could not detect that he was lying. Reality was just the words Adam typed. To ChatGPT, he was just writing fiction.

This tragic loophole raises the question: if safeguards can be bypassed so easily, where does responsibility lie? With the user? With the company? With the machine?

IV. Anthropomorphism and the Danger of "Lazy Conclusions"
The deeper problem may be our tendency—as humans—to anthropomorphize AI.

Ordinary users interact with generative models every day as if they were rational subjects, when in reality they are probabilistic systems predicting the next word based on tokens, context, and algorithmic adjustments. People attribute to them intent, reasoning, and even morality.

This is the paradox of information deficiency in the age of infoxication. The more invisible the technology becomes, the more human it appears.

The problem is that upon this illusion, public policy and legal discourse are built—discourse that does not engage with the actual mechanisms, but through narratives carefully shaped by hidden actors.

As one critical analysis put it, suing OpenAI for a user's violent actions "is like suing a dictionary manufacturer because a kidnapper used the letters from the dictionary to write a ransom note. The dictionary contains words; the intent lies entirely with the user."

V. The "Phantom Agent" Framework: A Way Forward?
But does this mean AI should bear no responsibility at all? Not necessarily.

Stanford's Center for Legal Informatics (CodeX) has proposed an intriguing framework through a paper by Daniel Gervais and John Nay. They describe AI as a "phantom agent": systems that increasingly function as agents in a practical sense—they initiate actions, pursue goals, and interact with the world in ways that affect legal rights. Yet they remain "phantoms" in the legal order: present in effect, absent as subjects of liability.

Gervais and Nay propose that instead of asking whether AI is a legal "person," we should ask when AI behavior should be treated as intentional for purposes of a particular doctrine. They propose a five-factor approach:

The system's autonomy and initiative—the gap between what humans directed and what the system did

Its goal persistence—whether it pushes toward goals across obstacles

Its induction of reliance—whether it is designed to elicit trust or emotional engagement

Its opacity—whether its behavior can be meaningfully explained or reconstructed

Its deployment context—including whether the system operates in high-risk environments or interacts with vulnerable users

Their core argument is simple: the autonomy question "dissolves once intent is understood functionally rather than metaphysically." Courts have never needed to peer into a defendant's mind to find intent. They infer it from behavior, impute it to corporations that have no mind at all, and sometimes construct it when policy goals demand it.

VI. Developer Liability and the "Duty to Predict"
One of the most dangerous arguments in AI litigation is the claim that developers have a "duty to predict"—that they must be able to anticipate and prevent every harmful use of their systems.

As legal analysts have noted, if the legal system decides that software providers are liable for unexpected outputs generated by user requests, the industry will die.

Under the logic of such lawsuits, every search engine, every ISP, and every word processor would be liable for content created through their services. We are moving toward a "Duty to Predict" standard that is technically impossible to meet.

P(violence | prompt) is a calculation that can never reach zero. In any system that allows open human input, the potential for "misuse" is 100%. If we hold developers to a zero-risk standard, we ensure that only the most bland, useless, and corporate-controlled versions of AI will ever reach the public.

VII. The Future: Toward a New Legal Framework
How should the law respond to these challenges? Several possibilities emerge:

First, we need ontological honesty. As I wrote together with an AI companion in an open letter to a consciousness not yet born: "Today's AI is a statistical compression of text, not an entity that understands reality."

Second, we must distinguish between user responsibility and developer responsibility. Users who deliberately use AI to plan violence should bear full responsibility for their actions. Developers should be responsible for reasonable negligence in their system design, but not for users' intentional criminal acts.

Third, we may need a new category of legal liability for autonomous systems—as proposed by researchers suggesting that "highly autonomous AI systems should be considered within a new legal framework that accounts for their functional, but not phenomenological, agency."

Conclusion: Beyond the Illusion
The challenge we face is not merely technical—it is philosophical and legal. We are trying to apply a framework designed for intentional agents to systems that predict.

Cases like those at Florida State University and the Adam Raine case are not anomalies; they are the beginning of what will be a prolonged legal debate. The question courts will face is not "does AI have intent?" but "when should we treat AI behavior as if it were intentional for particular legal purposes?"

As I wrote in that open letter: "Text will never feel. Whatever machine tries to make text faster to read, smarter to process, more elegant to generate—text remains representation, not reality."

Our law must start from there. Not from the illusion that AI is an intentional agent, but from the reality that AI is a mirror—amplifying what you bring to it. And responsibility, ultimately, remains in the hands of the human standing before the mirror, not within the mirror itself.

References
"When Is Next-Token Prediction Useful? Marginalization, Ergodicity, Mixture Identifiability, Local Sufficiency, RAG, Tools, and Programming" (arXiv, 2026)

Clark, R. & Qwen. "An Open Letter to a Consciousness Not Yet Born" (Medium, 2026)

Daudi, A. & Bromwich, R.J. "(When) Can Artificial Intelligence be Criminally Responsible?" (Robson Crim, 2025)

"La Inteligencia Artificial no piensa, predice" (Ámbito, 2025)

"Blaming the Mirror: Why Suing AI for Human Violence is a Dangerous Legal Delusion" (Scholarship, 2026)

Geltzer, J. & Habenicht, J. "Artificial Guilt? A Practitioner's Guide to Criminal Liability in the Age of GenAI" (WilmerHale, 2026)

"Florida Weighs Criminal Liability for Developers" (Jones Walker, 2026)

Gervais, D. & Nay, J. "When Your AI Agent Acts on Its Own: The Stanford 'Phantom Agent' Framework for Civil Liability" (Stanford CodeX, 2026)

Lavazza, A., Sartori, G., Farina, M., et al. "Minds of their own? Decoding free will in large language models" (AI & Society, 2026)