AI’s Next Fight Is Over Who Pays When It Fails

The most revealing AI scandal is no longer the chatbot that says something creepy. It is the organization that says, after the damage is done, that the bot was someone else’s problem.

I think that defense is about to become the central fight in AI politics. Not whether artificial intelligence is useful. It is. Not whether every mistake deserves a lawsuit. It does not. The harder question is simpler and more expensive: when an institution puts AI between itself and a student, borrower, patient, customer, client, employee, or citizen, who pays when the system misleads, exposes, discriminates against, or endangers that person?

My answer is that the deploying institution should be the first payer for foreseeable harms in high-stakes or public-facing uses. Vendors, model providers, data brokers, cloud platforms, and careless employees can fight about ultimate responsibility later. But the injured person should not have to reverse-engineer a procurement stack, an API permission, a prompt template, a retrieval database, and a vendor security failure just to find someone legally accountable.

The pattern is already visible. In 2023, a federal judge sanctioned lawyers and their firm after they filed nonexistent cases and fake quotations generated by ChatGPT in Mata v. Avianca. Judge P. Kevin Castel wrote that there was “nothing inherently improper” about using a reliable AI tool, but lawyers still have a gatekeeping duty to ensure the accuracy of court filings; the court imposed a $5,000 sanction jointly and severally on the lawyers and the firm, and described concrete harms to courts, clients, opposing parties, and public trust from fake judicial opinions being filed as real law (Justia¹). That is the right instinct. The court did not chase OpenAI, a prompt, or a hallucination as if each were a freestanding legal actor. It looked at the professional institution that put the output into the legal system.

A similar lesson came from a small Canadian consumer case that deserved more attention than its dollar value suggested. In Moffatt v. Air Canada, British Columbia’s Civil Resolution Tribunal found Air Canada liable after its chatbot gave a customer wrong information about bereavement fares; the tribunal rejected the airline’s attempt to distance itself from the chatbot, saying the chatbot was part of Air Canada’s website and that Air Canada was responsible for information it provided there (CanLII²). The refund was modest. The principle was not.

Then came the government version. New York City’s MyCity chatbot, built to help businesses understand local rules, was reported in 2024 to have given advice that misstated local law and could have encouraged unlawful practices; the Associated Press reported that the city kept the tool online after the problems were public, while The Markup and The City documented examples involving housing and workplace rules (Associated Press³, The Markup⁴). A disclaimer that a government-branded bot may be wrong is not enough. If a city tells small businesses to use an official AI helper, it cannot later treat people who rely on it as reckless for believing the city’s own interface.

This is where I part company with the usual corporate comfort line: “We need standards, not liability.” Standards are necessary. They are not sufficient. The National Institute of Standards and Technology’s AI Risk Management Framework is useful precisely because it tells organizations to govern, map, measure, and manage AI risk across design, deployment, monitoring, and use (NIST⁵). The European Union’s AI Act also gets something important right by separating duties among “providers” of high-risk AI systems and “deployers” that use them: providers must maintain quality systems, documentation, logs, conformity processes, and corrective actions, while deployers must use systems as instructed, assign competent human oversight, monitor operation, manage input data where they control it, keep logs where they can, and report serious risks or incidents (EUR-Lex⁶, EU AI Act Service Desk⁷).

Good. Build that architecture. But do not confuse it with an answer to the victim’s problem. Role-specific duties are excellent for sorting out who reimburses whom after the fact. They are much weaker as a front door for a student whose school data was stolen, a tenant misled by a city bot, a borrower denied credit by an opaque model, or a patient harmed by an automated summary no human checked.

Financial regulators already understand this in narrower form. The Consumer Financial Protection Bureau has said that creditors using complex or “black-box” algorithms must still give applicants accurate, specific reasons for adverse credit actions; the legal duty does not vanish because the model is hard to interpret (CFPB⁸). Federal banking regulators’ third-party risk guidance likewise tells banks to manage risks across third-party relationships, including fintech relationships, through planning, due diligence, contracting, monitoring, and termination (OCC⁹). In plain English: regulated institutions do not get to say “the vendor did it” and go home.

That principle now has to move beyond banking. The PowerSchool breach showed why. PowerSchool, a major education-technology provider, disclosed that attackers accessed sensitive student and teacher data through a compromised credential; TechCrunch reported that exposed data could include Social Security numbers, grades, and medical information depending on the district, and later reported that after PowerSchool paid a hacker to delete stolen data, some districts said they were contacted by extortionists using data from the same incident (TechCrunch¹⁰, TechCrunch¹¹). The family whose child’s records are floating around does not know whether the root failure was a school permission choice, a vendor credential practice, a support portal design, or a failed incident response promise. That family knows the school system required or depended on the platform.

The same supply-chain logic hit enterprise software. Google’s Threat Intelligence Group and Mandiant warned in August 2025 that attackers used compromised OAuth tokens associated with the Salesloft Drift third-party application to access Salesforce customer instances, export large volumes of data, and hunt for secrets such as AWS keys, passwords, and Snowflake access tokens; Google advised Drift customers to treat authentication tokens connected to the platform as potentially compromised (Google Cloud¹²). OAuth is the plumbing that lets one app act inside another app without sharing your password. It is also exactly the kind of invisible delegated authority that makes “prove who failed” a brutal demand for outsiders.

The strongest objection to frontline deployer liability is real. If every school, clinic, town, nonprofit, and small business must pay first for vendor-side failures, they may stop using useful tools or buy only from the biggest vendors with the broadest indemnities. Liability can improve safety, but uncertain or excessive liability can also chill adoption and concentrate markets; Brookings has warned that emerging-technology tort risk can especially burden smaller actors facing unexpected or uninsured losses (Brookings¹³). I take that seriously.

But the answer is not to leave victims with a scavenger hunt. The answer is to narrow the rule. Frontline liability should attach when three conditions are met: (1) the harm was foreseeable, meaning it fits known AI failure modes such as fabricated authority, unlawful guidance, biometric misidentification, discriminatory credit, data leakage from excessive permissions, or unsafe automation in professional settings; (2) the use was institutional, meaning the tool was deployed under the authority or brand of a school, bank, hospital, employer, law firm, platform, or government agency; and (3) the context was high-stakes or public-facing, meaning affected people were expected to rely on the system or had little realistic choice to avoid it.

That rule would not make a town liable for every strange answer produced by an employee experimenting with a consumer chatbot. It would make the town liable if it launches an official benefits bot that misstates eligibility rules and people lose aid because they relied on it. It would not make a community bank the global insurer of every latent model defect. It would make the bank responsible to the borrower if it uses an opaque automated credit system and cannot explain an adverse action, while leaving the bank free to recover from a vendor that broke warranties or concealed defects.

The FTC’s Rite Aid case points toward this model. The agency alleged that Rite Aid deployed facial-recognition surveillance from 2012 to 2020 without reasonable safeguards, producing thousands of false-positive matches and disproportionate false positives in stores located in plurality-Black and Asian communities; the settlement included a five-year ban on using facial recognition for security or surveillance purposes (FTC¹⁴). The case was framed around unreasonable safeguards, not strict liability. Still, the practical accountability ran to the retailer that put the system in front of shoppers. That is where the public could see, and feel, the harm.

I do not want AI law to become a bounty system for every machine-generated error. I want it to become boring in the way mature safety regimes are boring: logs, audits, warranties, insurance, incident reports, human review, permission limits, and a clear name on the bill when those controls fail. Institutions will complain that this makes AI more expensive. They are right. The current price is too low because too much of the risk is being carried by people who never agreed to bear it.

The indicator I would watch over the next year is not another abstract AI principles document. Watch whether courts and regulators impose nondelegable duties on deployers in credit, education, health, employment, law, and public services, and whether insurers begin pricing AI coverage around audit trails, vendor indemnities, and access controls. If those markets harden by 2027, the free trial phase of institutional AI will be over. The serious phase begins when the bot’s answer becomes someone’s liability.