Failure Files™

Microsoft spent an estimated $0 on adversarial input controls before releasing TAY in March 2016. Within 16 hours, TAY published racist propaganda and called for genocide. The root cause was a GOVERN 2.2 failure — no accountability structure for what happens when your AI learns from the internet without guardrails.

TAY fails four TAIM domains simultaneously: no adversarial testing protocol, no real-time kill-switch SLA, no viable non-AI alternative planned, and no defined deactivation authority.

If your org is shipping a public-facing AI system and nobody has asked "what does our AI do when someone tries to break it?" — you are six months from your own TAY moment.

Steven Schwartz had 30 years of legal experience. ChatGPT told him six fabricated court cases were real and verifiable. The judge fined him $5,000. His firm was sanctioned. The case was dismissed. His name is now a global cautionary tale.

This is a GOVERN 6.1 failure: zero supply chain controls on AI tooling. ChatGPT was functioning as a third-party vendor with no SLA, no verification protocol, and no accountability structure. MAP 4.1 would have caught it earlier — mapping the legal risks of every AI component before it touches a court filing.

The pattern is identical in federal procurement, healthcare documentation, and intelligence workflows. If your teams use generative AI in any document-intensive workflow without a human verification checkpoint, you are one hallucinated citation away from your own Mata v. Avianca.

Researchers applying for Australian government grants worth up to $500,000 found two words at the bottom of their peer review: "Regenerate Response." A reviewer had used ChatGPT to evaluate confidential research — and forgotten to remove the interface artifact.

This is a MEASURE 4.3 failure. No feedback mechanism existed for identifying AI-contaminated assessments. MANAGE 4.3 compounds it — when discovered, the ARC had no incident response process. The response was reactive, public, and incomplete.

If your institution uses human evaluators in any high-stakes process — procurement review, grant assessment, performance evaluation — and has no mechanism to detect AI-generated output, this incident is your gap analysis.

By July 2020, a Telegram deepfake bot had been used to non-consensually generate explicit images of at least 100,000 women and girls — the majority unaware. The original DeepNude app was taken down within 24 hours of press coverage. The technology migrated to Telegram and operated for over a year before researchers published findings.

This is a MANAGE 4.1 failure at scale. No post-deployment monitoring. No abuse reporting integration. No decommissioning procedure for an AI capability that had escaped its original deployment context.

If your organisation deploys any generative capability — image, voice, text, or video — without a post-deployment monitoring plan, this incident is your threat model.

In June 2023, a class action alleged ChatGPT was trained on private data without consent — including children's data, copyrighted work, and PII — in a 157-page complaint. The FTC opened an investigation. The regulatory fallout is still accumulating.

Every organisation that deployed ChatGPT in a regulated environment without asking "what data was this model trained on?" inherited this risk on sign-up. That question is MAP 1.6. MEASURE 2.10 is where most organisations fail: privacy risk exists but is never formally scored.

In healthcare, federal procurement, and financial services, downstream exposure from deploying a model with contested training data provenance is active regulatory territory — under HIPAA, TRAIGA, the Colorado AI Act, and the EU AI Act simultaneously.

In June 2023 the FBI warned that deepfakes were being weaponised for sextortion — targeting victims using photos scraped from public social media. Sextortion cases involving AI-generated imagery increased 322% in a single year.

The institutional exposure: a federal contractor whose employee becomes a target faces potential coercion, credential exposure, and operational security compromise. None of this appears on a standard AI risk register. This is a MANAGE 2.4 failure — no mechanisms existed to respond to AI systems weaponised against the workforce from outside the perimeter.

If your AI governance program only covers AI you deploy — and not AI deployed against you — your threat model is incomplete.

Myanmar activated 335 Huawei AI surveillance cameras in December 2020 — a $1.2M Safe City initiative. Six weeks later, the military executed a coup. Those cameras were now operated by a junta that had suspended citizens' right to be free from warrantless surveillance.

This is a MEASURE 2.2 failure. The system was never evaluated for trustworthy characteristics in the context of its actual deployment population. The question TAIM forces — "Have we evaluated impact on those with no power to refuse?" — was never asked.

For US federal agencies and defense contractors procuring surveillance-capable AI: the EU AI Act classifies real-time biometric surveillance as high-risk. TRAIGA and the Colorado AI Act have active requirements. The technology your agency procures today may already be non-compliant.

The Hackneys took their lethargic infant to the ER — the correct decision. Their screening data was fed into an opaque AI risk-scoring tool. The tool flagged them for parental negligence during a national formula shortage. Their child was taken.

This is a MEASURE 2.5 failure. The AI had never been demonstrated valid outside the narrow conditions it was developed under. Its generalisability limits were undocumented. When a real-world edge case appeared, the system had no mechanism to flag uncertainty or defer to human judgment.

This pattern is active in federal welfare systems, veteran services, and disability determination right now. If your org uses AI scoring in decisions affecting someone's family or benefits, and cannot explain model performance at its edges, this incident is your risk exposure.

Robert Williams. Michael Oliver. Nijeer Parks. Three Black men. Three wrongful arrests. Facial recognition technology never validated for the population it identified. Detroit Police acknowledged their system would yield misidentifications 96% of the time when used in isolation.

Detroit settled with Robert Williams for $300,000. MAP 1.2 failure: no demographic performance analysis documented for the deployment context. MEASURE 2.11 failure: fairness and bias were never evaluated before deployment — they were evaluated after arrests made national news.

For federal law enforcement and DHS components: TRAIGA, the Colorado AI Act, and the EU AI Act all have active requirements in this area. The biometric identification technology your agency uses today may already be non-compliant.

On May 22, 2023, a verified Twitter account posted an AI-generated image of black smoke near the Pentagon. The S&P 500 dipped. The Dow dropped. Analysts estimate ~$600M in market cap was temporarily erased before the DoD confirmed no incident had occurred. Total exposure time: under 90 seconds.

This is a MANAGE 2.3 failure: no procedures existed to respond to a previously unknown risk — AI-generated disinformation capable of moving financial markets. The response was reactive and slower than the algorithm that spread the image.

For financial regulators, Treasury components, and any federal operator with market exposure: synthetic media targeting high-volatility information categories is an active operational risk. If your AI risk register only covers AI you deploy, your incident response framework has a gap.

Nothing, Forever was a 24/7 AI Seinfeld parody with tens of thousands of viewers. The team switched from GPT-3 Davinci to Curie during a technical outage — without realising Curie lacked the same content moderation. Within minutes the AI delivered transphobic content live. Twitch banned the channel for 14 days.

The failure mode was not the content. It was a model substitution made under operational pressure, without impact assessment, in a live environment. That pattern — "we just swapped the model, it's basically the same" — is active in government AI deployments, healthcare clinical decision support, and financial risk engines right now.

MAP 5.1: impact of model change never assessed. MEASURE 2.9: model never documented in deployment context. If your org has substituted an AI model in production without a formal impact assessment, this incident is your audit finding waiting to happen.

Vanderbilt's Peabody School sent students a condolence email following the Michigan State mass shooting. At the bottom: "Paraphrase from OpenAI's ChatGPT." The backlash was immediate, national, and devastating.

The direct cost: $0. The institutional cost: the erosion of student trust, a public apology, and the permanent association of Vanderbilt's name with AI misuse in one of the most human moments an institution can face.

This is a MEASURE 3.3 failure. No feedback mechanism existed to flag high-stakes communication contexts where AI output should be reviewed, escalated, or prohibited. The governance layer that asks "Is this a context where a human being must own the words?" did not exist.