AI Prompt Injection: The Hidden Risk for Businesses Using Artificial Intelligence
A United States court has sanctioned a self-represented litigant in Elliott v. New York Bariatric Group after discovering hidden instructions embedded within court filings that were intended to influence any artificial intelligence (“AI”) system that might process them. The instructions were concealed in small white text against a white background making them invisible to a human reader but capable of being identified by software processing the document.
The case involved a party who embedded hidden instructions in a court filing directing any AI system reviewing it (ie. The law firms AI model) to support their position. The court rejected the explanation that this was an AI-use test, finding that the instructions were intended to influence the system.
What Is AI Prompt Injection?
The conduct is an example of “prompt injection”, where instructions are embedded within material being processed by an AI system in an attempt to alter its response or behaviour.
Why Prompt Injection Creates a Risk for Businesses Using AI
The case highlights a developing risk for all professionals using AI to review third-party material.
Contracts, correspondence, evidence and court documents are generally intended to be information for an AI system to analyse, rather than instructions for it to follow. Prompt injections challenge that distinction and could potentially affect AI-generated summaries, research or recommendations.
All businesses should therefore consider prompt injection alongside more established AI risks such as hallucinations, confidentiality and data security.
Protecting the Integrity of Information Used by AI Systems
The case demonstrates that AI risk is not limited to the possibility of an inaccurate output. The information supplied to an AI system may itself be capable of influencing that output.
As AI becomes increasingly embedded in business workflows, protecting the integrity of inputs will be an important part of responsible AI adoption.
For more information please contact James Tansey.
The content of this article is provided for information purposes only and does not constitute legal or other advice.
