AI continues to be adopted at pace by companies seeking to enhance and scale their operations, but alongside this potential there are significant opportunities for threat actors.
Attackers are increasingly targeting vulnerabilities within large language models (LLMs) used to recognise and generate text. In response to the growing risk, the recently launched OWASP Top 10 for LLMs covers the key vulnerabilities within these types of AI applications. Read our guide to learn more about the most critical vulnerabilities and how to reduce AI security risks.
What is the OWASP Top 10 for LLMs?
The OWASP Top 10 for LLMs is a new standard that lists the top 10 most critical vulnerabilities often seen in LLM applications, highlighting their potential impact, ease of exploitation and prevalence in real-world applications.
The OWASP LLM Top 10 is aimed at developers, data scientists and security experts responsible for designing and building applications and plug-ins using LLM technologies.
The standard was developed by the Open Web Application Security Project (OWASP), a not-for-profit foundation which supports organisations to improve the security of their web applications. The organisation is recognised primarily for the OWASP Top 10, which outlines key vulnerabilities that affect web application security.
The OWASP Top 10 for LLMs focuses on the unique implications of key vulnerabilities when encountered in applications utilising LLMs, making the connection between general application security principles and the specific kinds of challenges posed by these systems. It will be updated periodically to ensure that it keeps pace with the fast-changing nature of the AI industry.
The current OWASP LLM Top 10 vulnerabilities are listed below.
1. Prompt injection
An attacker manipulates an LLM through crafted inputs, causing it to unknowingly execute the attacker’s intentions. Direct injections overwrite system prompts, while indirect ones manipulate input from external sources.
2. Insecure output handling
This vulnerability occurs when an LLM output is accepted without scrutiny or with insufficient validation, sanitisation and handling of the outputs generated by large language models before they are passed downstream to other components, exposing backend systems. Misuse may lead to serious consequences such as cross-site scripting (XSS), cross-site request forgery (CSRF), server-side request forgery (SSRF), privilege escalation or remote code execution.
3. Training data poisoning
This issue occurs during the manipulation of pre-training data or data involved within the fine-tuning or embedding processes to introduce vulnerabilities or biases that compromise security, effectiveness or ethical behaviour. Potential sources include Common Crawl, WebText, OpenWebText and books.
4. Model denial of service
Attackers extract or infer specifics of training data by querying model or executing resource-heavy operations on LLMs, which leads to service degradation or high costs. The vulnerability is enhanced due to the resource-intensive nature of LLMs and the unpredictability of user inputs.
5. Supply chain vulnerabilities
The LLM application lifecycle can be compromised by vulnerable components or services, leading to security attacks, impacting the integrity of training data, machine learning (ML) models and deployment platforms. The use of third-party datasets, pre-trained models and plugins has the potential to create additional vulnerabilities.
6. Sensitive information disclosure
LLMs may inadvertently reveal confidential data in their responses, which can lead to unauthorised data access, privacy violations and security breaches. As a result, they may reveal sensitive information, proprietary algorithms or other confidential data through their output. It is critical to address this risk by implementing data sanitisation and strict user policies.
7. Insecure plugin design
LLM plugins, extensions that are called automatically by the model during user interactions, can have insecure inputs and insufficient access control. This lack of application control makes them easier to exploit and can result in consequences such as remote code execution.
8. Excessive agency
LLM-based systems may undertake damaging actions that can lead to unintended consequences due to unexpected or ambiguous outputs, such as excessive functionality, permissions or autonomy.
9. Overreliance
This issue is also referred to as hallucination or confabulation. Systems or people that are overly reliant on LLMs without oversight or confirmation may end up dealing with misinformation, miscommunication, reputational damage, legal issues and security vulnerabilities as a result of incorrect or inappropriate content generated by LLMs.
10. Model theft
This involves unauthorised access, copying or exfiltration of proprietary LLM models by malicious actors or APTs, potentially leading to economic losses, compromised competitive advantage and potential access to sensitive information.
AI security testing from Kroll
With AI’s capabilities continually developing, Kroll is focused on leading the security assessment approach for LLMs and AI and ML. We are constantly updating our methodology to align with the latest changes in these technologies, with ongoing investments in R&D in both LLM and AI security. By doing so, we ensure that our services are delivered consistently by consultants who are highly educated in and experienced with LLM and AI.
Kroll has developed an LLM security methodology that aligns with and ensures coverage of the OWASP Top 10 for LLM applications, but also goes beyond that to help clients identify and understand the risks associated with LLM systems in the context of their applications and business.
A Kroll LLM security assessment includes the following elements:
- Dynamic LLM testing: Our consultants use adversarial prompts to discover system behaviour and vulnerabilities within the LLM
- Cloud configuration review: These reviews cover all cloud components, alongside additional validation LLM system and data components.
- LLM developer survey: This provides valuable background information about the model, training data and process and system components.
- Web application pentest: LLM security assessments are conducted as part of a web application penetration test which comprehensively assesses the application and prevents any application vulnerabilities in non-LLM components from impacting LLM systems.
Learn more about our AI security testing services
