Is an open-source AI vulnerability next?

AI has captured widespread interest and offers numerous benefits. However, its rapid advancement and widespread adoption raise concerns, especially for those of us in cybersecurity. With so much interest, there are lots of insecure applications finding their way onto our devices and other endpoints, opening more pathways for the “bad guys” to steal our data.

Applications developed within open-source communities often face more significant security challenges because they are free and widely available, supported by volunteers, and because of other considerations. Even if a major open-source AI project hasn’t already been compromised, it’s only a matter of time until it is.

So, let’s explore why open-source AI security is lacking and what security professionals can do to improve it.

Back to the future: AI is software

First, it’s essential to acknowledge that AI is not something different from software; it is software. As such, it is part of the operation of IT systems and thus part of the software supply chain. AI should be treated the same as any other piece of code or artifact.

Of equal note, software supply chain security is not purely about web applications, command-line tooling, or other things that are most often thought of when referring to software. It protects every component and process as organizations develop, distribute, and deploy software. You can think of it as cybersecurity applied to the system delivery lifecycle (SDLC). Every stage of software development – from coding and building to production, deployment, and maintenance – is involved and needs to be secure.

What can go wrong in the AI software supply chain?

The challenges within the AI supply chain mirror those of the broader software supply chain, with added complexity when integrating large language models (LLMs) or machine learning (ML) models into organizational frameworks.

For instance, consider a scenario where a financial institution seeks to leverage AI models for loan risk assessment. This application demands meticulous scrutiny of the AI model’s software supply chain and training data origins to ensure compliance with regulatory standards, such as prohibiting protected categories in loan approval processes.

To illustrate, let’s examine how a bank integrates AI models into its loan risk assessment procedures. Regulations mandate strict adherence to loan approval criteria, forbidding the use of race, sex, national origin, and other demographics as determining factors. Thus, the bank must consider and assess the AI model’s software and training data supply chain to prevent biases that could lead to legal or regulatory complications.

This issue extends beyond individual organizations. The broader AI technology ecosystem faces concerning trends. Recent research indicates an inverse relationship between the security posture of open-source AI software tools and their popularity. Put simply, the more widely adopted an open-source AI tool or model, the greater the security vulnerabilities it may possess.

Furthermore, the prevalence of open-source AI models trained on potentially illegal or unethical data poses significant legal and regulatory risks for users. This exposure highlights the imperative for enhanced measures within the AI supply chain to guarantee safe and secure usage. While the future of AI holds promise, addressing these challenges is paramount to its responsible adoption and sustained success.

What security professionals can do

Securing open source requires focus across multiple avenues including:

• Security specifications: Advocate for greater transparency and accountability within the open-source community, demanding essential security metadata such as Software Bill of Materials (SBOMs), SLSA (Supply Chain Levels for Software Artifacts), and SARIF (Static Analysis Results Interchange Format).
• Open-source security tools: Collaborate with companies that offer support for security projects, such as Allstar, GUAC, and in-toto attestations, to bear some liability while still benefiting from open-source innovation.
• Industry contributions and funding to open source: Supporting organizations like the Open Source Security Foundation (OpenSSF), which develops specifications, tools, and initiatives to secure critical open source projects, is essential.

CISOs and their security teams need information about the software in their organization’s environments to ensure its security. With this information, CISOs can make informed, risk-based decisions about the software components they integrate into their environments. Relying on volunteer efforts for security without contribution or investment is unsustainable and ineffective.