The following piece is part of the U.S.-India AI Fellowship Program’s short-form series.
By: Vaibhav Garg
AI’s software supply chain problem
Artificial intelligence (AI), both generative and discriminative, has many use cases across sectors, domains, and customers. Developers who aim to build associated solutions rarely begin by developing an AI model from scratch; instead, they rely on model repositories or Model Lakes to leverage pre-trained models and libraries like HuggingFace, PyPi, and Kaggle. This is not necessarily unique to AI; generally, developers leverage a significant amount of third-party code when building proprietary products. In the cybersecurity context, these third-party components become a hidden risk for product teams.
Attackers recognize the complexity of addressing this risk and often use third-party components as a way to infiltrate entities. For example, a recent and prominent example of such an attack was by an actor attempting to exploit a backdoor in the XZ Utils utility used by a variety of Linux distributions. AI libraries can similarly be compromised, for instance, by using unsafe serialization. This problem may be exacerbated for AI libraries based on more opaque technologies; for example, attackers may be able to embed malware into model weights.
More pertinently, the risk to open-source models may be driven by broader geopolitical events. One example of this is Protestware, where maintainers or contributors to open-source projects may purposefully compromise underlying code in key upstream dependencies to disadvantage specific entities. This compromises key applications and can lead to national security concerns.
Limitations of security through transparency
Choosing the right AI models and associated libraries thus becomes critical not just for an application’s functionality, but also for larger geopolitical alliances such as the QUAD — a grouping of Australia, India, Japan, and the United States — and their national security concerns. These choices, as made by individual entities in the AI development lifecycle, must be made transparent to downstream consumers. Different researchers have made distinct proposals to address transparency in the AI development ecosystem, including but not limited to the AI Bill of Materials (AI BOMs) and Model Cards.
Manifest’s proposal for AI BOMs consists of metadata related to the model, its architecture, and its expected or intended usage. The Linux Foundation has proposed an AI BOM extension to the well-known System Package Data Exchange (SPDX) format canonically used for more generalized Software Bill of Materials (S BOMs). This extension separates the AI profile from the dataset that the AI is trained on. The Linux Foundation’s standard complements the proposal from Manifest by providing additional information about the model’s performance, ethical considerations, climate impact, and more.
AI BOMs are an extension of Model Cards — first proposed by researchers at HuggingFace. The intention behind Model Cards was to create transparency in the intended use of trained models, along with their performance in the context of potential target populations. The presence of Model Cards correlates with usage on HuggingFace, though researchers note that sections on environmental impact, limitations, and evaluations are often incomplete.
Thus, merely offering Model Cards on Model Lakes may be inadequate to drive a risk-based adoption of these models in AI products. More pertinently, in their current form, they would struggle to address the various geopolitical threats posed by Protestware to institutions such as the QUAD. For instance, current Model Card standards, such as those from the Linux Foundation, do not provide information on key security aspects such as adversarial robustness and lack of malicious manipulation.
Sustainable and secure Model Lake Ecosystems
One solution is for countries as well as institutions like the QUAD to create public-private Model Lakes. At a granular level, these can be created to target specific use cases, distinct sectors, or individual model types. Isolated Model Lakes can be combined to create an overarching Model Lake Ecosystem (MoLE) that covers multiple use cases, sectors, and model types. The governance of such ecosystems can then be multi-tiered where certain aspects are addressed at the ecosystem level and others are managed at the individual lake level.
The participants in these model ecosystems can be divided into Resource Producers (RPs), i.e. the entities that add new models or update existing models, and Resource Users (RUs), i.e. the entities that use the models in downstream AI products, wherein the resource unit would be defined as a model. Resource Producers have an inherent incentive to ensure that the ecosystem is managed appropriately so that they can drive the use of their models. Resource Users are similarly incentivized so that they can be assured of using high-quality models in their products, which — for instance — would not be victim to the next Protestware attacks. Given the alignment of these incentives, these participants can self-regulate to create new Model Card standards that address their security needs and simultaneously ensure that subsequent Model Cards are correctly and completely populated.
One such framework that facilitates this kind of self-governance, in the domain of natural resources such as fisheries, forests, and lakes, is Elinor Ostrom’s Nobel Prize-winning Common Pool Resources (CPR) framework. She identifies five criteria for the successful self-governance of these resources:
The cost of monitoring the resource must be low.
The rate of change in the resource’s ecosystem must be moderate.
Associated stakeholders should communicate frequently and have dense social connectivity to drive trust via social capital.
Non-stakeholders can be excluded.
Stakeholder support for both monitoring and associated rules enforcement.
In the context of Model Lakes, Model Cards provide a way to establish social capital for the models (and by extension the model contributors) to the model’s users. Monitoring entails ensuring that the information on the Model Cards is 1) relevant, 2) correct, and 3) complete. Stakeholders can support this by reporting Model Cards that do not meet these characteristics. Non-stakeholders can be defined as entities who either do not provide Model Cards or provide Model Cards of low quality. They can be excluded by having their cards removed, having their models deprecated, or having their access to the Model Lake itself revoked. The rate of change can be kept moderate by ensuring an upper bound on the number of new models being added and older models being removed, as well as providing a lineage between models, and grouping associated models together.
The creation of such Model Lake Ecosystems, which allow for self-governance via Ostrom’s CPR framework, offers several benefits in terms of managing risks in the AI Supply Chain:
Stakeholders can define and address the scope of the risks relevant to them. Thus, if Protestware is a relevant concern, then stakeholders would address it.
Governance will evolve with the nature of threats. This is important in AI as both the nature of AI and its associated threats are evolving quickly.
It allows stakeholders with fewer resources to have equitable participation in the management of the ecosystem.
Associated norms or rules will be less likely to impose cosmetic obligations, as the rules will be made by the entities that will bear the cost of compliance.
Governance based on the inherent extant incentives of the resource participants will be inherently non-coercive and thus should result in fewer instances of non-compliance.
Conclusion
While the opportunities and benefits of AI are many, the underlying risk is often obscured by the complex nature of the AI Supply Chain. Addressing these risks is paramount, especially when they impinge upon national security interests. For the United States and India, these interests may include ensuring that dominant AI ecosystems are imbued with democratic values and support for human rights. They would certainly include ensuring that reliance upon AI products does not create systematic invisible risks across critical infrastructure.
A US-India partnership to create Model Lake Ecosystems for critical national security use cases in AI may shed light on less visible risks and allow them to be addressed from the ground up by the stakeholders with vested interests. Channeling Ostrom, we must look beyond markets and states to those who hold both the knowledge and the capability for sustainable governance.
Vaibhav Garg is part of the U.S.-India AI Fellowship Program at ORF America. He is currently the Executive Director for Cybersecurity Research and Public Policy at Comcast Cable.