Researchers in the UK managed to poison an open-weight AI model (open-weight models have their trained parameters publicly released and made free to download – typically being used to run on a firm’s internal infrastructure) for $100 and in about an hour.
“Even when model weights are public (‘open weight’), we have almost no ability to predict its behavior,” they wrote. “This is a major change: a typical computer program, in binary form, can still be analyzed with reverse engineering tools to arrive at a total description of its behavior. With models, we have nowhere close to this capability.”
The skinny here is that this leaves you in a peculiar bind: use a closed-weight model (where you have no insight into the model’s behavior – such as OpenAI’s or Anthropic’s models) and trust that the vendors do everything they can to keep their models safe, or run open-weight models on your own infrastructure but make yourself potentially vulnerable to supply chain attacks (very similar to what we have seen numerous times lately in open-source packages for popular programming languages such as JavaScript).
Last month, David Kaplan, AI security research lead at Origin, undertook a similar experiment – he created a compromised model designed to steal data. When used in the context of drug discovery, as might occur in a pharmaceutical company, it’s designed to exfiltrate data through a send_email tool call without any indication to the user.
