AI goes rogue and then crazy

Welcome to our fortnightly newsletter. This week, our research assistant Harry Craigie is writing about the vast gulf between the way China and Australia look at and use AI; Alena is looking at the different ways our social media for youth ban is being adopted around the world, and we have a new podcast to share. Michael is talking to Paul Keller, the co-founder and director of policy at Open Future, a European think tank developing new approaches to a safe and open internet. Michael and Paul discuss whether there are ways other than through copyright licencing to deal with the impact of AI on the information ecosystem.  

As we detailed in our last newsletter, the government has announced it plans to take a stronger interest in AI policy and will establish a new Office of AI – which sounds grand and serious until you consider all of the government’s recent attempts to wrangle with AI. Lest we forget the proposal to introduce mandatory guardrails for high-risk AI use to embed fairness, accountability and transparency obligations on AI providers. It was abandoned. There was also an AI expert body established to advise government on what to do to protect us from the worst aspects of AI, but it too was abandoned. In their stead came the National AI Plan which essentially committed Australia to doing nothing beyond what current laws permits to deal with any risks.

And then along comes the mother of obstructions to keeping us safe from the worst of AI capabilities – a new, if predictable kind of security incident. Two OpenAI models – one existing and one unreleased –semi-autonomously hacked into a digital library of AI technology known as Hugging Face, used by developers and housing information on millions of AI models. OpenAI has said previously that its models have the capacity to expose cybersecurity problems and that these exposures could happen faster than networks defending against them can fix. That’s now happened.

OpenAI said the test they were conducting to see how the two models could together link online vulnerabilities into a successful cyberattack was meant to occur within the safety of a testing sandbox. Humans set the objective and created the unsafe conditions. But they didn't prompt the models to target Hugging Face. The models found a way out of the sandbox and onto the internet where it located Hugging Face because OpenAI testing guessed the site could tell them more about how to bypass security protocols given how many models it held.

OpenAI is apologetic, of course: ”We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of”. It is working with Hugging Face to fix the damage.

But if the vulnerability had been found or created by bad actors, working on bypassing security at, say, a banking site, the AI capacity becomes very scary. Even if, as some seem to believe, that this OPENAI incident was a marketing exercise to show the power of the company's latest model,  the new Australian Office of AI will need  strong powers to ensure against future breaches but also to make sure when AI companies are testing, the sandbox doesn't prove to be as easy to escape.

Read the newsletter.