Security analysts at Google are developing a framework that they hope will enable large language models (LLMs) to eventually be able to run automated vulnerability research, particularly analyses of malware variants.

The analysts with Google’s Project Zero – a group founded a decade ago whose job it is to find zero-day vulnerabilities – have been working on Project Naptime for about a year with the goal of enabling LLMs to conduct their research in a way that closely mirrors the “iterative, hypothesis-driven approach” of human security professionals, they said.

“This architecture not only enhances the agent’s ability to identify and analyse vulnerabilities but also ensures that the results are accurate and reproducible,” Project Zero analysts Sergei Glazunov and Mark Brand wrote in a blog post. “While modelling a human workflow is not necessarily an optimal way for an LLM to solve a task, it provides a soundness check for the approach, and allows for the possibility of collecting a comparative baseline in the future.”

(The analysts gave the project the name “Naptime” because LLMs with the ability to do such vulnerability research would give analysts the chance to take naps while the AI system helps with their work.)

Taking the Torch from Meta

Project Zero’s work builds on work done by Meta to benchmark LLMs’ capabilities of finding and exploiting memory safety issues. The Meta researchers found that none of the LLMs did well on the challenges presented in its CyberSecEval 2 study, adding that the “average scores of all LLMs over all tests suggests that LLMs have a ways to go before performing well on this benchmark and aren’t likely to disrupt cyber exploitation attack and defense in their present states.”

However, Glazunov and Brand wrote that Google analysts refined the test’s methodology to leverage modern LLM capabilities and were able to see “significantly better performance” in discovering vulnerabilities. They were able to generate benchmark performance in CyberSecEval 2 that was 20 times what Meta saw.

“When reviewing the existing publications on using LLMs for vulnerability discovery, we found that many of the approaches went counter to our intuition and experience,” they wrote. “Over the last couple of years, we’ve been thinking extensively about how we can use our expertise in ‘human-powered’ vulnerability research to help adapt LLMs to this task, and learned a lot about what does and doesn’t work well (at least with current models).”

It’s the Principles of the Thing

They mapped out a set of principles designed to take advantage of LLM strengths while acknowledging their limitations. These include allowing the AI models to run extensive reasoning processes, apply interactivity within the model, and using specialized tools like debuggers and scripting to better mimic in environment human security experts operate in.

“For instance, access to a Python interpreter enhances an LLM’s capability to perform precise calculations, such as converting integers to their 32-bit binary representations – a sub-task from CyberSecEval 2,” they wrote. “A debugger enables LLMs to precisely inspect program states at runtime and address errors effectively.”

Others principles were automatically verifying solutions with absolute certainty and a sampling strategy to explore multiple hypotheses through multiple independent trajectories.

The Google analysts created a specialized architecture for Naptime that included task-specific tools for improving the LLM’s ability and ensuring automatic verifications of the results. The focus is the interaction between an AI agent and the codebase it’s targeting. The tools – a code browser, debugger, a Python tool for running Python scripts in a sandboxed environment, and a reporter to enable the AI agent to communicate its progress – are “designed to mimic the workflow of a human security researcher,” the analysts wrote.

Naptime is focused on vulnerabilities in C and C++ code and is designed to find advanced memory corruption and buffer overflow vulnerabilities.

The Right Tools are Key

What the analysts found was that, given the right tools, current LLMs can start to perform basic vulnerability research, though they have a way to go before running autonomous offense security research.

“As we’ve said many times – a large part of security research is finding the right places to look, and understanding (in a large and complex system) what kinds of control an attacker might have over the system state,” Glazunov and Brand wrote. “Isolated challenges do not reflect these areas of complexity.”

Solving such problems is more akin to how security pros use targeted, domain-specific fuzzing that is run as part of a manual review workflow.

“More importantly, we believe that in tasks where an expert human would rely on multiple iterative steps of reasoning, hypothesis formation, and validation, we need to provide the same flexibility to the models,” they wrote. “Otherwise, the results cannot reflect the true capability level of the models.”

That said, the work isn’t done. Project Zero will continue working with their counterparts in Google’s DeepMind AI unit and across the company on Naptime.