Stanley Bak, an associate professor in the Department of Computer Science at Stony Brook University, led a team of researchers who designed a software tool that placed second and won Best Newcomer Tool award at the 2026 International Verification of Neural Networks Competition (VNN-COMP).
The tool, named vibecheck, competed with other entries under the guidelines of formal verification, which is the use of mathematical methodology to prove that a neural network satisfies a specified safety property for all possible inputs in a defined range.
Think of a formal verification tool as a test for your brain’s functionality.
More specifically, a neural network is a kind of AI program with a design inspired by the human brain. Both contain individual units, designed like neurons, which receive inputs and work together in a network to process that signal and decide on a finalized output to pass along.
VNN-COMP scored participating verification tools with benchmarks, sets of sample problems given for the tools to solve.
“The way the competition works is there’s not one algorithm that solves everything, ” Bak said. “There’s a collection of algorithms, and you have to use them for different parts of the computation together.”
Bak started working on perfecting the tool to do well on one specific benchmark.
“Quickly I was able to design a subset of the algorithms that were needed and actually do very well on that specific benchmark. And so in many cases it was also able to sort of match the state of the art,” he said. “So this sort of snowballed into, okay, why don’t we just try supporting all the benchmarks? I knew the competition was coming up in about two months from when I started. So it’s a little bit crazy to think that that would be enough time, especially since all the other tools were developed over many years.”
Despite the quick turnaround, vibecheck managed to win second place overall, only behind Alpha-beta-CROWN, who has placed first for the last five years of VNN-COMP, which has only been around for seven.
Instead of typing each line of code, vibecheck implemented “vibe coding,” where an AI, in this case Claude AI, is prompted to generate the code by the developer.
“It’s almost like working with a graduate student or working with a team where I’m not going to read every single line of code they write. Instead I demand evidence of correctness. You’ve implemented something? Okay, show me the results of how it works,” said Bak.
But Bak also expressed the major differences between working with one of his graduate students and working with an AI.
“When a grad student makes an obvious mistake, they will feel guilty, and they’re less likely to make an obvious mistake. With an AI, you don’t expect it to do that. There were all these incidents where it would make a mistake and then it would investigate it and then it would correct it. And then it would still have a mistake. I remember one time it told me it was providing a correction on its correction. If this were a student, I’d be like, what are you doing?”
While a supervisor on the AI to write the code proved successful for Bak and his team, he’s skeptical on how far researchers in the field should rely on this new technology.
“AI-based coding, it’s spectacular at producing products. Like if you’re producing an essay. But the goal of the university is to educate people. In the past, if you could write lots of great essays, then you’ve learned something about the writing process. Whereas now AI generates all the essays, you haven’t really learned anything. So for software engineering, it’s the same thing. If all you’re doing is using AI to just write the software, you haven’t really learned computer science, and that’s problematic.”
“So the fear is, if we just do nothing about this, there’ll be this missing layer of people who know how to design programs,” Bak said. “There’ll just be this huge gap of expertise that’ll be gone. That hasn’t happened yet. We know it sort of exists and we’re adapting to try to make sure we don’t lose this educational aspect, but it’s tricky.”
— Mattia Stradella
Add comment