Something's been bugging me

Over the past weeks, the Bitcoin community has come to realize what a few nerds in the space already suspected/knew for a while: most of our software, whether written by AI or humans, is buggy and broken in a bunch of ways. It’s unfortunate that this realization, and large scale mitigation efforts, only came after hundreds of millions of dollars (thousands of BTC) were recently stolen¹ with minimal cost to the attackers - presumably only hundreds of dollars in token spend.

Why are we only now seeing these exploits? My theory is that Bitcoin software (Bitcoin Core, Lightning, Liquid, …) has historically not been on the radar of threat actors, or not their priority, because the surface area of the wider cryptocurrency industry is much larger, and the cost of discovering vulnerabilities there was lower. Why invest in breaking a hardened codebase like Bitcoin Core when there is an ever-growing supply of less mature smart contracts to attack instead? I think this theory is supported by the amount of money stolen (from e.g. EVM bridges alone), as well as the scale of the smart contract auditing industry. Looking at recent examples, the game has obviously changed, a model pointed at hardware wallet firmware may quickly uncover something as consequential as a flaw in seed generation. The costs have clearly fallen dramatically.

The LLM scanning response

LLM vulnerability scanning has, unsurprisingly, become the dominant community response with hundreds of thousands of dollars poured into tokens for inspecting Bitcoin open source projects. This is great, as the exploits mentioned above clearly demonstrated the urgency, and I expect the responsible disclosures of critical findings to be published over time. For the Bitcoin Core project in particular (since I have some insights there), it is scanned by project Loupe, receives reports from the “Bitcoin Red Team”, and also has several contributors conducting scans of their own². These efforts have produced more than 1,000 reports. In my experience, most are hypothetical internal API issues or false positives, some identify real bugs, and only a tiny fraction are security-relevant. Given the volume, the project had to sift through them with LLMs to filter out the noise. No high or critical severity vulnerabilities have been found by these scans in Bitcoin Core thus far, nor have combinations of low or medium severity vulnerabilities led to high/critical impact.

If you ask me, despite the LLM scanning efforts, it is very likely that the situation will get worse without a mindset change in the developer community. LLM capabilities will go up, their costs go down, and access is unevenly distributed. What has been bugging me is a growing sense that software correctness now hinges on having access to the latest frontier model. That is not a sustainable security model. I think that LLM scanning should obviously be part of any serious project’s security efforts, but making it the primary strategy for a project’s security/correctness will result in a new wave of exploits each time the latest model drops. Projects should use the LLMs but, more importantly, work towards getting ahead of the LLMs, by investing in engineering practices that persist across model generations.

Getting ahead of the models

Two encouraging data points are secp256k1 and Bitcoin Core. During LLM scanning, they had either no issues found, or (as noted above) nothing severe enough to cause widespread disruption or loss of funds: This is awesome because 1) these projects are key infrastructure (dare I say load-bearing), and 2) I doubt it is entirely a coincidence that they have held up as well as they have so far, which means other projects can learn from them.

I am obviously biased given my work, but in my opinion one of (there are obviously more) the biggest contributing factors to the robustness of these projects is their exhaustive, continuous, and automated testing. Fuzzing/property-based testing, in particular, provides a level of assurance you can not achieve with example based testing, or code review alone (human or LLM). Bitcoin Core has spent more than a century of CPU time fuzzing individual functions and critical components (though oss-fuzz, Fuzzor, bitcoinfuzz and individual contributors), and decades of CPU time simulating small networks of full nodes under test (Antithesis, Fuzzamoto). I can’t establish causality from this alone, but my strong suspicion is that the sustained testing effort is an important reason these projects have become so robust.

It took these projects years to get to this point, and the work is still ongoing. Historically, it has been hard to convince projects to invest at this level in these techniques. Fortunately, LLMs are lowering the cost of adoption at exactly the moment when their usefulness is becoming much harder to ignore (Google Trends for “fuzzing”). They can help write fuzz harnesses, identify invariants, and make unfamiliar testing infrastructure easier to understand. LLMs made vulnerability discovery dramatically cheaper. The opportunity now is to also use them to lower the cost of building the testing infrastructure, so we can prevent entire classes of vulnerabilities in the first place.