Things go wrong on the Internet all the time. Software and hardware break in unexpected ways, bad guys worm their way into places they shouldn’t be, and private data winds up being not-so-private after all. But most of the time, things work the way they’re intended and the network hums along, overcoming minor--and sometimes major--problems without many people noticing.

The problem with things working the way they’re intended is that people tend not to notice. Attention usually only arrives when something goes haywire, and that’s of course what happened last week when the Meltdown and Spectre attacks on vulnerabilities in most modern processors were made public. These are clever side-channel attacks that allow an attacker to read memory from processes in ways that shouldn’t be possible.

“These hardware vulnerabilities allow programs to steal data which is currently processed on the computer. While programs are typically not permitted to read data from other programs, a malicious program can exploit Meltdown and Spectre to get hold of secrets stored in the memory of other running programs. This might include your passwords stored in a password manager or browser, your personal photos, emails, instant messages and even business-critical documents,” the description of the attack says.

These vulnerabilities are quite serious and they affect processors made by Intel, AMD, and ARM, and if you own a computer or handheld phoning device, then they probably affect you in one way or another. The bugs have been lurking in various processors for many years, and as Andy Greenberg explained in Wired this week, they could have been found at any time in the last two decades. In fact, the bugs were found several different times. But, in a crazy confluence of events and brain power, each of those discoveries occurred within the last few months and each was independent of the others. Paul Kocher, a highly respected cryptographer and security researcher, was one of the people who discovered the Spectre attack, and he told Greenberg that the group of nearly concurrent independent discovered was “a crazy coincidence”.

Coincidences like that are more common than you might think. Bug collisions--when more than one researcher (or attacker) finds the same vulnerability around the same time--happen from time to time, but they’re almost never as high-profile as Spectre and Meltdown. The seriousness of these vulnerabilities and the wide variety of vendors and systems affected put this situation in a category all its own. The hardware vendors would need considerable time to develop fixes and notify major customers and partners, and major software vendors such as Microsoft and Apple also had to be notified. And all of this needed to be done as quietly and quickly as possible, requirements that are pretty hard to meet these days.

In fact, rumors about a serious hardware problem had begun to spread in the security community as early as last summer’s Black Hat conference. Details were scarce, but high-level security folks at big vendors said something major was on the horizon. More specific rumors began circulating last week, pointing to a problem in Intel processors. While all of this was going on, in the background, dozens of people across the industry were working not only to fix the problem, but to coordinate the public disclosure of the information. The latter effort, while not as technically glamorous as finding and fixing the vulnerabilities, is equally important, especially in situations of this scale. If one researcher or vendor decides to go public before everyone else is ready, it can have disastrous effects for the other vendors involved, as well as for millions of users.

Somehow, improbably, it worked. The researchers and vendors involved ended up lifting the embargo a few days earlier than planned, but by then the response was set and patches were ready. The amount of effort involved in getting this done should not be underestimated, though. Take Microsoft, for example. In its guidance on this situation, the company said that 41 of the 45 currently supported versions of Windows already have fixes for Meltdown and Spectre.

“Just want it to sink in how utterly massive this effort was for all in the embargo, by just this one highlight of one vendor,” said Katie Moussouris, a former Microsoft security engineer who helped coordinate disclosure for hundreds of vulnerabilities there.

This coordinated disclosure process is difficult and includes dozens of moving parts. If any one of them fails, the whole thing falls down with potentially catastrophic consequences for vulnerable users. The fact that it worked so well in this case is impressive and a testament to the dedication of the people involved.