January 31, 2018, was a bitterly cold night in the Shirobad neighborhood of Tehran. In an exquisitely timed and synchronized operation, Mossad agents broke into six heavy steel vaults containing a vast trove of top-secret information that showed not only that Iran had once had a military nuclear program, but that it still did. As Yonah Jeremy Bob and Ilan Evyatar wrote in their recent book Target Tehran, the event constituted “perhaps the largest physical heist of intelligence materials from an enemy capital in the history of espionage.”

The morning hours of October 7, 2023, provided a very different picture of Israel’s intelligence establishment. A few thousand Hamas terrorists used drones and various low-tech measures to neutralize Israel’s sophisticated border surveillance and automated fire systems, swarm across the border, and carry out a massacre of unprecedented barbarity. The marauders broke into Israeli territory in more than 30 places, overrunning more than 22 villages, kibbutzim, and military bases. They penetrated as far as Ofakim, a city 15 miles into Israeli territory, halfway to Beersheba and a third of the way to Israel’s nuclear reactor at Dimona.

Much like the day the horrific Yom Kippur War broke out almost exactly 50 years earlier — also as the result of an inexcusable intelligence failure — October 7, 2023, will remain etched in the psyches of Israelis of this generation. Even worse, the IDF was woefully unprepared. The chaos in the early hours was lethal. It took more than a day before the attack was largely countered and a few days more until it was fully terminated. To the IDF’s credit, it wasn’t long before it rallied and got its act together.

How can one explain such colossal intelligence and operational ineptitude? There are various explanations that account for much of what happened, but none are truly satisfying. Let me give it a try.

Like all great strategic fiascos, October 7 was the result of many factors, first and foremost a failure of imagination.

The same thing has happened before. Russia was taken by surprise in 1941 because it had just signed a nonaggression pact with Germany and Stalin didn’t think Hitler would open a second front. Israel was taken by surprise in 1973 because it thought Egypt and Syria would not start a war without air superiority, which both adversaries lacked. America was taken by surprise on September 11, 2001, because it failed to imagine that someone would fly commercial airliners into skyscrapers and government buildings. In each of these cases, much of the raw intelligence was there. But it was discounted by intelligence, military, and political leaders because the possibilities were unthinkable.

Before October 7, the leaders of Israel’s intelligence establishment did not think that Hamas had anything near the capabilities required to pull off such a well-executed attack. Conversely, Israel correctly assessed that Hamas would not start a war without first gaining a commitment by Iran and Hezbollah to join them in the effort. What Israel did not know is that Hamas believed — mistakenly — that it had such a commitment.

A second factor was the overabundance of data, or what has been called the “noise-to-signal ratio,” in which the false signals, or “noise,” of raw information drown out the accurate signals that convey meaningful intelligence.

To an extent, the system did work. A low-level analyst in Unit 8200, Israel’s primary agency for collecting signals and cyber intelligence, succeeded in bringing her contrary assessment of Hamas’s intentions directly to the head of the unit and even to the head of Military Intelligence (MI). Unit 8200 got its hands on Hamas’s plan of attack more than a year before the war. Somehow, the chiefs were unaware of some of the information and blithely dismissed other parts. It simply did not fit with the overall assessment. Instead, there was an overreliance on fancy technology, including AI, which generates vast quantities of intelligence, much of it valuable, but without a comparable emphasis on the capabilities required to analyze and understand it.

Then there was wishful thinking.

In the years and months prior to October 7, Israeli intelligence had bought into Hamas’s intentionally misleading messaging, which indicated that it was increasingly focused on domestic affairs and economic reconstruction rather than its fundamental jihadi objective of Israel’s destruction. Not that anyone thought Hamas had given up on this long-term objective, but the dire economic straits in Gaza suggested that its priorities had changed. In the meantime, the IDF was focused on the growing terrorism in the West Bank during the previous months, at least some of which was part of Hamas’s overall deception plan.

That deception fed into the so-called conceptzia, Netanyahu’s specious conviction that the Palestinian issue could be sidelined for the long term by maintaining an ongoing separation between the West Bank and Gaza and strengthening Hamas’s rule in the latter. He also believed that Israel could enjoy peaceful relations with ever more Arab countries irrespective of the Palestinians, and he discounted evidence to the contrary.

A fourth factor was distraction.

In the months before October 7, the entire national leadership, political as well as military, was consumed by the domestic battle over Netanyahu’s judicial overhaul. Opposition to the plan erupted in the IDF’s most elite units, primarily among reservists, but also among conscripts and the professional military. The brass repeatedly warned Netanyahu of the devastating impact on the defense establishment, but, not for the first time, he ignored them. Once the war broke out, these units fully lived up to their responsibilities, with an overflow of reservists, including the leaders of the protest movement, immediately showing up for service. But things had looked very different just days earlier.

Another element in the failure is Israel’s long-standing decision-making pathologies, including an understandable predilection for micro-tactics, for dealing atomistically with the immediate issue, and for politicization.

For more than two decades, Israel sought to “mow the grass” as a way of keeping the Hamas threat down to size, rather than addressing the fundamental challenges it posed. Twenty years ago, Hamas was a small threat. Postponing the day of reckoning may be a human trait, but it has served only to exacerbate initial problems and turn them into a bigger one. Israel now faces a similar dilemma regarding Hezbollah in Lebanon, to say nothing of the regime in Tehran.

On a more mundane level, the IDF is constantly overextended, forced to cope with the entire gamut of possible military threats, from primitive but at times lethal rock-throwing, to sophisticated cyber attacks and nuclear programs. It thus has no choice but to conserve resources and take risks that might otherwise appear foolhardy. Personnel limitations are a particular problem, and outsiders would be stunned by how thin Israel’s peacetime deployments of forces have almost always been. Hamas was clearly aware of this, as were Egypt and Syria in October 1973. The postwar conclusion at that time was that Israel had to build a much larger military and deploy far greater forces. This rapidly proved both economically ruinous and an untenable burden on the reserve forces, as it will if a similar conclusion is reached once again. Moving forward, Israel will have to deploy somewhat greater forces, but it has no choice other than to continue relying primarily on intelligence and improved defenses.

All of this raises a final question, which really amounts to an unsolvable dilemma: Should intelligence assessments be based primarily on enemy capabilities or enemy intentions?

Intelligence agencies get it right most of the time — provided that developments fall within the bounds of the familiar. But they repeatedly fail when it comes to fundamental deviations thereof. Beyond the above examples of military surprises, one can readily add both the American and Israeli failures to anticipate the Arab Spring and the fall of the shah in Iran. A long-term reliance on assessments of capabilities is ultimately safer, but nearly impossible; one cannot remain fully mobilized for all eventualities, all the time. Intentions, on the other hand, are notoriously difficult to assess. Defensive capabilities must, therefore, be robust and designed to withstand surprises. This, however, brings us back to the issue of costs, leaving military officials and national leaders in an inescapable bind.

As with all strategic failures, there will be a tendency to look for quick fixes as well as substantial organizational changes. After the Yom Kippur War, new intelligence-assessment divisions were established in the Mossad and Shin Bet to complement and at times provide contrary assessments to those of MI. A short-lived proposal also called for the appointment of an adviser to the prime minister for intelligence affairs. After the 2006 war with Hezbollah, Israel’s National Security Staff (NSS) was charged with new responsibilities in the intelligence area. A more recent proposal called for a new bureaucratic body similar to the Office of the Director of National Intelligence established by the Bush administration after 9/11 to coordinate the work of all 16 American intelligence agencies.

With just three agencies (four if one includes the foreign ministry’s assessment division), the Israeli intelligence community is far smaller than America’s. It is therefore questionable whether there is a need for a major organizational fix, such as adding another level of bureaucracy between the agency heads and the premier.

A number of more limited and easily implemented reforms might be made. These could include strengthening the role of the premier’s military secretary by increasing the handful of intelligence officers already working for him; providing the NSS with a clearer intelligence oversight role; placing even greater emphasis on highly trained Arabists, human intelligence, and good-old signals intelligence, not just the “sexier” cyber; and developing closer ties between intelligence collection and the analytical side.

A more substantial proposal would be to significantly broaden and upgrade the role of the “Varash” (a Hebrew acronym for the Committee of the Heads of Intelligence Services: the chiefs of MI, the Mossad, and the Shin Bet), which in recent years has also included the head of the NSS, the prime minister’s military secretary, at times the premier himself, and which might also include the foreign ministry’s senior official. This long-standing and important forum meets regularly for purposes of interagency coordination but has yet to be institutionalized in statute. This should be changed, and an expanded version of the forum might also become the nucleus of an Israeli National Security Council. (The existing National Security Staffis just that: a support staff for the prime minister and the cabinet.)

Some believe that an entirely new civilian intelligence agency should be established in order to change the current situation in which MI bears primary responsibility for the national intelligence assessment in all areas: military, diplomatic, and socioeconomic. The role of MI is, indeed, an anomaly in the democratic world, in which civilian intelligence agencies usually bear this responsibility. Under this proposal, in addition to responsibility for the national intelligence assessment, the new bureaucratic body would also be responsible for the critically important national intelligence-collection plan, as well as for all intelligence budgets, which it would allot to the different agencies on this basis. In so doing, the new agency would provide for greater intelligence coordination and integration and would be an extremely influential new actor. A variation on this proposal might be to provide Varash or the NSS with the bureaucratic capabilities necessary to fulfill these functions.

“Concept” has been a dirty word in Israel ever since the intelligence failure of the Yom Kippur War. But intelligence agencies — and indeed, every one of us — have no alternative but to operate on the basis of preconceived conceptual frameworks. Imagine if this were not the case and we had to start from scratch every time an intelligence report came in, forcing us to ask: What is Hamas? What are its beliefs? Who are its leaders and what are their personalities? How capable are its forces? What are their intentions?

In practice, we all assess information and attach importance to parts and discard others on the basis of our existing conceptions. Conceptions are critical but become counterproductive when they are so deeply entrenched that they blind us to a changing reality, as happened in October 1973 and October 2023.

The Hebrew word for spying, rigul, derives from the three-letter root rgl, meaning foot, because in ancient times spies did much footwork to follow their adversaries and collect information on their capabilities and intentions. Biyun, which in modern usage refers to espionage, was based on the root word for understanding. Technology has changed over the millennia, but intelligence officials face the same human frailties and decision-making pitfalls they always did. They need the necessary information, but also the wisdom with which to understand it.

There is no foolproof solution to intelligence failure, and Israel needs far more than intelligence reform. It must rethink its overall national-security strategy and policies and elect a new national leadership with the ability to think afresh about all of Israel’s proliferating problems. As Daniel Patrick Moynihan once wryly put it, “Intelligence is not to be confused with intelligence.”