It's time for (r)evolution

Notes from Charl van der Walt's keynote at Insomni'hack '24.

In the weeks ahead of my keynote at Insomni'hack '24 in Switzerland last week (https://insomnihack.ch/talks-2024/#N7IFRX), I took the time to write some of my thoughts. It was a creative exercise, rather than a technical one, but I thought it would be worth sharing here for those who didn't make the talk, and also for those who did.

“They built us some toilets here, but we can't use them because there's no water" - Elsie Hanse

Pearl Harbour

I love to surf at the beaches of False Bay. Its jaws gulp south from Cape Point to Cape Hangklip to swallow the worst winds and swells of  the Atlantic Ocean, leaving only gentle and predictable waves for the city’s surfers to ride.

The Save our Seas foundation have their offices across the way from me, so that I can peer through their kitchen windows as I write this: “False Bay is home to one of the world’s largest white shark populations and a growing human community” their educational website explains, “This creates a number of challenges for both people and sharks”.

One big challenge for people .are surfers is to not get eaten by a shark, as this can severely foreshorten an otherwise enjoyable session. The idea is so distasteful that surfer friends who visit from abroad will refuse to enter the water, no matter how tempting the sets might be. I try to explain that they are more likely to perish in the car on the way to the beach, then in the jaws of a 6 meter, 2 tonne, 300 toothed eating machine, but that never seems to placate them.

We can’t blame them. That response is a function of the “availability heuristic” and is another real problem for people. The cognitive bias leads people to overestimate the likelihood of events that are more readily available in memory, which are often those that are dramatic, vivid, or emotionally charged. News stories and media coverage about surfers with disjoined limbs are more dramatic and receive more attention, making them more available in people’s memories. Fatal car accidents are so common in South Africa that not even my mother shares the stories on Facebook. So my friends overestimate the danger of being a shark Hors de Oeuvre and underestimate the risks they face on our roads.

Another very scary idea is “Cyber Pearl Harbour”, which is term first used in 2012 by then U.S. Defense Secretary Leon E. Panetta It’s a powerful metaphor that evokes images of clinical murderous intent, fiery explosions and young people dying in horrible ways. This, suggests Panetta, is the potential consequence of failure in cybersecurity.

The image has had its intended catalysing effect. It creates the backdrop for countless articles and papers on cyber threats, and the frame of reference for strategy and policy debates on cyber risks. It’s great white cyber.

Cynical as it may seem, however, death isn’t the worst thing that can happen to us. In fact, it happens to most people eventually. Dying can be worse than death. The slow decay of vision and hope and values and self, can be worse than death.

“Best get ready for the cyber Pearl Harbour” cautioned Dr Dan Lomas in January 2024, in response to concerns about Russian interference in western elections.

“Cyber Pearl Harbour” and the images it evokes, aren’t the worst thing that can happen to us either. Of course, any unnecessary death is a tragedy, and a cybersecurity failure that results in the loss of human life is an unforgivable.

But there are much more pernicious outcomes to consider, that are also much more insidious. Elections hacking and its implications for democracy is one good example. Like the creeping sands of the desert, the slow but inexorable loss of autonomy, the poignant death of truth and trust and the gasping demise of freedom and democracy themselves, scare me more than Pearl Harbour. And sharks.

Day Zero

The rain finally came on April 9th 2018, just 4 days before “day 0”.

The majority of Africa is desert, and the areas surrounding Cape Town are also semi-arid and impacted by water stress. With tall mountains that capture moist sea air and dump delicious rain on lush green gardens, wealthy suburban residents on the iconic mountain’s flanks barely notice, but the city’s 4 million+ residents are actually acutely dependent on the water collected in 6 major dams that surround the city proper. By November 2017, the middle of the dry summer, dam levels were at 35% and dropping. After several hot dry summers and disappointing winters the dams were running dry, and the city’s taps with them.

Day 0 was projected to arrive on April 12, 2018.

The idea of water actually running out, of taps that refused to fulfil their single, simple purpose, nearly caused a full-scale panic. As summer temperatures soared and contented tourists perfected tans, the city announced and enforced water restrictions. Just down the road from my village, a distant cousin of the fabled Table Mountain spills the water it collects from the moist sea air under the road and into the sea. It is one of Cape Town’s few natural water flows, and traffic on the road backed up in 2018 as residents from near and far heaved 25 litre bottles of fresh water across the road to waiting cars and “bakkies”. My mother bought water on the black market to keep her pet fish alive, while grocery stores enforced a quota on bottled-water sales. The city government expounded on plans for 200 city-wide water distribution points, where 25 litres of water per person would be made available daily when the last drops were eventually squeezed from the taps. We all feared riots and general social collapse.

“Day Zero is the stark reality we face when most taps will be turned off and residents will have to queue for water”  , warned then-Mayor Patricia de Lille. The city implemented draconian water restrictions, and residents did their best to fight the drought at home.

With no fresh rain expected, the city government rang the alarm bells, warning rich and poor alike to prepare for “the worst crisis any city has faced since 9/11”.

9/11

Everyone remembers where they were on September 11th 2001. Ironically, I was in a desert, in Namibia, with my friend and colleague Jaco. Jaco and I were contracted to a Namibian bank headquartered in the small oasis capital of Windhoek to perform ethical hacking and vulnerability assessments for them. The bank was modernizing, and at the time, “modernizing” meant “digitizing”. So the bank had procured an electronic banking solution to replace its old, manual systems. I had been delighted to learn that the entire system had been bought, almost fully functional, “off the shelf”. The system included components for bank tellers, Automatic Teller Machines, back-end and inter-bank switching, credit card transaction acquisition and Internet Banking. They had no real IT legacy to contend with, so the compatible components all plugged together tidily, and seemed to work rather well. I’m certain that we found and reported vulnerabilities during our testing, but I don’t recall that the systems appeared worse than other homebrewed equivalents that we’d tested.

I do recall that someone at the bank must’ve made a mistake somewhere, and I think it was a bad one. Peering in through the cracked open door of a server room late one afternoon, I observed a group of bank technicians and product consultants laboriously capturing financial transactions from a printed transcript directly into the banking software’s backend database. Some part of the digital version of the journal had been lost, and could only be recovered by referring to a paper record that fortunately still existed. The technicians were manually putting someone’s money back where it belonged.

I wondered if the bank’s clients would ever be able to conceive the ridiculous fragility of it. There was no money, there was only the record of money. And that record existed in a system, running on a computer, stored to a database and written to a disk. In a server room in Windhoek. Of course, modern banking is really more complex, sophisticated and robust than that. But also, not really. The world, I realized, was fragile. Incredibly fragile. And even more fragile would be the trust people place in the systems that constitute that world. One glance at what was happening in that server room that night, and the bank’s entire business would be replaced by thousands of shoeboxes under thousands of beds in the sparse desert country. That’s not as funny as it sounds. Investors would panic also. Taking no chances, they would rush to get out of the bank, and maybe other banks, and maybe even the markets - selling shares for whatever they could get and causing the markets to collapse even further. Businesses would fail, jobs would be lost, and economy would shrink. Probably not just in Namibia either, but also in neighbouring countries, the region, and the continent.

Faith is eternal, but trust is fragile.

The very shape of the geopolitical world we lived in turned out to be equally fragile on 9/11. As we stood in the bank’s fresh IT centre sometime during that project, we watched on the mounted televisions as two planes smashed into two towers a million miles away. The impact of that dreadful day on countless people’s lives and stories cannot be recorded on a transcript, and the lives and stories can never be recaptured, whether manually or otherwise.

I also learned that geopolitical status quo is just as fragile as those banking records in Windhoek, and just as vulnerable to the contagious collapse of trust. The world is also intimately interconnected, as the attack we watched on New York that day would impact the lives and stories of millions of people around the world for decades to come, no matter where they were when the planes hit.

Chain Reaction

Risk is a function of both likelihood and impact. Generally, when we think about our cybersecurity posture, we're thinking about likelihood. That is, how probable is it that our security will fail, and that bad things will happen? This is of course a valid and important perspective, but it allows us at times to ignore the other dimension of risk - the potential impact when something bad does happen.

In his book “The Doomsday Machine: Confessions of a Nuclear War Planner”, Daniel Ellsberg recounts how scientists planning the July 16, 1945, “Trinity[CvdW1] ” nuclear test, seriously contemplated the risk that the nuclear bomb could ignite hydrogen in the air and set the whole atmosphere on fire, eviscerating all life on the planet. An Inside Science article also describes an interaction between author Pearl S. Buck and Arthur Compton, the leader of the Metallurgical Laboratory in Los Alamos, about his conversation with J. Robert Oppenheimer – father of the atomic bomb - during the Manhattan Project:

"If, after calculation, [Compton] said, it were proved that the chances were more than approximately three in one million that the earth would be vaporized by the atomic explosion, he would not proceed with the project. Calculation proved the figures slightly less -- and the project continued."

That’s some dark arithmetic. Many have argued that in the context of the extraordinary threats of the time, the slight risk of planetary incineration from nuclear weapons testing was warranted. That may well be true, but it is thankfully seldom the case.

When assessing risk, we must recognise that when the potential impact is high enough - when the potential for harm or damage is so severe - even the slightest possibility of an event occurring cannot be tolerated.

The author Nicolas Taleb makes the colourful distinction between the realities "Mediocristan" and "Extremistan". Mediocristan refers to domains where outcomes are bounded and extreme events are rare, like human height, weight, or exam scores, which follow a normal distribution and don't exhibit wild variations. Extremistan is the reality of domains where outcomes are unbounded and extreme events are more common, like wealth (though not mine), book sales (though not mine), or stock market returns (sadly, also not mine). In Taleb’s view, “citizens” of Extremistan can experience dramatic and unpredictable fluctuations in their reality. Taleb argues that in Mediocristan, risk can be relatively well-calculated using statistical methods because extreme events are rare and typically have limited impact. In Extremistan, however, risk is much harder to calculate due to the presence of "black swan" events—extremely rare and unpredictable occurrences with significant consequences. These events can have a disproportionate impact on outcomes but are often overlooked or underestimated in traditional risk assessments.

Nuclear bombs rely on the rapid and uncontrolled chain reaction of nuclear fission to produce planet-melting explosions. In nuclear fission, a heavy atomic nucleus, such as uranium-235 (U-235) or plutonium-239 (Pu-239), absorbs a neutron, becoming unstable. The unstable nucleus splits into two or more smaller nuclei, called fission fragments, along with two or three neutrons and a tsunami of energy. The released neutrons can then collide with other nearby nuclei, causing a chain reaction where more nuclei undergo fission, releasing additional neutrons and yet more energy.

The rapid chain reaction results in the almost instantaneous release of an immense amount of energy, equivalent to millions of tons of conventional explosives, creating a devastating explosion.

Bombs belong in Extremistan. Chain reactions enable extreme impacts.

The intricate webs of interdependence make complex systems like cyberspace vulnerable to contagion, which places such systems in Taleb’s conception of Extremistan. As Dr Dan Geer puts it in his seminal “Rubicon” paper: “because the wellspring of risk is dependence, aggregate risk is a monotonically increasing function of aggregate dependence”. “Because dependence is transitive, so is risk. That you may not yourself depend on something directly does not mean that you do not depend on it indirectly. We call the transitive reach of dependence ‘interdependence,’ which is to say, correlated risk”.

At Orange Cyberdefense we focused keenly on interdependence as a core systemic factor that shapes the cyberthreat landscape.

“Interdependence” describes how IT systems and the businesses that use them do not operate in isolation. Risk cannot be assessed or managed for a single person, business or country in isolation. This holds true for cybersecurity also, and the impact of a security failure is never restricted to the primary victim alone. It’s not just about upstream or downstream technology exposure. It’s about the fundamental reality that we operate within a complex “web” of technical and business relationships, that make the probability and impact of security failures exponentially larger than we think.

In the interdependent Extremistan of cyber security, here's the key question: When we observe the growing number of breaches, leaks, thefts, extortion & service denials that fill the headlines every week, are we dealing with a set of independent, unrelated incidents that are unfortunate, but limited in their broader impact? Or is there potential for a contagion effect, and if so, what does that look like?

Shifting sands

Back in Windhoek, none of this was yet apparent to me, so our lives and our work there continued.

On one weekend during the project Jaco and I took some time off to drive the road west to the coastal town of Swakopmund. The route took us out through the city’s leafy suburbs and into the surrounding foothills, where the terrain turns rocky and vegetation looks craggy and primordial. Soon, even that meagre growth fades away, leaving nothing but rocks and sand.

And then, inexorably, there is only the sand.

Later in my life I took up endurance running as a hobby, and I sweated on foot across several of the world’s driest places - the Gobi, the Sahara and even Antarctica. Each desert is a wonder in its own way. The Gobi’s terrain was the most diverse, and there we climbed mountains and crossed icy rivers. The Sahara was the hottest, so that at times on the run there I collapsed beneath the tiniest shrub to seek shelter from the 50 degree heat. Many people are surprised to hear that Antarctica is a desert, but I ran there also and slept fitfully on the ice as the acrid odour of penguin guano burned my nostrils and the sound of giant ice shards crashing into the seas shocked me awake every time I heard it. The Atacama desert is even drier than the Namib, and there the endless salt pans that give way underfoot as you run almost broke my ankles, and my spirit. If you stop to pay attention, South Africa’s Little Karoo desert, where I did much of my training, looks how I imagine the first planet we discover with alien life will look. I think it might be my favourite.

But the Namib was my first real desert, and like many firsts in life, the encounter stuck with me.

The desert snakes down the west African coast for 450 kilometres, and the driest regions receive only 2mm of rainfall annually. But the numbers don’t do the experience of seeing it justice. The site of the endless, unbroken red sands evokes a deep sense of awe and a primal fear.

Southern Africa, we’re told, is desertifying. The vegetation is becoming sparser, and the sands are inching further and further outwards. As the planet’s climate changes, whether organically or from the impact of human activity, the terrain is changing in observable ways. The agriculture, industries and habitats that have evolved within a given terrain, are then forced to change also. I realized that our relationship with the planet, and the myriad of interdependencies that exist between people and their natural environment is also very fragile.

As the desert creeped and dams dried up, the people of Cape Town were starting to appreciate just how fragile our water ecosystem is.

If it's yellow, let it mellow

“If its yellow, let it mellow”, read the stickers on every hotel and restaurant restroom wall. It typically takes 9 litres to flush a toilet, 10 litres to enjoy a short shower, and 29 litres for the “economy” cycle of the typical dishwashing machine. The city produced colourful infographics to illuminate these obscure facts, and remind us of our duty to go dry.

My mother-in-law Doreen certainly did her part. At 73 years old, the retired school administrator lived in a simple semi-detached two-story in one of Cape Town’s middle-class suburbs. She had a plastic tub in the upstairs bathroom washbasin. From there, a garden hose ran out the window and down into the yard below.

“Brush teeth and wash hands” consumes 2 litres of water, per the helpful Cape Town city infographic. Thanks to scientific wonders like the gravity and Hydrostatic Equilibrium, Doreen’s tooth-and-face-water would trickle bravely from the basin through the hose, up through a yawning window, and down to the thirsting flower bed in the yard below. Two haggard litres of H20, twice a day, was what Doreen fed her garden plants. She washed with a damp cloth, queued at a nearby spring for her 25 litres, and flushed her toilet only when absolutely necessary.

“Fear works… the city did quite well at preaching the message of saving water, and we halved our water use", said Siyabonga Myeza of the Environmental Monitoring Group, proudly. Countless Doreen’s with countless water hoses were fending off disaster one spit at a time.

But the city’s estimated household water consumption at the time was about 600 million litres per day.. That’s a lot more than tooth brushing. The city’s population was also growing rapidly – from approximately 3,604,000 in 2010 to about 4,618,000 in 2020 (28%) and it’s water demand was growing with it. Doreen and other tooth-brushers were not the cause of the water shortage, and they were also not the solution. Something much bigger was happening.

In high school I had a friend called Gina. We went on Bible camp together, but she was much smarter than me. She eventually became an “Associate Professor, Environmental and Geographical Sciences” at the University of Cape Town, and published papers on themes like “Water governance and justice”. I was delighted when I saw her quoted on the subject of Day 0. “It doesn’t matter how much technical expertise you’ve got”, she said, “but you actually have to stand back and understand the system more broadly”. I told you she was smarter than me.

A broader glance at the water situation readily reveals obvious, systemic causes for the impending Day 0, including alien vegetation (that consumes much more water than native plants), a drought (potentially exacerbated by human-caused global warming), and a shortage of dam storage capacity. The latter was caused by poor planning and insufficient maintenance.

As kid in high school, aside from attending bible camps with Gina, I completed a few weeks of “work experience” with a civil engineering company in the city. The engineers specialized in large infrastructure projects, including dams, and were renowned for their (then) cutting edge application of IT. One of my tasks was to digitally capture rainfall and dam capacity data for a study they were conducting on the growing city’s water infrastructure. In the dimly lit computer centre, I painstakingly keyed in historic rainfall readings from kilometres of rolled up pluviometer tape, and traced out the dimensions of the city’s dams using a delightful mouse-like device that had a small window with crosshairs on it so I could follow the outline of the dams from printed maps. It was 1990 and the dawn of a new age. I actually shifted my career choice from engineering to computers because of that clever little mouse.

I was never involved in any of the analysis of the data I transcribed, but I do recall an engineer speculating to me that the city’s water requirements would probably exceed dam capacity within 10 years. Maybe he said 20 years. Either way, the government knew Day 0 was coming more than 30 years before it finally happened. According to a 2017 article in the local press, “since 1995 the city’s population has grown 79%, from about 2.4 million to an expected 4.3 million in 2018. Over the same period dam storage has increased by only 15%”. The Berg River Dam, completed in 2007, was the city’s only significant addition to water storage since 1995. It also became a nice place for a picnic.

In short, we didn’t have enough dams in the right places. But we didn’t look after the dams we had either. Maintenance and upgrades to dam infrastructure weren’t carried out adequately or in a timely manner, which resulted in leaks, sedimentation, and inefficiencies in water distribution.

The article quoted Kevin Winter, a lecturer in Environmental and Geographical Sciences, who probably worked with Gina at the University of Cape Town. Winter comments on the question of climate change due to human-caused global warming: “rainfall to the city’s catchment areas is coming later, dropping more erratically, and often missing the catchments altogether”. “We have to acknowledge that carbon dioxide is finding its way into the atmosphere and has reached a new high,” he said. “This is a global system, so the bigger systems are beginning to impact us … there is no doubt that pressure and temperature are related. So disturb the temperature, you disturb the pressure and you start to see different systems operating”.

But, fear works.

So Doreen and thousands of other grans across the city were making painful personal sacrifices to stave off disaster. In the long term, however, all their efforts would be futile. They are fighting a hurricane. On a boat that’s too small. With leaks.

And the effort is distracting us.

Running from Lions

“Every morning in Africa, a gazelle wakes up. It knows it must run faster than the fastest lion, or it will be killed. Every morning a lion wakes up. It knows it must outrun the slowest gazelle, or it will starve to death. It doesn't matter whether you are a lion or a gazelle. When the sun comes up, you better start running”. - Thomas L. Friedman

Cybersecurity feels like this to me. Regardless the particulars of our context or our role, we wake up in the morning and we run. We run to counter threats. We run to mitigate vulnerabilities. We run to learn and educate. We run to decide and document. We run to manage crises. We run to integrate a system or solution we’re not even sure we needed.

It’s a cynical scheme.

“As soon as possible and no later than 11:59PM on Friday February 2, 2024, disconnect all instances of Ivanti Connect Secure and Ivanti Policy Secure solution products from agency networks,” the U.S. Cybersecurity and Infrastructure Security Agency (CISA) said in a directive to federal agencies in the civilian executive branch on February 5, 2024.

Ivanti is an US IT security company headquartered in the bucolic Utah town of South Jordan, which was rebaptised by early Mormon settlers in 1859. In 2020, Ivanti purchased a security product called Pulse Connect Secure, which it subsequently rebaptised “Ivanti Connect Secure” (ICS).

“Ivanti Connect Secure is a next generation secure access product, which offers fast and secure connection between remote users and their organization’s wider network. Ivanti Connect Secure modernizes VPN deployments and is loaded with features such as new end user experience, increased overall throughput and simplified appliance management”.

It’s a cybersecurity product, obviously, and its built to be deployed on the exposed perimeter of enterprise’s network, where it can facilitate and orchestrate access by remote workers to the business’ digital crown jewels.

Pulse had been spun off from respected network technology giant Juniper in 2014, and celebrated “building on 15 years of security experience” and “delivering next generation secure access solutions to address the evolving security needs of enterprise customers”. Apparently it could boast of “20,000 enterprises and 80% of Fortune 500 companies” among its clients.

But the acquisition must’ve surfaced some awkward moments. For while negotiators were doubtlessly still posturing and parrying, CISA was urging businesses to “immediately patch CVE-2019-11510 —an arbitrary file reading vulnerability affecting Pulse Secure virtual private network (VPN) appliances”.

I remember wincing when we heard about the Ivanti acquisition, and wondering whether they understood just how much pain they were buying into.

It wasn’t just Pulse though. At Orange, we addressed this worrying issue of vulnerabilities in security products in a section of our 2020 Security Navigator report, discussing issues we’d released security advisory “Signals” over:

“Noteworthy over the last twelve months is also the visibility of several leading security product vendors in the very short list of technology vendors who featured multiple times in our Signals this year.

We noticed a distinctive ‘bump’ that occurred in May this year, where an unusually high number of vulnerabilities was reported in these security technologies. Indeed, there was a four-fold increase in vulnerabilities reported in selected security technologies between March and May 2020”.

We argued in the report that the extraordinary surge in security product vulnerabilities was the function of three factors:

The notable ‘success’ of Pulse Vulnerability, CVE-2019-11510, from May the previous year, which had been exploited in several high-profile attacks.

The rapid and sometimes reckless adoption or expansion of secure remote access capabilities during COVID, to accommodate remote workers, which made these technologies a very attractive target.

A cascade effect in which the discovery of one vulnerability creates knowledge, experience and ideas, and thus leads to the discovery of different vulnerabilities in the same product, or similar vulnerabilities in different product.

“There is no doubt that there is a surge in these kinds of vulnerabilities at this time”, we griped, “which, when combined with the apparent rush to deploy or scale remote access capabilities [to deal with lockdowns], is leaving critical perimeter security exposed and contributing in a direct way to compromises and breaches”. We urged our clients to “ensure that they have the people and processes in place to respond in a timely manner to vulnerabilities in security vendor products when they’re announced, or to engage with a provider that can assist with these functions”.

“Brush teeth and wash hands consumes 2 litres of water”, remember?

In our 2020 report we said nothing about the responsibility of vendors like Pulse to avoid repeats of this kind of crisis.

“Persistent vulnerabilities, attacks and compromises involving cyber security technologies continue to feature prominently in our advisories and are a cause of continued concern to us”, we blushed just a little in our 2021 report, “VMware, Pulse Secure, SonicWall, Citrix, Fortinet, F5, Palo Alto Networks and Juniper Networks have collectively appeared in 56 advisories this year”. “That’s 10% of all the bulletins we issued”.

“We believe”, suggested our ’21 report a little more boldly, that “an industry-wide discussion needs to be had to determine whether the problem is as real as we perceive it is, identify existing efforts that may already be underway to address the issue, or create some form of partnership to work toward a better situation for ourselves and our customers”.

I made a token effort to push this idea of a “forum” between Orange, our vendors and our customers. My idea was to initiate an open conversation about the perceived problem of vulnerabilities in security products, and the challenges service providers like us, and enterprises themselves, apparently faced in understanding and responding to vulnerabilities when they were reported or attacked. I fantasized that such a forum could eventually be codified as a permanent structure that produced best practices and standards, or even some form of assurance or certification for security vendors. And I wanted our customers to have a position of influence from which they could express their needs and requirements to us and our vendors.

My emails and calls about the idea achieved nothing, and my efforts soon petered out. Wind hole. The people I spoke to had lots to keep them busy with, and so did I. We were very busy selling more Ivanti products, for example, and even earned a generous commendation from the vendor for our exceptional sales.

Then, in January 2024, the Orange Cyberdefense CERT issued its first “Critical” threat advisory in over 12 months. Predictably, Pulse Secure was back in the news. Only this time under brand Ivanti. On January 31st, Ivanti had released fixes to address four critical vulnerabilities in their Connect Secure and Policy Secure solutions. Exploratory scans conducted by the CERT revealed that of the 24,986 exposed Ivanti devices detected, 3,254 were vulnerable, 684 had already been compromised, and 680 devices were leaking sensitive data. Indeed, “any asset exposed on Internet that did not apply one of the mitigations yet should be considered probably hacked”, the CERT warned.

Soon reports were flooding in of Ivanti devices being exploited in the wild. Chinese and other state actors were abusing the bugs, but criminal actors were in on the fun also. In fact, who wasn’t? Our Incident Responders frequently encountered the vulnerability when supporting clients who’d been hacked.

Meanwhile Ivanti blundered about in a frantic effort to get the issue under control, but largely failed. Hence the unambiguous directive from America’s CISA.

The idea that a security product that we sold and supported had caused a security crisis so severe that that government agencies were directed to do nothing less than completely remove it, horrified me.

Bailing water

January ’24 was a bad time for Ivanti, but it was a great time for us at Orange, and a boon for the industry in general. Our SOCs had 100’s of managed Ivanti devices to patch, and dozens more businesses approached us for assistance. Our Vulnerability Management teams furiously scanned and reported vulnerable systems. Our detection teams launched emergency procedures to implement suitable detections and threat hunts to scour logs and other artifacts for signs of attack and compromise. Our CERT team conscientiously tracked every acute detail of the developing saga and published updates to a blog post we updated almost daily. We briefed staff and customers and the press. We hosted a webinar to clarify the technical minutiae of the issue and its mitigation. Hundreds of people attended, and the webinar ran 30 minutes over the scheduled time. Our emergency response teams were run haggard responding to cries for help from businesses that had been attacked or compromised or feared they’d been attacked or compromised.

Like trained crew when crisis strikes a yacht, we were all hands on deck. Our partners and competitors, vendors, industry bodies, the media, and government partners all entered the fray with us, and we stood shoulder to shoulder as we grappled to face the threat and minimize the potential damage. Everyone was working overtime, but nobody minded. People cared about the issue, and this was what we did.

We were riding the storm. We were running.

Fear works. But the system isn’t broken, it's working as intended.

Journeys

Ivanti’s travails presented the industry with the perfect no-longer-enough narrative. “Zero Trust”, which is arguably impossible for most to achieve in practice, is the perfect “journey” to sell.

Over the 1st half of 2020, Wicus and I had worked on research for a Black Hat presentation we eventually titled “Virtually Private Networks[CvdW2] ”, in which we asked the question “How ‘secure’ can Remote Access ever be”?

Cleverly subtitled “Virtually good enough” the talk precis and marketing fluff were full of euphemisms, and honestly so was the talk itself. Because the findings of our research were actually quite startling. “While most (yet not all) attacks could be prevented in online/lockdown mode there is virtually no protection at all in offline/standard mode”.

Our findings weren’t earth shattering, and our dire prognostications never materially materialized, but the basic truth seemed clear to me: VPNs were an ancient solution to an ancient problem that didn’t need solving any more. Instead of pivoting to tackle more contemporary threats and TTPs, VPN vendors had invested decades into adding useability and manageability features, frantically chasing new operating systems and versions of operating systems, and creating narratives that would keep their clients on the “journey”. So their features multiplied and codebases bloated. New code was added while old code rotted. Deep in the technology’s bowels.

Our research spawned a few hopeful conversations with engineers and product managers within our business, but the only real response it got… was from marketing. My colleagues in marketing were smart and sincere and diligent and passionate. But they were doing their job. And their job was to breed and nurture narratives. So they did what they’re incentivised to do. They moulded our findings into a narrative, about a journey.

A social media post in April ’24 reminded me viscerally of this dynamic. A security advisory by software giant Oracle highlighted 400 security vulnerabilities across a pages-long list of product families in its suite. “Oracle announces 441 security issues, which is rather stunning” tooted Bert Hubert. “And then slaps a 'Contact Sales' button on that page, which tells you all you need to know”.

Recycling

The January version of the Ivanti crisis eventually faded, but a few busy weeks later in April ’24, CVE-2024-3400 hit. “Critical 0day in Palo Alto GlobalProtect gateway exploited in the wild” the Orange CERT advisory read. Like the cat that glitched the Matrix, we blinked, and crisis was back. A visual risk indicator on the vendor’s security advisory page strained to edge of the red zone, right on severity level 10. Urgency - HIGHEST, Attack Complexity - LOW, Privileges Required - NONE, User Interaction – NONE.

Palo Alto advisory initial suggested that “the vulnerability specifically affects the GlobalProtect gateway and can only be exploited if telemetry is enabled”, but that proved to be wishful thinking. The vendor conceded that disabling telemetry was not a viable mitigation. Blog posts and Proof of Concept code flooded the security internet. And on April 17th our advisory update cautioned that “We expect all exposed vulnerable instances to be attacked in the next hours”.

We had another storm to ride, and an enterprise perimeter security product that we sold and managed was again at the centre of it. We launched into action, like termites fixing a breach in their mound.

We were recycling.

The system isn't broken

The system isn’t broken, it's working exactly as designed.

I was thinking about Day 0 when I proposed a topic for my keynote at Insomni’hack in Lausanne. But there are still lots of water in Switzerland, so instead I proposed to speak about recycling.

“I feel so angry!”, my talk proposal read.

“When I go to the grocery store, I can take my own bags or pay for a paper bag, so my groceries aren’t packed in plastic. But it’s almost impossible for me to find a product that isn’t individually wrapped in some kind of plastic. When I get home, therefore, I spend hours sorting my trash for recycling. I separate the plastic, clean it, and group it by type. I carefully package it and carry it out to a special bin. The bin with plastics is always much more full then any of the others, but only 10% of it will ever be actually be recycled. The rest will end up in a landfill, eventually littering our fields, polluting our oceans and poisoning our food. While we dutifully work away to do our bit for the environment by sorting our trash, all of our efforts to recycle come to nothing. In the meantime, oil, retail and other big industries, with the support of governments worldwide, will produce and use more plastic than ever. Because they’re allowed to, and it earns them more profit.

It’s a cynical scheme. Recycling is a deception designed to keep us busy and distracted by our own sense of guilt and duty, while the system on the whole does what it was always designed to do – generate profits for shareholders – regardless of the impact on the environment and the societies that depend on it”.

It's ok that we all do our part for the environment. It’s necessary. Good. Essential even. But it’s also futile, and a distraction from the cold, hard truth: “In reality, you don’t ever change the hurricane. You just learn how to stay out of its path”. Jodi Picoult wrote for the DC series “Wonder Woman”, and I just found her quote on the internet, but it provides the perfect segue to the next subject.

Riding the hurricane

“It’s out there at sea that you are really yourself.” – Vito Dumas

I rode a hurricane once. For almost 10,000 kilometers from Cape Town to Sydney.

During my life I traversed hot deserts, and frozen deserts. But the ocean is a desert also. It rains sometimes of course, but nothing grows there. And the relentless crushing swells are more desolate and lonely and merciless and frightening than even the endless desert dunes.

In 2023, after years of planning and training, I managed to secure a position on a round-the-world ocean racing yacht for the leg from Cape Town to Sydney. Our 70-foot (22-meter) ocean racer followed the Clipper route, which runs from west to east through the Southern Ocean, riding a system of strong westerly winds called the Roaring Forties. We rode Force 12 hurricane winds for nearly half of the 26-day journey. Force 12 is the highest level on the Beaufort scale, which describes winds over 64 knots, or 118 km/h. “The air is filled with foam and spray; sea is completely white with driving spray; visibility very seriously affected”. Sailing in these conditions is physically, mentally, and emotionally exhausting. Racing – riding on the razor’s edge of disaster to eke out the very last drop of speed from the boat, every hour of the day and night – is even harder.

I was a helmsman – responsible for steering the boat – or a bowman – responsible for the thankless tasks that had to be done at the cold, wet, pointy part of the boat – and a rescue swimmer – who had to go into the ocean on purpose when anyone else goes in by accident. We worked in “watches” of four hours on and four hours off, day and night. I loved those busy and exciting roles. But every crewman also has to do the occasional watch below decks to take care of mundane but essential tasks like maintenance, cleaning and preparing hot drinks. You also have to make an entry in the ship’s log. On the hour, every hour.

A modern racing yacht carries almost as much technology as a spaceship. The boat has radar, a depth sounder, VHF, GPS, a chart plotter, satellite data links and weather charts that are updated every six hours. The charts present a vivid, colour-coded image of exactly what the weather is doing, so that the yacht can seek out and ride the edge of every terrifying storm system that drifts into orbit. The information that all this technology delivers is meticulously recorded in the log. So is the barometer reading.

A barometer is used to measure the air pressure in your vicinity. “Baros” meaning weight, and “metron” meaning measure. Several of our digital instruments reported barometer readings, but we used the traditional analogue instrument mounted on the bulkhead. Among the glowing high-tech of the yacht’s navigation room, it looks anachronistically like an old clock – round and chromed with a glass cover and a mosaic of numbers delicately printed on the face. A dial like a watch’s hand creeps around the face to indicate the air pressure. I always wanted to tap the glass with my finger before taking a reading – because I’d seen that done in movies – but I was told that I shouldn’t.

In the cold vacuum of space there is no air pressure. But the trillions of air particles trapped by gravity to create earth’s atmosphere press down relentlessly on its surface, compacting the molecules to a breathable density and exerting a measurable pressure. Barometric pressure is the force over an area exerted by air in the atmosphere on humans and other objects on earth. Our barometer on the boat measured air pressure in hectopascals (hPa), named for the French mathematician Blaise Pascal. At sea level, the average air pressure is thought of as one “atmosphere” and typically ranges between 970 and 1050 hPa.

When we recorded the barometric pressure hourly for the log, we would also compare it with the previously-recorded level. Per our standing orders, a change of 4hPa or more would require us to wake and inform the captain, no matter the time or circumstances.

In nature, changes in air pressure are big deal.

That’s because air pressure is an infallible predictor of the local weather. As the air rises like a balloon in an area of low pressure, it cools and condenses into clouds, often resulting in rain and storms. In high-pressure areas the air sinks, warming and leading to warm, dry weather. There are invaluable rules of thumb for interpreting the barometer reading. High pressure promises that good weather. Normal pressure at a given location suggests steady weather. Low pressure is associated with warm air and rainstorms. Changing pressure heralds changing weather, and so we would wake the captain up.

Due the sun’s varying impact on earth’s temperature, air pressure isn’t constant across the globe, even at sea level, and it also changes continuously at one place. Differences in air pressure are what makes the wind. In low-pressure areas the warmer air rises, leaving a vacuum that demands to be filled. The denser, colder air in high-pressure areas sinks and compresses, seeking somewhere more roomy to go. So the air rushes from high-pressure to low-pressure areas to equalize the difference, and thus creates wind. Changes in atmospheric pressure therefor predict changes in the wind. Essential to sailors, to deserts and the planet’s wellbeing in general.

But other factors affect the wind also. The tilt of the earth results in an uneven heating by the sun on the northern and southern hemispheres, creating the seasons that orchestrate our lives and the winds that rustle the autumn leaves. Temperatures also vary between day and night, shifting pressure systems around in diverse ways. The earth’s very rotation drags on moving air, to the right in the north, and to the left in the south. They call this Coriolis Effect. Cyclones and trade winds result from the Coriolis Effect.

Inert factors impact the wind also. Natural features like mountains and valleys have a significant effect, so that in Cape Town the wind tears across the Cape Flats with a fury, but leaves a “wind hole” in the lee of the mountain in Table Bay, which stalled our straining race yacht even before the noise of our departure fanfare had subsided. The built environment shapes the weather also. The asphalt streets of cities heat the air and affect the pressure, while its concrete buildings shape or block the flow of resulting winds.

As the alpha species of the planet, our consumption, our products and our waste are also having a measurable impact on the ancient balances between high pressure and low. The climate changes, and the weather follows. In the intricately interdependent reality that is our planet, everything else is effected also, whether for better or for worse.

A website called the Socratic Method expounds on a quote you often see at yacht clubs and at the dingy watering holes yachtsmen frequent. “When Jimmy Dean stated, 'I can't change the direction of the wind, but I can adjust my sails to always reach my destination,' he captured the essence of personal empowerment and resilience. This quote serves as a reminder that while we may not have control over external circumstances, we possess the power to adapt and make the best of them. It speaks to the importance of taking responsibility for one's own journey and finding ways to navigate through the challenges that life presents”.

We can’t change the weather, but we should adapt and make the best of it. So Doreen reuses her water, I recycle my plastics, and the captain has us trim the sails. We’re coping. Maybe progressing. Sometimes even thriving. But we’re still just responding.

But “you actually have to stand back and understand the system more broadly”, Gina’s voice echoes in my head.

On the boat, the weather charts tell us what’s happening around us, but the barometer allows us to hypothesize about what’s coming. Science and research allow us to understand the climate, which is why the weather is happening. Now we can begin to perceive our interdependence with our planet, and how our choices affect it. Now we can get ahead of the weather, instead of just responding to it.

Models

We wake up in the morning and we run. There are lions, and sharks, to run from, and terrible weather. That simple, singular threats can harm us is not in doubt, but to naively reduce the threat landscape to a loose collection of individual threats, like rocks raining from the sky, is to surrender to a future of constant recycling.

The cybersecurity threats we deal with daily emerge from a complex system of contributing forces that interact to create our reality on the ground like the forces that are climate collude to create the weather. By identifying and tracking the systemic factors that constitute the cyberthreat “climate”, we can begin to understand and predict the specific threats we experience daily, and therefore plan and prepare for them. We can perceive how our digital roads, buildings, and cities, our consumption, products and digital waste, impact the complex, intricate, interdependent system that is cybersecurity.

I’m fascinated by the potential power to be claimed by perceiving cybersecurity as a complex system with emergent properties, and I’ve strived for much of my career to describe the system in a useful way so that we can give ourselves the chance to stand back, understand cause and effect, and eventually, hopefully even dare to change it. Imagine the power of a chart that captures the interplay between these drivers, to describe how one force acts on another, to give birth to the vulnerabilities and threats that govern the schedule for our days. Imagine being able to see the nexuses on which these forces converge, so that our efforts to counter them can be concentrated on the pivotal points the system rests on, and not on the localized threat nodes that emerge apparently at random.

Imagine being able to see all this when it changes, so that we know when to wake the captain, and when to let him sleep.

I’ve never succeeded, but that doesn’t mean we should stop trying.

Contributing forces

I imagine the cyberthreats we experience daily as emerging from four layers of systemic force, which I describe as Root causes, Catalysts, Enablers and Shapers (RCES).

At the level of Root Causes, I’m searching for the sun - for the original sources of the energy that animate the whole system that security is supposed to secure. At this layer we attempt to identify the original causes of the cybersecurity problem. We try to answer the questions “Why is there software vulnerability”, and “why is there crime”? The sun in our particular solar system is surprisingly difficult to identify, and there is a frustrating “chicken and egg” dilemma. Like all efforts to comprehend the true origin of things, this promises to be a long and frustrating journey of excruciatingly small steps.

The Catalysts are systemic forces that exacerbate the challenges of cybersecurity. In the Southern Ocean I experienced Force 12 hurricane winds of the Roaring Forties and the memory of it will feed my nightmares for years to come. There is nothing on the open ocean to impede nature’s force in the vastness of the Southern Ocean so it reaches its full, unrestricted potential. But in Antarctica, in secluded bays among icebergs and jagged cliffs, I saw the winds that are called Katabatic, in which cold and heavy air that tears down the rugged mountains of ice and rock can turn a tranquil day to terror in minutes. Antarctica has the highest recorded wind speeds of any place in the world, at over 320 kph. Katabatic forces don’t emerge in the open ocean, because they require other factors to catalyse, agitate and accelerate them. I think of the Catalysts as factors that make the Root causes identified in the first layer “worse”. Here we are trying to answer questions like: “Why has cybersecurity become such a significant problem, and why does it continue to grow”?

The Enablers are a set of systemic factors that prevent us from acting to counter the cybersecurity problem. Unlike the Catalysts, these Enablers don’t have a perceptible impact on the size or the shape of the cybersecurity problem directly. Rather they are characteristics of the landscape that act as obstacles in our battle against cyber threats. In South Africa, by way of illustration, fires sparked by electrical faults or accidents with open flames frequently cause enormous loss and suffering in the slum areas we call townships. The quality of life and structures in the townships varies considerably, but many areas grow organically, like cracks on hot asphalt, with no proper roads or servitudes to separate or connect them. Emergency services, desperate to bring hope and succour to terrified township residents when the fires rage, are thwarted by the choked narrow pathways that were never built for vehicles. Thus, I think of the lack of proper city planning and building code enforcement as an enabler of those terrible fires with their terrible cost.

The Shapers in my model are systemic forces that act to give the problem of cybercrime the specific “form” it has today. The previous two layers may help us understand why cybersecurity is the growing problem we’re wrestling with today. But this layer helps us understand why it takes the specific forms through which we experience it most often. For example, “Why is cyber extortion the dominant form of cybercrime today, rather than something else, like credit card theft”? Or “why do attackers favour certain technical methods over others, like Phishing rather than zero-day exploits”? Nature provides another convenient metaphor for this concept in the form of cyclones, which rotate clockwise in the southern half of our planet, but counterclockwise in the north, because of the Coriolis effect.

“All models are wrong”. And no doubt so is this anaemic abstraction, “but some are also useful”, the popular aphorism heartens.

PEST

Francis Joseph Aguilar was a management scholar known for his contributions to strategic management frameworks. He has the kind of triple-barrel name that evokes respect and confidence, which is probably why I’m mentioning it here. One of his notable contributions is to the PEST framework, which is used for analyzing macro-environmental factors that can impact on a business or organization. The business boffins describe a process they call “Environmental Scanning”, which is “the study and interpretation of the political, economic, social and technological events and trends which influence a business, an industry or even a total market”.

PEST and its various acronym variations describe the major groups of systemic factors an organization could be influenced by – Political, Economic, Social, and Technological. Aguilar's work helped popularize the use of this framework as a tool for assessing the external environment and making strategic decisions based on those insights. This reminds me of the metronomic processes of reading, recording and responding to key instrument readings on the yacht, to ensure we stayed our course, stayed afloat and eventually stayed ahead, and strikes me as something that may help us make some sense of the complex environment that is cybersecurity. If not to truly understand it, then at least to describe what we believe have already understood.

The Political factors affecting the threat landscape include policies & regulations, and various other government legislations, as well as other government decisions, investments actions, or inactions.

The Economic factors reflect how finance, investments, incentives, trade, and business affect the threat landscape.

The Sociocultural aspect of the model refers to both social and cultural factors that impact the landscape. This includes things like the traditions, habits, patterns, and beliefs of both the victims and the criminals that may influence the problem in some way.

The Technological factors include all the diverse ways in which technology, the development of technology or our relationship with technology may contribute to the cybersecurity problem.

We could superimpose a framework like PEST over the RCES forces I described earlier, as a way of organizing the various systemic drivers into groups. The resulting matrix helps us to observe just how diverse the systemic drivers are, and thus avoid a myopic focus on just one factor (like the ever-so-tempting scapegoat that is technology). I’ve found it immensely useful to have such a framework through which to have thorough and structured conversations about why the problem of cybersecurity exists today, and what technologists, business leaders, law enforcement, policy makers, and regular people may be able to do to truly address it.

Although I hope that REST and RCES may prove to be useful to others, they are primarily just tools I use to organize my observations and insights for myself. You may notice me reaching for them from time to time in later chapters of this book. My goal here is not to convince you to use my model, but to suggest to you that we need a model – some means of standing back, as Gina extorted, and understanding that there is a system at work that may seem chaotic and unpredictable, but probably isn’t.