The Paradox of Automation

If you’re in the workforce you’re almost certainly working with technology. Whether that’s a work computer, specialized software and applications, or large physical equipment, the use and interaction between us and machines is unavoidable. For a few select occupations their interactions with technology go beyond mere use. Software programmers develop new applications and novel ways of doing things. Data Analysts determine which methods will be used for data collection, cleaning, and analysis. User Experience Designers design the human machine interactions. Engineers and project managers evaluate, select and integrate technologies into organizations with an expectation of making changes for the better.

The Responsibility for Good Technology

These technological decisions are a huge responsibility. The decisions carry with it the time money and resources of the organization, but will also dictate the way work is done within the organization for many years to come. Moreover, technology is not some unmitigated good. Just because a company pursues a new technology does not guarantee progress in the form of greater productivity, safety, or ease of use for the end user. When the wrong technology is chosen or when it is poorly implemented the results can be disastrous. Everybody who uses and makes decisions about technology should be mindful of a number of key considerations that determine the success or failure of technology.

Productivity through Automation.

At the heart technological decision is a hope Through some technology, app, automation, software, or algorithm, we will be able to solve our organization’s problems, whatever they might be. One of the most common problems to be solved is productivity - the ability to do more with less. And the technologies that promise the most productivity are those technologies that can automate tasks currently done by people.

The Paradox of Automation.

Automation means letting a machine do the work for you which necessarily entails giving full control, including and especially decision making, over to a technology, be it a computer program or a steam engine. When automation is employed, greater efficiency and safety is usually achieved than simply relying on individuals alone. But these results are achieved only when it works. When automation fails to perform correctly it creates new problems and challenges that humans are not prepared or equipped to manage. In his book “The Design of Everyday things”, Don Norman explains what he calls the Paradox of Automation:

“When the automation works, the tasks are usually done as well as or better than by people, moreover, it saves people from the dull, dreary routine tasks, allowing more useful productive use of time, reducing fatigue and error. But when the task gets too complex, automation tends to give up. This of course, is precisely when it is needed the most.”

Norman goes on to discuss many of the other issues surrounding automation, specifically the propensity for automation to fail abruptly and without warning and for individuals to have insufficient knowledge to deal with the crisis when the automation stops working. Indeed, as automation gets better and failures become rarer (though not completely eliminated) human actors will have less experience and situational knowledge than any time in the past. This means that increasingly, when it’s time to intervene, we won’t know how.

737 MAX, Driverless Vehicles, and The Arrogance of the Algorithm

One glaring example of the Paradox of Automation is by the Boeing 737 Max crashes. The Boeing 737 MAX was poorly designed with many known flaws and characteristics. Rather than use technology to craft a better plane, the Boeing aerospace engineers poorly designed a plane with bad aerodynamics. The poor design was known, but the new design was cheap. To overcome the poor design, A software was installed to automatically compensate for the plane’s bad aerodynamics.  The design of the plan caused the plane’s nose to elevate and the software automatically pushed the plane’s nose down. When the cause of the 2018 Lion Air Flight 610 and the 2019 Ethiopian Airlines Flight 302 were determined, it was ruled that the failure of this algorithm was the key contributing factor. Moreover, despite years or experience and training the pilots were unable to respond adequately in the crisis because they had never encountered such an issue. Between the two flights, the Paradox of Automation took 346 lives.

One person who has appreciated this paradox is American lawyer, consumer rights advocate, and former presidential candidate, Ralph Nader. Even before his grandniece was killed in the 2019 Ethiopian Airlines Flight 302, Nader has rallied against what he calls the “Arrogance in the Algorithm”. Even as far back as 2016, Ralph Nader has called for a restraint on the hubris inherent in some of today’s most ambitious technological undertakings. The hubris is one of delusion and control, that through some technological advancement we can finally achieve a level of total control. In 2016, Nader warns about algorithms and driverless vehicles:

“Too many complex variables cannot be anticipated with driverless cars, especially when the vast majority of cars on the road are traditional driver-directed cars, who behave in a whole variety of ways that aren’t predictable. Not to mention bicycles and puddles and potholes, and shiny sides of trucks, that’s why I call it the arrogance of the algorithm.”

Conclusion

The dream of technology as the ultimate means of productivity, control, and safety, are mains and will remain unfulfilled. If anything, our technological history should clearly direct us to curb our enthusiasm for promises that come from the machine. In truth, it should temper our beliefs regarding the limits of technology and its uses. The Paradox of Automation is an important consideration when making decisions about the use, development, and design of technology.