Articles   •   April 20, 2017

Too Fast Too Furious

One of the biggest appeals of using technology is the ability of computers to do the same thing over and over again with extreme precision. Unlike human employees, we see computers as things that never tire out, complain or make mistake.

With algorithms and AI getting smarter everyday, we start dreaming of making a huge one-off investment in technology, sit back and see the profits roll in. One company that made that dream a reality was Knight Capital Group. It was once the largest trader in US equities with a market share of around 17% on each the NYSE and NASDAQ. Knight ran an electronic trading group that relied on software algorithms and sophisticated computers to execute up to 109 trades per second. Tick-tock, that’s 218 trades. Knight could have made more than 50,000 trades by the time you’ve read this article.

Dreams don’t last forever, they say. On the 1st of August 2012, Knight woke up to face its worst nightmare. As usual, the system executed trades as soon as the stock markets opened. Within a minute, something went wrong. The system started making abnormal trades to such a scale that alarms went off on Wall Street. 45 minutes later, the company managed to pull the brakes but it was too late. Their closing position put them at a 460 million dollars loss.

What went wrong? Was it a glitch or problems with the hardware? It wasn’t. The servers were fine, working exactly as they were designed to, running the trading algorithm and executing trades with speed and precision. With one exception, it was executing trades based on the wrong algorithm.

During a manual software update, a technician had erroneously copied over some old code into the trading system. Without any review or test, the system was subsequently brought live and no one caught the error until it was too late.

When King Kong Slips on a Banana Peel

Today it’s a forgone conclusion that companies need to sell their products and services online. The idea is simple; a physical store can only reach out to a limited number of customers located within the store’s vicinity while an online store can reach anyone with an internet connection and postal address.

Dell was one of the early-birds in establishing their online delivery channels worldwide. In doing that they also learnt how size and scale can magnify a small mistake. Back in 2009, someone in Dell Taiwan mistakenly updated the price of an LCD monitor from USD 228.84 to USD 15.26. They had only planned to discount the price to USD 146.46. If this mistake was made in a physical store, you might end up with some very happy customers, a few thousand in losses and a new manager.

Not quite the same with an online store. Within 8 hours, the site racked up orders for 140,000 monitors and was subsequently ordered by the Taiwan Consumer Protection Commission to honour the sales. Dell found itself standing between the devil and the deep blue sea. It had to choose between a multi-million dollar loss by honouring the sales or a huge fine and severe damage to the brand by cancelling the orders.

Cyclops, Don’t Forget Your Glasses!

If you had watched the 2002 theatrical adaptation of Spider-man starring Tobey Maguire, you would be familiar with the saying “With great power comes great responsibility.” Shortly after Peter Parker realises his newfound powers, his uncle Ben shared those words of wisdom only to be dismissed by the young superhero.

We rely heavily on the “superpowers” of computing to enable all the magic in our business. A prudent and responsible initiative is to consider the possibility of that “superpower” or the people behind it going rouge. Some well thought out processes and protocols can mitigate the risk to our IT implementation.

Might Knight Capital still exist today if it had ensured checks and simulations were carried out after software updates? Could Dell have avoided the fiasco by building in limits and approvals for price changes? Recently Amazon showed us that something as minor as a typo can bring down whole data centre hosting millions of websites and services.

About the Author(s):

Jason Paul is the Head of the Technology Sector at Evant & Co., a management consulting firm with offices in Kuala Lumpur and Jakarta. Our purpose is to help our people and our clients to realise their aspirations while improving lives through business. We advise leaders on strategy, human capital, digital, and outsourcing.