Ghosts in the Machine

For 2006, the Tokyo stock exchange has decided to supplement its computer systems with an exotic new backup technology: people. In December an employee at the Japanese investment firm Mizuho entered a mistaken sell order into the TSE's computerized trading system, which didn't allow the trade to be canceled once the mistake was noticed. The result: a transaction that ended up costing the company about $346 million. Earlier, on Nov. 1, another computer glitch had shut down the entire bourse for four hours. Hence the announcement, just before Christmas, that a group of select officials who monitor trading at the exchange would henceforth be allowed to suspend transactions that look like mistakes--something that wasn't permitted under earlier rules.

You might think that a technological snafu of such proportions is a rare event in Japan. Remarkably, though, the TSE is just one of a host of institutions that ran into software-related problems during the past year. Glitches around the country resulted in drivers being overcharged for tolls, failed applicants receiving driver's licenses and airplanes taking to the skies without permission. Even some of the country's most tech-savvy companies weren't immune. Telecom giant NTT saw 68,000 lines disrupted when a computer bug crippled its corporate data network in September. In addition, a tech glitch caused Toyota's prized orchestra of robot musicians to lose their mojo at the World Expo in Aichi; Toyota had to control the robots manually to make sure they performed as planned. But it was the troubles plaguing financial institutions that seemed the most disturbing of all. Software malfunctions knocked out ATMs at leading banks. And in the fall the General Insurance Association of Japan revealed that many companies had underpaid upwards of 160,000 customers because of programming errors.

How do such problems occur in one of the world's leading high-tech nations? Experts cite several issues, but the biggest is that the software used by Japanese firms isn't as sophisticated or reliable as the nation's vaunted hardware. When a leading Japanese business magazine ran a cover story on the software malaise last April, it noted that software suffers from a status problem in a country that still tends to be obsessed with making "things" rather than integrating systems. "[T]he status of Japanese programmers is not high," noted Nikkei Business. "Unless we change the thinking that 'software is something that comes with hardware,' we cannot stop quality corruption." The publication cited numerous problems in the software that runs Japan's much-vaunted mobile phones, as well as bugs that sabotaged the speed controls on a bullet-train line in one particularly nerve-rattling case.

The status gap may contribute to a lag in the numbers of qualified engineers who choose to go into programming. Even though the IT business in Japan exploded from 1998 to 2003, the magazine noted that the number of workers in the industry increased by only 6 percent during that period. "While the workload expands, the work force hasn't," observed the authors. As a result, the magazine added, Japan's programmers are overburdened.

And then there's the tendency, so common to the companies of postwar Japan Inc., to do everything in-house. "Historically, instead of outsourcing, big Japanese companies often had their own IT subsidiaries that did [software] work for them," says Carl Kay, a Tokyo-based business consultant. "They'd take their business processes and digitize them. But if the processes were poorly designed to begin with, you've just made them faster, not better." Outside Japan, he argues, companies tend to buy standardized software packages that embody "world-standard methodologies," and by using such software "you're able to get the benefit of that experience and knowledge."

With Japanese firms developing their own software in isolation, it should come as no surprise that problems result--and they often show up when there are corporate mergers. One of the country's recent bank mergers was slowed by financial regulators who feared a repeat of a catastrophic 2002 glitch resulting from the creation of megabank Mizuho Financial Group (formed when Mizuho Bank merged with Dai-Ichi Kangyo Bank, Fuji Bank and the Industrial Bank of Japan).

Japan's economic recovery, which looks more secure each day, isn't vulnerable to the software sickness. But the problem needs attention--and if Japan wants investors both inside and outside the country to keep believing, it may be time for the corporate elite to start ponying up for some really good bug spray.