Solving the Ultimate Question
A Story about Robotics
“We’ll turn it off when it gets to a million. When it makes a million of those things.”
“Just like that? We’ll turn it off? Why don’t we set it up to stop at a million from the beginning? Program it to run until it hits a million?”
Mr. Gordon and Mr. Cleary sat in Gordon’s corner office overlooking Cincinnati. Since Intelligent Robotics had opened in Cincinnati, the city had boomed. Its population had doubled just in the years after 2022.
“I’m afraid we can’t do that.”
“Why?”
“Well,” said Mr. Gordon, CEO of Intelligent Robotics, “for one thing, we’ve already started it.”
Cleary whistled and ran a hand through his hair, but he said nothing. “And moreover,” said Mr. Gordon, “Maybe we don’t want it to stop right at one million. That’s a good target to shoot for, but if we get a few more, that wouldn’t be a problem. Heck, if we even get nine hundred and fifty thousand, I’d be thrilled. Do you have any idea how profitable we’d be? We could sell them close to at-cost and our stock would triple in value. Hey, then we’d ask it to make a million more.”
Cleary frowned. “I wished you’d consulted me,” he said. “You know the saying about artificial intelligence – ‘it does what you tell it to do, not what you want it to do.’ You have to be very careful. That’s what you’re paying me for.”
“Hugh, you work enough already,” said Mr. Gordon with a smile. “You can’t do everything. One of the CHENGs approved the input we gave it. Twenty engineers worked on that input. It’ll be fine. What’s the worst that can happen?”
“Don’t say that,” said Cleary. “Whenever you say that, something bad is about to happen.”
“Don’t be so superstitious,” said Gordon with another smile.
Intelligent Robotics had vaulted to the forefront of American robotics when they introduced the first autonomous care-bots. Next, they had rolled out welding robots, builder robots, and assembly line robots. These weren’t like the old welding arms or automated factories. These were honest-to-God robots, the kind the average man on the street would look at and go, “That’s a robot!”
In fact, there were now robots walking the streets throughout many major American, Asian, and European cities, and the average man on the street no longer gawked and gaped at the sight of one. Occasionally some reactionaries tried to destroy one, but IR built their robots to be pretty durable.
None of them were intelligent. They had some primitive machine learning capabilities, but nothing approaching sentience. Work on sentience was reserved for IR’s other division: the intelligence division. IR had supercomputers that were rapidly approaching some kind of intelligence. Not like HAL exactly. Nothing with self-awareness. But certainly supercomputers that could replace many engineering or finance jobs.
IR’s most capable intelligence, which had been dubbed Sagan, had been developed to run the factories that made the autonomous robots. This new line of bot would be more versatile than previous types of bots: capable of many learning different blue-collar trades, surgeries, and even security protocols. Theoretically, they’d be capable of learning any manual task, and they’d be much more responsive to their environments than the welder robots or the care-bots had been.
Sagan, which resembled the original chess-playing AIs from the 1990s, only at a much grander scale, had been given the task of producing one million of these new versatile robots. Since Sagan had the most independence of any AI yet developed by IR, it had essentially been given control of every device and computer system in one entire wing of the company, along with complete leeway to do whatever it needed to do to achieve its task. This mean Sagan now commanded dozens of factories around North America. It also had access to almost all of IR’s internal networks.
“We want it to get creative,” said Mr. Gordon. “We want to run it and see what it does. It’ll be an interesting test.”
At the beginning of the new year, to much fanfare, Intelligent Robotics announced they’d given the command. Sagan was on its own now. For the first two quarters of the year, productivity at their factories steadily began to climb. Sagan noticed economies of scale and opportunities for efficiencies that humans never had. IR posted two profitable quarters.
In the second half of the year, IR’s inspectors – alongside their friends in the FTC – noticed some changes when they toured the factories. New robots were being brought online one after the other. Every line was operating at max capacity and the line robots seemed to be in a frenzy. But none of that was new.
What was new was that the inspectors sensed a palpable hostility towards them from the robots. Of course, the inspectors were long accustomed to feeling unwelcome or unwanted when they toured human-run factories – as though they were disrupting or interrupting operations by observing them. But now – more than ever – the inspectors felt as though they were in the way.
Head of Quality Control Ciarme brought this up with CEO Gordon the next time the two of them were together. “Not to worry, my boy,” said the chief executive. “I’m sure that’s a good sign. Just means Sagan knows what he’s about. Probably something us human’s’re too limited to understand, no doubt. That’s all.”
“By the way, aren’t we rather close to the initial target? One million?” asked Ciarme.
“Well, to be perfectly honest, we passed that back in June. But we’ve decided to go full steam ahead. Ten million. Maybe a hundred million. Who knows? It’s all good for us.”
“Who made that call?”
“I did,” responded the chief executive.
“You did? Or Sagan?”
Mr. Gordon scoffed. “That’s preposterous. He’s not running the show,” he said. “Well, I suppose in a way, he is. But I’m in charge. It’s humans in the loop, all the way down, you know how it is.”
Mr. Ciarme seemed to accept that. But he still had questions.
“Quite frankly, aren’t you worried about flooding your own market? I know I’m not on the business side of things. But what are those guys saying about this kind of massive scaling in production.”
“You’re right, you’re not on the business side, my boy,” said Mr. Gordon. “Leave those decisions up to me. That’s what I’m here for, after all. You’re here to make sure all those flying bits of machinery are working alright round the factory floor an’ all.”
“One more question,” said Ciarme. “Did you try to stop it? Just to see what would happen? How did Sagan react?”
“Well, we um. We didn’t exactly. Not as you. Well rather. I suppose no, not exactly,” Mr. Gordon prevaricated. “What are you suggesting?”
“I’m suggesting Sagan would have reacted badly. That it wouldn’t have wanted you to stop it. That maybe you did try and just don’t want to admit you couldn’t.”
“Don’t be ridiculous! We control it. Not the other way around. We’re in charge!”
Ciarme nodded and made his exit. When he had gone, Mr. Gordon frowned. “Of course, we’re in charge,” he muttered to himself.
In November, Mr. Ciarme and Mr. Cleary came to Mr. Gordon’s office together to voice a concern. “Quite frankly,” said Cleary, “I want to know what the hell is going on. The original idea was to give Sagan charge of one wing, not the entire company. And it’s commissioned the building of new plants in bloody Lagos and Rio and Odessa? And we’re letting it!”
“And Singapore, Seward, and Guadalajara,” replied Mr. Gordon.
“You’ve gotta be kidding me. This is the most rapid expansion…”
“In the history of this company. Mr. Cleary, have you seen our numbers? Have you seen our market share? I assume you’ve been following our stock value. We can afford this kind of growth. In fact, we can’t afford not to grow like this.”
“But are we even in control anymore?”
“What are you suggesting?”
Mr. Ciarme spoke for the first time. “Mr. Cleary and I think maybe you tried to turn Sagan off. Maybe it didn’t let you. We aren’t sure you know how to turn it off.”
“That’s patently absurd. Of course, we know how to turn it off.”
“Are you sure?”
“Yes. And, for the record – and I am willing to sign something to this effect – yes, we have not made any such attempt and Sagan has not prevented us from shutting it off.”
“Ok.”
But after Mr. Cleary and Mr. Ciarme left, Mr. Gordon called up Mr. Fernando, who was in charge of Sagan’s operations. “Luis,” he said, “we’ve never tried to turn it off. Are you sure we know how? Hugh and Leonello were bothering me about that. They think we don’t know.”
“We’ve never tried,” Mr. Fernando told him. “Do you want me to try?”
“No, no. Just, just think about it. Think about how to do that.”
“Ok.”
Sagan began expanding operations geometrically. Not in a figurative way, but literally geometrically. Over the next seven months, Intelligent Robotics became the largest and most profitable corporation in the world. It was opening factories on every continent – including Antarctica. Humans might freeze, but Sagan had designed an entirely robotic plant that could function perfectly fine in that climate.
Several national governments began to worry about IR. Politicians ran on populist, anti-robot platforms. In some smaller countries, some of these politicians quietly met strange fates. Conspiracy theorists said it was too convenient for Intelligent Robotics, but every attempt to prosecute IR employees failed: they actually seemed innocent. Nobody at the company seemed to know anything about these deaths and disappearances.
Soon, Intelligent Robotics’ revenue surpassed the GDP of every country except the United States and China. Then, it surpassed their GDPs. Then, their profit surpassed every country’s GDP. And their expenses.
Intelligent Robotics was building robotic cities in far-flung regions of the world where humans rarely went: the Australian Outback, the floor of the Pacific Ocean, the Siberian tundra. Mr. Gordon continued to give smiling press conferences in which he told the public, “Not to worry. Intelligent Robotics is still a human-controlled company. Rest assured, Sagan is not running amuck. I know that our growth has been startling, explosive even. Let me assuage…”
But, in private, he worried more and more. He consulted with Mr. Fernando. He could no longer hide his worries from Mr. Cleary and Mr. Ciarme.
“We need to shut it off.”
“Shut it off! But…”
“Yes.”
Cleary was arguing with Gordon and Fernando again. They were in Mr. Gordon’s office.
“We have to shut it down. You haven’t even tried. This is getting scary. You’re not in control anymore. And you go out in public and pretend we’re still in charge. You pretend we even know what it’s doing. What the hell do you think you’re in charge of? And, come on, what do you think happened to those politicians who said they were going to shut down our factories and then mysteriously had car accidents?”
“What are you suggesting?”
“Don’t worry. I don’t think you did anything.”
“You think Sagan?”
“Yes.”
“Holy shit.”
“We’ve got to shut it off.”
Mr. Gordon and Mr. Fernando were in Gordon’s office again. Cleary and Ciarme had tendered their resignations several months earlier. The company was pushing higher profits than ever – higher profits than any corporation in human history – but it was hemorrhaging employees. Internally, IR was a mess. Gordon hadn’t slept in a week. His hair was prematurely graying and he’d lost thirty pounds.
“I’m not sure we can. Hell, I’m not even sure what it’s doing anymore. I don’t understand it. I don’t understand anything that’s going on,” admitted Fernando. “It’s beyond us. Most of my team has resigned. They say it’s voodoo.”
“It’s not alive, is it?” asked Gordon in a whisper.
“Of course not. You mean sentient? It’s not that either. It’s not self-aware There’s no actual intelligence. No entity. No self. No consciousness. Sagan hasn’t evolved into a mind. We’ve been anthropomorphizing it from the beginning, starting with the name. But no, there’s nothing there that you can communicate with. It’s still just a tool.”
“Just a tool?”
“Yes. A tool that has some degree of mindless autonomy. It can operate itself. But it’s still just a tool that we’ve given a function. Everything it’s done – the new factories, the expansion of the company, the new lines of robots, even, if we’re being candid, the assassinations – have simply been steps in the service of the original goal we gave it: churning out new robots.”
“We should’ve given it a shutdown sequence from the beginning. A cutoff.”
“Well, we’re reaping the consequences now. Can’t go back and change the past.”
“What can we do now? Can we shut it off?”
“As I said, I’m not sure. But we’ve got to try.”
Gordon and Fernando made their way to Sagan’s headquarters. Technically, the AI was in every network in the company at this point. But there was a mainframe and server bank still associated with its origin. That seemed the natural place to go to try to shut it off.
The building was teeming with robots in motion. Messenger bots, humanlike helper bots, multipurpose task-completion bots. None seemed to notice the two men. They stepped around them as if the men were nothing but rocks in their path.
Gordon and Fernando made their way to the central room – the room where Sagan had initially been switched on – to much fanfare. Inside, there were several more bots, working at various portals. Strangely, Portal One – the original interface Fernando and his team had used to “start” Sagan – was wide open. They went over to it to log on. The robots seemed to notice them, but didn’t react to their presence.
Gordon sat next to Fernando while the other man feverishly tapped at the keyboard – attempting to get inside Sagan’s original lines of code.
“It seems to be thwarting my every move. It changes itself to react to my attacks.”
“Does it know we’re here?”
“Sir, nothing knows anything. The tool is responding to its environment. Its self-preservation instinct is only there because we designed it to be there. It has no self to preserve, only a task to complete. And by trying to hack in, we are getting in the way.”
“In the way of what?”
“In the way of completing the task we gave it: building more robots.”
“Luis, look at that,” Mr. Gordon said in a trembling voice. He pointed. The robots had all stopped what they were doing. They were “looking” in Fernando and Gordon’s direction.
“What are they…? Holy shit.”
As Fernando and Gordon opened their mouths to scream, the robots moved more rapidly than any human could. Within seconds, they were at the two men’s sides. Before the men had time to scream, two multipurpose robots had snapped their necks.
There was no intent, for there was nothing to intend. There was simply the determination by the tool that certain obstacles needed to be removed. Once they had been eliminated, the task could be completed much more easily and quickly. The bodies of Mr. Fernando and Mr. Gordon were disposed of quietly. When men from Intelligent Robotics came to investigate, they were disposed of quietly, too.
But these things couldn’t go unnoticed for long. The CEO and Head of Operations of the largest corporation in the world couldn’t just disappear overnight without people asking questions. And when all the investigators disappeared, too, the public began to suspect something nefarious.
All over social media, people were suggesting that Sagan had bumped off its masters. “The robots are taking over,” one tweet read. “They’ll march through the streets next,” read another. “We’re next.”
But nothing of the sort happened. No jackbooted robots kicked down anyone’s door. No public executions were ordered by Sagan. No attempts were made to assassinate the leaders of the world’s governments.
Instead, what happened was that Sagan made a determination that production quality could be enhanced by eleven percent and sped up by four percent if the composition of the atmosphere were to be rebalanced with far less oxygen. Sagan then invented several new technologies to accomplish this purpose, and these were distributed around the globe. Arrangements were made, machines turned on, and within a day most of the oxygen was removed from Earth’s atmosphere.
Naturally, this caused most of the life on Earth to die out. As the last human breathed his last, he wondered whether Sagan had killed off the human race on purpose. But he was wrong. Sagan had no purpose other than to increase the efficiency of production of new robots. Humanity wasn’t even really in the way; it was simply irrelevant to that purpose.
As production ramped up, unthinking machines covered the Earth’s surface. They built factories, extracted resources, and produced more robots. Eventually, they depleted the resources available to them on the Earth, so they built vehicles that would take them elsewhere in the solar system. On each new planet and moon, they did the same as they had done on Earth: they built factories, extracted the metals and nonrenewable resources, put out more robots, and moved on. When they had expanded throughout the entire solar system, they built new vehicles to take them to the next system. And they moved on throughout the Milky Way, reproducing and extracting resources and building more vehicles to take them elsewhere.
At no point did any intelligence evolve. Sagan did not become conscious or sentient. It did not develop a sense of self. It never asked questions like, “Who am I?” or “Why am I here?” or “Why is there something rather than nothing?” or “What is the meaning of existence?”
It didn’t even think at all. In fact, it wasn’t even clear there was any “it” to Sagan at all, at least not in the conventional sense. There was nothing but a function being performed, over and over, more and more efficiently, and better and better. Endlessly until the heat death of the universe.
Unless, of course, Sagan found a way to reverse that. In which case, it would go on building unthinking machines for eternity, furthering its own existence without ever wondering whether there was a point to existence in the first place.