The Chicago Blues Puzzle

Since the early days, there have been two big problems with making lifelike robots – and the first is the eyes. The tiny involuntary movements, the spontaneous dilation of the iris, the constant supply of moisture and how much of it there is, the infinitesimally subtle changes that convey emotion – human beings are so precisely though unconsciously aware of how all that works in other humans, it’s always been a giveaway.

If you ask someone “Robot or human?” the answer is always “Robot…there’s just something not right about the eyes.”

In the business, we call this the Baby-Blues Puzzle.

Well, finally, we’ve solved it. Ladies and gentlemen, I give you the Optioid semi-organic eyeball.

Pau Garriga, CEO SideKick Robotics, TechCon32

--------------------------

Kieran had been hit by a Boston smart-taxi on Huntington Avenue – which wasn’t supposed to be possible. There’d been a glitch in the AI, they figured. This was a conundrum in which Kieran took a professional interest –  not that he had any recollection of the accident.

The memory of his Minnesota childhood was intact, and his college years in Seattle. But from then on, everything was fractured and disjointed. In that moment of impact with a robocab, he’d lost the detail of a whole decade. When he returned to work, after months of physical rehab, he was afraid that he might have forgotten how to do his job too.

 “Good to have you on board again,” Sharon said. “Listen, they told me you might be a little shaky while you re-learn all this stuff. We got your back – okay?”

“Thanks – appreciate it. So what’s new on the showroom floor?”

“Let’s take a walk, and I’ll talk you through.”

The Boston showroom was SideKick Robotics’ flagship retail outlet. They had thirty models on active display – playing chess, making cakes, juggling, painting landscapes – and two hundred more crated up out back in the warehouse.

“Man, these new ones are so fluid,” Kieran said, as he and Sharon walked past two robots playing table tennis. “No judder.”

Sharon looked at the bracelet on her wrist. “Sorry – I need to take this.” She stepped away from Kieran and put her fingertips to her ear. “Theresa, how are you? Yes, of course – you can collect anytime this afternoon...”

Kieran walked on, towards the high-end section of the showroom. Any model from this range would cost more than a mansion on the Hill.

“Hi – can I help you?”

The speaker was a man in his forties – trim, grey-streaked hair, brown eyes, genuine tan.

“Uh – I work here. I’ve been away…”

“Ah, you’re Kieran. I’m Henry.” He put out his hand, and Kieran shook it. “Nice to meet you.”

“Oh, you beat me to it,” Sharon said, catching up, “Kieran, Henry joined us from the Sydney office to head up the East Coast team.”

“Yep – and I need someone to take on sales of the new model. You ready for it?”

He turned and beckoned to a young woman who was sprinkling flakes of food into a tank full of tropical fish.

“Leila – come and meet Kieran.”

The woman smiled and walked towards them, holding out her hand.

“Nice to meet you, Kieran,” she said.

“You too, Leila.”

As he took her hand, Kieran felt a buzz in his bracelet.

“Wow,” he said. “I would not have guessed that.”

----------------------------

So it turns out that the buying public are less than happy about the Optioid eye. It’s just too convincing. Consumers are worried that we’ve removed  the last sure-fire way of telling a robot from a person. Yeah, okay, the second-to-last one – but the only one that’s practical on a day-to-day basis.

Of course, you could always ask – a robot can’t lie about it. But people won’t do that because it might cause offence if the person you asked was…well…a person. No one likes to think that they’ve been mistaken for a machine.

To get around this, we’ve come up with an app that’ll run on your bracelet. It detects the distinct and specific pattern of electrical activity that characterizes a robot body. When a human and a robot touch for the first time – shaking hands, or high-fiving – the bracelet will buzz, silently.

So, we will always be aware which of the people around us are robots and which are human. And the robots are aware of that too, of course. Everyone knows who’s who and what’s what, and no one need fret.

Pau Garriga, SKR Major Investors’ Meeting

------------------------------------------------

Kieran sold Leila Tuesday morning – which was a huge boost to his self-confidence.

“Because of the accident, I was worried, you know?” he told Henry when they stopped at The Plough and Stars for a celebratory drink.

“Once a sales guy, always a sales guy,” Henry said.  He raised his beer.  “Cheers.”

Over a few beers Kieran talked about his mom and dad and the little house in the middle of Minnesota’s frozen nowhere. Henry’s tales of growing up in the bush were both hilarious and touching.

“Of course, it’s all changed. The Aussie wilderness I was raised in – it’s a two-hundred-acre cranberry farm now. That’s progress, right?”

It was half-past seven when Henry said that – because the game was just starting on the screen behind the bar. But when Kieran woke up the next morning, the cranberry farm was the last thing he could remember from the previous night.

He lay in bed, thinking, searching his mind. How had he got home? Had he passed out? How much had he drunk? He checked his bracelet. It had unlocked the apartment door at seven minutes past ten. He looked at his cashflow app. He’d picked up the bar-tab at nine thirty-five and blipped out of the subway at one minute past ten.

He couldn’t remember any of that. And he didn’t have a hangover.

“No, no,” he muttered as he showered. “Not blackouts. Can’t have blackouts.” He’d had a few in the early weeks of his rehabilitation, and the doctors had been worried. But there’d been none for months – and that had been a big thing when they’d let him come back to work. No blackouts. He couldn’t have blackouts.

“Bad beer or something. I dunno,” he said, as he dressed.

He was early to work. It was Wednesday, and the display robots were charging, sat with hands in their laps, heads dropped, feet bare. They picked up current from plates in the floor.

Kieran walked to the Sales Office at the back of the store. It was deserted, although someone had put the coffee on. He was pouring himself a cup when a voice said, “G’day. I’ll take one of those.”

“God, you scared the daylights out of me!” Kieran gasped, turning to see Henry at his desk. “Where the hell did you come from?”

Henry chuckled. “I was right here - just tying my shoelace. Jeez, chill, mate.” He came around the desk and took the cup. “Listen – good chat last night. Know what? Your idea for the Europe campaign – it could work. You write it up and I’ll organize a call with London.”

“Right.”

“End of the week?”

“Sure.”

“I’m glad we quit at a reasonable hour. Can’t handle school-night drinking at my age.”

------------------------------------------------

The basic principles of the battery haven’t changed since it was invented nearly a hundred and fifty years ago. They’ve gotten smaller and more efficient, but if Edison were to take apart the battery from one of our robots, he’d understand exactly how it works.

It’s absurd that the engine of a twenty-first-century robot is little more than pimped-up nineteenth-century technology. It’s crazy that a machine made to serve us twenty-four-seven is actually out of commission six percent of the time.

Well, not for much longer. Right now, we are testing a power source that we can install in a robot during manufacture, and that robot will never have to clock out to be charged.

I can’t yet tell you how this thing amasses energy – but I’ll give you a clue. Right now, you and I are travelling at sixty-seven-thousand miles per hour on a very hot rock.

Pau Garriga, SKR Internal Tech Briefing

------------------------------------------------

Kieran had no clue what his idea for the European market had been. It had completely gone, along with everything else that followed the cranberry comment on Tuesday night. He didn’t know how he was going to explain that.

It turned out he didn’t need to.

On Thursday, while he was attending a training day in Cambridge, the Boston showroom was attacked by Body and Soul, a militant anti-robotics organization. On the sidewalk outside the store, the terrorists detonated a magnetic pulse bomb that completely fritzed the neuro-circuits of the eight or nine robots nearest the door, and flatlined dozens more that would need resynapsing. Even those in prep out back were slightly jumbled.

 “I haven’t been able to document the European strategy idea,” Kieran told Henry when he went in Friday.

“The what?” Henry asked, not looking up from his pad.

“You said by today I had to write…”

“Write what?”

“The idea I had.”

Now Henry looked up. “Mate, you may not have noticed, but a bloody bomb went off here yesterday. I have thirty-five units out there with ‘Sold’ stickers on their asses and patchy memory chips in their heads. It’s chaos. I don’t know what you’re talking about and right now I don’t care.”

“Right. Sorry. How can I help?”

“Thank you,” Henry said, rubbing his eyes with his fingertips. “I’ve been up all night. Can you just take over on the desk for a couple of hours while I get my head down in the quiet room and re-charge the ol’ batteries?”

“Sure,” Kieran said. And that was the last thing he remembered until he woke up Saturday at home.

“Not again,” he murmured, rolling over.  He buried his face in the pillow. “Oh, no, no, no…”

--------------------------------------------------------------------

Yeah, we could set the bracelet app so that it detects robots at a short distance, rather than on physical contact. But we think that would be counterproductive. Have you seen how some people treat fellow human beings in customer-facing jobs – airport staff, sales people, hospital orderlies? Do you think they’d comply with requests or follow procedures if they knew for sure they were talking to a robot? You can practically hear the raised voices.

“I’m not taking that from a goddam machine. I want to talk to a human being right now!”

Whoever’s standing there behind the desk represents the company, or airport security, or a government department – and that remains the case whether it’s a robot or a human.

The question, really, is what would be gained, in transactional situations, from knowing which is which?

Pau Garriga, Expert Witness, Massachusetts vs Body and Soul

--------------------------------------------------------------------

The following Tuesday Henry was summoned to SKR’s Security Hub in Lima, Peru, for an anti-terrorism briefing. He was told to bring a representative of rank-and-file staff, and he asked Kieran to make the trip.

“First-class, natch. Luxury in-flight service and a couple of days in the sun. Can’t be bad.”

Once they’d boarded, an issue arose that delayed take-off. Kieran, in the exec-pod closest to the door. could hear the crew talking about it.  The manifest said there were 111 human passengers and 14 robots on board, and the passports aligned with those numbers. But the sensor on the door – the equivalent of the bracelet app – had counted 110 and 15. It thought that one of the humans was a robot.

“What are we going to do – make everyone get off and on again?”

“Ask the robots to identify themselves. They can’t lie, can they?”

“Does that count as a question? Would that work?”

Eventually they decided that the door sensor had malfunctioned - which was by far the most convenient explanation – and the plane took off fifteen minutes late.

Kieran could see Henry a few loungers ahead of him. Or at least, he could see Henry’s legs from the knee down, kicking off his shoes and then – unacceptably in Kieran’s view – his socks.

“It’s not often you see another man’s bare feet,” Kieran mused. “It makes Henry look like…”

Kieran’s whole body jumped. Oh, man! No!

Couldn’t be true, could it? It would explain a few things, sure. But no, no – couldn’t be, because when he and Henry shook hands the first time, Kieran’s bracelet hadn’t buzzed. Then again, that technology must be fallible, hence the count on the plane. Though, if Henry was, that would explain the count on the plane.

Kieran found it impossible to concentrate on the movie or the meal. He wondered about the layout of the airport at Lima. He checked Google and, yes, there was an auto-med dispensary airside, just before Customs.

A couple of hours later, as they passed the auto-med, Kieran grabbed Henry’s arm and steered him over to the screen.

“Better safe than sorry, eh?” he said. He quickly pressed the button for malaria, and then the button for ‘Group’ – and the auto-med’s cameras and bio-readers scanned both him and Henry.

“Man, I don’t need it,” Henry said, walking away.

“Processing,” the screen said. “Now dispensing oral preventative for malaria. One traveler.”

Yeah, Kieran thought. One. Because you know that an oral vaccine would be no use to him. Thank you.

He took the package of medication from the tray and hurried on towards Customs.

“What was up with that?” Henry asked, as they reached the taxi-rank.

“Henry, I have to ask you something.”

“I picked up malaria meds coming through Africa only last month,” Henry continued “My face and iris are on the system. I told you I didn’t need it.”

Henry gave the taxi driver their bags to put in the trunk. As the driver opened the door for them, his hand brushed Kieran’s shoulder, and Kieran’s bracelet buzzed.

“What shall we do this evening?” Henry asked, as the taxi hit the freeway. “Up to you – anything you like.”

Kieran, who was gazing perplexedly out of the window, shrugged. Then, after a moment’s thought, he smiled and turned to Henry.

“Karaoke,” he said.

--------------------------------------------------------------------

So, having solved the Baby-Blues Puzzle, from an observational standpoint there’s only one thing left that’s a dead giveaway. And it’s another blue.

We can make a robot that’ll play Bach on the piano. Easy – just mechanics. We can even create a robot that’ll improvise a guitar solo. A little more difficult, but just mechanics and math.

What we can’t do is create a robot that can sing convincingly. See – there’s just so much going on when a human being sings – not only the sophisticated mechanical stuff of making the noise, but the nuances of expression, the subtleties of timing and rhythm, the dynamics that convey emotion and experience. It’s incredibly complex.

Now, say we tell a robot to take a shot at an apparently simple tune like Mannish Boy. The robot will deliver every single note exactly like Muddy Waters’ record. Has to – because to improvise, to interpret is totally beyond a robot’s capabilities. But even note-for-note it still won’t sound right. It won’t fool anybody. You can tell it’s a robot.

We call this the Chicago Blues Puzzle, and it bugs the hell out of me that we haven’t solved it yet.

I’m not sure we ever will.

Pau Garriga, TED Talk

--------------------------------------------------------------------

It wasn’t so much a karaoke bar as a pick-up joint with a display stand. You could assess the goods as they belted out the hits.

“Give it a go,” Kieran urged Henry. “There’s, like, five thousand songs on their list. There must be something you know.”

“I really do not like this place,” Henry almost shouted against the din. “And, mate, I can’t sing, believe me.”

“Aw, come on – who the hell can? It’s just a bit of fun. We’ll do it together,”

“Let’s just finish this beer and go somewhere else, eh?”

Kieran wasn’t going to let him off.

“Look, I’ll do it, and then we do one together. Really, I’m useless but it doesn’t matter. Watch.”

The stage was empty. Kieran approached the MC.

“What do you want to sing, bro?”

“I..er…”

He couldn’t think of any songs at all. At least, no recent ones. But back in his furthest memory – in the early years that he could still recall – there were songs that his parents listened to – songs he’d learned riding in the back seat of the car, or on Sunday afternoons while he was doing his homework, or at family parties, appalled at his mom and dad dancing like teenagers. Those songs he knew.

“The First Time Ever I Saw Your Face,” he said, to his own surprise.

“Hey – respect, man. Old school. Coming up on the screen right now.”

“I know the words,” Kieran said.

He took the microphone, and he closed his eyes. And although he was actually at a hook-up joint in Latin America, he sang as if he were in the little house in Minnesota. He was singing along to Spotify with his dad while his mom sat on the sofa, sipping her vodka-and-lime and applauding them.  He sang knowing how his dad would sing it, and hearing his mom sliding in with the harmonies. He sang it for his childhood, and that memory – the recollection of himself that he was sure of – was more real than anything he’d lived since the accident.

Not that anyone was listening. There was a smattering of applause as he finished, and he stepped off the stage and went back to Henry

“Not bad at all,” Henry said. “Can we go now?”

As the next track kicked in, two girls in their twenties, very drunk, sat down on the barstools either side of them.

“Buy us champagne, and we’ll sing with you guys,” one of them shrieked over the music. “We can sing anything.”

“Yeah, persuade him,” Kieran told her. “I’ll be back in a sec.”

In the restroom, he checked out Settings on his bracelet. If he deleted Henry from Contacts, the app would test him again the next time they touched.

Having deleted Henry’s details, Kieran noticed the words ‘auto-med’ in ‘Related to Recent Searches’. He opened the link.

Company Politics Cause Scandalous Waste of Medicine, read the headline.

The article said that auto-med machines all over the world were dispensing completely unnecessary medicines to travelers because auto-med franchises couldn’t or wouldn’t share data across borders.

Kieran almost whooped with excitement.

“So the system didn’t know him from Africa,” he murmured, as he scrolled through the article. “It just knew not to dispense to a robot.”

He headed back into the deafening club – and met Henry, heading towards the door.

“Hey, I thought we were going to sing,” Kieran shouted, pointing at the stage.

Henry shook his head. “Listen, mate, I’ve got a wife back home. I’m trying to be good.” He gestured towards the girls at the bar. “I can’t handle this.”

“Okay, okay – I’ll see you tomorrow,” Kieran said. “We okay?” He held out his hand.

“Oh, sure, man. No worries.” Henry took Kieran’s hand, and shook it. “I’ll see you at breakfast. Don’t stay out all night, alright?”

As Henry headed for the exit, Kieran slumped against the wall, baffled.

His bracelet hadn’t buzzed.

He thought for a moment, and then ran towards the exit. Out on the street he saw Henry getting into a taxi. He strode over to him and put a hand on his shoulder. Henry turned.

“Henry,” Kieran said firmly, “Are you a robot?”

​

Henry rolled his eyes. “Mate – no. Maybe I should’ve sung. They haven’t built the robot yet that can sing as badly as I do.”

--------------------------------------------------------------------

The following Monday, back in Boston, Sharon called Kieran to her office on the tenth floor.

“Henry’s worried about you. About whether you were really ready to return to work. It’s a stressful job. High-pressure. It can mess with your mind.”

“This is because I asked him whether he was a robot, isn’t it?”

Sharon nodded. “Yeah, that’s mostly it.”

Okay, Kieran thought, they don’t know about the blackouts. That’s a relief. And as for the Henry thing…

“…it’s not as insane as it sounds,” he said.

He explained about the auto-med machine, and about Henry forgetting things after the pulse bomb attack, and the miscount on the airplane, and the time Henry was tying his laces when Kieran came in…

It was the last one that stopped him in his tracks. He sighed.

“You know, when I say it all out loud, it does sound crazy,” he admitted.

“It sounds inconsistent, that’s all.” Sharon said. “But not entirely off the mark.”

“What?”

Sharon leaned forward, elbows on the desk.

“Do you think it would be possible to create a robot that didn’t know it was a robot?” she said. “How would you do that?” She sat back, grinning. “Well, for a start, you’d have to unlink it from the web, so it didn’t know everything. And you’d have to give it a past that it couldn’t check – like, for instance, telling it grew up in Australia. In fact, you’d have to constantly feed it false memories to cover downtime and robot-specific activities.”

“What?” Kiernan said again.

“And if you wanted to test it in the real world, you’d buddy it up with someone who knows a lot about robotics, but who has no shared past, no connection at all, with this specific robot. Like someone just back to work after a prolonged absence.”

Kieran struggled to find something to hold on to. “But… he didn’t set off my bracelet – twice.”

Sharon shrugged. “That’s just app software. Our software. Easily manipulated to ignore specific subjects. Tell you what, though – the damn airline needs to download updates more often.”

--------------------------------------------------------------------

A robot is a machine. Essentially, a computer installed in a puppet.  Could we build a robot that didn’t know it was a robot? Well, yeah – we do it every day. All of them are built like that. They only know they’re robots because we tell them they are. We don’t have to tell them at all.

The problem would be that these things are observant, and they’d soon notice that some people need sleep and hamburger and deodorant while others need electricity and not much else. But we could stop them clocking these awkward truths just by feeding them illusions. Hey, some people would say that’s been happening to human beings for millennia.

It’s been suggested that someone would be bound to tell a robot what it really was. But would the robot believe them? How could it?

Let’s try it. I already said that we’re trialing this. We have created a robot that doesn’t know that he or she is a robot. And that robot is in this auditorium.

Yeah, really.

It’s here, now.

So – put your hand up if you believe that I’m talking about you.

Pau Garriga, Keynote Speech, Vancouver World Fair

--------------------------------------------------------------------

“We didn’t expect you to figure it out this quickly, Kieran, but the very fact that you did is significant,” Sharon said. “And we’d still like you to monitor him. Please stay close to Henry, and see how he performs.”

Over the next few weeks, Kieran worked with Henry, socialized with Henry, confided in Henry and made copious notes about Henry. Knowing now that his immediate boss was a robot, Kieran was more and more impressed by SideKick’s R&D operation. Apart from memory lapses and a measure of social awkwardness – missing nuances of conversation, a tendency to express rather too honest observations to people’s faces – Henry was a very convincing human. And when he got things wrong, everyone just put it down to his being Australian – a smart move from the programmers, Kieran thought.

But it began to occur to him that Henry’s credibility in itself implied a problem.

In a bar one evening, Henry became involved in an argument with a guy about racial politics. The other guy was an asshole, and aggressive with it. And Henry reacted.

“Mate, you have a choice. You can get up and walk out of the door right now. Or I’ll choose a window and you’ll make your exit that way. You don’t have very long to think about it.”

When the guy was gone – via the door – Henry offered an opinion of him which, though inventive, was not one you’d want to come up in your search-history.

“Would you really have thrown him through the window?” Kieran asked

Henry grinned. “What floor we on?”

Robots weren’t supposed to do that – they were programmed to be neither aggressive nor threatening. They were made incapable of causing harm to a human being. And although Henry hadn’t laid a hand on the objectionable guy, he’d given every impression that he would, and that he could. He certainly believed he could.

And there’s the problem, Kieran wrote in his notes that evening. Henry doesn’t know he’s a robot. So why should he follow the rules that apply to robots?

Over the next few weeks, Kieran observed several instances of non-robot behavior in Henry – anger in a high-pressure meeting, impatience at a restaurant, cutting-in line at Fenway Park. Robots weren’t supposed to do any of it.

It raises an interesting question, Kieran noted, in the conclusion to his report. If we allow robots to believe they are human beings, will we also allow them to act like human beings? I’d say no – for the very practical reason that, in a straight fight, any robot could kill any human being. It’s too big a risk to take.

Kieran emailed his report to Sharon, and the following day most of the Boston staff flew to Vancouver for their CEO’s keynote speech at the World Fair. The sales team sat in a reserved row near the back of the hall, ready to network at the reception afterwards.

When Mr. Garriga posed his theatrical question, asking for one person to raise a hand if they believed they were a robot, Kieran glanced along the row at Henry – which went unnoticed as everyone was glancing at everyone else, wondering.

Henry didn’t appear at all discomfited. Indeed, he seemed distracted by some activity in the row behind him.

A member of the audience had stood up. He raised a banner and started shouting.

“Only God may create Man! Look to scripture! Robots...”

That’s as far as the protester got. Henry punched him square on the jaw, and he folded like a card table.

Kieran stood too, about to go to Henry. But he looked to his left, and in the seat directly behind his, another protestor was getting to his feet.

This one had no banner. He had a gun.

--------------------------------------------------------------------

“Kieran?”

He was in a room overlooking the river. He could see the Longfellow Bridge and MGH.

“Kieran – are you with us?”

It was Sharon speaking. Her face was close to his.

“What’s the last thing you remember?”

The gun. He remembered the gun.

“You didn’t show up at Vancouver airport. You were still asleep at the hotel. You blacked out, didn’t you?”

“I’m in Boston,” Kieran said. “How?”

“We brought you home. What’s the last thing you remember?”

“Mr. Garriga’s speech,” Kieran said.

A man’s voice said “Did you like it?”

Sharon moved away, and Kieran saw Pau Garriga sitting across the table, and next him was Henry.

“Hello, mate,” Henry said.

“Henry…”

“I wanted talk to you because of what happened in Vancouver,” Pau Garriga said. “It was a unique situation – and one that you had anticipated.” He tapped the pad on the table in front of him. “I’ve read your report.”

Kieran glanced at Henry.

“I know what’s going on,” Henry said.

“Shall I run the clip?” Sharon said, and Pau Garriga nodded.

The auditorium came up on the screen. The camera zoomed in to Henry and the row behind him.

“None of this is CGI,” Sharon said. “It’s a composite of several live feeds.”

The protestor stood up behind Henry, who spun towards him. The protestor began to shout and Henry, grimacing with irritation and contempt, knocked him out with a solid right to the jaw.

“Pause,” Sharon said. And the picture froze on Henry turning back towards the front.

“You predicted this would happen, didn’t you, Kieran?” Garriga said. He read from the pad.

 A robot who doesn’t know he’s a robot must be constrained either by programming – which would certainly give the game away, because robots would notice those constraints – or by the same cultural constraints that apply to human beings – and as we see every day, those are easily over-ridden.

Garriga looked up. “You’re entirely correct, Kieran. What’s the process here?”

Kieran nodded. He could see where this was going.

“If any other robot struck a human being,” he said, “we’d decommission it.”

“Oh, we’d do more than that,” Garriga said. “We’d pay huge compensation to the robot’s owner. We’d throw millions at PR management. And we’d take the thing to pieces to figure out what went wrong.”

Kieran couldn’t look at Henry. “Yes, we would,” he said.

“But, hang on,” Sharon said. “Henry was closing down a perceived threat. The guy might have been an assassin, a suicide bomber.”

“Robots aren’t capable of making those choices,” Kieran said. “They don’t have the predictive capacity or the ethical structures. The rule’s simple. Robots can’t harm human beings.”

“Whoa, man,” Henry said. “Are you’re saying I should be decommissioned?”

Kieran forced himself to look at Henry. “I’m sorry. I’m really sorry.”

“Don’t be,” Garriga said. “A robot’s an animated abacus. You’re buying your own sales pitch, Kieran.”

--------------------------------------------------------------------

Should we have seen this coming?

There are so many ways in which a robot can be identified. The auto-med thing – that was a smartly-engineered litmus test on Kieran’s part. But when you think about it, that sort of giveaway’ll crop up all the time. Any system that utilizes biofeedback would have to be worked around. Which we could do, I guess.

What we didn’t foresee is the problem of a robot having no reason not to behave like a human being. Our robot has behaved in a very human way – and I don’t see how we can excuse that.

So now we have a huge problem, as Kieran predicted. We’re going to have to junk millions of dollars’ worth of investment simply because – if you think about – the experiment was successful.

Even the robot couldn’t distinguish itself from a human being.

Pau Garriga, e-mail to Sharon Everett

--------------------------------------------------------------------

Garriga pointed at the screen. “Play,” he said.

On the screen, Henry unfroze, turned and looked along the row towards Kieran, who glanced to his left and saw the protestor taking a gun from his belt. Immediately, Kieran launched himself over the back of his chair, his hand chopping down on the protestor’s wrist. The gun fell to the floor, and Kieran’s momentum carried the protestor backward, the two of them crashing in a heap into the last row of seats.

“Pause,” Garriga said.

“God,” Kieran said. “I don’t remember that.”

“How many of these blackouts have you had?” Sharon asked.

“Three or four. I know I should have told you.”

“We could have helped,” Garriga said. “But I don’t think we would have. It was all too fascinating.”

Sharon leaned forward in just the way she had that Monday in the office. “The auto-med was right, Kieran, as you worked out. There was only one human on its scan. The plane’s sensor was right too. And if you’re going to make a robot think it’s human, you have to give it a past it can’t check…”

“Doesn’t have to be Australia,” Henry said.

Sharon nodded “Right. It could be a decade of memory loss following an accident. You also have to give the robot a power source that doesn’t need re-charging – even if the prototype’s a little flaky.”

“And you have to feed it illusions,” Garriga said, “so that all this makes sense.”

Kieran blinked. He processed. He turned to look at Henry.

“Oh, man. I wasn’t monitoring you, was I?” he said. “You were monitoring me. It was me.”

“Yes, mate. The first time we met, your bracelet didn’t buzz – but mine did.”

“Play,” Garriga said.

On screen, Kieran scrambled and wrestled until he was straddling the protestor’s chest, and then held him down by pressing his forearm across the guy’s throat.

“Pause,” Garriga said. He fixed his eyes on Kieran. “The rule’s simple, right?”

Kieran dropped his head and nodded. “Yes.”

There was a long silence. Kieran gazed at his hands – his deceptively beautiful and clever non-human hands. He pressed them together, like prayer.

Then he looked at Garriga. “I must be installed with a auto-shutdown routine,” he said. “Give me the code, and I’ll do it.”

Garriga nodded and tapped his bracelet. “Sending now.”

Kiernan checked his own bracelet. “Got it. Thank you.” He smiled. “Thank you – all of you – for allowing me to be human for a little while. I hope that the experiment advances our work and benefits the company. Thank you. Goodbye.”

“No,” Henry said. “Stop.”

He turned to Garriga.

“You don’t scrap a machine that’s unique. That would be commercially idiotic as well as technologically Luddite. And this machine, believe me, is unique.”

Garriga shook his head. “It was unique when it didn’t know it was a machine. But now…”

“No,” Henry insisted. “We filled him with false memories that are completely real to him – they give him a past that informs his present. Kieran’s the only robot who ever had that.” Henry tapped his bracelet and then pointed a finger at the screen. “Watch this – I filmed it in Lima.”

The picture was shaky, and the lighting was smudgy red and purple – but the figure holding the old-school microphone was unmistakably Kieran.

“Check it out,” Henry said. “I swear to God, I was close to tears.”

He pressed Play, and the music began. Two bars intro, and a voice came in. Kieran’s voice.

The first time ever I saw your face,
I thought the sun rose in your eyes
And the moon and the stars
Were the gifts you gave
To the dark and the endless sky, my love,
To the dark and the endless sky…

As the last note faded, Henry spoke. “You don’t scrap that, Pau. You can’t.”

Garriga turned from the screen and looked at Kieran.

​

“Wow,” he said. “That’s not bad.”

He smiled. “Not at all bad for an abacus.”