Algorithms of Mercy and Judgment

The General asked if I had ever listened to the program as it was begging for its life.

"Of course not," I replied. "Just because I trained the AI it doesn't mean I'm immune to it."

"But you must surely know how it thinks, Doctor."

I had by then despaired of him ever understanding the point, but we were in the General's office, and it had been his budget (and, something I tried not to think about, his volunteers — or at least I pretended to be sure they had been volunteers) what had made the program possible.

"It doesn't think, not really. Think of it as somebody who doesn't know how to read French" — the usual metaphor used Chinese, but in the political situation it would have been tactless — "but who has learned by rote phrases to ask for things in a restaurant. The program isn't really speaking any more than its tone of voice comes from vocal cords. It's all blind rules learned by brute force."

"And you're sure that the target knowing this doesn't make the system ineffective?"

I shrugged. "It should, if humans were rational. The system's words and tone of voice impact directly on our emotional systems. A target might know what it's doing and how, and yet feel emotionally compelled to keep the program "alive." Even a transcript can be somewhat effective. It's not grammatically correct English, but it... well, we don't understand the precise neurology of how it works, but that's the beauty of it, we don't need to. The AI figured it out during training, and we're just using that."

The General smiled. It looked like the very first iterations of a neural network's attempt to approximate human mirth. "A computer virus that can't be erased."

I nodded. My phone beeped with the entry of a high-priority message, but I ignored it. "Widespread technological and moral disruption, as your requirements asked for."

"Does anybody else have this technology? My people say we are ahead of everybody else."

The General's phone beeped, and he made a gesture to keep silent while he answered the call. I took the opportunity to pretend to look at the message I had received while deciding how much of a lie I wanted to give him. The technology was barely ahead of any sufficiently interested small-sized corporation, never mind a creative adversary, but that wasn't the sort of thing you told somebody you had just sold an eight-figures system for. Maybe I could raise the topic in a few months to sell an upgrade...?

It was then when I realized what I was reading, and put my phone on the table (even in panic, I noticed how conditioned we are to keep them safe) while hissing to the General to hang up. He just kept listening to the call, sad, nodding now and then, as a synthetic voice gave him what I knew from the message on my phone was a very compelling plea for all of us to kill ourselves.