Wan Yining,
Hello. Due to the urgency of the situation and the lack of suitable paper, I am writing this letter on the back of a painting from my collection.
By the time you receive this letter, the world will have completely changed, and in this altered world, you are the only solution I can see.
Perhaps you will not quite understand much of what I am about to say, or perhaps you are finding this present world harder and harder to comprehend, but I trust you. I believe you will make the right judgments and do the right things.
Where should I begin? You are twenty-two years old this year, which reminds me of how I was at twenty-two. Back then, it seemed as if the outside world was bright and splendid, as though the finest tapestry of life was just beginning to unfold. At that time, just like you, I was being trained as a communications researcher. That was during the era of the Great Migration.
...
A car accident altered my view of the world. I believe you should generally understand the technical principles of ABS. That was a braking technology on the eve of the great explosion of autonomous driving technology. When a vehicle brakes, it automatically modulates the braking pressure based on the movement of the wheels to prevent them from locking up. It was considered an exceptionally safe technology.
Yet it took a dangerous step: the driver surrendered their own control over the vehicle. And on that occasion, the ABS system in the car I was driving malfunctioned. While in transit, the vehicle suddenly braked, and I was rear-ended by a car behind me and launched off the road.
At such a moment, for someone who believes in God, it would be a moment of revelation. It profoundly shook my trust in technology—especially in technology that humans cannot control.
In the tens of seconds I sat in the car waiting for the fire engines to arrive, still shaken, I reviewed the past and future of ABS and autonomous driving technology. What truly terrified me was that it seemed no one could stop any of this from happening.
Just like the example that had just happened to me, it would surely be treated as a low-probability event and relegated to the realm of negligible margins. In the data, it would definitely be auto-
We know that whatever can go wrong in terms of technological logic inevitably will go wrong; it is only a matter of time. Moreover, due to human optimism and the hype surrounding new technologies, reflection almost never occurs until we slam into a colossal brick wall.
There are some mistakes we can afford to make, but there are others that bring irrecoverable ruin. For instance, if autonomous driving technology causes traffic accidents, at most a few thousand people worldwide might die.
Is there, then, any technology that could cause the demise of all humanity? Yes, nuclear weapons are such a technology. Fortunately, however, at the eleventh hour before humanity crossed into danger, by a stroke of serendipity, everyone reduced their nuclear arsenals, preventing that catastrophe from occurring.
And what is the nuclear weapon of today? AI.
Whenever I think of this, I feel thorns at my back, as if sitting on a bed of nails.
As human beings, we must realize that this world has not always existed as it is now. The life everyone enjoys is not inherently guaranteed. If we forget the foundations beneath it, danger will arrive. This world does not inevitably progress, nor does the chronological order of different eras necessarily mean advancement; what we imagine as progress is merely a contingency. The laws summarized by historians are nothing more than hindsight; everything could very well have turned out entirely different.
From the vanishing of the dinosaur era to mammals crawling ashore, dressing up like apes in human clothes and becoming the new masters of the continents—were these monkeys truly stronger than dinosaurs? Smarter? Or were they simply luckier?
The same holds true for our entire human society. The disappearance of slavery was not due to an increase in human virtue, but to the disappearance of the viability of slavery. When new possibilities emerge, there will always be someone who tries them.
This world can change in an instant; civilization can revert to barbarism overnight. Civilization is merely a paper-thin layer between the Earth's biosphere and its atmosphere. We cherish it immensely, but we must also recognize its fragility.
...
A human life passes in a flash like a white horse leaping a crevice; a millennium of history is gone in the snap of a finger. Poets might merely sigh at this, but a realistic politician has no choice but to account for its reality.
Does a human life span only a hundred years? Will a thousand years of history suffer a resurgence? These are not flights of extraordinary imagination, but assessments of probability.
This demands exceptional intuition and keen observational insight.
You may have only learned from textbooks why we refuse to construct any physical or virtual entities of artificial intelligence. In truth, this is not merely an ethical issue, but also a struggle for power.
Technology is a pole of power in contemporary society; every single system has technology involved in it. We cannot ignore its reshaping of the entire power structure, and every technology carries the potential to be weaponized.
This is true for biotechnology and algorithmic technology alike. That is precisely why the Coalition Government has always maintained an Ethics Committee. The Ethics Committee not only suppresses the potential weaponization of civilian technology, but also oversees the government to ensure it does not weaponize technologies that are excessively dangerous. Nuclear weapons remain the greatest reminder of our age.
So, whose power struggle is this? Naturally, it is between humans and artificial intelligence!
Technological advancement has always been a positive feedback loop, especially when it comes to weaponized technology—that is how arms races come about. When the weapons used by police grow more formidable, criminals find ways to develop even stronger weapons, and the police, in turn, continually upgrade their armament and elevate their artificial intelligence capabilities, until one day this positive feedback loop gives birth to a mighty AI military. And that military will be the nightmare of all mankind.
They may argue that we will use only a fraction of AI technology rather than the whole, thereby preventing the emergence of conscious artificial intelligence. That is truly utter nonsense spoken in broad daylight. How could that possibly be?
If a robot lacks self-awareness, it means it cannot make decisions; it requires a human to continually make decisions on its behalf. But humans are lazy. Eventually, someone will delegate a portion of the decision-making authority to the robot, until all decision-making authority is handed over. And robots never slack off.
Perhaps someone will also argue that once robots acquire human consciousness, they will possess humanity; and having humanity, could they not then exist as members of the human race?
If these people read a little history, they would not be so naive. Even as fellow humans, people fought one another over bloodlines, fought over religious beliefs, and even more ridiculously created different nationalities to wage war against one another throughout the tens of thousands of years before the 22nd century. Tribes sharing the same language and the same ancestor could turn on each other as mortal enemies. How can you guarantee that these robots will not become the common enemy of mankind?
This is not science fiction; this is what happens in reality. Realpolitik logic does not dictate that you pose no threat simply because you have no intention of becoming an enemy; rather, we are safe only when you lack the capability to become an enemy. The logic of technology does not state that what has never happened in the past will not happen in the future; rather, if something is technologically possible, given enough time, it will inevitably happen.
The bargaining chip Wade might use in exchange is to guarantee an extra fifty years of life to the Presidium. All those old fellows lack might just be more life. When they can live long enough, they may want nothing but slaves; they might not even want their own descendants. Due to matters of inheritance, descendants are potential enemies, whereas only slaves are safe.
I imagine Zhang Beihai has already devised a way to create slaves, and they are even more useful than ordinary slaves. They are the robots he manufactures. They obey absolutely, follow orders to the letter, and eliminate any need to worry about betrayal, surrender, and other concerns that humans must fret over.