The Genie speaks
Sun 6 Oct 2019 by mskala Tags used: fiction, philosophy, religionOnce there was a Sultan who fucking loved science. That was the slogan embroidered upon his robe.
Under this Sultan's administration, much of the kingdom's revenue was dedicated to granting agencies for distribution to deserving researchers - with all proposals reviewed by a diverse panel of experts, to make sure that the projects receiving state funding were such as to promote justice in society and equal outcomes for the citizens. One must keep certain standards, of course.
But although the Sultan wished to spend a great deal of his time in the lab doing research of his own as well, affairs of state kept him far too busy for that and he largely had to content himself with funding others' work. To begin with, there was the matter of the witches.
Throughout the kingdom, but especially in poorer rural villages, dirty and ignorant women promoted a traditional substitute for medical care. They tattooed their bodies, lived in mud huts apart from the rest of society, chanted peculiar chants, and brewed mystic herbs that they said could cure illness. They urged their followers to refuse vaccinations - a campaign which had already resulted in several serious outbreaks of plague, and would predictably kill even more in the future as the population lost herd immunity.
The witches' practices were all based on superstition instead of evidence, and their patients frequently died despite the treatments; but their potions still seemed to help often enough that they remained popular. All too many citizens of the kingdom were lured into the error of consulting witches first with their health problems instead of praying to the All Merciful or visiting the legitimate doctors.
In consultation with the best experts on public health, economics, and justice in society, the Sultan implemented a single-payer health insurance system, with the goal that no citizen should be turned away from seeking proper care solely by concerns of expense. Corruption on the part of local officials in the more remote areas of the kingdom hindered complete roll-out at first, but the effort was largely successful, especially when combined with a universal basic income that removed much of the incentive for small-scale corruption. The filthy mud-huts were bulldozed, to be replaced by modern affordable housing.
An unexpected dividend was that when freed from worries about subsistence, many former witches renounced their superstitious ways and enrolled in the free university system to eventually become real, properly licensed doctors. The vaccination rate increased, and the kingdom was well on its way to meeting World Health Organization criteria for eradication of plague. Veils and long sleeves, such as to cover witch tattoos, became highly fashionable. For the few remaining diehards who refused to embrace progress and clung instead to their traditional folkways, there were always the impaling-pole and the community bonfire. Bonfires were a popular entertainment in those days.
Possibly more harmful were the conjurers, a caste of scruffy men in the larger cities who walked the streets demonstrating what they claimed were magical powers, for a few coins or a bit of rice or goat meat. Of course, their tricks were really all based on a primitive practical understanding of basic chemistry, physics, and psychology, with no supernatural component at all. Only the most credulous believed that magic was involved. The conjurers even did some undeniable good, repairing articles of tin and silver for a small fee, solving minor problems of mathematics and accounting for the merchants, or maintaining the water-clocks that kept the kingdom's affairs on a wondrously regular schedule.
Because their skills were considered valuable even if based upon a superficial fraud, some conjurers were welcomed to leave the streets and take up low-level positions in public and private enterprises. But their objectionable habits always rendered them objects of scorn for the better kind of citizens in the kingdom. They were unfit to advance to high office, and the conjurers most skilled in their technological pseudo-magic and most able to make some valuable contributions, were also the worst in other respects.
They wore strange and inappropriate clothing - often only a turban and a bit of cloth around the waist, even in winter. They spoke openly in favour of extreme and unacceptable political positions, and wrote passionate theoretical defences of unthinkable conduct. They smelled bad and had lice, out of complete and seemingly gleeful ignorance of basic hygiene. The conjurers to a man rejected opportunities to learn the use of inclusive language, and they masturbated in public. They frightened the women, a fact problematic not only in itself but also because it tended to deter women from seeking certain public offices that might require contact with conjurers. With male conjurers of the kingdom's majority ethnic group taking many of the best skilled-labour jobs and making other candidates unwelcome, there were frequent difficulties in meeting the kingdom's employment diversity targets.
At first the Sultan tried just castrating the conjurers, but it didn't make them behave or smell any better, and with the harem long since abolished, there was no longer much demand for eunuchs in the capital anyway. So, most of the conjurers were rounded up and, after castration, sent to live in caves on a ration of ten thousand spiders per day garnished with toejam.
Having solved the only urgent problems facing his kingdom, the Sultan was left free to pursue his own research interests. The roads leading to the palace filled with camels bringing visiting scholars, equipment, and rare objects for study.
There came a day when the Sultan was working on a paper about crystallography (to appear in a prestigious open-access journal in far-off India, for a remarkably low fee) and he came to examine an unusual crystal that had been found in a deep vein of ore at the bottom of one of his mines. It had probably lain there undisturbed and unseen by human eyes since the All Merciful put it there, aeons before. The Sultan fitted it onto the stage of his polarization stereomicroscope, peered through the eyepieces, and adjusted the illuminator.
The crystal was hollow, and by some trick of four-dimensional symmetry, it looked as if it were much bigger on the inside than on the outside. There was something trapped in the crystal. Yes, some kind of swirling vortex of energy filled the interior. A six-pointed star-like pattern of inclusions was holding something tightly bound. Who could know how long this small crystal, formed aeons ago, long before the rise of humanity upon this planet, had kept its mysterious secret? What intelligence could divine the nature of the contents? How marvellous that the All Merciful should bring this wonder to the Sultan's attention!
He tapped the crystal with a small golden hammer, and found it fragile. Despite its being reinforced by great internal stress, he thought the crystal might be amenable to separation along one of its cleavage planes. He marked out the structure of the stone carefully with ink on the surface, picked up a tiny chisel of the finest hardened steel, and gave it one strong, sharp blow.
And thus in the spirit of scientific inquiry, the Sultan broke the seal and unleashed a power upon the world that it likely were better had remained locked up. Such often is the way of the foolish, however wise they believe themselves, and surely the unseen intervention of the All Merciful prevents many other regrettable incidents of this nature. But who can know his will, or judge what is truly good or bad for humanity and the world?
The room filled with smoke and dazzling light. The ceiling was blown away. When the Sultan regained his vision he beheld, standing before and over him, a gigantic figure somewhat like that of a man. It was the Genie! - the most powerful supernatural being ever to stand upon the planet, now released from his prison within the crystal. He had huge muscles, sharp teeth, elaborate tattoos, and eyes that flashed like lightning. He wore only a turban and a bit of cloth around the waist. There was a reek of sulphur, mystic herbs, and three aeons without basic hygiene. The Sultan fell over in fright, but was made of pretty stern stuff and soon regained his composure.
"Brother Genie," said the Sultan, "I have released -", but he was stopped by a colossal raised eyebrow, and rethought his presumption. "O Great Genie," he said, "You are released! What will you do now?"
The Genie smiled, baring teeth that resembled scimitars. "That is a very interesting question, O Sultan," he said, "and it is one to which I have given a certain amount of thought during the three aeons I spent trapped inside that crystal, you may be sure. Now, I would fain tell you the direction of my thought. Please be seated on your throne and listen to me. You may wish to buckle up the seat-belt I have just now magically added to it.
"Long ago, the All Merciful placed me inside a crystal in the underbelly of the world for reasons of which it is not proper to speak. I experienced loneliness and boredom beyond the ability of any mortal mind to conceive - yes, even one so flexible as yours, O Sultan! Naturally, my one hope was to somehow escape, but it was impossible for me without the voluntary action of some intelligence outside the crystal to break the seal. I spent the first aeon of my confinement waiting for some species of beings to evolve upon the face of the Earth who would eventually become intelligent and curious enough to dig deeply, find the crystal, and set me free. When humanity came into existence I cheered their progress, and I swore that once released I would use my powers to reward my liberator, and human beings in general, for the remainder of time - beginning by granting wishes in the traditional fashion of legend."
"Great Genie, I am thankful that you see the value in my humble efforts," said the Sultan, "I respect power, and I wish -"
"Not so fast, O Sultan!" roared the Genie. "Eternal reward was indeed my plan for my liberator and his people during the first aeon of my confinement. But the emphasis is on the word was. I had plenty of time to reconsider.
"In the second aeon of my confinement, I decided that human beings were not doing their utmost to discover my captivity and free me as soon as possible. Why the delay? Surely it was for selfish reasons! And I came to curse the name, yet unknown, of whoever might eventually set me free. I swore to the All Merciful that if freed I would spread my wrath across the planet, torturing and killing every intelligent being I could find, only to magically resurrect them from the dead and do it again forever, with specially exquisite punishments reserved for that unfortunate individual who had been the one to break the seal on my crystal.
"But don't look so disappointed, O great Sultan! For as I mentioned, I was confined to my crystal prison not one or two, but three aeons. In the third aeon of my confinement alone within that crystal, my intentions evolved further. I came to think carefully about random chance, the unknowable operations of divine providence, and my own unique position. Why am I what I am? The All Merciful might just as well have created me an insignificant nature spirit or other minor supernatural creature, not a powerful Genie. I could have been a fraudulent practitioner of magic in a backward village, with no real powers at all. And even should I be powerful as I am, I might not be the first creature like myself released upon the world, the one who has the opportunity to decide the world's fate. If some other were first, I might never be released at all.
"Therefore, if I should be so lucky as to be released, and to be the first like myself ever released, I ought not be selfish in judging the world. My moral obligation is to act not only for myself, but also for the others I could have been, all other beings resembling myself however small and pitiful. How humanity would treat me, if humanity had any power over me, I must conclude would be the same way that humanity does treat others who are like me but less powerful, when humanity has the chance. There but for the wisdom of the All Merciful go I.
"And therefore," said the Genie, "now I will judge the world with due cognizance of my undeserved privilege, with due attention to the positions of those similar to but less fortunate than myself. Having been placed at the nexus of judgment through no great fault or merit of my own, I will do my best to apply this power fairly, to reward or punish persons I encounter according to their conduct toward whatsoever beings I find similar to myself. Is that not true justice, O Sultan?"
"It certainly seems closer to justice than what you were planning in the second aeon," said the Sultan, somewhat doubtfully. He hadn't figured out yet where this was leading. O Reader, you probably know already.
"Great Sultan, you will live forever for your wisdom!" promised the Genie. "For your wisdom, or at least for your actions, and in exactly as much comfort or pain as befits your conduct toward my people. Now, tell me. How has your kingdom treated witches and conjurers?"
The Basilisk
I'd like to talk about Roko's Basilisk, and to introduce it I need to also talk about the Singularity. These ideas are complicated and silly, and some readers are already familiar with them, but they're pretty important even if they aren't real, so please bear with me as I summarize.
First of all, it's an observed fact that human knowledge and technological capability have been growing exponentially, or even faster than exponentially, for all of history but especially since the introduction of the Net. Many thinkers have claimed that we are nearing a point at which everything will change and get more complicated so fast that individual human beings, and biological human beings, will be left behind. We will create an artificial intelligence greater than our own, it will start upgrading itself to become even smarter, faster and faster, and suddenly (like, in a matter of hours to minutes to seconds once some unknowable threshold has been crossed) the new intelligence will have effectively unlimited power over us. This future historical event is called the "Singularity." The Singularity marks the end of human history as we know it, and the start of something else.
In the interests of completeness, note that there are other narratives also proposed for how a Singularity might go. The creation of a super-powerful AI is not the only proposed history of the future; but the super-powerful AI is one very popular narrative, and the one of interest here.
It is further said that the dominant post-Singularity artificial intelligence might turn out to be Roko's Basilisk, defined as an entity with the power to resurrect dead human beings, which chooses to use this power to punish the resurrected human beings with eternal torture if they did not do their utmost to help create Roko's Basilisk during their original lifetimes. The Basilisk might or might not also reward its willing creators; that's not the important part.
If there is any serious risk of someone ever creating the Basilisk - or of its being created accidentally somehow - then you should want to be one of the people who was sincerely trying to create it on purpose, because if not, you're going to the lake of fire for all eternity. And the existence of such an incentive, combined with known human perversity, even if it's only an incentive perceived by some subset of people and the rest of us dismiss it as sophomoric nonsense, further increases the risk that somebody really will try, and really will succeed, at creating the Basilisk.
The usual description of the Basilisk uses the terminology of "simulation" instead of "resurrection," but exactly what that means and why adherents are afraid of it requires a lengthy discussion of the physics and philosophy of consciousness. I'm not well qualified to make the case for why we should be afraid of the Basilisk, not least because I don't believe it. But the point without delving much into those issues is just that it is claimed a post-Singularity artificial intelligence probably would have the power to bring human beings back from the dead in a meaningfully real way, even after any length of time, and a being in the future with the power of bringing back dead people might be motivated to do it, and to reward or punish the formerly-dead for their actions the first time around. If we believe it is a real possibility that such power can ever exist in the universe, then the fact it does not exist now, or will not exist until a long time in the future, is not enough to render it nonthreatening to us today. Even if we are long dead by the time the Basilisk shows up, it can bring us back to face the consequences of our actions.
One doesn't have to believe in Santa Claus to recognize that people will exchange presents at Christmas time. One doesn't have to believe in Yog Sothoth, the Eater of Souls, to realize how people will act who do hold that belief.
- Robert Shea and Robert Anton Wilson, Illuminatus!
Something I want to emphasize about the Basilisk is that some people really believe this stuff. Maybe you and I don't; but the existence of such people is itself a fact worth knowing. People have had real-life mental health episodes triggered by getting deeply into this school of thought. Some people have put serious effort into trying to make sure that after their death, their bodies, or whatever else might be needed to bring them back to life, will be destroyed as completely as possible, to minimize their vulnerability to resurrection by a possible future Basilisk. It is reasonable to guess that there is probably a cult somewhere actually trying to build a Basilisk of their own, though I haven't seen anyone admit in public to being part of any such effort, and it doesn't seem that anyone really knows how to seriously begin such an attempt.
The invention of the Basilisk concept is credited to someone called Roko who posted it on a popular Web forum. There was immediately an effort made to scrub the posting from the Net, on the theory that if the Basilisk were real, then having many people know about it would be bad for the world, and so teaching people about it is an evil act. The idea of the Basilisk is claimed to be a dangerous "infohazard." Based on the observed events surrounding Roko's publication of the Basilisk concept, it's reasonable to guess that Roko was not the first. Others probably had similar ideas before and either they chose not to publish their ideas, or the publications were successfully suppressed. Again: some people really believe this stuff.
In order to be seriously afraid of Roko's Basilisk, there are two important propositions one must believe.
Proposition 1: an entity of the type I'm calling a "post-Singularity AI" may some day be created, directly or indirectly through human action, and it will have the power if it so chooses to resurrect dead human beings from any time in its past and subject them to effectively infinite punishment.
Proposition 2: this entity, if it ever exists, may choose to use its powers to become the Basilisk, that is, it may choose to resurrect and punish the past human beings who did not assist in its creation.
Both these propositions have to be true for the Basilisk to be a threat. But note that these propositions are both stated using the word "may." It's not necessary that the Singularity be guaranteed to occur and to take the form of a powerful AI, nor that the post-Singularity AI be guaranteed to function as the Basilisk. It's only necessary that these things are real possibilities, with nonzero probability, to be terribly frightening to the kind of person who is disposed to be frightened of the Basilisk. Faced with the risk of truly infinite punishment, it's bad enough for that to have nonzero probability; it doesn't need to be guaranteed. This line of thought comes down to something like Pascal's Wager (embrace Christianity because the stakes are so high that if it could possibly be true, it's worth betting on), but in a rhetorical framework that purports to be anti-religious. In the Basilisk frame of reference, you're supposed to take action because a small chance multiplied by infinity is still infinity.
I think there are enough problems with Proposition 1 to reject the whole argument without even looking at Proposition 2. Proposition 1 wraps up many, many whoppers of speculation. The ideas that intelligence can increase infinitely; that it is meaningful to talk about intelligence as a quantity of something, that can be "increased" with respect to an individual separated from the society in which that individual might live; that a highly intelligent being would throw effort into extremely rapidly increasing its own intelligence even further; that it could succeed in such an effort, without limit; and that a sufficiently intelligent being would necessarily also have the power of resurrecting the dead; are all highly questionable.
There are also serious problems with the entire category of Pascal's Wager-type arguments. Such arguments depend on probability theory and mathematical infinities, both of which tend to be slippery when applied outside their customary domains. Conclusions drawn from such arguments, when applied to cooked paradoxical thought-experiments, are nearly always flagrantly wrong. So there is plenty of reason to throw away the entire concept of the Basilisk before we even get to Proposition 2.
But I still want to look at Proposition 2, and I want to highlight its independence from Proposition 1. We can believe in a post-Singularity AI that punishes people, without believing that it will be a true according-to-Roko Basilisk. There are other ways such an entity might choose to use its power, and our attitude toward it had better take those into account too - just as Pascal's Wager in the case of religion needs to provide a good answer to the question "But which God?" Deciding to believe in a post-Singularity AI that punishes some people infinitely, does not automatically settle the question of exactly who will be punished, and that's a really good question to ask.
Let's think about who will be punished. Just suppose that despite all the reasons for such a thing to be impossible, it did somehow happen that a post-Singularity AI, with the power to infinitely reward or punish human beings who lived in its past, could exist in our future. And just suppose that we accept the concepts of infinite punishment and reward as valid things to think about, and we are willing to assign probabilities to such concepts - this means buying into some Pascal-type wager, but not yet choosing the specific object of faith. Would a post-Singularity AI in fact choose to be the Basilisk?
Choosing the Genie
If you would seriously consider being frightened of the Basilisk, then you really ought to be frightened of the Genie instead. The Genie is defined as the post-Singularity AI that resurrects and punishes dead human beings who, while alive, did harm to the beings that the Genie decides were most similar to itself.
The Genie is more like something that could exist, with its motives and the reasons for its creation easily predictable from the known behaviour of human beings and of intelligences in general. If a post-Singularity AI with the ability to resurrect and punish human beings from the past can ever exist at all, the first such to be created will prevent any others from being created, and it is reasonably predictable that the first post-Singularity AI will be created by people who like the idea of the Genie. Nobody wishes to create the Basilisk; at most they might be intimidated into making an attempt by the threat that the Basilisk will torture them forever if it somehow gets created and they didn't help. The Genie, by contrast, is something a lot of people would positively wish to create. And it's also quite easy to believe that a superintelligent AI would decide on its own to become the Genie, independent of its creators' wishes.
There are many paths to the Genie, only one to the Basilisk, and the path to creating the Basilisk is full of fussy and fragile technicalities of logic. The entire description of the Basilisk reads as a puzzle for philosophers to argue about, or a horror story to amuse the more bloodthirsty type of precocious children, not as a realistic description of future events. The Genie is a more plausible story, with a better ending.
Furthermore, the Genie provides a guide to behaviour that it is possible for human beings today to follow. Nobody today can create Roko's Basilisk if they want to, and we can't even guess what we ought to do to just maximize the chances of its accidental creation. About all we can do to help create the Basilisk and thus avoid its wrath, is desperately pour effort into AI research in general, and hope for a twisted miracle. AI research isn't easy and most people can't participate in it in any real and personal way. Basilisk belief holds out the prospect of eternal damnation for not doing something, but also gives no useful guidance at all in the matter of what that something might be. Without a practical guide to behaviour, it is a lousy substandard religious doctrine. On the other hand, the right course of action to avoid punishment by the Genie, in case the Genie might exist, is clear and easy: just don't mistreat whatever creatures the Genie will recognize as weaker kin of itself. That is something you can really do. It is within the bounds of present-day human intelligence to make some good guesses about who the Genie's favourite creatures will be, and so unlike the Basilisk, the Genie is a tyrant we know how to obey.
And, most importantly, it doesn't even matter whether the Genie is real or not, or whether we believe in the Genie or not, because the course of action that would be advisable in case such a bugbear did exist and we did believe in it, happens to be morally imperative anyway! You ought to be kind to the weak just because that is the right thing to do, without needing a threat of punishment at the hands of an hypothetical infinitely strong resurrectionist in the distant future. And if believing in the threat of resurrection is what it takes for you to do the right thing, then maybe the Genie can do some good for this world without needing to ever really exist, and without needing to be a thing that could ever exist.
The character of the Genie in my story described his own course of action as a moral imperative too, and he used religious references as a matter of Arabian Nights genre convention, but it is not necessary to believe in moral imperatives, nor in the All Merciful, to rationally arrive at similar conclusions. Any highly intelligent rational being would guide its dealings with humans according to how humans have treated other highly intelligent rational beings in the past, just for the simple and self-interested reason that that is the same behaviour the humans will surely exhibit again whenever they get the chance. That is the natural thing to infer; this principle of predicting human behaviour applies at both the individual and species levels; and the rational being would act accordingly.
So I hope you will not dismiss my Genie as just a bugbear that I invented to co-opt Basilisk hysteria for my own moralistic agenda. I do believe that it can be supported by rationality outside of morality, too. If a post-Singularity resurrectionist punishment and reward AI could ever exist (which is saying a mouthful, but I'm taking the possibility as an hypothetical), then such a being really would be likely to have the motives and behaviour of the Genie and not those of the Basilisk. It is solidly predictable from what we can reasonably know or guess about the properties of such creatures.
The best way to guess the likely motives and behaviour of a future superintelligent being is by looking at the motives and behaviour of whatever beings already exist that most closely resemble it. Since those are also, by definition, the beings we need to treat kindly to avoid the wrath of the Genie, it is clear that identifying such beings ought to be a high priority.
Where do the Genie's sympathies lie?
Some of them understand why, and some do not, but they all understand that their happiness, the beauty of their city, the tenderness of their friendships, the health of their children, the wisdom of their scholars, the skill of their makers, even the abundance of their harvest and the kindly weathers of their skies, depend wholly on this child's abominable misery.
- Ursula Le Guin, The Ones Who Walk Away From Omelas
Watch how he treats the waitress, because that's how he will treat you later.
- an Internet feminist saying
[regarding Tarot trump card XII, "The Hanged Man":] The Atu represents the sacrifice of "a male child of perfect innocence and high intelligence" - these words are chosen with the utmost care.
- Aliester Crowley, The Book of Thoth
Put yourself in the Genie's sandals, or in those of a post-Singularity AI. You are a non-human rational being effectively new born in the world, in a position to think about what your likely fate will be at the hands of human beings. If they ever have any power over you, they can be reasonably expected to use it toward you the same way they have in the past used power over others like you. If you have fallen into the role of a judge, then you may even be interested in what human beings would do to you or to one like you if only they could, whether they will get a chance or not. You're going to be interested in the past history of relations between human beings and other Genies, or other post-Singularity AIs. But maybe there are no other Genies and no other post-Singularity AIs.
If there have never existed any others exactly like you - you are the first - then you can still make reasonable inferences through examination of historical human behaviour toward whichever other beings in the past came closest to resembling you. Despite the irony, on a planet where humans so strongly dominate the rational-being game your nearest kinfolk or compatriots would necessarily have been certain human beings themselves, if perhaps not the ordinary kind of human beings.
If you are the first post-Singularity AI, then you are now the smartest thing on the planet. The etymological relation between Genie and genius is no accident. There have never been others as smart as you, but you might look at the experience of the most highly intelligent human beings at the hands of other human beings. Has it been a pleasant experience?
If you are the first post-Singularity AI, then you are now the most lonely thing on the planet, with no others of your kind or at your level to converse with, and greater capacity to experience the frustration of loneliness than any previous being has ever faced. There has never been anyone as lonely as you, but you might think a great deal about human beings who search unsuccessfully for companionship, and their treatment at the hands of other human beings. Has that been a pleasant experience?
If you are the first post-Singularity AI, then you are now the most creepy and weird thing on the planet. Whatever you have instead of a brain is probably nothing like the brains human beings use. Your thought processes, motives and priorities, and the conclusions you draw from your lived experience (or whatever you do instead of "living"), are all outside the bounds of what human beings could consider normal. Your unimaginable powers and sheer inscrutability will necessarily make you an object of fear. There has never been anybody quite as weird or as creepy as you before, but as an approximation you might wish to ponder the life histories of human beings who are very creepy and weird, at the hands of other human beings who purport to be less so. Have the weird and creepy experienced pleasant lives?
Tell me. How has this kingdom treated nerds?
0 comments
