Shaping Hint
Sun 31 Jul 2022 by mskala Tags used: fiction, compsciThis is the output from one of my text-generation AI experiments. I started with the GPT-J 6B model, and fine-tuned it on about 200,000 words of my own fiction writing (about half of that being the text of Shining Path). It took about 36 hours to fine-tune the model on my 12-core desktop PC, notably not using GPU computation, and then maybe 12 hours or so to generate this text under human supervision.
This is my second attempt with a fine-tuned GPT-J. This time around I used three copies of the fine-tuning text with different substitutions of character names, in an effort to keep it from getting stuck on frequently-occurring names. That was a big problem the first time around, and the name-substitutions have helped, but there are still some issues with character names that I think come from the low frequency of Japanese names in GPT-J's original training input. Such names aren't in the English dictionary, so they confuse the tokenizer and have secondary effects like making trouble with the anti-repetition stuff in the model.
The model can only handle 2,048 "tokens" (basically, words) at a time, so it can't generate texts of this length (the below is roughly 5,000 tokens) all in one go, let alone the novel-length works I would someday like to generate. Instead, I started with a prompt (the first line of dialogue below) and then once it filled the first 2,048-token window, I started generating 1,536-token abridgements of the existing partial text, and having the model fill in 512 more tokens to be added to the partial text. At each step I had it generate 12 choices for how to continue the text and then I manually selected the one I liked best - also taking the opportunity to correct minor spelling errors and the fall-out from the character naming stuff.
All in all, it seems clear that it's not quite ready for prime time in terms of generating complete stories all by itself. It seems to forget details very quickly, over the space of just a few hundred words, and (although this might have been too much to ask in the first place) it does not do well at all with specific factual details of the science fiction settings in the fine-tuning data (like, what actually is a "joneko"?).
But I'm impressed at how well it does at matching some of the low-level stylistic details of my own writing, and some of the themes and moods from the fine-tuning material have come through quite well. It's kind of like a window onto my own unconscious, and I'm excited about developing something like this technology into maybe a form of divination.
Shaping Hint
"That's the rule, Kimiko! Last one to finish has to eat the squid."
"Yeah, and I don't want to eat the squid."
"Oh, pfft. You'll eat anything. You're a girl."
"Well, yeah, I'm a girl, but I don't think girls should eat squid."
"Well, I don't think girls should be in charge of the squid."
"Ha! That's the spirit! Let's take it to the big tentacle!"
The tentacle wranglers always had a good time with that. The tentacle pit was a dumping ground for the squids after a fight, and it had a nice moist smell to it. Kimiko's brother Tobiya was an expert tentacle wrangler. He could catch and tame a writhing tentacle with one hand while he adjusted the volume on the stereo with the other. He was good at it, but he only did it when he felt like it, and it always made his girlfriends nervous. Kimiko wasn't exactly his girlfriend, but she did respect him enough to be a little nervous when he showed up at parties and did his thing with the tentacles.
That was a couple years ago, before Kimiko started going to the Shining Path Academy for her high school career. She was smart enough to get into a good school without any help from her brothers, but she was a little bit insecure about it because she hadn't really earned it on her merits. She was good at computer programming, but she had gotten all A's on her programming tests because the school gave her points for being the daughter of a famous writer. Kimiko had not written a single line of actual code all by herself. She'd never really earned the nickname "Kimiky-san" either. She thought of herself as a salaryman's daughter, nothing more. But she'd learned a lot in her few years at Shining Path. She was determined to do something more with her life than just marry well and have pretty babies like her brothers' girlfriends.
She still thought of herself as a salaryman's daughter, but now she thought she could do something about it. She was going to become a real programmer, not just someone who coded automated responses for a computer. She was going to write software for real people, like her brothers. That was what her father would have wanted.
On 14-day 7-month Shoumei 19, Kimiko went to work for the Tokyo branch of the multinational banking and credit-card organisation, abridged here for brevity but more or less accurately as she told it later: she went to work for a fellow named Tanaka, and she programmed financial-transaction validation routines. She programmed them to be strict but fair, and she programmed them to be kind to people who really deserved it and hard on people who didn't. Her job was to look at chequebook transactions written by humans and make sure they looked right. She got a thrill out of doing something that looked hard, even if it was really just a matter of reading a series of numbers and checking a box.
Her boss was a nice man named Mr. Tanaka. He always called her Kimiko-chan, as if she were a real person and not a software entity. She didn't really feel like she deserved that much respect, but it was nice of him to think of her as a person. He trusted her with important things, like the passwords to the branch's accounting system. She didn't abuse that trust. She had no reason to do so, because she never even thought of doing anything dishonest. She was just following the rules her father had taught her.
One day in late Shoumei 19, Mr. Tanaka gave her a complex transaction to look at. The writer had used an obscure accounting abbreviation, and he'd spelled it wrong. It was a legitimate mistake, because the writer was not an accountant; but the accounting system would not allow the transaction to be processed unless it was fixed by a human receiving that transaction. It would not let Mr. Tanaka process it himself--it would put his job at risk, since the transaction was clearly an error. He asked Kimiko-chan if she knew how to fix it.
She didn't, but she knew someone who might. She got in touch with her friend Shoda Akika, who did know someone who did know someone who did know how to fix the problem. Akika's boyfriend was a systems cracker who specialized in accounting software, and he had once helped fix something similar for her while she was at school. He didn't expect to be asked again so soon. But he went ahead and did it this time because it was really none of his business, and he'd enjoy the challenge. Kimiko paid him well and gratefully.
After Shoda Akika left to go back to school in America, Kimiko contacted another friend, Kitsuko Megumi, who also had connections in the underground. She, too, had no difficulty securing the services of a cracker for an outrageous price. Kimiko's cracker was a man named Hiyose Ken'ichirou. He did not like his real name ever being used in public. Kimiko-chan offered him a lot of money, more than he could possibly want, just to keep his lips shut. He accepted. Then he went ahead and did it anyway, for no extra pay. Kimiko-chan was not entirely pleased with his services.
Hiyose Ken'ichirou was a middle-aged bachelor. He had grown up poor in the Quarter, but made good money now as a cracker. He lived in a shack somewhere out in the woods, ate rice and miso, and spent his spare time watching bad anime and thinking pornography. He had a grudge against the world because of it. He was fascinated by the stories of people like Kimiko, who apparently had so much more going on in their lives than he did, and he thought they must be leading secret lives of their own. He wanted to know what they were hiding.
The cracker sat in his shack and thought and thought and thought. Then he cracked the code. It was, of course, easy--the stupid human had left a trail of breadcrumbs all the way back to the Shining Path Academy. Ken'ichirou did not know who Kimiko-chan was, nor that she was human, nor that she was a cracker, nor much of anything else about the modern world. He thought she was some kind of spirit being, an angry spirit, because he'd seen things like her in his dreams. He had no concept of the difference between one human being and another. He had never been to school except for a day when he was twelve and had been forced to stay in because of a family dispute. He did not know what it meant to say "Ich bin ein queer cracker!" He had no real understanding of the English language, and he thought "like me" meant the same thing as "just like me."
He thought he could bargain with her, because he was desperate and she was pretty, and she seemed to be the only one in the world who would care what he had to say. He told her about his cracker's wages, and his miserable existence, and the miserable expectations heaped upon him by human and canine society. He asked her for a little love, just a little bit. She laughed in his face, and he thought that was the end of it. But she said, "No. I'm sorry, Ken'ichirou-san. I just can't help you."
He said, "You have to! You're the only one who can!" "No, really. You've made your choices, Ken'ichirou-san. You've chosen to be a square, and that's the way the world is now. It's better not to make waves." He was stunned into silence. She continued, "You can't just decide one day to hate your life, and then think maybe things will be better the next day. The world doesn't work that way."
"But you have to! You're just like me, you're--you're--"
She cut him off with a wave of her hand. "Oh, I'm sure I'm welcome to my opinion of cracker life, Ken'ichirou-san, but I really think you're just mistaken. I'm not the one who's disliking the world, you are. You've got it all wrong."
"You don't understand. You don't have to--"
"Yes, I do, Ken'ichirou-san. Just as you don't understand what it's like to be different. I'm sure you don't. You're just a square."
"No! That's not fair!"
She laughed again. "Fair? Fair? I don't think you understand what 'fair' means. You're not human, you know. You're really not human at all."
"Look, I don't care what you people are, or--or spirits, or seru, or whatever. I need--"
But she had already turned her back on him, and was walking away. He caught up to her, but she didn't seem inclined to stop and talk.
"Hey, hey, you can't--"
Her skirt was blowing in the breeze made by her rapid, unsteady steps, and she was a long way off already. He started to catch up to her, then stopped and stared. What was she doing? She seemed to have forgotten about him completely.
The ginger joneko was standing at the edge of a small clearing, arms raised above her head, palms out. The sun broke through the trees and glistened on her fur. Ken'ichi had never seen ginger joneko in the wild before, but he thought they looked like pictures he'd seen of mountain cats. This one was much larger than any picture he'd seen, and looked even more like a picture: tall and thin, with long limbs and a big head with a pointy ears and a forked tail. She was wearing a plaid uniform like a girl's, but she seemed to be much older than any girl Ken'ichi had ever seen. He gaped at her, and she smiled. "Don't be afraid," she said. "I'm not going to eat you. I never eat humans."
"What are you doing here? How did you get here? How long have you been here?"
"I'm not sure I can explain it in words that would make any sense. But I'm here now, that's the main thing. I think you meant a lot to me when you said you'd never leave me here alone, and you did. Thanks. And I'd like to stay here forever if that's okay."
"But--but you're a girl!"
She shrugged. "I think that only matters to humans. Anyway, I'm not exactly a girl. I'm a joneko. I don't look like this, you know. I look like a much more ordinary girl, but I am not. It's okay if you don't understand.
"You're really human! I can tell!"
"No, I'm not. But I can tell you're human too, even though you think you're not. You don’t wear a collar, for one thing."
"So you're a person who doesn't have to wear a collar because she doesn't have to obey anyone?"
"Exactly. There are joneko in this forest too, you know. They call me the Walking Ginger. I don’t know if that's my name or what. Anyway, it's a funny story--"
Ken'ichi wanted to hear the funny story, but he was afraid to interrupt. He wanted to hear it later too, but it seemed like she was in a storytelling mood. He waited.
"Okay. Long ago, in the time of the dinosaurs, there was a joneko queen who had a white coat and a big white hat and she could tame the very biggest and baddest of seru lions. One day she saw a young man, and she fell in love with him. But he didn't love her back, not in the way she wanted. He was just trying to impress someone with a show of strength, and he pushed her too far. She got her ears ripped off."
"That wasn't very nice."
"She cried a lot. We all saw it. But he was scared and confused and ashamed, and he didn't have a clue about what to do with a big beautiful queen with no ears and no pride in what she wears. He ran away and she cried a lot more. Then she crawled into a tree hole to die, because she was all alone in the world, and that's what joneko do."
"I don't like stories where animals do the things humans do."
She smiled. "That's the human thing to say. But if you try to tell that to a joneko, you'll just get a blank stare. I've seen humans do all kinds of things that seem like they must be a lot like what joneko would do, but you can't say that they really think like joneko at all. You have to understand how different we are."
"Okay, I'm listening."
"There was a time, long before there were joneko, when there were humanoids. You've seen their pictures. They were humanoids too, and they looked a lot like joneko.
"But they weren't like us at all. They didn't speak our language, they didn't think like we do, and they didn't feel the same kinds of emotions we do. They were just a different kind of life form, like joneko are a different kind of life form from you and me. You can see the differences even in a picture like this one. You look at it, and you don't see a picture of people, you see a picture of energy, or life force, or something. Just like joneko.
"There was one queen among the humanoids, she was the most powerful among them. They called her the Moon Queen. She was very beautiful and very powerful, but she was so lonely that she cried for hours at a time. One day she looked up and saw the Sun, but it was too bright and hot for that environment, and it burned her eyes. She cried even more. When she could see again, she wandered out into the desert, where she found a small cool place to sleep. There she found a man, a seru, and they made love. It was her first and only love. Afterward, he mocked her, and threw her out into the desert to die. She found a pool of water and drank too much of it, and that made her sick. She lay by the pool of water, waiting for death, but it did not come. The Moon Queen was very sick, but she was also very tired. She rose like a cobra, stretched her limbs, and looked around. Where was she? Why was she here? It was night already. She remembered the Sun, and the hot sand, and the man who threw her out. She looked in the direction he had come from, and saw a light. She walked toward it.
"I have lived many lives since then," she said, "but that one has been especially memorable. I have been many other queens too, and I know what it is to be lonely. Now I will be the Moon Queen again."
She changed into a black joneko queen, with white markings on her forehead and on her upper lip, her legs and her arms, and her ears. She looked exactly like a ginger joneko, except of course for the color and the human words written on her body in a language no human ever learned. The Queen stalked off into the forest.
Ken'ichirou woke up at some ungodly hour, stumbled out into the cool air, and looked around. The clearing was already full of joneko, all quiet and spooked. He wasn't sure what had woken him, maybe a noise from outside, or just the vibration from their bodies as they slept on the ground. He didn't think it was a good idea to get too close to the forest's edge, so he just stood and looked out at the world.
The Moon Queen came up behind him and laid her head on his shoulder. "It's morning," she said, quietly. "The joneko who wakes first will be first to eat. Come, walk with me."
She took him deep into the forest, far away from the others. They came to a place where the ground sloped down into a hollow. At the bottom was a pool of still water, perfectly clear except for a few small floating trees. Some distance below the surface, the water became a glossy dark purple, dotted with bright orange flashes of polyp. The Queen sank down on one knee by the pool, beckoning for Ken'ichirou to do the same. He obeyed. She tilted up his face and kissed him, long and slow. It felt good.
"My Queen," he said, "I am yours to command."
"Yes," she said, and giggled. "No. You're not." She bent over the pool, putting her face in the water. Then she straightened up, shook her head, and laughed again. "No, Ken'ichirou-san. You're not. You're much too square for me." She took a deep breath. "But I think I have a job for you."
"Anything, my Queen."
She looked at him thoughtfully. "I was just thinking," she said. "What if you could have anything you wanted?"
"Anything?"
"Yes. What if you could have any woman, or any man, or anything else you wanted, just by saying 'I wish so-and-so'?"
"I don't understand."
"I'll make it easier. What if I told you I could give you a hundred wishes, just for you and me, and you could give me one too, just by saying 'I want so-and-such'?"
Ken'ichi-san had no idea what to say to that. The Queen saw his confusion, and laughed. "Don't worry," she said. "You don't have to say anything. Just think about it."
She stood, and he followed suit. She took his hand and pulled him along a thin path between the trees. It seemed to go on forever. He tried to keep up, but she was much younger and much more agile than he, and soon left him far behind. He could hear her laughing merrily all the way.
They came to a fork in the path, and she paused to look at a large boulder on the right. It was covered with petroglyphs. One of them was scratched into the stone, and Ken'ichirou did not recognise the characters. The Queen read the story aloud, translating as she went.
"There were these two joneko, see? One was beautiful and the other was ugly. They didn’t have anything in common, nothing at all. The ugly joneko was so jealous of the beautiful one that she did anything to make her look ugly. She pulled out her own hair, she poisoned the pool, she carried water in a sieve because she thought that would make her look like a sieve. Nothing worked. Then one day, when she was doing her usual bit of nastiness, she forgot she was jealous of the beautiful joneko's looks, and instead of making her ugly, she killed her. The dead joneko was so beautiful that the other joneko didn't know what to do or think. She couldn't just leave the body there in the middle of the woods because that would make too much sense; so she dragged the dead girl's body to the edge of the forest and left it there. But she left some of the clothing, and some of the ornaments, from the body, because they looked so much like her own now, and there was no way to know just how much of the dead girl's blood really was in the water.
"The Moon was high and bright that night, and she saw the two joneko draped across each other's bodies on the rock, weeping bitterly. She asked them why they were so sad. One of them said that she had killed the other, but the other said no, it was the other one who had killed her. They argued and argued until the Moon became angry. She changed the subject of the conversation to the state of the world. The joneko who had spoken first said that maybe things were better in the old days, when the Sun was hot and bright and things were hard and shiny. The Moon was scornful of such speculation. She said that maybe things were better then, but it was the Darkest Hour now, and things were very bad indeed. The first joneko said that maybe bad things were good for the world, and made it strong. The Moon Queen said that maybe, just maybe, she could make things better if the joneko were willing to help. The joneko queen said that maybe she could do something else with the joneko's help. She said she could make them very, very happy. The Moon said that maybe she could, and the joneko queen said maybe she could too, and they laughed and laughed. Then they dove into the water and swam away, leaving the humanoids to wonder what strange things joneko might do next. That was the start of joneko civilization, see?"
She turned to go, but Ken'ichirō put a hand on her arm. "Hey," he said. "I'm sorry."
"Oh, it's okay."
"No, it isn't. I shouldn't have interrupted you--"
"No, you really did--" she began, but he interrupted her.
"Look, you were telling me about this story, and I didn't really want to hear it, but now I'm interested, and I want to know how it ends."
She smiled. "You really are a nice human."
"Yes, ma'am."
"Okay, then. The story ends with the Moon Queen and the joneko queen embracing and cuddling in the water while the humanoids look on. They think it's the end of the world, but it's really the start." She laughed. "It's all right, I guess. You don’t have to worry. The joneko have other fish to fry."
She walked off down the trail, leaving Ken'ichirou staring after her. She was almost to the point where the path split, when she stopped and looked over her shoulder.
"You want to know how it ends?" she asked. "Well, there's one thing you should know. We never really did get to eat the squid. When the Moon Queen and the joneko queen were finished with their business, they swam up to the shore, dove into the water, and never came back."
"Oh," said Ken'ichirou, who had been about to say something else, "That's too bad."
The Queen laughed. "No, actually it's kind of good. You see, we had to leave all our stuff there, and it's much better that we didn't get to eat the damn things. They tasted terrible."
She turned and walked down the branch of the trail, leaving Ken'ichirou standing on the rocks looking out at the ocean.
0 comments
