« LaTeXで文章の空白の植... | Home | The Pudding Fallacy »

Serializing Shining Path

Sun 13 Dec 2015 by mskala Tags used: , , ,

Shortly after I finished my PhD in 2008, I took about half a year off from other work, and wrote a 100,000-word science fiction thriller called Shining Path. That was half a year for the actual writing. It was a synthesis of notes and other material I'd been collecting for a number of years previously. I then spent the next three years or so looking for an agent to represent it.

I sent query letters to 40 agents, four of whom responded by asking to see the full or partial manuscript. That's a 10% rate of non-rejection responses to the initial query; the most trustworthy statistics I have are that the industry average for this number was 1% at the time, so it would appear that my queries were considerably better than average. All of the agents who asked for partials proceeded to request the full manuscript. One of them was from a big-name agency, with whom I hadn't thought I'd have any chance.

But none of those communications resulted in a representation deal. And at the end of that period, I decided to put the project on the shelf. Everybody who's written about the process of selling a book emphasizes that it's important to never ever give up, but I saw a number of reasons to stop throwing energy into the project. First, some of what I'd hoped to achieve by publishing the book could only have happened by publishing it quickly, and that was no longer possible. In particular, one possible outcome if it had sold quickly and done well in the market was that it could have saved me at least one (extremely stressful) academic job-search cycle.

Second, the fact that it didn't sell quickly was a pretty strong indicator that the manuscript just wasn't as good objectively as I thought. Shining Path remains one of my favourite novels, but my tastes are not everybody's. And so even if I did manage to sell it at some point, the benefits of doing so might not be all that great. Writing a novel that sells a few hundred copies is just as much work as writing one that sells tens of thousands; but the return on investment differs greatly between those two cases, for both author and publisher. If the author wants to make money (the publisher always wants to) then it's important to be in the high-sales category and not the low-sales category.

A related factor was that some of the topical content (I thought at the time) was closely tied to historical events of the late 2000s and would go out of date if I couldn't get it into the market very soon. In fact, I was wrong on that last point. I had actually anticipated issues that would become much more relevant circa 2013-2015 - but with the information available to me or anyone at the time, I think it was a reasonable analysis.

Third, the number of agents in the business (at all, but especially those willing to represent "genre" fiction like mine) isn't all that large and they do not turn over rapidly. After querying 40 of them I'd already really queried all the agents who were suitable. Trying to continue would mean sending my future queries almost entirely to agents who were not really a good match, and hoping that they would both be willing to look at my work despite its being outside their usual line, and assuming they liked it, that they'd unexpectedly turn out to be competent to represent it despite it being outside their specialities. I don't like those odds.

Probably the most decisive factor was that at that time the course of my life meant I was spending a whole lot of energy on approaching people who owed me nothing, running around knocking on doors to beg for favours that were more important to me than to them. That was what I was going through at that time both while looking for a job and while looking for a partner. I hate the activity of running around knocking on doors to beg. I wrote the book hoping that doing so could be part of a plan to change my life into one that would not feature that activity; it in fact coloured some of the characterizations in the text; if I could have had an agent it would have been the agent's job to shield me from running around knocking on doors and begging; and so the last thing I needed was to remain committed to a process that would mean more running around knocking on doors and begging with no guarenteed end to it.

I had set two targets, things I wanted to accomplish with the project, and one of them was that I wanted 10,000 people to read my book. That's ambitious for a first novel, but well within the reasonable range of possibility for a first novel that is actually good, is properly represented, sold to a mainstream publisher, and printed on paper. Having 10,000 readers is not within the reasonable range of possibility for a self-published book, and it wasn't (at the time) within the reasonable range of possibility for any kind of "e" book. Posting a book freely on the Net also won't in general get you 10,000 readers unless you are already for other reasons a famous author. There really seem to be more people willing to pay for a book from an unknown than to read it without paying. And so I did not seriously consider any of those options.

There is also the fact that people who self-publish have to self-promote. Self-publication is all about running around knocking on doors. So self-publication would mean a very great increase, not a decrease, in the amount of that activity in my life, exactly what I didn't want, and the only thing that could compensate would be not just the (already ambitious) level of success I had been hoping for with third-party publication, but also another order of magnitude on top of that to compensate for having to do self-promotion, at worse odds. So it was right off the table.

The manuscript has remained on my shelf through the years since then, and I'm starting to think it may be time to do something else with it. As I mentioned, it has become more topically relevant - and I don't know how long that will remain true. The world is continuing to change, and the standard publication model is crumbling. Already five years ago it was clear that the writing was on the wall for printed fiction, and "e" readers have only continued to become more and more popular. "E" books are now a serious option, albeit still with much lower sales numbers than real books. It's pushing it to think my manuscript could hit 10,000 on electronic publication alone even today, and the idea of doing that with electronic self-publication is even more implausible than before because of the much greater competition these days.

But the Shining Path manuscript does me no good at all just sitting on the shelf until it's completely outdated. Any course of action that cannot cost me much in the failure case, and has a significant chance of succeeding, seems better than just letting it evaporate. The trick is to construct a plan that can plausibly get at least 10,000 people to read the book, and does not require me to run around knocking on doors and begging.

Here's what I'm considering.

  • I post Chapter 1 here. There are a total of 42 chapters. I announce the posting on Twitter.
  • For integers n from 1 to 41, when the announcement of Chapter n has been retweeted floor(10^(4n/41)) times, I post Chapter n+1. And I make this plan known as a promise.

The exponential formula means that Chapters 2 through 5 go up after one retweet of the previous announcement each, but after that the requirement increases fairly rapidly. It's a 25.2% increase per iteration, although that factor is somewhat distorted when the numbers are small, due to the effects of integer rounding. Chapter 12 requires ten retweets of the Chapter 11 announcement; Chapter 22 requires a hundred retweets, 32 a thousand, and the final chapter only goes up when ten thousand people have retweeted the announcement of the penultimate chapter (with thousands of retweets of the other chapters in between, too). The number of chapters, the final goal of 10,000, and the formula specified above work out nicely so that it hits the numbers ten, one hundred, and one thousand exactly, along the way to 10,000.

The number of retweets of an announcement isn't the same as number of readers of the thing being announced. But it certainly helps. The total of 48599 retweets implied by the completion of this plan to cover all the announcements certainly means that there'd be some large if hard to measure number of readers for at least parts of the actual story, at least a few thousand. The actual number of Twitter users who see a retweet is much larger than the number who retweeted it. And having a large number of retweets has secondary effects that are also valuable in themselves - it would certainly attract a lot of general attention to other things I write.

How plausible is it? Well, ten thousand retweets is certainly well within the range of what the major celebrities of Twitter achieve. Nearly all of Justin Bieber's tweets hit this threshold; his median seems to be a few tens of thousands. A significant minority of Barak Obama's tweets go over 10,000; median might be in the low thousands. @ProBirdRights, which is a popular, though niche, comedy account, typically ranges between a few hundred and about 3,000, rarely spiking higher; they've probably gotten 10,000 once or twice but I didn't see any examples in a quick glance at their current timeline. The most famous people on Twitter whom I can legitimately call friends frequently get a few tens of retweets per tweet. I myself have occasionally hit that level on my most popular tweets, but my median is zero.

Of course, the "I will post the next chapter when I get Y number of retweets" thing distorts the entire analysis. How many people would choose to retweet an announcement on those terms is completely different from how many choose to retweet just any arbitrary tweet without the distorting incentive. What I foresee happening is that there is some audience of some size who would actually be interested in following the story, would retweet all the announcements, and join in the game of trying to encourage others to do so. I don't know how big that audience is, nor how effective its reach is. The level of attention would grow to encompass all of that audience, and then stagnate, as nobody else would care enough to participate. If it's less than 10,000, too bad; the postings just stop when we run out of new readers who are interested enough to continue.

And that's really the problem. If I commit to this course of action, it's a one-way street. The value of the manuscript to a traditional publisher plunges after it's been posted for free on the Net. The loyal readers are disappointed if they can't finish the book, though I guess I'd remain free to just throw in the towel and post the rest for them as a nice gesture, even if it doesn't hit its promotion goals. It might be important that I don't allow myself to appear so willing to do that, as to dissuade anybody from participating in the promotion.

There are a few people other than me who have copies of the entire manuscript already and could "leak" it, but a moment's thought should reveal why I'm not worried about that.

If the forced-viral promotion scheme actually works well, then I can reasonably expect agents and publishers to come to me instead of my running around knocking on their doors. At the 10,000 level or maybe even the 1,000 level, there's the possibility of selling it into the printed-book market despite its having been posted on the Net, because at those levels it has proven marketability. But if the audience tops out at about a hundred people, which seems like honestly the most likely possibility, then I'm stuck with about half the manuscript posted and not much way forward. That's still better than doing nothing, but there's the opportunity cost that then I no longer have the possibility of implementing some other plan in the future.

This project is easily interesting enough to my friends that I can expect two or three retweets for the first few chapters pretty much automatically without needing to ask for them. (I can even predict who those would be.) If I trade on my social capital asking my friends to retweet it, I might be able to hit 10 or so; good through Chapter 12. To take it to the next level, though, it'd have to be the case that enough strangers have read it by that point, and have liked it well enough, that it starts attracting third-party attention and people who choose to retweet it on the merits without my asking them personally.

I think the only plausible way it can get to the next level after that, would be if the reach gets big enough to attract attention from what might be called "big" players. If a few pros or semi-pros with multi-thousands readerships find out about it and think the project (the book itself, or the unusual promotion scheme, or both) is interesting enough to write about in their Web logs or whatever (NOT just retweeting an announcement but actually writing about it), that could carry it into the kind of propagation that would allow it to really finish on the stated terms. It comes down to a race between how much attention is needed to reach those "thought leaders," and how much attention is available in my local circle of basically unknown persons. I don't think it's a gimmie either way on which of those factors would win, but my inclination is toward pessimism.

There might be some danger of someone gaming the system - for instance, by paying for spam robots to generate a lot of fake retweets. That does cost money to do, but I don't think it costs so much as to be inconceivable. I could imagine that there might be one dedicated reader who loves the book more than they love me and would do that to ensure I posted it all, never mind that by doing so they'd destroy the objective of getting lots of real people to read it. It'd have nasty side effects, too, as Twitter would likely punish me for supposedly benefiting from spam even if I didn't ask for it and didn't want it to happen. I'm not sure what can be done about this kind of attack. But at least it's something that would cost the attacker real money, and that should filter out most of the random crazies.

I'd be interested to hear your thoughts on this proposal.


I like this proposal. The novel deserves to be released and this is an innovative way to release it.

I think the threshold formula should include terms for time since last chapter published and/or time since chapter 1 published. That way you could still release the novel while providing a strong incentive to get chapter announcements retweeted.

Since I have two drafts somewhere in my files, I would be tempted to leak chapters if they were taking too long to get posted. I would exercise final editorial control over the work though: I'd add more inexplicable romantic pairings, provide implausible explanations for the Deconstruction, throw in plenty of pseudo-science, and in general not reference the unpublished chapters in any way.
Gord - 2015-12-13 19:15
I agree with Gord that guaranteed releases are important (release them weekly unless there are enough tweets). You seem to think that 10k readers is optimistic and a pessimistic reader might decide not to read something that will never be completed. A plan to maximize the number of readers would probably generate more readers than this one targeted at 10k readers.

You may also need some social engineering so that people who are just starting to read the book at a slow rate tweet about chapter 35. Under your plan, if chapters don't cause tweetstorms then people may end up seeing promotions of chapter N over a lengthy period and become blind to notices about your book. Towards the end, will people think 'rt @mskala Does little Nell die? Ch 37 http://bit.ly/aesf35' is the work of a spam campaign?

The directed Twitter social graph intersected with people who will like your book might not even have 10k people reachable from you. Do people on Twitter even have an attention span that can cover that many characters? To reach your goal you will need magnification, which could be a single (well-connected) positive review. If you can get a few hundred readers part way through the book then having them all download a $1 Kindle copy on the same day might cause you to rank well enough to get your 10k sales (and remuneration).
Michael - 2015-12-13 19:43
I'm going to use the words "monkeys" and "bananas" here instead of "people" and "money," because that's more fun and makes just as much sense.

Gord: what sign should your proposed added terms have? It sounds like you have in mind that it should be negative (fewer retweets required if it's been a long time since the last release) but I'm not sure that it shouldn't actually be positive (more retweets required if it's been a long time). Getting 10,000 monkeys reading the book in a month is better than 10,000 readers in a year; although I'm stating the goal in terms of total number of readers, there is some degree to which the first derivative (rate at which number of readers increases) is also valuable. Having the target total *decrease* over time means treating a negative first derivative as contributing positively to the goal, which seems to be backwards. This is actually also a reason to prefer the exponential as the shape of the curve: all its derivatives and integrals of all orders are also exponentials, so doing that allows a sort of scale or order agnosticism - I'm not committing to whether I really care about number of monkeys or rate of new monkeys or what.

Michael: I think you're envisioning a slightly different measurement, the number of tweets about the book. I was thinking of measuring the specific number of retweets of the one chapter announcement tweet for each chapter. That tweet would probably be worded something like "Chapter 23 of Shining Path is out! (URL) Next chapter when this receives 158 RTs" The URL would point to a "table of contents" page with links to the first and latest chapters prominent at the top, intended to make it easy both for current and new readers. Any other discussion or sharing (on Twitter or elsewhere) may be desirable, but is not what's being measured. There are a number of reasons for doing it that way, one simply being that number of retweets of one specific tweet is something easily and objectively measurable - I and anyone else who cares can easily see the number and know whether it has reached the threshold. It reduces any suggestion of me or others tampering with the process. Retweets on Twitter in particular are also valuable compared to "likes" and such because they uniquely put the message in front of third parties, which is what it takes for the population of readers to grow; and unlike anything that goes on on Facebook, they're not beholden to a secret curation algorithm.

It also sounds like you're postulating a goal of maximizing the expected[*] number of monkeys. I am not; I am talking about maximizing the probability of at least 10,000 monkeys. A plan with 99% chance of 1,000 readers and 1% chance of 10,000 (1,100 in expectation) is not as good as one with 95% chance of zero readers and 5% chance of 10,000 (500 in expectation); the utility function is importantly concave upward in the relevant part of the domain. Although I prefer to talk about it as a "quantity premium," it'd also be reasonable to say that Web site promotion is very strongly "risk-seeking." I've written about that here before.

[*] I am using the word "expected" here in a technical math sense that basically means "average"; I'm sure Michael knows that, but to clarify for other readers with different background, it's not one of the other senses of "expected" in English that might have to do with what I'd deserve or demand.

Exponential growth means that at any time, some significant fraction of the monkeys seeing announcements have seen no, or very few, such announcements before. The average number of announcement tweets a current participant has seen is a constant, no matter how long the process continues, because as current participants see more tweets, they continue to be outnumbered by new participants. With the coefficients I stated, almost exactly half the population of retweeters counted do so for their first time on one of the last three announcements. (It's 5011 retweets for posting Chapter 39; then 40, 41, and 42 necessarily count at least 4989 who did not contribute to meeting that threshold). The "Oh, man, not another of these boring repetitive chapter-announcement tweets, I've seen dozens of these!" experience is only available to the tiny core who joined at the very beginning. That issue of losing existing readers so the total declines would be relevant if instead of growing exponentially, the process hit a plateau with a smaller population full saturated some time before the end, but if it hits any kind of plateau where it stops growing, or even if it grows at 10% instead of 25%, then the whole thing is doomed anyway (10% growth per chapter, with 1 at the start, means 49 readers at the end). At that point it doesn't matter whether I lose a few readers or lose them all.

The directed social network graph certainly contains a strongly connected component covering almost everybody who participates in social media in the world at all, and I am certainly part of that SCC. That's how social network graphs work; those simple facts don't require further research at this point. Intersecting the one large SCC with the set of monkeys interested in my book, the result is (near enough that we don't care about the difference) all monkeys interested in the book who participate in social media at all. It's not at all clear that 10,000 such monkeys exist in the world, and if they don't then there's nothing that can be done.

The possible world I'm more concerned about, because it's one where both success and failure are possible depending on my actions, would be the one where at least 10,000 monkeys interested in my book exist; then they are certainly reachable through the social network by the known properties of social networks; but the news about the book may or may not reach them. That's a question about how the news percolates through the social network. If I just tell my friends, most of them won't be interested and the few who are probably won't tell their friends, and similarly at each successive level. The overall percentages are too small for it to reach the entire network (as evidenced by the fact that Shining Path is *not* famous today). The point of the post-when-retweeted scheme is to increase the "chance someone who is interested will tell others" percentage enough that the whole thing percolates - that is, the news does in fact reach the entire population of interested monkeys. The properties of that percolation are an interesting math research question about which I should probably do more reading and maybe some simulation experiments.

Monkeys given the choice between waiting and spending bananas may choose to spend bananas. That is how "free to play" games manage to be incredibly successful, and so if I wanted bananas then a guaranteed-posting scheme would make sense. But I don't want bananas, I want to have 10,000 readers, and the only way I see for that to happen is if monkeys spend *social capital* by telling their friends. Social capital is the most valuable thing in the universe to a monkey. Given the choice between spending social capital, spending bananas, or waiting, they will certainly not spend social capital. So I fear that offering monkeys a chance to read the book without exponential growth occurring (for instance, in a guaranteed-posting scheme where postings can occur without an exponentially growing number of monkeys choosing to spend social capital), that will guarantee that exponential growth does NOT in fact occur. This is unfair because it means monkeys’ ability to achieve their own goals becomes conditional on something outside their individual control, but life isn’t fair, and that’s exactly the same unfairness I face every day.

This comment is getting too long already, but if I were to extend it I’d talk about Gresham’s Law: bad bananas drive out good, and you can expect that anything you sell will be paid for with the worst bananas you accept, if you offer a choice.
Matt - 2015-12-14 01:38
I understood what you meant for retweeting, I think that it is better to make your tweets focus on the story rather than the pyramid scheme. Someone seeing a sequence of 'Ch 12: In which the bonobo conspiracy is revealed' may be come intrigued; in the latter case the repeat observer will think about unfollowing your loyal reader (growth reaches new people, but the same uninterested people will be spammed with every chapter). Tell people why your book is worth reading with every tweet. Back when you made ads for your comic I think that they went to the most recent comic and the ad content struggled for relevance. It probably would have been better if you had separate campaigns for each theme (such as 'cat girls') the went to a landing page with your best work in that area or a search on that subtopic.

I also realized that your goal was based on hitting a certain number of readers. I believe that your scheme has a lower probability of reaching that goal than one that doesn't require thousands of people to have faith that you will reach that goal. Years ago, many of your potential readers were told 'This Rothfuss guy wrote a great book and the rest of the series has been finished so you won't be left hanging by him like that GRRM bastard' and have never recovered from it. Telling people you have an entire book and might let them read the end can result in people saying 'I don't want to start reading that; it would really suck if I got hooked by it'.

The people who have high out degree can't afford to plug you for any chapter other than the second to last and might still hold back for fear of a prank. The graph restricted to people who would read your work and then retweet according to the current scheme might have more than 10k monkeys in it, but is probably disjoint. OTOH, if Oprah likes your work and tweets 'everybody gets a free ebook', your mission is accomplished.

Andy Weir already implemented the best case scenario of going from long-finished webcomic to book and movie deal. Theft of Pride may still be dead in the water, however. Wish I asked him about that when I had the chance, but I think that I only put in the effort to get a copy because of The Martian (read because C&A was awesome).
Michael - 2015-12-14 13:24
Thanks for thinking it through carefully. I just don't understand what motive you think would cause a large number of monkeys to propagate something like this at all, if postings are guaranteed without their doing so. Merely because the book is good is evidently not enough - I *never* see text fiction propagated that way on social media, even though many eligible works exist and statistics guarantee that at least a few of them must be good. And the warm feeling of helping out seems only to be effective at at most one degree of separation from the original poster.
Matt - 2015-12-15 00:08
Oh, on the BoCon ads - my experience of clicking on ads for other comics was that if they went to a "landing page" instead of the current strip, that was almost a dealbreaker. It seemed to me to be very important to NOT pull that shit on my readers. YMMV, of course.
Matt - 2015-12-15 00:19
Referring to your readers are "monkeys" has pretty much doomed this approach from the start. It's not cute, it's condescending. People's time is their most precious resource, in this time of infinite information and entertainment available at one's fingertips. Not treating your readers with respect pretty much guarantees you won't be getting a piece of that pie.
Bo - 2015-12-15 05:18
I think you've completely missed the point of why I was doing that, but others may as well, so it's useful information.
Matt - 2015-12-15 05:24
Serial fiction normally leaves people wanting more at the end of each issue and shouting at boats for spoilers. If people don't want the next part right away then your story will never get to 10k readers; people will stop reading if there are long gaps. If a webcomic doesn't update weekly or faster and doesn't have an RSS feed, I will probably forget that it exists. Once stories span a year there is little hope that I will remember key plot points. The usual kudos are 'page turner', 'I couldn't put it down', 'I read it in one sitting', 'I stayed up all night reading it'. As an exception, What the Dog Saw seems to be an amazing book; I read one (self-contained) chapter at a time and set it aside for months to save more for later. For any other book, putting it down after a single chapter is death. (Will a kid really let you read just one chapter at bedtime?)

I don't think that large numbers of monkeys propagating is the solution (or a desirable mental image). I think that you need to attract the interest of well-connected individuals/automated systems (alpha geeks on social media, Kindle top sellers, Ars Technica) or make yourself part of a promotion (American Express was sponsoring free ebooks at YYZ, Humble Bundle Books, stretch goal to an existing KS campaign or whatever). Your proposed structure inhibits the monkeys and will not attract the well-connected.

If your only goal is to hit 10k readers, a Warhol worm that encrypts hard drives until victims read your book and answer reading comprehension questions would not only succeed, it would generate lots of free publicity.

As for ad campaigns, they are either so general that they aren't worth clicking on or so specific that your latest page is probably inapplicable. If I click on an ad for cat girls and there aren't any, my first thought will not be 'this fine gentleman payed good money to show me some cat girls; if they aren't on this page I should probably read his comic for from start to finish to find them'. If you have a long work, ads should probably go to self-contained stories that you are proud of (most people seem to hate their initial work) or otherwise answering needs. Sending me to the most recent page of something that only makes sense in context is asking for failure.
Michael - 2015-12-15 12:16
You're still describing reasons why it *won't* work *with* a firm requirement and incentive for propagation. I was more interested in reasons why it *will* work (or at least, might work) *without* that requirement and incentive, which I thought was the course you're advocating. Unless such reasons exist, it sounds like your position, and mine that it can't work without the requirement and incentive, do not contradict each other but together entail that it can't work at all.

The half-life of social media propagation is only a few hours (with some weird cyclic distortions because of day and night). The total number of sharing events since first posting isn't a linear increase that might take a week or a month to hit a threshold; it's a thing like the integral of exponential decay, that either hits the threshold almost immediately, or never does. If the last update is more than a week ago (really, two days) and the threshold hasn't been met yet, then the experiment is over. The idea that it might drag on long enough between updates for readers to forget about it, BUT that the whole thing wasn't over at that point anyway, isn't such a real possibility that it should outweigh considerations applicable to cases where continuing remains worthwhile.

What I'm getting from this discussion is that actually reaching the goal with a project of this type isn't a real enough possibility either way, with or without a firm requirement on propagation, to be worth pursuing further at this time.
Matt - 2015-12-16 00:27
Paul N
My uninformed opinion: if this publishing strategy will be fun and interesting for you regardless of the outcome, then go ahead. If not reaching your goal will make you sad or bitter, then please refrain.

There are more things I could write, but that is the most important thing I want to express.

I feel like a jerk saying so, but I would almost certainly not participate in such a social media campaign even though I enjoy your blog and think you are a brilliant human being.

Paul N - 2015-12-22 03:32
I strive to define goals in terms of objective facts.
Matt - 2015-12-22 05:25
What about the project? Any news on that?

By the way, I also happen to think that the exponential scheme that you designed would decrease the number of people willing to read. I, for one, would retweet, but refrain from reading until it is complete. Life is too short to read unfinished books (apart from Kafka's...)
Andrea - 2016-03-18 02:16
I'm not planning to pursue it further at the moment. The discussion here certainly doesn't lend any support to the proposition that it would work, and it's worse than useless if it doesn't work.
Matt - 2016-03-23 08:00

(optional field)
(optional field)
Answer "bonobo" here to fight spam. ここに「bonobo」を答えてください。SPAMを退治しましょう!
I reserve the right to delete or edit comments in any way and for any reason.