I’m back. And with much relief 🙂 (thank you for obliging me last week, friends. All is well. Sometimes understanding probabilities is not fun.)
I thought I’d talk about that lurking monster in the black lagoon, one that seems to terrify so many people…
For no good reason, he’s just misunderstood. Actually they’re a bit of an old soppy, really.
Here. Come and meet him. He does bite but only if you do something silly like run away, or approach him without his favorite titbit, logic. Give him logic and Numbers will be your friend. (Unlike Leviticus, who will at best be your ally.)
It is of course possible to get numbers to lie, or at least allow yourself to be fooled by them, especially if you don’t understand them. Lies, damned Lies and Statistics…
Still, as writers, particularly of sf and particularly those of us venturing into independent publishing, they’re important. Some of this is slowly getting through to publishing It is not too long a leap of logic to say that those numbers tell another story too: Losing bookstores is a much bigger problem for publishers than it is for _writers_ too. Look, the publishers were the gatekeepers… not to being published – that could be got around – but to retail space. Without a publisher, and without one that could exert push to get lots of retail customer eyeballs an author was doomed, it didn’t matter if the book was good, bad, or even brilliant. In fact mediocre could do fairly well, except that this generated the probability of diminishing return customers. As probability has a lot to do with my friend numbers, let me dwell on these, just briefly. I’m sorry if I am boring you with something almost all of you know, but I’ve had my nose rubbed in the fact that most publishers, and it seems a fair number of the grandees of SFWA do not even begin to understand concepts like ‘probability’, ‘ratio’, or ‘average’, let alone ‘normal Gaussian distribution’ or ‘cohort analysis’. Probability is ‘the odds’ of something happening. ie if you have a ten sided dice for your wargame, and it is equally weighted and properly shaken and thrown, you’re going to throw a 1 one time out of ten. You have a 0.1 or 10% probability of throwing a 1, the next time you throw you still have 0.1 chance, not a 1 in 9 chance. It does not mean if you buy kippers it will not rain.
If you’re a reader… You’ve all been there… finished the last page of a book, checked there isn’t one hidden somewhere, and read the damned appendix because you’ve enjoyed it so much, and either gone back and started it again, or rushed off to try and find another in same series or by the same author. That’s a probability of 1(or 100%, or 10:10) that you’ll buy another book by the same author/in the same world. The probability that you’ll buy more books of the same genre/setting/ series could all be calculated. And this is important – all of those figures differ for every reader of that book. Some books will get a very high average, and if you plotted all the probabilities on a graph, you’d get the typical distribution curve – a big fat and wide one if almost everyone liked it, and a narrow skinny one if only a small sector of the population does. So: if the Book A has a lot of appeal (approaching 1 that they’d buy another) to 90% of men, regardless of their orientation, and 50% of women, it’s going to sell a lot and generate a lot more sales of other books. If you looked at the curve you’ve got a big tortoise helmet covering a lot. This curve is ideal, but very, very rare. Book B has a lot of appeal (approaching 1) to Communist vegetarian Latvian lesbians — it’s going to sell well to its niche, but will in general terms not generate the sales of many other books. If you look at its curve, it’s a thin tall porcupine spike in one place. Kind of obvious… but what seems to be missed is that if a read is so-so… say a probability of 0.45 that I’d buy a book by that author or in that genre… AND it has narrow appeal base… (a thin short spike in one place) you’ll kill that market quite quickly. ESPECIALLY if you’re putting books for Communist vegetarian Latvian lesbians which don’t have a very high ‘buy by same author/genre’ with a low probability that even the target audience will buy… into bookstores as the only choice. In other words putting a short thin soft porcupine spot on place and telling everyone it is a tortoise. This is what publishers have been doing for years, because they could. It has been steadily losing readers from the underserved parts of the distribution.
Part of this comes down to straight arrogance and command economy. Part of it is ‘inclusivity’ It has become very politically correct to confuse diversity/inclusivity with ‘representative’
The latter has considerable commercial value, and makes a lot sense. The former only makes sense in politics (ie. None, really). The latter generates, as a by-product, quite a degree of ‘diversity’. If you have a hundred possible customers, and 50 are male, only selling Y front underpants and boxers means you lose 50 customers for sure… Selling a range of underwear is your best option. It costs quite a lot switch your manufacturing to something else… so if you have to choose, and 4% of your customers like G-strings and the rest somewhat more ordinary nickers, if you decide to be diverse and have just as much invested in G-strings as everything else… you’re diverse. And broke. If you decide diverse means getting rid of the offensive-to-G-string-wearers Y front and boxer line to concentrate on the G-strings… you’re what seems to be the SFWA idea of ‘inclusive’. This has been a trend in traditional publishing (SFWA reflects the mainstream of that, which is why it is what it is) What does the monster numbers suggest your fate will be as a big flabby beast in the nasty world of commerce, with a few short, soft little spines in one spot, and maybe a few long thin ones, also in the same spot, running around shrilly yelling ‘I’m a tortoise, I’m a truly inclusive tortoise?’
It’s not pretty. Which is why indy is steadily eating large traditional publishing’s lunch, because they might have narrow appeal, but at least their porcupine spikes are all over the place, and they’re getting audiences that the ‘inclusive’ couldn’t reach, by being closer to ‘representative’. The end result, say some pundits is that those writers who have large web audiences will take over where publishers left off, once they lost the gatekeeper to retail space job.
And it is perfectly possible that Amazon will, once it has a lock on retail space, do what other large retailers have done once in that position, and screw suppliers and tell customers: “We only have size 11 left boots. Take it (at this inflated price) or leave it.”
Or not. The one thing Amazon, alone among the book retailers (and publishers) has done is to embrace numbers. The poor creature from the black lagoon has been much loved and is eagerly providing them with a lot of marketing data. We’re still at the early stages now, but I’ve seen a little of the sort of data-use that is getting used in commerce (and, interestingly, in Australian political planning) where they’re down to the street and house level of analysis. They know what the probabilities are on damn near everything. When they target a product at an area, they have a strong idea just how to target. You don’t write or read sf, if you don’t realize, probably a lot sooner than you think, the product recommendations you’ll be getting will be really well tailored to your tastes and likes – far more precisely than a recommendation by a popular blogger. And it won’t be reaching the popular blogger’s 5000, or 50 000. It’ll reach every person with a computer. And their likes are not alike. Reaching _all_ of your potential real hardcore fans is a lot more effective than reaching a bigger group of eyeballs which don’t all buy, and certainly don’t all love and put on their must-buy this author list. A narrow interest group but very tall probability spike (always buy the next Kate Paulk or Tom Kratman, and who knows, even the Communist Vegetarian Latvian Lesbian’s book) is probably enough to support a lot more authors than now
Maybe the advice that the great advisors to the publishing industry should be giving is that they need to learn to love Numbers. Not the Bookscan level GIGO (garbage in, garbage out) but the real McCoy. Find out what the market actually is and cater to giving it what it wants. But to do that, you can’t just guess. Or be PC and ‘inclusive’. They need to start collecting worthwhile data and get professionals to help you to understand it, and to target the customers. The thing is right now, that sort of targetting requires a lot of data input, a lot of analysis, and huge amount of computing power. Only a large organization could do it… now. But that too will change. I can see a targetting of product widget being available to writers in say… 20 years.
And that’s enough numbers for one day.
Mad Genius Club 
35 responses to “Numbers”
“Only a large organization could do it… now.”
Well, Amazon has that whole Cloud thing, and its just… sittin there…
(btw your post the other week made me go look you up… and a few minutes later, i had Dragon Ring and Dog And Dragon, both… just sayin..)
You have just discovered the one writer in SF whose worth is most seriously undervalued market wise. Enjoy.
yeah, i dug through both in a night each, sound familiar? 😀
Yes. I hoard my Freer and only read him when I can afford to lose the night 😉
As my daughter puts it,
“I know that feel.”
🙂
Yeah, there is no doubt the clud is 1)going to be used, 2)going to be raided.
Dragons… Ah good. The plague spreads 😉 Pass it on.
I like your tortoise helmet and porcupine quill image so well I simply must steal it.
I note that porcupine quills may throw off enough money to keep an indie writer in beer and pretzels, but a big publisher and/or a bookstore chain needs a steady supply of tortoises to keep the lights on.
Perhaps indie writers will serve as a farm team wherein many scribblers produce a mix of dreck, porcupines, and tortoises. The challenge for the big publisher will be finding tortoises and approaching their authors with suitcases of money. Devising a discriminator should be interesting: in small samples tortoises will look more like dreck than like porcupines.
I think you’ve hit on a problem that publishing simply has no tools to deal with at the moment: It’s relatively easy to pick a pick a porcupine quill- especially if the narrow part of the demographic it appeals to matches editor tastes closely. Given that the overwhelming bulk – outside of Baen – of traditional publishing editors are from similar backgrounds and socio-political strata, that ends up with very samey picks, even if they are not trying to educate the hoi polloi. Picking a story with broad spread of appeal -or even stories with a spread of porcupine spikes means getting it sampled by a range – not just probably female, East Coast, urban, liberal arts graduate, left to extreme left political doctrine… and if you happen to be one of the above, you’re finding many great reasons why this is a stupid idea, before power and quite possibly your job is lost.
First, thank you for giving me hives – I’m still allergic to regressions after having survived the Lotke-Volterra population analyses in Intro to Ecosystem Biology (talk about your weeder courses!)
Second, great Torah pun. Third, yes, and it’s interesting to look at what is selling in history as compared to sci-fi. The publishers are still selling non-PC history quite well, while blatently PC histories are not being pushed as hard as they were a few years ago. Granted, we’re comparing apples and artichokes, but I almost wonder if the editors at the major houses caught on that they need more turtles, and so have been pushing turtles instead of single-quill porcupines.
I don’t know if it’s that far off a comparison TX — look, years ago, in an all-sf writers list, we found most of us were reading popular history BECAUSE WE COULDN’T FIND ANYTHING TO READ IN OUR FIELD. I.e. the field that first seduced us, drew us in, and made us so ardent we wanted to write it, was not just not hospitable, it was actively pushing us out. this group, btw, was “diverse” and left leaning, but I found it amusing their reasons for not reading were the same as mine “it’s not fun” even as they cranked out “worthy” books. (There’s a problem with confusing a market stall with a soap box.)
So your comparison is not as out — people move between genres when one is becomes inhospitable, even that far out. Also, for the record and weirdly, I’m getting more short story invites than they ever got. If I were willing, I could sell a short story a week — and I would if I had time — and not just to Baen. I thought coming out of the political closet would give me leprosy — and I still cringe but again the blog is how I keep from making my books long screeds. I KNOW better, but it’s not wholly under my control — but instead I seem to be hot-in-demand-flavor. Which also scares me a little, but that’s something different.
This. I got away from sci-fi some years ago, once I’d worked myway through Heinlein and Pournelle, Drake and Weber et al. After that, most things looked pretty grim in comparison. And not in the interesting, grimdark and fascinating kind of way- GFHH grim. No fun.
The something different I and several of my friends who read are looking for is pretty simple. Engaging, fascinating stories that don’t talk down to us. Sure we all have our niche interests, and I get the fantasy/urban fantasy bug every now and again. But good storytelling will always sell to whichever market it fits. And for a good while, “good storytelling” wasn’t very common on my local bookstore’s shelves.
Also, regarding numbers, I thought many of those publishing house giants of the industry went to private schools and such? I know public school mathematics has suffered in quality, but what excuse have they for not following the numbers?
Dan, I’m not sure about their schooling, as this is so not my set of people. I have a feeling you’ll find they scraped through on rote rather than comprehension, left that for the arts side as soon as possible, forgot the indignity of it as quickly as they could… I’m pretty sure however that the chants of ‘la la la’ are very loud there. I think a lot of them suspect at least that there maybe problems, but they’re kind of like someone who won’t see a doctor and get tests done, because they’d rather go on kidding themselves that all is well, even if they’re losing weight and throwing up every time they eat. That’s normal human behavior to some extent anyway, but when it goes into business – or maintaining heavy machinery, firearms or scuba tanks, it should be a fire-able offence.
You’ll find good storytelling is our core credo here on MCG. I can recommend all of my fellow writers here. The Baen site is worth a look in too.
Barbara’s rule: Thou shall not not ecourage punsters!;-) (I’m always delighted when readers pick up the puns, subtext, plays on words. And she’s right. It makes me worse.)
I suspect the history comparison has one curious corrolary – I bet the editors are also from the same demographic group that make up NY’s sf/fantasy editors. So they’re finding some tool to at least guess what non-PC history will sell. (one has to wonder how accurate that is, but it’s something, especially if it outsells the hard-push)
It’s a truism in process improvement circles that “if you can’t measure it, you can’t fix it.”
Bravo! encore! And you don’t even know if it is broken – which it may not be – unless you measure it. And do it properly, understanding what you do, because yes, you can cherry-pick and massage figures. But in the medium term that will almost inevitably destroy your business.
Hence the opposition to standardized testing.
“They’d only teach to the test!” they cry.
“Well at least then they’d be teaching them SOMETHING.” I’d answer.
Meanwhile–over in software engineering land–there’s a related warning: “Be very careful what you choose to measure. That choice will greatly influence what you attempt to fix… for good or ill.”
This is of great concern for SE, because we’re still trying to figure out which metrics and interventions are (a) relevant, (b) helpful, (c) worthwhile, (etc., etc.)
and it’s much easier to measure what readers have purchased, than what they might purchase (especially if it’s in a different porcupine quill than anything you have data on)… part of the numbers-to-marketing problem then, is how to use marketing (including pricing) to generate more samples.
But the question is, do you really want to find it and fix it? Run into that in my field (process engineering) all the time. There is a problem, but spending the time and resources to find it may not match what management wants to focus on today.
Yep, got it in one. There is huge amount of ‘we don’t want to fix it, so we won’t measure it, so we can pretend it is all fine or a minor issue. The only problem with that attitude is… what if it isn’t (and I think we can fairly safely say:it isn’t)?
American systems are trying to use it politically too, but there are limits to it, as proven by the fact that a certain faction of a certain political party hounded us before the elections — specifically hounded ME — and the chances of my voting for them were slightly lower than those of my growing wings and flying. But eh.
On the rest, agree with you — I’m hearing amazing things from friends who have been at this indie game a few years longer than I. if I can stay alive with the work…
Statistical analysis can never be perfect, but it doesn’t need to be. It just needs to be sufficiently good to have a better sales return on investment than broad based marketing.
That US political faction that wasted resources on you, after all, did end up winning the election 😦 .
thank you for saying it for me :-). That’s the point with probabilities. They are not necessarily right about one individual. Nor does an incorrect guess on those probabilities prove anything. “There is a 95% probability that a Chinese person will have an IQ of over 107…. but General Cash Mi Chek (extra points for anyone who remembers the source of this) is as stupid as a stump… therefore it is wrong!” Well, no. He’s just the 5%. And it is possible (a 5% chance) – not probable (a 95% chance), that the next person (Li Tin Wheedle, I believe) is also dim (or stoned). Just like throwing the 10 sided dice the odds do not change with each individual throw. They’re an individual and this is a population probability. On the other hand, with an infinite improbability drive… you can get lynched by a rampaging mob of respectable physicists.
On the other other hand, you can also get a sperm whale and a bowl of petunias from a pair of guided nuclear missiles, so it’s something of a wash.
Probability works best with accurate data points- for books, I’d say sales, for elections, votes. Given how the political process works in this country, they are heavily invested in the math, but have unreliable data until the final votes are counted. Much simpler (and faster!) to have the data on what a given reader has bought in the past to predict what he may buy next.
For politics… I just consider the game rigged when one side says “give us votes, we give you stuff. Like money. From those other guys.” And the other side says “vote for us, we’re not as bad. We waste money slower.” *facepalm* Some elections, I’d say we are approaching distinction without difference.
(apologies for the digression).
In the end, I can only write the Characters who choose to show up in my head. And I can only cast them in Stories where they belong …. Maybe those stories will become turtles, in which case I get to retire to a beautiful house in TX without any near neighbors. But I don’t know how to write whatever it may be “they” want from me ….
A prolific writer of nice sharp porcupine quills can do as well as an occasional writer of turtles. If you write ten books in two years and make ten thousand a piece off them, the only difference from writing one book in two years and making a hundred grand; is your word count.
Oh and the fact that your readers are happier than the readers of the one turtle every two years author.
Yes, A porcupine can do very reasonably — IF the quill is available to and known about by the right people. If you (true story) put vast numbers of Michael Moore books on sale in a Red state, largely rural population bookstore, it won’t sell.
Actually it’s more a case of aiming THE MARKETING at the right place, rather than the author aiming their writing.
“Communist vegetarian Latvian lesbians” . . .
I think you’ve just defined my next niche market writing project. Except that I’m going to have them wielding assegais and knobkerries. While wearing karosses.
🙂
🙂 I’m sure the agents and publishers from NYC are beating a path to your door as we speak.
Very good article, Yes Sarah, I will have to check out Kate’s books. Vox recommended you, you recommend Kate, and my collection grows. Dave’s books already grace my shelves. To the point, I see the author/publishing/reader field more like the American expansion. First the thinly scattered natives (publishers/bookstores) not really advancing in technological civilization but making large sale in narrow fields. Like Indians burning a large grassland to drive buffalo over a cliff. Run out of buffalo. In come the trapper/traders called ‘Indies’ followed by the General Store for pioneers and squatters. (Amazon and other Internet book sites, including Ebay) which brings us up to today- I can only see the Independent writers plus Amazon and its copies continuing to grow and the bookstore moving into the back seat (moving to reservations) I don’t think any of us can see what this is going to morph into in a few years. I think it is going to be very interesting though.
You’re assuming that people are predictable. They aren’t. You’re also ignoring the bread and butter of anyone who really wants to make it as an independent – the books that only a few people read, and those few people want more. As for the rest, kiddo, dream on.
“You’re assuming that people are predictable. They aren’t.” The gospel according St Ellis the Ancient.
Oh I didn’t know that! Are you sure? I mean absolutely sure? Well, I shall rush off and tell those immensely profitable and successful businesses, who actually base their entire success on predicting what people (not an individual, people in general) will or won’t buy. I’ll tell them Ellis Lonharno says you are all dead wrong and must immediately change to the ‘system’ publishing uses. Where they’re surviving by selling the family silver, some of which doesn’t actually belong to them. I’m sure they’ll be even more impressed than I was. Trot out your credentials for making this startling insight, Ellis, so I can tell them where your expertise comes from. As for ‘ignoring bread and butter’ have any real what you’re talking about, in dollars and cents? Like, have you done this? Got the figures? Got the figures from a reasonable sample? Or is is this just a miraculous insight?
Wherever you normally hang out it might give you street cred to come and casually dismiss argument and treat the host of the discussion with disrespect. Here we expect grown-up behavior. You’re welcome to dissect, try to show why or where I am wrong. You’re even welcome to snark about stupid. I’m capable of stupid, and have the balls to admit that I have been and been wrong. But the behavior you’ve just displayed will get a real swat next time, not just a gentle tap. Got that, kiddo?
Amazing restraint. :o)
Years of practice. We note he hasn’t come back. Strange. 😉