We spoke to Jodie Archer and Matthew Jockers, authors of the upcoming The Bestseller Code (out 20th September). Jodie bought and edited books for Penguin UK before decamping for the doctoral program in English at Stanford University. After her PhD, she worked at Apple as their research lead on literature. Matthew is Susan J. Rosowski Associate Professor of English at the University of Nebraska-Lincoln where he teaches and directs the Nebraska Literary Lab. His text mining research has been profiled in The New York Times, The LA Review of Books and more. We talked to them about how they became research partners, their research process and their favourite books.

1. Aside from being a literature-focused research team since 2010, you are both established in your respective fields. Can you tell us a little bit about what you did before starting work on the research behind The Bestseller Code, and how you each came to recognise a need for a research partner with a set of skills different to your own?

Jodie: I was born in the UK and did my undergraduate degree at Cambridge in English, concentrating on literary criticism and contemporary literature. That led me to a two-year trainee scheme at Penguin, which taught me the different aspects of the business. After that I became an editor in London. Most of the rest of our story is in the book: I got a scholarship to study in the US for six years, and did a PhD in English at Stanford, again concentrating on literature and literary criticism after 1960. Because of my interest in publishing, I wanted to focus on questions of literary taste, and most specifically to give some rare attention to the books thousands of living people actually read – bestsellers. Matt’s capabilities with text mining were the only way I could answer my intuitions and questions satisfactorily, namely “Are there things most bestsellers have in common, regardless of genre or even whether they are considered literary or mass market?” I was interested in the hidden patterns of authorship and readership – both how we could say something new about success in fiction writing and how we could take a snapshot of American taste in the moment. Matt was the only person who could have helped me with this project.

Matt: I earned a traditional PhD in English (1997) with an emphasis on Irish and Irish-American literature, but since the late 1980s I’d been very interested in both computers and quantification. These interests first came together during the 1990s when I was invited to work on several text digitisation projects as a graduate research assistant. After earning my degree, I continued to develop my programming chops, and my dual expertise in computing and literature led to a position at Stanford. At Stanford, I ran a little seminar/workshop series called “Beyond Search”. The goal of the series was to explore, in experimental ways, how the computational analysis of text could take us beyond simple searching of those texts. Jodie and I first met when she attended one of those meetings.

2. What was your central motivation behind setting out to identify a formula for a popular book? Do you think that this will have a positive effect on the commercial performance of books by new authors? How about self-publishing authors? Any other predictions for the publishing industry?
Jodie: I don’t think formula is the right word: there is no formula offered in the book and there could never be. We use 2,800 points of data to explain bestselling: while there are tips for writers definitely, it is not a “paint by numbers” approach. Neither of us would be quite as interested in writing a book like that, and we are happy that very diverse novels with different voices can still capture the latent DNA.

The original motivation for me was really to seek an answer to a question. Critics and scholars had always said that there were simply no patterns in bestsellers and that they were unpredictable. I didn’t think that was true. I wanted to show how fiction works from an additional perspective.

We hope that the book will help self-published writers, yes. We are interested in book discovery and the idea that we could help talented new writers to be discovered. But at this point, the data is not available to the industry or to individual writers beyond what is in the book.

3. The study was comprised of 20,000 books, analysed over the course of five years. Can you shed some light on the process you went through to amass relevant data? At what point is this data deemed meaningful?

We did two studies, one of 20,000 novels to test the theory, and then a second on a smaller corpus of contemporary bestsellers and other books that did not sell so well. That second study, represented in the book, had a corpus of just under 5,000 novels. Over time we developed the algorithm to include style, theme, then plot, setting, character and so on. At each stage we tested to see if the data was meaningful to our question. We analysed over 28,000 points of data from every novel, and in the end something like 2,799 were important or useful in distinguishing between a novel that had hit the NYT bestseller list and one that did not sell very well.

4. What findings from your research surprised you the most?

Jodie: Two happy surprises were that The Da Vinci Code and Fifty Shades of Grey – two massive sellers – had similar emotional rhythms. The similarities between their emotional “plot lines” that we show in the book were very compelling to me. The second was likely that The Circle by Dave Eggers was deemed to be the best demonstration we had of the moves involved in hitting the list. I had wondered if the algorithm would choose someone obvious like Stephen King, but the choice of what is called a more middlebrow novel in the UK or a more commercial novel in the US (as opposed to a genre novel) was a happy surprise. The algorithm picks those novels with both deep characters and well-wrought plots. Finally, I was surprised that setting doesn’t have that much influence. Yes, it matters if you set your book in a city, but not whether it is New York or Chicago or Frankfurt.

5. What are each of your favourite books, bestseller or otherwise?
Jodie: We talk in the last chapter about how anyone in books dreads that question! I have about two full bookcases just of my favourite books! If we just include contemporary writers, as a student my favourite novel for years was Margaret Atwood’s The Handmaid’s Tale. I have written so many essays about that book! Over the past few years of reading bestsellers for this research, there have been many brilliant reads. Hard to pick just one, but the latest one that had me addicted was Jojo Moyes’ Me Before You. I think that is a five-star bestseller.

Matt: The favourite book question, ugh…

I’ve spent a lifetime with books. I have many favourites. As a scholar I never get tired of reading and teaching Joyce’s Ulysses; honestly, I love the book, and I love it more every time I re-read it. But really, I don’t recommend it to folks, and I don’t wrap it up as a gift for someone’s birthday. Except for my wife – when we were dating she got a copy. My reasons for liking Ulysses are complicated and have a lot to do with things that likely have nothing to do with your question. So, maybe I should say Dostoevsky’s Crime and Punishment; I became an English major after reading Crime and Punishment. It’s the one that launched me on this path. Maybe that is my favourite book? Then again, as a kid, I read myself to sleep every night with Louis L’Amour – all those Westerns about the Sackett boys must have made some impression. So maybe one of those is my favourite? Then again, thinking about Westerns reminds me of how much I enjoyed Cormac McCarthy’s Blood Meridian. Blood Meridian is a book that I do recommend over and over again. It’s a great book, but it’s not the kind of book that you “fall in love” with. It’s not the sort of book that I can recommend without a long list of caveats and explanations. It’s no beach read. The favourite book question is complicated. As far as bestsellers go, I’ve liked a lot of them and all of them for different reasons. I totally enjoyed The Da Vinci Code; Alice Sebold’s The Lovely Bones is incredibly moving and I think that The Devil Wears Prada is a delightfully good read. So, what’s my favourite book? Ask me again tomorrow.